istio-Multi Cluster issue with ingress gateway

swetha · September 30, 2019, 10:24am

Hi all,

I am working on setting up istio in a multi cluster environment following the link below

but i am unable to bring up istio ingress gateway on remote cluster
where describe ingress gateway pod give “Readiness probe failed: HTTP probe failed with statuscode: 503” and logs has "2019-09-30T10:23:59.853775Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-09-30T10:24:01.853643Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-09-30T10:24:03.854045Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-09-30T10:24:05.853654Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-09-30T10:24:07.854168Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-09-30T10:24:09.854165Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-09-30T10:24:11.853729Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-09-30T10:24:13.853751Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-09-30T10:24:15.853950Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
[2019-09-30 10:24:16.905][22][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
"
can someone please help me here.

Thanks,
Swetha

edgarHz · October 7, 2019, 3:10pm

I’m having this same issue.
I’ll also appreciate any help.

ericvn · October 7, 2019, 4:12pm

I know I’ve worked successfully through this example many times over the last week using Istio 1.2.6 and 1.3.1 on IKS clusters. I’m not an expert, but I’m thinking maybe there was a problem in the n2-k8s-config or with the LOCAL_GW_ADDR. When I first started working this example many months ago I hit similar issues and it was usually something in that area of the example.

edgarHz · October 7, 2019, 5:53pm

Thanks, Eric.

I’ll go through the example again and report back.
BTW, I’m using minikube, so I’m remapping some ports in the “remote” cluster for the pilot, telemetry and tracing services using the “targetPort” attribute. I’m not sure if this is OK.

Is there any pod logs that would help to further diagnose if I don’t succeed?

ericvn · October 7, 2019, 6:57pm

I’m not sure about using MiniKube and remapping ports. I’ve heard on calls that it should work, but not sure if there are differences with the current instructions. I probably used Minikube as one of the two clusters during some early work when Istio would run on free IKS clusters so that could be my second.

I’m not sure what logs to really check. Maybe one of the networking people can chime in. It just looks to me like the ingress gateway in the second cluster can’t talk to Pilot in the first cluster. The Pilot IP is gotten early in the second step. The secrets are later as well. You could maybe exec into the ingress and try to ping the pilot IP (not sure what’s installed inside that container).

edgarHz · October 9, 2019, 5:10pm

Hi, Eric

I’ve retried the setup with the same results.

After finishing the setup (start watching cluster 2) I see these logs in Pilot in the main cluster:

2019-10-09T16:57:38.709976Z	info	Processing add: istio-system/n2-k8s-secret
2019-10-09T16:57:38.714401Z	info	Adding new cluster member: n2-k8s-config
2019-10-09T16:57:38.715559Z	info	Service controller watching namespace "" for services, endpoints, nodes and pods, refresh 1m0s
2019-10-09T16:57:38.715709Z	info	Number of remote clusters: 1
gc 21 @1256.426s 0%: 0.008+20+0.039 ms clock, 0.017+1.3/5.4/15+0.079 ms cpu, 9->10->4 MB, 10 MB goal, 2 P
2019-10-09T16:57:38.916886Z	info	Handle service istio-pilot in namespace istio-system
2019-10-09T16:57:38.917458Z	info	Handle service istio-policy in namespace istio-system
2019-10-09T16:57:38.917636Z	info	Handle service kubernetes-dashboard in namespace kube-system
2019-10-09T16:57:38.917687Z	info	Handle service istio-ingressgateway in namespace istio-system
2019-10-09T16:57:38.917754Z	info	Handle service istio-sidecar-injector in namespace istio-system
2019-10-09T16:57:38.918016Z	info	Handle service istio-telemetry in namespace istio-system
2019-10-09T16:57:38.918075Z	info	Handle service kubernetes in namespace default
2019-10-09T16:57:38.918116Z	info	Handle service kube-dns in namespace kube-system
2019-10-09T16:57:38.918165Z	info	Handle service istio-citadel in namespace istio-system
2019-10-09T16:57:38.918231Z	info	Handling event add for pod kubernetes-dashboard-7b8ddcb5d6-2s5jz in namespace kube-system -> 172.17.0.2
2019-10-09T16:57:38.918295Z	info	ads	Label change, full push 172.17.0.2 
2019-10-09T16:57:38.918381Z	info	Handling event add for pod istio-ingressgateway-76679b7b4f-f8msj in namespace istio-system -> 172.17.0.6
2019-10-09T16:57:38.918425Z	info	ads	Label change, full push 172.17.0.6 
2019-10-09T16:57:38.918503Z	info	Handling event add for pod nginx-ingress-controller-5d9cf9c69f-zm79t in namespace kube-system -> 172.17.0.5
2019-10-09T16:57:38.918849Z	info	Handling event add for pod kube-apiserver-minikube in namespace kube-system -> 10.0.2.15
2019-10-09T16:57:38.918942Z	info	ads	Label change, full push 10.0.2.15 
2019-10-09T16:57:38.919043Z	info	Handling event add for pod istio-citadel-56fd64bf48-2ckt6 in namespace istio-system -> 172.17.0.7
2019-10-09T16:57:38.919328Z	info	ads	Label change, full push 172.17.0.7 
2019-10-09T16:57:38.919424Z	info	Handling event add for pod kube-proxy-t6bqk in namespace kube-system -> 10.0.2.15
2019-10-09T16:57:38.919455Z	info	ads	Label change, full push 10.0.2.15 
2019-10-09T16:57:38.919537Z	info	Handling event add for pod coredns-5c98db65d4-qmdbt in namespace kube-system -> 172.17.0.4
2019-10-09T16:57:38.919563Z	info	ads	Label change, full push 172.17.0.4 
2019-10-09T16:57:38.919592Z	info	Handling event add for pod kube-controller-manager-minikube in namespace kube-system -> 10.0.2.15
2019-10-09T16:57:38.919655Z	info	ads	Label change, full push 10.0.2.15 
2019-10-09T16:57:38.919685Z	info	Handling event add for pod etcd-minikube in namespace kube-system -> 10.0.2.15
2019-10-09T16:57:38.920238Z	info	ads	Label change, full push 10.0.2.15 
2019-10-09T16:57:38.920362Z	info	Handling event add for pod istio-sidecar-injector-56d8c7bf94-qsj9t in namespace istio-system -> 172.17.0.8
2019-10-09T16:57:38.920393Z	info	ads	Label change, full push 172.17.0.8 
2019-10-09T16:57:38.920422Z	info	Handling event add for pod coredns-5c98db65d4-42jkp in namespace kube-system -> 172.17.0.3
2019-10-09T16:57:38.920698Z	info	Handling event add for pod storage-provisioner in namespace kube-system -> 10.0.2.15
2019-10-09T16:57:38.920835Z	info	ads	Label change, full push 10.0.2.15 
2019-10-09T16:57:38.920918Z	info	Handling event add for pod kube-addon-manager-minikube in namespace kube-system -> 10.0.2.15
2019-10-09T16:57:38.920942Z	info	ads	Label change, full push 10.0.2.15 
2019-10-09T16:57:38.920969Z	info	Handling event add for pod kube-scheduler-minikube in namespace kube-system -> 10.0.2.15
2019-10-09T16:57:38.921035Z	info	ads	Label change, full push 10.0.2.15 
2019-10-09T16:57:38.921072Z	info	Handle EDS endpoint kube-scheduler in namespace kube-system -> []
2019-10-09T16:57:38.921115Z	info	Handle EDS endpoint kubernetes in namespace default -> [192.168.99.116]
2019-10-09T16:57:38.921202Z	info	Handle EDS endpoint kube-dns in namespace kube-system -> [172.17.0.3 172.17.0.4]
2019-10-09T16:57:38.921336Z	info	Handle EDS endpoint kubernetes-dashboard in namespace kube-system -> [172.17.0.2]
2019-10-09T16:57:38.921497Z	info	Handle EDS endpoint istio-pilot in namespace istio-system -> [192.168.99.115]
2019-10-09T16:57:38.921605Z	info	Handle EDS endpoint istio-telemetry in namespace istio-system -> [192.168.99.115]
2019-10-09T16:57:38.921646Z	info	Handle EDS endpoint kube-controller-manager in namespace kube-system -> []
2019-10-09T16:57:38.921977Z	info	Handle EDS endpoint istio-ingressgateway in namespace istio-system -> []
2019-10-09T16:57:38.922041Z	info	Handle EDS endpoint istio-citadel in namespace istio-system -> [172.17.0.7]
2019-10-09T16:57:38.922109Z	info	Handle EDS endpoint istio-sidecar-injector in namespace istio-system -> [172.17.0.8]
2019-10-09T16:57:38.922159Z	info	Handle EDS endpoint istio-policy in namespace istio-system -> [192.168.99.115]
2019-10-09T16:57:39.022624Z	info	ads	Push debounce stable[10] 29: 100.366287ms since last change, 103.622935ms since last push, full=true
2019-10-09T16:57:39.025444Z	info	ads	XDS: Pushing:2019-10-09T16:57:39Z/9 Services:11 ConnectedEndpoints:1
2019-10-09T16:57:39.029306Z	info	ads	Cluster init time 3.806076ms 2019-10-09T16:57:39Z/9
2019-10-09T16:57:39.029606Z	info	ads	Pushing router~172.17.0.7~istio-ingressgateway-b9c48469c-nwv7b.istio-system~istio-system.svc.cluster.local-4
2019-10-09T16:57:39.032665Z	info	ads	CDS: PUSH for node:istio-ingressgateway-b9c48469c-nwv7b.istio-system clusters:74 services:11 version:2019-10-09T16:57:39Z/9
2019-10-09T16:57:39.036232Z	info	ads	EDS: PUSH for node:istio-ingressgateway-b9c48469c-nwv7b.istio-system clusters:72 endpoints:72 empty:[]
2019-10-09T16:57:39.036426Z	info	ads	LDS: PUSH for node:istio-ingressgateway-b9c48469c-nwv7b.istio-system listeners:0
2019-10-09T16:57:44.950898Z	info	ads	Push Status: {
    "ProxyStatus": {}
}

IP of the main cluster is 192.168.99.115. IP of the “remote” one is 192.168.99.116.
In Pilot logs I can see only one line with the IP of the remote cluster:

2019-10-09T16:57:38.921115Z	info	Handle EDS endpoint kubernetes in namespace default -> [192.168.99.116]

which makes me believe that Pilot is communicating with the remote cluster. But ingress gateway in the remote cluster is still having issues and logging these messages:

[2019-10-09 17:08:19.172][16][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
2019-10-09T17:08:19.949873Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected

As you said, if someone from networking can help, I will appreciate it.

edgarHz · October 9, 2019, 10:45pm

UPDATE:

Now ingressgateway in the “remote” cluster is reporting as ready. I deleted “clusterIP: none” attributes from the Service resources of istio-telemetry, istio-pilot and istio-policy. By removing these attributes, the port mappings are now working and communication seems to happen.

After creating the sample apps, I can see logs in Pilot indicating that the services of the remote cluster are being registered:

2019-10-09T22:36:22.895744Z	info	ads	ADS: "127.0.0.1:56636" router~172.17.0.7~istio-ingressgateway-b9c48469c-rrxmc.istio-system~istio-system.svc.cluster.local-21 terminated rpc error: code = Canceled desc = context canceled
2019-10-09T22:36:22.914141Z	info	ads	ADS: "127.0.0.1:56636" sidecar~172.17.0.14~sleep-69c766786-vz8dl.sample~sample.svc.cluster.local-22 terminated rpc error: code = Canceled desc = context canceled
2019-10-09T22:36:22.917337Z	info	ads	ADS: "127.0.0.1:37100" sidecar~172.17.0.9~helloworld-v2-85bc988875-wvhgp.sample~sample.svc.cluster.local-23 terminated rpc error: code = Canceled desc = context canceled
2019-10-09T22:31:22.893432Z	info	ads	LDS: PUSH for node:helloworld-v2-85bc988875-wvhgp.sample listeners:34
2019-10-09T22:31:22.894953Z	info	ads	ADS:CDS: REQ 127.0.0.1:56636 router~172.17.0.6~istio-ingressgateway-76679b7b4f-r476z.istio-system~istio-system.svc.cluster.local-24 459.24µs version:2019-10-09T22:21:21Z/19
2019-10-09T22:31:22.896158Z	info	ads	CDS: PUSH for node:istio-ingressgateway-76679b7b4f-r476z.istio-system clusters:78 services:13 version:2019-10-09T22:21:21Z/19
2019-10-09T22:31:22.897135Z	info	ads	EDS: PUSH for node:helloworld-v2-85bc988875-wvhgp.sample clusters:38 endpoints:38 empty:[]
2019-10-09T22:31:22.899642Z	info	ads	EDS: PUSH for node:istio-ingressgateway-76679b7b4f-r476z.istio-system clusters:76 endpoints:76 empty:[]
2019-10-09T22:31:22.903451Z	info	ads	LDS: PUSH for node:istio-ingressgateway-76679b7b4f-r476z.istio-system listeners:0
2019-10-09T22:31:22.906654Z	info	ads	RDS: PUSH for node:helloworld-v2-85bc988875-wvhgp.sample routes:14
2019-10-09T22:31:22.902010Z	info	ads	LDS: PUSH for node:sleep-69c766786-vz8dl.sample listeners:34
2019-10-09T22:31:22.912159Z	info	ads	RDS: PUSH for node:sleep-69c766786-vz8dl.sample routes:14

I see there are some “context canceled” logs. Not sure if this is right.

The sidecars in the remote cluster take some time to become ready, but they do. Now, the issue is that the traffic doesn’t passthrough when pinging the demo services:

kubectl exec --context=$CTX_CLUSTER1 -it -n sample -c sleep $(kubectl get pod --context=$CTX_CLUSTER1 -n sample -l app=sleep -o jsonpath='{.items[0].metadata.name}') -- curl helloworld.sample:5000/hello
upstream connect error or disconnect/reset before headers. reset reason: connection failure

I see this in the sidecar of the workload trying to connect:

[2019-10-09 22:31:22.390][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-10-09T22:34:04.315Z] "GET /hello HTTP/1.1" 503 UF,URX "-" "-" 0 91 31 - "-" "curl/7.64.0" "32af88f7-399f-4d75-a010-819ffdbd1ec3" "helloworld.sample:5000" "192.168.99.118:31390" outbound|5000||helloworld.sample.svc.cluster.local - 10.105.138.247:5000 172.17.0.14:53352 - default
[2019-10-09T22:34:51.089Z] "GET /hello HTTP/1.1" 503 UF,URX "-" "-" 0 91 28 - "-" "curl/7.64.0" "807f08c1-7611-444b-9426-cb51ff5a8e9d" "helloworld.sample:5000" "192.168.99.118:31390" outbound|5000||helloworld.sample.svc.cluster.local - 10.105.138.247:5000 172.17.0.14:53834 - default

And in the istio-ingress svc of the remote cluster, I see these messages are logged each few minutes:

[2019-10-09 22:19:48.456][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
[2019-10-09 22:20:18.436][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 14, upstream connect error or disconnect/reset before headers. reset reason: connection failure
[2019-10-09 22:26:21.995][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-10-09 22:31:22.403][15][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,

Any other suggestions? I think I’m close to have it working

edgarHz · October 11, 2019, 5:24pm

Today, when re-starting my minikube env, I saw it worked for a moment and, then stopped working. These are the logs in the sidecar for that very brief moment it worked:

2019-10-11T16:07:30.400390Z	info	Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-10-11T16:07:32.402739Z	info	Envoy proxy is ready
[2019-10-11T16:16:14.903Z] "GET /hello HTTP/1.1" 200 - "-" "-" 0 60 136 136 "-" "curl/7.64.0" "031144f0-7820-4883-9fbc-8e2422d3fdee" "helloworld.sample:5000" "192.168.99.118:31390" outbound|5000||helloworld.sample.svc.cluster.local - 10.105.138.247:5000 172.17.0.12:39500 - default
[2019-10-11T16:16:30.022Z] "GET /hello HTTP/1.1" 200 - "-" "-" 0 60 168 167 "-" "curl/7.64.0" "b206933e-8250-454b-bb25-b42e218362e0" "helloworld.sample:5000" "192.168.99.118:31390" outbound|5000||helloworld.sample.svc.cluster.local - 10.105.138.247:5000 172.17.0.12:39658 - default
[2019-10-11 16:18:02.622][16][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13, 
[2019-10-11T16:20:11.250Z] "GET /hello HTTP/1.1" 503 UF,URX "-" "-" 0 91 44 - "-" "curl/7.64.0" "0fdc640b-c87c-4e4b-aa3d-5a24b975ea2d" "helloworld.sample:5000" "192.168.99.118:31390" outbound|5000||helloworld.sample.svc.cluster.local - 10.105.138.247:5000 172.17.0.12:41868 - default

Once it stopped working, it didn’t recover and traffic don’t go through.

Given that it briefly worked, I’m not sure if the troubleshooting page is will bring any insight. Anyway, when walking through the troubleshooting, these are the results:

$ istioctl proxy-status
NAME                                                   CDS        LDS        EDS        RDS          PILOT                            VERSION
helloworld-v2-85bc988875-wvhgp.sample                  SYNCED     SYNCED     SYNCED     SYNCED       istio-pilot-86bdbfbc55-l8hx2     1.3.1
istio-ingressgateway-76679b7b4f-r476z.istio-system     SYNCED     SYNCED     SYNCED     NOT SENT     istio-pilot-86bdbfbc55-l8hx2     1.3.1
istio-ingressgateway-b9c48469c-rrxmc.istio-system      SYNCED     SYNCED     SYNCED     NOT SENT     istio-pilot-86bdbfbc55-l8hx2     1.3.1
sleep-69c766786-jjjpv.sample                           SYNCED     SYNCED     SYNCED     SYNCED       istio-pilot-86bdbfbc55-l8hx2     1.3.1
sleep-69c766786-sbnr5.sample                           SYNCED     SYNCED     SYNCED     SYNCED       istio-pilot-86bdbfbc55-l8hx2     1.3.1

Fetching Envoy config:

$ istioctl proxy-config cluster -n sample sleep-69c766786-jjjpv
SERVICE FQDN                                              PORT      SUBSET         DIRECTION     TYPE
BlackHoleCluster                                          -         -              -             STATIC
InboundPassthroughClusterIpv4                             -         -              -             ORIGINAL_DST
PassthroughCluster                                        -         -              -             ORIGINAL_DST
helloworld.sample.svc.cluster.local                       5000      -              outbound      EDS
istio-citadel.istio-system.svc.cluster.local              8060      -              outbound      EDS
istio-citadel.istio-system.svc.cluster.local              15014     -              outbound      EDS
istio-galley.istio-system.svc.cluster.local               443       -              outbound      EDS
istio-galley.istio-system.svc.cluster.local               9901      -              outbound      EDS
istio-galley.istio-system.svc.cluster.local               15014     -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       80        -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       443       -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       853       -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       8060      -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       15004     -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       15011     -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       15020     -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       15029     -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       15030     -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       15031     -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       15032     -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       15443     -              outbound      EDS
istio-ingressgateway.istio-system.svc.cluster.local       31400     -              outbound      EDS
istio-pilot.istio-system.svc.cluster.local                8080      -              outbound      EDS
istio-pilot.istio-system.svc.cluster.local                15010     -              outbound      EDS
istio-pilot.istio-system.svc.cluster.local                15011     -              outbound      EDS
istio-pilot.istio-system.svc.cluster.local                15014     -              outbound      EDS
istio-policy.istio-system.svc.cluster.local               9091      -              outbound      EDS
istio-policy.istio-system.svc.cluster.local               15004     -              outbound      EDS
istio-policy.istio-system.svc.cluster.local               15014     -              outbound      EDS
istio-sidecar-injector.istio-system.svc.cluster.local     443       -              outbound      EDS
istio-sidecar-injector.istio-system.svc.cluster.local     15014     -              outbound      EDS
istio-telemetry.istio-system.svc.cluster.local            9091      -              outbound      EDS
istio-telemetry.istio-system.svc.cluster.local            15004     -              outbound      EDS
istio-telemetry.istio-system.svc.cluster.local            15014     -              outbound      EDS
istio-telemetry.istio-system.svc.cluster.local            42422     -              outbound      EDS
kube-dns.kube-system.svc.cluster.local                    53        -              outbound      EDS
kube-dns.kube-system.svc.cluster.local                    9153      -              outbound      EDS
kubernetes-dashboard.kube-system.svc.cluster.local        80        -              outbound      EDS
kubernetes.default.svc.cluster.local                      443       -              outbound      EDS
mgmtCluster                                               15020     mgmt-15020     inbound       STATIC
prometheus.istio-system.svc.cluster.local                 9090      -              outbound      EDS
prometheus_stats                                          -         -              -             STATIC
sleep.sample.svc.cluster.local                            80        -              outbound      EDS
sleep.sample.svc.cluster.local                            80        http           inbound       STATIC
xds-grpc                                                  -         -              -             STRICT_DNS
zipkin                                                    -         -              -             STRICT_DNS

Envoy routes. I see there is no route to the “helloworld” service. But since that service is in the remote cluster, I’m not sure if it’s OK if it’s not listed:

$ istioctl proxy-config routes -n sample sleep-69c766786-jjjpv
NOTE: This output only contains routes loaded via RDS.
NAME                                                          VIRTUAL HOSTS
80                                                            4
5000                                                          2
8060                                                          2
8080                                                          2
9090                                                          2
9091                                                          3
9901                                                          2
15004                                                         3
15010                                                         2
15014                                                         7
istio-telemetry.istio-system.svc.cluster.local:42422          1
kubernetes-dashboard.kube-system.svc.cluster.local:80         1
istio-ingressgateway.istio-system.svc.cluster.local:15020     1
kube-dns.kube-system.svc.cluster.local:9153                   1
inbound|80|http|sleep.sample.svc.cluster.local                1
inbound|80|http|sleep.sample.svc.cluster.local                1
                                                              1

Envoy endpoints. In this case, helloworld is present:

$ istioctl proxy-config endpoints -n sample sleep-69c766786-jjjpv
ENDPOINT                 STATUS      OUTLIER CHECK     CLUSTER
10.106.98.215:15011      HEALTHY     OK                xds-grpc
127.0.0.1:80             HEALTHY     OK                inbound|80|http|sleep.sample.svc.cluster.local
127.0.0.1:15000          HEALTHY     OK                prometheus_stats
127.0.0.1:15020          HEALTHY     OK                inbound|15020|mgmt-15020|mgmtCluster
172.17.0.10:8080         HEALTHY     OK                outbound|8080||istio-pilot.istio-system.svc.cluster.local
172.17.0.10:15010        HEALTHY     OK                outbound|15010||istio-pilot.istio-system.svc.cluster.local
172.17.0.10:15011        HEALTHY     OK                outbound|15011||istio-pilot.istio-system.svc.cluster.local
172.17.0.10:15014        HEALTHY     OK                outbound|15014||istio-pilot.istio-system.svc.cluster.local
172.17.0.11:9091         HEALTHY     OK                outbound|9091||istio-telemetry.istio-system.svc.cluster.local
172.17.0.11:15004        HEALTHY     OK                outbound|15004||istio-telemetry.istio-system.svc.cluster.local
172.17.0.11:15014        HEALTHY     OK                outbound|15014||istio-telemetry.istio-system.svc.cluster.local
172.17.0.11:42422        HEALTHY     OK                outbound|42422||istio-telemetry.istio-system.svc.cluster.local
172.17.0.13:53           HEALTHY     OK                outbound|53||kube-dns.kube-system.svc.cluster.local
172.17.0.13:9153         HEALTHY     OK                outbound|9153||kube-dns.kube-system.svc.cluster.local
172.17.0.14:53           HEALTHY     OK                outbound|53||kube-dns.kube-system.svc.cluster.local
172.17.0.14:9153         HEALTHY     OK                outbound|9153||kube-dns.kube-system.svc.cluster.local
172.17.0.15:80           HEALTHY     OK                outbound|80||sleep.sample.svc.cluster.local
172.17.0.2:9090          HEALTHY     OK                outbound|80||kubernetes-dashboard.kube-system.svc.cluster.local
172.17.0.4:443           HEALTHY     OK                outbound|443||istio-sidecar-injector.istio-system.svc.cluster.local
172.17.0.4:15014         HEALTHY     OK                outbound|15014||istio-sidecar-injector.istio-system.svc.cluster.local
172.17.0.5:443           HEALTHY     OK                outbound|443||istio-galley.istio-system.svc.cluster.local
172.17.0.5:9901          HEALTHY     OK                outbound|9901||istio-galley.istio-system.svc.cluster.local
172.17.0.5:15014         HEALTHY     OK                outbound|15014||istio-galley.istio-system.svc.cluster.local
172.17.0.6:80            HEALTHY     OK                outbound|80||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:443           HEALTHY     OK                outbound|443||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:853           HEALTHY     OK                outbound|853||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:8060          HEALTHY     OK                outbound|8060||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15004         HEALTHY     OK                outbound|15004||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15011         HEALTHY     OK                outbound|15011||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15020         HEALTHY     OK                outbound|15020||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15029         HEALTHY     OK                outbound|15029||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15030         HEALTHY     OK                outbound|15030||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15031         HEALTHY     OK                outbound|15031||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15032         HEALTHY     OK                outbound|15032||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15443         HEALTHY     OK                outbound|15443||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:31400         HEALTHY     OK                outbound|31400||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.7:8060          HEALTHY     OK                outbound|8060||istio-citadel.istio-system.svc.cluster.local
172.17.0.7:15014         HEALTHY     OK                outbound|15014||istio-citadel.istio-system.svc.cluster.local
172.17.0.8:9090          HEALTHY     OK                outbound|9090||prometheus.istio-system.svc.cluster.local
172.17.0.9:9091          HEALTHY     OK                outbound|9091||istio-policy.istio-system.svc.cluster.local
172.17.0.9:15004         HEALTHY     OK                outbound|15004||istio-policy.istio-system.svc.cluster.local
172.17.0.9:15014         HEALTHY     OK                outbound|15014||istio-policy.istio-system.svc.cluster.local
192.168.99.117:8443      HEALTHY     OK                outbound|443||kubernetes.default.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15004||istio-policy.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15004||istio-telemetry.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15010||istio-pilot.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15011||istio-pilot.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15014||istio-citadel.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15014||istio-pilot.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15014||istio-policy.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15014||istio-sidecar-injector.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15014||istio-telemetry.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15020||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15029||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15030||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15031||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15032||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|15443||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|31400||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|42422||istio-telemetry.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|443||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|443||istio-sidecar-injector.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|443||kubernetes.default.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|5000||helloworld.sample.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|53||kube-dns.kube-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|8060||istio-citadel.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|8080||istio-pilot.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|80||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|80||kubernetes-dashboard.kube-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|80||sleep.sample.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|9091||istio-policy.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|9091||istio-telemetry.istio-system.svc.cluster.local
192.168.99.118:31390     HEALTHY     OK                outbound|9153||kube-dns.kube-system.svc.cluster.local

I’m not showing the listeners. I’m not sure if it’s meaningful.

I’m not pasting the bootstrap info, because envoy is properly communicating with Pilot. So I think that should be OK.

Testing connectivity to Istio Pilot: On the main cluster all is OK. On the “remote” cluster, I get:

root@istio-ingressgateway-76679b7b4f-r476z:/# curl http://istio-pilot:8080/debug/edsz
curl: (7) Failed to connect to istio-pilot port 8080: Connection refused

However, port 8080 is not exposed in IngressGateway. So, I think in the remote cluster this test it not valid.

Envoy version:

$ kubectl exec -it -n sample sleep-69c766786-jjjpv -c istio-proxy pilot-agent request GET server_info
{
   "version": "dc3aafe38c5af924462e09e471a8a8804c0395f5/1.12.0-dev/Clean/RELEASE/BoringSSL",

I don’t fully understand all these data. Most of it looks OK to me.

Ankit_Chopra · February 17, 2020, 10:17am

Hey,

We also tried to build the same setup and we witnessed the similar issue. Istio-Ingress doesnt work in the remote cluster unless we explicitly creates the VS and Gateway in the cluster with control plane.

So in order to make the ingress controller of remote cluster to work, we have to create gateway, virtual service and service entry of a particular service of remote cluster in the cluster with control plane.
Once we create these resources in the Control Plane cluster, the Ingress-gateway of Remote Ingress controller starts working.

Hence anyone is able to figure out the solution for the same.

Topic		Replies	Views
Services in remote istio cluster in a multi cluster istio setup are not accessible	2	1732	November 13, 2019
Istio ingress & egress gateway readiness probe failed in pure IPv6 kubernetes Cluster [Istio 1.6.8] Networking	0	611	October 9, 2020
istio-Ingressgateway pod is failing with 503 error Networking	1	1923	June 25, 2019
Multicluster installation - cross-cluster routing connection failure Networking	4	1096	March 10, 2021
Istio Pods fail Readiness Probes Kiali	0	819	March 9, 2020

istio-Multi Cluster issue with ingress gateway

Related topics