Today, when re-starting my minikube env, I saw it worked for a moment and, then stopped working. These are the logs in the sidecar for that very brief moment it worked:
2019-10-11T16:07:30.400390Z info Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
2019-10-11T16:07:32.402739Z info Envoy proxy is ready
[2019-10-11T16:16:14.903Z] "GET /hello HTTP/1.1" 200 - "-" "-" 0 60 136 136 "-" "curl/7.64.0" "031144f0-7820-4883-9fbc-8e2422d3fdee" "helloworld.sample:5000" "192.168.99.118:31390" outbound|5000||helloworld.sample.svc.cluster.local - 10.105.138.247:5000 172.17.0.12:39500 - default
[2019-10-11T16:16:30.022Z] "GET /hello HTTP/1.1" 200 - "-" "-" 0 60 168 167 "-" "curl/7.64.0" "b206933e-8250-454b-bb25-b42e218362e0" "helloworld.sample:5000" "192.168.99.118:31390" outbound|5000||helloworld.sample.svc.cluster.local - 10.105.138.247:5000 172.17.0.12:39658 - default
[2019-10-11 16:18:02.622][16][warning][config] [bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:87] gRPC config stream closed: 13,
[2019-10-11T16:20:11.250Z] "GET /hello HTTP/1.1" 503 UF,URX "-" "-" 0 91 44 - "-" "curl/7.64.0" "0fdc640b-c87c-4e4b-aa3d-5a24b975ea2d" "helloworld.sample:5000" "192.168.99.118:31390" outbound|5000||helloworld.sample.svc.cluster.local - 10.105.138.247:5000 172.17.0.12:41868 - default
Once it stopped working, it didn’t recover and traffic don’t go through.
Given that it briefly worked, I’m not sure if the troubleshooting page is will bring any insight. Anyway, when walking through the troubleshooting, these are the results:
$ istioctl proxy-status
NAME CDS LDS EDS RDS PILOT VERSION
helloworld-v2-85bc988875-wvhgp.sample SYNCED SYNCED SYNCED SYNCED istio-pilot-86bdbfbc55-l8hx2 1.3.1
istio-ingressgateway-76679b7b4f-r476z.istio-system SYNCED SYNCED SYNCED NOT SENT istio-pilot-86bdbfbc55-l8hx2 1.3.1
istio-ingressgateway-b9c48469c-rrxmc.istio-system SYNCED SYNCED SYNCED NOT SENT istio-pilot-86bdbfbc55-l8hx2 1.3.1
sleep-69c766786-jjjpv.sample SYNCED SYNCED SYNCED SYNCED istio-pilot-86bdbfbc55-l8hx2 1.3.1
sleep-69c766786-sbnr5.sample SYNCED SYNCED SYNCED SYNCED istio-pilot-86bdbfbc55-l8hx2 1.3.1
Fetching Envoy config:
$ istioctl proxy-config cluster -n sample sleep-69c766786-jjjpv
SERVICE FQDN PORT SUBSET DIRECTION TYPE
BlackHoleCluster - - - STATIC
InboundPassthroughClusterIpv4 - - - ORIGINAL_DST
PassthroughCluster - - - ORIGINAL_DST
helloworld.sample.svc.cluster.local 5000 - outbound EDS
istio-citadel.istio-system.svc.cluster.local 8060 - outbound EDS
istio-citadel.istio-system.svc.cluster.local 15014 - outbound EDS
istio-galley.istio-system.svc.cluster.local 443 - outbound EDS
istio-galley.istio-system.svc.cluster.local 9901 - outbound EDS
istio-galley.istio-system.svc.cluster.local 15014 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 80 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 853 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 8060 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15004 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15011 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15020 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15029 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15030 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15031 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15032 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 31400 - outbound EDS
istio-pilot.istio-system.svc.cluster.local 8080 - outbound EDS
istio-pilot.istio-system.svc.cluster.local 15010 - outbound EDS
istio-pilot.istio-system.svc.cluster.local 15011 - outbound EDS
istio-pilot.istio-system.svc.cluster.local 15014 - outbound EDS
istio-policy.istio-system.svc.cluster.local 9091 - outbound EDS
istio-policy.istio-system.svc.cluster.local 15004 - outbound EDS
istio-policy.istio-system.svc.cluster.local 15014 - outbound EDS
istio-sidecar-injector.istio-system.svc.cluster.local 443 - outbound EDS
istio-sidecar-injector.istio-system.svc.cluster.local 15014 - outbound EDS
istio-telemetry.istio-system.svc.cluster.local 9091 - outbound EDS
istio-telemetry.istio-system.svc.cluster.local 15004 - outbound EDS
istio-telemetry.istio-system.svc.cluster.local 15014 - outbound EDS
istio-telemetry.istio-system.svc.cluster.local 42422 - outbound EDS
kube-dns.kube-system.svc.cluster.local 53 - outbound EDS
kube-dns.kube-system.svc.cluster.local 9153 - outbound EDS
kubernetes-dashboard.kube-system.svc.cluster.local 80 - outbound EDS
kubernetes.default.svc.cluster.local 443 - outbound EDS
mgmtCluster 15020 mgmt-15020 inbound STATIC
prometheus.istio-system.svc.cluster.local 9090 - outbound EDS
prometheus_stats - - - STATIC
sleep.sample.svc.cluster.local 80 - outbound EDS
sleep.sample.svc.cluster.local 80 http inbound STATIC
xds-grpc - - - STRICT_DNS
zipkin - - - STRICT_DNS
Envoy routes. I see there is no route to the “helloworld” service. But since that service is in the remote cluster, I’m not sure if it’s OK if it’s not listed:
$ istioctl proxy-config routes -n sample sleep-69c766786-jjjpv
NOTE: This output only contains routes loaded via RDS.
NAME VIRTUAL HOSTS
80 4
5000 2
8060 2
8080 2
9090 2
9091 3
9901 2
15004 3
15010 2
15014 7
istio-telemetry.istio-system.svc.cluster.local:42422 1
kubernetes-dashboard.kube-system.svc.cluster.local:80 1
istio-ingressgateway.istio-system.svc.cluster.local:15020 1
kube-dns.kube-system.svc.cluster.local:9153 1
inbound|80|http|sleep.sample.svc.cluster.local 1
inbound|80|http|sleep.sample.svc.cluster.local 1
1
Envoy endpoints. In this case, helloworld is present:
$ istioctl proxy-config endpoints -n sample sleep-69c766786-jjjpv
ENDPOINT STATUS OUTLIER CHECK CLUSTER
10.106.98.215:15011 HEALTHY OK xds-grpc
127.0.0.1:80 HEALTHY OK inbound|80|http|sleep.sample.svc.cluster.local
127.0.0.1:15000 HEALTHY OK prometheus_stats
127.0.0.1:15020 HEALTHY OK inbound|15020|mgmt-15020|mgmtCluster
172.17.0.10:8080 HEALTHY OK outbound|8080||istio-pilot.istio-system.svc.cluster.local
172.17.0.10:15010 HEALTHY OK outbound|15010||istio-pilot.istio-system.svc.cluster.local
172.17.0.10:15011 HEALTHY OK outbound|15011||istio-pilot.istio-system.svc.cluster.local
172.17.0.10:15014 HEALTHY OK outbound|15014||istio-pilot.istio-system.svc.cluster.local
172.17.0.11:9091 HEALTHY OK outbound|9091||istio-telemetry.istio-system.svc.cluster.local
172.17.0.11:15004 HEALTHY OK outbound|15004||istio-telemetry.istio-system.svc.cluster.local
172.17.0.11:15014 HEALTHY OK outbound|15014||istio-telemetry.istio-system.svc.cluster.local
172.17.0.11:42422 HEALTHY OK outbound|42422||istio-telemetry.istio-system.svc.cluster.local
172.17.0.13:53 HEALTHY OK outbound|53||kube-dns.kube-system.svc.cluster.local
172.17.0.13:9153 HEALTHY OK outbound|9153||kube-dns.kube-system.svc.cluster.local
172.17.0.14:53 HEALTHY OK outbound|53||kube-dns.kube-system.svc.cluster.local
172.17.0.14:9153 HEALTHY OK outbound|9153||kube-dns.kube-system.svc.cluster.local
172.17.0.15:80 HEALTHY OK outbound|80||sleep.sample.svc.cluster.local
172.17.0.2:9090 HEALTHY OK outbound|80||kubernetes-dashboard.kube-system.svc.cluster.local
172.17.0.4:443 HEALTHY OK outbound|443||istio-sidecar-injector.istio-system.svc.cluster.local
172.17.0.4:15014 HEALTHY OK outbound|15014||istio-sidecar-injector.istio-system.svc.cluster.local
172.17.0.5:443 HEALTHY OK outbound|443||istio-galley.istio-system.svc.cluster.local
172.17.0.5:9901 HEALTHY OK outbound|9901||istio-galley.istio-system.svc.cluster.local
172.17.0.5:15014 HEALTHY OK outbound|15014||istio-galley.istio-system.svc.cluster.local
172.17.0.6:80 HEALTHY OK outbound|80||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:443 HEALTHY OK outbound|443||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:853 HEALTHY OK outbound|853||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:8060 HEALTHY OK outbound|8060||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15004 HEALTHY OK outbound|15004||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15011 HEALTHY OK outbound|15011||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15020 HEALTHY OK outbound|15020||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15029 HEALTHY OK outbound|15029||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15030 HEALTHY OK outbound|15030||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15031 HEALTHY OK outbound|15031||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15032 HEALTHY OK outbound|15032||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:15443 HEALTHY OK outbound|15443||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.6:31400 HEALTHY OK outbound|31400||istio-ingressgateway.istio-system.svc.cluster.local
172.17.0.7:8060 HEALTHY OK outbound|8060||istio-citadel.istio-system.svc.cluster.local
172.17.0.7:15014 HEALTHY OK outbound|15014||istio-citadel.istio-system.svc.cluster.local
172.17.0.8:9090 HEALTHY OK outbound|9090||prometheus.istio-system.svc.cluster.local
172.17.0.9:9091 HEALTHY OK outbound|9091||istio-policy.istio-system.svc.cluster.local
172.17.0.9:15004 HEALTHY OK outbound|15004||istio-policy.istio-system.svc.cluster.local
172.17.0.9:15014 HEALTHY OK outbound|15014||istio-policy.istio-system.svc.cluster.local
192.168.99.117:8443 HEALTHY OK outbound|443||kubernetes.default.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15004||istio-policy.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15004||istio-telemetry.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15010||istio-pilot.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15011||istio-pilot.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15014||istio-citadel.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15014||istio-pilot.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15014||istio-policy.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15014||istio-sidecar-injector.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15014||istio-telemetry.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15020||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15029||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15030||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15031||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15032||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|15443||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|31400||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|42422||istio-telemetry.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|443||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|443||istio-sidecar-injector.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|443||kubernetes.default.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|5000||helloworld.sample.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|53||kube-dns.kube-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|8060||istio-citadel.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|8080||istio-pilot.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|80||istio-ingressgateway.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|80||kubernetes-dashboard.kube-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|80||sleep.sample.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|9091||istio-policy.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|9091||istio-telemetry.istio-system.svc.cluster.local
192.168.99.118:31390 HEALTHY OK outbound|9153||kube-dns.kube-system.svc.cluster.local
I’m not showing the listeners. I’m not sure if it’s meaningful.
I’m not pasting the bootstrap info, because envoy is properly communicating with Pilot. So I think that should be OK.
Testing connectivity to Istio Pilot: On the main cluster all is OK. On the “remote” cluster, I get:
root@istio-ingressgateway-76679b7b4f-r476z:/# curl http://istio-pilot:8080/debug/edsz
curl: (7) Failed to connect to istio-pilot port 8080: Connection refused
However, port 8080 is not exposed in IngressGateway. So, I think in the remote cluster this test it not valid.
Envoy version:
$ kubectl exec -it -n sample sleep-69c766786-jjjpv -c istio-proxy pilot-agent request GET server_info
{
"version": "dc3aafe38c5af924462e09e471a8a8804c0395f5/1.12.0-dev/Clean/RELEASE/BoringSSL",
I don’t fully understand all these data. Most of it looks OK to me.