Hi,
i have a simple cluster with 3 masters and 2 worker nodes, and i have enabled automatic sidecar injection in some of the namespaces. If i deploy an application then the injection is not working on the worker node where istiod runs, but works on the other worker node. Any idea what could cause this or how i could investigate it? I see the following error messages in the istio-proxy container on the working node and on the non-working node…
non working:
2021-09-21T08:26:30.027820Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
9/21/2021 10:26:30 AM 2021-09-21T08:26:30.119232Z warn ca ca request failed, starting attempt 2 in 181.53978ms
9/21/2021 10:26:30 AM 2021-09-21T08:26:30.301004Z warn ca ca request failed, starting attempt 3 in 383.361528ms
9/21/2021 10:26:30 AM 2021-09-21T08:26:30.684655Z warn ca ca request failed, starting attempt 4 in 729.187635ms
9/21/2021 10:26:31 AM 2021-09-21T08:26:31.414179Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing dial tcp 10.43.54.97:15012: i/o timeout”
9/21/2021 10:26:31 AM 2021-09-21T08:26:31.720153Z warn ca ca request failed, starting attempt 1 in 97.396333ms
9/21/2021 10:26:31 AM 2021-09-21T08:26:31.817786Z warn ca ca request failed, starting attempt 2 in 201.303555ms
9/21/2021 10:26:32 AM 2021-09-21T08:26:32.019258Z warn ca ca request failed, starting attempt 3 in 417.970541ms
9/21/2021 10:26:32 AM 2021-09-21T08:26:32.028152Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
9/21/2021 10:26:34 AM 2021-09-21T08:26:34.028075Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
9/21/2021 10:26:36 AM 2021-09-21T08:26:36.027803Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
9/21/2021 10:26:38 AM 2021-09-21T08:26:38.027759Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected
9/21/2021 10:26:40 AM
working:
| 2021-09-21T06:09:22.061881Z | warning | envoy config | StreamAggregatedResources gRPC config stream closed: 0, |
|---|---|---|---|
| 2021-09-21T06:09:22.544236Z | info | xdsproxy | connected to upstream XDS server: istiod.istio-system.svc:15012 |
| 2021-09-21T06:41:38.827001Z | warning | envoy config | StreamAggregatedResources gRPC config stream closed: 0, |
| 2021-09-21T06:41:39.181322Z | info | xdsproxy | connected to upstream XDS server: istiod.istio-system.svc:15012 |
| 2021-09-21T07:10:58.202346Z | warning | envoy config | StreamAggregatedResources gRPC config stream closed: 0, |
| 2021-09-21T07:10:58.531760Z | info | xdsproxy | connected to upstream XDS server: istiod.istio-system.svc:15012 |
| 2021-09-21T07:42:40.851021Z | warning | envoy config | StreamAggregatedResources gRPC config stream closed: 0, |
| 2021-09-21T07:42:41.081856Z | info | xdsproxy | connected to upstream XDS server: istiod.istio-system.svc:15012 |
Connectivity check:
working container:
istio-proxy@sleep-5dbff8dbd4-c9jz8:/$ nc -v -w 2 istiod.istio-system.svc 15012
istiod.istio-system.svc.cluster.local [10.43.54.97] 15012 (?) open
not working container:
istio-proxy@sleep-58bb86d655-7r7sh:/$ nc -v -w 2 istiod.istio-system.svc 15012
istiod.istio-system.svc.cluster.local [10.43.54.97] 15012 (?) : Connection timed out
regards
Zoltan