Hi Team
We discovered an issue when deploying an application in the Istio-mesh.
Output from command “kubectl logs [application-pod] -c isito-proxy”
2023-02-23T22:33:59.904365Z warnEnvoy proxy is NOT ready: config received from XDS server, but was rejected: cds updates: 96 successful, 0 rejected; lds updates: 0 successful, 96 rejected
Then we use “istioctl proxy-status” command and find out that ALL the envoy proxies are in that STALE stage in LDS.
istioctl proxy-status -i istio-internal --context ops-cluster | less
NAME CLUSTER CDS LDS EDS RDS ISTIOD VERSION
application_pod-5d4f4df service-cluster SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-5dbd6b 1.14.6
application_pod-5d6b8dd service-cluster SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-5dbd6 1.14.6
application_pod-5f6ff958 service-cluster SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-5dbd6b 1.14.6
application_pod-7898d4 service-cluster SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-5dbd6 1.14.6
application_pod-796f68b8 service-cluster SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-5dbd 1.14.6
application_pod-7cf99b service-cluster SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-5dbd6 1.14.6
application_pod-89c8dc service-cluster SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-5dbd6 1.14.6
application_pod-975cdd service-cluster SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-5dbd6b4 1.14.6
Istio version: 1.14.6
However after 1 hour, ALL LDS are back to SYNCED stage, and we can deploy application without issue. Everything back to normal.
Please suggest what we can do to investigate the root cause or anything we can do to prevent this issue happen agains.