I have recently upgraded to Istio 1.7.0 and am running in to an issue where the root cert appears to be invalid, preventing my pods from starting (this includes ingress gateway pods) until 5 minutes have passed and the root cert then appears to get rotated and the service starts up successfully. Any thoughts?
> istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:45:22.785962Z info JWT policy is third-party-jwt
istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:45:22.785982Z warn Using existing certificate ./etc/certs istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:45:22.785996Z info PilotSAN []string{"istiod.istio-system.svc"} istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:45:22.786000Z info MixerSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account"} istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:45:22.786031Z info sa.serverOptions.CAEndpoint == istiod.istio-system.svc:15012 istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:45:22.786038Z info Using user-configured CA istiod.istio-system.svc:15012 istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:45:22.786041Z info istiod uses self-issued certificate ****<cert redacted>**** istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:50:23.214177Z info cache Root cert has changed, start rotating root cert for SDS clients istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:50:23.214306Z info sds resource:default pushed key/cert pair to proxy istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:50:23.214336Z info sds Dynamic push for secret default istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:50:24.006840Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:50:26.007206Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:50:28.006953Z warn Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 0 rejected; lds updates: 0 successful, 0 rejected istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:50:28.388171Z info sds resource:ROOTCA new connection istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:50:28.388249Z info sds Skipping waiting for gateway secret istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:50:28.388437Z info cache adding watcher for file ./etc/certs/root-cert.pem istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:50:28.388478Z info cache GenerateSecret from file ROOTCA istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:50:28.388610Z info sds resource:ROOTCA pushed root cert to proxy istio-ingressgateway-69fd56779c-595mp istio-proxy 2020-10-14T16:50:30.007461Z info Envoy proxy is ready
This seems to happen every time so where is it getting the invalid root cert from? I have checked /etc/certs where it reports it is pulling the certs from when it starts up and these do not appear to change