We have installed two istiod in a cluster, facing the CA certificate issue intermittently, when we delete app pod manually in the namespace. Sometimes deployment successfully completed and some time it fails with CA certificate issue.
Istio Version 1.18.2
Any experts can help on this issue?
We are using outboundTrafficPolicy to REGISTRY_ONLY, when I change it to ALLOW_ANY everything working fine no issue, sidecar able to fetch the CA certificates.
spec:
meshConfig:
outboundTrafficPolicy:
mode: REGISTRY_ONLY
Sidecar proxy Error Log:
Envoy command: [-c etc/istio/proxy/envoy-rev.json --drain-time-s 45 --drain-strategy immediate --local-address-ip-version v4 --file-flush-interval-msec 1000 --disable-hot-restart --allow-unknown-static-fields --log-format %Y-%m-%dT%T.%fZ %l envoy %n %g:%# %v thread=%t -l warning --component-log-level misc:error --concurrency 2]
2023-10-18T17:50:35.401101Z warn ca ca request failed, starting attempt 1 in 95.997032ms
2023-10-18T17:50:35.497603Z warn ca ca request failed, starting attempt 2 in 218.441706ms
2023-10-18T17:50:35.717206Z warn ca ca request failed, starting attempt 3 in 436.533937ms
2023-10-18T17:50:36.167508Z warn ca ca request failed, starting attempt 4 in 807.244185ms
2023-10-18T17:50:36.975109Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = “transport: authentication handshake failed: tls: failed to verify certificate: x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "cluster.local")”
2023-10-18T17:50:37.725320Z warn ca ca request failed, starting attempt 1 in 99.116073ms
2023-10-18T17:50:37.824595Z warn ca ca request failed, starting attempt 2 in 192.845891ms
2023-10-18T17:50:38.027748Z warn ca ca request failed, starting attempt 3 in 423.957088ms
Log from Istiod:
error when updating configmap istio-ca-root-cert: Operation cannot be fulfilled on configmaps “istio-ca-root-cert”: the object has been modified; please apply your changes to the latest version and try again controller=namespace controller
Please let me know if any additional detials required.
Thanks
Gokul