After upgrading from istio 1.4 to 1.5 everything initially proceeded smoothly, however recently my Ingress Gateway started failing to connect to an internal service, and was logging SSL certificate expired errors.
Looking into this further, it appears that the gateway may still have been using the file-system mounted certificates generated by istio-citadel, and with citadel removed from the 1.5 installation these then expired without being renewed.
I had initially thought that the problem was that I was still running the (deprecated) node-agent workload in the cluster, and that this might be preventing the loading of certs from istiod
Re-installing citadel renewed the certs, and fixed the issue - but I’d much prefer to run using the ‘new’ istiod/sds setup. Even after uninstalling citadel and node-agent again, the ingress still appeared to be mounting certs from the citadel-generated secrets into /etc/certs.
Currently I’ve found that removing citadel/node-agent, and removing the key-and-cert secrets created by citadel seems to result in the ‘correct’ (new) behaviour - certs are no longer mounted to /etc/certs and the certificates used by the ingress gateway appear to have a much shorter expiry (~24h?) compared with the citadel-generated certificates.
So - my assumption is that for some reason with the original cert secrets (from citadel) present, the ingress gateway was using these, rather than loading certs from istiod - and that now that these secrets have been deleted the behaviour is as expected. Does this sound reasonable, or have I got myself into a situation where everything works ‘for now’, but is likely to fail in the future?
Thanks!