Issue with Istio 1.3.2 SDS not reloading tls cert and key

Security
1

I can create a cert-manager 0.10 Certificate with DNS resolvers and I can see that the secret is ultimately created with the tls.key and tls.crt entries. If I export the secret and base64 decode it, I can see that it was issued by Let’s Encrypt Staging/Prod. However, cert-manager generates a temporary cert while the real cert is being fetched and SDS loads this cert. I can execute a

echo | openssl s_client -showcerts -servername foo.example.com -connect foo.example.com:443 2>/dev/null | openssl x509 -inform pem -noout -text

and see the cert returned from the istio-ingressgateway is cert-manager’s temporary cert, even though cert-manager has created/updated the secret which is in the same namespace as the istio-ingressgateway.

A workaround is to kill the istio-ingressgateway and then the SDS agent in the pod will pick up the correct credentials.

Any idea’s as to why this is or how I can debug what’s going on?

2

There was an issue with Ingress SDS and the fix is in 1.3.5.
If the ingress-sds container has log like this 2019-09-26T12:41:12.701283Z warn secretFetcherLog unexpected server key/cert change in secret application-connector-certs, then you hit the same issue. You can upgrade to 1.3.5 in that case.
More context is here https://github.com/istio/istio/issues/17409.