We want to plug in our own CA certificate as described at plugin-ca-cert.
However, we have an unusual requirement: when istiod starts, we’ll use an init-container to:
generate a new CA private key,
request a CA cert from our company’s root CA,
and write them to an emptyDir for istiod to read from.
Is this ‘ephemeral CA cert’ approach supported?
I’m worried about what will happen after one or more of our istiod pods restart. The new istiod will get a new key and CA certificate. Proxy-to-istiod and proxy-to-proxy communications will then use a mixture of certificates signed by the old and new intermediate CAs.
This may break the chain of trust, depending on how/when Istio distributes the intermediate CA certificate. I read root-transition which suggests everything might automatically work, via the magic of Envoy hot restarts, but I’m not sure.
The reason I have this unusual requirement is that my company has tight controls on the use of internal CAs. However, I expect anyone plugging in a long-lived CA cert will have a similar problem when it eventually expires. Also note that I want to use an emptyDir to store my CA’s private key rather than a Secret because my company generally doesn’t treat Kubernetes Secrets as sufficiently secure for storing TLS keys.
Yes, this will work. Write them to /etc/cacerts on the istiod pod with filenames of:
“ca-key.pem”
“ca-cert.pem”
“cert-chain.pem”
Istiod tries to mount cacerts into this dir, which you could just change to be an empty volume.
As for the chain of trust, as long as they derive from the same root, it should be okay.
But I’m still unclear how/whether the newly-restarted istiod will trust a long-running proxy (myproxy1) that connects to it.
This would work if the proxy presents myproxy cert and the original ca-cert.pem that signed it. Istiod would complete the chain of trust using its mounted root-ca.pem. Another good alternative would be if the proxy gets dynamically issued a brand new cert (myproxy’), signed by the new ca-cert.pem’, in which case I don’t care whether istiod completes the chain of trust by receiving ca-cert.pem’ from the proxy or looking it up from its local storage.
But the chain of trust wouldn’t complete if the proxy presents its original myproxy cert, and doesn’t send the original ca-cert.
I have similar concerns about proxy1-to-proxy2 TLS, in cases where their certs were issued by the ca-cert and ca-cert’ respectively.
I plan to test all this out but wanted to get a rough idea of feasibility and pitfalls first!
it will work as long as the root ca is same and old ca-cert.pem is not expired, i tested this mixed ca-cert.pem combination with self signed ca and it worked. I wanted to test the scenario of what happens when the signing cert (ca-cert.pem) is rotated before its expiry.
Our security team also has concerns on using k8 as storing the cert, so I am also looking init sidecar solution similar to that @Phil_Harvey mentioned above.
@ceposta so far I have seen that by default istio creates istio-ca-secret secret and uses it, if cacerts secret is also present then istio gives precedence to that.
But if /etc/certs is created by custom init container then will istio not try to re-create the self signed certs?
Also i think this problem will be solved once vault integration is available.
No, it checks on the file system for the certs in /etc/cacerts … it shouldn’t matter whether it’s mounted in or if it’s created by an init-container before istiod starts
When the init-container requests a CA cert from company’s root CA, does the returned cert-chain contain root CA cert as well? If so, then the cert-chain will push to Envoy sidecar, and mTLS handshake should work if envoy A has cert-chain and B has cert-chain’ (different intermediate CA cert + same root CA cert). If not, then there will be a traffic downtime, until all proxy sidecars receive the same cert-chain’.