Hi all!
We’re running 6 K8s clusters (2 prod, 2 dev, 2 test) and are in the process of enabling multicluster between the 2 clusters in the respective environment.
These clusters are all “in production” and already run workloads.
When rolling out the shared CA cert to the dev clusters, we noticed that all Pods, on the cluster where the new CA cert had been deployed, lost communication to the control plane - as their sidecars still had the cert from the “old”, non-multicluster setup.
In hindsight, this was kinda to be expected, but it took us a moment to notice and fix by restarting all Pods with sidecars - which seems to be the only way to make them talk to the control plane again?
(FYI: both the test and the dev clusters have it enabled now, and work as expected)
Before we now also roll out a new, shared CA cert to our 2 prod clusters, and also get an outage there until all pods with sidecars have been restarted, we were wondering if there is any chance of doing this with zero downtime - e.g. with the old and the new CA certs present in the control plane at the same time, so that the already running sidecars, who are only trusting the old CA cert, can still talk to the control plane that has the new CA cert.