Best practices for mounting egress caCertificates (Trouble with: Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 6 rejected; lds updates: 5 successful, 0 rejected)

Security
1

Hello,
i’m using Istio 1.5.3.

I’m trying to set up secure communication with mesh-external resources for which i need to import own caCertificates to use in the DestinationRule:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: my-host-name
spec:
  host: my.host.name
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
      - port:
          number: 9443
        tls:
          caCertificates: /etc/istio/egressgateway-ca-certs/CA-CERTS.pem
          mode: SIMPLE

I understand that the file /etc/istio/egressgateway-ca-certs/CA-CERTS.pem needs to be mounted to the sidecar proxy from which the egress traffic originates. This can be done either with annotations in the deployment spec, or apparently by using a dedicated egress gateway and using the egressgateway-ca-certs secret.

Now, here is the problem: The cluster is scaled down every night, and upon scaling up in the morning the Istio Sidecars of almost all pods fail to start with the following error:

Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 0 successful, 6 rejec ted; lds updates: 5 successful, 0 rejected

This seems to affect those pods that do NOT have that /etc/istio/egressgateway-ca-certs/CA-CERTS.pem file available in their sidecar. Those that do start up just fine, and if i remove the caCertificate line from the destination rule, then all others start up too.

It kinda does make sense - those pods would not be able to perform what is required by that destination rule. However, they would never be required to do it, as the egress traffic won’t happen from them.

Do i really have to mount the caCertificates that are used by the Egress of only one specific Pod to ALL Istio sidecars of all pods in the mesh (even in all namespaces!), even though it is just for egress from one specific pod? Is there some better solution?

Thanks a lot in advance!

2

My two cents would be trying to see if you are able to use Sidecar limit the scope of this DestinationRule, thus it only applies to the workloads you care about. Other pods thus won’t get this effect in the CDS push.

@howardjohn might recommend better stuff.

1 Like
3

Thanks! What i’ve done so far is to use exportTo to limit the destinationRule to the namespace that i want it to apply to at least, but i haven’t found a way to limit it to specific sidecars. The latter would solve the problem of course.

Right now it seems that the default behaviour of a destinationrule is that it becomes available in all namespaces despite being installed into a specific single namespace, and it will prevent proper pod startup across the entire mesh in all namespaces for all pods that do not mount the referenced caCertificates file.

Isn’t that even a security problem?
Someone who can install destinationrules to one namespace might be able to negatively impact namespaces that he has no access to otherwise. One could write a “destinationrule-bomb” that prevents proper pod startup across the entire mesh.

4

Have you tried to use istio-egressgateway and mount the certificate only there? You could limit the DR only for egress namespace.

1 Like
5

Thanks! Yeah, that works…but it feels a bit like a workaround to be forced to put the egress gateway into an own namespace just to deal with that destinationRule behaviour there.

Also, redirecting the traffic through an egress gateway doesn’t look as nice in kiali, as the service to the egress gateway then shows up in red if there are problems, but that one might be used by many different outgoing connections.

(I’m not trying to be negative here, just pointing out disadvantages of that approach)

6

I’m glad that I could help. I agree that it’s not ideal, but this is open source project so that everybody can submit his/her ideas. :slight_smile:

7

:slight_smile: I think the best solution would be that the envoy proxies do start up with missing caCertificates, but then refuse to serve that traffic if it happens. Just like they would do with a wrong caCertificate or server certificate.
But that’d be a change in envoy, not istio.