Custom client certificates for service to service calls

Security
1

Background
We’re currently using vanilla Envoy with mostly static config files in our org. Most services across the company do authorization based on client certificate CNs, and this works fine via Envoy since it supports forwarding the cert details to the service as a header. Now we’re looking to move to Istio 1.1, and I’m trying to replicate roughly the same behavior. There are two scenarios for this (mtls is globally enabled):

  • Ingress calls from outside of the cluster
  • Service to service calls

Following Securing Gateways with HTTPS, I was able to have ingress require mtls and forward client cert details to the backend service, as expected.

For service to service calls, I can use Destination Rules to have each service submit a custom client certificate to upstream. I was able to successfully mount a volume to the istio-proxy sidecar with the right certificates.

Problem
After applying destination rules with paths to client certs, calls from App1 to App2 were getting an upstream reset error. There were no logs for either proxies (App1’s or App2’s), but I exec’d into App1’s proxy and attempted to make an https call with the custom cert to App2, and I noticed the call was failing because App2’s validation context for its ingress listener uses the default, self-signed root-cert.pem file generated by Citadel, which explains why it wasn’t accepting the custom certs:

"validation_context": {
 "trusted_ca": {
  "filename": "/etc/certs/root-cert.pem"
 }
}

Question
What is the best way to approach this? I understand Istio supports Plugging in External CA Key and Certificate, however, in our case we have a dedicated CA team that manages certificates at the company level, and my team doesn’t have access to their private key to plug into Istio - we only have the root cert. So my question is: is there a way to have Citadel continue to generate self-signed server certs BUT override the root-cert.pem only for this CA validation context?

I tried playing around with the sidecar-injector-configmap helm template to have it load sidecar certs from a different secret (one that I’d be able to create myself) instead of one created by Citadel. I ended up replacing root-cert.pem portion of an existing istio.app1 secret with our CA’s root cert, applied it under a new secret called certs.app1 and verified that the sidecar mounted it correctly. However, after inspecting the contents of etc/certs/root-cert.pem on the sidecar, I noticed it was still using the default self-signed root cert, even though the mounted secret had a different pem file. I’m not sure how this is possible, but I’m thinking something realizes that the root cert isn’t correct and attempts to overwrite it with what Citadel gives it. I saw these in the sidecar logs, though not sure if it’s relevant:

|2019-03-25T17:40:33.960106Z|info|watchFileEvents: "/etc/certs//..2019_03_25_17_40_33.353356507": CREATE|
|2019-03-25T17:40:33.960161Z|info|watchFileEvents: "/etc/certs//..2019_03_25_17_40_33.353356507": MODIFY|ATTRIB|
|2019-03-25T17:40:33.960168Z|info|watchFileEvents: "/etc/certs//..data_tmp": RENAME|
|2019-03-25T17:40:33.960228Z|info|watchFileEvents: "/etc/certs//..data": CREATE|
|2019-03-25T17:40:33.960267Z|info|watchFileEvents: "/etc/certs//..2019_03_25_17_39_09.147894698": DELETE|
|2019-03-25T17:40:43.960257Z|info|watchFileEvents: notifying|

What are the implications of having the option to explicitly specify the path for CA validation context built natively in Istio? Any help would be appreciated!

2

For service-to-service calls, we use citadel, whose root can come from external or self-signed. Can we use citadel issued key/cert for service-to-service calls ?

3

This is currently not supported. But there are three possible solutions (require implementations):

  1. Invent a way on Pilot to override the validation cert for specific Envoy xDS sections.
  2. Adding an extra root of trust to every Envoy, for authenticating your own system’s certs.
  3. Introducing “trust bundles” (each root is scoped to a specific “trust domain”) to Envoy and Istio. This is a work under discussion with the SPIFFE team.

Both 1) and 3) are more complex solutions. And IMO, 1) is less preferable compared with 3), for introducing security related complexity to Pilot. 2) is a workaround that doesn’t comply with the “trust bundle” model. We are designing 3) now and it would be great if we can get some help from the community as well.

4

Thanks Oliver. With option 3, all external certificates will need to be regenerated to have their SAN fields populated with their SPIFFE IDs, right?

5

That’s a good point. The “trust bundle” here is a SPIFFE standard. I don’t think we thought about how to extend it to non-SPIFFE certs.

Just thinking aloud: maybe using the domain name in the cert to scope the trust bundles? For example, if you have a trust bundle for domain name “example.com”, then all the certs with DNS name as its subdomains, like “prod.example.com” are suitable for the trust bundle.

6

I’m just learning about SPIFFE; going over their documentation, it doesn’t look like there’s anything inherently different in an SVID x509 document than a normal cert other than the fact that there needs to be an SPIFEE-compliant ID in the SAN field:

In an X.509 SVID, the corresponding SPIFFE ID is set as a URI type in the Subject Alternative Name extension (SAN extension, see RFC 5280 section 4.2.16). An X.509 SVID MUST contain exactly one URI SAN. It MAY contain any number of other SAN fields, including DNS SANs.

This would require submitting a one time CSR to the CA for every client certificate, which doesn’t sound too bad.

7

Yes, you can make your CA SPIFFE-compliant :slight_smile: Then you can adopt the trust bundle when it’s available in the future.

8

Is there a github issue for tracking the work for adding trust bundles?

9

In the interim another approach could be to setup an intermediate proxy that trusts Istio (your CA) and your Service (company CA). I found that it is feasible to use mesh expansion as a method to chain together services. Mutual TLS from Istio to the intermediate proxy (envoy?) and mutual TLS from the proxy to your external Service. I’ve only tested this out in a lab environment. If you setup your own CA you could create longer-lived certs for your intermediate proxy.