Certificate Validation with SDS enabled on Ingress Gateway and mode set to MUTUAL

#1

I have been having some difficulty understanding the mechanism by which certificates are validated by either party in a mutual TLS handshake.

I have SDS enabled on my ingress gateway(s) and the certificates are read by the Ingress SDS container (secretFetcher) from a Secret of type kubernetes.io/generic with cacert, cert and tls keys/value pairs present.

My question however is, how exactly does envoy trust a certificate down to the RootCA based on the cacert (or maybe I am completely off and I am mixing concepts up ) ?
A RootCA is usually a trusted entity and I understand how browsers and mobile devices can establish a chain of trust using a Trusted CA/ICA store.
However with Envoy, how does it know to trust a RootCA like verisign/docusign/letsencrypt ? Are these preloaded into the container at build time (or maybe as a part of the base image) ?

So if we have the following scenario,

How would Ingress Gateway validate a certificate presented by Service A (which was signed by ICA_A and RootCA_A) provided that the ICA -> RootCA used by the Ingress Gateway itself are ICA_B and RootCA_B ?

PS: This might be a very basic question for someone with a better understanding of certificate validation than me. I am only trying to get a better understanding of the concepts. I would really appreciate if some one could explain the mechanics of how this works for Envoy specifically.

#2

Also based on reading https://www.envoyproxy.io/docs/envoy/latest/intro/arch_overview/ssl#enabling-certificate-verification, does envoy implicitly trust the stored CAs even in SDS mode ? The link shows a static configuration where the validation context is configured with the path

validation_context:
          trusted_ca:
            filename: /etc/ssl/certs/ca-certificates.crt

I wonder if this is the case with SDS as well ?!

#3

I peeked at the dynamic envoy config and see no traces of trusted-ca’s in the validation context when SDS is enabled,

{
    "tls_context": {
      "common_tls_context": {
        "alpn_protocols": [
          "h2",
          "http/1.1"
        ],
        "tls_certificate_sds_secret_configs": [
          {
            "name": "foobar-certs",
            "sds_config": {
              "api_config_source": {
                "api_type": "GRPC",
                "grpc_services": [
                  {
                    "google_grpc": {
                      "target_uri": "unix:/var/run/ingress_gateway/sds",
                      "stat_prefix": "sdsstat"
                    }
                  }
                ]
              }
            }
          }
        ],
        "combined_validation_context": {
          "default_validation_context": {},
          "validation_context_sds_secret_config": {
            "name": "foobar-certs-cacert",
            "sds_config": {
              "api_config_source": {
                "api_type": "GRPC",
                "grpc_services": [
                  {
                    "google_grpc": {
                      "target_uri": "unix:/var/run/ingress_gateway/sds",
                      "stat_prefix": "sdsstat"
                    }
                  }
                ]
              }
            }
          }
        }
      },
      "require_client_certificate": true
    }
  }

Makes me wonder this is handled in the cases where the certificate is signed by a trusted-ca but the cacert is not passed in through the SDS api ? :thinking:

#4

validation_context_sds_secret_config is Envoy asking for CA certs over the SDS API.

#5

So how can it be configured to use the trusted-ca location in addition to the sds config ? Is there a fallback default validation context that could be set ?

#6

My question however is, how exactly does envoy trust a certificate down to the RootCA based on the cacert (or maybe I am completely off and I am mixing concepts up ) ?

You need to manually configure the rootCA to k8s secret so ingress can pick it up. Please refer here for more info: https://istio.io/docs/tasks/traffic-management/secure-ingress/sds/, which shows config TLS ingress for single host, multiple hosts and mTLS ingress.

A RootCA is usually a trusted entity and I understand how browsers and mobile devices can establish a chain of trust using a Trusted CA/ICA store.
However with Envoy, how does it know to trust a RootCA like verisign/docusign/letsencrypt ? Are these preloaded into the container at build time (or maybe as a part of the base image) ?

It’s not preloaded but dynamically configured via k8s secret by mesh operator.

#7

@Tao_Li So in the case of Secrets that only have the tls.crt and tls.key is it not possible to set the validation context to a default root certificate through some static envoy configurations ?

So taking the example of the SDS based tls context configuration block from the istio-proxy

           "combined_validation_context": {
            "default_validation_context": {},
            "validation_context_sds_secret_config": {
             "name": "foobar-certs-cacert",
             "sds_config": {
              "api_config_source": {
               "api_type": "GRPC",
               "grpc_services": [
                {
                 "google_grpc": {
                  "target_uri": "unix:/var/run/ingress_gateway/sds",
                  "stat_prefix": "sdsstat"
                 }
                }
               ]
              }
             }
            }
           }

can we somehow program the

"default_validation_context": {
    trusted_ca:
        filename: /etc/ssl/certs/ca-certificates.crt
}

to hold a reference to /etc/ssl/certs/ca-certificates.crt ? Maybe something that could be configured (in the future) via the Gateway CRD ?

#8

Can this PR meet your need? https://github.com/istio/istio/pull/13363

1 Like
#9

Yea I ve been keeping a watch on that PR. It would definitely help. Didn’t see much activity for 2 weeks on the PR so I thought it wasn’t being pursued any longer.

#10

Just pinged that PR. Hopefully we can hear from him soon.

1 Like