Trust custom Root CA on Egress Gateway

Hallo

I want to route traffic to services outside of Kubernetes over the Egress Gateway.

I don’t want to trust our internal Root Certificates on all pods so I’m asking if its possible to add/trust them on the Egress Gateway so that I call them without trusting them on each Pod.

I am not sure I completely understand the question, but my hunch is that you are looking to set up TLS origination at the Egress Gateway. See https://istio.io/docs/tasks/traffic-management/egress/egress-gateway-tls-origination/ for instructions on how to do this.

In addition you can set the DestinationRule.TLSMode to SIMPLE and provide the CA certificate to be used by the Gateway to validate the external server’s certificate using DestinationRule.TLSSettings.caCertificate.
DISCLAIMER: I haven’t tried this personally but I suppose it should work.

The TLSSettings.caCertificates documentation states,

caCertificates	string	

OPTIONAL: The path to the file containing certificate authority certificates to use in verifying a presented server certificate. If omitted, the proxy will not verify the server’s certificate. Should be empty if mode is ISTIO_MUTUAL.

My understanding is that, if you don’t specify the caCertificates value the Gateway will skip validation of Server certificates, while if you do it will validate it based on the ca-certs you have uploaded.

Is my understanding accurate @Tao_Li ?

@nitishm you are right. The only issue is that you have to deploy the egress gateway in a separate namespace. See https://github.com/istio/istio/issues/15517.

1 Like

Thank you, it works. :slight_smile:

I’ve a bit strange behavior, the connection was working but now I get a 503.

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: myservice-ext
  namespace: mynamespace
spec:
  hosts:
  - myservice.domain.local
  ports:
  - number: 80
    name: http
    protocol: HTTP
  - number: 443
    name: https
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway-myservice
  namespace: mynamespace
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 80
      name: https
      protocol: HTTPS
    hosts:
    - myservice.domain.local
    tls:
      mode: MUTUAL
      serverCertificate: /etc/certs/cert-chain.pem
      privateKey: /etc/certs/key.pem
      caCertificates: /etc/certs/root-cert.pem
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-myservice
  namespace: mynamespace
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: myservice
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 80
        tls:
          mode: ISTIO_MUTUAL
          sni: myservice.domain.local
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-myservice-through-egress-gateway
  namespace: mynamespace
spec:
  hosts:
  - myservice.domain.local
  gateways:
  - istio-egressgateway-myservice
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: myservice
        port:
          number: 80
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway-myservice
      port: 80
    route:
    - destination:
        host: myservice.domain.local
        port:
          number: 443
      weight: 100
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-tls-for-myservice
  namespace: mynamespace
spec:
  host: myservice.domain.local
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: SIMPLE 
        caCertificates: /etc/certs/myrootcert.pem

In the proxy logs I see:

gRPC config for type.googleapis.com/envoy.api.v2.Cluster rejected: Error adding/updating cluster(s) outbound|443||myservice.domain.local: Invalid path: /etc/certs/myrootcert.pem

The certificate is mounted on the egress gateway pods.

Any hints whats wrong in my config?

So you had this working with the same caCertificate mounted and it randomly stopped working ? Could you make it work again with a clean setup ?

Yes it was working with the same config.

I think it has nothing to do with the ca certificate, its still working with another service.

I’ve opened a new Topic -> https://discuss.istio.io/t/upstream-connect-error-or-disconnect-reset-before-headers-reset-reason-connection-terminatio/

@nicolasmi I forgot to mention, sorry - you have to use exportTo: '.' in your destination rule, so the destination rule will be applied in the egress gateway’s namespace only and the certificate will be loaded in that namespace only (for the egress gateway only).

Still getting a “503 Service Unavailable” and “upstream connect error or disconnect/reset before headers. reset reason: connection Termination”

I can access my service from the istio-proxy container in the same pod.

Access from my app container in the same pod doesn’t work.

Really don’t understand that, also that its working for other services.

How can I debug the 503?

Is it possbile that there’s an issue with the certificate used on the service?

I have two services and they run on the same webserver and also share the same certificate.

The CN matches for the fqdn of service that is working, the fqdn of the other service is only a subject alternative name.

The only difference between these services is that they have different fqdn and this certificate thing…

Can you provide the cert detail? Not sure what do you mean “fqdn of other service is only a subject alt name” and why do we have the difference of the two services?

The certificate has:

CN = my-service.domain.local

and

SAN =
DNS Name=my-service.domain.local
DNS Name=my-service2.domain.local

Found out that on my-service2.domain.local I get HTTP_1_1_REQUIRED when I try to access it with curl.

On my-service.domain.local I can use HTTP/2.

Maybe thats the issue. Can I force the Egress Gateway to use HTTP/1.1?

@vadimeisenbergibm, any ideas?

@Tao_Li @nicolasmai I will have to check the issue, will take me some time.

I have some more details.

The services are running on a IIS Webserver (Windows Server 2016).

When I set the SSL option “Client Certificate” to accept or required I get:

“503 Service Unavailable” and “upstream connect error or disconnect/reset before headers. reset reason: connection Termination”

When I set the SSL option “Client Certificate” to ignore it works.

In the docs from Microsoft they state:

In a few cases, HTTP/2 can’t be used in combination with other features. In these situations, Windows will fall back to HTTP/1.1 and continue the transaction. This may involve negotiating HTTP/1.1 during the handshake, or sending an error code to the client instructing it to retry over an HTTP/1.1 connection.

Looks like the “negotiating HTTP/1.1 during the handshake, or sending an error code to the client instructing it to retry over an HTTP/1.1 connection” does not work with Istio.