Istio: Unable to set up mutual TLS origination with an egress gateway, custom certificates

Hello,

I am having problem with configuration of mutual TLS origination with an egress gateway. I am using configuration examples provided in the documentation:

but with certificates being added to egress gateway as kubernetes secrets.

I am getting following error message for curl:

kubectl exec "$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})" -c sleep -- curl -s -v 'http://ADDRESS_HERE/Service/something'

[2021-02-08T10:32:50.151Z] "GET /Service/something HTTP/1.1" 503 UF,URX "TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED" 0 195 47 - "10.240.0.14" "Apache-HttpClient/4.5.3 (Java/11.0.10)" "9777cc5a-a181-4961-b551-0af558c24e43" "<ADDRESS_HERE>" "<IP_ADDRESS_HERE>:9443" outbound|9443||<ADDRESS_HERE> - 10.240.0.164:9443 10.240.0.14:51206 <ADDRESS_HERE> -
[2021-02-08T10:32:50.253Z] "GET /Service/something HTTP/1.1" 503 UF,URX "TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED" 0 195 33 - "10.240.0.14" "Apache-HttpClient/4.5.3 (Java/11.0.10)" "9777cc5a-a181-4961-b551-0af558c24e43" "<ADDRESS_HERE>" "<IP_ADDRESS_HERE>:9443" outbound|9443||<ADDRESS_HERE> - 10.240.0.164:9443 10.240.0.14:51208 <ADDRESS_HERE> -

It looks like there is an issue related to certificate but I believe that the certificates are correct because I am able to connect to the target service from outside the Istio mesh (curl -k --cert client.crt --key client.key -v https://ADDRESS_HERE:9443/Service/something).

Here’s my configuration:

Istio 1.8.0:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: externalservice-egress
spec:
  hosts:
  - api.externalservice.com
  ports:
  - number: 9443
    name: https
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: externalservice-egress
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 9443
      name: https
      protocol: HTTPS
    hosts:
    - api.externalservice.com
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: externalservice-egress
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: externalservice-egress
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 9443
        tls:
          mode: ISTIO_MUTUAL
          sni: api.externalservice.com
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: externalservice
spec:
  hosts:
    - api.externalservice.com
  gateways:
    - externalservice-egress
    - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: externalservice-egress
        port:
          number: 9443
      weight: 100
  - match:
    - gateways:
      - externalservice-egress
      port: 9443
    route:
    - destination:
        host: api.externalservice.com
        port:
          number: 9443
      weight: 100
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: externalservice-egress-tls-origination
  namespace: istio-system # namespace other than for other configuration items - like for example from documentation
spec:
  host: api.externalservice.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 9443
      tls:
        mode: MUTUAL
        sni: api.externalservice.com
        caCertificates: /etc/istio/egressgateway-ca-certs/ca.crt
        clientCertificate: /etc/istio/egressgateway-certs/tls.crt
        privateKey: /etc/istio/egressgateway-certs/tls.key

Kubernetes secrets:

kubectl describe secret -n istio-system istio-egressgateway-certs 

Name:         istio-egressgateway-certs
Namespace:    istio-system
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.crt:  1956 bytes
tls.key:  1679 bytes

kubectl describe secret -n istio-system istio-egressgateway-ca-certs 

Name:         istio-egressgateway-ca-certs
Namespace:    istio-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
ca.crt:  3792 bytes

I can see both secrets mounted successfully inside istio-egressgateway pod under /etc/istio/ directories.

api.externalservice.com is an external service listening only on https/9443.

Thank you for helping me with this. I appreciate any ideas how to resolve this issue.

Could anyone advise on this issue, please? Is there anything other than incorrect certificate that may cause this behavior?

are you seeing any request logs on your egressgateway? can you determine where the connection is failing? and if i understand correctly this is your setup? app <-----Istio mTLS ----> EgressGateway <-----Custom mTLS —> API

Thank you for replying. Here’s the log from istio-egressgateway:

2021-02-11T07:59:25.379414Z	info	xdsproxy	Envoy ADS stream established
2021-02-11T07:59:25.379506Z	info	xdsproxy	connecting to upstream XDS server: istiod.istio-system.svc:15012
2021-02-11T07:59:25.390224Z	warning	envoy main	there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
2021-02-11T07:59:26.648667Z	info	Envoy proxy is ready
2021-02-11T08:01:02.052124Z	info	sds	resource:ROOTCA new connection
2021-02-11T08:01:02.052170Z	info	sds	resource:default new connection
2021-02-11T08:01:02.052210Z	info	sds	Skipping waiting for gateway secret
2021-02-11T08:01:02.052243Z	info	sds	Skipping waiting for gateway secret
2021-02-11T08:01:02.083063Z	info	sds	resource:file-root:/etc/istio/egressgateway-ca-certs/ca.crt new connection
2021-02-11T08:01:02.083164Z	info	sds	Skipping waiting for gateway secret
2021-02-11T08:01:02.083229Z	info	sds	resource:file-cert:/etc/istio/egressgateway-certs/tls.crt~/etc/istio/egressgateway-certs/tls.key new connection
2021-02-11T08:01:02.083312Z	info	sds	Skipping waiting for gateway secret
2021-02-11T08:01:02.083478Z	info	cache	adding watcher for file /etc/istio/egressgateway-ca-certs/ca.crt
2021-02-11T08:01:02.083647Z	info	cache	GenerateSecret from file file-root:/etc/istio/egressgateway-ca-certs/ca.crt
2021-02-11T08:01:02.083683Z	info	cache	adding watcher for file /etc/istio/egressgateway-certs/tls.crt
2021-02-11T08:01:02.083803Z	info	cache	GenerateSecret from file file-cert:/etc/istio/egressgateway-certs/tls.crt~/etc/istio/egressgateway-certs/tls.key
2021-02-11T08:01:02.084189Z	info	sds	resource:file-root:/etc/istio/egressgateway-ca-certs/ca.crt pushed root cert to proxy
2021-02-11T08:01:02.084183Z	info	sds	resource:file-cert:/etc/istio/egressgateway-certs/tls.crt~/etc/istio/egressgateway-certs/tls.key pushed key/cert pair to proxy
2021-02-11T08:01:02.349635Z	info	cache	Root cert has changed, start rotating root cert for SDS clients
2021-02-11T08:01:02.349686Z	info	cache	GenerateSecret default
2021-02-11T08:01:02.349781Z	info	sds	resource:default pushed key/cert pair to proxy
2021-02-11T08:01:02.652938Z	info	cache	Loaded root cert from certificate ROOTCA
2021-02-11T08:01:02.653016Z	info	sds	resource:ROOTCA pushed root cert to proxy
[2021-02-11T08:19:03.174Z] "GET /Service/something HTTP/1.1" 503 UF,URX "TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED" 0 195 59 - "10.240.0.101" "Apache-HttpClient/4.5.3 (Java/11.0.10)" "a12a5df0-f83c-48c6-b57e-c7c5a1fd0f1c" "<ADDRESS_HERE>" "<IP_ADDRESS_HERE>:9443" outbound|9443||<ADDRESS_HERE> - 10.240.1.0:9443 10.240.0.101:35326 <ADDRESS_HERE> -

I am trying to reproduce “Perform mutual TLS origination with an egress gateway” configuration from Istio / Egress Gateways with TLS Origination (File Mount), so I think that mutual tls should be performed by istio-egressgateway talking to external service on behalf of our application. The application is configured to use http/80 which is translated to https/9443 by the egress gateway.

Does anyone have any ideas how to solve it or confirm that the certificate is what needs to be changed? I may be wrong here but maybe there is a way to tell Istio to trust the custom CA certificate and establish the connection regardless of verification error?