Istio: Unable to set up mutual TLS origination with an egress gateway, custom certificates

Networking
1

Hello,

I am having problem with configuration of mutual TLS origination with an egress gateway. I am using configuration examples provided in the documentation:

but with certificates being added to egress gateway as kubernetes secrets.

I am getting following error message for curl:

kubectl exec "$(kubectl get pod -l app=sleep -o jsonpath={.items..metadata.name})" -c sleep -- curl -s -v 'http://ADDRESS_HERE/Service/something'

[2021-02-08T10:32:50.151Z] "GET /Service/something HTTP/1.1" 503 UF,URX "TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED" 0 195 47 - "10.240.0.14" "Apache-HttpClient/4.5.3 (Java/11.0.10)" "9777cc5a-a181-4961-b551-0af558c24e43" "<ADDRESS_HERE>" "<IP_ADDRESS_HERE>:9443" outbound|9443||<ADDRESS_HERE> - 10.240.0.164:9443 10.240.0.14:51206 <ADDRESS_HERE> -
[2021-02-08T10:32:50.253Z] "GET /Service/something HTTP/1.1" 503 UF,URX "TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED" 0 195 33 - "10.240.0.14" "Apache-HttpClient/4.5.3 (Java/11.0.10)" "9777cc5a-a181-4961-b551-0af558c24e43" "<ADDRESS_HERE>" "<IP_ADDRESS_HERE>:9443" outbound|9443||<ADDRESS_HERE> - 10.240.0.164:9443 10.240.0.14:51208 <ADDRESS_HERE> -

It looks like there is an issue related to certificate but I believe that the certificates are correct because I am able to connect to the target service from outside the Istio mesh (curl -k --cert client.crt --key client.key -v https://ADDRESS_HERE:9443/Service/something).

Here’s my configuration:

Istio 1.8.0:

apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: externalservice-egress
spec:
  hosts:
  - api.externalservice.com
  ports:
  - number: 9443
    name: https
    protocol: HTTPS
  resolution: DNS
  location: MESH_EXTERNAL
---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: externalservice-egress
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 9443
      name: https
      protocol: HTTPS
    hosts:
    - api.externalservice.com
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: externalservice-egress
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
  - name: externalservice-egress
    trafficPolicy:
      loadBalancer:
        simple: ROUND_ROBIN
      portLevelSettings:
      - port:
          number: 9443
        tls:
          mode: ISTIO_MUTUAL
          sni: api.externalservice.com
---
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: externalservice
spec:
  hosts:
    - api.externalservice.com
  gateways:
    - externalservice-egress
    - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: externalservice-egress
        port:
          number: 9443
      weight: 100
  - match:
    - gateways:
      - externalservice-egress
      port: 9443
    route:
    - destination:
        host: api.externalservice.com
        port:
          number: 9443
      weight: 100
---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: externalservice-egress-tls-origination
  namespace: istio-system # namespace other than for other configuration items - like for example from documentation
spec:
  host: api.externalservice.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 9443
      tls:
        mode: MUTUAL
        sni: api.externalservice.com
        caCertificates: /etc/istio/egressgateway-ca-certs/ca.crt
        clientCertificate: /etc/istio/egressgateway-certs/tls.crt
        privateKey: /etc/istio/egressgateway-certs/tls.key

Kubernetes secrets:

kubectl describe secret -n istio-system istio-egressgateway-certs 

Name:         istio-egressgateway-certs
Namespace:    istio-system
Labels:       <none>
Annotations:  <none>

Type:  kubernetes.io/tls

Data
====
tls.crt:  1956 bytes
tls.key:  1679 bytes

kubectl describe secret -n istio-system istio-egressgateway-ca-certs 

Name:         istio-egressgateway-ca-certs
Namespace:    istio-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
ca.crt:  3792 bytes

I can see both secrets mounted successfully inside istio-egressgateway pod under /etc/istio/ directories.

api.externalservice.com is an external service listening only on https/9443.

Thank you for helping me with this. I appreciate any ideas how to resolve this issue.

2

Could anyone advise on this issue, please? Is there anything other than incorrect certificate that may cause this behavior?

3

are you seeing any request logs on your egressgateway? can you determine where the connection is failing? and if i understand correctly this is your setup? app <-----Istio mTLS ----> EgressGateway <-----Custom mTLS —> API

4

Thank you for replying. Here’s the log from istio-egressgateway:

2021-02-11T07:59:25.379414Z	info	xdsproxy	Envoy ADS stream established
2021-02-11T07:59:25.379506Z	info	xdsproxy	connecting to upstream XDS server: istiod.istio-system.svc:15012
2021-02-11T07:59:25.390224Z	warning	envoy main	there is no configured limit to the number of allowed active connections. Set a limit via the runtime key overload.global_downstream_max_connections
2021-02-11T07:59:26.648667Z	info	Envoy proxy is ready
2021-02-11T08:01:02.052124Z	info	sds	resource:ROOTCA new connection
2021-02-11T08:01:02.052170Z	info	sds	resource:default new connection
2021-02-11T08:01:02.052210Z	info	sds	Skipping waiting for gateway secret
2021-02-11T08:01:02.052243Z	info	sds	Skipping waiting for gateway secret
2021-02-11T08:01:02.083063Z	info	sds	resource:file-root:/etc/istio/egressgateway-ca-certs/ca.crt new connection
2021-02-11T08:01:02.083164Z	info	sds	Skipping waiting for gateway secret
2021-02-11T08:01:02.083229Z	info	sds	resource:file-cert:/etc/istio/egressgateway-certs/tls.crt~/etc/istio/egressgateway-certs/tls.key new connection
2021-02-11T08:01:02.083312Z	info	sds	Skipping waiting for gateway secret
2021-02-11T08:01:02.083478Z	info	cache	adding watcher for file /etc/istio/egressgateway-ca-certs/ca.crt
2021-02-11T08:01:02.083647Z	info	cache	GenerateSecret from file file-root:/etc/istio/egressgateway-ca-certs/ca.crt
2021-02-11T08:01:02.083683Z	info	cache	adding watcher for file /etc/istio/egressgateway-certs/tls.crt
2021-02-11T08:01:02.083803Z	info	cache	GenerateSecret from file file-cert:/etc/istio/egressgateway-certs/tls.crt~/etc/istio/egressgateway-certs/tls.key
2021-02-11T08:01:02.084189Z	info	sds	resource:file-root:/etc/istio/egressgateway-ca-certs/ca.crt pushed root cert to proxy
2021-02-11T08:01:02.084183Z	info	sds	resource:file-cert:/etc/istio/egressgateway-certs/tls.crt~/etc/istio/egressgateway-certs/tls.key pushed key/cert pair to proxy
2021-02-11T08:01:02.349635Z	info	cache	Root cert has changed, start rotating root cert for SDS clients
2021-02-11T08:01:02.349686Z	info	cache	GenerateSecret default
2021-02-11T08:01:02.349781Z	info	sds	resource:default pushed key/cert pair to proxy
2021-02-11T08:01:02.652938Z	info	cache	Loaded root cert from certificate ROOTCA
2021-02-11T08:01:02.653016Z	info	sds	resource:ROOTCA pushed root cert to proxy
[2021-02-11T08:19:03.174Z] "GET /Service/something HTTP/1.1" 503 UF,URX "TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED" 0 195 59 - "10.240.0.101" "Apache-HttpClient/4.5.3 (Java/11.0.10)" "a12a5df0-f83c-48c6-b57e-c7c5a1fd0f1c" "<ADDRESS_HERE>" "<IP_ADDRESS_HERE>:9443" outbound|9443||<ADDRESS_HERE> - 10.240.1.0:9443 10.240.0.101:35326 <ADDRESS_HERE> -

I am trying to reproduce “Perform mutual TLS origination with an egress gateway” configuration from Istio / Egress Gateways with TLS Origination (File Mount), so I think that mutual tls should be performed by istio-egressgateway talking to external service on behalf of our application. The application is configured to use http/80 which is translated to https/9443 by the egress gateway.

5

Does anyone have any ideas how to solve it or confirm that the certificate is what needs to be changed? I may be wrong here but maybe there is a way to tell Istio to trust the custom CA certificate and establish the connection regardless of verification error?

6

Hello @bertini86 , We are having the same issues and trying to find a solution. Were you able to get this resolved ? we would really appreciate if you can share the steps to fix the issue. Thanks !!

7

Hi - we’re facing the same issue with Istio 1.13.2 …Anyone got this working?

1 Like
8

istio-1.14.3 , please exec ’ kubectl get pod | awk ‘{print $1}’ | xargs kubectl delete pod’