Cannot intercept TLS traffic with istio egress gateway

Marco · June 11, 2021, 3:00pm

I am trying to intercept generic TLS traffic (including HTTPS) directed to external applications with the istio egress gateway. The reason is that I want to be able to define which source pods can access a given set of external resources.
I have already tried the TLS origination approach by following the procedure described in Istio / Egress Gateways with TLS Origination (File Mount) . While this approach seems to work most of the time, I would still like to be able to send encrypted traffic directly from my application.

Now, I’m wondering if it is possible at all to do something like that. I would imagine the flow to be like this: Pod X wants to access an external service (e.g., google), the egress gateway intercepts the traffic and then it sends it to the intended destination. Now, this works if I use PASSTHROUGH as tls mode on the Gateway. However, this won’t allow me to enforce policies of any kind, because the gateway will just let traffic flow without inspecting it.

This is why I tried to setup ISTIO_MUTUAL as tls mode. This works perfectly with generic tcp traffic, but I would like to do it for TLS as well in order to rely on DNS resolution of domains (with generic tcp traffic I have to assign the ip addresses manually, which is not always sustainable. With TLS I should be able to rely on SNIs). So, the flow would look like the following: Pod X encrypts some traffic that is already encrypted (application encryption + mutual TLS), it sends it to the egress gateway, the egress gateway decrypts the mutual TLS layer (without trying to understand the content since it is encrypted), it decides whether to allow or deny such traffic (since now it is able to determine the source pod identity and AuthorizationPolicies can be used easily) and it sends the traffic to the intended (external) destination (if the source pod matches an allow policy).

Is this possible? The only reference that I found is Istio Prelim 1.11 / Consuming External MongoDB Services , and I followed it quite accurately without any success. It looks like the traffic does not even get to the egress gateway. Is this approach completely wrong and should TLS origination be used instead, all the times? What would be an alternative to filter encrypted traffic sent directly by a pod?

Here are some configuration details.
I am using istio 1.9.2 and I installed it with the istio operator.
I deployed an additional egress gateway with the IstioOperator:

    egressGateways:
      - enabled: true
        k8s:
          service:
            ports:
              - name: tls
                port: 443
        label:
          app: istio-egressgateway-tls
          istio: istio-egressgateway-tls
        name: istio-egressgateway-tls

This generates an associated deployment and a service:

apiVersion: v1
kind: Service
metadata:
  annotations:
  labels:
    app: istio-egressgateway-tls
    install.operator.istio.io/owning-resource: istiocontrolplane
    install.operator.istio.io/owning-resource-namespace: istio-system
    istio: istio-egressgateway-tls
    istio.io/rev: default
    operator.istio.io/component: EgressGateways
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.9.2
    release: istio
  name: istio-egressgateway-tls
  namespace: istio-system
spec:
  clusterIP: 10.33.7.154
  ports:
  - name: tls
    port: 443
    protocol: TCP
    targetPort: 443
  selector:
    app: istio-egressgateway-tls
    istio: istio-egressgateway-tls
  sessionAffinity: None
  type: ClusterIP

Here are all the manifests that I used to configure the egress gateway (it’s based on Istio / Egress Gateways with TLS Origination (File Mount) ):

---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: egress-gw-tls
  namespace: istio-system
  annotations:
    argocd.argoproj.io/sync-wave: "-1"
    argocd.argoproj.io/sync-options: SkipDryRunOnMissingResource=true
spec:
  selector:
    istio: istio-egressgateway-tls
  servers:
    - hosts:
        - www.google.com
      port:
        number: 443
        name: tls
        protocol: TLS
      tls:
        mode: ISTIO_MUTUAL

---
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: google-test
  namespace: istio-system
spec:
  host: istio-egressgateway-tls.istio-system.svc.cluster.local
  subsets:
    - name: google-test-subset
      trafficPolicy:
        portLevelSettings:
          - port:
              number: 443
            tls:
              mode: ISTIO_MUTUAL
              sni: www.google.com

---
apiVersion: networking.istio.io/v1beta1
kind: ServiceEntry
metadata:
  name: google-test
  namespace: istio-system
spec:
  hosts:
    - www.google.com
  ports:
    - number: 443
      name: tls
      protocol: TLS
  location: MESH_EXTERNAL
  resolution: DNS

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: google-test
  namespace: istio-system
spec:
  hosts:
    - www.google.com
  gateways:
    - mesh
    - istio-system/egress-gw-tls
  tls:
    - match:
        - gateways:
            - mesh
          port: 443
          sniHosts:
            - www.google.com
      route:
        - destination:
            host: istio-egressgateway-tls.istio-system.svc.cluster.local
            subset: google-test-subset
            port:
              number: 443
          weight: 100
  tcp:
    - match:
        - gateways:
            - istio-system/egress-gw-tls
          port: 443
      route:
        - destination:
            host: www.google.com
            port:
              number: 443
          weight: 100

---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: allow-all-tls
  namespace: istio-system
spec:
  rules:
    - {}
  selector:
    matchLabels:
      istio: istio-egressgateway-tls

Some Debugging info

I spawn a container with an istio sidecar that is able to forward traffic.
If I curl www.google.com:

root@command-demo-privileged-57c9c99d77-5sgrm:/# curl https://www.google.com
curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com:443

I see nothing in the egress gateway logs. In the command-demo-privileged-57c9c99d77-5sgrm sidecar logs I see:

{
  "response_code_details": null,
  "bytes_received": 0,
  "authn_source_principal": null,
  "authn_email": null,
  "downstream_local_address": "172.217.168.4:443",
  "start_time": "2021-06-11T14:53:38.606Z",
  "upstream_service_time": null,
  "authn_name": null,
  "bytes_sent": 0,
  "method": null,
  "request_id": null,
  "upstream_cluster": "outbound|443|google-test-subset|istio-egressgateway-tls.istio-system.svc.cluster.local",
  "downstream_remote_address": "10.32.13.216:35314",
  "protocol": null,
  "authority": null,
  "duration": 2,
  "authn_upn": null,
  "connection_termination_details": null,
  "upstream_local_address": null,
  "user_agent": null,
  "requested_server_name": null,
  "route_name": null,
  "x_forwarded_for": null,
  "upstream_host": "10.32.3.56:443",
  "response_flags": "UF,URX",
  "authn_request_auth_principal": null,
  "upstream_transport_failure_reason": null,
  "path": null,
  "response_code": 0
}

If I use telnet

root@command-demo-privileged-57c9c99d77-5sgrm:/# telnet www.google.com 443
Trying 172.217.168.4...
Connected to www.google.com.
Escape character is '^]'.

^CConnection closed by foreign host.

I see nothing in the egress gateway logs and in the sidecar container it’s pretty clear that traffic is being sent through the passthrough cluster:

{
  "downstream_local_address": "172.217.168.4:443",
  "bytes_received": 9,
  "authn_upn": null,
  "route_name": null,
  "method": null,
  "authn_name": null,
  "upstream_host": "172.217.168.4:443",
  "authn_email": null,
  "requested_server_name": null,
  "upstream_local_address": "10.32.13.216:39622",
  "upstream_service_time": null,
  "upstream_transport_failure_reason": null,
  "downstream_remote_address": "10.32.13.216:39256",
  "authn_source_principal": null,
  "protocol": null,
  "user_agent": null,
  "connection_termination_details": null,
  "x_forwarded_for": null,
  "start_time": "2021-06-11T14:55:29.970Z",
  "upstream_cluster": "PassthroughCluster",
  "bytes_sent": 7,
  "path": null,
  "request_id": null,
  "authn_request_auth_principal": null,
  "authority": null,
  "response_code": 0,
  "duration": 11333,
  "response_code_details": null,
  "response_flags": "-"
}

Unfortunately, this is as far as I can go. I don’t have additional ideas about things that I could check. Any pointer would be appreciated.

leitang · June 14, 2021, 6:06pm

Regarding to the error of “OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to www.google.com”, you may turn on curl verbose debugging flags to get more details of the error.

Topic		Replies	Views
mTLS origination for egress traffic with custom mTLS between client istio-proxy and egress gateway Security	1	3301	June 9, 2021
Decrypt tls traffic at istio gateways Security	0	230	November 6, 2023
Egress Gateways with TLS Origination AND TLS passthrough for egress chokepoint Networking security	1	2683	May 12, 2020
Egress TLS with TCP Networking	9	4298	June 7, 2023
Using Istio egress as HTTPS proxy for pods without sidecar	1	1204	February 9, 2022

Cannot intercept TLS traffic with istio egress gateway

Related Topics