Egress TLS with TCP

Objective:
To have the resources & certificates configured such that:

  1. Plain TCP only traffic from application container to istio-proxy.
  2. istio-proxy to egress g/w using mTLS
  3. egress g/w to external TLS-TCP server.

For HTTPS traffic, I could get it working but since this is TCP with TLS, I’m not able to configure it end to end. I’ve following example on istio.io and consuming external service - mongo DB example as shared by @vadimeisenbergibm
Client application for testing is simple TCP client in python and external TCP server is simple python TCP with TLS server. Application pods are in same namespace as egress g/w pod.

I’ve following resources:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-mtls-for-tcpext
  namespace: test-ns
spec:
  host: tcpext.external.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 7777
      tls:
        caCertificates: /etc/fed/ca-certs/ca-chain.cert.pem
        clientCertificate: /etc/fed/certs/tls.crt
        mode: MUTUAL
        privateKey: /etc/fed/certs/tls.key
        sni: tcpext.external.com

---

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-tcpext
  namespace: test-ns
spec:
  exportTo:
  - '*'
  host: istio-egressgateway-ext.test-ns.svc.cluster.local
  subsets:
  - name: tcpext

---

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway-ext
  namespace: test-ns
spec:
  selector:
    istio: test-ns-egressgateway-ext
  servers:
  - hosts:
    - '*'
    port:
      name: tls
      number: 443
      protocol: TLS
    tls:
      caCertificates: /etc/certs/root-cert.pem
      mode: MUTUAL
      privateKey: /etc/certs/key.pem
      serverCertificate: /etc/certs/cert-chain.pem
  - hosts:
    - '*'
    port:
      name: tcp
      number: 7777
      protocol: TCP

---

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: tcpext
  namespace: test-ns
spec:
  addresses:
  - 10.15.10.158/32
  endpoints:
  - address: 10.15.10.158
  hosts:
  - tcpext.external.com
  location: MESH_EXTERNAL
  ports:
  - name: tls
    number: 7777
    protocol: TLS
  resolution: STATIC

---

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-tcpext-through-egress-gateway
  namespace: test-ns
spec:
  exportTo:
  - '*'
  gateways:
  - istio-egressgateway-ext
  - mesh
  hosts:
  - 10.15.10.158
  tcp:
  - match:
    - destinationSubnets:
      - 10.15.10.0/32
      gateways:
      - mesh
      port: 7777
    route:
    - destination:
        host: istio-egressgateway-ext.test-ns.svc.cluster.local
        port:
          number: 7777
        subset: tcpext
  - match:
    - gateways:
      - istio-egressgateway-ext
      port: 7777
    route:
    - destination:
        host: tcpext.external.com
        port:
          number: 7777
      weight: 100

I don’t see message coming to egress proxy itself with above config.
Following log at istio-proxy sidecar for app pod:
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][filter] [external/envoy/source/extensions/filters/listener/original_dst/original_dst.cc:18] original_dst: New connection accepted
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][filter] [src/envoy/tcp/mixer/filter.cc:30] Called tcp filter: Filter
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][filter] [src/envoy/tcp/mixer/filter.cc:40] Called tcp filter: initializeReadFilterCallbacks
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][filter] [external/envoy/source/common/tcp_proxy/tcp_proxy.cc:204] [C1434755] new tcp proxy session
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][filter] [src/envoy/tcp/mixer/filter.cc:133] [C1434755] Called tcp filter onNewConnection: remote 192.168.148.97:49756, local 10.15.10.158:7777
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][connection] [external/envoy/source/common/network/connection_impl.cc:104] [C1434755] closingdata_to_write=0 type=1
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][connection] [external/envoy/source/common/network/connection_impl.cc:193] [C1434755] closingsocket: 1
[2020-06-15T13:14:35.012Z] “- - -” 0 NR “-” “-” 0 0 0 - “-” “-” “-” “-” “-” - - 10.15.10.158:7777 192.168.148.97:49490 - -

Note that for TCP traffic you have to specify the port’s protocol as TCP. So in your case you probably need two ServiceEntries: one with addresses and the port specified as TCP, another one with the host and the port specified as TLS.

Also note that currently mTLS between Istio proxy and gateway is broken, see https://github.com/istio/istio/issues/23910

Thanks @vadimeisenbergibm for checking this.
Another thing to note is that external host: tcpext.external dot com is not DNS resolved. Here we know only the IP, the host used here is dummy as the application would directly open socket to external IP and Port and communicate.

Few updates to skip istio mTLS between proxy and egress and 2 service entries.

---

apiVersion: networking.istio .io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway-ext
  namespace: test-ns
spec:
  selector:
    istio: test-ns-egressgateway-ext
  servers:
  - hosts:
    - '*'
    port:
      name: tcp
      number: 8888
      protocol: TCP

---

apiVersion: networking.istio. io/v1alpha3
kind: VirtualService
metadata:
  name: direct-tcpext-through-egress-gateway
  namespace: test-ns
spec:
  gateways:
  - istio-egressgateway-ext
  - mesh
  hosts:
  - 10.15.10.236
  tcp:
  - match:
    - gateways:
      - mesh
      port: 7777
    route:
    - destination:
        host: istio-egressgateway-ext.test-ns.svc.cluster.local
        port:
          number: 8888
        subset: tcpext
  - match:
    - gateways:
      - istio-egressgateway-ext
      port: 8888
    route:
    - destination:
        host: tcpext.external.com
        port:
          number: 7777
      weight: 100

---

apiVersion: networking.istio. io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-tcpext
  namespace: test-ns
spec:
  host: istio-egressgateway-ext.test-ns.svc.cluster.local
  subsets:
  - name: tcpext

---

apiVersion: networking.istio. io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-mtls-for-tcpext
  namespace: test-ns
spec:
  host: tcpext.external.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 7777
      tls:
        caCertificates: /etc/fed/ca-certs/ca-chain.cert.pem
        clientCertificate: /etc/fed/certs/tls.crt
        mode: MUTUAL
        privateKey: /etc/fed/certs/tls.key
        sni: tcpext.external.com

---

apiVersion: networking.istio. io/v1alpha3
kind: ServiceEntry
metadata:
  name: mdf2-tls-236
  namespace: test-ns
spec:
  addresses:
  - 10.15.10.236
  endpoints:
  - address: 10.15.10.236
  hosts:
  - tcpext.external.com
  location: MESH_EXTERNAL
  ports:
  - name: tcp
    number: 7777
    protocol: TCP
  resolution: STATIC

---

apiVersion: networking.istio. io/v1alpha3
kind: ServiceEntry
metadata:
  name: tcpext-tls
  namespace: test-ns
spec:
  hosts:
  - tcpext.external dot com  # new user limitation to put only 4 links, read as . com
  location: MESH_EXTERNAL
  ports:
  - name: tls
    number: 7777
    protocol: TLS

---

On above though local istio-proxy is not forwarding to egress and trying to route directly.

First thing that I see - the hosts in the Service Entry and VIrtual Service must match (even if they are dummy hosts).

I wrote a blog post in 2018 about directing TCP traffic through the egress gateway https://istio.io/latest/blog/2018/egress-mongo/#direct-tcp-egress-traffic-through-an-egress-gateway.

I am not sure if instructions in the blog post are up to date, though. So please change the host in VirtualEntry to match the host in the Service Entry - hopefully the sidecar proxy will start forwarding traffic to the Egress Gateway.

I tried this change of specifying the host in virtualhost that which is used in serviceentry. Also matched the destinationSubnet field as in example and also updated gateway with same host, still no luck.

---

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: mdf2-tls-236
  namespace: test-ns
spec:
  addresses:
  - 10.15.10.236/32
  endpoints:
  - address: 10.15.10.236
  hosts:
  - tcpext.external.com
  location: MESH_EXTERNAL
  ports:
  - name: tcp
    number: 7777
    protocol: TCP
  resolution: STATIC

---

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-tcpext-through-egress-gateway
  namespace: test-ns
spec:
  gateways:
  - istio-egressgateway-ext
  - mesh
  hosts:
  - tcpext.external.com
  tcp:
  - match:
- destinationSubnets:
  - 10.15.10.236/32
  gateways:
  - mesh
  port: 7777
route:
- destination:
    host: istio-egressgateway-ext.test-ns.svc.cluster.local
    port:
      number: 8888
    subset: tcpext
  - match:
- gateways:
  - istio-egressgateway-ext
  port: 8888
route:
- destination:
    host: tcpext.external.com
    port:
      number: 7777
  weight: 100

---

istio-proxy logs:

   [Envoy (Epoch 0)] [2020-06-19 06:33:10.803][26][debug][filter] [src/envoy/tcp/mixer/filter.cc:133] [C1681] Called tcp filter onNewConnection: remote 192.168.148.124:54018, local 10.15.10.236:7777
    [Envoy (Epoch 0)] [2020-06-19 06:33:10.803][26][debug][connection] [external/envoy/source/common/network/connection_impl.cc:104] [C1681] closing data_to_write=0 type=1
    [Envoy (Epoch 0)] [2020-06-19 06:33:10.803][26][debug][connection] [external/envoy/source/common/network/connection_impl.cc:193] [C1681] closing socket: 1
    [Envoy (Epoch 0)] [2020-06-19 06:33:10.803][26][debug][filter] [src/envoy/tcp/mixer/filter.cc:177] [C1681] Called tcp filter onEvent: 1
    [Envoy (Epoch 0)] [2020-06-19 06:33:10.803][26][debug][filter] [src/istio/control/client_context_base.cc:139] Report attributes: attributes {
      key: "connection.duration"
      value {
        duration_value {
          nanos: 137000
        }
      }
    }
    attributes {
      key: "connection.event"
      value {
        string_value: "close"
      }
    }
    att
    [Envoy (Epoch 0)] [2020-06-19 06:33:10.803][26][debug][filter] [src/envoy/tcp/mixer/filter.cc:35] Called tcp filter : ~Filter

Listeners:
At app pod:
10.15.10.236 7777 TCP

At egress g/w:
0.0.0.0 8888 TCP

Note that here client is basic TCP python3 client which opens a socket to (10.15.10.236:7777) which has to go via istio-proxy -> egress - TLS --> external python tcp tls server running on 10.15.10.236:7777

Following listener config i see for istio-proxy of application pod

{
    "name": "10.15.10.236_7777",
    "address": {
        "socketAddress": {
            "address": "10.15.10.236",
            "portValue": 7777
        }
    },
    "filterChains": [
        {
            "filterChainMatch": {
                "prefixRanges": [
                    {
                        "addressPrefix": "10.15.10.236",
                        "prefixLen": 32
                    }
                ]
            },
            "filters": [
                {
                    "name": "mixer",
                    "typedConfig": {
                        "@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
                        "transport": {
                            "networkFailPolicy": {
                                "policy": "FAIL_CLOSE",
                                "baseRetryWait": "0.080s",
                                "maxRetryWait": "1s"
                            },
                            "checkCluster": "outbound|9091||istio-policy.fed-istio.svc.cluster.local",
                            "reportCluster": "outbound|9091||istio-telemetry.fed-istio.svc.cluster.local",
                            "reportBatchMaxEntries": 100,
                            "reportBatchMaxTime": "1s"
                        },
                        "mixerAttributes": {
                            "attributes": {
                                "context.proxy_version": {
                                    "stringValue": "1.4.2"
                                },
                                "context.reporter.kind": {
                                    "stringValue": "outbound"
                                },
                                "context.reporter.uid": {
                                    "stringValue": "kubernetes://hello-world-1-0-0-220-main-int-dbg-7c4c9f7d57-jxgwl.test-ns"
                                },
                                "destination.service.host": {
                                    "stringValue": "tcpext.external.com"
                                },
                                "destination.service.name": {
                                    "stringValue": "tcpext.external.com"
                                },
                                "destination.service.namespace": {
                                    "stringValue": "test-ns"
                                },
                                "source.namespace": {
                                    "stringValue": "test-ns"
                                },
                                "source.uid": {
                                    "stringValue": "kubernetes://hello-world-1-0-0-220-main-int-dbg-7c4c9f7d57-jxgwl.test-ns"
                                }
                            }
                        },
                        "disableCheckCalls": true
                    }
                },
                {
                    "name": "envoy.tcp_proxy",
                    "typedConfig": {
                        "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                        "statPrefix": "outbound|8888|tcpext|istio-egressgateway-ext.test-ns.svc.cluster.local",
                        "cluster": "outbound|8888|tcpext|istio-egressgateway-ext.test-ns.svc.cluster.local",
                        "accessLog": [
                            {
                                "name": "envoy.file_access_log",
                                "typedConfig": {
                                    "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                                    "path": "/dev/stdout",
                                    "format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
                                }
                            }
                        ]
                    }
                }
            ]
        },

From the routes it appeared that istio-egressgateway-ext.test-ns.svc.cluster.local needed a port 8888 to be configured, did that and its getting on egress g/w now. Currently getting no healthyhost for TCP connection pool, must be serviceentry config.

Yeah looks like I could get it working, dropped the second tls serviceentry. Thanks @vadimeisenbergibm for all the help, appreciate your support.

@kunalekawde Great, I am happy to hear that!