Egress TLS with TCP

kunalekawde · June 15, 2020, 2:11pm

Objective:
To have the resources & certificates configured such that:

Plain TCP only traffic from application container to istio-proxy.
istio-proxy to egress g/w using mTLS
egress g/w to external TLS-TCP server.

For HTTPS traffic, I could get it working but since this is TCP with TLS, I’m not able to configure it end to end. I’ve following example on istio.io and consuming external service - mongo DB example as shared by @vadimeisenbergibm
Client application for testing is simple TCP client in python and external TCP server is simple python TCP with TLS server. Application pods are in same namespace as egress g/w pod.

I’ve following resources:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-mtls-for-tcpext
  namespace: test-ns
spec:
  host: tcpext.external.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 7777
      tls:
        caCertificates: /etc/fed/ca-certs/ca-chain.cert.pem
        clientCertificate: /etc/fed/certs/tls.crt
        mode: MUTUAL
        privateKey: /etc/fed/certs/tls.key
        sni: tcpext.external.com

---

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-tcpext
  namespace: test-ns
spec:
  exportTo:
  - '*'
  host: istio-egressgateway-ext.test-ns.svc.cluster.local
  subsets:
  - name: tcpext

---

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway-ext
  namespace: test-ns
spec:
  selector:
    istio: test-ns-egressgateway-ext
  servers:
  - hosts:
    - '*'
    port:
      name: tls
      number: 443
      protocol: TLS
    tls:
      caCertificates: /etc/certs/root-cert.pem
      mode: MUTUAL
      privateKey: /etc/certs/key.pem
      serverCertificate: /etc/certs/cert-chain.pem
  - hosts:
    - '*'
    port:
      name: tcp
      number: 7777
      protocol: TCP

---

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: tcpext
  namespace: test-ns
spec:
  addresses:
  - 10.15.10.158/32
  endpoints:
  - address: 10.15.10.158
  hosts:
  - tcpext.external.com
  location: MESH_EXTERNAL
  ports:
  - name: tls
    number: 7777
    protocol: TLS
  resolution: STATIC

---

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-tcpext-through-egress-gateway
  namespace: test-ns
spec:
  exportTo:
  - '*'
  gateways:
  - istio-egressgateway-ext
  - mesh
  hosts:
  - 10.15.10.158
  tcp:
  - match:
    - destinationSubnets:
      - 10.15.10.0/32
      gateways:
      - mesh
      port: 7777
    route:
    - destination:
        host: istio-egressgateway-ext.test-ns.svc.cluster.local
        port:
          number: 7777
        subset: tcpext
  - match:
    - gateways:
      - istio-egressgateway-ext
      port: 7777
    route:
    - destination:
        host: tcpext.external.com
        port:
          number: 7777
      weight: 100

I don’t see message coming to egress proxy itself with above config.
Following log at istio-proxy sidecar for app pod:
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][filter] [external/envoy/source/extensions/filters/listener/original_dst/original_dst.cc:18] original_dst: New connection accepted
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][filter] [src/envoy/tcp/mixer/filter.cc:30] Called tcp filter: Filter
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][filter] [src/envoy/tcp/mixer/filter.cc:40] Called tcp filter: initializeReadFilterCallbacks
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][filter] [external/envoy/source/common/tcp_proxy/tcp_proxy.cc:204] [C1434755] new tcp proxy session
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][filter] [src/envoy/tcp/mixer/filter.cc:133] [C1434755] Called tcp filter onNewConnection: remote 192.168.148.97:49756, local 10.15.10.158:7777
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][connection] [external/envoy/source/common/network/connection_impl.cc:104] [C1434755] closingdata_to_write=0 type=1
[Envoy (Epoch 0)] [2020-06-15 13:14:42.626][23][debug][connection] [external/envoy/source/common/network/connection_impl.cc:193] [C1434755] closingsocket: 1
[2020-06-15T13:14:35.012Z] “- - -” 0 NR “-” “-” 0 0 0 - “-” “-” “-” “-” “-” - - 10.15.10.158:7777 192.168.148.97:49490 - -

vadimeisenbergibm · June 16, 2020, 2:03am

Note that for TCP traffic you have to specify the port’s protocol as TCP. So in your case you probably need two ServiceEntries: one with addresses and the port specified as TCP, another one with the host and the port specified as TLS.

Also note that currently mTLS between Istio proxy and gateway is broken, see https://github.com/istio/istio/issues/23910

kunalekawde · June 18, 2020, 6:22pm

Thanks @vadimeisenbergibm for checking this.
Another thing to note is that external host: tcpext.external dot com is not DNS resolved. Here we know only the IP, the host used here is dummy as the application would directly open socket to external IP and Port and communicate.

Few updates to skip istio mTLS between proxy and egress and 2 service entries.

---

apiVersion: networking.istio .io/v1alpha3
kind: Gateway
metadata:
  name: istio-egressgateway-ext
  namespace: test-ns
spec:
  selector:
    istio: test-ns-egressgateway-ext
  servers:
  - hosts:
    - '*'
    port:
      name: tcp
      number: 8888
      protocol: TCP

---

apiVersion: networking.istio. io/v1alpha3
kind: VirtualService
metadata:
  name: direct-tcpext-through-egress-gateway
  namespace: test-ns
spec:
  gateways:
  - istio-egressgateway-ext
  - mesh
  hosts:
  - 10.15.10.236
  tcp:
  - match:
    - gateways:
      - mesh
      port: 7777
    route:
    - destination:
        host: istio-egressgateway-ext.test-ns.svc.cluster.local
        port:
          number: 8888
        subset: tcpext
  - match:
    - gateways:
      - istio-egressgateway-ext
      port: 8888
    route:
    - destination:
        host: tcpext.external.com
        port:
          number: 7777
      weight: 100

---

apiVersion: networking.istio. io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-tcpext
  namespace: test-ns
spec:
  host: istio-egressgateway-ext.test-ns.svc.cluster.local
  subsets:
  - name: tcpext

---

apiVersion: networking.istio. io/v1alpha3
kind: DestinationRule
metadata:
  name: originate-mtls-for-tcpext
  namespace: test-ns
spec:
  host: tcpext.external.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 7777
      tls:
        caCertificates: /etc/fed/ca-certs/ca-chain.cert.pem
        clientCertificate: /etc/fed/certs/tls.crt
        mode: MUTUAL
        privateKey: /etc/fed/certs/tls.key
        sni: tcpext.external.com

---

apiVersion: networking.istio. io/v1alpha3
kind: ServiceEntry
metadata:
  name: mdf2-tls-236
  namespace: test-ns
spec:
  addresses:
  - 10.15.10.236
  endpoints:
  - address: 10.15.10.236
  hosts:
  - tcpext.external.com
  location: MESH_EXTERNAL
  ports:
  - name: tcp
    number: 7777
    protocol: TCP
  resolution: STATIC

---

apiVersion: networking.istio. io/v1alpha3
kind: ServiceEntry
metadata:
  name: tcpext-tls
  namespace: test-ns
spec:
  hosts:
  - tcpext.external dot com  # new user limitation to put only 4 links, read as . com
  location: MESH_EXTERNAL
  ports:
  - name: tls
    number: 7777
    protocol: TLS

---

On above though local istio-proxy is not forwarding to egress and trying to route directly.

vadimeisenbergibm · June 18, 2020, 7:51pm

First thing that I see - the hosts in the Service Entry and VIrtual Service must match (even if they are dummy hosts).

I wrote a blog post in 2018 about directing TCP traffic through the egress gateway https://istio.io/latest/blog/2018/egress-mongo/#direct-tcp-egress-traffic-through-an-egress-gateway.

I am not sure if instructions in the blog post are up to date, though. So please change the host in VirtualEntry to match the host in the Service Entry - hopefully the sidecar proxy will start forwarding traffic to the Egress Gateway.

kunalekawde · June 19, 2020, 7:02am

I tried this change of specifying the host in virtualhost that which is used in serviceentry. Also matched the destinationSubnet field as in example and also updated gateway with same host, still no luck.

---

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: mdf2-tls-236
  namespace: test-ns
spec:
  addresses:
  - 10.15.10.236/32
  endpoints:
  - address: 10.15.10.236
  hosts:
  - tcpext.external.com
  location: MESH_EXTERNAL
  ports:
  - name: tcp
    number: 7777
    protocol: TCP
  resolution: STATIC

---

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-tcpext-through-egress-gateway
  namespace: test-ns
spec:
  gateways:
  - istio-egressgateway-ext
  - mesh
  hosts:
  - tcpext.external.com
  tcp:
  - match:
- destinationSubnets:
  - 10.15.10.236/32
  gateways:
  - mesh
  port: 7777
route:
- destination:
    host: istio-egressgateway-ext.test-ns.svc.cluster.local
    port:
      number: 8888
    subset: tcpext
  - match:
- gateways:
  - istio-egressgateway-ext
  port: 8888
route:
- destination:
    host: tcpext.external.com
    port:
      number: 7777
  weight: 100

---

istio-proxy logs:

   [Envoy (Epoch 0)] [2020-06-19 06:33:10.803][26][debug][filter] [src/envoy/tcp/mixer/filter.cc:133] [C1681] Called tcp filter onNewConnection: remote 192.168.148.124:54018, local 10.15.10.236:7777
    [Envoy (Epoch 0)] [2020-06-19 06:33:10.803][26][debug][connection] [external/envoy/source/common/network/connection_impl.cc:104] [C1681] closing data_to_write=0 type=1
    [Envoy (Epoch 0)] [2020-06-19 06:33:10.803][26][debug][connection] [external/envoy/source/common/network/connection_impl.cc:193] [C1681] closing socket: 1
    [Envoy (Epoch 0)] [2020-06-19 06:33:10.803][26][debug][filter] [src/envoy/tcp/mixer/filter.cc:177] [C1681] Called tcp filter onEvent: 1
    [Envoy (Epoch 0)] [2020-06-19 06:33:10.803][26][debug][filter] [src/istio/control/client_context_base.cc:139] Report attributes: attributes {
      key: "connection.duration"
      value {
        duration_value {
          nanos: 137000
        }
      }
    }
    attributes {
      key: "connection.event"
      value {
        string_value: "close"
      }
    }
    att
    [Envoy (Epoch 0)] [2020-06-19 06:33:10.803][26][debug][filter] [src/envoy/tcp/mixer/filter.cc:35] Called tcp filter : ~Filter

Listeners:
At app pod:
10.15.10.236 7777 TCP

At egress g/w:
0.0.0.0 8888 TCP

Note that here client is basic TCP python3 client which opens a socket to (10.15.10.236:7777) which has to go via istio-proxy → egress - TLS → external python tcp tls server running on 10.15.10.236:7777

kunalekawde · June 19, 2020, 3:19pm

Following listener config i see for istio-proxy of application pod

{
    "name": "10.15.10.236_7777",
    "address": {
        "socketAddress": {
            "address": "10.15.10.236",
            "portValue": 7777
        }
    },
    "filterChains": [
        {
            "filterChainMatch": {
                "prefixRanges": [
                    {
                        "addressPrefix": "10.15.10.236",
                        "prefixLen": 32
                    }
                ]
            },
            "filters": [
                {
                    "name": "mixer",
                    "typedConfig": {
                        "@type": "type.googleapis.com/istio.mixer.v1.config.client.TcpClientConfig",
                        "transport": {
                            "networkFailPolicy": {
                                "policy": "FAIL_CLOSE",
                                "baseRetryWait": "0.080s",
                                "maxRetryWait": "1s"
                            },
                            "checkCluster": "outbound|9091||istio-policy.fed-istio.svc.cluster.local",
                            "reportCluster": "outbound|9091||istio-telemetry.fed-istio.svc.cluster.local",
                            "reportBatchMaxEntries": 100,
                            "reportBatchMaxTime": "1s"
                        },
                        "mixerAttributes": {
                            "attributes": {
                                "context.proxy_version": {
                                    "stringValue": "1.4.2"
                                },
                                "context.reporter.kind": {
                                    "stringValue": "outbound"
                                },
                                "context.reporter.uid": {
                                    "stringValue": "kubernetes://hello-world-1-0-0-220-main-int-dbg-7c4c9f7d57-jxgwl.test-ns"
                                },
                                "destination.service.host": {
                                    "stringValue": "tcpext.external.com"
                                },
                                "destination.service.name": {
                                    "stringValue": "tcpext.external.com"
                                },
                                "destination.service.namespace": {
                                    "stringValue": "test-ns"
                                },
                                "source.namespace": {
                                    "stringValue": "test-ns"
                                },
                                "source.uid": {
                                    "stringValue": "kubernetes://hello-world-1-0-0-220-main-int-dbg-7c4c9f7d57-jxgwl.test-ns"
                                }
                            }
                        },
                        "disableCheckCalls": true
                    }
                },
                {
                    "name": "envoy.tcp_proxy",
                    "typedConfig": {
                        "@type": "type.googleapis.com/envoy.config.filter.network.tcp_proxy.v2.TcpProxy",
                        "statPrefix": "outbound|8888|tcpext|istio-egressgateway-ext.test-ns.svc.cluster.local",
                        "cluster": "outbound|8888|tcpext|istio-egressgateway-ext.test-ns.svc.cluster.local",
                        "accessLog": [
                            {
                                "name": "envoy.file_access_log",
                                "typedConfig": {
                                    "@type": "type.googleapis.com/envoy.config.accesslog.v2.FileAccessLog",
                                    "path": "/dev/stdout",
                                    "format": "[%START_TIME%] \"%REQ(:METHOD)% %REQ(X-ENVOY-ORIGINAL-PATH?:PATH)% %PROTOCOL%\" %RESPONSE_CODE% %RESPONSE_FLAGS% \"%DYNAMIC_METADATA(istio.mixer:status)%\" \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\" %BYTES_RECEIVED% %BYTES_SENT% %DURATION% %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% \"%REQ(X-FORWARDED-FOR)%\" \"%REQ(USER-AGENT)%\" \"%REQ(X-REQUEST-ID)%\" \"%REQ(:AUTHORITY)%\" \"%UPSTREAM_HOST%\" %UPSTREAM_CLUSTER% %UPSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_LOCAL_ADDRESS% %DOWNSTREAM_REMOTE_ADDRESS% %REQUESTED_SERVER_NAME% %ROUTE_NAME%\n"
                                }
                            }
                        ]
                    }
                }
            ]
        },

kunalekawde · June 19, 2020, 9:07pm

From the routes it appeared that istio-egressgateway-ext.test-ns.svc.cluster.local needed a port 8888 to be configured, did that and its getting on egress g/w now. Currently getting no healthyhost for TCP connection pool, must be serviceentry config.

kunalekawde · June 19, 2020, 9:45pm

Yeah looks like I could get it working, dropped the second tls serviceentry. Thanks @vadimeisenbergibm for all the help, appreciate your support.

vadimeisenbergibm · June 21, 2020, 4:27am

@kunalekawde Great, I am happy to hear that!

kaitaklam · June 7, 2023, 2:38pm

@kunalekawde Do you hvae the complete set of yaml files?

Topic		Replies	Views
Egress tls originated from istio-proxy to external service for a tcp protocol	1	329	October 19, 2023
Mutual TLS origination by Egress Gateway Security	2	772	April 12, 2019
mTLS origination for egress traffic with custom mTLS between client istio-proxy and egress gateway Security	1	3371	June 9, 2021
Direct Egress TLS for TCP Networking	1	1003	July 12, 2020
gRPC+mTLS through Egress GW Networking	0	1632	October 14, 2019

Egress TLS with TCP

I’ve following resources:

Related topics