Egress Gateways with TLS Origination AND TLS passthrough for egress chokepoint

Some context:

We have an AWS EKS cluster, using the same VPC subnet as EC2 instances
In EC2, each component has it’s own security group, with default-deny on ingress

Now, we need to allow a workload in a pod access to a specific microservice running in EC2.

The trouble is, AWS doesn’t currently allow assigning a security group to a pod. So, our thought is to allow ingress from the EKS security group (that all worker nodes get assigned), and instead control access via egress restrictions in the istio cluster. Specifically, we’d force all egress traffic (http and https, but also tcp for e.g. RDS mysql/postgres) through an egress gateway, and control access via NetworkPolicy from the application pods to the egress pods. (side-note, I’m open to other options here if someone has come up with something better).

In researching, I found the article ‘Egress Gateways with TLS Origination’ (https://istio.io/docs/tasks/traffic-management/egress/egress-gateway-tls-origination/) promising. This is close, but it breaks https queries from app pods (I created an issue to track the problem: https://github.com/istio/istio/issues/23740). I recognize that changing applications to use http is the likely response to this, but this is not an option for some sensitive microservices.

I’ve manipulated some settings from that example to try to get outbound https to route through the egress gateway, but haven’t had much success. Any guidance/advice appreciated.

Current config:

---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: edition-cnn-com-egress
  namespace: istio-system
spec:
  host: istio-egressgateway
  subsets:
  - name: tls-origination
  - name: tls-passthrough
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: edition-cnn-com-origination
  namespace: istio-system
spec:
  host: edition.cnn.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: edition-cnn-com
  namespace: istio-system
spec:
  selector:
    istio: egressgateway
  servers:
  - hosts:
    - edition.cnn.com
    port:
      name: http-port-for-tls-origination
      number: 80
      protocol: HTTP
  - hosts:
    - edition.cnn.com
    port:
      name: tls-passthrough
      number: 334
      protocol: TLS
    tls:
      mode: PASSTHROUGH
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: edition-cnn-com
  namespace: istio-system
spec:
  hosts:
  - edition.cnn.com
  ports:
  - name: http
    number: 80
    protocol: HTTP
  - name: https
    number: 443
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: edition-cnn-com
  namespace: istio-system
spec:
  gateways:
  - edition-cnn-com
  - mesh
  hosts:
  - edition.cnn.com
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway
        port:
          number: 80
        subset: tls-origination
  - match:
    - gateways:
      - edition-cnn-com
      port: 80
    route:
    - destination:
        host: edition.cnn.com
        port:
          number: 443
      weight: 100
  tls:
  - match:
    - gateways:
      - mesh
      port: 443
      sniHosts:
      - edition.cnn.com
    route:
    - destination:
        host: istio-egressgateway
        port:
          number: 334
        subset: tls-passthrough

Update: Found another article talking about egress through a gateway: https://istio.io/docs/tasks/traffic-management/egress/wildcard-egress-hosts/#configure-egress-gateway-traffic-to-a-wildcard-host

That article is specifically looks at routing outbound https requests through a gateway. Merging those changes into my original config, I now have this:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: edition-cnn-com-egress
  namespace: istio-system
spec:
  host: istio-egressgateway
  subsets:
  - name: tls-origination
  - name: tls-passthrough
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: edition-cnn-com-origination
  namespace: istio-system
spec:
  host: edition.cnn.com
  trafficPolicy:
    loadBalancer:
      simple: ROUND_ROBIN
    portLevelSettings:
    - port:
        number: 443
      tls:
        mode: SIMPLE
---
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: edition-cnn-com
  namespace: istio-system
spec:
  selector:
    istio: egressgateway
  servers:
  - hosts:
    - edition.cnn.com
    port:
      name: http-port-for-tls-origination
      number: 80
      protocol: HTTP
  - hosts:
    - edition.cnn.com
    port:
      name: tls
      number: 443
      protocol: TLS
    tls:
      mode: PASSTHROUGH
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: edition-cnn-com
  namespace: istio-system
spec:
  hosts:
  - edition.cnn.com
  ports:
  - name: http
    number: 80
    protocol: HTTP
  - name: https
    number: 443
    protocol: HTTPS
  resolution: DNS
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: edition-cnn-com
  namespace: istio-system
spec:
  gateways:
  - edition-cnn-com
  - mesh
  hosts:
  - edition.cnn.com
  http:
  - match:
    - gateways:
      - mesh
      port: 80
    route:
    - destination:
        host: istio-egressgateway
        port:
          number: 80
        subset: tls-origination
  - match:
    - gateways:
      - edition-cnn-com
      port: 80
    route:
    - destination:
        host: edition.cnn.com
        port:
          number: 443
      weight: 100
  tls:
  - match:
    - gateways:
      - mesh
      port: 443
      sniHosts:
      - edition.cnn.com
    route:
    - destination:
        host: istio-egressgateway
        port:
          number: 443
        subset: tls-passthrough
      weight: 100
  - match:
    - gateways:
      - edition-cnn-com
      port: 443
      sniHosts:
      - edition.cnn.com
    route:
    - destination:
        host: edition.cnn.com
        port:
          number: 443
      weight: 100

proxy logs from the client pod:
[2020-05-12T15:52:17.862Z] "- - -" 0 - "-" "-" 517 0 10584 - "-" "-" "-" "-" "172.25.66.219:443" outbound|443|tls-passthrough|istio-egressgateway.istio-system.svc.cluster.local 172.25.69.92:56432 151.101.193.67:443 172.25.69.92:55882 edition.cnn.com -

so that’s progress - at least the request is getting routed to the gateway.

proxy logs from the egress gateway:
[2020-05-12T15:53:15.373Z] "- - -" 0 - "-" "-" 517 0 300123 - "-" "-" "-" "-" "151.101.129.67:443" outbound|443||edition.cnn.com 172.25.66.219:39316 172.25.66.219:443 172.25.69.92:56706 edition.cnn.com -

I’m still getting the 3 minute hang then timeout in the app pod.