TCP egress gateway to AWS RDS cluster

Networking
1

Hi

I’m trying to setup an egress gateway for pods to access a mysql database hosted in Amazon RDS.
I started following the example @ https://istio.io/blog/2018/egress-mongo/#verify-that-egress-traffic-is-directed-through-the-egress-gateway and adapted it to my reality. The bulk of changes is that the endpoint does not have a fixed IP address, but rather three subnets within our VPC.
I started by adding the service entry as:

apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: mysql-external
  namespace: truota
spec:
  hosts:
  - mysql.rds.svc
  addresses:
  - 10.63.30.0/24
#  - 10.63.29.0/24
#  - 10.63.28.0/24
  ports:
  - name: tcp
    number: 3306
    protocol: TCP
  exportTo: 
  - "*"
  location: MESH_EXTERNAL
  resolution: NONE
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: mysql-external-2
  namespace: truota
spec:
  hosts:
  - mysql2.rds.svc
  addresses:
#  - 10.63.30.0/24
  - 10.63.29.0/24
#  - 10.63.28.0/24
  ports:
  - name: tcp
    number: 3306
    protocol: TCP
  exportTo: 
  - "*"
  location: MESH_EXTERNAL
  resolution: NONE
---
apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: mysql-external-3
  namespace: truota
spec:
  hosts:
  - mysql3.rds.svc
  addresses:
#  - 10.63.30.0/24
#  - 10.63.29.0/24
  - 10.63.28.0/24
  ports:
  - name: tcp
    number: 3306
    protocol: TCP
  exportTo: 
  - "*"
  location: MESH_EXTERNAL
  resolution: NONE

With this setup, the pods can access the services with no problem:

[2019-07-31T09:30:50.342Z] "- - -" 0 UF,URX "-" "-" 0 0 10001 - "-" "-" "-" "-" "10.63.30.92:3306" outbound|3306||mysql.rds.svc - 10.63.30.92:3306 10.63.26.235:52042 -

Then I went and created the egress resources (I have an egress gateway controller in a separate namespace)

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: mysql-gateway
  namespace: istio-egressgateway
spec:
  selector:
    istio: egressgateway
  servers:
  - port:
      number: 7777
      name: tcp
      protocol: TCP
    hosts:
    - mysql.rds.svc
    - mysql2.rds.svc
    - mysql3.rds.svc
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egressgateway-for-mysql
  namespace: truota
spec:
  host: istio-egressgateway.istio-egressgateway.svc.cluster.local
  exportTo: 
  - "*"
  subsets:
  - name: mysql
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: mysql
  namespace: truota
spec:
  host: mysql.rds.svc
  exportTo: 
  - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: mysql2
  namespace: truota
spec:
  host: mysql2.rds.svc
  exportTo: 
  - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: mysql3
  namespace: truota
spec:
  host: mysql3.rds.svc
  exportTo: 
  - "*"
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: direct-mysql-through-egress-gateway
  namespace: truota
spec:
  exportTo:
  - "*"
  hosts:
  - mysql.rds.svc
  - mysql2.rds.svc
  - mysql3.rds.svc
  gateways:
  - mesh
  - istio-egressgateway/mysql-gateway
  tcp:
  - match:
    - gateways:
      - mesh
      destinationSubnets:
      - 10.63.30.0/24
#      - 10.63.29.0/24
#      - 10.63.28.0/24
      port: 3306
    route:
    - destination:
        host: istio-egressgateway.istio-egressgateway.svc.cluster.local
        subset: mysql
        port:
          number: 7777
  - match:
    - gateways:
      - mesh
      destinationSubnets:
#      - 10.63.30.0/24
      - 10.63.29.0/24
#      - 10.63.28.0/24
      port: 3306
    route:
    - destination:
        host: istio-egressgateway.istio-egressgateway.svc.cluster.local
        subset: mysql
        port:
          number: 7777
  - match:
    - gateways:
      - mesh
      destinationSubnets:
#      - 10.63.30.0/24
#      - 10.63.29.0/24
      - 10.63.28.0/24
      port: 3306
    route:
    - destination:
        host: istio-egressgateway.istio-egressgateway.svc.cluster.local
        subset: mysql
        port:
          number: 7777
  - match:
    - gateways:
      - istio-egressgateway/mysql-gateway
      port: 7777
    route:
    - destination:
        host: stag-eks-mysql-services.cjqornbhu9ah.eu-west-1.rds.amazonaws.com
        port:
          number: 3306
      weight: 100

After this, the pods route traffic correctly to the egress gateway:

[2019-07-31T11:29:53.105Z] "- - -" 0 - "-" "-" 0 0 1 - "-" "-" "-" "-" "10.63.17.36:7777" outbound|7777|mysql|istio-egressgateway.istio-egressgateway.svc.cluster.local 10.63.26.235:58616 10.63.30.92:3306 10.63.26.235:48024 -

But the egress gateway controller does not send the traffic to the outside service:

[2019-07-31T11:29:53.106Z] "- - -" 0 NR "-" "-" 0 0 0 - "-" "-" "-" "-" "-" - - 10.63.17.36:7777 10.63.26.235:58616 -

Has anyone come across this issue? Did you manage to solve it?

Thanks

1 Like
2

I would start with a single subnet first. If it works, I would define three gateways and see if it works. Then I would try to combine the gateways and the virtual services into one.

3

Got it working. The problem was at the virtual service created in the “truota” namespace. Although I used the exportTo stanza with “*”, the routing did not become available to the “istio-egressgateway” namespace (where the actual controller resides).
Given that, the gateway controller had no route to the database. I ended up splitting the virtual service in two:

  • one in the “truota” namespace that routed mesh traffic to the egress gateway
  • one in the “istio-egressgateway” that routed port 7777 traffic to the db
4

Cool, glad to hear it worked! Recently I tried two virtual services for HTTP egress traffic control and it worked for me too.