503 Error when accessing External Service via Egress Gateway

1

Having some weird issues I’m hoping someone can help with:

Istio 1.6.x
Kubernetes GKE 1.16 (private cluster)
Shared Control Plane Topology
auto mTLS enabled

We are trying to route to a Non-Mesh service defined with a service entry via an egress gateway. The service is deployed on one of the clusters, but for “reasons” cant be joined to the mesh. However, other mesh services need to be able to access it, preferably via an egress gateway (so we have a pattern for external services as well)

When we generate the ServiceEntry on its own, I can see traffic flows correctly on the same cluster. As soon as we add a DestinationRule, Gateway and VirtualService, while it is able to resolve the DNS, we get 503 errors.
when running the following from an nginx container in a istio-enabled namespace
curl -vI http://mynginx.ns.svc.cluster.local:PORT

I see the following result:
Trying 10.174.1.133 Connected to my-nginx.test-testing.svc.cluster.local (10.174.1.133) port 8039 (#0) >HTTP/1.1 503 Service Unavailable

Can anyone shed any light on why or have any troubleshooting pointers?

I’ve checked the routes on the egress-gateway pods using istioctl pc routes istio-egressgateway-xxxx -n istio-system -o json and can see a virtualHosts for blackhole:portnumber which I assume means the route is in place?

```apiVersion: networking.istio.io/v1alpha3
kind: ServiceEntry
metadata:
  name: mynginx
spec:
  hosts:
  - my-nginx.temp-testing.svc.cluster.local
  ports:
  - number: 8039
    name: http-port
    protocol: HTTP
  resolution: DNS
  location: MESH_EXTERNAL
---
  apiVersion: networking.istio.io/v1alpha3
  kind: Gateway
  metadata: 
    name: local-egress
  spec:
    selector:
      istio: egressgateway
    servers:
    - port:
        number: 8039
        name: http
        protocol: HTTP
      hosts:
      - my-nginx.temp-testing.svc.cluster.local
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: egress-for-mynginx
spec:
  host: istio-egressgateway.istio-system.svc.cluster.local
  subsets:
    - name: mynginx
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata: 
  name: direct-mynginx-via-gw
spec:
  hosts:
  - my-nginx.temp-testing.svc.cluster.local
  gateways:
  - istio-egressgateway
  - mesh
  http:
  - match:
    - gateways:
      - mesh
      port: 8039
    route: 
    - destination: 
        host: istio-egressgateway.istio-system.svc.cluster.local
        subset: mynginx
        port:
          number: 8039
      weight: 100
  - match:
    - gateways:
      - istio-egressgateway
      port: 8039
    route:
    - destination:
        host: my-nginx.temp-testing.svc.cluster.local
        port:
          number: 8039
      weight: 100
```