NGINX ingress controller with ISTO proxy sidecar does not forward the traffic to upstream

Networking
1

We are having an internal NGNIX ingress controller, which has been injected with istio side-car proxy.
From Istio components usage, we don’t use istio gateway.
When STRICT mTLS for our application on namespace level via PeerAuthentication CR.
With this strict mTLS, inbound traffic to internal NGNIX ingress controller gets upstream connect error.
But the upstream service could be accessed directly.

With ingress pod annotation "traffic.sidecar.istio.io/includeInboundPorts: ", if we bypass the proxy it is working.
But we cannot bypass, as we have built few logics like rate limit, AuthorizationPolicy , RequestAuthentication.

Any idea/suggestions, how to come around this problem?
//
[2022-04-11T08:26:19.595Z] “GET /auth/realms/mxe HTTP/1.1” 503 UC upstream_reset_before_response_started{connection_termination} - “-” 0 95 0 - “127.0.0.6” “curl/7.66.0” “78f124bf-51aa-4656-b40f-2f562d2022ef” “mxe-commons-mxe-ingress-controller” “10.111.180.90:8080” PassthroughCluster 192.168.202.62:51832 10.111.180.90:8080 127.0.0.6:0 - allow_any
[2022-04-11T08:26:19.588Z] “GET /auth/realms/mxe HTTP/1.1” 503 - via_upstream - “-” 0 95 8 8 “-” “curl/7.66.0” “78f124bf-51aa-4656-b40f-2f562d2022ef” “mxe-commons-mxe-ingress-controller” “192.168.202.62:80” inbound|80|| 127.0.0.6:41971 192.168.202.62:80 192.168.202.39:44806 outbound_.80_..mxe-commons-mxe-ingress-controller.mxe.svc.cluster.local default
[2022-04-11T08:26:19.632Z] “GET /auth/realms/mxe HTTP/1.1” 503 UC upstream_reset_before_response_started{connection_termination} - “-” 0 95 0 - “127.0.0.6” “curl/7.66.0” “78f124bf-51aa-4656-b40f-2f562d2022ef” “mxe-commons-mxe-ingress-controller” “10.111.180.90:8080” PassthroughCluster 192.168.202.62:51842 10.111.180.90:8080 127.0.0.6:0 - allow_any
[2022-04-11T08:26:19.631Z] “GET /auth/realms/mxe HTTP/1.1” 503 - via_upstream - “-” 0 95 1 1 “-” “curl/7.66.0” “78f124bf-51aa-4656-b40f-2f562d2022ef” “mxe-commons-mxe-ingress-controller” “192.168.202.62:80” inbound|80|| 127.0.0.6:41971 192.168.202.62:80 192.168.202.39:53974 outbound.80_._.mxe-commons-mxe-ingress-controller.mxe.svc.cluster.local default
////

2

According to these threads (1) (2), the NGINX ingress controller does not use Services to route traffic to the pods. Instead it uses uses a list of all endpoints (Pod IP/port) in the NGINX upstream configuration in order to bypass kube-proxy to allow NGINX features like session affinity and custom load balancing algorithms.

Instead of routing the outbound traffic to the list of endpoints in the NGINX upstream configuration, you should configure NGINX ingress to route to a single upstream service, so that the outbound traffic is intercepted by the istio sidecar.

The nginx.ingress.kubernetes.io/service-upstream annotation disables that behavior and instead uses a single upstream in NGINX, the service’s Cluster IP and port.

The nginx.ingress.kubernetes.io/upstream-vhost annotation allows you to control the value for host in the following statement: proxy_set_header Host $host, which forms part of the location block.

The following annotations need to be added to each Ingress resource in order to make Istio work properly, so that:

  • Outbound traffic is intercepted by the istio sidecar
  • Traffic is sent to the right service entry.

So can you please add below to your Ingress resource as annotation -

nginx.ingress.kubernetes.io/service-upstream: "true"
nginx.ingress.kubernetes.io/upstream-vhost: <service-name>.<namespace>.svc.cluster.local

And don’t forget to mention the below labels to your Ingress Controller deployment manifest -

  traffic.sidecar.istio.io/includeInboundPorts: ""
  traffic.sidecar.istio.io/excludeInboundPorts: "443"
  sidecar.istio.io/inject: 'true'

Hope this helps!

1 Like