AWS Nlb HealthCheck fails for TargetGroup when ExternalTrafficPolicy is Local

Networking
1

Running Istio 1.19.5 with gateway chart of same version on a 1.27 aws eks cluster when configuring the gateway chart to have an ExternalTrafficPolicy of Local then the healthchecks of the corresponding aws nlb fail even though the services are running on the same node as the gateway. How can I investigate further as to what the issue maybe?

2

I’m running similar setup with Istio 1.15 and it works fine:

apiVersion: v1
kind: Service
metadata:
  name: udp-proto-srv
  namespace: udp-proto
  labels:
    app.kubernetes.io/name: udp-proto-srv
    app.kubernetes.io/instance: udp-proto-srv
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-scheme: internet-facing
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: "80"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: TCP
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-healthy-threshold: "3"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-unhealthy-threshold: "3"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-timeout: "10"
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-interval: "10"
spec:
  selector:
    app: udp-proto-app
  ports:
    - port: 123
      name: ntp
      protocol: UDP
      targetPort: 123
  externalTrafficPolicy: Local
  type: LoadBalancer

I would suggest to start from checking NLB errors, follow this guide (it talks about Fargate, but you can apply it to pure EKS as well).

3

What I have found is that switching the nlb-target-type to ip as you have done from instance then the load balancer works but I do not believe it is following the externalTrafficPolicy as I can still connect to a pod on a different node to the node the gateway is on. My understanding is that local traffic policy means you can only connect to pods that are local (ie on the same node) as the gateway.