I’d like to use AWS NLB for our ingressGateway. And the only way to preserve client PI is to set Service externalTrafficPolicy=Local. And as it describes in K8S Service Doc
Nodes without any Pods for a particular LoadBalancer Service will fail the NLB Target Group’s health check on the auto-assigned .spec.healthCheckNodePort and not receive any traffic.
And this works as expected.
The default Health check params for Target Groups in Kubernetes 1.15.6:
- Unhealthy threshold is set to 3
- Interval 30s.
- Timeout 6s
And seems it brings an issue in case we have enabled HPA for ingressGateway. When hpa scales down the pod - there is a delay in NLB/TargetGroups when it moves instance from healthy -> unhealthy. And for 90+ seconds NLB routes traffic to this nodes.
What would be a proper solution or workaround here?