we are having issues with our Ingress gateway being overloaded by config changes. The issue looks pretty much like what is described here: https://github.com/istio/istio/issues/23371 Ingress gateways readiness probes start to time out under load. We currently work around this by increasing the timeouts. However, there has to be a better solution.
Our architecture looks like this:
We have a single ingress gateway handling all traffic going to our AWS load balancer. We serve multiple instances of a single-tenant application for multiple customers. Customers can add and remove instances as they see fit. Every customer has its own namespace. New customers are added frequently.
Any ideas on how we can prevent our ingress gateway from falling over under load? The only idea we had was layering ingress gateways (one global gateway plus one for each customer) but dynamically spinning up ingress gateways isn’t documented so I want to double check if we are running in a wrong direction.