We’ve been seeing some performance issues in our production environment and we are starting to suspect Istio is the culprit. Here’s the behavior we are seeing… We use MongoDB so we have a bunch of mongos pods (Mongo routers) running. When we make some changes in how other services talk to our mongos pods, after a while everything falls over. When it falls over, the Mongo clients are getting connection resets whenever they try to talk to our mongos pods, which explains why nothing is working.
When we ask the mongos processes about their connection count, most of them seem suspiciously close to 1024. We found some references to an Envoy setting cluster.CircuitBreakers.Thresholds.max_connections that defaults to 1024, but we don’t explicitly enable any circuit breakers, and it sounds like that has to do with outbound connections (“upstream”) and the issue we are seeing is with incoming connections to the mongos pods – as far as we can tell, no one client is making anywhere near 1024 connections.
Does anyone know if Istio has any default settings that would limit incoming TCP connections to 1024? I’ve looked around but the surface area of the Istio + Envoy documentation is huge, and I may just be looking for the wrong terms…
We are running Istio 1.1.4, and both the mongos and client pods have the istio-proxy sidecar automatically injected and are in the same namespace. We also have mixer.policy.enabled: false and policyCheckFailOpen: true so this shouldn’t be related to policies in any way…
Any help would be appreciated… Thanks,
Jeff