Istio (1.5.1) ingress gateway (envoy) heap memory grows unbounded, causing OOM when memory limit is exceeded

Performance and Scalability
1

In Istio 1.5.1 , we are seeing what appears to be a very similar issue with istio ingressgateway memory usage linearly increasing ending in an OOM due to memory limit exceeded. The tcp connections seem to not grow with the memory. But envoy heap size seems to be increasing monotonously and never being reclaimed, eventhough the “allocated” memory shrinks down when connections to the gateway drop.

istio_gateway_envoy_heap_vs_allocated
istio_gateway_envoy_heap_vs_allocated1834×760 48.8 KB

I am seeing a few different suggestions to address this:

  1. upgrade to istio 1.6 / 1.7
  2. configure envoy with envoy.overload_actions.shrink_heap as suggested here

Could someone please advise / provide some guidance ?

I’ve also created an issue for this in istio GHE

Thank you.