since a few weeks we see a significant increase in memory usage on istio-proxy and we can’t pin down the cause of it.
I’ve enabled the profiler on istio-proxy and see that the vast majority of the memory is consumed by
Envoy::Buffer::OwnedImpl::reserveWithMaxLength
any insights on what could cause this?
I am not sure if this could be applicable to your case, but we also experienced high memory usage and we fixed it by tuning envoy initial_stream_window_size and initial_connection_window_size as mentioned in Configuring Envoy as an edge proxy — envoy 1.29.0-dev-c1ffb9 documentation.
You can see some other reference to such problems in envoy issue tracker, eg, High memory usage with long HTTP/2 connections (GRPC) · Issue #9891 · envoyproxy/envoy · GitHub.
Also, I would suggest to check if the concurrency in your envoy proxy is set adequately, see Command line options — envoy 1.29.0-dev-c1ffb9 documentation for some explanation of what this is. We found that the concurrency needs to be set correctly because each worker keeps a separate connection pool, so too many workers mean too many connections and thus too many buffers.