We have a problem in our env. When we stream grpc from service A to service B, sidecar of service A is killed with OOM on about 4 to 5 MB per second traffic. We tried different options, like:
1.one connection per stream
2. one connection for all streams
3. we strip everything from both services, just stream generation in A and consumption(just consumption, nothing else) in B. And yet sidecar of A still fails.
The only thing that helps now is to set initial_connection_window_size for A’s sidecar to some small amount, like 1MB. By default in envoy it is 256MB. The problem is, I can’t make it work for all involved pods. Some pods have websocket traffic also. And when I apply envoy filter, websockets start failing. I assume it is because it’s applied to all ports and protocols, while I want it to apply only if I have grpc traffic.
Here is my config:
So my question are:
- How to change initial_connection_window_size only to subset of ports/protocols?
- Why does envoy has 256MB initial_connection_window_size by default? It seems like to much even if we use only one http connection to service?
- Any other thoughts what could go wrong?