Gateway randomly returns 507 "exceeded request buffer limit while retrying upstream"

Networking
1

Ingress gateway is retrying if the upstream returns 502. Most of the time it is working as expected. Sometimes gateway returns 507 “exceeded request buffer limit while retrying upstream” without retrying.

From the logs I can see this: app 502 → istio-proxy sidecar 502 → ingress gateway 507 → client. Unable to find other errors in logs related to ingress-gateway.
Requests are 1-30Mb in size.

  • Any ideas where to look for the issue?
  • How to inspect Envoy configuration with the applied filter?
  • Is per_connection_buffer_limit_bytes setting responsible for increasing request buffer size (for retrying)?

VirtualService:

...
      retries:
        attempts: 4
        retryOn: 502,retriable-status-codes,connect-failure
        retryRemoteLocalities: true
...

EnvoyFilter (Increase per connection buffer limit - #5 by ethicalaakash):

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: buffer-limit
spec:
  configPatches:
    - applyTo: LISTENER
      ListenerMatch: 0.0.0.0_8080
# Should it be?      listenerMatch: 0.0.0.0_8080
      patch:
        operation: MERGE
        value:
          per_connection_buffer_limit_bytes: 100000000

Using Istio 1.8.3

Thanks for your help.