Ingress gateway is retrying if the upstream returns 502. Most of the time it is working as expected. Sometimes gateway returns 507 “exceeded request buffer limit while retrying upstream” without retrying.
From the logs I can see this:
app 502 →
istio-proxy sidecar 502 →
ingress gateway 507 →
client. Unable to find other errors in logs related to
Requests are 1-30Mb in size.
- Any ideas where to look for the issue?
- How to inspect Envoy configuration with the applied filter?
per_connection_buffer_limit_bytessetting responsible for increasing request buffer size (for retrying)?
... retries: attempts: 4 retryOn: 502,retriable-status-codes,connect-failure retryRemoteLocalities: true ...
EnvoyFilter (Increase per connection buffer limit - #5 by ethicalaakash):
apiVersion: networking.istio.io/v1alpha3 kind: EnvoyFilter metadata: name: buffer-limit spec: configPatches: - applyTo: LISTENER ListenerMatch: 0.0.0.0_8080 # Should it be? listenerMatch: 0.0.0.0_8080 patch: operation: MERGE value: per_connection_buffer_limit_bytes: 100000000
Using Istio 1.8.3
Thanks for your help.