In here https://istio.io/docs/concepts/traffic-management/, it is stated that
" when all instances in a load balancing pool have failed, Envoy returns an HTTP 503 code. The application must implement any fallback logic needed to handle the HTTP 503 error code…"
In our case, the application is the browser, the browser will simply display the return code and we can’t change its behavior.
What we have is a DNS based GTM setup for app. test. com.
app. test. com -> IP1 -> Istio Ingress Gateway in DC1
app. test. com -> IP2 -> Istio Ingress Gateway in DC2
We have a health-check mechanism so that when the backend failed in DC1, app.test.com will only resolve to IP2.
However, in this failover situation, an existing browser session that uses IP1 will simply display 503, and will not refresh the DNS entry for app.test.com, as 503 is a valid response code.
(BTW, a new browser session works as it will start with IP2. if we shutdown istio totally, it will also work, as browser will refresh DNS and get IP2.)
So the question is, can Istio ingress-gateway instead of returning 503, simply do a disconnect or any behavior that can handle this situation?
We are looking at adding an extra hop, a LTM in front, that will simulate this behavior, translating 503 to a refuse connection. But why adding extra hops just for this. Any help or suggestion is greatly appreciated.