We faced a weird situation recently when out of 3 replicas of a service 1 pod was returning 503 “no healthy upstream error” for any request routed to it.
On further investigation, we found that the liveness and readiness probe of the pod were passing and error was being returned by the istio-proxy container as there were no logs in the service container for any requests other than health check requests. Terminating the pod fixed the issue.
I want to understand how to avoid such a situation in the future.