Hi Team
we are having Kubernetes cluster version 1.23.8 and Istio version 1.14.5
Some time after the application deployment application gives 503 “no healthy upstream” intermittently
we have only one pod deployed for the application and its up and running
some call will return 200 and some will returning 503,
Even after pod restarting its not fixing,
But if we restarts the istiod the issue is getting fixed.
can some body help me to debug this issue and and suggest the solution for this
Sample log from ingress gate way for 503:
{“method”:“GET”,“authority”:xxx.pqa.wus.cld.abc.com”,“route_name”:null,“bytes_sent”:19,“bytes_received”:0,“downstream_remote_address”:“10.53.0.21:16420”,“duration”:0,“request_id”:“0934c1cb-b357-4b04-aec1-3ee1ecd3a0ef”,“path”:“/status”,“requested_server_name”:xxx.pqa.wus.cld.abc.com”,“start_time”:“2023-03-01T01:08:04.440Z”,“upstream_local_address”:null,“response_code_details”:“no_healthy_upstream”,“upstream_service_time”:null,“upstream_cluster”:“outbound|80|main|abc-xxx-service.xxx.svc.cluster.local”,“response_flags”:“UH”,“response_code”:503,“protocol”:“HTTP/2”,“upstream_host”:null,“upstream_transport_failure_reason”:null,“x_forwarded_for”:“10.53.0.21”,“connection_termination_details”:null,“downstream_local_address”:“172.18.37.24:8443”,“user_agent”:“curl/7.84.0”}
Sample log from ingress gate way for 200:
{“authority”:”xxx.pqa.wus.cld.abc.com”,“protocol”:“HTTP/2”,“connection_termination_details”:null,“upstream_host”:“172.18.29.208:8080”,“upstream_service_time”:“10”,“upstream_transport_failure_reason”:null,“duration”:10,“downstream_remote_address”:“10.53.0.24:7488”,“bytes_received”:0,“response_code_details”:“via_upstream”,“path”:“/status”,“response_code”:200,“method”:“GET”,“requested_server_name”:”xxx.pqa.wus.cld.abc.com”,“user_agent”:“curl/7.84.0”,“response_flags”:“-”,“upstream_local_address”:“172.18.37.24:60324”,“request_id”:“eb0ca4f0-6c86-4773-b966-f01ea0ec8a47”,“route_name”:null,“downstream_local_address”:“172.18.37.24:8443”,“upstream_cluster”:“outbound|80|main|abc-xxx-service.xxx.svc.cluster.local”,“x_forwarded_for”:“10.53.0.24”,“start_time”:“2023-03-01T01:35:03.561Z”,“bytes_sent”:32}