We are using the warmupDurationSecs property for gradually warming up our service but we observed that it works sometimes and sometimes it does not:
On the left, we can see a pod getting warmed up but when we test it again, warmup does not happen.
Note that the pod we are expecting to warmup is newer than the other pods which are running for a good amount of time.
How can we debug this further?
you mean the violet line indicates the new pod that is not being warmed?
Correct, the purple line denotes the requests rate for the pod which did not get warmed up and got equal traffic.
According to the official doc, warmupDurationSecs will fail when the pod is changed frequently. But I don’t know the exact algorithm。Somebody knows？
Hello guys, note that we are using istio 1.15.4
In these cases, we observed that there are some initial health check failures with following message:
[2023-05-24T16:26:33.192Z] "GET /api/health HTTP/1.1" 503 UF upstream_reset_before_response_started{connection_failure,delayed_connect_error:_111} - "delayed_connect_error:_111" 0 145 0 - "-" "Go-http-client/1.1" "95cea709-b5b9-4430-b4a0-9a54ab65d8c2" "192.168.53.44:9091" "192.168.53.44:9091" inbound|9091|| - 192.168.53.44:9091 192.168.44.230:52866 - default
I confirmed that the health check API is working by hitting the health check endpoint from a shell before the above requests come in. How can we debug why the health checks are failing with the above error when the request is going via istio?
Please ignore the previous message, we prevented these requests but the issue is still happening.