Istio has several default metrics, such as istio_requests_total, istio_request_bytes, istio_tcp_connections_opened_total. Istio envoy proxy computes and exposes these metrics. On the Istio website, it shows that istio_requests_total is a COUNTER incremented for every request handled by an Istio proxy.
We made some experiments where we let a lot of requests go through the Istio envoy to reach a microservice behind Istio envoy, and at the same time we monitored the metric from Istio envoy. However, we found that istio_requests_total does not include the requests that have got through Istio envoy to the backend microservice but their responses have not arrived at Istio envoy from the backend microservice. In other words, istio_requests_total only includes the number of served requests, and does not include the requests in flight.
My question is: is our observation right? Why does istio_requests_total not include the requests in flight?