What is the metric to find http retry attempts in prometheus?
In Prometheus, I tried all the metrics that have ‘retry’ in its name, but all values are 0. Either I am not testing properly or Prometheus ‘retry’ metrics aren’t working as expected.
I have a virtual service defined with this:
timeout: 5s
retries:
attempts: 3
retryOn: 5xx,gateway-error,reset,connect-failure,refused-stream,retriable-status-codes
fault:
delay:
percent: 100
fixedDelay: 10s
I am also puzzled here - the retries seem only relevant to grpc as per:
kubectl exec -it echoserver-v1-cbc695fcb-pvj8d -c istio-proxy -n gps-test pilot-agent request GET stats | grep -i ret
cluster.xds-grpc.circuit_breakers.default.rq_retry_open: 0
cluster.xds-grpc.circuit_breakers.high.rq_retry_open: 0
cluster.xds-grpc.client_ssl_socket_factory.downstream_context_secrets_not_ready: 0
cluster.xds-grpc.client_ssl_socket_factory.upstream_context_secrets_not_ready: 0
cluster.xds-grpc.retry_or_shadow_abandoned: 0
cluster.xds-grpc.upstream_rq_retry: 0
cluster.xds-grpc.upstream_rq_retry_overflow: 0
cluster.xds-grpc.upstream_rq_retry_success: 0
Do values of 0 indicate default of no attempts?
policy.istio.io/checkRetries in https://istio.io/docs/reference/config/annotations/
Any help appreciated.
In Prometheus - one can use sum(istio_requests_total{request_protocol=“http”,response_flags=“URX”})
flags via https://www.envoyproxy.io/docs/envoy/v1.11.1/configuration/access_log
As I can’t pause a pod - I’ll look for an App to simulate URX code.