How to show request path In Istio metrics?

1

I have one workload where I want to be able to see the request path for requests going into the pod.
When I look in istio_requests_total{} prometheus metrics, I dont see any request path ie: /api etc…
Is there an easy way to do this in setting up a the virtual service or do I have to build a custom handler, instance, rule?
I dont necessarily need to see the request path for every request globally, just for one particular microservice.

Thanks

2

There is some discussion going on now around exposing the request.host in the telemetry. But it’s not yet available in the default requestcount metric. I’ve got some branches in kiali that can expose the info if you add request.host to the telemetry, but you could also look at prometheus directly. To add it you can do the following if you’re installing via helm charts (note - this will not limit it to one workload, i don’t think that’s possible):

edit: install/kubernetes/helm/istio/charts/mixer/templates/config.yaml

add request.host/request_host in the requestcount instance and also in the prometheus handler

then re-run the helm template.

oc project istio-system helm template _output/istio-1.3.0-rc.3/install/kubernetes/helm/istio | oc apply -f-

it may take a few minutes to propagate the change, but then I think you should be able to see request_host in istio_requests_total and hopefully it will provide the info you are looking for.

3

Can you kindly elaborate? Will request.host provide the request URI? I have a similar requirement where i need to be able to breakdown requests by the individual APIs exposed by our Restful micro-services.

4

Hi @zzkhan,
Some work has been done since this topic was originally posted. I think this should describe what you want: https://istio.io/latest/docs/tasks/observability/metrics/classify-metrics/

And to leverage that in Kiali: https://kiali.io/documentation/latest/features/#_graph_operation_nodes

5

BTW: to have the full URI (including, e.g. query params) would be a HUGE resource drain on your Prometheus storage because the cardinality for your Prometheus timeseries would be very, very large (just think of all the different permuations/combinations of URIs you would have). So I highly doubt the Istio core will ever support putting full URIs (including query params and the like) in a Prometheus label.

This is why it isn’t so easy to do what you want. We had a previous discussion (actually a few discussions) when people bring this up. The answer is normally, “Use Jaeger / some tracing solution if you want to analyze performance on a per-URI basis” - that’s the tool that should be used for this kind of thing, not metrics.

But, as @jotak points out, Kiali actually does now support the new Istio experimental feature of defining “operations” which can provide metrics for particular endpoints. But that requires some additional configuration in your mesh to classify your requests in a manner that you deem important (i.e. you classify particular requests as operations and Istio will store the operation data in the Prometheus telemetry).

6

We ended up handling this by having the server return a URI pattern as a header with out the ids and query parameters. Then updated the Istio metrics to put this pattern in the metrics labels.

7

Hi, can you share how did you read the values from request headers to add it to istio telemetry ?