Folks,
I’ve starting to use ISTIO to our EKS cluster, but now we’re trying to read the metrics on a few apps and I’ve don’t found anything about it.
It’s possible to have metrics based on the application path?
For example, I have an app that has 2 paths “/resolve” and “/test”, it’s possible to have metrics based on the path?
I’ve looked to use the DestinationRule to send to a subset but doesn’t seem right for this purpose.
@jeanguirro it is possible. there are attributes that contains information on the request path (request.path, request.url_path). you can find more info in the attribute vocabulary docs.
With those attributes, you could add dimensions (aka labels) to the default metrics definitions. And then set the attribute expression to something like: path: request.path | "unknown".
The reason that this isn’t the default is because of cardinality concerns. If you have a very small number of paths (it sounds like you have two) and can be certain you aren’t receiving bad requests to an unconstrained set of paths (like perhaps by being exposed to the public internet), it should be OK to add such a dimension to your metrics.
If you do have a large number of paths or any services that use part of a path to refer to some kind of unique ID or value then you have to be very careful using request.path or request.url_path for the same reason that @douglas-reid stated that it was not a default in Istio. This can cause an unbounded cardinality explosion and could potentially overwhelm your metrics provider or result in high cost.
The way that we handled this was to have our services respond with a header that will mask the unique portions of the path and prevent this cardinally explosion. In the event that the header is not there we write unknown as the value.
You would enhance the default metrics by adding something like this to the metric definition:
method_name: response.headers[“uri”] | “unknown”
This will take the response header uri and add it as a dimension to the metric called method_name. The service would then respond with a header called uri in a format like “/saveDocument/{DocID}”. We then have a standardized library that we import into the application that implicitly adds this header on outgoing responses.
The only downside to this approach is that if the response is not sent (due to timeout, outage, etc…) then the value will be “unknown”. This seemed like an alright tradeoff to control cardinality.
@douglas-reid Following docs here: Istio / Classifying Metrics Based on Request or Response
Was able to get request_operation=“Httpbinstatus” for /status/*.
But this value comes only when I am hitting it from inside pod, when hitting same request using INGRESS_HOST:INGRESS_PORT, value comes as unknown.
Any help?
It is impossible to say exactly what is happening. I’d check to make sure that you are on the same path, or going to the same pod. Is there a path rewrite operation happening at the gateway or similar? Also make sure you made the change for all types of traffic in your EnvoyFilter.