Prometheus to collect headers as metrics

Policies and Telemetry
1

Hello,

For the ease of troubleshooting, We want a solution that is well integrated with istio so that we dont have to go for other solutions such as ELK.

I have two questions about this:

  1. Is there a way to get correlation-IDs and other headers from a call as a “metric” in prometheus ?
  2. How can I capture pod logs (full logs not just metrics), We are using GKE (Google cloud’s managed kubernetes service) with disabled stackdriver ?

Regards,
Sourabh W

2

It is possible to add your own metrics (and/or customize the out-of-box config that is installed by default) to add x-request-id and other headers to metrics. You would just need to draft up a few config artifacts (along the lines of the Metrics and Logs tasks on istio.io).

Beware, however, that including unbounded fields like request ids in metrics could be problematic. From https://prometheus.io/docs/practices/naming/:

CAUTION: Remember that every unique combination of key-value label pairs represents a new time series, which can dramatically increase the amount of data stored. Do not use labels to store dimensions with high cardinality (many different label values), such as user IDs, email addresses, or other unbounded sets of values.

Istio doesn’t currently offer any special solution to getting pod logs. Mixer can provide access logs for a pod, but it doesn’t have any mechanism for accessing the full application logs, etc. I believe it is possible to update the fluentd config on GKE to export to a different place (if not using Stackdriver) however.

Hope that helps,
Doug.

3

Thanks Doug.

What all config. files need to be changed, what all new CRDs or kubernetes native objects need to be deployed to get it working with prometheus ?