I am currently setting up a Kubernetes workload, We want to use Istio and we use Prometheus for metrics. I am using this helm chart for Prometheus. I am deploying Prometheus in the
default namespace, which has the
istio-injection=enabled label set (so Prometheus itself has an Envoy side-car proxy, I’m not sure if that makes a difference for my problem, though…)
I want various pods to export metrics to Prometheus, and I use annotations to that effect. For example, I set the following annotations for a given pod:
prometheus.io/port: "3100" and
The Istio documentation says here that Istio will override those annotations and will merge the Envoy metrics with the app metrics. I can see in my cluster that the annotations are indeed overridden, but no merging takes place. When I curl port 15020 at
/stats/prometheus, I can see only istio and envoy metrics, and no metrics from the app. The app metrics work fine when the cluster is run without Istio.
Could anyone please let me know what I should do to ensure the metrics are properly merged? Thank you very much!