Annotations for Prometheus overriden, and metrics not merged

Hello everyone,

I am currently setting up a Kubernetes workload, We want to use Istio and we use Prometheus for metrics. I am using this helm chart for Prometheus. I am deploying Prometheus in the default namespace, which has the istio-injection=enabled label set (so Prometheus itself has an Envoy side-car proxy, I’m not sure if that makes a difference for my problem, though…)

I want various pods to export metrics to Prometheus, and I use annotations to that effect. For example, I set the following annotations for a given pod: prometheus.io/scrape: "true", prometheus.io/port: "3100" and prometheus.io/path: /metrics.

The Istio documentation says here that Istio will override those annotations and will merge the Envoy metrics with the app metrics. I can see in my cluster that the annotations are indeed overridden, but no merging takes place. When I curl port 15020 at /stats/prometheus, I can see only istio and envoy metrics, and no metrics from the app. The app metrics work fine when the cluster is run without Istio.

Could anyone please let me know what I should do to ensure the metrics are properly merged? Thank you very much!

Can you check the proxy logs? Should see something like Prometheus scraping configuration: {true 3100}

Also can check istio_agent_scrapes_total and istio_agent_scrape_failures_total from 15020/stats/prometheus?

Hi @howardjohn , thanks for your suggestions. Actually, I decided to re-install everything, and now everything works fine… I don’t know what happened, I guess I messed up my cluster somehow. Anyway, thanks a lot for your help!