Annotations for Prometheus overriden, and metrics not merged

1

Hello everyone,

I am currently setting up a Kubernetes workload, We want to use Istio and we use Prometheus for metrics. I am using this helm chart for Prometheus. I am deploying Prometheus in the default namespace, which has the istio-injection=enabled label set (so Prometheus itself has an Envoy side-car proxy, I’m not sure if that makes a difference for my problem, though…)

I want various pods to export metrics to Prometheus, and I use annotations to that effect. For example, I set the following annotations for a given pod: prometheus.io/scrape: "true", prometheus.io/port: "3100" and prometheus.io/path: /metrics.

The Istio documentation says here that Istio will override those annotations and will merge the Envoy metrics with the app metrics. I can see in my cluster that the annotations are indeed overridden, but no merging takes place. When I curl port 15020 at /stats/prometheus, I can see only istio and envoy metrics, and no metrics from the app. The app metrics work fine when the cluster is run without Istio.

Could anyone please let me know what I should do to ensure the metrics are properly merged? Thank you very much!

2

Can you check the proxy logs? Should see something like Prometheus scraping configuration: {true 3100}

Also can check istio_agent_scrapes_total and istio_agent_scrape_failures_total from 15020/stats/prometheus?

3

Hi @howardjohn , thanks for your suggestions. Actually, I decided to re-install everything, and now everything works fine… I don’t know what happened, I guess I messed up my cluster somehow. Anyway, thanks a lot for your help!

4

@howardjohn i have k8 1.20 running on minikube, deployed a sample app in default namesapce, that app provides below metrics:

$ curl -s 172.17.0.4:8080/metrics |grep -v ^# | head -5
http_request_duration_seconds_bucket{code="200",handler="found",method="get",le="0.005"} 8
http_request_duration_seconds_bucket{code="200",handler="found",method="get",le="0.01"} 8
http_request_duration_seconds_bucket{code="200",handler="found",method="get",le="0.025"} 8
http_request_duration_seconds_bucket{code="200",handler="found",method="get",le="0.05"} 8
http_request_duration_seconds_bucket{code="200",handler="found",method="get",le="0.1"} 8

deployment has annotations

$ k describe deploy prometheus-example-app |grep prometheus.io
  Annotations:  prometheus.io/port: 8080
                prometheus.io/scrape: true

pod annotations are overwritten by istio

$ k describe po prometheus-example-app-5556bc98cf-c4lvh |grep prometheus.io
              prometheus.io/path: /stats/prometheus
              prometheus.io/port: 15020
              prometheus.io/scrape: true

sidecar logs show log confirming that the annotations are accepted

$ k logs prometheus-example-app-5556bc98cf-c4lvh -c istio-proxy |grep scraping
2022-05-23T15:50:17.971930Z	info	Prometheus scraping configuration: {true  8080}

istio sidecar doesnt have app nor istio_agent_scrapes_* metrics

$ curl -s 172.17.0.4:15090/stats/prometheus |grep http_request_duration_seconds_bucket

$ curl -s 172.17.0.4:15090/stats/prometheus |grep istio_agent_scrapes

istio version: 1.13.4

what to check here?

I think its a bug, raised metrics merging not working in 1.13 · Issue #39091 · istio/istio · GitHub