enablePrometheusMerge and ISTIO_PROMETHEUS_ANNOTATIONS

Trying to understand the doc for using enablePrometheusMerge and env var ISTIO_PROMETHEUS_ANNOTATIONS.

Does not mention what port the users app should be using in order for the merge with Envoy metrics to work. What port should be used?

The enablePrometheusMerge will automatically add the annotations:

prometheus.io path: /stats/prometheus
prometheus.io port: "15020"
prometheus.io scrape: "true"

That is the “merged” metrics port, but what to understand the envoy and app ports that feed into that.

From the code, looks like that env VAR ISTIO_PROMETHEUS_ANNOTATIONS is used and default to “80:/metrics”
That doc is also missing: https://istio.io/latest/docs/reference/commands/pilot-agent/#envvars

cc @howardjohn @rvennam

https://github.com/istio/istio/blob/f508fdd78eb0d3444e2bc2b3f36966d904c5db52/pilot/cmd/pilot-agent/status/server.go#L131-L144 is the code and also https://istio.io/latest/docs/reference/commands/pilot-agent/#envvars is missing doc for that env var ISTIO_PROMETHEUS_ANNOTATIONS

Does not mention what port the users app should be using in order for the merge with Envoy metrics to work. What port should be used?

Whatever port you want. If you configure prometheus.io/port=1234 we use 1234. If you don’t set that annotation, we fall back to 80 (aligned with prometheus defaulting)

I see that if the app has the annotation defined already, this will create the following env vars with that info to allow the app to continue using the same port for scrapes/merges of the metrics.

 - name: ISTIO_METAJSON_ANNOTATIONS
      value: |
        {"kubernetes.io/psp":"ibm-privileged-psp","prometheus.io/path":"/metrics","prometheus.io/port":"8080","prometheus.io/scrape":"true","prometheus.istio.io/merge-metrics":"true"}
    - name: ISTIO_PROMETHEUS_ANNOTATIONS
      value: '{"scrape":"true","path":"/metrics","port":"8080"}'

else it will look for “:80/metrics”.

ISTIO_PROMETHEUS_ANNOTATIONS is an implementation detail, you don’t need to touch it yourself - you just configure the prometheus.io annotations

Thx @howardjohn, maybe the doc needs a bit of tweaking/examples to help with this understanding.

@howardjohn just to clarify, the merge support is looking for existing deployment annotations only at the deployment.metadata.annotations level? If set at the spec.template.metadata level, it won’t find them, right?

Its looking at the pods (ie spec.template.metadata)

@howardjohn Another question. With the merge, we are seeing that the metrics are now coming from the istio-proxy container. For Sysdig using this, we can only see the metrics under the istio-proxy container, it cannot see the true application containers name anymore. Therefore with multiple data plane pods, they all have the same container name.

Is this just a limitation of the merge, or is there some way to allow both scrapes to work, one for the proxy and one for the app container to allow both to show up in Sysdig?
Or allow the customer app name to be applied to the merged metrics instead of the fixed istio-proxy name?

Would need to go up to the Pod labels and use something there for the queries to make them unique I believe.