Stats/prometheus do not include my pod stats with enablePrometheusMerge=true

Policies and Telemetry
1

I describe the pod and while i would expect to find my metrics on localhost:15090/stats/prometheus, i only find the envoy / istio ones.
Why doesnt it include the localhost:1399/actuator/prometheus?

@howardjohn any clues?

  ISTIO_META_POD_PORTS:          [

                                     {"name":"http","containerPort":1399,"protocol":"TCP"}
                                 ]
  ISTIO_META_APP_CONTAINERS:     app
  ISTIO_META_CLUSTER_ID:         <some-non-default-id>
  ISTIO_META_INTERCEPTION_MODE:  REDIRECT
  ISTIO_METAJSON_ANNOTATIONS:    ...
                                 
  ISTIO_META_WORKLOAD_NAME:      dev-...
  ISTIO_META_OWNER:              ...
  ISTIO_META_MESH_ID:            <some-non-default-name>.local
  DNS_AGENT:                     
  ISTIO_PROMETHEUS_ANNOTATIONS:  {"scrape":"true","path":"/actuator/prometheus","port":"1399"}

I also changed istio-proxy logging to debug and i dont see anything about scraping, is that normal?

2

It is on port 15020. 15020 has the merged ones, 15090 has the envoy stats only.

3

Same deal unfortunately

iam@dev-iam-backend-cf4dbfb5d-k7hm5:/iam-be$ curl localhost:1399/actuator/prometheus -s | tail -3   
# TYPE jvm_buffer_count_buffers gauge
jvm_buffer_count_buffers{id="mapped",} 0.0
jvm_buffer_count_buffers{id="direct",} 13.0
iam@dev-iam-backend-cf4dbfb5d-k7hm5:/iam-be$ curl localhost:15020/stats/prometheus -s | grep jvm
iam@dev-iam-backend-cf4dbfb5d-k7hm5:/iam-be$

in fact it’s only istio / envoy metrics

iam@dev-iam-backend-cf4dbfb5d-k7hm5:/iam-be$ curl localhost:15020/stats/prometheus -s | grep -v envoy | grep -v istio | grep -v -e '^$'
iam@dev-iam-backend-cf4dbfb5d-k7hm5:/iam-be$

Can i somehow see the scraping job logs? I really tried to troubleshoot this, via configuration or log browsing but i could not find anything

4

@howardjohn it’s not the port, the metrics are just not scraped as you can see on my above post.
Is there any way to debug ? Some configuration to try? How can i see those scraping logs?

6
  1. I am using PeerAuthentication: STRICT in a govcloud cluster
  2. Therefore, I was very-much hoping to get merged metrics from the istio-agent.
  3. We are using the istio sidecar injector in our namespace to add istio-proxy (w/istio-agent inside).
  4. To guarantee metrics merging was turned on, I ran:

$ istioctl install --set meshConfig.enablePrometheusMerge=true

This will install the Istio 1.16.2 default profile with ["Istio core" "Istiod" "Ingress gateways"] components into the cluster. Proceed? (y/N) y
✔ Istio core installed
✔ Istiod installed
✔ Ingress gateways installed
✔ Installation complete
Making this installation the default for injection and validation.

  1. I am seeing these problems so far
  • Only port 15090 is injected by the injector in istio-proxy. I understand this has istio metrics only.
  • How do i get the sidecar injector to also declare port 15020 ??
  • I have discovered that if I add the weird prometheus annotations declaring the application monitoring port, metrics merging will occur and the prometheus annotations will be rewritten:
(in the deployment)
spec:
  template:
    metadata:
      annotations:
        prometheus.io/path: /metrics
        prometheus.io/port: "9538"
        prometheus.io/scrape: "true"

from the deployment it is rewritten in the pod to:

(in the pod)
metadata:
  annotations:
    prometheus.io/path: /stats/prometheus
    prometheus.io/port: '15090'
    prometheus.io/scrape: 'true'
    sidecar.istio.io/status: >-
      {"initContainers":["istio-init"],"containers":["istio-proxy"],"volumes":["workload-socket","credential-socket","workload-certs","istio-envoy","istio-data","istio-podinfo","istio-token","istiod-ca-cert"],"imagePullSecrets":null,"revision":"default"}
spec:
  containers:
    - name: istio-proxy
      image: docker.io/istio/proxyv2:1.16.2
      ports:
        - name: http-envoy-prom
          containerPort: 15090
          protocol: TCP

Which is certainly not very helpful because I want prometheus to scrape 15020, not 15090.

I have temporarily hacked my deployment by changing “containerPort” from 15090 to 15020 on L346 of the ConfigMap istio-sidecar-injector (in namespace istio-system), but this is not something that’s easy to hack with terraform or helm.