Incorrect metrics collected by prometheus

Policies and Telemetry Kiali
1

I’m using an old version of istio (v1.10).
I’m trying to config a custom prometheus instance to scrape the metrics.
From the PrometheusUI, I can see that pods are discovered, however, the metrics collected seem strange.

I’m deploying a deployment wbt2-1710475010716020737-app, a Service wbt2-1710475010716020737-app and a VirtualService wbt2-1710475010716020737-app-vs.
The hosts of the VirtualService is configured as wbt2-1710475010716020737-app.wbt.com.

I can access the deployment from outside the cluster by using the domain name https://wbt2-1710475010716020737-app.wbt.com/xxx.
I access the deployment many times, but when I check the metrics istio_requests_total on prometheus, the value didn’t increase.

I’ve tested more and this is what I’ve observed for now:

I kubectl exec into the pod of the deployment.

  • When I access this deployment by curl localhost:9080/xxxx, the istio_requests_total doesn’t increase.
  • When I access by curl https://wbt2-1710475010716020737-app.wbt.com/xxx, the istio_requests_total doesn’t increase.
  • When I access by curl wbt2-1710475010716020737-app:9080/xxx ( which is the service name), the istio_requests_total do increase.

All the above three access methods can get correct http response.

So I’m confused what may be the problem here.


Here is my PodMonitor config for prometheus.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  labels:
    app.kubernetes.io/component: prometheus
    app.kubernetes.io/name: prometheus
    app.kubernetes.io/part-of: kube-prometheus
    app.kubernetes.io/version: 2.26.0
  name: istio-sidecars
  namespace: monitoring
spec:
  podMetricsEndpoints:
  - path: /stats/prometheus
    relabelings:
      - action: keep
        sourceLabels:
          - __meta_kubernetes_pod_container_port_name
        regex: '.*-envoy-prom'
      - sourceLabels: 
          - __meta_kubernetes_pod_annotation_prometheus_io_path
        separator: ;
        regex: (.+)
        targetLabel: __metrics_path__
        replacement: $1
        action: replace
      - sourceLabels: 
          - __address__
          - __meta_kubernetes_pod_annotation_prometheus_io_port
        separator: ;
        regex: ([^:]+)(?::\d+)?;(\d+)
        targetLabel: __address__
        replacement: $1:$2
        action: replace
      - separator: ;
        regex: __meta_kubernetes_pod_annotation_prometheus_io_param_(.+)
        replacement: __param_$1
        action: labelmap
      - separator: ;
        sourceLabels: 
          - __meta_kubernetes_pod_label_app
        regex: (.+)
        targetLabel: app
        replacement: $1
        action: replace
      - separator: ;
        sourceLabels: 
          - __meta_kubernetes_pod_label_version
        regex: (.+)
        targetLabel: version
        replacement: $1
        action: replace  
      - sourceLabels: 
          - __meta_kubernetes_pod_name
        separator: ;
        regex: (.*)
        targetLabel: pod
        replacement: $1
        action: replace
      - sourceLabels: 
          - __meta_kubernetes_pod_phase
        separator: ;
        regex: Pending|Succeeded|Failed|Completed
        replacement: $1
        action: drop
  namespaceSelector:
    any: true

And here is the metrics istio_requests_total collected by prometheus:

istio_requests_total{app="wbt2-1710475010716020737-app", connection_security_policy="mutual_tls", container="istio-proxy", destination_app="wbt2-1710475010716020737-app", destination_canonical_revision="wbt2-1710475010716020737-app", destination_canonical_service="app", destination_cluster="Kubernetes", destination_principal="spiffe://k8s.local/ns/wbtest2/sa/default", destination_service="wbt2-1710475010716020737-app.wbtest2.svc.cluster.local", destination_service_name="wbt2-1710475010716020737-app", destination_service_namespace="wbtest2", destination_version="wbt2-1710475010716020737-app", destination_workload="wbt2-1710475010716020737-app", destination_workload_namespace="wbtest2", instance="172.30.113.239:15020", job="monitoring/istio-sidecars", namespace="wbtest2", pod="wbt2-1710475010716020737-app-658d984f7f-zfz4c", reporter="destination", request_protocol="http", response_code="200", response_flags="-", source_app="wbt2-1710475010716020737-app", source_canonical_revision="wbt2-1710475010716020737-app", source_canonical_service="app", source_cluster="Kubernetes", source_principal="spiffe://k8s.local/ns/wbtest2/sa/default", source_version="wbt2-1710475010716020737-app", source_workload="wbt2-1710475010716020737-app", source_workload_namespace="wbtest2", version="wbt2-1710475010716020737-app"} 
4

istio_requests_total{app="wbt2-1710475010716020737-app", connection_security_policy="unknown", container="istio-proxy", destination_app="wbt2-1710475010716020737-app", destination_canonical_revision="wbt2-1710475010716020737-app", destination_canonical_service="app", destination_cluster="Kubernetes", destination_principal="spiffe://k8s.local/ns/wbtest2/sa/default", destination_service="wbt2-1710475010716020737-app.wbtest2.svc.cluster.local", destination_service_name="wbt2-1710475010716020737-app", destination_service_namespace="wbtest2", destination_version="wbt2-1710475010716020737-app", destination_workload="wbt2-1710475010716020737-app", destination_workload_namespace="wbtest2", instance="172.30.113.239:15020", job="monitoring/istio-sidecars", namespace="wbtest2", pod="wbt2-1710475010716020737-app-658d984f7f-zfz4c", reporter="source", request_protocol="http", response_code="200", response_flags="-", source_app="wbt2-1710475010716020737-app", source_canonical_revision="wbt2-1710475010716020737-app", source_canonical_service="app", source_cluster="Kubernetes", source_principal="spiffe://k8s.local/ns/wbtest2/sa/default", source_version="wbt2-1710475010716020737-app", source_workload="wbt2-1710475010716020737-app", source_workload_namespace="wbtest2", version="wbt2-1710475010716020737-app"}
4
2

Stuck several days, I checked the the metrics endpoint :15020/stats/prometheus of the Pod and found that metrics were not generated when I made requests.

I turn on the logging output of the envoy proxy by the envoy admin endpoint :15000/logging?router=debug and checked the log of the istio-proxy container of the Pod.
I can see that the prometheus requesting endpoint /stats/prometheus was logging.
But when I made request from outside cluster (i.e. by the hosts of the VirtualService), the request was not logged.

On the other hand, when I made request from inside the cluster by the service domain name, I can see the request was logged and metrics were generated.

My assumption was that requests made from outside the cluster (ingress-gateway → Gateway → VirtualService → Service → Pod) are send to the pod directly, passing-through the istio-proxy sidecars.

I’ve no idea why, maybe it’s a silly config problem somewhere. Still investigating…

3

Found the problem.

The service’s port name should start with ‘http’, otherwise traffic is not routed correctly.