ServiceMonitors needed for using Istio 1.7 with Prometheus Operator on Kubernetes 1.19

I installed Istio default with --set values.prometheus.enabled=true, which worked fine, but I wanted to move to a production configuration.
So I reinstalled Istio without Prometheus, and installed the kube-prometheus stack (head).
There is a section in the Istio documentation for using Istio with an existing Prometheus


I have to add some scraping configuration somewhere, but not sure what to add and where.
The only prometheus configmap I have is prometheus-k8s-rulefiles-0.

The Prometheus Operator documentation says I should add ServiceMonitors.
Are there ready-to-use ServiceMonitors for the Istio Control and Data plane, or instruction how to make them?
Ideally, an IstioCtl option to install Prometheus ServiceMonitors would be great.

here is what i have for 1.5 istio, you may need to adjust this for 1.7 but it should at least give you some idea on scraping control plane and data plane

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istio-component-monitor
  namespace: istio-system
  labels:
    monitoring: istio-components
    release: <label that prometheus listens for>
spec:
  jobLabel: istio
  selector:
    matchExpressions:
      - {key: istio, operator: In, values: [mixer,pilot,galley,citadel,sidecar-injector]}
  namespaceSelector:
    any: true
  endpoints:
  - port: http-monitoring
    interval: 15s
  - port: http-policy-monitoring
    interval: 15s
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: envoy-stats-monitor
  namespace: istio-system
  labels:
    monitoring: istio-proxies
    release: <label that prometheus listens for>
spec:
  selector:
    matchExpressions:
      - {key: istio-prometheus-ignore, operator: DoesNotExist}
  namespaceSelector:
    any: true
  jobLabel: envoy-stats
  endpoints:
  - path: /stats/prometheus
    targetPort: 15090
    interval: 15s
    relabelings:
    - sourceLabels: [__meta_kubernetes_pod_container_port_name]
      action: keep
      regex: '.*-envoy-prom'
    - action: labeldrop
      regex: "__meta_kubernetes_pod_label_(.+)"
    - sourceLabels: [__meta_kubernetes_namespace]
      action: replace
      targetLabel: namespace
    - sourceLabels: [__meta_kubernetes_pod_name]
      action: replace
      targetLabel: pod_name

Thank you!
I’ll try to use these as a basis for ServiceMonitors for 1.7

Hi, did you get this to work. I’m going to have to tackle this same problem soon and would be great if you could share any results you’ve had.

Yes, I’ve got it to work. Let’s get my notes :slight_smile:

I found some (parts of) examples in different places, and combined them to make it work.
I created a ServiceMonitor for the Istio components, and a PodMonitor for my application pods that I want to monitor and see in Kiali. The monitors are in the monitoring namespace. Furthermore I created a ClusterRole to have the necessary access to other namespaces.
I will add the yaml files below. Note that the ServiceMonitor matchExpression can be simpler, because 1.7 just has one pod. I think you can just match on pilot.
The PodMonitor just select the default namespace. These are things that should be easy to change, but I didn’t get around to it yet. I don’t know what the relabelings are for and if they are needed. I you find out please let me know.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-proxy-monitor
  namespace: monitoring
  labels:
    istio: proxy
spec:
  jobLabel: component
  selector:
    matchLabels:
      security.istio.io/tlsMode: "istio"
    matchExpressions:
    - key: migration
      operator: NotIn
      values: [ "true", "1" ]

  namespaceSelector:
    matchNames:
    - default

  podMetricsEndpoints:
  - port: http-envoy-prom
    path: /stats/prometheus
    relabelings:
    - action: labeldrop
      regex: __meta_kubernetes_pod_label_skaffold_dev.*
    - action: labeldrop
      regex: __meta_kubernetes_pod_label_pod_template_hash.*
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istio-component-monitor
  namespace: monitoring
  labels:
    k8s-app: istio-components
    monitoring: istio-components
spec:
  jobLabel: istio
  selector:
    matchExpressions:
      - {key: istio, operator: In, values: [mixer,pilot,galley,citadel,sidecar-injector]}
  namespaceSelector:
    matchNames:
    - istio-system
  endpoints:
  - port: http-monitoring
    interval: 15s
  - port: http-policy-monitoring
    interval: 15s
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
2 Likes

Awesome, thanks. This will surely help me tackle the next few steps. Greatly appreciate the help!

Good information thanks for sharing
vmware

Hi @robert thanks for sharing, I would work this in istio1.8

the relabeling is if you want to change the variables names for something more readable.

See this discussion.

It links to some semi-official service monitors.

This is an old thread, but I just basically asked the same question for Istio 1.9.5 in a new thread here: Scraping Istio metrics from Prometheus Operator (e.g. using ServiceMonitor)

If anyone who had this working on Istio 1.7 has a working configuration for 1.9.5 (or 1.10) that would be amazing…