ServiceMonitors needed for using Istio 1.7 with Prometheus Operator on Kubernetes 1.19

#1

I installed Istio default with --set values.prometheus.enabled=true, which worked fine, but I wanted to move to a production configuration.
So I reinstalled Istio without Prometheus, and installed the kube-prometheus stack (head).
There is a section in the Istio documentation for using Istio with an existing Prometheus


I have to add some scraping configuration somewhere, but not sure what to add and where.
The only prometheus configmap I have is prometheus-k8s-rulefiles-0.

The Prometheus Operator documentation says I should add ServiceMonitors.
Are there ready-to-use ServiceMonitors for the Istio Control and Data plane, or instruction how to make them?
Ideally, an IstioCtl option to install Prometheus ServiceMonitors would be great.

#2

here is what i have for 1.5 istio, you may need to adjust this for 1.7 but it should at least give you some idea on scraping control plane and data plane

---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istio-component-monitor
  namespace: istio-system
  labels:
    monitoring: istio-components
    release: <label that prometheus listens for>
spec:
  jobLabel: istio
  selector:
    matchExpressions:
      - {key: istio, operator: In, values: [mixer,pilot,galley,citadel,sidecar-injector]}
  namespaceSelector:
    any: true
  endpoints:
  - port: http-monitoring
    interval: 15s
  - port: http-policy-monitoring
    interval: 15s
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: envoy-stats-monitor
  namespace: istio-system
  labels:
    monitoring: istio-proxies
    release: <label that prometheus listens for>
spec:
  selector:
    matchExpressions:
      - {key: istio-prometheus-ignore, operator: DoesNotExist}
  namespaceSelector:
    any: true
  jobLabel: envoy-stats
  endpoints:
  - path: /stats/prometheus
    targetPort: 15090
    interval: 15s
    relabelings:
    - sourceLabels: [__meta_kubernetes_pod_container_port_name]
      action: keep
      regex: '.*-envoy-prom'
    - action: labeldrop
      regex: "__meta_kubernetes_pod_label_(.+)"
    - sourceLabels: [__meta_kubernetes_namespace]
      action: replace
      targetLabel: namespace
    - sourceLabels: [__meta_kubernetes_pod_name]
      action: replace
      targetLabel: pod_name
#3

Thank you!
I’ll try to use these as a basis for ServiceMonitors for 1.7

#4

Hi, did you get this to work. I’m going to have to tackle this same problem soon and would be great if you could share any results you’ve had.

#5

Yes, I’ve got it to work. Let’s get my notes :slight_smile:

I found some (parts of) examples in different places, and combined them to make it work.
I created a ServiceMonitor for the Istio components, and a PodMonitor for my application pods that I want to monitor and see in Kiali. The monitors are in the monitoring namespace. Furthermore I created a ClusterRole to have the necessary access to other namespaces.
I will add the yaml files below. Note that the ServiceMonitor matchExpression can be simpler, because 1.7 just has one pod. I think you can just match on pilot.
The PodMonitor just select the default namespace. These are things that should be easy to change, but I didn’t get around to it yet. I don’t know what the relabelings are for and if they are needed. I you find out please let me know.

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: istio-proxy-monitor
  namespace: monitoring
  labels:
    istio: proxy
spec:
  jobLabel: component
  selector:
    matchLabels:
      security.istio.io/tlsMode: "istio"
    matchExpressions:
    - key: migration
      operator: NotIn
      values: [ "true", "1" ]

  namespaceSelector:
    matchNames:
    - default

  podMetricsEndpoints:
  - port: http-envoy-prom
    path: /stats/prometheus
    relabelings:
    - action: labeldrop
      regex: __meta_kubernetes_pod_label_skaffold_dev.*
    - action: labeldrop
      regex: __meta_kubernetes_pod_label_pod_template_hash.*
    - action: labelmap
      regex: __meta_kubernetes_pod_label_(.+)
---
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: istio-component-monitor
  namespace: monitoring
  labels:
    k8s-app: istio-components
    monitoring: istio-components
spec:
  jobLabel: istio
  selector:
    matchExpressions:
      - {key: istio, operator: In, values: [mixer,pilot,galley,citadel,sidecar-injector]}
  namespaceSelector:
    matchNames:
    - istio-system
  endpoints:
  - port: http-monitoring
    interval: 15s
  - port: http-policy-monitoring
    interval: 15s
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus-k8s
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
2 Likes
#6

Awesome, thanks. This will surely help me tackle the next few steps. Greatly appreciate the help!

#7

Good information thanks for sharing
vmware