I am deploying Prometheus/Alertmanager/Grafana to my cluster using the latest kube-prometheus-stack helm chart (formerly known as the prometheus operator helm chart).
This chart provides the Helm Operator CRDs like ServiceMonitor and PodMonitor, which are really nice for exposing metrics to Prometheus. I’ve got a ServiceMonitor defined to pull in my custom workload metrics and the operator already includes monitors to pull in general Kubernetes cluster metrics, but I’m not sure how to proceed with Istio.
I’m curious about both the architecture of Istio metrics collection and the practical configuration side of things. With the current Istio architecture, am I right to guess that all metrics for the mesh will be exposed from Istiod? Or do I need to have Prometheus pull metrics directly from envoy containers in the mesh somehow?
Assuming all metrics are collected and exposed by Istiod, I would imagine that I can create a ServiceMonitor which targets the Istiod service in the istio-system namespace and scrape that? If I run kubectl get endpoints -n istio-system istiod-1-9-5 -o yaml, I see an endpoint named http-monitoring. Is that maybe a metrics endpoint?
I guess I’m wondering also if anyone out there has already tackled this problem. Is there any configuration available to pull Istio metrics into the Prometheus Operator, using a ServiceMonitor or PodMonitor? Thanks.