I’m attempting to deploy Istio 1.5.1 with Telemetry v2 enabled and no Mixer applications (Telemetry, Policy services) and no Prometheus from Istio. I get the ServiceMonitors in the istio-system namespace upon deployment, but my Prometheus Operator in another namespace (monitoring) cannot seem to scrape all of these monitors. Namely, classic Istio metrics like istio_requests_total do not appear in Prometheus. I’ve tried numerous approaches from adjusting my scrape configs to what exists upstream to mounting the istio.default secret to Prometheus Operator (since we have mTLS enabled cluster-wide) but cannot seem to get these metrics into my Operator. Targets like envoy-statistics load fine and are healthy, but istio-mesh and anything marked with “secure” fails.
Does anybody have any sage wisdom for how to get this configured properly? I know using your own Prometheus Operator is a supported practice, I just have found the documentation surrounding how to get it up and rolling to be nonexistent.