We have a strict requirement for encryption of all data in transit. We are already using Istio for our apps to handle this, but we also need to integrate Prometheus into this. I’ve used Prometheus many times without Istio with success, but Istio seems to be causing a few issues.
It’s a common thing in Prometheus to connect to pods directly, such as with the Alert Manger peering, the node exporters, and I think any of the pods it scrapes, and this seems to be outside of the normal pattern with Istio. For example, for the node exporters, Prometheus will make a http connection to http://10.48.177.162:9100/metrics. The Envoy logs show something like this:
[2019-05-23T19:06:44.349Z] "- - -" 0 UF,URX "-" 0 0 1 - "-" "-" "-" "-" "10.48.177.162:9100" outbound|9100||prometheus-node-exporter.monitoring.svc.cluster.local - 10.48.177.162:9100 10.48.176.103:56676 -
Has anyone else here had success with getting their own Prometheus integrated with Istio mTLS and would mind sharing configs? If not, are there any particular Istio objects that need to be created to handle these “direct-to-pod” connections?
I’m deploying from the standard stable/prometheus helm chart, only adjustments are using statefulsets for the alertmanager and server to enable HA.