BYO Prometheus with mTLS

We have a strict requirement for encryption of all data in transit. We are already using Istio for our apps to handle this, but we also need to integrate Prometheus into this. I’ve used Prometheus many times without Istio with success, but Istio seems to be causing a few issues.

It’s a common thing in Prometheus to connect to pods directly, such as with the Alert Manger peering, the node exporters, and I think any of the pods it scrapes, and this seems to be outside of the normal pattern with Istio. For example, for the node exporters, Prometheus will make a http connection to http://10.48.177.162:9100/metrics. The Envoy logs show something like this:

[2019-05-23T19:06:44.349Z] "- - -" 0 UF,URX "-" 0 0 1 - "-" "-" "-" "-" "10.48.177.162:9100" outbound|9100||prometheus-node-exporter.monitoring.svc.cluster.local - 10.48.177.162:9100 10.48.176.103:56676 -

Has anyone else here had success with getting their own Prometheus integrated with Istio mTLS and would mind sharing configs? If not, are there any particular Istio objects that need to be created to handle these “direct-to-pod” connections?

I’m deploying from the standard stable/prometheus helm chart, only adjustments are using statefulsets for the alertmanager and server to enable HA.

1 Like

@firebert have you looked at the addon configuration for secure pods by any chance? Prom job config and deployment spec ?

Do those help at all in configuring your server? Could you use a similar setup for node exporter, etc. ?

Thanks! I will take a look in more detail, but looks promising so far.

One of the issues with the node exporter is it doesn’t seem to take the Envoy sidecar, most likely due to it using the host’s network space. I’ll experiment with this a bit and see what I come up with and share back here if I get a working config.