BYO Prometheus with mTLS

[firebert]

We have a strict requirement for encryption of all data in transit. We are already using Istio for our apps to handle this, but we also need to integrate Prometheus into this. I’ve used Prometheus many times without Istio with success, but Istio seems to be causing a few issues.

It’s a common thing in Prometheus to connect to pods directly, such as with the Alert Manger peering, the node exporters, and I think any of the pods it scrapes, and this seems to be outside of the normal pattern with Istio. For example, for the node exporters, Prometheus will make a http connection to http://10.48.177.162:9100/metrics. The Envoy logs show something like this:

[2019-05-23T19:06:44.349Z] "- - -" 0 UF,URX "-" 0 0 1 - "-" "-" "-" "-" "10.48.177.162:9100" outbound|9100||prometheus-node-exporter.monitoring.svc.cluster.local - 10.48.177.162:9100 10.48.176.103:56676 -

Has anyone else here had success with getting their own Prometheus integrated with Istio mTLS and would mind sharing configs? If not, are there any particular Istio objects that need to be created to handle these “direct-to-pod” connections?

I’m deploying from the standard stable/prometheus helm chart, only adjustments are using statefulsets for the alertmanager and server to enable HA.

[douglas-reid]

@firebert have you looked at the addon configuration for secure pods by any chance? Prom job config and deployment spec ?

Do those help at all in configuring your server? Could you use a similar setup for node exporter, etc. ?

[firebert]

Thanks! I will take a look in more detail, but looks promising so far.

One of the issues with the node exporter is it doesn’t seem to take the Envoy sidecar, most likely due to it using the host’s network space. I’ll experiment with this a bit and see what I come up with and share back here if I get a working config.

[eperdeme]

Hi, sorry to hijack this a bit but I’ve implemented the solution which works, but I don’t understand why it works.

Every article I read says that the prometheus envoy sidecar will intercept the traffic destined to the destination pod IP and then create an envoy->envoy TLS session and send the traffic over it. So why does Prometheus need to know about the istio certs and use HTTPS ?

Is it because Prometheus uses IP addresses to talk to apps and MTLS via sidecar only works if using hostnames?

[douglas-reid]

@eperdeme the default recommendation is to not use a sidecar for prometheus. if you look at the configuration, you will see that sidecar injection is specifically disabled.

[chainswagger]

Hi @douglas-reid,
Would there be an alternative to this post 1.5 since istio.default secret does not get created anymore?

[111]

For prometheus to integrate with istio, there is a blog here: https://istio.io/blog/2020/proxy-cert/

please note that the sidecar here is only used for certificate management instead of handling traffic (for pod-to-pod mtls issue which is not istio supported pattern currently.)

[chainswagger]

@111
I just reviewed the helm chart.
This does not seem like this integrates well with a self managed prometheus operator instance and assumes that you are only going to use the bundled installation with istio.

I’m also looking at the prometheusoperator template in the repo, and it seems like it tries to mount a secret called istio.prometheus to scrape mtls enabled targets but it doesn’t indicate where that secret comes from.

Is this the only supported way to accomplish this task?

[douglas-reid]

As an updated FYI: Prometheus integration with Istio has been changing with the releases. The latest set of changes move to eliminate any custom configuration required to scrape Istio targets.

[chainswagger]

Can we get access to read that document?

[douglas-reid]

It is available on the Istio Community drive. Access is covered here.

Anybody can access the team drive for reading and commenting. To get access simply join the istio-team-drive-access@ group. Once you’ve done that, head to Team Drive and behold all the docs.

[111]

@chainswagger, you can read the istio.io page first to find there are 2 options here, option 2 is disabled by default and has some limitation (not tls for both control plane and application for scraping)

istio.io doc: https://istio.io/latest/docs/ops/integrations/prometheus/
github: https://github.com/istio/istio/pull/22318
design doc: https://docs.google.com/document/d/1TTeN4MFmh4aUYYciR4oDBTtJsxl5-T5Tu3m3mGEdSo8/edit

[naveen.adisesha]

In our cluster, where we have our internal prometheus we plan to scrape the pod endpoints by making the pod endpoints as part of a headless service. That way prometheus can have istio-proxy container injected and also reach pod ips directly. No need to mount the certs to reach mtls endpoints. I want to know if or why the istio team does/does not recommend to create a headless service when they want to reach pod/port endpoints so that mtls is auto applied ? Any views on this method ?

Describe our approach here.

Istio-mTLS and POD IP, Port

Have seen that lot of teams are having issue to reach podIP and ports when applying mtls-strict. Infact we had some difficult trying to achieve that using service entry and rules as maintenance is tough. We worked around the problem using the following approach. Basically the pod and port combination we created as a service entry. We make that service as headless. In that way we can avoid managing the endpoints manually using Service Entry, Also all the endpoints,port are treated as service an…

[Satish]

Hi,
I have similar issue. Can you please share if you managed to integrate prometheus with istio using mtls. thanks

[douglas-reid]

@Satish do you require mTLS for prometheus scraping, or can you use the prometheus merge functionality to scrape in plaintext ?

[Satish]

@douglas-reid
thanks for reply.
I managed to solve this. I deployed our own prometheus with istio side car including istio_init. All I did is mounted istio-certs folder into prometheus container and configured service monitors to scrape using rotated istio certs while in strict mode. prometheus scraping and grafana all in mtls using istio proxy.

However I have not tested or verified side car metrics which I understand will be in plain text. For istio proxy metrics, can I simply create servicemonitors to scrape them. Can you please point me to an example to create service monitors for istio proxy in plan text.

[deepak_deore]

i did same

created a feature request with prometheus operator to provide options to specify certs for scraping some targets

this feature request also has configuration in case any one wants to refer.