Recommended Strategy for Gathering Metrics with Strict mTLS on

Is there a recommended method of gathering metrics from a secure application (in service mesh) with strict mTLS and managing certificates?

I think I’m making a few assumptions so would just like some advice I guess.

My understanding is that Prometheus is intended to be deployed outside of the service mesh, and comes with a default configuration to scrape from http endpoints outside of the service mesh and https endpoints inside the service mesh.

Prometheus also has a directory with Istio certificates, which I assume is how the sidecar is allowing this through and accepting it’s client certificates with mutual TLS forced.

I am also assuming there is no easy way of translating a regular http endpoint from my application to be accessed as https from the sidecar, so that would mean I would have to implement a custom method of verifying/producing certificates, (or self signing certificates?) and publish a https endpoint in my application?

Alternatively is there a mechanism within Istio for producing valid certificates and loading them into my docker image on deployment.

This seems like quite a bit of effort for something relatively simple, so am I missing anything, and wouldn’t it be easier to just place Prometheus inside the service mesh? (or have one in and one out and use Prometheus federation)

Alternatively is there a mechanism within Istio for producing valid certificates and loading them into my docker image on deployment.

AFAIK, this is not supported, cc @incfly @Oliver

This seems like quite a bit of effort for something relatively simple, so am I missing anything, and wouldn’t it be easier to just place Prometheus inside the service mesh?

What do you mean to place Prometheus inside the service mesh? Like injecting a sidecar for it? @incfly @douglas-reid may know more about this.

Placing Prometheus inside the service mesh is a bit complicated, given that Prom uses its own discovery to find endpoints and go direct to those endpoints. But, we are working on a way to allow Prom to have its own sidecar presently. In the meantime, it uses file-mounted certs (but it isn’t doing the mutual part of mTLS).

FWIW, Prom can scrape both HTTP and HTTPS endpoints within a mesh. The only issue comes with PERMISSIVE mode settings, AFAIK.