Is there a recommended method of gathering metrics from a secure application (in service mesh) with strict mTLS and managing certificates?
I think I’m making a few assumptions so would just like some advice I guess.
My understanding is that Prometheus is intended to be deployed outside of the service mesh, and comes with a default configuration to scrape from http endpoints outside of the service mesh and https endpoints inside the service mesh.
Prometheus also has a directory with Istio certificates, which I assume is how the sidecar is allowing this through and accepting it’s client certificates with mutual TLS forced.
I am also assuming there is no easy way of translating a regular http endpoint from my application to be accessed as https from the sidecar, so that would mean I would have to implement a custom method of verifying/producing certificates, (or self signing certificates?) and publish a https endpoint in my application?
Alternatively is there a mechanism within Istio for producing valid certificates and loading them into my docker image on deployment.
This seems like quite a bit of effort for something relatively simple, so am I missing anything, and wouldn’t it be easier to just place Prometheus inside the service mesh? (or have one in and one out and use Prometheus federation)