Istio 1.5.1 Operator, Mixerless Metrics with Prometheus Operator

1

Hi,

I’m attempting to deploy Istio 1.5.1 with Telemetry v2 enabled and no Mixer applications (Telemetry, Policy services) and no Prometheus from Istio. I get the ServiceMonitors in the istio-system namespace upon deployment, but my Prometheus Operator in another namespace (monitoring) cannot seem to scrape all of these monitors. Namely, classic Istio metrics like istio_requests_total do not appear in Prometheus. I’ve tried numerous approaches from adjusting my scrape configs to what exists upstream to mounting the istio.default secret to Prometheus Operator (since we have mTLS enabled cluster-wide) but cannot seem to get these metrics into my Operator. Targets like envoy-statistics load fine and are healthy, but istio-mesh and anything marked with “secure” fails.

Does anybody have any sage wisdom for how to get this configured properly? I know using your own Prometheus Operator is a supported practice, I just have found the documentation surrounding how to get it up and rolling to be nonexistent.

Thanks.

2

@AlexDHoffer could you share how did you achieve mounting istio.default secret with prometheus operator?

I have something similar issue and trying to make prometheus scrape the metrics from strict mtls enabled namespaces.

see here and here for my issues, not sure if that will help you

3

@deepak_deore this comment on a GitHub issue was the approach I used to manually mount the secret onto my operator: https://github.com/istio/istio/issues/7352#issuecomment-439617432

4

@AlexDHoffer 1.5.x istio doesn’t create istio.default secret in namespaces, do you provide some option to istio that makes it create this secret?

5

@deepak_deore I would assume that’s being populated by our cert-manager instance, at least on our cluster with cert-manager on the ingress gateway. For our cluster without cert-manager, it could be a holdover from older Istio versions.

I should also clarify that I never got this working, so I don’t want to make it appear as though this solution worked.

6

We haven’t tried 1.5 but on 1.4 we got prometheus operator scraping strict mtls targets. We needed to disable sidecar injection for prometheus pods and then mount the mtls certs like in here: https://github.com/istio/istio/issues/7352#issuecomment-439617432

This could have all changed with 1.5 though. What sort of errors are you seeing on prometheus when targeting mtls services?

7

@dima-123 We were trying to incorporate the service monitors created (from prometheusOperator = true) in the Istio namespace into our own Prometheus Operator. It was not working. But we got this working by simply writing our own service monitors that just scrape the endpoints of the sidecars. All normal data is loaded in. This may not be the practice Istio suggests but we much prefer it to using a bundled Prometheus instance.

8

Did you ever get this working? Reading this thread, I realized that the only reason my prometheus operator is able to successfully scrape off istio injected pods is because of the legacy istio.default secret left behind throughout istio installations. I rather get it working on 1.5 recommended way

9

in my setup i will be creating the istio.default secret manually (or through some k8 job to keep it updated) and mounting inside ptometheus like the 1.4 way

having sidecar to manage this with prometheus that is installed by prometheus operator makes it more complex.

10

if someone is looking how I created the cert manually (is simple to automate but i havent done that yet):

  1. save istio key and cert from istio-system/istio-ca-secret to /tmp/ca-key.pem and /tmp/ca-cert.pem
  2. create key - openssl genrsa -out /tmp/default.key 2048
  3. create csr - openssl req -new -key /tmp/default.key -out /tmp/default.csr -config /tmp/istio_openssl.conf
  4. create cert - openssl x509 -sha256 -req -days 3650 -in /tmp/default.csr -CA /tmp/ca-cert.pem -CAkey /tmp/ca-key.pem -extensions v3_req -out /tmp/default.crt -extfile /tmp/istio_openssl.conf

istio_openssl.conf (adjust URI.1 to the namespace and serviceaccount used by prometheus first “default” is namespace and second “default” is sertviceaccount used by the pod that will use this cert):

[req]
distinguished_name = req_distinguished_name
req_extensions = v3_req
prompt = no

# req_distinguished_name may be removed but i kept it with my company details
[req_distinguished_name]
C = COUNTRY
ST = STATE
L = LOCATION
O = ORG NAME
OU = OU

[v3_req]
keyUsage = critical, digitalSignature, keyEncipherment
extendedKeyUsage=serverAuth,clientAuth
basicConstraints = critical, CA:FALSE
subjectAltName = critical, @alt_names

[alt_names]
URI.1 = spiffe://cluster.local/ns/default/sa/default