Hi all, I’ve been working on getting the community kube prometheus stack helm chart deployment to scrape the metrics endpoints from my Istio Strict mTLS enabled cluster. I feel like I’m fairly close to getting it all working as I can see the istiod and envoy endpoints in the Prometheus Service Discovery tab. However I can see that the related target endpoints under the Prometheus Targets tab are failing due to an http ->> https request issue.
I originally posted this issue with all of the details on the prometheus-community/helm-charts Github but haven’t had any replies yet. Considering this is a bit of a grey area between both communities I think it’s probably worth a cross-post, so here it is:
opened 04:43PM - 06 Apr 22 UTC
bug
### Describe the bug a clear and concise description of what the bug is.
Ok, so… on my journey into getting Prometheus working with Istio using STRICT mutual TLS, here is where I'm up to:
- I've managed to get the promstack deployed successfully with an istio sidecar correctly mounting the Istio certs:
- I've also managed to get the Istio additional scrape config to work as I can now see the endpoints showing up in the Service Discovery:
<img width="621" alt="image" src="https://user-images.githubusercontent.com/4973075/162021975-8af37909-4b5c-49ac-9cc5-e15c820833a9.png">
However, I know something is not working 100% because I can't see any of the istio related queries in the graph query dropdown i.e 'istio_request_totals' etc and because the Istio Targets aren't working with the error 'server gave HTTP response to HTTPS client':
I've also tried ServiceMonitors and PodMonitors instead of the custom scrapers but same result. I feel like I'm pretty close to getting this working but after perusing what must be 100+ Github issues and other related posts I'm a little stuck and in need of community advice.
Do I need to be changing a value or annotation somewhere in the prometheus operator or CR spec to fix this http -> https issue? Any advice would be welcome on the issue!
Here are my relevant files:
[prometheus-stack-values.txt](https://github.com/prometheus-community/helm-charts/files/8428795/prometheus-stack-values.txt)
[istio.custom.operator_ha.txt](https://github.com/prometheus-community/helm-charts/files/8428806/istio.custom.operator_ha.txt)
### What's your helm version?
version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.5"}
### What's your kubectl version?
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"19bca9208454ad7ac76f6fc71b5a2520922e395b", GitTreeState:"clean", BuildDate:"2022-03-10T18:21:44Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
### Which chart?
prometheus-community/kube-prometheus-stack
### What's the chart version?
34.1.1
### What happened?
_No response_
### What you expected to happen?
_No response_
### How to reproduce it?
_No response_
### Enter the changed values of values.yaml?
prometheusOperator:
podAnnotations:
sidecar.istio.io/rewriteAppHTTPProbers: "true"
admissionWebhooks:
patch:
podAnnotations:
sidecar.istio.io/inject: "false"
grafana:
adminPassword: strongpassword
podAnnotations:
sidecar.istio.io/rewriteAppHTTPProbers: "true"
prometheus:
serviceMonitor:
scheme: "https"
tlsConfig:
caFile: /etc/istio-output-certs/root-cert.pem
keyFile: /etc/istio-output-certs/key.pem
certFile: /etc/istio-output-certs/cert-chain.pem
insecureSkipVerify: true
prometheusSpec:
logLevel: info
additionalScrapeConfigs:
- job_name: 'istiod'
scheme: https
tls_config:
ca_file: /etc/istio-output-certs/root-cert.pem
key_file: /etc/istio-output-certs/key.pem
cert_file: /etc/istio-output-certs/cert-chain.pem
insecure_skip_verify: true # Prometheus does not support Istio security naming, thus skip verifying target pod ceritifcate
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- istio-system
relabel_configs:
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: istiod;http-monitoring
- job_name: 'envoy-stats'
scheme: https
tls_config:
ca_file: /etc/istio-output-certs/root-cert.pem
key_file: /etc/istio-output-certs/key.pem
cert_file: /etc/istio-output-certs/cert-chain.pem
insecure_skip_verify: true # Prometheus does not support Istio security naming, thus skip verifying target pod ceritifcate
metrics_path: /stats/prometheus
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_port_name]
action: keep
regex: '.*-envoy-prom'
alertingEndpoints:
- name: promstack-prometheus-operator-alertmanager
namespace: default
port: web
scheme: https
pathPrefix: /
apiVersion: v2
tlsConfig:
caFile: /etc/istio-output-certs/root-cert.pem
keyFile: /etc/istio-output-certs/key.pem
certFile: /etc/istio-output-certs/cert-chain.pem
insecureSkipVerify: true
volumeMounts:
- name: istio-certs-dir
mountPath: /etc/istio-output-certs/
readOnly: true
volumes:
- name: istio-certs-dir
emptyDir:
medium: Memory
podMetadata:
annotations:
sidecar.istio.io/inject: "true"
sidecar.istio.io/rewriteAppHTTPProbers: "true"
traffic.sidecar.istio.io/includeInboundPorts: "" # do not intercept any inbound ports
traffic.sidecar.istio.io/includeOutboundIPRanges: "" # do not intercept any outbound traffic
proxy.istio.io/config: | # configure an env variable `OUTPUT_CERTS` to write certificates to the given folder
proxyMetadata:
OUTPUT_CERTS: /etc/istio-output-certs
sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs-dir", "mountPath": "/etc/istio-output-certs"}]' # mount the shared volume at sidecar proxy
labels:
sidecar.istio.io/inject: "true"
# securityContext:
# runAsGroup: 1337
# runAsNonRoot: true
# runAsUser: 1337
# fsGroup: 1337
alertmanager:
alertmanagerSpec:
podMetadata:
annotations:
sidecar.istio.io/rewriteAppHTTPProbers: "true"
serviceMonitor:
tlsConfig:
caFile: /etc/istio-output-certs/root-cert.pem
keyFile: /etc/istio-output-certs/key.pem
certFile: /etc/istio-output-certs/cert-chain.pem
insecureSkipVerify: true
kube-state-metrics:
podAnnotations:
sidecar.istio.io/rewriteAppHTTPProbers: "true"
### Enter the command that you execute and failing/misfunctioning.
helm install promstack prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace --values prometheus-stack-values.yaml
### Anything else we need to know?
_No response_
I’ve included the relevant prometheus values config and the istio cr/operator yaml files in the above linked github post.
Any advice on how to help me push this over the finishing line or a steer in the direction of what I might be missing would be much appreciated!
As you can read in the documentation about scraping metrics with mTLS: The control plane, gateway, and Envoy sidecar metrics will all be scraped over plaintext. This means that you cannot scrape Istio metrics with mTLS. However, you can use mTLS to scrape application metrics exposed by an app on a custom port.
