Hi all, I’ve been working on getting the community kube prometheus stack helm chart deployment to scrape the metrics endpoints from my Istio Strict mTLS enabled cluster. I feel like I’m fairly close to getting it all working as I can see the istiod and envoy endpoints in the Prometheus Service Discovery tab. However I can see that the related target endpoints under the Prometheus Targets tab are failing due to an http ->> https request issue.
I originally posted this issue with all of the details on the prometheus-community/helm-charts Github but haven’t had any replies yet. Considering this is a bit of a grey area between both communities I think it’s probably worth a cross-post, so here it is:
opened 04:43PM - 06 Apr 22 UTC
bug
### Describe the bug a clear and concise description of what the bug is.
Ok, so… on my journey into getting Prometheus working with Istio using STRICT mutual TLS, here is where I'm up to:
- I've managed to get the promstack deployed successfully with an istio sidecar correctly mounting the Istio certs:
- I've also managed to get the Istio additional scrape config to work as I can now see the endpoints showing up in the Service Discovery:
<img width="621" alt="image" src="https://user-images.githubusercontent.com/4973075/162021975-8af37909-4b5c-49ac-9cc5-e15c820833a9.png">
However, I know something is not working 100% because I can't see any of the istio related queries in the graph query dropdown i.e 'istio_request_totals' etc and because the Istio Targets aren't working with the error 'server gave HTTP response to HTTPS client':
I've also tried ServiceMonitors and PodMonitors instead of the custom scrapers but same result. I feel like I'm pretty close to getting this working but after perusing what must be 100+ Github issues and other related posts I'm a little stuck and in need of community advice.
Do I need to be changing a value or annotation somewhere in the prometheus operator or CR spec to fix this http -> https issue? Any advice would be welcome on the issue!
Here are my relevant files:
[prometheus-stack-values.txt](https://github.com/prometheus-community/helm-charts/files/8428795/prometheus-stack-values.txt)
[istio.custom.operator_ha.txt](https://github.com/prometheus-community/helm-charts/files/8428806/istio.custom.operator_ha.txt)
### What's your helm version?
version.BuildInfo{Version:"v3.8.0", GitCommit:"d14138609b01886f544b2025f5000351c9eb092e", GitTreeState:"clean", GoVersion:"go1.17.5"}
### What's your kubectl version?
Client Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"b695d79d4f967c403a96986f1750a35eb75e75f1", GitTreeState:"clean", BuildDate:"2021-11-17T15:48:33Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"} Server Version: version.Info{Major:"1", Minor:"22", GitVersion:"v1.22.4", GitCommit:"19bca9208454ad7ac76f6fc71b5a2520922e395b", GitTreeState:"clean", BuildDate:"2022-03-10T18:21:44Z", GoVersion:"go1.16.10", Compiler:"gc", Platform:"linux/amd64"}
### Which chart?
prometheus-community/kube-prometheus-stack
### What's the chart version?
34.1.1
### What happened?
_No response_
### What you expected to happen?
_No response_
### How to reproduce it?
_No response_
### Enter the changed values of values.yaml?
prometheusOperator:
podAnnotations:
sidecar.istio.io/rewriteAppHTTPProbers: "true"
admissionWebhooks:
patch:
podAnnotations:
sidecar.istio.io/inject: "false"
grafana:
adminPassword: strongpassword
podAnnotations:
sidecar.istio.io/rewriteAppHTTPProbers: "true"
prometheus:
serviceMonitor:
scheme: "https"
tlsConfig:
caFile: /etc/istio-output-certs/root-cert.pem
keyFile: /etc/istio-output-certs/key.pem
certFile: /etc/istio-output-certs/cert-chain.pem
insecureSkipVerify: true
prometheusSpec:
logLevel: info
additionalScrapeConfigs:
- job_name: 'istiod'
scheme: https
tls_config:
ca_file: /etc/istio-output-certs/root-cert.pem
key_file: /etc/istio-output-certs/key.pem
cert_file: /etc/istio-output-certs/cert-chain.pem
insecure_skip_verify: true # Prometheus does not support Istio security naming, thus skip verifying target pod ceritifcate
kubernetes_sd_configs:
- role: endpoints
namespaces:
names:
- istio-system
relabel_configs:
- source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
action: keep
regex: istiod;http-monitoring
- job_name: 'envoy-stats'
scheme: https
tls_config:
ca_file: /etc/istio-output-certs/root-cert.pem
key_file: /etc/istio-output-certs/key.pem
cert_file: /etc/istio-output-certs/cert-chain.pem
insecure_skip_verify: true # Prometheus does not support Istio security naming, thus skip verifying target pod ceritifcate
metrics_path: /stats/prometheus
kubernetes_sd_configs:
- role: pod
relabel_configs:
- source_labels: [__meta_kubernetes_pod_container_port_name]
action: keep
regex: '.*-envoy-prom'
alertingEndpoints:
- name: promstack-prometheus-operator-alertmanager
namespace: default
port: web
scheme: https
pathPrefix: /
apiVersion: v2
tlsConfig:
caFile: /etc/istio-output-certs/root-cert.pem
keyFile: /etc/istio-output-certs/key.pem
certFile: /etc/istio-output-certs/cert-chain.pem
insecureSkipVerify: true
volumeMounts:
- name: istio-certs-dir
mountPath: /etc/istio-output-certs/
readOnly: true
volumes:
- name: istio-certs-dir
emptyDir:
medium: Memory
podMetadata:
annotations:
sidecar.istio.io/inject: "true"
sidecar.istio.io/rewriteAppHTTPProbers: "true"
traffic.sidecar.istio.io/includeInboundPorts: "" # do not intercept any inbound ports
traffic.sidecar.istio.io/includeOutboundIPRanges: "" # do not intercept any outbound traffic
proxy.istio.io/config: | # configure an env variable `OUTPUT_CERTS` to write certificates to the given folder
proxyMetadata:
OUTPUT_CERTS: /etc/istio-output-certs
sidecar.istio.io/userVolumeMount: '[{"name": "istio-certs-dir", "mountPath": "/etc/istio-output-certs"}]' # mount the shared volume at sidecar proxy
labels:
sidecar.istio.io/inject: "true"
# securityContext:
# runAsGroup: 1337
# runAsNonRoot: true
# runAsUser: 1337
# fsGroup: 1337
alertmanager:
alertmanagerSpec:
podMetadata:
annotations:
sidecar.istio.io/rewriteAppHTTPProbers: "true"
serviceMonitor:
tlsConfig:
caFile: /etc/istio-output-certs/root-cert.pem
keyFile: /etc/istio-output-certs/key.pem
certFile: /etc/istio-output-certs/cert-chain.pem
insecureSkipVerify: true
kube-state-metrics:
podAnnotations:
sidecar.istio.io/rewriteAppHTTPProbers: "true"
### Enter the command that you execute and failing/misfunctioning.
helm install promstack prometheus-community/kube-prometheus-stack --namespace monitoring --create-namespace --values prometheus-stack-values.yaml
### Anything else we need to know?
_No response_
I’ve included the relevant prometheus values config and the istio cr/operator yaml files in the above linked github post.
Any advice on how to help me push this over the finishing line or a steer in the direction of what I might be missing would be much appreciated!
