Hey team,
We have already istio running in the production environment and we have our own Grafana/Prometheus deployment which is installed via helm charts. My task is to check and analyze current Istio deployment and make improvements for the future. I have realized that we have only istiod pod running in istio-system namespace, so I started with installing the Kiali deployment. Per the documentation, only Prometheus integration is enough to collect metrics for populating the graph, since we are not using Grafana/Prometheus deployment from Istio itself I understand that I need to manually update CM of Kiali and restart deployment with the current service name of Prometheus like below.
spec: external_services: prometheus: # Prometheus service name is "metrics" and is in the "telemetry" namespace url: "http://metrics.telemetry:9090/"
I already did it and I restarted deployment, I don’t see any logs in the Kiali pod regarding Prometheus and I can see the correct version of it from the Kiali dashboard. I can see only logs regarding tracing which I believe I can skip for now.
I also went to the Prometheus console and tried to use istio_requests_total in the expression, but did not get anything as a result. Could you please help me to check what could be the issue?