Kiali only showing Prometheus traffic?

Policies and Telemetry Kiali
1

I have Kiali running and Istio sidecar injection virtually everywhere in the cluster I am testing against.

On Kiali, I can see all the Prometheus traffic throughout the cluster. However, all of my service communication seems to be missing.

I can see in the Istio-Proxy that there are metrics present in the service istio-proxy logs.

istio_requests_total{response_code="200",reporter="destination",source_workload="nginx-ingress-ingress-nginx-controller",source_workload_namespace="platform",source_principal="unknown",source_app="unknown",source_version="unknown",destination_workload="service",destination_workload_namespace="default",destination_principal="unknown",destination_app="service",destination_version="unknown",destination_service="service.default.svc.cluster.local",destination_service_name="service",destination_service_namespace="default",request_protocol="http",response_flags="-",grpc_response_status="",connection_security_policy="none",source_canonical_service="ingress-nginx",destination_canonical_service="service",source_canonical_revision="latest",destination_canonical_revision="latest"} 36

However, Prometheus seems to not show those stats:

image
image1918×287 12.2 KB

The target however, is there:

image
image1525×65 6.35 KB

Does something specific need to be configured on the individual services to view the traffic between them?

This is running Istio 1.6 and Kiali 1.24.2.

2

If I understand, you are seeing traffic in Kiali, such as the graph or in the detail page tabs. But you are not able to find your metrics directly in Prometheus? Kiali is mainly driving off of the prometheus metrics so if you see traffic in Kiali the prom data must be there. Are you sure your prometheus console is connected to the correct prom instance?

3

I didn’t realise I had not replied to this. It ended up being something with the Prometheus metrics not coming in and the way Prometheus was scraping.

All sorted through adjusting the Prometheus rules, this was prior to the 1.16 upgrade which included prometheusMerge, so it was a bit snowflakey.