Prometheus traffic marked as "unknown" on custom endpoints


#1

Hello all,
Not sure if anyone has run into this issue, but it seems that when I define custom metrics endpoints on my workloads and prometheus scrapes them, istio marks them as “unknown” source, since Prometheus is not within my service mesh. I have previously tried to include Prometheus within my mesh, but seems that Envoy is unable to handle direct IP traffic; Prometheus’ service discovery on Kubernetes uses IP addresses. Is there a way to filter out traffic from Prometheus so that metrics are not clouded by “false unknowns”?


#2

One possible solution: update the match clause on your metrics rules to exclude url paths that are metrics specific (example: add && request.url_path != "/metrics").