Not sure if anyone has run into this issue, but it seems that when I define custom metrics endpoints on my workloads and prometheus scrapes them, istio marks them as “unknown” source, since Prometheus is not within my service mesh. I have previously tried to include Prometheus within my mesh, but seems that Envoy is unable to handle direct IP traffic; Prometheus’ service discovery on Kubernetes uses IP addresses. Is there a way to filter out traffic from Prometheus so that metrics are not clouded by “false unknowns”?
One possible solution: update the
match clause on your metrics rules to exclude url paths that are metrics specific (example: add
&& request.url_path != "/metrics").