Can't see metric in prometheus UI

Policies and Telemetry
1

Hi all,

I am new to istio and i have installed on GKE. i am trying to create custom metric for prometheus and see the metric detail in UI. I am successfully able to run prometheus configuration explained in istio documentation.

But when i try to create my own custom metric using different namespace , i don’t see metric in UI but i see that configuration exist on kubernetes which i checked using kubectl command.

I also followed the https://istio.io/help/ops/telemetry/missing-metrics/ and all services and confiuration are correct as per this URL.

Here are the configuration details , i am using

# Configuration for metric instances
apiVersion: "config.istio.io/v1alpha2"
kind: metric
metadata:
  name: myrequestcount
  namespace: my-namespace
spec:
  value: "1" # count each request
  dimensions:
    reporter: conditional((context.reporter.kind | "inbound") == "outbound", "client", "server")
    source: source.workload.name | "unknown"
    destination: destination.workload.name | "unknown"
    message: '"Health check service called!"'
  monitored_resource_type: '"UNSPECIFIED"'
---
# Configuration for a Prometheus handler
apiVersion: "config.istio.io/v1alpha2"
kind: prometheus
metadata:
  name: myrequesthandler
  namespace: my-namespace
spec:
  metrics:
  - name: my_request_count # Prometheus metric name
    instance_name: myrequestcount.metric.my-namespace # Mixer instance name (fully-qualified)
    kind: COUNTER
    label_names:
    - reporter
    - source
    - destination
    - message
---
# Rule to send metric instances to a Prometheus handler
apiVersion: "config.istio.io/v1alpha2"
kind: rule
metadata:
  name: myprom
  namespace: my-namespace
spec:
  actions:
  - handler: myrequesthandler.prometheus
    instances:
    - myrequestcount.metric
---

Namespace and other things are available in GKE and config.

-Kapil

2

What namespace are your deployments in (for which you are trying to see myrequestcount.metric)?

The istio-system namespace is currently special (for better or worse). Configuration applied in that namespace is considered global (applied for every request). Configuration in other namespaces only impacts traffic in those namespaces.