Default namespace metrics not reported

Policies and Telemetry
1

Setup

Service

  • one simple web service(one API: ping) running on k8s under namespace default
  • no sidecar is injected
  • istio-ingressgateway is used to access the service

Metric

  • one simple metric: request count. configured with one instance, one prometheus handler and one rule. similar to example

Tests

access service API: curl
monitor metric: port-forward to query via prometheus UI

1: metric configured under default namespace + no sidecar injected

the configured metric has no data point:
it exists in the query dropdown menu, but no data point.
yet some other default metrics have valid data point, for example istio_requests_total

2: metric configured under istio-system namespace + no sidecar injected

the configured metric has correct data point.

3: metric configured under default namespace + sidecar injected

same result as test 1

4: metric configured under istio-system namespace + sidecar injected

same result as test 2

Questions

  1. seems like metric need to be defined under istio-system namespace in order to be recorded correctly. I may have missed this while reading documentation, can someone point me to where this behavior is explained?

  2. what’s the best practice on what namespace to use for non-default metrics? if I need to created a metric not under istio-system namespace, is there a way to configure mixer to accept that?

  3. I would like to confirm: attributes will be gathered regardless of whether recipient uses an istio sidecar / envoy. i.e. a request from a pod / service running on k8s with istio sidecar injected will generate instances asynchronously inside mixer, regardless of where the desired recipient is.

Thanks in advance!

2

This is the expected behavior based on your configuration (no sidecar in the default namespace). The rules you are defining in the default namespace are restricted to the default namespace – but you have nothing reporting to mixer in that namespace.

A few responses:

  1. Rules defined in istio-system apply to all traffic within the mesh. istio-system is special in this regard. Rules defined in other namespaces only apply to traffic in those namespaces. Since you aren’t injecting any proxies in your namespace, your rule in that namespace has no effect – and the result is expected.

  2. The best practice for non-default rules is to apply them in the namespace that you wish them to function. Again though, this requires proxies in that namespace.

    Alternatively, you can override behavior in mesh-wide rules via the match clause. In your case, you could have a clause like: match: destination.namespace == "default".

  3. Without a sidecar proxy, there is nothing to do any Istio-specific gathering, etc. The concept guide on policies and telemetry should cover how this works.

In your particular situation, the reason you are seeing any metrics at all is because your configuration in istio-system covers reports from your ingressgateway. Istio is generating client-side metrics for the call to the other service (similar to reporter="source"). In order to tell Istio to generate those metrics, the rule must apply in the namespace of the client (which in your case is istio-system, explaining how you see some metrics in those cases).

Hope that helps,
Doug

3

@douglas-reid Thank you so much!
it makes a lot sense now that you pointed the traffic somewhat belongs to a namespace.