Problems adding custom metric with "Stats" extension

Policies and Telemetry
1

Hi!

I’m trying to add a custom metric as documented in https://istio.io/latest/docs/tasks/observability/metrics/customize-metrics/ but the metric doesn’t not appear in Prometheus.

Here is my config

...
- applyTo: HTTP_FILTER
  match:
    context: SIDECAR_INBOUND
    listener:
      filterChain:
        filter:
          name: envoy.http_connection_manager
          subFilter:
            name: envoy.router
    proxy:
      proxyVersion: ^1\.6.*
  patch:
    operation: INSERT_BEFORE
    value:
      name: istio.stats
      typed_config:
        '@type': type.googleapis.com/udpa.type.v1.TypedStruct
        type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
        value:
          config:
            configuration: |
              {
                "debug": "true",
                "stat_prefix": "istio",
                "definitions": [
                  {
                    "name": "my_custom_metric",
                    "value": "response.code",
                    "type": "COUNTER"
                  }
                ],
                "metrics": [
                  {
                    "name": "requests_total",
                    "dimensions": {
                      "my_custom_tag": "string(response.code)"
                    }
                  }
                ]
              }
            root_id: stats_inbound
            vm_config:
              code:
                local:
                  inline_string: envoy.wasm.stats
              runtime: envoy.wasm.runtime.null
              vm_id: stats_inbound
...

"value": "response.code" - is just for example, I plan to use a more meaningful attribute, just trying to make it work with something simple first.
I also tried "value": "1.1" - no luck.

I don’t see the metric in Prometheus by querying "{__name__=~".*my_custom_metric.*"}".
And nothing in kubectl exec $MY_PODNAME -c istio-proxy -- curl 'localhost:15000/stats/prometheus' | grep 'my_custom_metric'

While I see that my_custom_tag in either of the places, so adding custom tag works, but custom metrics does not.

And no metrics-related errors in Envoy logs kubectl logs $MY_PODNAME -c istio-proxy | grep -e "Config Error" -e "envoy wasm"

Anything I’m doing wrong?

Istio version: 1.6.3

2

@kuat can you check the config ?

3

It looks right (tested locally). Maybe add a dimension to the custom metric? It might be skipped somewhere due to lack of dimensions.

4

hm… still no luck after adding the dimension @kuat

            configuration: |
              {
                "debug": "true",
                "stat_prefix": "istio",
                "definitions": [
                  {
                    "name": "my_custom_metric",
                    "value": "response.code",
                    "type": "COUNTER"
                  }
                ],
                "metrics": [
                  {
                    "name": "requests_total",
                    "dimensions": {
                      "my_custom_tag": "string(response.code)"
                    }
                  },
                  {
                    "name": "my_custom_metric",
                    "dimensions": {
                      "my_custom_tag": "string(response.code)"
                    }
                  }
                ]
              }

I have Istio installed via istioctl manifest apply --set profile=demo
And Kubernetes version is 1.16.5

5

New dimension names require extraStatTags. Can you try an existing dimension name, such as reporter?

6

That worked! However, I was having the extraStatTags annotation set and it was working at least partially, I understand - because I was getting that tag in Prometheus for the built-in requests_total metric.

apiVersion: apps/v1
kind: Deployment
  ...
  template:
    metadata:
      labels:
        app: myappname
      annotations:
        sidecar.istio.io/extraStatTags: my_custom_tag

So, then… how to be able to have a custom, not built-in tag, for a custom metric?
Also, you mean it’s not possible to have a metric with just a name and without the dimensions?

7

The extra stat tag requires a proxy restart, so perhaps, the relevant proxy was just stale? The case of a metric without dimensions is a corner case, so I assume it gets filtered out somewhere in istio or envoy pipeline.

8

(the Envoy proxy, right?)
no… I’m doing the restarts of the relevant pods after adding sidecar.istio.io/extraStatTags, anyway I additionally did kubectl rollout restart deployments/myapp - no luck.

basically if I configure

                "definitions": [
                  {
                    "name": "my_custom_metric",
                    "value": "1",
                    "type": "COUNTER"
                  }
                ],
                "metrics": [
                  {
                    "name": "my_custom_metric",
                    "dimensions": {
                      "reporter": "'xxx'"
                    }
                  }
                ]

then it works

but if I add a custom tag, which is in sidecar.istio.io/extraStatTags of the relevant Deployment:

                "metrics": [
                  {
                    "name": "my_custom_metric",
                    "dimensions": {
                      "reporter": "'xxx'",
                      "my_custom_tag": "'xxx'"
                    }
                  }
                ]

then it stops working

9

That’s interesting. Perhaps, you can try query envoy admin port :15000/stats and share what it looks like for my_custom_metric? Also, can you get the config dump from envoy (especially bootstrap) to make sure the custom tag actually makes it there?

10 
  $ kubectl exec $HELLODS_PODNAME -c istio-proxy -- curl -s 'localhost:15000/stats' | grep 'my_custom_metric'
  reporter=.=xxx;.;_istio_my_custom_metric: 2

I did 2 requests before adding my_custom_tag - we see it’s working.
Then I add my_custom_tag, did some more requests but the counter stopped growing, and if I restart the pod - then nothing is returned from /stats.

11 
kubectl exec $HELLODS_PODNAME -c istio-proxy -- curl -s 'localhost:15000/config_dump'
12

I think 1.6.3 has a nasty segfault that causes envoy to restart on a configuration change for configurable metrics (see https://github.com/envoyproxy/envoy-wasm/pull/547). That has been fixed on master and I think made it to 1.6.4.

13

Same with Istio 1.6.4.

Envoy version is 1.14.3

kubectl exec -it hellods-57794d5747-mzsp5 -c istio-proxy  pilot-agent request GET server_info
{
 "version": "80ad06b26b3f97606143871e16268eb036ca7dcd/1.14.3-dev/Clean/RELEASE/BoringSSL",
 "state": "LIVE",
 "hot_restart_version": "11.104"

Looks like it has been released after your PR was merged…

Or it should be in the next Istio release?

The new Envoy config dump, if needed: https://drive.google.com/file/d/1GfhhCtJ5nIq9MZmYba2XGl11vT_kil1e/view?usp=sharing

14

I think it didn’t make it to 1.6.4 since CVE fixes took precedence. 1.6.5 should be coming soon, and I believe it should have it. For now, using MeshConfig extraStatTags during installation and modifying EnvoyFilter during installation works.

15

Thanks a lot @kuat !