Customize Metrics with EnvoyFilter... Invalid expression?

Policies and Telemetry
1

Background:
Recently I am trying to deploy prometheus according to the official documentation. My k8s cluster had istio deployed a long time ago and it was working fine for a while. At the same time, a set of microservice backend architecture is deployed on my cluster.
However, after successfully deploying prometheus and scraping data from istio proxy, I realized that I need to obtain information at the http request level.
According to the official documentation: Istio / Customizing Istio Metrics, I did successfully got the request.url_path label. The only regret is that request.url_path will generate metrics data with a particularly wide range of values for the case of path parameters(like GET /api/v1/product/:id), which causes my prometheus memory usage to increase sharply.
My solution is to add fullpath in the response header (I use gin as my backend framework), thus aggregating my metrics.
But for my rpc service, I don’t want to add extra label (such as header) again. Instead, I hope to solve this by Envoy Filter. The logic is very simple:
If it is a grpc request, the label of the metrics uses “request.url_path”. Otherwise use my custom label “response.headers[‘X-Prometheus-Url-Path’]”.

Question:
Is there any way to achieve this? I’ve tried using various Envoy Attributes, but it does not work.
I also tried using cel like “has (response.headers[‘X-Prometheus-Url-Path’]) ? response.headers[‘X-Prometheus-Url-Path’] : request.url_path”. which doesn’t work either and says “cannot create expression”

My Envoy Filter configuration:

  • applyTo: HTTP_FILTER
    match:
    context: SIDECAR_INBOUND
    listener:
    filterChain:
    filter:
    name: envoy.filters.network.http_connection_manager
    subFilter:
    name: envoy.filters.http.router
    proxy:
    proxyVersion: ^1.15.*
    patch:
    operation: INSERT_BEFORE
    value:
    name: istio.stats
    typed_config:
    ‘@type’: type.googleapis.com/udpa.type.v1.TypedStruct
    type_url: type.googleapis.com/envoy.extensions.filters.http.wasm.v3.Wasm
    value:
    config:
    configuration:
    ‘@type’: type.googleapis.com/google.protobuf.StringValue
    value: |
    {
    “debug”: “false”,
    “stat_prefix”: “istio”,
    “disable_host_header_fallback”: true,
    “metrics”: [
    {
    “dimensions”: {
    “destination_cluster”: “node.metadata[‘CLUSTER_ID’]”,
    “source_cluster”: “downstream_peer.cluster_id”
    }
    },
    {
    “name”: “requests_total”,
    “dimensions”: {
    “request_url_path”: “has (response.headers[‘X-Prometheus-Url-Path’]) ? response.headers[‘X-Prometheus-Url-Path’] : request.url_path”
    }
    }
    ]
    }
    root_id: stats_inbound
    vm_config:
    code:
    local:
    inline_string: envoy.wasm.stats
    runtime: envoy.wasm.runtime.null
    vm_id: stats_inbound
1 Like