Trouble Adding Dimensions to Custom Metrics

Policies and Telemetry
1

I’m trying to add dimensions to existing metrics using Istio 1.8.1

I followed the steps in Istio / Customizing Istio Metrics to add destination_port and request_host to the requests_total metric (exactly the scenario in the documentation), but I’m not seeing the new dimensions in prometheus.

I see the IstioOperator has been updated correctly. I’ve redeployed pods within my mesh. Still nothing.

telemetry:
  v2:
    prometheus:
      configOverride:
        inboundSidecar:
          metrics:
            - name: requests_total
              dimensions:
                destination_port: string(destination.port)
                request_host: request.host
            - name: istio_tcp_connections_closed_total
              dimensions:
                destination_ip: string(destination.ip)
        outboundSidecar:
          metrics:
            - name: requests_total
              dimensions:
                destination_port: string(destination.port)
                request_host: request.host
            - name: istio_tcp_connections_closed_total
              dimensions:
                destination_ip: string(destination.ip)
        gateway:
          metrics:
            - name: requests_total
              dimensions:
                destination_port: string(destination.port)
                request_host: request.host
            - name: istio_tcp_connections_closed_total
              dimensions:
                destination_ip: string(destination.ip)
2

Followup:
I’m not seeing any of the EnvoyFilter’s being updated so looks like the IstioOperator isn’t actually doing anything with those new values.

3

I wasn’t able to get the configs in the IstioOperator to work. Instead I updated the stats-filter-1.8 and tcp-stats-filter-1.8 manually with the new metrics/destinations.

I also couldn’t get the extraStatsConfig in the IstioOperator meshConfig to work. I moved those to be defined as pod annotations and that worked for me.