[EnvoyFilter] local rate limit with whitelisted IPs

Networking
1

Hello,

I am trying to apply the local rate limit filter on my workload. I found on internet the filter to use to be able to add rate limit and it works, but I need to remove this filter for a list of whitelisted IP.
So I added an ip tagging filter and tried to use the value of the header set by the first filter in the rate limit one, but it does not seems to work.
I would like to disable the rate limit when we match the whitelisted CIDR (or change the limit to a big one, like I tried in the following file)

What am I missing, I am using istio 1.9.3

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: rate-limit-ip-tag
spec:
  workloadSelector:
    labels:
      app.kubernetes.io/instance: <a value>
      app.kubernetes.io/name: <a value>
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          portNumber: 8080
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
              subFilter:
                name: "envoy.filters.http.router"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.ip_tagging
          typed_config:
            "@type": type.googleapis.com/envoy.extensions.filters.http.ip_tagging.v3.IPTagging
            request_type: BOTH
            ip_tags:
              - ip_tag_name: "whitelisted"
                ip_list:
                  - address_prefix: 192.168.1.1
                    prefix_len: 24
---
apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: rate-limit
spec:
  workloadSelector:
    labels:
      app.kubernetes.io/instance: <a value>
      app.kubernetes.io/name: <a value>
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: SIDECAR_INBOUND
        listener:
          filterChain:
            filter:
              name: "envoy.filters.network.http_connection_manager"
      patch:
        operation: INSERT_BEFORE
        value:
          name: envoy.filters.http.local_ratelimit
          typed_config:
            "@type": type.googleapis.com/udpa.type.v1.TypedStruct
            type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
            value:
              stat_prefix: http_local_rate_limiter
    - applyTo: HTTP_ROUTE
      match:
        context: SIDECAR_INBOUND
        routeConfiguration:
          vhost:
            name: "inbound|http|8080"
            route:
              action: ANY
      patch:
        operation: MERGE
        value:
          route:
            rate_limits:
            - actions:
              - header_value_match:
                  headers:
                  - name: "x-envoy-ip-tags"
                    contains_match: "whitelisted"
                  descriptor_value: "whitelisted"
          typed_per_filter_config:
            envoy.filters.http.local_ratelimit:
              "@type": type.googleapis.com/udpa.type.v1.TypedStruct
              type_url: type.googleapis.com/envoy.extensions.filters.http.local_ratelimit.v3.LocalRateLimit
              value:
                stat_prefix: http_local_rate_limiter
                descriptors:
                  - entries:
                    - key: header_match
                      value: whitelisted
                    token_bucket:
                      max_tokens: 500000
                      tokens_per_fill: 500000
                      fill_interval: 60s
                token_bucket:
                  max_tokens: 5
                  tokens_per_fill: 5
                  fill_interval: 60s
                filter_enabled:
                  runtime_key: local_rate_limit_enabled
                  default_value:
                    numerator: 100
                    denominator: HUNDRED
                filter_enforced:
                  runtime_key: local_rate_limit_enforced
                  default_value:
                    numerator: 100
                    denominator: HUNDRED
                response_headers_to_add:
                  - append: false
                    header:
                      key: x-local-rate-limit
                      value: 'true'

Thank you