Envoy filter not working on specific gateway

Hi there

I’m using istio 1.10.3

I deployed kubeflow with its default gateway, protected by ext_auth filter:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: authn-filter
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_FILTER
      match:
        context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: ""
      patch:
        # For some reason, INSERT_FIRST doesn't work
        operation: INSERT_BEFORE
        value:
          # See: https://www.envoyproxy.io/docs/envoy/v1.17.0/configuration/http/http_filters/ext_authz_filter#config-http-filters-ext-authz
          name: "envoy.filters.http.ext_authz"
          typed_config:
            '@type': type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
            http_service:
              server_uri:
                uri: http://$(AUTHSERVICE_SERVICE).$(AUTHSERVICE_NAMESPACE).svc.cluster.local
                cluster: outbound|8080||$(AUTHSERVICE_SERVICE).$(AUTHSERVICE_NAMESPACE).svc.cluster.local
                timeout: 10s
              authorization_request:
                allowed_headers:
                  patterns:
                    # XXX: MUST be lowercase!
                    - exact: "authorization"
                    - exact: "cookie"
                    - exact: "x-auth-token"
              authorization_response:
                allowed_upstream_headers:
                  patterns:
                    - exact: "kubeflow-userid"

But I also have other services in the same cluster, so I want to bypass the ext_auth filter for some VirtualServices

Also, I have multiple Gateways, which caused alot trouble and I want to remove them (only keep the kubeflow-gateway)

When I applied the bypass-auth-filter on other gateways, it works well:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: bypass-grafana-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
    - applyTo: HTTP_ROUTE
      match:
        context: GATEWAY
        routeConfiguration:
          vhost:
            route:
              name: grafana-route
      patch:
        operation: MERGE
        value:
          name: envoy.ext_authz_disabled
          typed_per_filter_config:
            envoy.ext_authz:
              "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
              disabled: true

But when I moved the VirtualServices to point to kubeflow-gateway, the bypass-auth-filter did not work anymore

(I have also configured the gateway hosts)

Anyone knows why?