Patching ExtAuthzPerRoute to sidecar's virtual host

I am using ExtAuthzPerRoute to send metadata to the ext_authz server.

The static config when using Envoy looks like this:

    filter_chains:
    - filters:
      - name: envoy.filters.network.http_connection_manager
        typed_config:  
          "@type": type.googleapis.com/envoy.extensions.filters.network.http_connection_manager.v3.HttpConnectionManager
          stat_prefix: ingress_http
          route_config:
            virtual_hosts:
            - name: default
              domains: "*"
              typed_per_filter_config:
                envoy.filters.http.ext_authz:
                  "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
                  check_settings:
                    context_extensions:
                      key: value
              routes:
                 ...

Now I am trying to configure the same thing on an Istio sidecar with the following EnvoyFilter Patch:

...
- applyTo: VIRTUAL_HOST
    match:
      context: SIDECAR_INBOUND
      routeConfiguration:
        vhost:
          name: inbound|http|80
    patch:
      operation: MERGE
      value:
        per_filter_config:
          "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthzPerRoute
          check_settings:
            context_extensions:
              key: value
...

but have had no success.

Any one has any insight on this? Thanks.