VirtualService header routing from AuthorizationPolicy

Networking
1

So I’m trying to set up a custom authz plugin which works with a PKI infrastructure. This plugin injects some headers which I have some VirtualServices that route to different resources based on the injected headers.

$ istioctl version
client version: 1.14.1
control plane version: 1.15.0
data plane version: 1.15.0 (8 proxies)

For the sake of example, lets say my auth plugin talks over grpc with envoyproxy v3 and injects the header X-TEST-HEADER: SUCCESS. I’m using the httpbin example, with the following mesh config.

extensionProviders:
  - name: "ext-authz-plugin"
    envoyExtAuthzGrpc:
      service: "my-plugin-svc.istio-system.svc.cluster.local"
      port: "50051"

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: httpbin
  namespace: default
spec:
  gateways:
  - istio-system/my-gateway
  hosts:
  - '*'
  http:
  - match:
    - headers:
        X-TEST-HEADER:
          regex: (?:.*?\bSUCCESS\b).*
      uri:
        prefix: /headers
    route:
    - destination:
        host: httpbin
        port:
          number: 8000

and the following gateway

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: my-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: istio-ingress-credential
      mode: MUTUAL

I’ve tried creating an AuthorizationPolicy against the ingress gateway itself to attempt to apply a global auth.

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ext-authz
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: CUSTOM
  provider:
    name: ext-authz-plugin
  rules:
  - {}

I’ve also tried applying that policy on a namespace basis, and workload basis for my deployment, but my logs never show an auth request being received. Instead what I see in the ingress gateway logs are

"GET /headers HTTP/2" 404 NR route_not_found - ... "

That makes sense if the plugin isn’t being called, and based on this it appears that routing happens before auth. So then I tried to remove the AuthorizationPolicy, and instead use an EnvoyFilter to inject the authz before the routing filter. This was built based on istio docs and envoy docs.

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: authz-envoyfilter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      listener:
        filterChain:
          filter:
            name: envoy.filters.network.http_connection_manager
            subFilter:
              name: envoy.filters.http.router
    patch:
      operation: INSERT_FIRST
      value:
        name: envoy.filters.http.ext_authz
        typed_config:
          "@type": type.googleapis.com/envoy.extensions.filters.http.ext_authz.v3.ExtAuthz
          grpc_service:
            envoy_grpc:
              cluster_name: ext-authz-plugin-filter
            timeout: 0.5s
          transport_api_version: V3
          include_peer_certificate: true
  - applyTo: CLUSTER
    patch:
      operation: ADD
      value: # cluster specification
        name: ext-authz-plugin-filter
        connect_timeout: 0.25s
        typed_extension_protocol_options:
          envoy.extensions.upstreams.http.v3.HttpProtocolOptions:
            "@type": type.googleapis.com/envoy.extensions.upstreams.http.v3.HttpProtocolOptions
            explicit_http_config:
              http2_protocol_options: {}
        type: LOGICAL_DNS
        lb_policy: ROUND_ROBIN
        load_assignment:
          cluster_name: ext-authz-plugin-filter
          endpoints:
          - lb_endpoints:
            - endpoint:
                address:
                  socket_address:
                    protocol: TCP
                    address: my-plugin-svc.istio-system.svc.cluster.local
                    port_value: 50051

But still, authz fires after routing with the gateway showing the same logs as above (also note that I’ve tried adding match contexts with no luck). The only way I’ve gotten this working is either converting the above EnvoyFilter to a NETWORK_FILTER, or using a nearly identical EnvoyFilter with some Lua that makes an http request to my plugin instead. I’ve also had luck with WASM plugins, but that’s not a desirable choice based on some custom auditing and monitoring.

I’d rather not use the NETWORK_FILTER, because there isn’t as much metadata (ex. x-forwarded-client-cert) and I’ll have to do a lot of refactoring. I’d rather not use the Lua filter, because I’ve run into issues with things like the content-length header being overriden and it’s just more difficult/inconvenient to manage.

I’m hoping it’s just a config issue, is there any way to achieve what I’m wanting with the flow AuthZ header inject → Virtual service match?