Istio External Authorizer - AuthenticationPolicy expected response

I am trying to setup Istio’s External Authorizer so I can handle user sessions. The problem I am running into is that I am always getting RBAC access denied, no matter the status code I return. I am using FastAPI as the authorizer, and all it is currently doing is returning a couple headers and setting the status code as 200

Mesh config:

    extensionProviders:
    - name: "sample-ext-authz-http"
      envoyExtAuthzHttp:
        service: "demo-api.demo.svc.cluster.local"
        port: "8080"
        includeRequestHeadersInCheck: ["x-ext-authz", "cookie"]
        headersToUpstreamOnAllow: ["testing"]
        headersToDownstreamOnAllow: ["set-cookie"]

RequestAuthentication and AuthorizationPolicy:

apiVersion: security.istio.io/v1beta1
kind: RequestAuthentication
metadata:
  name: demo
  namespace: wick
spec:
  jwtRules:
    - issuer: "johnwick.com
      jwksUri: "https://johnwick.com/.well-known/jwks.json"
      fromHeaders:
        - name: "Authorization"
          prefix: "Bearer "
      fromParams:
        - "token"
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: demo-inner-comms
  namespace: wick
spec:
  action: ALLOW
  rules:
    - from:
        - source:
            namespaces: [ "wick" ]
---
apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ext-authz
  namespace: wick
spec:
  action: CUSTOM
  provider:
definition.
    name: sample-ext-authz-http
  rules:
    - to:
        - operation:
            hosts: [ "*.johnwick.com", "*.johnwick.com:*" ]

FastAPI code:

from fastapi import FastAPI, Request, status
from fastapi.responses import Response
from starlette.middleware.sessions import SessionMiddleware

app = FastAPI()
app.add_middleware(SessionMiddleware, secret_key='DEMODONTUSE')


@app.get('/{path:path}')
async def root(path: str, request: Request):
    headers = {
        'testing': 'testing',
        'x-ext-authz': 'allow'
    }

    response = Response(status_code=200, headers=headers)

    return response

Here’s what the istio-proxy logs look like on the demo.com service:

2022-09-13T16:06:55.842891Z     debug   envoy rbac      checking request: requestedServerName: , sourceIP: REPLACED IP:50251, directRemoteIP: REPLACED IP:50251, remoteIP: REPLACED IP:0,localAddress: 10.225.2.28:9090, ssl: none, headers: ':authority', 'john-wick-webui.dev.johnwick.com'
':path', '/'
':method', 'GET'
':scheme', 'https'
'user-agent', 'curl/7.79.1'
'accept', '*/*'
'authorization', 'Bearer REPLACED JWT TOKEN'
'x-cloud-trace-context', '2869289499fc9dea7dcb46e55f3fdaed/13733345605759396675'
'via', '1.1 google'
'x-forwarded-for', 'REPLACED IP, REPLACED IP'
'x-forwarded-proto', 'https'
'x-request-id', '22ff1432-b368-4e88-925c-28fd0105222d'
, dynamicMetadata:
2022-09-13T16:06:55.842904Z     debug   envoy rbac      shadow denied, matched policy istio-ext-authz-ns[wick]-policy[john-wick-webui-tools-testing]-rule[0]
2022-09-13T16:06:55.842915Z     debug   envoy rbac      no engine, allowed by default
2022-09-13T16:06:55.842978Z     debug   envoy router    [C0][S3635838946293304567] cluster 'outbound|8080||demo-api.wick.svc.cluster.local' match for URL '/'
2022-09-13T16:06:55.843016Z     debug   envoy router    [C0][S3635838946293304567] router decoding headers:
':method', 'GET'
':path', '/'
':authority', 'john-wick-webui.dev.johnwick.com'
':scheme', 'https'
'content-length', '0'
'authorization', 'Bearer REPLACED JWT TOKEN'
'accept', '*/*'
'x-forwarded-proto', 'https'
'x-forwarded-for', 'REPLACED IP, REPLACED IP,10.225.2.28'
'user-agent', 'curl/7.79.1'
'x-b3-traceid', '22073f983cfe9931fea2388a5b952dc4'
'x-b3-spanid', '1b9f65422a502ff3'
'x-b3-parentspanid', 'fea2388a5b952dc4'
'x-b3-sampled', '0'
'x-envoy-internal', 'true'
'x-envoy-expected-rq-timeout-ms', '600000'

2022-09-13T16:06:55.843034Z     debug   envoy pool      [C107466] using existing connection
2022-09-13T16:06:55.843039Z     debug   envoy pool      [C107466] creating stream
2022-09-13T16:06:55.843049Z     debug   envoy router    [C0][S3635838946293304567] pool ready
2022-09-13T16:06:55.845948Z     debug   envoy router    [C0][S3635838946293304567] upstream headers complete: end_stream=false
2022-09-13T16:06:55.846027Z     debug   envoy http      async http request response headers (end_stream=false):
':status', '200'
'date', 'Tue, 13 Sep 2022 16:06:53 GMT'
'server', 'uvicorn'
'testing', 'testing'
'x-ext-authz', 'allow'
'set-cookie', 'Authorization="Bearer REPLACED JWT TOKEN"; Path=/; SameSite=lax'
'transfer-encoding', 'chunked'
'x-envoy-upstream-service-time', '2'

2022-09-13T16:06:55.846038Z     debug   envoy client    [C107466] response complete
2022-09-13T16:06:55.846119Z     debug   envoy jwt       Called Filter : decodeHeaders
2022-09-13T16:06:55.846134Z     debug   envoy jwt       Prefix requirement '/' matched.
2022-09-13T16:06:55.846149Z     debug   envoy jwt       extract authorizationBearer
2022-09-13T16:06:55.846166Z     debug   envoy jwt       origins-0: JWT authentication starts (allow_failed=false), tokens size=1
2022-09-13T16:06:55.846171Z     debug   envoy jwt       origins-0: startVerify: tokens size 1
2022-09-13T16:06:55.846178Z     debug   envoy jwt       origins-0: Parse Jwt REPLACED JWT TOKEN
2022-09-13T16:06:55.846378Z     debug   envoy jwt       origins-0: Verifying JWT token of issuer johnwick.com
2022-09-13T16:06:55.846450Z     debug   envoy jwt       origins-0: JWT token verification completed with: OK
2022-09-13T16:06:55.846463Z     debug   envoy jwt       Jwt authentication completed with: OK
2022-09-13T16:06:55.846495Z     debug   envoy filter    AuthenticationFilter::decodeHeaders with config
policy {
  origins {
    jwt {
      issuer: "johnwick.com"
    }
  }
  origin_is_optional: true
  principal_binding: USE_ORIGIN
}
skip_validate_trust_domain: true
disable_clear_route_cache: true

2022-09-13T16:06:55.846501Z     debug   envoy filter    No method defined. Skip source authentication.
2022-09-13T16:06:55.846509Z     debug   envoy filter    Validating request path / for jwt issuer: "johnwick.com"

2022-09-13T16:06:55.846615Z     debug   envoy filter    ProcessJwtPayload: json object is {"admin":true,"exp":1663128000,"iss":"johnwick.com","services":["wick.service.ui"],"upn":"jwick@johnwick.com","user":{"first_name":"John","id":"REPLACED ID","last_name":"Wick","username":"jwick@johnwick.com"}}
2022-09-13T16:06:55.846643Z     debug   envoy filter    JWT validation succeeded
2022-09-13T16:06:55.846654Z     debug   envoy filter    Set principal from origin:
2022-09-13T16:06:55.846657Z     debug   envoy filter    Origin authenticator succeeded
2022-09-13T16:06:55.846771Z     debug   envoy filter    Saved Dynamic Metadata:
fields {
  key: "request.auth.claims"
  value {
    struct_value {
      fields {
        key: "iss"
        value {
          list_value {
            values {
              string_value: "johnwick.com"
            }
          }
        }
      }
      fields {
        key: "services"
        value {
          list_value {
            values {
              string_value: "wick.service.ui"
            }
          }
        }
      }
      fields {
        key: "upn"
        value {
          list_value {
            values {
              string_value: "jwick@johnwick.com"
            }
          }
        }
      }
      fields {
        key: "user"
        value {
          struct_value {
            fields {
              key: "first_name"
              value {
                list_value {
                  values {
                    string_value: "John"
                  }
                }
              }
            }
            fields {
              key: "id"
              value {
                list_value {
                  values {
                    string_value: "REPLACED ID"
                  }
                }
              }
            }
            fields {
              key: "last_name"
              value {
                list_value {
                  values {
                    string_value: "Wick"
                  }
                }
              }
            }
            fields {
              key: "username"
              value {
                list_value {
                  values {
                    string_value: "jwick@johnwick.com"
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
fields {
  key: "request.auth.raw_claims"
  value {
    string_value: "{\"admin\":true,\"upn\":\"jwick@johnwick.com\",\"exp\":1663128000,\"services\":[\"wick.service.ui\"],\"user\":{\"username\":\"jwick@johnwick.com\",\"id\":\"REPLACED ID\",\"first_name\":\"John\",\"last_name\":\"Wick\"},\"iss\":\"johnwick.com\"}"
  }
}

2022-09-13T16:06:55.846946Z     debug   envoy rbac      checking request: requestedServerName: , sourceIP: REPLACED IP:50251, directRemoteIP: REPLACED IP:50251, remoteIP: REPLACED IP:0,localAddress: 10.225.2.28:9090, ssl: none, headers: ':authority', 'john-wick-webui.dev.johnwick.com'
':path', '/'
':method', 'GET'
':scheme', 'https'
'user-agent', 'curl/7.79.1'
'accept', '*/*'
'x-cloud-trace-context', '2869289499fc9dea7dcb46e55f3fdaed/13733345605759396675'
'via', '1.1 google'
'x-forwarded-for', 'REPLACED IP, REPLACED IP'
'x-forwarded-proto', 'https'
'x-request-id', '22ff1432-b368-4e88-925c-28fd0105222d'
, dynamicMetadata: filter_metadata {
  key: "envoy.filters.http.jwt_authn"
  value {
    fields {
      key: "johnwick.com"
      value {
        struct_value {
          fields {
            key: "admin"
            value {
              bool_value: true
            }
          }
          fields {
            key: "exp"
            value {
              number_value: 1663128000
            }
          }
          fields {
            key: "iss"
            value {
              string_value: "johnwick.com"
            }
          }
          fields {
            key: "services"
            value {
              list_value {
                values {
                  string_value: "wick.service.ui"
                }
              }
            }
          }
          fields {
            key: "upn"
            value {
              string_value: "jwick@johnwick.com"
            }
          }
          fields {
            key: "user"
            value {
              struct_value {
                fields {
                  key: "first_name"
                  value {
                    string_value: "John"
                  }
                }
                fields {
                  key: "id"
                  value {
                    string_value: "REPLACED ID"
                  }
                }
                fields {
                  key: "last_name"
                  value {
                    string_value: "Wick"
                  }
                }
                fields {
                  key: "username"
                  value {
                    string_value: "jwick@johnwick.com"
                  }
                }
              }
            }
          }
        }
      }
    }
  }
}
filter_metadata {
  key: "envoy.filters.http.rbac"
  value {
    fields {
      key: "istio_ext_authz_shadow_effective_policy_id"
      value {
        string_value: "istio-ext-authz-ns[wick]-policy[john-wick-webui-tools-testing]-rule[0]"
      }
    }
    fields {
      key: "istio_ext_authz_shadow_engine_result"
      value {
        string_value: "denied"
      }
    }
  }
}
filter_metadata {
  key: "istio_authn"
  value {
    fields {
      key: "request.auth.claims"
      value {
        struct_value {
          fields {
            key: "iss"
            value {
              list_value {
                values {
                  string_value: "johnwick.com"
                }
              }
            }
          }
          fields {
            key: "services"
            value {
              list_value {
                values {
                  string_value: "wick.service.ui"
                }
              }
            }
          }
          fields {
            key: "upn"
            value {
              list_value {
                values {
                  string_value: "jwick@johnwick.com"
                }
              }
            }
          }
          fields {
            key: "user"
            value {
              struct_value {
                fields {
                  key: "first_name"
                  value {
                    list_value {
                      values {
                        string_value: "John"
                      }
                    }
                  }
                }
                fields {
                  key: "id"
                  value {
                    list_value {
                      values {
                        string_value: "REPLACED ID"
                      }
                    }
                  }
                }
                fields {
                  key: "last_name"
                  value {
                    list_value {
                      values {
                        string_value: "Wick"
                      }
                    }
                  }
                }
                fields {
                  key: "username"
                  value {
                    list_value {
                      values {
                        string_value: "jwick@johnwick.com"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
    fields {
      key: "request.auth.raw_claims"
      value {
        string_value: "{\"admin\":true,\"upn\":\"jwick@johnwick.com\",\"exp\":1663128000,\"services\":[\"wick.service.ui\"],\"user\":{\"username\":\"jwick@johnwick.com\",\"id\":\"REPLACED ID\",\"first_name\":\"John\",\"last_name\":\"Wick\"},\"iss\":\"johnwick.com\"}"
      }
    }
  }
}

2022-09-13T16:06:55.846974Z     debug   envoy rbac      enforced denied, matched policy none
2022-09-13T16:06:55.846990Z     debug   envoy http      [C107474][S9741496262757324086] Sending local reply with details rbac_access_denied_matched_policy[none]

Without using the external authorizer and removing the CUSTOM action everything works as expected. Without JWT token or with a bad JWT I get RBAC access denied, with a valid JWT token I am able to get access as expected