External AuthrorizationPolicy for IgressGateway

I want to route all requests from a particular hostname to be routed to the ext-authz service to validate the requests. This ext-authz server will add additional headers if the request is valid. After that I want to route the traffic based on the additional header which is added by ext-authz server. I’ve configured AuthorizationPolicy on Ingress gateway service and also deployed ext-authz on istio-system namespace. The requests are going to ext-authz service but ingress gateway pod is throwing error like envoy rbac shadow denied, matched policy istio-ext-authz-ns[istio-system]-policy[ext-authz]-rule[0]. Can someone help me with this?

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ext-authz
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: CUSTOM
  provider:
    # The provider name must match the extension provider defined in the mesh config.
    # You can also replace this with sample-ext-authz-http to test the other external authorizer definition.
    name: istio-ext-authz-grpc
  rules:
  # The rules specify when to trigger the external authorizer.
  - to:
    - operation:
        hosts: ["foo.com"]
        paths: ["/*"]