IP Access restriction using Istio not working

Security
1

My Setup : Istio Ingress Gateway is an Azure Internal Load Balancer and we have an Azure Application Gateway sitting on top of the Azure Internal Load Balancer (Istio Ingress Gateway) . The application Gateway receives traffic from external world and sends it to the Istio Ingress Gateway (Internal Load Balancer / Internal IP).

I am trying to implement IP-based access restriction using Istio where I would like Istio to block all requests apart from Certain IPs. I got to a point where It’s blocking the requests but it’s not allowing the IPs I add in the YAML Manifest. I suspect the IP that it is seeing is different that the original Client IP as I have a couple of Load Balancers in front of my Istio.

I read from some documents that X-Forwarder-For is what we should be using to get the actual IP and I have used this in some other implementations but unclear how to get Istio to pick the actual X-Forwarder-Host. Can we create some custom headers and make Istio read from there? Please help

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ingress-policy
  namespace: istio-system
spec:
  selector:
    matchLabels:
      app: istio-ingressgateway
  action: ALLOW
  rules:
  - from:
    - source:
        remoteIpBlocks: ["98.XXX.XXX.66"]

I enabled Debug Logs in my Envoy and below is what is getting recorded.

[2023-05-13T03:22:05.387Z] "GET /cluster/XXXX/XXXX/hari5 HTTP/1.1" 403 - rbac_access_denied_matched_policy[none] - "-" 0 19 0 - "98.XXX.XXX.66:52063,10.240.208.62" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.0.0.0 Safari/537.36" "37ccfaa6-7deb-414f-9f5c-72926c997994" "XXX-XX-XXX.XXX.com" "-" outbound|80||XXX.XXX-XXX.svc.cluster.local - 10.240.208.23:8080 10.240.208.62:40755 - -