Authorization policy for restriction based on Client IP not working

1

Deployed Istio 1.10 on AKS cluster.
We are using Azure Application Gateway as the frontend and Istio gateway as the backend.
Traffic from the internet will be routed like this :

Traffic >> Azure Application Gateway >> Istio gateway >> Microservice

We have some microservices which we want to be accessible from VPN.
Therefore we are using Authorization policy which will check the Client IP and restrict the access to microservice based on Client IP.

Below Authorization policy will restrict the access to http://hostName/httpbin if not accessed from given IP.

Authorization policy :

apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: ip-restriction-policy
  namespace: istio-system
spec:
  selector:
    matchLabels:
      istio: ingressgateway
  action: DENY
  rules:
    - from:
      - source:
          notIpBlocks: [ "24.198.223.80" ]
      to:
        - operation:
            paths: [ "/httpbin/*" ]

But this policy is not getting applied as we can see from the istio gateway logs, two ips are sent as Client IP, i.e, 24.198.223.80,172.48.28.4

One more IP of proxy server is getting appended to Client IP.
Does anyone handle such a scenario?