Istio applies mutualTLS check before IP whitelist for gateway

I have the following setup: Google Cloud TCP Network Load Balancer → Istio Gateway (mTLS mutual authentication for a domain) + Authorization Policy for IP whitelisting → Virtual Service pointing to deployment.
My question is - the auth policy does not seem to work, or at least it is applied after the mTLS verification. The moment I switch from MUTUAL to SIMPLE for the TLS mode and access from unatuthorized IP, I get the correct RBAC error.
To sum up, my issue is mutualTLS check is enforced before (or instead?) of IP whitelist. Is there any way to change that.
Thank you