IP whitelisting trouble - ingress gateway is always seeing the cluster IP

I’m currently running Istio 1.3.5 on a bare metal cluster and I’m trying to get the IP whitelist example (https://istio.io/docs/tasks/policy-enforcement/denial-and-list/) to work. The ingress gateway service is of type LoadBalancer, IP adresses are provided by MetalLB.
I’ve already set externalTrafficPolicy to Local for the ingress gateway service. But it looks like the gateway is still seeing some cluster IP, not the real client IP address (in the source.ip attribute), since the reply always looks like this:
PERMISSION_DENIED:whitelistip.default: is not whitelisted

I’m currently out of ideas, so any hints will be appreciated.

Were you able to resolve and implement IP white listing

The only working way we found was for HTTP connections using some container containing the IP-Adresse, e.g. X-Forwared-For or X-Real-IP.