Istio proxy used up all ports for websocket connections

Hi all, I am using istio proxy for one of my services that handles WebSocket connections from clients, and it seems to be the case that for each connection, the envoy will occupy one port for some reason, and the maximum number of connection is bound to the number of available ports. So I was wondering if anyone else has encountered the same problem, and how I am able to remove such limitation?