A lot of memory with many idle connections on gateway

Hello, we are using istio and have like 18K permanent connections from outside of our cluster, (other part of organization). The connections usually dont do anything however in total the memory used on the istio gateway for that workload is over 12GB of memory. I guess it has something to do with some kind of buffer per connection. Is there a way how can we tune this memory usage down? Thanks in advance!