This probably is linked to the number of services you have exposed on the mesh. I learned last week, on one of solo.io’s workshop, that using the Sidecar object to restrict the services visibility can help reduce the memory usage.
Try look at that and see if it applies to your scenario.
I hope this helps.