We are pretty new to running Kubernetes and seem to be having issues with finding information on how many istiod instances we should be running in our cluster and how much cpu/mem requests should be assigned.
Our istiod instances seems to use a high number of cores and ram and if we scale up the number of instances, it causes things to grind to a halt.
We are running 1.7.2 and with about 4600 pods running in our cluster with sidecars running.
Currently istiod has 7cpu and 15gb or ram assigned and around 6 instances running, the HPA is configured to allow up to 30 instances which is sometimes does get to. This seems excessive but the cpu/mem requests were increased as it seemed to be constantly under load and when we increased the instance count it seem to cause the cluster to grind to a halt.
It looks like documentation mentions it should be around 1cpu and 1.5gb of ram but no mention of how many instances should be set.
What is the guidance here to find the sweet spot with assigned reources and number of instances?