we are facing some troubles with istio discovery as it consumes almost all CPU power. According to documentation one pilot can serve thousands of pods. We have roughly 300 and pilot is not able to handle it.
From the log I do not see any issue as every line is INFO level. I tried to switch to warning level, but nothing pop up. Well, nothing above INFO level, but it seems that pilot is constantly pushing configurations even if nothing new is going on in cluster. (I’m not sure if this is the correct behaviour or not.) Also there no pods that are in restart loop.
Istio validations are ok, at least istioctl says that. I’ve tried to search this up on google, but it seems like this issue was fixed in 1.1.0 and we are using 1.4.0 respectively 1.4.3 ( I hoped that upgrade will help. )
According previous information I feel that we have something wrong within our configuration. Is here anyone who could help me out, please?
If you need anything from configuration to logs I’ll gladly provide that. Thanks in advance.
Edit: monitoring says that push to envoy takes up to 30s to push configuration into a proxy. Which is very strange and could lead to solution.