I’m running a relatively-small kops cluster on AWS (3 masters, 6 nodes) and I’m wondering: is it a good idea to run the Istio control plane components (pilot, mixer, galley, etc.) on the master nodes?
Our masters consistently have a lot of CPU + RAM free while our nodes are better-utilized. I was thinking it might make sense to free up some resources on the nodes by moving the Istio control plane components to the masters.
It seems like this should be possible using defaultNodeSelector and defaultTolerations in Istio’s values.yaml when deploying. Searching around, I can’t find any examples of other people doing this, which leads me to ask: is there some obvious downside to this setup that I’m overlooking?