Hi, We are using istio 1.02 on our EKS 1.11 cluster with 4 core cpu worker nodes and 3 replicas of pilot.
We have 288 pods, and 152 services running in our cluster. We keep seeing spikes in CPU usage and we have set a 2 core cpu resource request to pilot.
We keep seeing higher time taken for config propagation which evident from looking at pilot logs…
{“log”:“2019-01-18T16:55:55.822487Z\u0009info\u0009ads\u0009Push finished: 25.171395449s {\n”,“stream”:“stdout”,“time”:“2019-01-18T16:55:55.822687031Z”}
{“log”:“2019-01-18T16:57:41.829520Z\u0009info\u0009ads\u0009Push finished: 17.190185229s {\n”,“stream”:“stdout”,“time”:“2019-01-18T16:57:41.900057111Z”}
{“log”:“2019-01-18T17:00:02.248743Z\u0009info\u0009ads\u0009Push finished: 11.459144162s {\n”,“stream”:“stdout”,“time”:“2019-01-18T17:00:02.248925382Z”}
{“log”:“2019-01-18T17:00:31.420665Z\u0009info\u0009ads\u0009Push finished: 11.725450191s {\n”,“stream”:“stdout”,“time”:“2019-01-18T17:00:31.42133389Z”}
{“log”:“2019-01-18T17:10:36.598676Z\u0009info\u0009ads\u0009Push finished: 11.836136377s {\n”,“stream”:“stdout”,“time”:“2019-01-18T17:10:36.601939693Z”}
{“log”:“2019-01-18T17:11:08.068074Z\u0009info\u0009ads\u0009Push finished: 15.080455919s {\n”,“stream”:“stdout”,“time”:“2019-01-18T17:11:08.070558291Z”}
{“log”:“2019-01-18T17:11:11.500967Z\u0009info\u0009ads\u0009Push finished: 14.902825228s {\n”,“stream”:“stdout”,“time”:“2019-01-18T17:11:11.501156841Z”}
{“log”:“2019-01-18T17:11:39.615958Z\u0009info\u0009ads\u0009Push finished: 30.25543483s {\n”,“stream”:“stdout”,“time”:“2019-01-18T17:11:39.61968538Z”}
{“log”:“2019-01-18T17:11:44.672438Z\u0009info\u0009ads\u0009Push finished: 34.576678818s {\n”,“stream”:“stdout”,“time”:“2019-01-18T17:11:44.67578737Z”}
{“log”:“2019-01-18T17:11:46.238052Z\u0009info\u0009ads\u0009Push finished: 35.55138106s {\n”,“stream”:“stdout”,“time”:“2019-01-18T17:11:46.238168287Z”}
When i looked at the blog How to scale Pilot, and how Pilot scales? for pilot scaling and compare it with our clusters, we are running pretty low amount of pods/services compared to wht is mentioned in the blog.
Does anyone know of any performance improvement we can make till Istio releases 1.1 version (hoping that version has performance improvements for Pilot)? Any guidance on pilot performance tuning in general would help.
Thanks in advance.