Need Help about pilot proxy status issue

1

Istio: 1.0.5
K8s: 1.11.5

It looks that the proxy status is fine shown as below, but when we run one of them using istioctl ps cloud-ali-v1-57cd9dc45c-rrhbq.default, it shows that there are differences between Pilot clusters/listerns and Envoy clusters/listeners. See details below.

Anyone know this issue? Thansk a lot.

istioctl ps
Stderr when execute [/usr/local/bin/pilot-discovery request GET /debug/syncz ]: gc 1 0.011s 7%: 0.023+0.71+0.78 ms clock, 0.095+0.21/0.60/0.87+3.1 ms cpu, 4->4->1 MB, 5 MB goal, 4 P
gc 2 0.018s 10%: 0.005+0.79+0.90 ms clock, 0.023+0.24/0.71/1.0+3.6 ms cpu, 4->4->2 MB, 5 MB goal, 4 P

PROXY CDS LDS EDS RDS PILOT VERSION
cloud-ali-v1-57cd9dc45c-5cjj4.default SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-8b4bd5947-kk5np 1.0.2
cloud-ali-v1-57cd9dc45c-8q5r5.default SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-8b4bd5947-kk5np 1.0.2
cloud-ali-v1-57cd9dc45c-9hs8b.default SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-8b4bd5947-kk5np 1.0.2
cloud-ali-v1-57cd9dc45c-9zczm.default SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-8b4bd5947-kk5np 1.0.2
cloud-ali-v1-57cd9dc45c-g28sz.default SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-8b4bd5947-kk5np 1.0.2
cloud-ali-v1-57cd9dc45c-jrn2p.default SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-8b4bd5947-kk5np 1.0.2
cloud-ali-v1-57cd9dc45c-lgdzh.default SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-8b4bd5947-kk5np 1.0.2
cloud-ali-v1-57cd9dc45c-nt4zm.default SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-8b4bd5947-kk5np 1.0.2
cloud-ali-v1-57cd9dc45c-rrhbq.default SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-8b4bd5947-kk5np 1.0.2
cloud-ali-v1-57cd9dc45c-tcjrb.default SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-8b4bd5947-kk5np 1.0.2
istio-ingressgateway-84b4868f6b-6lwvh.istio-system SYNCED SYNCED SYNCED (100%) SYNCED istio-pilot-8b4bd5947-kk5np 1.0.2

istioctl ps cloud-ali-v1-57cd9dc45c-rrhbq.default
Stderr when execute [/usr/local/bin/pilot-discovery request GET /debug/config_dump?proxyID=cloud-ali-v1-57cd9dc45c-rrhbq.default ]: gc 1 0.011s 7%: 0.021+0.82+0.80 ms clock, 0.084+0.24/0.68/0.74+3.2 ms cpu, 4->4->1 MB, 5 MB goal, 4 P
gc 2 0.019s 10%: 0.005+0.78+1.0 ms clock, 0.022+0.25/0.70/1.0+4.1 ms cpu, 4->4->2 MB, 5 MB goal, 4 P

Please check the details form the link below. thanks.
istioctl-ps-result

2

That seems like a lot, could you run istioctl version --remote to check that your pilot/istioctl versions are in sync?

Could you put this in pastebin and link as well, it’s difficult to read in that format :slightly_smiling_face:.

3

Hello. This sounds similar to https://github.com/istio/istio/issues/10293 in which the Routes are showing differences with requestHeadersToAdd. I also agree with Liam that the output is difficult to read.

4

thanks @liamawhite the versions of istioctl and pilot are same, 1.0.5.
From the result of “istioctl ps”, it shows that all are in sync, but it is different for one pod named “istioctl ps cloud-ali-v1-57cd9dc45c-rrhbq.default”. Very intresting that these results are not in sync.

5

Change to use pastebin to make the output clear, thanks…please get from istio-ps-diff-result - Pastebin.com

From the output, it showns that there are differences between clusters and listeners, but routes are in sync. very intresting. Thanks

6

I think this is a different issue. That one is to do with ordering and is an issue with the ps diff command itself, this is not from what I can see. Are you able to communicate properly with cloud-ali.default.svc.cluster.local?

Could you raise an issue in istio/istio including all the info, link back to this, and give some more information on the virtual services, gateways, etc associated with this service.