Hello,
I have three ingress gateways on version 1.7.4, the are based on the default installation.
They were running fine on GCloud Kubernetes clusters and at some point all the instances started showing this in the logs:
2021-01-21T23:15:19.204158Z warning envoy config gRPC config for type.googleapis.com/envoy.config.listener.v3.Listener rejected: Error adding/updating listener(s) 0.0.0.0_0: cannot bind '0.0.0.0': Read-only file system
|2021-01-21T23:15:20.530655Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
|2021-01-21T23:15:22.529830Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
|2021-01-21T23:15:24.530011Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
|2021-01-21T23:15:26.529798Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
|2021-01-21T23:15:28.529719Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
|2021-01-21T23:15:30.529526Z|warn|Envoy proxy is NOT ready: config not received from Pilot (is Pilot running?): cds updates: 1 successful, 0 rejected; lds updates: 0 successful, 1 rejected|
istioctl proxy-status
EDITED-istio-ingressgateway-86c8d96d65-98lc9.istio-system SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-1-7-4-677bfc785-mx6sr 1.7.4
EDITED-istio-ingressgateway-86c8d96d65-c5v9c.istio-system SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-1-7-4-677bfc785-mx6sr 1.7.4
EDITED-istio-ingressgateway-86c8d96d65-cx2rs.istio-system SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-1-7-4-677bfc785-mx6sr 1.7.4
EDITED-istio-ingressgateway-86c8d96d65-j6s9q.istio-system SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-1-7-4-677bfc785-mx6sr 1.7.4
EDITED-istio-ingressgateway-86c8d96d65-qrrfh.istio-system SYNCED STALE (Never Acknowledged) SYNCED SYNCED istiod-1-7-4-677bfc785-mx6sr 1.7.4
So, as fart as I understand the issue is that:
Pilot is trying to configure Envoy but envoy is not able to do it due to what it seems to be a readonly issue.
I have the exact same configuration on other 2 clusters and they are running fine.
I tried restarting the pods but the issue is still the same and also to just change the image version to 1.7.6 again without success.
Any ideas?
Please let me know if you need more information.