Pilot discovery failure

Hi folks,

What would cause this (from pilot’s discovery container)? It’s timing out while trying to resolve the galley hostname.

2020-03-05T19:58:33.226509Z info mcp (re)trying to establish new MCP sink stream
2020-03-05T19:58:33.226607Z error mcp Failed to create a new MCP sink stream: rpc error: code = Unavailable desc = all SubConns are in TransientFailure, latest connection error: connection error: desc = "transport: Error while dialing dial tcp: lookup istio-galley.istio-system.svc on read udp> i/o timeout"
2020-03-05T19:58:33.236845Z info Configuration not synced: first push for [istio/authentication/v1alpha1/policies istio/authentication/v1alpha1/meshpolicies istio/rbac/v1alpha1/serviceroles istio/rbac/v1alpha1/clusterrbacconfigs istio/security/v1beta1/authorizationpolicies istio/networking/v1alpha3/destinationrules istio/networking/v1alpha3/envoyfilters istio/config/v1alpha2/httpapispecbindings istio/mixer/v1/config/client/quotaspecbindings istio/rbac/v1alpha1/rbacconfigs istio/mixer/v1/config/client/quotaspecs istio/rbac/v1alpha1/servicerolebindings istio/networking/v1alpha3/virtualservices istio/networking/v1alpha3/gateways istio/networking/v1alpha3/serviceentries istio/networking/v1alpha3/sidecars istio/config/v1alpha2/httpapispecs] not received

Here are my pods:

NAME                                      READY   STATUS      RESTARTS   AGE
istio-citadel-594657f7bd-2jvb9            1/1     Running     0          3h18m
istio-galley-6cc846cb55-22wbq             1/1     Running     0          110m
istio-ingressgateway-6469996f79-ssrhg     0/1     Running     0          108m
istio-init-crd-10-1.4.2-jt6rg             0/1     Completed   0          3h18m
istio-init-crd-11-1.4.2-8rfms             0/1     Completed   0          3h18m
istio-init-crd-14-1.4.2-mhqvz             0/1     Completed   0          3h18m
istio-pilot-7fd7cf99f-s4bbb               1/2     Running     0          35m
istio-security-post-install-1.4.2-6pvhd   0/1     Completed   0          3h18m
istio-sidecar-injector-7c78fb47bf-sgrqv   1/1     Running     0          3h18m


It might be your DNS pod were not in ready status, or the host runs pilot pod had a bad iptables rule.