Istio 1.6 kube-apiserver cannot calling webhook TLS handshake timeout

Hi everyone, I want to ask something

Spec:

  • Kubernetes 1.15.9
  • Istio 1.6.0 multicluster replicated control-plane

I have problem with Istio service-mesh. Auto injection seems not working (already label the namespace).

when checking replicaset in particular namespace

Warning FailedCreate 5m20s (x56 over 10h) replicaset-controller Error creating: Internal error occurred: failed calling webhook “sidecar-injector.istio.io”: Post https://istiod.istio-system.svc:443/inject?timeout=30s: net/http: TLS handshake timeout

when checking kube-apiserver logs

I0603 20:35:34.291242 1 trace.go:81] Trace[2115523713]: “Create /apis/networking.istio.io/v1alpha3/namespaces/istio-system/gateways” (started: 2020-06-03 20:35:24.284742163 +0000 UTC m=+572813.401043450) (total time: 10.006451608s):
W0603 20:35:45.298539 1 dispatcher.go:105] Failed calling webhook, failing open validation.istio.io: failed calling webhook “validation.istio.io”: Post https://istiod.isti-system.svc:443/validate?timeout=30s: net/http: TLS handshake timeout
E0603 20:35:45.298608 1 dispatcher.go:106] failed calling webhook “validation.istio.io”: Post https://istiod.istio-system.svc:443/validate?timeout=30s: net/http: TLS handshake timeout

Log from istiod pod

2020-06-04T07:36:13.982455Z info validationController Reconcile(enter): retry dry-run creation of invalid config
2020-06-04T07:36:23.987667Z info http: TLS handshake error from 10.XX.XX.XX:60024: EOF
2020-06-04T07:36:23.991184Z info validationController Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-06-04T07:36:23.992004Z info validationController validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=10645333) is up-to-date. No change required.

note that 10.XX.XX.XX is node IP.

related information:

  • my kube-apiserver pod use dnsPolicy:ClusterFirst
  • when exec directly into kube-apiserver pod I cant resolve istiod.istio-system.svc but I can curl directly to istiod pod IP.

what is the problem here? should I

Fix this issue. This issue is because TLS communication between API server & Istiod not working. In my case different MTU.