Calling webhook "sidecar-injector.istio.io" failed. Post failed with i/o timeout

1

Hi,

I m new to istio, trying to learn how to deploy applications with istio-sidecar. Tried the following steps to install the istio and the sample application(bookinfo) over it

Istio is installed succesfully.

[root@k8s-master istio-1.4.5]# kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
grafana-6b65874977-hkcbs 1/1 Running 1 33h
istio-citadel-7d4689c4cf-wktnr 1/1 Running 1 33h
istio-egressgateway-679b746848-j2gsz 1/1 Running 1 33h
istio-galley-6b8dfcc549-7zncl 1/1 Running 1 33h
istio-ingressgateway-db547d98-9p5w8 1/1 Running 1 33h
istio-pilot-85d8f75c4-qltpb 1/1 Running 1 33h
istio-policy-6845468548-vtccn 1/1 Running 2 33h
istio-sidecar-injector-6fdc95467f-2qqst 1/1 Running 1 33h
istio-telemetry-5b994fddc6-nntrl 1/1 Running 5 33h
istio-tracing-c66d67cd9-kwhqd 1/1 Running 1 33h
kiali-8559969566-7wqvr 1/1 Running 1 33h
prometheus-66c5887c86-r2nz7 1/1 Running 1 33h

I tried the following commands to install bookinfo application.

kubectl label namespace default istio-injection=enabled
kubectl apply -f samples/bookinfo/platform/kube/bookinfo.yaml

But the pods are not coming up. Error can be seen below.

kubectl describe rs productpage-v1-596598f447

Warning FailedCreate 9m49s replicaset-controller Error creating: Internal error occurred: failed calling webhook “sidecar-injector.istio.io”: Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: context deadline exceeded (Client.Timeout exceeded while awaiting headers)
Warning FailedCreate 3m34s (x6 over 10m) replicaset-controller Error creating: Internal error occurred: failed calling webhook “sidecar-injector.istio.io”: Post https://istio-sidecar-injector.istio-system.svc:443/inject?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)

Kindly let me know how to resolve this.
Following are the versions of kubernetes and istio
[root@k8s-master istio-1.4.5]# kubectl version
Client Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.2”, GitCommit:“59603c6e503c87169aea6106f57b9f242f64df89”, GitTreeState:“clean”, BuildDate:“2020-01-18T23:30:10Z”, GoVersion:“go1.13.5”, Compiler:“gc”, Platform:“linux/amd64”}
Server Version: version.Info{Major:“1”, Minor:“17”, GitVersion:“v1.17.3”, GitCommit:“06ad960bfd03b39c8310aaf92d1e7c12ce618213”, GitTreeState:“clean”, BuildDate:“2020-02-11T18:07:13Z”, GoVersion:“go1.13.6”, Compiler:“gc”, Platform:“linux/amd64”}
[root@k8s-master istio-1.4.5]# istioctl version
client version: 1.4.5
control plane version: 1.4.5
data plane version: 1.4.5 (2 proxies)
[root@k8s-master istio-1.4.5]#

Regards
Sriram

2

Can you please update the kube-apiserver noproxy variable with adding .svc in it

3

Hi,
I am facing the same issue in istio 1.5.2. Can you tell me how to update this flag in kube apiserver?

4

Some question:

  1. are you behind proxy? (if you behind proxy, try setting noproxy)
  2. is the MTU of the cluster correct?

I have a problem related to this before, and the MTU of some master node is wrong.

5

Thanks for the reply.

  1. The cluster is not behind the proxy
  2. the mtu value is 1450. I am using flannel as CNI.
1 Like
6

Usually it is a firewall rule to add between the server and node. You need to make sure your K8s server is able to access the istio pilot pod.

The error is cannot access https://istio-sidecar-injector.istio-system.svc:443 but it is wrong. On earlier version (up to 1.5 I think) this error was linked to the logical url to call and did not reflect the real endpoint the server called. On 1.6 I think I saw an error with the correct url (pod name + port) (the istio injector svc doesn’t exist anymore so url should reflect the correct one directly).

Anyway, you need to check which port your pilot pod listen to (it should be 9443 or 15017). Just check the service “istio-sidecar-injector” in the “istio-system” namespace, you the port to open is the target port linked to the 443 service port.

7

@Gregoire There is no firewall running in the master or worker nodes. All the webhook requests are getting timed-out. Please find the error:

failed calling webhook “validation.istio.io”: Post https://istiod.istio-system.svc:443/validate?timeout=30s: context deadline exceeded

Environment Details:
OS: Centos 7 (VM)
Kubernetes : 1.18.1
Istio : 1.5.2
CNI : Flannel

1 Like
8

Did any one made istio work in an on-premise mode using Centos 7? Could this be a certificate issue? If so, does Istio allow webhook with http instead of https?

1 Like
9

I guess there is some answer there: https://github.com/istio/istio/issues/21058

  • apparently flannel can have some issue, so people add a network rule manually on their master.
10

@Gregoire for now able to fix this issue by using backend as host-gw instead of vxlan in flannel cni. But not sure why vxlan didn’t work.

11

Hi, I am facing same issue in my lab setup on my laptop.

Environment:

Istio version installed 1.7. Pods are up and working

vagrant@master-1:~$ kubectl get pods -n istio-system

NAME READY STATUS RESTARTS AGE
grafana-75b5cddb4d-5t5lq 1/1 Running 1 16h
istio-egressgateway-695f5944d8-s7mbg 1/1 Running 1 16h
istio-ingressgateway-5c697d4cd7-vpd68 1/1 Running 1 16h
istiod-76fdcdd945-tscgc 1/1 Running 0 17m
kiali-6c49c7d566-8wbnw 1/1 Running 1 16h
prometheus-9d5676d95-zxbnk 2/2 Running 2 14h

Cluster information:-

  • Cluster is deployed by hard way

1 LB in front of master, 1 Master, 2 worker nodes in setup deployed on VMbox Ubuntu VMs . I am using weave as the CNI for my cluster.

Error from Kube API server

Aug 31 02:48:22 master-1 kube-apiserver[1750]: I0831 02:48:22.521377 1750 trace.go:116] Trace[51800791]: “Call mutating webhook” configuration:istio-sidecar-injector,webhook:sidecar-injector.istio.io,resource:/v1, Resource=pods,subresource:,operation:CREATE,UID:9b96e1b2-3bbe-41d6-a727-0e19cdd9fbd1 (started: 2020-08-31 02:47:52.521061627 +0000 UTC m=+1080.518695497) (total time: 30.000277923s):
Aug 31 02:48:22 master-1 kube-apiserver[1750]: Trace[51800791]: [30.000277923s] [30.000277923s] END
Aug 31 02:48:22 master-1 kube-apiserver[1750]: W0831 02:48:22.521529 1750 dispatcher.go:181] Failed calling webhook, failing closed sidecar-injector.istio.io: failed calling webhook “sidecar-injector.istio.io”: Post https://istiod.istio-system.svc:443/inject?timeout=30s: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
Aug 31 02:48:22 master-1 kube-apiserver[1750]: I0831 02:48:22.521814 1750 trace.go:116] Trace[491776795]: “Create” url:/api/v1/namespaces/dev/pods,user-agent:kubectl/v1.18.0 (linux/amd64) kubernetes/9e99141,client:192.168.5.30 (started: 2020-08-31 02:47:52.510910326 +0000 UTC m=+1080.508544152) (total time: 30.010883231s):
Aug 31 02:48:22 master-1 kube-apiserver[1750]: Trace[491776795]: [30.010883231s] [30.003030474s] END

12

I get same the error with you and I must open fw on ports 443, 15017, 10250 with source is ip range of master and distination is network tag of node pool.
ref here

1 Like