Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded

Networking
1

I created istio-injection=enabled label to specific namespace. But, Replicaset of that namespace occur below error.

Warning  FailedCreate  12m (x20 over 53m)  replicaset-controller  Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded

I think the problem is with the link below.

$ kubectl get --raw /api/v1/namespaces/istio-system/services/https:istiod:https-webhook/proxy/inject -v4

I0803 20:26:06.413013   28163 helpers.go:216] server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "error trying to reach service: dial tcp 10.0.8.144:15017: connect: connection timed out",
  "reason": "ServiceUnavailable",
  "code": 503
}]
Error from server (ServiceUnavailable): error trying to reach service: dial tcp 10.0.8.144:15017: connect: connection timed out

$ curl https://istiod.istio-system.svc:443/inject -k # other container

no body found

$ curl https://localhost:15017/inject -k # self container

no body found

How can i solve this problem? I use EKS.

3 Likes
2

Having the same exact issue. Did you find a solution?

1 Like
3

Also having this issue in GKE. Haven’t seen it before, but now happening in our dev and staging clusters. Both using pre-emptible machines.

The other symptom is that I don’t seem to be able to get logs from any pods in the cluster.

Seems to be fixed by

kubectl rollout restart deployment  -n kube-system

I haven’t yet managed to find out the root cause though.

Istio 1.14.2

4

For those who come here with GKE issues:

Have you done the Firewall edits on GCP? I gave the roll out a shot and had no luck.

I had the error below

      message: Deployment does not have minimum availability.
    - type: ReplicaFailure
      status: 'True'
      lastUpdateTime: '2022-08-26T15:16:04Z'
      lastTransitionTime: '2022-08-26T15:16:04Z'
      reason: FailedCreate
      message: >-
        Internal error occurred: failed calling webhook
        "namespace.sidecar-injector.istio.io": failed to call webhook: Post
        "https://istiod.istio-system.svc:443/inject?timeout=10s": context
        deadline exceeded

And once I updated the firewall rules for the master, it worked!

Link to firewall documentation

I am also running a private pre-emptive cluster on GKE. Version 1.24.3-gke.900

Comments I make are mine and mine only, and do not reflect that of my Employer

1 Like
5

having the exact issue on EKS v1.23 with Istio 1.16.0, any fix/solution?
no luck with rollout restart deployment

server response object: [{
  "metadata": {},
  "status": "Failure",
  "message": "error trying to reach service: dial tcp 10.4.78.203:15017: connect: connection timed out",
  "reason": "ServiceUnavailable",
  "code": 503

}]

6

Try checking if communication between nodes and eks control plane is open on port 15017. In my case i had to open this port on ingress in node security group. Those curls you tested proves that communication between nodes is possible, istio in order to create sidecar needs to be able to communicate with hook and for that network passage to EKS master is needed.

2 Likes
7

Check the PROXY settings for k8s API server(sorry checking in on-premise cluster)
Removed the PROXY env variables api server yaml in /etc/kubernetes/manifest.

K8s automatically set these proxy setting variables if system already has proxy settings.

1 Like
8

same! thanks! took some searching to find this one.

9

Hi,
I see the following error
- lastTransitionTime: ‘2023-03-10T10:15:06Z’
message: >-
Internal error occurred: failed calling webhook
“namespace.sidecar-injector.istio.io”: failed to call webhook: Post
“https://istiod.istio-system.svc:443/inject?timeout=10s”: context

In the kube-apiserver.yaml there is no PROXY setting.

10

Having the same exact issue. Did you find a solution?

11

[Istio / Google Kubernetes Engine](Istio/GKE Private Cluster Firewall)

This solution worked for me

Create a env variable CLUSTER_NAME with your cluster name or replace ${CLUSTER_NAME} for your cluster name directly in following command

gcloud compute firewall-rules list --filter="name~gke-${CLUSTER_NAME}-[0-9a-z]*-master"

Get firewall-rule-name returned by previous command and replace in next command

gcloud compute firewall-rules update <firewall-rule-name> --allow tcp:10250,tcp:443,tcp:15017
12

No. Instead, I reinstalled, K8 cluster & fresh Istio 1.17.1, and it is working now.

13

i did downgrade the istio from 1.17.2 to 1.17.1 but it is showing still the same error , am working on multi-cluster setup

https://istio.io/latest/docs/setup/install/multicluster/primary-remote/


Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject/cluster/k3d-me1mber/net/network16?timeout=10s": context deadline exceeded

can anyone please provide me the solution

14

I am getting the exact same issue:
EKS 1.25
VPC-CNI 1.12.2.eksbuild.1
Istio 1.17.2 installed with

istioctl install --set values.gateways.istio-ingressgateway.type=NodePort

k get po -n istio-system
NAME                                   READY   STATUS    RESTARTS   AGE
istio-egressgateway-85649899f8-6wd7c   1/1     Running   0          14h
istio-ingressgateway-f56888458-tlbsf   1/1     Running   0          14h
istiod-64848b6c78-tqqff                1/1     Running   0          14h

symptom appears when trying to deploy a RS to a istio-injection labeled NS:

Error creating: Internal error occurred: failed calling webhook "namespace.sidecar-injector.istio.io": failed to call webhook: Post "https://istiod.istio-system.svc:443/inject?timeout=10s": context deadline exceeded

The same diagnostics as the original post:

kubectl get --raw /api/v1/namespaces/istio-system/services/https:istiod:https-webhook/proxy/inject -v4
...
metadata": {},
  "status": "Failure",
  "message": "error trying to reach service: dial tcp 10.0.0.21:15017: connect: connection timed out",
  "reason": "ServiceUnavailable",
  "code": 503
}]
Error from server (ServiceUnavailable): error trying to reach service: dial tcp 10.0.0.21:15017: connect: connection timed out
.....

I also get the same results for the curl command for pods on the same node/same namespace and for pods on different nodes on different namespaces.

curl https://istiod.istio-system.svc:443/inject -k

no body found

actions taken:
I have opened the ingress to the node sg and the acls on the subnets look ok.
Is there something else to try ?
Has anyone found the fix for this on EKS ?

15

Ok the fix for my issue was locating the launch template security groups and adding ingress from suorce cluster security group for tcp 443 and 15017.
That did it !

16

great!!!
This solved my problem.

17

Yes, I had same issue while deploying istio v1.20 on EKS v1.27 and it fixed by modify Security for EKS allow traffic all from istio and it’s worked