Service "istio-sidecar-injector" not found

1

Hi,

I’m new to Istio in general, recently I took over a kubeflow project using istio on EKS.

I recently moved the istio deployments to a newly created node group (using NodeSelector in the deployment) to help protect it from being autoscaled. And now I’m having this weird problem.

At first, istio-ingressgateway had 0/1 pods, and investigation showed that it was happening because of the following error:
$ kubectl describe deploy istio-ingressgateway -n istio-system

  Type     Reason        Age                  From                   Message
  ----     ------        ----                 ----                   -------
  Warning  FailedCreate  14m (x115 over 26h)  replicaset-controller  Error creating: Internal error occurred: failed calling webhook "sidecar-injector.istio.io": Post https://istio-sidecar-injector.istio.svc:443/inject?timeout=30s: service "istio-sidecar-injector" not found

But when I run
$ kubectl get service istio-sidecar-injector -n istio-system
I get

NAME                     TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)   AGE
istio-sidecar-injector   ClusterIP   10.100.85.254   <none>        443/TCP   66d

So the service appears to be there. I tried backing up, deleting and restoring the istio-sidecar-injector deployment, and now also istio-sidecar-injector fails with the same error.

Any idea what can be causing this?

Thanks in advance

2

The inrgess gateway should not be getting auto injected. I suggest inspecting the configuration of the webhook (kubectl get mutatingwebhookconfigurations.admissionregistration.k8s.io -oyaml) to determine how it is configured. Likely this mean istio-system is label istio-injection=enabled maybe?

1 Like
3

Thank you @howardjohn That was the issue! I have no idea how that label go there, but the services are running now.

But I can’t seem to connect to my cluster (kubeflow on port 80) and when I run get ingress i see it does not have an address.

$  kubectl get ingress -n istio-system
NAME            HOSTS   ADDRESS   PORTS   AGE
istio-ingress   *                 80      67d
4

Hi i am also running with same issue now, but i am not able to find thee value istio-injection=enabled inside istio-sidecar-injector when i am doing json format

kubectl -n istio-system get configmap istio-sidecar-injector -o=jsonpath=’{.data.config}’

could you please help me