Istio-ingressgateway pods tries to come up before Istiod pods and fails with image pull error

Contributors
1

Istio-ingressgateway pods tries to come up before Istiod pod and fails with image pull error.

Error Output:

$ k get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-ingressgateway-58f67444d8-8ntm8 1/1 ImagePullError 0 4h13m
istiod-7b76584b5b-7mcr4 1/1 Running 0 12h

kubectl describe pod output:

Normal Scheduled 54m default-scheduler Successfully assigned istio-system/istio-ingressgateway-58f67444d8-kdpvw to aks-workersxxxxxxxxxxxxxx
Normal Pulling 53m (x4 over 54m) kubelet Pulling image “auto”
Warning Failed 53m (x4 over 54m) kubelet Failed to pull image “auto”: rpc error: code = Unknown desc = failed to pull and unpack image “Docker”: failed to resolve reference “Docker”: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
Warning Failed 53m (x4 over 54m) kubelet Error: ErrImagePull
Warning Failed 53m (x6 over 54m) kubelet Error: ImagePullBackOff
Normal BackOff 4m47s (x222 over 54m) kubelet Back-off pulling image “auto”

istio-ingressgateway pod logs:
Error from server (BadRequest): container “istio-proxy” in pod “istio-ingressgateway-58f67444d8-kdpvw” is waiting to start: trying and failing to pull image

Expectations:
istio-ingressgateway pod should wait until istiod pod is up.

2

This seems to be an error with pulling the image, not necessarily an issue with which pod comes up before the other.

How are you installing Istio?

3

This issue gets resolved if I delete the istio-ingressgateway pod after the istiod pod is Up.
so which means istio-ingressgateway pod needs something from istiod pod to come up.

we are using helm charts to install the istio.

4

Can you check if there are any errors in istiod pod? Also, where are you running your cluster?

5

I had the same exact issue with the istio-system pod, when installing Istio using Helm charts.

alisson@alisson-ubuntu:~/repos/devops$ kubectl get pods
NAME                            READY   STATUS             RESTARTS   AGE
istio-system-7bc9d7fb64-bxdnk   0/1     ImagePullBackOff   0          2m21s
istiod-977466b69-gnwtt          1/1     Running            0          2m21s

alisson@alisson-ubuntu:~/repos/devops$ kubectl logs istio-system-7bc9d7fb64-bxdnk
Error from server (BadRequest): container "istio-proxy" in pod "istio-system-7bc9d7fb64-bxdnk" is waiting to start: trying and failing to pull image

Then after deleting the istio-system pod, a new one was able to run just fine.

6

thea “auto” image is a place holder for the istio mutating webhhok. Once istiod is up the mutating webhook replaces the “auto” image with envoy that corresponds to the istiod version