I have created istio enabled service on EKS 1.19 with istio 1.9 The service was created but the proxy was not able to authenticate. I have entered the proxy pod and have found that ./var/run/secrets/tokens/istio-token is not accessible by the istio-proxy account. The token has root permissions:lrwxrwxrwx 1 root root 18 Mar 16 20:06 ./var/run/secrets/tokens/istio-token → …data/istio-token. The proxy has “failed to fetch token from file: open ./var/run/secrets/tokens/istio-token: permission denied” message in the logs. Is there a way to fix the problem?
Hi, may you show some details on the deployed service/pod in yaml and how it was created? Do you still face the problem?
I have recreated cluster with eksctl and istio works now. I suspect that upgrade to k8s 1.19 created the problem.
I was facing the same issue. Are you using EKS 1.18 AMI? It will work until the node restarts, and won’t work with new nodes. EKS 1.19 AMI has been working fine for me. I believe EKS 1.19 broke backward compatibility with previous version of Kubelet, and below might be the root cause:
You’re no longer required to provide a security context for non-root containers that need to access the web identity token file for use with IAM roles for service accounts. For more information, see IAM roles for service accounts andproposal for file permission handling in projected service account volume on GitHub.
I had initially created the cluster using Pulumi and then upgraded it to EKS 1.9. I don’t know the root of the problem, but when I recreated cluster using eksctl without upgrading to 1.19 it works fine. There are two options here, but I more suspect that the upgrade broke it.
Are you sure you have updated both masters and workers?
I had the same issue with AKS but i had forgotten to update worker nodes…
As soon as i have updated also the worker all goes fine 
I got same issue when after upgrade eks version from 1.8 to 1.9.
So i deleted previous nodepool and provisioned new nodepool.
After that, error was gone.
Created a bug to track this: Istio-token permissions problem on EKS - failed to fetch token from file · Issue #31740 · istio/istio · GitHub
Can someone experiencing this try to set --set values.pilot.env.ENABLE_LEGACY_FSGROUP_INJECTION=true on installation and see if it occurs?
I started having this problem a few days ago.
Installing with --set values.pilot.env.ENABLE_LEGACY_FSGROUP_INJECTION=true makes it go away.
Like AlexF, this happened when I upgraded to k8s 1.19. I am using IBM Kubernetes Service.
1 Like
thanks @ed.snible , we also faced the same issue in 1.19, install with --set values.pilot.env.ENABLE_LEGACY_FSGROUP_INJECTION=true works for me as well, we are using GKE btw.
Thanks a lot, same problem occurred on my environment with k8s v1.21 and istio v1.13 on my own cluster.