Istio-cni " No such file or directory"

I’m looking to use istio with the istio-cni so I don’t need to have an istio-init container with NET_RAW and NET_ADMIN starting with every pod.

I’ve gone through the documentation at:

and I’m install istio-cni in the kube-system namespace like

istioctl install --set components.cni.enabled=true --set components.cni.namespace=kube-system

and it appears to be starting correctly, I can see the running istio-cni-node-XXXXX pods in the kube-system.

The problem is when I try to start pods that rely on istio-cni. the pods are stuck in the Init:0/1. describe pod shows:

Warning FailedCreatePodSandBox 3m53s (x1547 over 123m) kubelet, (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to set up sandbox container “514b71c470c1eb625909b4412d90b730d8c6e0af1f4c3696c703bceb7e8e7f29” network for pod “httpbin-654c6cbbb9-gj8kd”: NetworkPlugin cni failed to set up pod “httpbin-654c6cbbb9-gj8kd_with-istio” network: exit status 1

and journalctl -xu kubelet on the node shows

Jun 22 18:41:18 kubelet-wrapper[1491]: {“level”:“info”,“time”:“2020-06-22T18:41:18.374659Z”,“msg”:“nsenter args”,“nsenterArgs”:["–net=/proc/10153/ns/net","/opt/cni/bin/","-p",“15001”,"-u",“1337”,"-m",“REDIRECT”,"-i","","-b","","-d",“15020,15021,15090”,"-o",“15020”,"-x","","-k",""]}
Jun 22 18:41:18 kubelet-wrapper[1491]: {“level”:“error”,“time”:“2020-06-22T18:41:18.375738Z”,“msg”:“nsenter failed”,“out”:“nsenter: failed to execute /opt/cni/bin/ No such file or directory\n”,“error”:“exit status 1”}
Jun 22 18:41:18 kubelet-wrapper[1491]: {“level”:“info”,“time”:“2020-06-22T18:41:18.375758Z”,“msg”:“nsenter out: nsenter: failed to execute /opt/cni/bin/ No such file or directory\n”}

I’ve verified that /opt/cni/bin/ exists on the kubelet and in the istio-cni docker containers.

where should I be looking to figure out what’s wrong?

edit: this is on version

client version: 1.6.3
control plane version: 1.6.3
data plane version: 1.6.3 (2 proxies)

I’ve tried with 1.5.4, same result. :expressionless: