Hi, I am trying to install upstream Istio into OpenShift 4.3.
I have found some links in Istio site where they explain how to do it, but some pods are stuck at ContainerCreating status:
istio-citadel-7bcc69d486-v6qcf 0/1 ContainerCreating 0 3m25s
istio-galley-79c57df7b7-x2zkw 0/2 ContainerCreating 0 3m24s
istio-ingressgateway-7d897944-bxg29 0/1 ContainerCreating 0 3m25s
istio-pilot-565b4c4c9c-7sfcd 0/2 Pending 0 3m24s
istio-policy-76f487b7df-hddrq 0/2 ContainerCreating 0 3m20s
istio-sidecar-injector-79ccdf54c4-pm7vw 0/1 ContainerCreating 0 3m24s
istio-telemetry-864654ddd8-p2s9g 0/2 Pending 0 3m25s
prometheus-66c5887c86-krgwq 0/1 ContainerCreating 0 3m25s
I have waited like 5 minutes but didn’t succeed, checking the events it seem some kind of permission errors with the network.
Then the links I found are:
The OpenShift 4.3 is installed in Azure and after the installation I logged with admin account and I run:
./oc adm policy add-scc-to-group anyuid system:serviceaccounts -n istio-system
istioctl manifest apply --set values.global.proxy.privileged=true --set cni.components.cni.namespace=kube-system --set values.cni.cniBinDir=/var/lib/cni/bin --set values.cni.cniConfDir=/var/run/multus/cni/net.d --set cni.enabled=true
So I am enabling the CNI as well.
But not success, do anyone have experience on installing Istio in OpenShift 4.1, 4.2, or 4.3 that can tell me if I need to do anything else before/after?
Thank you very much.
Don’t know what version of Istio you are trying to install… that list of pods doesn’t look like the latest upstream Istio, correct? (1.5.x) Because many of those pods are gone now and there should be an “istiod”.
The stuff you need to do on OpenShift changed between Istio 1.5 and earlier (particularly dealing with CNI settings).
You can compare how some of us on the Kiali team do this - we have a script. Compare what is done here with what you are doing:
https://github.com/kiali/kiali/blob/7cfa1c2cf661dccad0563f35c98a4c3bb17fc05a/hack/istio/install-istio-via-istioctl.sh
This script works for upstream Istio 1.5.0 - however, there was a report it did not work on 1.5.1 due to this Istio bug: https://github.com/istio/istio/issues/22496
This should work on OpenShift 4.2 and up (and maybe even 4.1, but its been a while since I tried on that version of OpenShift, so no guarantees).
This script also worked for Istio 1.4, but I believe you need an earlier version of that script due to changes in the CNI settings. So just look at the history of that script to see what changed:
https://github.com/kiali/kiali/commits/master/hack/istio/install-istio-via-istioctl.sh
Currently I am using Istio 1.4 as it is the next version supported for Maistra so I do not want to bump to Istio 1.4, so yes you are right.
And thank you very very much for helping me on this.
@jmazzitelli I tried the script you mention but using one compatible with 1.4.6 instead of 1.5 then I configured everything and I run it and I got the next errors:
kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-citadel-7bcc69d486-htv7p 1/1 Running 0 20m
istio-egressgateway-6b6f694c97-5dzkg 0/1 Running 0 20m
istio-galley-6cf8f58d7b-q9v4p 0/1 ContainerCreating 0 4m48s
istio-ingressgateway-8c9c9c9f5-rg7jm 0/1 Running 0 20m
istio-pilot-c6dbc54b9-28tpq 0/1 Running 0 20m
istio-policy-66c4cf95c5-spj4v 0/1 CrashLoopBackOff 11 19m
istio-sidecar-injector-79ccdf54c4-7d5h8 0/1 ContainerCreating 0 6m16s
istio-telemetry-7bf96cb54f-4dx76 0/1 CrashLoopBackOff 11 20m
prometheus-66c5887c86-dfgv6 0/1 ContainerCreating 0 20m
Then I think that most of the errors are because one service is failing, let me put here the events:
Istio-galley: ReadinessIndicatorFile: timed out waiting for the condition
Istio Policy: Liveness probe failed: Get http://10.131.0.17:15014/version: dial tcp 10.131.0.17:15014: connect: connection refused
Isito Pilot: Readiness probe failed: HTTP probe failed with statuscode: 503
Istio Sidecar Injector: PollImmediate error waiting for ReadinessIndicatorFile: timed out waiting for the condition
Istio Telemetry: Liveness probe failed: Get http://10.128.2.16:15014/version: dial tcp 10.128.2.16:15014: connect: connection refused
So I think that the readiness probes are failing because of pods with livenesss probes failing, but no idea why.
When I tried to uninstall I get it Error from server (NotFound): error when deleting “STDIN”: services “istio-policy” not found
Error from server (NotFound): error when deleting “STDIN”: serviceaccounts “istio-policy-service-account” not found
So it seems not everything is installed.
And more news I tried Istio 1.5.0 and istiod stucks failing with the readiness probe too, I am not sure why this is happening, if could be a problem in Azure or in OpenShift 4.3 or what.
Hiya Alex
There are quite a few reasons the pods may not be deploying but without more context it’s hard to say what is going on. It could be that you don’t have sufficient resources in the cluster to deploy the pods, especially if using the default installation configuration which comes with OpenShift, or that there are permission issues.
The best thing is to take a look at the pods themselves and see why they are not deploying, a describe should be able to tell you more. Look at all the pods which are stuck in ContainerCreating and ClashLoopBackOff, starting with galley.
As for the question on CNI you will need to deploy this but it likely has no impact on whether the pods will deploy or not. OpenShift 4.1+ uses nftables (not iptables) on the host so the init container approach will not work unless it has been updated to configure nftables. If this fails then the interception will not occur but the sidecar would still deploy.
I am having the same issue with OpenShift 4.3.29 (running on AWS installed via IPI). istio-cni is running successfully under kube-system namespace but galley (1.4.7) fails to start with Multus error on ReadinessIndicatorFile.
FWIW, I am still troubleshooting this issue, but istio-cni installation breaks OpenShift 4.3.29 Multus because it renames the file /var/run/multus/cni/net.d/80-openshift-network.conf to /var/run/multus/cni/net.d/80-openshift-network.conflist . The same file is “readinessindicatorfile” as multus expects (see multus daemonset under openshift-multus).
If you rename the file to the original name on each node, galley starts.
Warning FailedCreatePodSandBox 15m kubelet, ip-10-0-176-253.us-west-2.compute.internal Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istio-galley-867b9f8bc6-g7rgn_istio-system_dbda1038-df96-4444-b8a2-bc629e99585a_0(ed16213387afb9e037c819ab919989de94bbcd83999b0b2390ed31cced36e103): Multus: [istio-system/istio-galley-867b9f8bc6-g7rgn]: PollImmediate error waiting for ReadinessIndicatorFile: timed out waiting for the condition
Warning FailedCreatePodSandBox 81s (x9 over 13m) kubelet, ip-10-0-176-253.us-west-2.compute.internal (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istio-galley-867b9f8bc6-g7rgn_istio-system_dbda1038-df96-4444-b8a2-bc629e99585a_0(a299cdb04fc13eb356d1dafcc490f451a9d3b1eaded62560faa8b6b99d5fcb67): Multus: [istio-system/istio-galley-867b9f8bc6-g7rgn]: error adding container to network "k8s-pod-network": delegateAdd: error invoking conflistAdd - "k8s-pod-network": conflistAdd: error in getting result from AddNetworkList: failed to find plugin "istio-cni" in path [/opt/multus/bin /var/lib/cni/bin /usr/libexec/cni]
Normal Pulling 68s kubelet, ip-10-0-176-253.us-west-2.compute.internal Pulling image "docker.io/vmwareallspark/galley:1.4.7"
Normal Pulled 64s kubelet, ip-10-0-176-253.us-west-2.compute.internal Successfully pulled image "docker.io/vmwareallspark/galley:1.4.7"
Normal Created 64s kubelet, ip-10-0-176-253.us-west-2.compute.internal Created container galley
Normal Started 64s kubelet, ip-10-0-176-253.us-west-2.compute.internal Started container galley
UPDATE 07/30:
The way to solve this issue is setting cniConfFileName=istio-cni.conf as documented on Istio 1.6 page:
The Istio 1.4 documentation does not make any reference on this (suggest someone to add it):
Another word of caution is after installing Istio 1.4 with this parameter, the istio-cni pod keep generating warning messages, multiple times a minute:
WARNING: Configured CNI config file “/host/etc/cni/net.d/istio-cni.conflist” preempted by “80-openshift-network.conf”.
WARNING: Configured CNI config file “/host/etc/cni/net.d/istio-cni.conflist” preempted by “80-openshift-network.conf”.
WARNING: Configured CNI config file “/host/etc/cni/net.d/istio-cni.conflist” preempted by “80-openshift-network.conf”.