New Istio version in OpenShift 4.3

lordofthejars · March 29, 2020, 9:21am

Hi, I am trying to install upstream Istio into OpenShift 4.3.

I have found some links in Istio site where they explain how to do it, but some pods are stuck at ContainerCreating status:

istio-citadel-7bcc69d486-v6qcf 0/1 ContainerCreating 0 3m25s
istio-galley-79c57df7b7-x2zkw 0/2 ContainerCreating 0 3m24s
istio-ingressgateway-7d897944-bxg29 0/1 ContainerCreating 0 3m25s
istio-pilot-565b4c4c9c-7sfcd 0/2 Pending 0 3m24s
istio-policy-76f487b7df-hddrq 0/2 ContainerCreating 0 3m20s
istio-sidecar-injector-79ccdf54c4-pm7vw 0/1 ContainerCreating 0 3m24s
istio-telemetry-864654ddd8-p2s9g 0/2 Pending 0 3m25s
prometheus-66c5887c86-krgwq 0/1 ContainerCreating 0 3m25s

I have waited like 5 minutes but didn’t succeed, checking the events it seem some kind of permission errors with the network.

Then the links I found are:

The OpenShift 4.3 is installed in Azure and after the installation I logged with admin account and I run:

./oc adm policy add-scc-to-group anyuid system:serviceaccounts -n istio-system

istioctl manifest apply --set values.global.proxy.privileged=true --set cni.components.cni.namespace=kube-system --set values.cni.cniBinDir=/var/lib/cni/bin --set values.cni.cniConfDir=/var/run/multus/cni/net.d --set cni.enabled=true

So I am enabling the CNI as well.

But not success, do anyone have experience on installing Istio in OpenShift 4.1, 4.2, or 4.3 that can tell me if I need to do anything else before/after?

Thank you very much.

jmazzitelli · March 29, 2020, 1:13pm

Don’t know what version of Istio you are trying to install… that list of pods doesn’t look like the latest upstream Istio, correct? (1.5.x) Because many of those pods are gone now and there should be an “istiod”.

The stuff you need to do on OpenShift changed between Istio 1.5 and earlier (particularly dealing with CNI settings).

You can compare how some of us on the Kiali team do this - we have a script. Compare what is done here with what you are doing:

https://github.com/kiali/kiali/blob/7cfa1c2cf661dccad0563f35c98a4c3bb17fc05a/hack/istio/install-istio-via-istioctl.sh

This script works for upstream Istio 1.5.0 - however, there was a report it did not work on 1.5.1 due to this Istio bug: https://github.com/istio/istio/issues/22496

This should work on OpenShift 4.2 and up (and maybe even 4.1, but its been a while since I tried on that version of OpenShift, so no guarantees).

This script also worked for Istio 1.4, but I believe you need an earlier version of that script due to changes in the CNI settings. So just look at the history of that script to see what changed:

https://github.com/kiali/kiali/commits/master/hack/istio/install-istio-via-istioctl.sh

lordofthejars · March 29, 2020, 1:11pm

Currently I am using Istio 1.4 as it is the next version supported for Maistra so I do not want to bump to Istio 1.4, so yes you are right.

lordofthejars · March 29, 2020, 1:12pm

And thank you very very much for helping me on this.

lordofthejars · March 29, 2020, 3:42pm

@jmazzitelli I tried the script you mention but using one compatible with 1.4.6 instead of 1.5 then I configured everything and I run it and I got the next errors:

kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
istio-citadel-7bcc69d486-htv7p 1/1 Running 0 20m
istio-egressgateway-6b6f694c97-5dzkg 0/1 Running 0 20m
istio-galley-6cf8f58d7b-q9v4p 0/1 ContainerCreating 0 4m48s
istio-ingressgateway-8c9c9c9f5-rg7jm 0/1 Running 0 20m
istio-pilot-c6dbc54b9-28tpq 0/1 Running 0 20m
istio-policy-66c4cf95c5-spj4v 0/1 CrashLoopBackOff 11 19m
istio-sidecar-injector-79ccdf54c4-7d5h8 0/1 ContainerCreating 0 6m16s
istio-telemetry-7bf96cb54f-4dx76 0/1 CrashLoopBackOff 11 20m
prometheus-66c5887c86-dfgv6 0/1 ContainerCreating 0 20m

Then I think that most of the errors are because one service is failing, let me put here the events:

Istio-galley: ReadinessIndicatorFile: timed out waiting for the condition
Istio Policy: Liveness probe failed: Get http://10.131.0.17:15014/version: dial tcp 10.131.0.17:15014: connect: connection refused
Isito Pilot: Readiness probe failed: HTTP probe failed with statuscode: 503
Istio Sidecar Injector: PollImmediate error waiting for ReadinessIndicatorFile: timed out waiting for the condition
Istio Telemetry: Liveness probe failed: Get http://10.128.2.16:15014/version: dial tcp 10.128.2.16:15014: connect: connection refused

So I think that the readiness probes are failing because of pods with livenesss probes failing, but no idea why.

lordofthejars · March 29, 2020, 3:46pm

When I tried to uninstall I get it Error from server (NotFound): error when deleting “STDIN”: services “istio-policy” not found
Error from server (NotFound): error when deleting “STDIN”: serviceaccounts “istio-policy-service-account” not found

So it seems not everything is installed.

lordofthejars · March 29, 2020, 4:10pm

And more news I tried Istio 1.5.0 and istiod stucks failing with the readiness probe too, I am not sure why this is happening, if could be a problem in Azure or in OpenShift 4.3 or what.

kconner · March 30, 2020, 1:50pm

Hiya Alex

There are quite a few reasons the pods may not be deploying but without more context it’s hard to say what is going on. It could be that you don’t have sufficient resources in the cluster to deploy the pods, especially if using the default installation configuration which comes with OpenShift, or that there are permission issues.

The best thing is to take a look at the pods themselves and see why they are not deploying, a describe should be able to tell you more. Look at all the pods which are stuck in ContainerCreating and ClashLoopBackOff, starting with galley.

As for the question on CNI you will need to deploy this but it likely has no impact on whether the pods will deploy or not. OpenShift 4.1+ uses nftables (not iptables) on the host so the init container approach will not work unless it has been updated to configure nftables. If this fails then the interception will not occur but the sidecar would still deploy.

brito-rafa · July 29, 2020, 7:00pm

I am having the same issue with OpenShift 4.3.29 (running on AWS installed via IPI). istio-cni is running successfully under kube-system namespace but galley (1.4.7) fails to start with Multus error on ReadinessIndicatorFile.

brito-rafa · July 30, 2020, 3:38pm

FWIW, I am still troubleshooting this issue, but istio-cni installation breaks OpenShift 4.3.29 Multus because it renames the file /var/run/multus/cni/net.d/80-openshift-network.conf to /var/run/multus/cni/net.d/80-openshift-network.conflist . The same file is “readinessindicatorfile” as multus expects (see multus daemonset under openshift-multus).

If you rename the file to the original name on each node, galley starts.

  Warning  FailedCreatePodSandBox  15m                kubelet, ip-10-0-176-253.us-west-2.compute.internal  Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istio-galley-867b9f8bc6-g7rgn_istio-system_dbda1038-df96-4444-b8a2-bc629e99585a_0(ed16213387afb9e037c819ab919989de94bbcd83999b0b2390ed31cced36e103): Multus: [istio-system/istio-galley-867b9f8bc6-g7rgn]: PollImmediate error waiting for ReadinessIndicatorFile: timed out waiting for the condition
  Warning  FailedCreatePodSandBox  81s (x9 over 13m)  kubelet, ip-10-0-176-253.us-west-2.compute.internal  (combined from similar events): Failed create pod sandbox: rpc error: code = Unknown desc = failed to create pod network sandbox k8s_istio-galley-867b9f8bc6-g7rgn_istio-system_dbda1038-df96-4444-b8a2-bc629e99585a_0(a299cdb04fc13eb356d1dafcc490f451a9d3b1eaded62560faa8b6b99d5fcb67): Multus: [istio-system/istio-galley-867b9f8bc6-g7rgn]: error adding container to network "k8s-pod-network": delegateAdd: error invoking conflistAdd - "k8s-pod-network": conflistAdd: error in getting result from AddNetworkList: failed to find plugin "istio-cni" in path [/opt/multus/bin /var/lib/cni/bin /usr/libexec/cni]
  Normal   Pulling                 68s                kubelet, ip-10-0-176-253.us-west-2.compute.internal  Pulling image "docker.io/vmwareallspark/galley:1.4.7"
  Normal   Pulled                  64s                kubelet, ip-10-0-176-253.us-west-2.compute.internal  Successfully pulled image "docker.io/vmwareallspark/galley:1.4.7"
  Normal   Created                 64s                kubelet, ip-10-0-176-253.us-west-2.compute.internal  Created container galley
  Normal   Started                 64s                kubelet, ip-10-0-176-253.us-west-2.compute.internal  Started container galley

UPDATE 07/30:
The way to solve this issue is setting cniConfFileName=istio-cni.conf as documented on Istio 1.6 page:

The Istio 1.4 documentation does not make any reference on this (suggest someone to add it):

Another word of caution is after installing Istio 1.4 with this parameter, the istio-cni pod keep generating warning messages, multiple times a minute:
WARNING: Configured CNI config file “/host/etc/cni/net.d/istio-cni.conflist” preempted by “80-openshift-network.conf”.

WARNING: Configured CNI config file “/host/etc/cni/net.d/istio-cni.conflist” preempted by “80-openshift-network.conf”.