K8S Istio sidecar injection with other init containers

I have istio-injection: enabled in my project’s namespace.
I have a job that uses an initContainer and a normal container (to provide some ordering).

This seems to be incompatible with Istio sidecar…
It seems that istio-init is an initContainer which finishes before istio-proxy.
istio-proxy must be running before anything else on the pod can connect to outside the pod, correct?

So, because I have my own initContainer, that must finish before the normal containers are started, and this container requires network traffic… so it fails because istio-proxy is not yet running…

Are there some configuration options within Istio that would help here?
There is no way to enforce ordering of containers other than initContainers and containers

Or is the answer: exclude jobs that use initContainers from sidecar injection, accepting the loss of traffic monitoring/security?

Thanks for any help.

1 Like

There is no easy solution. Your init container will run before the sidecar starts. If your container runs before Istio’s init container it will not be secure. If your container runs after Istio’s it will not have network access.

If you can avoid doing network I/O in your init containers you should. If you must use init containers that expect connectivity, you’ll need a work-around.

If an application uses an Init Container for orchestration (for example trying to contact a remote CouchDB before starting up an app that depends on immediate connectivity to CouchDB) the pod initialization will not complete.

If your init container is just a tiny script you can sometimes move it from its own container to the app container with a hack like the following in your Deployment .yaml.

command: ["/bin/bash", "-c"]
args: ["until curl --head localhost:15000 ; do echo Waiting for Sidecar; sleep 3 ; done ; echo Sidecar available; ./init-stuff.sh && ./startup.sh"]

(where init-stuff.sh gets replaced by whatever your init used to do, and startup.sh is whatever command starts your app container).

1 Like

There are a number of things coming down the pipeline which will help address this, including some changes that are being suggested within upstream kubernetes, however this is an issue for the current istio installations.

One topic of interest is the sidecar proposal, this will allow us to mark containers specifically as sidecars and have them follow a different lifecycle to the application containers (and init containers). This is likely many months away if not further out.

Another is enhancements we are making to the CNI plugin so it can start/stop proxy. We have a PoC of this running now but have one last problem to solve before we bring it to the community, accountability for the resource usage. This will also help us remove the requirement for the application’s service account to have elevated privileges.

The same person on our team is involved with both efforts, Marko Luksa.

2 Likes

Thank you @ed.snible for the suggestion, helps us going forward for now.
Thank you @kconner for the information, we’ll be keeping an eye on this - looking forward to that future.

I had the same problem with a NodeJS microservice inside an Istio mesh that need to connect to a MongoDB instance at startup outside of the mesh. My NodeJs application always started before the istio proxy and the connection to MongoDB used to fail.
I solved my proble using an HTTP livenessProbe which call a route (/ping) that is created only if the Mongo connection succeed.
So, when the pod is deployed, the mongo connection fails, then the istio proxy starts, and then the probe fails which restart the NodeJS container. After that, the mongo connection succeed.

1 Like

So this effectively means having to remove all init-containers and move all that logic back into the main application container :cold_sweat:

Any progress on this issue? I see discussion from a year ago about solutions that are “months” out. Has anything materialized?

I am new to Istio, I just hit the same problem as I am using Vault initContainer sidecar as part of the same pod to inject secrets.

You may work around this problem with Vault initContainers by define them in your pod spec instead of using the auto injection. InitContainers in the pod spec are loaded first than injected ones, so, if your network allows, you can load the secrets before Istio changes the network rules for your app.

This feature would be very much appreciated to be able to use a initContainer for data migrations, like described in the project of Sebastian

Hey so would it be a long shot to build a tooling/init container based off of the istio proxy itself, add one’s tools and scripts needed, then include in the initContainers section explicitly :thinking: ? Just starting hands-on with Istio, so this could be quite terribly stupid idea, but yeah this specific use case seems like quite a blocker to some scenarios indeed.

I think I have the same situation as described although my db is deployed as a subchart in the same namespace. This is for pull request testing purposes not production deploys.

My db migration script has prehooks and it needs to reach the Postgres db. It uses secrets that are created by Postgres when it’s deployed. To ensure they are there and that Postgres is up I run an into container that checked that it’s reachable.This knit container definition is in the pod spec. Because it’s defined there, it fires before the istio initContainer that was injected via annotations.
I need istio’s init container to complete so that the proxy is running so that my initContainer can reach the db.

I think I’m restating the same case as the original poster except their case may be outside the namespace.

Much like another poster I’m wondering if I can now define some snippet of istio’s init container config in my pod spec, purely for the purpose of forcing it’s execution to be first, such that the remainder of the configuration comes from the cluster config: same as when the annotation is used to enable / disable it.

Hi,
i’ve just faced the same issue, here are my findings:

  • as per k8s docs, init containers are started in the order in which they appear in the manifest
  • i could not find yet in which order admission webhooks are invoked by k8s admission controller, so I’m guessing it’s undefined
  • also I could not find any parameters which could influence the order istio injects it’s init container, so I’m guessing it depends on the webhook invocation order
  • interestingly, in my case out of >100 pods only 1 has istio-init come first, in which case vault-init is unable to access GKE metadata meaning it cant authenticate to the vault server meaning the pod gets stuck in initializing state forever
  • at least to me at least things are consistent, so the same pod with the same manifest will get the initcontainers injected in the same order at all times
  • i still fail to see what is different with the one pod not working. The main difference I see, that it has the vault init container config as pod annotation, while the others have it in a separate configmap. I fail to see how that would influence the webhook invocation order…

IMHO istio could make sure that it would inject itself as the last initcontainer and make sure this is the case even if some other admission webhook is called afterwards and adds it’s init container after it (by setting reinvocationPolicy and getting reinvoked if other mutating admission webhooks alter the pod as well)

We faced this problem yesterday and was able to resolve it using
securityContext:
runAsUser: 1337

Pod are fail to start due to init-containers not starting with ISTIO CNI Enabled

Problem

When trying to deploy pods with Istio CNI enabled, pods status are uninitialized or redeploying.

Environment

  • Managed Anthos service mesh on Google Kubernetes Engine
  • Istio CNI Enabled
  • init-containers are executing before the sidecar proxy has started

Solution

Update the UID of the init-container to 1337, which is the same UID used by the sidecar proxy and traffic sent by this uid will not be captured in Istio’s iptable rule.

For example, to update the uid of the init-container named “nginx” in the “nginx” namespace, you would use the following command:

kubectl run asuser nginx --container nginx

This would open up a shell prompt in the container, where you can change the UID value. Once you have made your changes, you can exit the shell prompt and the init-container’s configuration will be updated with your changes.

Cause

When init-containers execute before the sidecar proxy starts, traffic is loss during their execution and causes pods to redeploy. By setting the uid to the same as the sidecar proxy, traffic won’t be sent, allowing for the sidecar proxy to start and the pods to be able to deploy.