[Resolved] CronJob and mutual tls

#1

I want to run a backup through CronJob like described here. Unfortunately, no connection can be established from CronJob’s pod. Sidecar is successfully injected and there no errors in istio-init and istio-proxy

Adding a DestinationRule doesn’t help (or I did it wrong). Is there any information how to connect to mutual tls secured service from a cronjob?

#2

There is an open issue on CronJobs [ https://github.com/istio/istio/issues/11659 ] but I don’t think it is related to the problem you are having with connections not being established.

Kubernetes starts your job and the sidecar in parallel. It may take a few seconds before networking is ready. When I have encountered that in regular pods I modified my pod descriptor. Perhaps you can modify your job in a similar way.

K8s deployment descriptor command injection that waits for sidecar then runs original startup

command: ["/bin/bash", "-c"]
args: ["until curl --head localhost:15000 ; do echo Waiting for Sidecar; sleep 3 ; done ; echo Sidecar available; ./origin-startup.sh"]
#3

Thank you for your response!

Yes, I’ve seen this issue and faced it too, but it’s not related to the current issue. I’ve tested with:

      initContainers:
      - name: init-wait
        image: alpine
        command: ["sh", "-c", "for i in $(seq 1 300); do nc -zvw1 mariadb.ns1.svc.cluster.local 3306 && exit 0 || sleep 3; done; exit 1"]

connection itself is open, but the behavior is the same like described in #10062 (broken mtls). I had this issue with my services before, but fixed with explicit ISTIO_MUTUAL, like described in the issue. But have no clue how to do it with CronJob.

#4

The proposals on #10062 should work identically with a Job and a Pod.

When I get into this situation I use the command: override as above with a pure sleep 999999999 and kubectl exec in to run interactive versions of the commands hoping for more insight into the problem.

Your example shows a svc.cluster.local in ns1. By default Istio when a namespace is created Istio does mTLS but doesn’t inject the sidecar. Make sure the namespace is labeled.

When I get really stuck I do a “4 point check”.

  1. Try to run curl or the client using localhost from the app container or sidecar to verify the app is up and listening on 127.0.0.1
  2. Try to run curl or the client from the sidecar to the IP address of the pod. Using curl arguments --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem -k if mTLS enabled.
  3. Try to run curl or the client from the proxy of the client pod, using curl arguments --key /etc/certs/key.pem --cert /etc/certs/cert-chain.pem --cacert /etc/certs/root-cert.pem -k if mTLS enabled.
  4. Try to run curl or the client from the client app interactively, exactly as the app itself would.
#5

Thank you very much. Indeed, this was timing issue. There is no need for DestinationRule for CronJob/Pod.

Now fighting with 11659 :frowning: