Applyig mTLS breaks the communication between jaeger collector and cassandra

Jaeger helm charts are deployed having jaeger agent, collector, cassandra , query. The whole set-up works fine on a namespace having injection enabled.
However when we apply the mTLS policy, the jaeger collector fails to connect to the hosts. The istio-proxy of jaeger collector fails to find the JWT token. Copied below are the details:

mtls Policy:
apiVersion: authentication.istio.io/v1alpha1
kind: Policy
metadata:
name: default
spec:
peers:

  • mtls: {}

Destination route:
apiVersion: “networking.istio.io/v1alpha3
kind: “DestinationRule”
metadata:
name: “default”
spec:
host: “*.local”
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
~
~

LOGS from Jaeger collector pod:
2020/05/28 05:47:52 gocql: dns error: lookup jaeger-cassandra on 10.96.0.10:53: no such host
{“level”:“fatal”,“ts”:1590644872.2643485,“caller”:“collector/main.go:91”,“msg”:“Failed to init storage factory”,“error”:"gocql: unable to create session: failed to resolve any of the provided hostnames

{“level”:“info”,“ts”:1590644962.0080297,“caller”:“flags/admin.go:100”,“msg”:“Admin server started”,“http-port”:14269,“health-status”:“unavailable”}
2020/05/28 05:49:23 gocql: unable to dial control conn 10.32.0.20: EOF
{“level”:“fatal”,“ts”:1590644963.022509,“caller”:“collector/main.go:91”,“msg”:“Failed to init storage factory”,“error”:"gocql: unable to create session: control: unable to connect to initial hosts: EOF

LOGS from istio-proxy side car of jaeger collector

2020-05-28T06:07:41.151144Z info JWT policy is third-party-jwt
2020-05-28T06:07:41.151258Z warn Missing JWT token, can’t use in process SDS ./var/run/secrets/tokens/istio-tokenstat ./var/run/secrets/tokens/istio-token: no such file or directory
2

cc @incfly. Can you share the detailed steps for reproducing the issue? Which version of Istio are you using?

Istio version: 1.5.2

Details below:

Helm Charts: https://github.com/jaegertracing/helm-charts/tree/master/charts/jaeger

Configuration steps used:

  1. Open the values.yaml and configure cassandra as follows:
    provisionDataStore:
    cassandra: true
    elasticsearch: false

  2. create a kubernetes namespace “jaeger” and enable istio auto injection

  3. Deploy mTLS policies. Create a directory called policies under templates and place the mTLS and destination route policies.

  4. Deploy the hem charts.
    helm install -n jaeger --namespace jaeger .

  5. Observe the pods

[kubectl get pods -n jaeger

NAME READY STATUS RESTARTS AGE
jaeger-agent-jn8kv 2/2 Running 0 7m38s
jaeger-agent-k7v5k 2/2 Running 0 7m38s
jaeger-cassandra-0 2/2 Running 0 7m38s
jaeger-cassandra-1 1/2 CrashLoopBackOff 3 5m55s
jaeger-collector-6965df75-lmfxl 1/2 CrashLoopBackOff 6 7m38s
jaeger-query-5bbcb9cb84-d589c 2/3 CrashLoopBackOff 6 7m38s

  1. check the pod logs of jaeger-collector and istio-proxy under it.

Please let me know if you need further details.

For now the issue seems to be resolved after the cassandra schema is configured for a higher timeout. Brought up the jaeger services with mTLS applied. Since the cassandra schema job service was crashing early, changed a setting in values.yaml

activeDeadlineSeconds : 1200