Istio 1.9.1 - Kiali don't connect to Grafana and Jaeger

Hi all,
I have a Istio 1.9.1 installed using istio operator.
When I issue

istioctl d kiali

The kiali console says
grafana Unreachable
jaeger Unreachable

I installed the addons using:

kubectl apply -f
kubectl apply -f
kubectl apply -f
kubectl apply -f
kubectl apply -f

Then I shutted down the cluster and then rebooted

Kubecl for istio-system namespace says

sysop@m-serv:~$ kubectl get all -n istio-system
NAME                                        READY   STATUS    RESTARTS   AGE
pod/jaeger-7f78b6fb65-xk2xg                 1/1     Running   1          4h30m
pod/grafana-784c89f4cf-4hmxv                1/1     Running   1          4h33m
pod/zipkin-7fcd647cf9-7m5wg                 1/1     Running   1          4h22m
pod/istiod-d4f7d664f-mpxcb                  1/1     Running   1          4h59m
pod/istio-egressgateway-bd477794-wzmk5      1/1     Running   1          4h59m
pod/istio-ingressgateway-79df7c789f-22d4z   1/1     Running   1          4h59m
pod/prometheus-7bfddb8dbf-s6dl5             2/2     Running   2          4h36m
pod/svclb-istio-ingressgateway-4mdtv        5/5     Running   5          4h59m
pod/svclb-istio-ingressgateway-xtj7n        5/5     Running   5          4h59m
pod/kiali-dc84967d9-bzgvj                   1/1     Running   1          4h43m
pod/svclb-istio-ingressgateway-cs5h5        5/5     Running   5          4h59m

NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP                                       PORT(S)                                                                      AGE
service/istiod                 ClusterIP   <none>                                            15010/TCP,15012/TCP,443/TCP,15014/TCP                                        4h59m
service/istio-egressgateway    ClusterIP    <none>                                            80/TCP,443/TCP,15443/TCP                                                     4h59m
service/kiali                  ClusterIP     <none>                                            20001/TCP,9090/TCP                                                           4h43m
service/prometheus             ClusterIP   <none>                                            9090/TCP                                                                     4h36m
service/grafana                ClusterIP   <none>                                            3000/TCP                                                                     4h33m
service/jaeger-collector       ClusterIP   <none>                                            14268/TCP,14250/TCP                                                          4h30m
service/tracing                ClusterIP    <none>                                            80/TCP                                                                       4h30m
service/zipkin                 ClusterIP      <none>                                            9411/TCP                                                                     4h30m
service/istio-ingressgateway   LoadBalancer,,   15021:30380/TCP,80:31104/TCP,443:31620/TCP,31400:32241/TCP,15443:32064/TCP   4h59m

NAME                                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/svclb-istio-ingressgateway   3         3         3       3            3           <none>          4h59m

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jaeger                 1/1     1            1           4h30m
deployment.apps/grafana                1/1     1            1           4h33m
deployment.apps/zipkin                 1/1     1            1           4h22m
deployment.apps/istiod                 1/1     1            1           4h59m
deployment.apps/istio-ingressgateway   1/1     1            1           4h59m
deployment.apps/istio-egressgateway    1/1     1            1           4h59m
deployment.apps/prometheus             1/1     1            1           4h36m
deployment.apps/kiali                  1/1     1            1           4h43m

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/jaeger-7f78b6fb65                 1         1         1       4h30m
replicaset.apps/grafana-784c89f4cf                1         1         1       4h33m
replicaset.apps/zipkin-7fcd647cf9                 1         1         1       4h22m
replicaset.apps/istiod-d4f7d664f                  1         1         1       4h59m
replicaset.apps/istio-egressgateway-bd477794      1         1         1       4h59m
replicaset.apps/istio-ingressgateway-79df7c789f   1         1         1       4h59m
replicaset.apps/prometheus-7bfddb8dbf             1         1         1       4h36m
replicaset.apps/kiali-dc84967d9                   1         1         1       4h43m

“istioctl d grafana” works correctly
But jaeger dashboard says

sysop@m-serv:~/software/kubernetes$ istioctl d jaeger
2021-03-08T16:42:22.647224Z	error	klog	error copying from remote stream to local connection: readfrom tcp4> write tcp4> write: broken pipe

What I’m doing wrong?


If you use the operator to install Istio, probably you may need to personalize your Kiali installation telling where is located the prometheus and jaeger services.

To run this you can personalize the helm charts, or better, use the Kiali operator, which is designed and recommended for these kind of work.

Also, Christian Postal prepared some steps for combining operators in Istio and Kiali as well:

But please, tell us your details if using the operator doesn’t work for you.

We’ll look if there is any step missing in the doc.

Thanks !

P.S.: Feel free to join into the #kiali room in Istio Slack and/or Freenode IRC where probably other colleagues can help you out.

Thanks Lucas,
I’ll try asap.


I’m trying to use operator because it seems the only method to set the jwksResolverExtraRootCA (see issue 29366 ).

I started with Istio 1.4.4 and always I stomped on authentication.

Everytime things messed up, I started again from scratch… and Istio changed version.
With Istio 1.6.3 I stomped on istio-ingressgateway not restarting if authentication fails (see 25578 )

I’m not sure; but seems that to use an external keycloak server with certificates signed by a private certification authority you have to:

  • install Istio using istio operator
  • during the installation set jwksResolverExtraRootCA

I expect that this will not work; but I’ll have to try and then report.
Unfortunately things that always worked installing Istio with istioctl now seems that need some type of configuration.

I’ve found a solution to the #29366 issue.
Now I’ll try your suggestion.