Istio 1.9.1 - Kiali don't connect to Grafana and Jaeger

MirtoBusico · March 8, 2021, 4:45pm

Hi all,
I have a Istio 1.9.1 installed using istio operator.
When I issue

istioctl d kiali

The kiali console says
grafana Unreachable
jaeger Unreachable

I installed the addons using:

kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/addons/kiali.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/addons/prometheus.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/addons/grafana.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/addons/jaeger.yaml
kubectl apply -f https://raw.githubusercontent.com/istio/istio/release-1.9/samples/addons/extras/zipkin.yaml

Then I shutted down the cluster and then rebooted

Kubecl for istio-system namespace says

sysop@m-serv:~$ kubectl get all -n istio-system
NAME                                        READY   STATUS    RESTARTS   AGE
pod/jaeger-7f78b6fb65-xk2xg                 1/1     Running   1          4h30m
pod/grafana-784c89f4cf-4hmxv                1/1     Running   1          4h33m
pod/zipkin-7fcd647cf9-7m5wg                 1/1     Running   1          4h22m
pod/istiod-d4f7d664f-mpxcb                  1/1     Running   1          4h59m
pod/istio-egressgateway-bd477794-wzmk5      1/1     Running   1          4h59m
pod/istio-ingressgateway-79df7c789f-22d4z   1/1     Running   1          4h59m
pod/prometheus-7bfddb8dbf-s6dl5             2/2     Running   2          4h36m
pod/svclb-istio-ingressgateway-4mdtv        5/5     Running   5          4h59m
pod/svclb-istio-ingressgateway-xtj7n        5/5     Running   5          4h59m
pod/kiali-dc84967d9-bzgvj                   1/1     Running   1          4h43m
pod/svclb-istio-ingressgateway-cs5h5        5/5     Running   5          4h59m

NAME                           TYPE           CLUSTER-IP      EXTERNAL-IP                                       PORT(S)                                                                      AGE
service/istiod                 ClusterIP      10.43.181.186   <none>                                            15010/TCP,15012/TCP,443/TCP,15014/TCP                                        4h59m
service/istio-egressgateway    ClusterIP      10.43.44.211    <none>                                            80/TCP,443/TCP,15443/TCP                                                     4h59m
service/kiali                  ClusterIP      10.43.58.65     <none>                                            20001/TCP,9090/TCP                                                           4h43m
service/prometheus             ClusterIP      10.43.137.192   <none>                                            9090/TCP                                                                     4h36m
service/grafana                ClusterIP      10.43.210.188   <none>                                            3000/TCP                                                                     4h33m
service/jaeger-collector       ClusterIP      10.43.199.121   <none>                                            14268/TCP,14250/TCP                                                          4h30m
service/tracing                ClusterIP      10.43.138.19    <none>                                            80/TCP                                                                       4h30m
service/zipkin                 ClusterIP      10.43.4.23      <none>                                            9411/TCP                                                                     4h30m
service/istio-ingressgateway   LoadBalancer   10.43.172.16    192.168.202.101,192.168.202.102,192.168.202.103   15021:30380/TCP,80:31104/TCP,443:31620/TCP,31400:32241/TCP,15443:32064/TCP   4h59m

NAME                                        DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/svclb-istio-ingressgateway   3         3         3       3            3           <none>          4h59m

NAME                                   READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/jaeger                 1/1     1            1           4h30m
deployment.apps/grafana                1/1     1            1           4h33m
deployment.apps/zipkin                 1/1     1            1           4h22m
deployment.apps/istiod                 1/1     1            1           4h59m
deployment.apps/istio-ingressgateway   1/1     1            1           4h59m
deployment.apps/istio-egressgateway    1/1     1            1           4h59m
deployment.apps/prometheus             1/1     1            1           4h36m
deployment.apps/kiali                  1/1     1            1           4h43m

NAME                                              DESIRED   CURRENT   READY   AGE
replicaset.apps/jaeger-7f78b6fb65                 1         1         1       4h30m
replicaset.apps/grafana-784c89f4cf                1         1         1       4h33m
replicaset.apps/zipkin-7fcd647cf9                 1         1         1       4h22m
replicaset.apps/istiod-d4f7d664f                  1         1         1       4h59m
replicaset.apps/istio-egressgateway-bd477794      1         1         1       4h59m
replicaset.apps/istio-ingressgateway-79df7c789f   1         1         1       4h59m
replicaset.apps/prometheus-7bfddb8dbf             1         1         1       4h36m
replicaset.apps/kiali-dc84967d9                   1         1         1       4h43m
sysop@m-serv:~$

“istioctl d grafana” works correctly
But jaeger dashboard says

sysop@m-serv:~/software/kubernetes$ istioctl d jaeger
http://localhost:16686
2021-03-08T16:42:22.647224Z	error	klog	error copying from remote stream to local connection: readfrom tcp4 127.0.0.1:16686->127.0.0.1:52022: write tcp4 127.0.0.1:16686->127.0.0.1:52022: write: broken pipe

What I’m doing wrong?

lucas.ponce · March 9, 2021, 7:45am

Hi,

If you use the operator to install Istio, probably you may need to personalize your Kiali installation telling where is located the prometheus and jaeger services.

To run this you can personalize the helm charts, or better, use the Kiali operator, which is designed and recommended for these kind of work.

https://kiali.io/documentation/latest/installation-guide/#_install_kiali_latest

Also, Christian Postal prepared some steps for combining operators in Istio and Kiali as well:

github.com

christian-posta/workshops/blob/d164af8b9300b960acf5006a8a21fb35462cde28/istio-day2/1-understand-istio/03-observability.md

# Lab 3 :: Connecting to observability systems

One of the most powerful parts of Istio is using the mesh to quickly troubleshoot and diagnose issues that inevitably come up in microservices networking. Where are requests slowing down? Where are they failing? Where are things becoming overloaded? Having something like Envoy proxy in the request path on both sides of a transport between services acts like a flood light to help uncover these issues.

We saw in Lab 01 that Envoy surfaces a large set of stats, metrics, guages, and histograms. In this lab we look at connecting that source of information to an observability system. In many ways the mesh makes these systems more valuable.

Istio comes with a few [sample addons](https://istio.io/latest/docs/ops/integrations/prometheus/#option-1-quick-start) for adding observability components like Prometheus, Grafana, and Kiali. In this lab **we will assume you are not going to use these components** as they are intended for very simple deployments and **not intended for a realistic production setup**. We will install something that looks more realistic and use that.

## Prerequisites

Verify you're in the correct folder for this lab: `/home/solo/workshops/istio-day2/1-understand-istio/`. 

We will be using a realistic observability system that uses Prometheus and many other components out of the box called [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus). This project tries to curate and pre-integrate a realistic deployment, highly available deployment of Prometheus with the Prometheus operator, Grafana, and a lot of ancillary pieces like alertmanager, node-exporters, adapters for the Kube API, and others. Please see the [kube-prometheus](https://github.com/prometheus-operator/kube-prometheus) docs for more.

We will be using the [helm](https://github.com/prometheus-community/helm-charts/tree/main/charts/kube-prometheus-stack) charts to install `kube-prometheus`. Just note, for this lab we will slightly trim down the number of components installed to stay within some of our VM resource limits (specifically for the instructor-led labs.. on your own cluster, carry on :) ). 

## Install kube-prometheus and highly-available Prometheus

To get started, let's create the namespace into which we'll deploy our observability components:

This file has been truncated. show original

But please, tell us your details if using the operator doesn’t work for you.

We’ll look if there is any step missing in the doc.

Thanks !

P.S.: Feel free to join into the #kiali room in Istio Slack and/or Freenode IRC where probably other colleagues can help you out.

MirtoBusico · March 9, 2021, 11:40am

Thanks Lucas,
I’ll try asap.

TL;DR

I’m trying to use operator because it seems the only method to set the jwksResolverExtraRootCA (see issue 29366 ).

I started with Istio 1.4.4 and always I stomped on authentication.

Everytime things messed up, I started again from scratch… and Istio changed version.
With Istio 1.6.3 I stomped on istio-ingressgateway not restarting if authentication fails (see 25578 )

I’m not sure; but seems that to use an external keycloak server with certificates signed by a private certification authority you have to:

install Istio using istio operator
during the installation set jwksResolverExtraRootCA

I expect that this will not work; but I’ll have to try and then report.
Unfortunately things that always worked installing Istio with istioctl now seems that need some type of configuration.

MirtoBusico · March 23, 2021, 10:30am

I’ve found a solution to the #29366 issue.
Now I’ll try your suggestion.

Topic		Replies	Views
Jaeger url not valid in Kiali configuration Kiali	9	7370	February 19, 2020
Kiali istio Component status masthead Policies and Telemetry	8	1995	June 18, 2020
Problems when i am trying add components kiali, grafana and prometheus in Istio version 18.0 Config	8	1519	May 7, 2021
Kaili integration with Garafana URL Istio 1.14.1 is not working Kiali	3	394	April 27, 2023
Runtimes discovery failed, cannot load dashboards in namespace istio-system. Error was: the server could not find the requested resource (get monitoringdashboards.monitoring.kiali.io	3	2578	September 6, 2019

Istio 1.9.1 - Kiali don't connect to Grafana and Jaeger

Related topics