Kiali not working: Empty graph

Policies and Telemetry Kiali
1

Hello everyone,

I am trying to deploy Kiali to a Kubernetes cluster that already has Istio installed. When I try the graphs in the Kiali UI, I only get “Empty graph”, despite traffic clearly flowing through my apps.

Istio is installed in the istio-system namespace, Kiali in the observability namespace. I am using the kiali-server helm chart with the following values:

istio_namespace: istio-system

auth:
  strategy: anonymous

deployment:
  ingress_enabled: false
  service_type: NodePort
  node_port: 30002

external_services:
  prometheus:
    url: http://prometheus-server.observability:80
  grafana:
    url: http://grafana.observability:80
  tracing:
    url: http://jaeger-query.observability:16686

Any idea?

2

First step is to look at the Kiali pod logs - look for any errors or warnings that look suspicious.

3

You will also want to ensure that Prometheus server is successfully scraping the Istio mesh applications and collecting telemetry. See this for more: Kiali: Service mesh observability and configuration

4

Hi @jmazzitelli ,

I did have a look but didn’t see any error. There are warnings, though:

2021-04-14T07:08:41Z INF Kiali: Version: v1.32.0, Commit: 166ffcc9edfc6fdd8c6ce8e74eb5670414b949f5

2021-04-14T07:08:41Z INF Using authentication strategy [anonymous]
2021-04-14T07:08:41Z WRN Kiali auth strategy is configured for anonymous access - users will not be authenticated.
2021-04-14T07:08:41Z INF Kiali: Console version: 1.32.0
2021-04-14T07:08:41Z INF Updating base URL in index.html with [/kiali]
2021-04-14T07:08:41Z INF Generating env.js from config
2021-04-14T07:08:41Z INF Server endpoint will start at [:20001/kiali]
2021-04-14T07:08:41Z INF Server endpoint will serve static content from [/opt/kiali/console]
2021-04-14T07:08:41Z INF Starting Metrics Server on [:9090]
2021-04-14T07:10:29Z INF [Prom Cache] Enabled
2021-04-14T07:10:29Z INF Kiali Cache is active for namespaces [.*]
2021-04-14T07:10:30Z INF Waiting for Kiali cache for [namespace: istio-system] to sync
2021-04-14T07:10:30Z WRN Kiali has not [istio-system] namespace cached. It is required to fetch Istio Status correctly
2021-04-14T07:10:30Z INF Kiali cache for [namespace: istio-system] started
2021-04-14T07:10:30Z INF Waiting for Kiali cache for [namespace: observability] to sync
2021-04-14T07:10:30Z INF Kiali cache for [namespace: observability] started
2021-04-14T07:10:30Z INF Waiting for Kiali cache for [namespace: default] to sync
2021-04-14T07:10:30Z INF Kiali cache for [namespace: default] started
2021-04-14T07:10:30Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:30Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:30Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:30Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:30Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:30Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:30Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:30Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:30Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:30Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:45Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:45Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:45Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:45Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:45Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:45Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:45Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:45Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:45Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:10:45Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:00Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:00Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:00Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:00Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:00Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:00Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:00Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:00Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:00Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:00Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:15Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:15Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:15Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:15Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:15Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:15Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:15Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:15Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:15Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:15Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:30Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:30Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:30Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:30Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:30Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:30Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:30Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:30Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:30Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:30Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
5

@jmazzitelli In Prometheus, I can see a lot of metrics starting with istio_*. Which metrics should I look for in particular?

Thanks for your help!

6

@jmazzitelli I went through the link you provided me with, and I can see that Prometheus is showing up the metrics Kiali is interested in, as far as I can tell.

Could it be because Kiali can’t connect to Prometheus?

7

@jmazzitelli I just checked and Kiali can connect to Prometheus:

$ k -n observability exec -it kiali-86f457bcf9-hsrzf -c kiali -- curl -v http://prometheus-server.observability:80
* Rebuilt URL to: http://prometheus-server.observability:80/
*   Trying 10.107.90.71...
* TCP_NODELAY set
* Connected to prometheus-server.observability (10.107.90.71) port 80 (#0)
> GET / HTTP/1.1
> Host: prometheus-server.observability
> User-Agent: curl/7.61.1
> Accept: */*
> 
< HTTP/1.1 302 Found
< content-type: text/html; charset=utf-8
< location: /graph
< date: Wed, 14 Apr 2021 09:27:56 GMT
< content-length: 29
< x-envoy-upstream-service-time: 1
< server: envoy
< 
<a href="/graph">Found</a>.

* Connection #0 to host prometheus-server.observability left intact
8

I tried to move Prometheus, Jaeger, Kiali and Grafana into the istio-system namespace, and that didn’t make any difference. Kiali is still showing me “Empty graph”.

9

@lucas.ponce @jshaughn Do you know what these errors in the logs mean? Would this affect the ability of Kiali to generate the graph ? I’ve never seen these warnings before:

2021-04-14T07:11:30Z WRN Workload prometheus-node-exporter of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
2021-04-14T07:11:30Z WRN Workload promtail of type DaemonSet has not a ReplicaSet as a child controller, it may need a revisit
10

DaemonSet didn’t have a good support in Kiali as controllers but we have recently fix that in upstream.

That warning in the context of Kiali means that there are Pods without a known controller (on this case DaemonSet).

That was an edge case in the early versions of Kiali that we hope it’s fixed now.

11

If you are not seeing a Kiali graph it is because Kiali is finding no telemetry. This can happen if:

  • Kiali is connecting to the wrong prometheus.
  • Istio is not scraping telemetry to prometheus.
  • You have selected incorrect namespaces.
  • There is no telemetry for your selected time period.

That last one catches people, if you are looking at say the last N minute of telemetry, and you have not generated traffic in last N minutes, you will have nothing for the graph.

12

This is an excellent, pithy checklist that should go in kiali.io FAQ :wink: