Configure Kiali with your own Grafana Instance

Hello,

I’m wondering if it is possible to integrate your own Grafana instance into the Kiali Dashboard. I’ve read that it is, but when I enable Grafana in external_services section of my kiali operator yaml file and provide the url to my grafana instance, I’m not seeing any reference to grafana at all in the kiali dashboard.

external_services:
** grafana:**
** enabled: true**
** url: ‘https://my-grafana.com’**

Am I missing something? Any help with this would be greatly appreciated.

You didn’t say what version of Istio or Kiali you have. But Istio 1.5 has the following two settings that get passed to Kiali - maybe the comments help?:

If you installed Kiali via the Kiali Operator, you have these settings available to help configure your own Grafana:

Thanks for the response. Using istio version 1.4.6 and kiali operator 1.15.0. In my operator I only have:
external_services:
** grafana:**
** enabled: true**
** url: ‘https://my-grafana.com’**

I looked at the Kiali configmap, and it is showing the URL that I configured, but during the creation of the operator, I don’t see the Grafana url environment variable being set:

"GRAFANA_URL= "

Will try implementing some of the settings to see if that works. If all is well, I should see a link to Grafana when viewing metrics correct?

Yes, you’ll see a Grafana link on the metrics pages. @jotak would be able to give more details on that.

As for an environment variable - don’t expect to see any “GRAFANA_URL” env var set anywhere. The operator will put this value directly in the configmap from which the Kiali server will obtain the value.

As @jmazzitelli said. Let me know if it didn’t work, with Kiali logs to better figure out what happened.

Hoping to try this out in the next day or two. Will keep you posted.

So was able to try this (finally). Have my own Grafana instance setup. I can see the metrics on the Kiali Outbound and Inbound tabs, but not seeing any link to my Grafana instance in Kiali. The configuration in my kiali-cr.yaml file is:

external_services:
grafana:
auth:
ca_file: “”
insecure_skip_verify: true
password: “”
token: “”
type: “none”
use_kiali_token: false
username: “”
dashboards:
- name: “Istio Workload Dashboard”
enabled: true
in_cluster_url: “http://grafana.svc:3000
url: “https://my.grafana.com/?orgId=1

Is there something else I could be missing?

Variables are missing!

     dashboards:
    - name: 'Istio Service Dashboard'
      variables:
        namespace: var-namespace
        service: var-service
    - name: 'Istio Workload Dashboard'
      variables:
        namespace: var-namespace
        workload: var-workload

Yes, as @bethmage says, there are missing variables, but perhaps it’s not the only problem.
Some debugging hints:

  1. Make sure Grafana API & Istio dashboard is accessible:

    curl https://my.grafana.com/api/search?query=Istio%20Workload%20Dashboard

You should have a result similar to:

[{"id":8,"uid":"UbsSZTDik","title":"Istio Workload Dashboard","uri":"db/istio-workload-dashboard","url":"/d/UbsSZTDik/istio-workload-dashboard","slug":"","type":"dash-db","tags":[],"isStarred":false,"folderId":1,"folderUid":"i5rjgqrZk","folderTitle":"istio","folderUrl":"/dashboards/f/i5rjgqrZk/istio"}]

If not, then perhaps it’s because you don’t have Istio dashboards. Since you run your own Grafana instance you must make sure to have installed on it the grafana dashboards that Istio provides (Kiali doesn’t do it on its own).

Next step would be to check the same from Kiali pod, using in_cluster_url, to make sure Kiali can reach Grafana. Run the following with your KIALI_POD name:

kubectl exec KIALI_POD -n istio-system -ti /bin/bash
curl http://grafana.svc:3000/api/search?query=Istio%20Workload%20Dashboard

You should see the same result as before. If not, then it’s probably the in_cluster_url that isn’t correct. (note that if it’s the case, you should also see some errors in Kiali logs).

Also, I’ve opened an issue on Kiali because currently, the generated links would be broken with “?orgId=1” is base URL: https://github.com/kiali/kiali/issues/2623 ; will fix asap, an easy one.

Thanks for the replies. I pulled the istio-workload-dashboard.json dashboard from the Istio Grafana instance and imported it into my Grafana instance. I then tried the above recommendations. I added the namespace and service variables to my config:

auth:
ca_file: “”
insecure_skip_verify: true
password: “”
token: “”
type: “none”
use_kiali_token: false
username: “”
dashboards:
- name: “Istio Workload Dashboard”
variables:
namespace: “foo”
service: “my-service”
enabled: true
in_cluster_url: “http://grafana.svc:3000
url: “https://grafana.com

I am able to curl the dashboard:
curl https://grafana.com/api/search?query=istio+workload+dashboard

[{“id”:30,“uid”:“UbsSZTDik”,“title”:“Istio Workload Dashboard”,“uri”:“db/istio-workload-dashboard”,“url”:"/d/UbsSZTDik/istio-workload-dashboard",“slug”:"",“type”:“dash-db”,“tags”:,“isStarred”:false}]%

One difference I do see between what you posted and my results is that I don’t have any folder options. My Grafana version does not use folders.

I exec’d into the Kiali pod and was also able to curl Grafana using the internal url:
curl http://grafana.svc:3000/api/search?query=Istio%20Workload%20Dashboard

[{“id”:30,“uid”:“UbsSZTDik”,“title”:“Istio Workload Dashboard”,“uri”:“db/istio-workload-dashboard”,“url”:"/d/UbsSZTDik/istio-workload-dashboard",“slug”:"",“type”:“dash-db”,“tags”:,“isStarred”:false}]sh-4.2# %

But still when I go to the Inbound or Outbound Metrics tab in Kiali for the workload, there are no links to Grafana. I also don’t see any notifications in the message center that would indicate an issue and also don’t see in errors in my kiali pod logs. Is there anything else I could be missing?

You must be close to getting it working if there’s nothing bad in logs :slight_smile:

The “variables” part in config map is wrong, actually, you shouldn’t change the default unless you have modified the grafana dashboards or created new ones.

  • The Istio Workload Dashboard will be linked from Workload metrics pages in Kiali, it needs to have “namespace” and “workload” variables, so you should put the default back (and don’t modify “var-namespace”: this is the query parameters that Grafana is expecting; Kiali will translate those into query params, such as “…/mydashboard?var-namespace=foo&var-workload=bar”, which Grafana will understand)

      - name: "Istio Workload Dashboard"
        variables:
          namespace: "var-namespace"
          workload: "var-workload"
    
  • The Istio Service Dashboard is the one linked from Service metrics pages in Kiali and it needs “namespace” and “service” variables:

      - name: "Istio Service Dashboard"
        variables:
          namespace: "var-namespace"
          service: "var-service"
    

As said, you can modify these settings only if you change the grafana dashboards or build new ones. We made it configurable to make Kiali more flexible with custom grafana dashboards. For instance you could create a new dashboard that aggregates metrics per “app”, and add it in Kiali config (“app” variable is also recognized by Kiali) :

    - name: Istio App Dashboard
      variables:
        app: var-app
        namespace: var-namespace

That worked! Thanks so much for your help with this!!!

Was able to get the Service and Workload Dashboards added and am seeing the links to Grafana on the respective Workloads and Services metrics tabs. I also added the dashboards for the Istio applications (pilot, citadel, galley, etc…). I’m not seeing any metrics for these applications in Kiali nor am I seeing the Grafana links on the application tab. I used the above configuration of:

  • name: Istio App Dashboard
    variables:
    app: var-app
    namespace: var-namespace
    Is there something else I could be missing? Also, I see that Istio has Istio Mesh and Performance dashboards. Is it possible to add Grafana links in Kiali for these Dashboards?

Hi @rberna

Sorry I missed your last message, seeing it only now.
About the “App” dashboard: I gave this as an example of custom dashboard that you could add, but you need to create it in Grafana first, did you do that? The “name” field in Kiali yaml must equal the name of a dashboard within Grafana.

About other provided dashboards such as Istio Mesh: today it’s not configured in Kiali but we could add links to them indeed. Example:

    - name: Istio Mesh Dashboard
      variables:
        namespace: ignored
        service: ignored
        workload: ignored
        app: ignored

However, these dashboards are not per service, like the other ones, so the use case is a little bit different and links from within Kiali were not intended for that per design (that’s why there are those variables / ignored in yaml above). I’ll start a discussion with kiali team to see if we could make that more good-looking.