Configure Kiali with your own Grafana Instance

rberna · March 20, 2020, 8:21pm

Hello,

I’m wondering if it is possible to integrate your own Grafana instance into the Kiali Dashboard. I’ve read that it is, but when I enable Grafana in external_services section of my kiali operator yaml file and provide the url to my grafana instance, I’m not seeing any reference to grafana at all in the kiali dashboard.

external_services:
** grafana:**
** enabled: true**
** url: ‘https://my-grafana.com’**

Am I missing something? Any help with this would be greatly appreciated.

jmazzitelli · March 20, 2020, 8:46pm

You didn’t say what version of Istio or Kiali you have. But Istio 1.5 has the following two settings that get passed to Kiali - maybe the comments help?:

github.com

istio/istio/blob/9942cd4e534c94400c31d47dd60e4c644b8aa408/manifests/istio-telemetry/kiali/values.yaml#L78-L79


grafanaURL: "" # If you have Grafana installed and it is accessible to client browsers, then set this to its external URL. Kiali will redirect users to this URL when Grafana metrics are to be shown.
grafanaInClusterURL: "http://grafana:3000" # In Kubernetes cluster with ELB in front this option is needed, since public IP of ELB is not reachable from inside the cluster

If you installed Kiali via the Kiali Operator, you have these settings available to help configure your own Grafana:

github.com

kiali/kiali/blob/ff1786aaee5e790bce8beb7ded30c77d7cdba686/operator/deploy/kiali/kiali_cr.yaml#L380-L425


# **Grafana-specific settings:
# auth: authentication settings to connect to Grafana:
#   ca_file: The certificate authority file to use when accessing Grafana using https. An empty string means no extra
#       certificate authority file is used. Default is an empty string.
#   insecure_skip_verify: Set true to skip verifying certificate validity when Kiali contacts Grafana over https.
#   password: Password to be used when making requests to Grafana, for basic authentication. User only requires viewer permissions. May refer to a secret - see note above.
#   token: Token / API key to access Grafana, for token-based authentication. It only requires viewer permissions. May refer to a secret - see note above.
#   type: The type of authentication to use when contacting the server from the Kiali backend. Use "bearer" to send the
#       token to the Grafana server. Use "basic" to connect with username and password credentials. Use "none" to not use any authentication.
#       Default is "none"
#   use_kiali_token: When true and if auth.type is "bearer", the same OAuth token used for authentication in Kiali will be used for the API calls to Grafana,
#       and auth.token config is ignored then.
#   username: Username to be used when making requests to Grafana, for basic authentication. User only requires viewer permissions.
# dashboards: A list of Grafana dashboards that Kiali can link to. Each item contains:
#   name: The name of the dashboard in Grafana
#   variables:
#     app: The name of a variable that holds the app name, if used in that dashboard (else it must be omitted)
#     namespace: The name of a variable that holds the namespace, if used in that dashboard (else it must be omitted)
#     service: The name of a variable that holds the service name, if used in that dashboard (else it must be omitted)
#     workload: The name of a variable that holds the workload name, if used in that dashboard (else it must be omitted)

This file has been truncated. show original

rberna · March 20, 2020, 9:18pm

Thanks for the response. Using istio version 1.4.6 and kiali operator 1.15.0. In my operator I only have:
external_services:
** grafana:**
** enabled: true**
** url: ‘https://my-grafana.com’**

I looked at the Kiali configmap, and it is showing the URL that I configured, but during the creation of the operator, I don’t see the Grafana url environment variable being set:

"GRAFANA_URL= "

Will try implementing some of the settings to see if that works. If all is well, I should see a link to Grafana when viewing metrics correct?

jmazzitelli · March 20, 2020, 11:09pm

Yes, you’ll see a Grafana link on the metrics pages. @jotak would be able to give more details on that.

As for an environment variable - don’t expect to see any “GRAFANA_URL” env var set anywhere. The operator will put this value directly in the configmap from which the Kiali server will obtain the value.

jotak · March 23, 2020, 8:19am

As @jmazzitelli said. Let me know if it didn’t work, with Kiali logs to better figure out what happened.

rberna · March 23, 2020, 8:23pm

Hoping to try this out in the next day or two. Will keep you posted.

rberna · April 6, 2020, 8:50pm

So was able to try this (finally). Have my own Grafana instance setup. I can see the metrics on the Kiali Outbound and Inbound tabs, but not seeing any link to my Grafana instance in Kiali. The configuration in my kiali-cr.yaml file is:

external_services:
grafana:
auth:
ca_file: “”
insecure_skip_verify: true
password: “”
token: “”
type: “none”
use_kiali_token: false
username: “”
dashboards:
- name: “Istio Workload Dashboard”
enabled: true
in_cluster_url: “http://grafana.svc:3000”
url: “https://my.grafana.com/?orgId=1”

Is there something else I could be missing?

bethmage · April 7, 2020, 7:27am

Variables are missing!

     dashboards:
    - name: 'Istio Service Dashboard'
      variables:
        namespace: var-namespace
        service: var-service
    - name: 'Istio Workload Dashboard'
      variables:
        namespace: var-namespace
        workload: var-workload

jotak · April 7, 2020, 7:54am

Yes, as @bethmage says, there are missing variables, but perhaps it’s not the only problem.
Some debugging hints:

Make sure Grafana API & Istio dashboard is accessible:

curl https://my.grafana.com/api/search?query=Istio%20Workload%20Dashboard

You should have a result similar to:

[{"id":8,"uid":"UbsSZTDik","title":"Istio Workload Dashboard","uri":"db/istio-workload-dashboard","url":"/d/UbsSZTDik/istio-workload-dashboard","slug":"","type":"dash-db","tags":[],"isStarred":false,"folderId":1,"folderUid":"i5rjgqrZk","folderTitle":"istio","folderUrl":"/dashboards/f/i5rjgqrZk/istio"}]

If not, then perhaps it’s because you don’t have Istio dashboards. Since you run your own Grafana instance you must make sure to have installed on it the grafana dashboards that Istio provides (Kiali doesn’t do it on its own).

Next step would be to check the same from Kiali pod, using in_cluster_url, to make sure Kiali can reach Grafana. Run the following with your KIALI_POD name:

kubectl exec KIALI_POD -n istio-system -ti /bin/bash
curl http://grafana.svc:3000/api/search?query=Istio%20Workload%20Dashboard

You should see the same result as before. If not, then it’s probably the in_cluster_url that isn’t correct. (note that if it’s the case, you should also see some errors in Kiali logs).

Also, I’ve opened an issue on Kiali because currently, the generated links would be broken with “?orgId=1” is base URL: https://github.com/kiali/kiali/issues/2623 ; will fix asap, an easy one.

rberna · April 7, 2020, 3:36pm

Thanks for the replies. I pulled the istio-workload-dashboard.json dashboard from the Istio Grafana instance and imported it into my Grafana instance. I then tried the above recommendations. I added the namespace and service variables to my config:

auth:
ca_file: “”
insecure_skip_verify: true
password: “”
token: “”
type: “none”
use_kiali_token: false
username: “”
dashboards:
- name: “Istio Workload Dashboard”
variables:
namespace: “foo”
service: “my-service”
enabled: true
in_cluster_url: “http://grafana.svc:3000”
url: “https://grafana.com”

I am able to curl the dashboard:
curl https://grafana.com/api/search\?query\=istio+workload+dashboard

[{“id”:30,“uid”:“UbsSZTDik”,“title”:“Istio Workload Dashboard”,“uri”:“db/istio-workload-dashboard”,“url”:“/d/UbsSZTDik/istio-workload-dashboard”,“slug”:“”,“type”:“dash-db”,“tags”:,“isStarred”:false}]%

One difference I do see between what you posted and my results is that I don’t have any folder options. My Grafana version does not use folders.

I exec’d into the Kiali pod and was also able to curl Grafana using the internal url:
curl http://grafana.svc:3000/api/search?query=Istio%20Workload%20Dashboard

[{“id”:30,“uid”:“UbsSZTDik”,“title”:“Istio Workload Dashboard”,“uri”:“db/istio-workload-dashboard”,“url”:“/d/UbsSZTDik/istio-workload-dashboard”,“slug”:“”,“type”:“dash-db”,“tags”:,“isStarred”:false}]sh-4.2# %

But still when I go to the Inbound or Outbound Metrics tab in Kiali for the workload, there are no links to Grafana. I also don’t see any notifications in the message center that would indicate an issue and also don’t see in errors in my kiali pod logs. Is there anything else I could be missing?

jotak · April 8, 2020, 9:12am

You must be close to getting it working if there’s nothing bad in logs

The “variables” part in config map is wrong, actually, you shouldn’t change the default unless you have modified the grafana dashboards or created new ones.

The Istio Workload Dashboard will be linked from Workload metrics pages in Kiali, it needs to have “namespace” and “workload” variables, so you should put the default back (and don’t modify “var-namespace”: this is the query parameters that Grafana is expecting; Kiali will translate those into query params, such as “…/mydashboard?var-namespace=foo&var-workload=bar”, which Grafana will understand)
```
  - name: "Istio Workload Dashboard"
    variables:
      namespace: "var-namespace"
      workload: "var-workload"
```
The Istio Service Dashboard is the one linked from Service metrics pages in Kiali and it needs “namespace” and “service” variables:
```
  - name: "Istio Service Dashboard"
    variables:
      namespace: "var-namespace"
      service: "var-service"
```

As said, you can modify these settings only if you change the grafana dashboards or build new ones. We made it configurable to make Kiali more flexible with custom grafana dashboards. For instance you could create a new dashboard that aggregates metrics per “app”, and add it in Kiali config (“app” variable is also recognized by Kiali) :

    - name: Istio App Dashboard
      variables:
        app: var-app
        namespace: var-namespace

rberna · April 9, 2020, 6:11pm

That worked! Thanks so much for your help with this!!!

rberna · April 13, 2020, 8:09pm

Was able to get the Service and Workload Dashboards added and am seeing the links to Grafana on the respective Workloads and Services metrics tabs. I also added the dashboards for the Istio applications (pilot, citadel, galley, etc…). I’m not seeing any metrics for these applications in Kiali nor am I seeing the Grafana links on the application tab. I used the above configuration of:

name: Istio App Dashboard
variables:
app: var-app
namespace: var-namespace
Is there something else I could be missing? Also, I see that Istio has Istio Mesh and Performance dashboards. Is it possible to add Grafana links in Kiali for these Dashboards?

jotak · April 30, 2020, 7:10am

Hi @rberna

Sorry I missed your last message, seeing it only now.
About the “App” dashboard: I gave this as an example of custom dashboard that you could add, but you need to create it in Grafana first, did you do that? The “name” field in Kiali yaml must equal the name of a dashboard within Grafana.

About other provided dashboards such as Istio Mesh: today it’s not configured in Kiali but we could add links to them indeed. Example:

    - name: Istio Mesh Dashboard
      variables:
        namespace: ignored
        service: ignored
        workload: ignored
        app: ignored

However, these dashboards are not per service, like the other ones, so the use case is a little bit different and links from within Kiali were not intended for that per design (that’s why there are those variables / ignored in yaml above). I’ll start a discussion with kiali team to see if we could make that more good-looking.

Topic		Replies	Views
Kaili integration with Garafana URL Istio 1.14.1 is not working Kiali	3	389	April 27, 2023
Kiali not finding grafana	0	619	December 4, 2020
Istio 1.2.3 grafana URL is not set in Kiali configuration	5	6889	August 7, 2019
Kiali-operator v1.36.0 grafana istio-dashboards "view in Grafana" link is not available Policies and Telemetry	5	1024	September 1, 2021
Kiali does not connect to non istio-system namespace prometheus/grafana Kiali	2	4557	March 21, 2020

Configure Kiali with your own Grafana Instance

Related topics