Kiali does not connect to non istio-system namespace prometheus/grafana

Policies and Telemetry Kiali
1

I am trying to configure kiali v1.9.1 started using istio deployments to use our external prometheus and grafana instances.

I have modified the kiali configmap so that the loaded config looks like the following:

cat config.yaml
istio_namespace: fed-istio
deployment:
  accessible_namespaces: ['**']
auth:
  strategy: login
server:
  port: 20001
  web_root: /kiali
external_services:
  tracing:
    url: http://10.15.10.211:16686
    in_cluster_url: http://10.15.10.211:16686
  grafana:
    enabled: true
    url: http://10.15.12.31:3000
    in_cluster_url: http://10.15.12.31:3000
    service: "grafana"
    service_namespace: "fed-grafana"
    auth:
      username: "admin"
      password: "admin"
  prometheus:
    url: http://10.15.12.31:9090
    custom_metrics_url: http://10.15.12.31:9090

I still see this a client_error in the kiali logs,

~ kubectl logs -f kiali-1-0-0-109-main-int-dbg-74bfd545fb-ffbpc
I0319 17:16:39.970246       1 kiali.go:66] Kiali: Version: v1.9.1, Commit: 2431f9b8844202e27ec1c92f404fe7485253ffec
I0319 17:16:39.970828       1 kiali.go:210] Using authentication strategy [login]
I0319 17:16:39.970870       1 kiali.go:87] Kiali: Console version: 1.9.1
I0319 17:16:39.970875       1 kiali.go:272] Updating base URL in index.html with [/kiali]
I0319 17:16:39.971118       1 kiali.go:253] Generating env.js from config
I0319 17:16:39.997325       1 server.go:57] Server endpoint will start at [:20001/kiali]
I0319 17:16:39.997343       1 server.go:58] Server endpoint will serve static content from [/opt/kiali/console]
I0319 17:16:39.997350       1 metrics_server.go:18] Starting Metrics Server on [:9090]
I0319 17:16:39.997522       1 kiali.go:142] Secret is now available.
E0319 17:17:05.272883       1 workloads.go:570] Workload fed-istio-pod-redis-server is not found as StatefulSet
E0319 17:17:05.289635       1 workloads.go:570] Workload alertmanager-prometheus-alerts is not found as StatefulSet
E0319 17:17:05.289658       1 workloads.go:570] Workload prometheus-prometheus is not found as StatefulSet
E0319 17:17:05.310482       1 workloads.go:570] Workload fed-service-reg-pod-service-reg-server is not found as StatefulSet
E0319 17:17:13.008165       1 config.go:103] Failed to fetch Prometheus configuration: client_error: client error: 404
E0319 17:17:13.009351       1 config.go:103] Failed to fetch Prometheus flags: client_error: client error: 404
E0319 17:17:13.584961       1 workloads.go:570] Workload alertmanager-prometheus-alerts is not found as StatefulSet
E0319 17:17:13.584985       1 workloads.go:570] Workload prometheus-prometheus is not found as StatefulSet
E0319 17:17:13.585912       1 workloads.go:570] Workload fed-istio-pod-redis-server is not found as StatefulSet
E0319 17:17:13.604050       1 workloads.go:570] Workload fed-service-reg-pod-service-reg-server is not found as StatefulSet
E0319 17:17:47.608302       1 dashboards.go:305] runtimes discovery failed, cannot load dashboards in namespace fed-example. Error was: the server could not find the requested resource (get monitoringdashboards.monitoring.kiali.io)
2

Hi,
I think there is a Github Issue opened where some colleagues are commenting on this topic.
I put here the Kiali CR https://github.com/kiali/kiali/blob/bd87e06f5f0d60902a1028b635b111b82d4b2bb9/operator/deploy/kiali/kiali_cr.yaml which contains additional detailed information about the config.

Note that in general, Kiali needs two usecases:

  • First, Kiali pod need to access to these services internally - within the cluster- , to collect info, apply some logic (build dashboards, filters, etc).
  • Second, in order to link Kiali UI with Tracing/Grafana, Kiali needs to know how these services are exposed from the cluster to the external traffic.

Hope this helps.
Lucas

3

Hey Lucas. Thanks for the response. The kiali_cr definitely helps understanding the config option (though we are using just the helm variant of Kiali, however there is some overlap in the options).

The second use case is what I am trying to acheive but cannot figure out which config flags to set. No matter what I do it doesn’t seem to address the issue of not being able to connect to the services.

I tried using the pod Ip and had some success but that is a simulation and not really feasible as the services could be external to our cluster (deployed in a secondary cluster)