Runtimes discovery failed, cannot load dashboards in namespace istio-system. Error was: the server could not find the requested resource (get monitoringdashboards.monitoring.kiali.io

1

I have used following command to install istio with kiali, grafana and jaeger.

helm upgrade istio install/kubernetes/helm/istio --install \

–namespace istio-system \

–set pilot.traceSampling=100.0 \

–set tracing.enabled=true \

–set tracing.jaeger.tag=1.14.0 \

–set kiali.enabled=true \

–set kiali.tag=v1.4.0 \

–set kiali.createDemoSecret=true \

–set kiali.dashboard.jaegerURL=http://localhost:16686 \

–set global.mtls.enabled=true \

–set grafana.enabled=true \

–set kiali.dashboard.grafanaURL=http://localhost:3000

with this command, I can access grafana and jaeger through port-forwarding technique from localhost but have some problem integrating with kiali.
kiali is showing some following errors which I can not figure out.
for grafana:
Get localhost:3000/api/search?query=Istio%20Service%20Dashboard: dial tcp 127.0.0.1:3000: connect: connection refused
for jaeger:
Get localhost:16686: dial tcp 127.0.0.1:16686: connect: connection refused

logs in the kiali workload:

doesn’t return a memquotas list
2019-09-04T11:26:44.700201207Z W0904 11:26:44.700082 1 istio_rules_service.go:114] Querying template got an error: istio-system doesn’t return a listentries list
2019-09-04T11:26:44.700899425Z W0904 11:26:44.700173 1 istio_rules_service.go:114] Querying adapter got an error: istio-system doesn’t return a zipkins list
2019-09-04T11:26:44.700922548Z W0904 11:26:44.700261 1 istio_rules_service.go:114] Querying template got an error: istio-system doesn’t return a quotas list
2019-09-04T11:26:44.700929327Z W0904 11:26:44.700356 1 istio_rules_service.go:114] Querying template got an error: istio-system doesn’t return a kuberneteses list
2019-09-04T11:26:44.700935631Z W0904 11:26:44.700410 1 istio_rules_service.go:114] Querying template got an error: istio-system doesn’t return a logentries list
2019-09-04T11:26:44.700941489Z W0904 11:26:44.700451 1 istio_rules_service.go:114] Querying adapter got an error: istio-system doesn’t return a cloudwatches list
2019-09-04T11:26:44.700946879Z W0904 11:26:44.700496 1 istio_rules_service.go:114] Querying adapter got an error: istio-system doesn’t return a kubernetesenvs list
2019-09-04T11:26:44.700952047Z W0904 11:26:44.700520 1 istio_rules_service.go:114] Querying adapter got an error: istio-system doesn’t return a noops list
2019-09-04T11:26:44.700956908Z W0904 11:26:44.700527 1 istio_rules_service.go:114] Querying adapter got an error: istio-system doesn’t return a bypasses list
2019-09-04T11:26:44.700962074Z W0904 11:26:44.700535 1 istio_rules_service.go:114] Querying adapter got an error: istio-system doesn’t return a prometheuses list
2019-09-04T11:26:44.700967305Z W0904 11:26:44.700553 1 istio_rules_service.go:114] Querying template got an error: istio-system doesn’t return a apikeys list
2019-09-04T11:26:44.70098493Z W0904 11:26:44.700601 1 istio_rules_service.go:114] Querying adapter got an error: istio-system doesn’t return a statsds list
2019-09-04T11:26:44.700990491Z W0904 11:26:44.700661 1 istio_rules_service.go:114] Querying adapter got an error: istio-system doesn’t return a stackdrivers list
2019-09-04T11:26:44.70099573Z W0904 11:26:44.700685 1 istio_rules_service.go:114] Querying adapter got an error: istio-system doesn’t return a circonuses list
2019-09-04T11:26:44.701000842Z W0904 11:26:44.700769 1 istio_rules_service.go:114] Querying adapter got an error: istio-system doesn’t return a dogstatsds list
2019-09-04T11:26:44.788972838Z E0904 11:26:44.784662 1 istio_config.go:73] sidecars.networking.istio.io is forbidden: User “system:serviceaccount:istio-system:kiali-service-account” cannot list resource “sidecars” in API group “networking.istio.io” in the namespace “istio-system”
2019-09-04T11:34:24.058643251Z E0904 11:34:24.055499 1 grafana.go:35] Get localhost:3000/api/search?query=Istio%20Service%20Dashboard: dial tcp 127.0.0.1:3000: connect: connection refused
2019-09-04T11:34:24.098578332Z I0904 11:34:24.098364 1 versions.go:168] DEBUG: Detected Istio version [root@9ad856b9-c627-11e9-abca-26bcb80ec4e0-docker.io/istio-1.2.5-d9e231eda0e163d0f3df0103546c7a06b72cc48d-dirty-Modified]
2019-09-04T11:34:24.140176547Z E0904 11:34:24.139756 1 jaeger.go:27] Get localhost:16686: dial tcp 127.0.0.1:16686: connect: connection refused
2019-09-04T11:35:42.450039702Z E0904 11:35:42.448166 1 dashboards.go:30] runtimes discovery failed, cannot load dashboards in namespace istio-system. Error was: the server could not find the requested resource (get monitoringdashboards.monitoring.kiali.io)

Can anybody help me out here, please?

2

Sorry for the delay to response.
I didn’t see it.
Please, could you tell us which version are you using of Istio and Kiali ?

I suspect if you install Kiali using the operator that should be gone:
https://www.kiali.io/documentation/getting-started/

cc @jotak

3

Yes, the errors about the dashboards suggest Kiali wasn’t successfully installed and a CRD seems to be missing (or maybe you tried to update over an existing installation?) in any case I would suggest as Lucas says, to install with the operator, I think it’s the easiest way to do.

Also, about port-forwarding to localhost for Jaeger and Grafana: you need to do some extra-config because the Kiali pod will not be able to resolve these URLs (“localhost” for Kiali is its own pod). In Kiali config you can set up in-cluster URL for Grafana as such: “in_cluster_url: http://grafana.istio-system:3000” (hence using kubernetes domain names system) , then Kiali will use this URL instead of the external one. This is documented in Kiali config: https://github.com/kiali/kiali/blob/d97ee0fbcfafc2d3b80c9d2e43667d7ae1a88c99/operator/deploy/kiali/kiali_cr.yaml#L261 .

4

PS: if you want to try with just installing what seems to be a missing CRD, you can grab that file here https://github.com/kiali/kiali/blob/d97ee0fbcfafc2d3b80c9d2e43667d7ae1a88c99/operator/deploy/crd.yaml and run kubectl apply -f crd.yaml on it

But you will still see some errors due to Kiali not being able to fetch Grafana & Jaeger, hence Grafana & Jaeger integrations will be disabled