Jaeger url not valid in Kiali configuration

Policies and Telemetry Kiali
1

Hi all,
I’m using Istio 1.4.3 installed with the demo profile and Kiali gives this error in the home page:

Jaeger URL in cluster is not set in Kiali configuration

But in the Kiali config map I have:

{
	"config.yaml": "istio_component_namespaces:
		  grafana: istio-system
		  tracing: istio-system
		  pilot: istio-system
		  prometheus: istio-system
		istio_namespace: istio-system
		deployment:
		  accessible_namespaces: ['**']
		server:
		  port: 20001
		  web_root: /kiali
		external_services:
		  istio:
		    url_service_version: http://istio-pilot.istio-system:8080/version
		  tracing:
		    url: http://tracing:80
		  grafana:
		    url: http://graphana:3000
		  prometheus:
		    url: http://prometheus.istio-system:9090
		  jaeger:
		    url: http://tracing:80
		"
}

And I see these services:

sysop@hoseplak3s:~/software/istio-1.4.3$ sudo kubectl get svc -n istio-system
[sudo] password di sysop: 
NAME                     TYPE           CLUSTER-IP      EXTERNAL-IP      PORT(S)                                                                                                                      AGE
jaeger-agent             ClusterIP      None            <none>           5775/UDP,6831/UDP,6832/UDP                                                                                                   4d5h
jaeger-collector         ClusterIP      10.43.206.133   <none>           14267/TCP,14268/TCP,14250/TCP                                                                                                4d5h
jaeger-query             ClusterIP      10.43.82.190    <none>           16686/TCP                                                                                                                    4d5h
tracing                  ClusterIP      10.43.241.194   <none>           80/TCP                                                                                                                       4d5h
zipkin                   ClusterIP      10.43.13.172    <none>           9411/TCP                                                                                                                     4d5h
kiali                    ClusterIP      10.43.148.223   <none>           20001/TCP                                                                                                                    4d5h
istio-galley             ClusterIP      10.43.138.175   <none>           443/TCP,15014/TCP,9901/TCP,15019/TCP                                                                                         4d5h
prometheus               ClusterIP      10.43.67.79     <none>           9090/TCP                                                                                                                     4d5h
istio-sidecar-injector   ClusterIP      10.43.160.53    <none>           443/TCP                                                                                                                      4d5h
istio-citadel            ClusterIP      10.43.94.59     <none>           8060/TCP,15014/TCP                                                                                                           4d5h
grafana                  ClusterIP      10.43.35.24     <none>           3000/TCP                                                                                                                     4d5h
istio-policy             ClusterIP      10.43.131.225   <none>           9091/TCP,15004/TCP,15014/TCP                                                                                                 4d5h
istio-egressgateway      ClusterIP      10.43.60.152    <none>           80/TCP,443/TCP,15443/TCP                                                                                                     4d5h
istio-pilot              ClusterIP      10.43.194.84    <none>           15010/TCP,15011/TCP,8080/TCP,15014/TCP                                                                                       4d5h
istio-telemetry          ClusterIP      10.43.107.127   <none>           9091/TCP,15004/TCP,15014/TCP,42422/TCP                                                                                       4d5h
istio-ingressgateway     LoadBalancer   10.43.52.147    192.168.202.10   15020:30829/TCP,80:31047/TCP,443:31520/TCP,15029:30626/TCP,15030:32534/TCP,15031:30744/TCP,15032:31932/TCP,15443:30726/TCP   4d5h
sysop@hoseplak3s:~/software/istio-1.4.3$

The URL (corresponding to tracing service)

http://10.43.241.194/jaeger/search

shows the Jaeger UI home page

What I’m doing wrong?

2

That configmap data looks wrong to me. That should be under “data”:

$ kubectl get cm kiali -n istio-system -o json
{
    "apiVersion": "v1",
    "data": {
        "config.yaml": "api:\n  namespaces:\n   .... and the rest ...

Also, there is no “jaeger” item under external_services… (at least not in the newer versions, can’t remember if really old versions of Kiali had that)… but the name is “tracing”:

external_services:
  tracing:
    url: ....the url here...

Does any of this FAQ help? https://kiali.io/faq/distributed-tracing/

What does the About box say is the URL?

See the kiali_cr.yaml file for documentation on the external_services.tracing setting - https://github.com/kiali/kiali/blob/d62b4057b1ef2614b363afb343414a819af59828/operator/deploy/kiali/kiali_cr.yaml#L433-L470

Some work has gone into this tracing config in past releases, so make sure you are looking at the correct version of that example file for the version of Kiali you have. For example, here’s the v1.9 version.

3

Well, the Kiali version is

Kiali-ui
    1.9.1 (83c6a25f96163c26b4959d5b89ac67a6a2b52da9)
Kiali
    v1.9.1 (2431f9b8844202e27ec1c92f404fe7485253ffec)

Components 

Istio
    1.4.3 
Prometheus
    2.12.0 
Kubernetes
    v1.17.2+k3s1 
Grafana URL
    http://graphana:3000
Jaeger URL
    http://tracing:80

and seems that the jaeger url is correct; but the error persist:

Could not fetch Jaeger info. Turning off Jaeger integration.
Jaeger URL in cluster is not set in Kiali configuration
14/02/2020    11:27:14

Here is th kiali config map obtained with “sudo kubectl get cm kiali -n istio-system -o yaml >kiali-cm.yaml”

apiVersion: v1
data:
  config.yaml: |
    istio_component_namespaces:
      grafana: istio-system
      tracing: istio-system
      pilot: istio-system
      prometheus: istio-system
    istio_namespace: istio-system
    deployment:
      accessible_namespaces: ['**']
    server:
      port: 20001
      web_root: /kiali
    external_services:
      istio:
        url_service_version: http://istio-pilot.istio-system:8080/version
      tracing:
        url: http://tracing:80
      grafana:
        url: http://graphana:3000
      prometheus:
        url: http://prometheus.istio-system:9090
      jaeger:
        url: http://tracing:80
kind: ConfigMap
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"v1","data":{"config.yaml":"istio_component_namespaces:\n  grafana: istio-system\n  tracing: istio-system\n  pilot: istio-system\n  prometheus: istio-system\nistio_namespace: istio-system\ndeployment:\n  accessible_namespaces: ['**']\nserver:\n  port: 20001\n  web_root: /kiali\nexternal_services:\n  istio:\n    url_service_version: http://istio-pilot.istio-system:8080/version\n  tracing:\n    url: \n  grafana:\n    url: \n  prometheus:\n    url: http://prometheus.istio-system:9090\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app":"kiali","operator.istio.io/component":"Kiali","operator.istio.io/managed":"Reconcile","operator.istio.io/version":"1.4.3","release":"istio"},"name":"kiali","namespace":"istio-system"}}
  creationTimestamp: "2020-02-09T13:02:48Z"
  labels:
    app: kiali
    operator.istio.io/component: Kiali
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.4.3
    release: istio
  name: kiali
  namespace: istio-system
  resourceVersion: "50059"
  selfLink: /api/v1/namespaces/istio-system/configmaps/kiali
  uid: 1bfa4f56-fea2-4177-9c70-a9d30aa9dd3d

About the jaeger entry, it was only my attempt to see if was required an “jaeger” entry instead of a “tracing” entry.

Any hint?

4

Could not fetch Jaeger info. Turning off Jaeger integration.

There is an FAQ for that error message here: https://kiali.io/faq/distributed-tracing/#turn-off-jaeger-integration

It says:

Although Kiali discovered the Jaeger URL the signed up user 
doesn’t have access to the Jaeger query endpoint. Accessing
to the discovered URL (shown in the "About" modal) shouldn’t
be working. Once the endpoint will be accessible by the signed
up user, the Jaeger integration will be enabled in Kiali.

That’s about all I know… but there are other Kiali folks that would know more things to look into to fix this (@lucas.ponce - who would be the best person for this?)

5

Hi,

You’re missing the jaeger in cluster URL: Kiali needs it to talk to jaeger from its own pod. This is probably something like: http://tracing.istio-system/jaeger (assuming jaeger is in namespace istio-system, with service name ‘tracing’)

In yaml it gives something like:

  external_services:
    tracing:
      url: 'http://tracing:80'
      in_cluster_url: 'http://tracing.istio-system/jaeger'
6

Hi,

I think there is a pending pr for this

https://github.com/kiali/kiali.io/pull/203

The Kiali CR should be updated.

Alberto is back on Monday and can also add more details.

7

Hi,
a little step forward.
Also adding the in_cluster-url didn’t work; but I realized that copyng the urls in the browser did’nt work.
So I changed the tracing references using the IP address.
Now the external_service lines are:

server:
  port: 20001
  web_root: /kiali
external_services:
  istio:
    url_service_version: http://istio-pilot.istio-system:8080/version
  tracing:
    url: http://10.43.241.194:80
    in_cluster_url: 'http://10.43.241.194/jaeger'
  grafana:
    url: http://graphana:3000
  prometheus:
    url: http://prometheus.istio-system:9090

And now Kiali don’t show the error message

Seems that I cannot access the kubernetes dns.

My framework is:

  • Ubuntu + Mate desktop running in a KVM virtual machine
  • K3S kubernetes installation acting as master and worker
  • Istio 1.4.3 installed using istioctl

I run the browser inside the virtual machine. Maybe it is a resolv.conf problem

8

I can’t tell why Kubernetes DNS wouldn’t work, but indeed if it doesn’t, that would be a problem. However this is a bit weird because it seems to work for your prometheus URL, isn’t it? Or do you see no metric / no graph in Kiali, either?

9

Really it was only the first try to see if the Kiali error goes away.
Now to have everything work I have to use all IP addresses:

server:
  port: 20001
  web_root: /kiali
external_services:
  istio:
    url_service_version: http://10.43.194.84:8080/version
  tracing:
    url: http://10.43.241.194:80
    in_cluster_url: 'http://10.43.241.194/jaeger'
  grafana:
    url: http://10.43.35.24:3000
  prometheus:
    url: http://10.43.67.79:9090

Still I don’t know how to diagnose the dns server.

10

In a fresh istio 1.4.5 installation the kiali config map is:

kind: ConfigMap
apiVersion: v1
metadata:
  name: kiali
  namespace: istio-system
  selfLink: /api/v1/namespaces/istio-system/configmaps/kiali
  uid: e615c003-d308-463b-a7da-b98f459523bb
  resourceVersion: '6615'
  creationTimestamp: '2020-02-19T11:20:06Z'
  labels:
    app: kiali
    operator.istio.io/component: Kiali
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.4.5
    release: istio
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: >
      {"apiVersion":"v1","data":{"config.yaml":"istio_component_namespaces:\n 
      grafana: istio-system\n  tracing: istio-system\n  pilot: istio-system\n 
      prometheus: istio-system\nistio_namespace: istio-system\ndeployment:\n 
      accessible_namespaces: ['**']\nserver:\n  port: 20001\n  web_root:
      /kiali\nexternal_services:\n  istio:\n    url_service_version:
      http://istio-pilot.istio-system:8080/version\n  tracing:\n    url: \n 
      grafana:\n    url: \n  prometheus:\n    url:
      http://prometheus.istio-system:9090\n"},"kind":"ConfigMap","metadata":{"annotations":{},"labels":{"app":"kiali","operator.istio.io/component":"Kiali","operator.istio.io/managed":"Reconcile","operator.istio.io/version":"1.4.5","release":"istio"},"name":"kiali","namespace":"istio-system"}}
data:
  config.yaml: |
    istio_component_namespaces:
      grafana: istio-system
      tracing: istio-system
      pilot: istio-system
      prometheus: istio-system
    istio_namespace: istio-system
    deployment:
      accessible_namespaces: ['**']
    server:
      port: 20001
      web_root: /kiali
    external_services:
      istio:
        url_service_version: http://istio-pilot.istio-system:8080/version
      tracing:
        url: 
      grafana:
        url: 
      prometheus:
        url: http://prometheus.istio-system:9090

With these settings tracing and grafana urls are missing.
The prometheus url geves a “server not found” error.
All services are running and are accessible using their IP address.

Any hint?