Error with istio-system port-forward svc to kiali ( Unable to listen on port 20001)

1

Hi Everyone,

I have istio version 1.9 installed. I have deployed kiali and I am trying to access kiali, but when I try to forward it gives me an error.

How can I configure it, will you help me please?

type → kiali is → clusterIP
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S)
kiali ClusterIP 1.x.x.x 20001/TCP,9090/TCP

./istioctl d kiali → error
Unable to listen on port 20001: Listeners failed to create with the following errors: [unable to create listener: Error listen tcp4 127.0.0.1:20001: bind: address already in use unable to create listener: Error listen tcp6 [::1]:20001: socket: address family not supported by protocol]
http://localhost:46633/kiali
Failed to open browser; open http://localhost:46633/kiali in your browser.

kubectl -n istio-system port-forward svc/kiali 20001:20001 → error
Unable to listen on port 20001: Listeners failed to create with the following errors: [unable to create listener: Error listen tcp4 127.0.0.1:20001: bind: address already in use unable to create listener: Error listen tcp6 [::1]:20001: socket: address family not supported by protocol]
error: unable to listen on any of the requested ports: [{20001 20001}]

Many thanks

2

The error “bind: address already in use” is a common error, and it means you have something listening on port 20001 already. You must find out what is listening to port 20001 on localhost and kill it. Without killing whatever is listening on 20001, those commands you are trying to run will never work.

To confirm you already have something on your local host listening at port 20001, try this test (note this is just using the simple Linux utility “nc” - make sure you have that utility available to you, feel free to install it yourself if you do not have nc on your machine - but you probably do, most Linux machines have it by default).

  • Run the command: nc -l localhost 20001

If you see an error print out immediately that says something like “Ncat: bind to ::1:20001: Address already in use. QUITTING.” then my hypothesis is correct and you must find out what is listening to 20001 and kill it (it could be anything - you must investigate on your machine and find it and kill it).

3

Hi jmazzitelli,

Firstly. Thanks for helping. did what you told me and now this happens.
./istioctl dashboard kiali
http://localhost:20001/kiali

Failed to open browser; open http://localhost:20001/kiali in your browser.

I have tried setting it up with this, but I am not successful. would this be necessary?
or what did I need?

cat <<EOF | kubectl apply -f -

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
kind: Gateway
metadata:
metadata:
name: kiali-gateway
name: kiali-gateway
namespace: istio-system
spec:
servers:
selector:
istio: ingressgateway
servers:

  • port:
    mode: SIMPLE
    number: 443
    name: https-kiali
    protocol: HTTPS
    tls:
    apiVersion: networking.istio.io/v1alpha3
    mode: SIMPLE
    kind: VirtualService
    credentialName: telemetry-gw-cert
    hosts:
    • “kiali.${INGRESS_DOMAIN}”

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: kiali-vs
namespace: istio-system
spec:
hosts:

  • “kiali.${INGRESS_DOMAIN}”
    gateways:
  • kiali-gateway
    http:
    number: 20001
  • route:
    • destination:
      host: kiali
      port:
      number: 20001

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
name: kiali
namespace: istio-system
spec:
host: kiali
trafficPolicy:
tls:
mode: DISABLE

EOF
gateway.networking.istio.io/kiali-gateway created
virtualservice.networking.istio.io/kiali-vs created
destinationrule.networking.istio.io/kiali created

thanks for your help,

4

Failed to open browser; open http://localhost:20001/kiali in your browser.

Sounds like your system doesn’t have the necessary utilities to launch the browser from istioctl. You haven’t mentioned what your environment is - what operating system? What kubernetes cluster environment? You can see here that the operating system is what determines what utility istioctl will try to use to launch the browser - xdg-open on Linux (I’ll assume you are on Linux, though, again, you didn’t specify). So if you do not have xdg-open (or whatever utility your specific operating system needs) then you will get that error you see.

In that case, just start the browser on your machine manually and point it to the URL the istioctl output tells you to point to (open http://localhost:20001/kiali in your browser.) - did you point your browser to that URL? What did your browser say?

You should not need a Gateway if you are port-forwarding directly. But if you want to expose Kiali with a gateway, follow the full instructions on how to do that correctly - you can find the docs here: Istio / Remotely Accessing Telemetry Addons

5

Hi jmazzitelli,

I answer your questions. It is a kuberntes cluster environment with linux operating system, the problem is that I think I am not forwarding the ports well. Try this, but not with much success.

kubectl -n istio-system port-forward svc/kiali 20001:20001 &
[1] 49946

Forwarding from 127.0.0.1:20001 → 20001

curl http://127.0.0.1:20001/kiali

URL: GET http://127.0.0.1/kiali

Categoría: Uncategorized URLs

Motivo: UNKNOWN

Notificación: GATEWAY_TIMEOUT

I don’t know if I’m forwarding well. Sorry I have my doubts.I do it well ?

Thank you very much for your help.

kind Regards,

6

hi jmazzitelli,
good news ! . kiali already works, now what does not work for me is prometheus with the error, Could not fetch health: Error while fetching app health: Post “http: //prometheus.istio-system: 9090 / api / v1 / query”: dial tcp : lookup prometheus.istio-system on 10.xxx:x: no such host.

prometheus → type balancer.
How can i fix it ?

many many thanks,

7

Where do you see that error? Is that in the Kiali logs? If so, it sounds like you need to tell Kiali how to access Prometheus by properly configuring Kiali. The config for that is here: kiali-operator/kiali_cr.yaml at f82d0b8b18df160515d2d36330212ad635b85876 · kiali/kiali-operator · GitHub

8

Hi Jmazzitelli ,
I saw the error in kiali. The error is because I am trying to configure kiali so that it can use external prometheus and grafana from another cluster.
I already have prometheus and grafana running but in another cluster. So in order not to use several, I would like to configure kiali to take them from the other cluster. Can you help me configure it up?. its possible ?

Many many thanks,

9

All I can say is read the documentation in the example kiali_cr.yaml that I linked to earlier - those are all the config settings it supports. You’ll need to make sure Prometheus is exposed properly (so it is reachable from the Kiali pod) and configure Kiali appropriately. I’ve never deployed Prometheus in one cluster while Kiali is in a completely different cluster - I’ve never heard of such a deployment before. Usually Istio, Prometheus and Kiali are all in the same cluster (most times even in the same namespace in the same cluster). You will have to do some custom configuration to get all of that to work. Sorry, that’s all I know.

10

hi Jmazzitelli,
I’ve been investigating and I think the solution to my case would be to install istio-operator? , to be able to configure external grafana and prometheus?
I only have installed → samples/addons/kiali.yaml
But , Is there any way that I can display some .yaml file so that I can declare this data?. how to do it ?.
I think this is the solution to my problem. but I don’t know how to install it.

I need to know how I can do it, help me please .
for example:
external_services:
grafana:
in_cluster_url: ‘http://grafana.istio-system:3000’
url: ‘http://grafana-istio-system.apps.ocp4-kqe1.maistra.upshift.redhat.com’
prometheus:
in_cluster_url: ‘http://prometheus.istio-system:9090’
url: ‘http://prometheus-istio-system.apps.ocp4-kqe1.maistra.upshift.redhat.com’
tracing:
in_cluster_url: ‘http://tracing.istio-system/jaeger’
url: >-
http://jaeger-query-istio-system.apps.ocp4-kqe1.maistra.upshift.redhat.com
version: default

note ;
I did the Install with Istioctl installation, but I think my solution could be to install it with Istio Operator Install?
Many many thanks,

11

All I can say is: read the docs and become more familiar with Kubernetes, Istio, and Kiali.

For the option to install Kiali Server via Helm, see the Kiali Quick Start Guide section on Helm installation. For the option to install Kiali Server via Kiali Operator, see the Kiali Installation Guide on Kiali Operator option. Those may help. The Kiali CR docs explain the setting values, and the Kiali FAQ may provide some insight.

You are doing something I have never done – putting Kiali in a completely different cluster than where Istio is deployed. So I can’t help other than tell you to read the docs and configure your clusters so they expose the different services externally (Prometheus, Grafana, etc). Those Kiali external_services URLs must be the URLs that can connect the Kiali pod in one cluster to the Prometheus/Grafana/etc pod in the other cluster.

Side note: looking at the URLs you posted, they have “maistra” in them - are you actually using Maistra (the upstream version of OpenShift Service Mesh)?? Kiali installation is very different in Maistra compared to the Istio installation using istioctl. I suspect, though, you just found those on the internet somewhere as examples. In any event, if you are using Maistra (aka OpenShift Service Mesh) be aware Kiali install is very different and you should read the Maistra/OpenShift Service Mesh docs to know how to do it.

12

hi jmazzitelli.
Let’s see that I have explained myself wrong. I have istio and kiali on the same server, so I need to know how and where to configure external urls to be able to connect the kiali, prometheus and grafana pod through urls from other clusters.

I explain better and forgive me ;

  • I have a cluster 1, where I have istio and kiali installed.
  • on the other hand, I have in another cluster the services of prometheus and grafana.
    and them , what I want to configure is from cluster 1 the urls of grafana and prometheus of cluster2. Just the urls.
    -result :
    cluster 1, configured with istio and kiali with the urls of grafana and prometheus. So as not to have repeated monitoring. and then also to be able to monitor for example bookinfo locally. ¿ its possible ?

and forgive me if I did not know how to explain myself, thank you very much

Many many thanks

13

hi jmazzitelli.
can you help me please. I have reinstalled istio, I need to configure prometheus and grafana to collect external metrics, how do I configure it? . you help me ?

Errros:
Could not fetch health: Error while fetching app health: Post “http://prometheus.istio-system:9090/api/v1/query”: dial tcp: lookup prometheus.istio-system on x.x.x:x: no such host

istio components status:
grafana → Unreachable.
prometheus → Unreachable
custom dashboard → Unreachable

note , i know that Unreachable status means that Kiali hasn’t been succesfuly able to communicate with the component (Prometheus, Grafana) , but I don’t know how to solve it.

i believe that i need to personalize my Kiali installation telling where is located the prometheus and grafana services, but… how do i do it?

Many thanks for your patience !

Regards,

14

I do not know. As I said before, I have never installed addons in a place different from the default addons location, so I do not know how to do what you want to do.

You should start a new thread or ask in the #general room in Istio Slack. The subject of this thread is “error with istio-system port-forward svc to Kiali” and people who would know the answer probably will be ignoring this thread

As for the question: “i need to personalize my Kiali installation telling where is located the prometheus and grafana services, but… how do i do it?” - see these links for the settings you need to specify in your Kiali CR for Grafana and Prometheus - those personalize your Kiali installation to tell it where Prometheus and Grafana are located. If you know your cluster’s Prometheus and Grafana setup (which you should since you installed them), you should know the values you need. So read the comments and look at the settings there and it should be obvious what you need to set in order for Kiali to talk to Prometheus and Grafana

15

hi jmaziitelli,
I have managed to configure prometheus in istio, but when configuring the config.map of kiali the external services of grafana gives me unreachealble. But inside the pod if I curl if it arrives without problem. So I don’t know if I’m putting this part correctly.

Kiali → configmap,

its that correct ?

external_services:
custom_dashboards:
enabled: true
prometheus:
url: http://prometheus.cluster.local/
grafana:
url: http://grafana.cluster.local/
auth:
type: basic
use_kiali_token: false
username: user
password: password

Many thanks ,