Istio components and gateway are not getting installed on a private GKE cluster

1

I’m trying to install Istio on a private GKE cluster. I have downloaded the version 1.4.3 of Istio and then applied the default profile. But not all the components are getting installed from the manifest. Below are the error logs.

$ istioctl manifest apply

This will install the default Istio profile into the cluster. Proceed? (y/N) y
- Applying manifest for component Base...
✔ Finished applying manifest for component Base.
- Applying manifest for component Pilot...
- Applying manifest for component Policy...
- Applying manifest for component Galley...
- Applying manifest for component Citadel...
- Applying manifest for component IngressGateway...
- Applying manifest for component Prometheus...
- Applying manifest for component Telemetry...
- Applying manifest for component Injector...
✘ Finished applying manifest for component Pilot.
✘ Finished applying manifest for component Telemetry.
✔ Finished applying manifest for component Prometheus.
✔ Finished applying manifest for component Citadel.
✔ Finished applying manifest for component Galley.
✔ Finished applying manifest for component Policy.
✔ Finished applying manifest for component Injector.
✔ Finished applying manifest for component IngressGateway.

Component Pilot - manifest apply returned the following errors:
Error: error running kubectl: signal: killed


Component Kiali - manifest apply returned the following errors:
Error: error running kubectl: exit status 1

Error detail:

Unable to connect to the server: dial tcp 192.168.0.2:443: i/o timeout (repeated 1 times)


apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""



Component Grafana - manifest apply returned the following errors:
Error: error running kubectl: exit status 1

Error detail:

Unable to connect to the server: dial tcp 192.168.0.2:443: i/o timeout (repeated 1 times)


apiVersion: v1
items: []
kind: List
metadata:
  resourceVersion: ""
  selfLink: ""



Component Telemetry - manifest apply returned the following errors:
Error: error running kubectl: exit status 1

Error detail:

Unable to connect to the server: net/http: request canceled (Client.Timeout exceeded while awaiting headers) (repeated 1 times)






✘ Errors were logged during apply operation. Please check component installation logs above.

Failed to generate and apply manifests, error: errors were logged during apply operation

Also, the ingress gateway is not getting created from any of the sample applications (helloworld, bookinfo). Below is the error:

$ kubectl apply -f samples/bookinfo/networking/bookinfo-gateway.yaml

Error from server (Timeout): error when creating "samples/bookinfo/networking/bookinfo-gateway.yaml": Timeout: request did not complete within requested timeout 30s
Error from server (Timeout): error when creating "samples/bookinfo/networking/bookinfo-gateway.yaml": Timeout: request did not complete within requested timeout 30s

However, I tried to use istio along with GKE on the same private cluster by following the guide here.

This worked and all the components are installed successfully, along with the ingress gateway.
I have enabled the ports 80,8080,1000-2000,22,443,9443 on the network as well.
Can please someone tell us what would be the issue causing this error.

Kubernetes version:

Client Version: version.Info{Major:"1", Minor:"17", GitVersion:"v1.17.3", GitCommit:"06ad960bfd03b39c8310aaf92d1e7c12ce618213", GitTreeState:"clean", BuildDate:"2020-02-11T18:14:22Z", GoVersion:"go1.13.6", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"13+", GitVersion:"v1.13.12-gke.25", GitCommit:"654de8cac69f1fc5db6f2de0b88d6d027bc15828", GitTreeState:"clean", BuildDate:"2020-01-14T06:01:20Z", GoVersion:"go1.12.11b4", Compiler:"gc", Platform:"linux/amd64"}

Istio version:

client version: 1.4.3
control plane version: 1.4.3
data plane version: 1.4.3 (1 proxies)

Platform: GKE

OS: Ubuntu

2

I tried to get a private GKE cluster going with Istio 1.4.5 and ran into similar problems. As context, I’ve been running the same configuration with Istio 1.3.3, and I’m just trying to upgrade. I figured using istioctl was the way to go, since helm installation is deprecated.

For the timeout error, I found it was necessary to create a firewall rule enabling the master nodes to access the worker nodes on port 9443, as discovered and well documented by a very helpful Stack Overflow user. That allowed me to configure an Istio Gateway resource and put a couple pods in the cluster.

At this point I can talk between pods, but I can’t reach into the cluster from outside. Not from a Global Load Balancer health check, and not from a bastion VM in the same project and network as the GKE cluster.

Next thing I noticed was that if you use NodePort for your Ingress, the HTTP node port has moved from 31380 to 30636, and I couldn’t find a way to change it back with istioctl. Fine, I can change my firewall rules etc to match. Still can’t get anything out of the cluster, though.

So back to helm, and everything works. Is anyone out there successful at getting Istio installed via istioctl on a private GKE cluster? It’s feeling a little untested to me.

1 Like
3

Thanks for pointing out the firewall issue with port 9443. It indeed worked and I’m able to install Istio 1.4.5 as well on a private GKE cluster.
And regarding your issue where you are not able to reach the cluster, have you configured a NAT for the cluster? Also, please check if you have allowed the required IPs to access the kubernetes cluster master in the master authorized networks.
More details can be found here:
https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters
https://cloud.google.com/nat/docs/gke-example
https://cloud.google.com/kubernetes-engine/docs/concepts/private-cluster-concept

4

Pretty sure it’s none of that.

  1. NAT helps process deployed to the cluster make outbound requests, which I don’t need.
  2. The master can talk to the worker nodes, or else it wouldn’t be possible to deploy Istio at all. It uses a custom namespace, but Istio’s processes are definitely deployed to the cluster. Also I am able to kubectl apply services into the cluster, and make http requests between pods. None of that would install without master access.

If I set up my cluster with helm, everything works fine. With istioctl nothing is reachable, even from a GCE VM in the same network and using wget. Same with 1.4.5 and 1.4.6. It’s not worth a lot of debugging unless I can verify the same problem with 1.5, I guess.