Hi, I’ve installed a self managed cluster in AWS with kubeadm on Ubuntu 18.04. Ive got a 2 node cluster up and running with the the Calico CNI. I recently installed Istio following this this guide - Istio / Getting Started which worked and i managed to expose Kiali via Istio Ingress and all was good. I then realised that my worker node needed more CPU’s and Memory so i killed the worker node and rebuilt it on a larger ec2 instance. The cluster was restored and stable.
I’ve installed Istio again following the same guide as above but my install fails/times out when installing the Istio gateway. If I look at the istio gateway pod I can see the following error:
Readiness probe failed: Get “http://10.100.0.74:15021/healthz/ready”: dial tcp 10.100.0.74:15021: connect: connection refused
and the following response is returned to the terminal:
Ingress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition Deployment/istio-system/istio-ingressgateway
The AWS ELB which is instantiated by the install process for the istio ingress gateway has a keep alive that also shows ‘out of service’.
I’m struggling to find out what’s going on or what I’ve done differently.
Anyone got any suggestions?
Cheers
Garry
Some more information. I realised my docker and kube components were different version between the master and node. I’ve fixed that but the problem still exists.
I’ve looked into the istio-ingressgateway service which is deployed as part of the Istio install and i can see the EndPoints are empty as per below. The POD associated with the istio-ingressgateway is 10.100.0.66. Should this be listed in the output below? I’ve not got a working config to compare.
Cheers
kubectl describe svc istio-ingressgateway -n istio-system
Name: istio-ingressgateway
Namespace: istio-system
Labels: app=istio-ingressgateway
install.operator.istio.io/owning-resource=unknown
install.operator.istio.io/owning-resource-namespace=istio-system
istio=ingressgateway
istio.io/rev=default
operator.istio.io/component=IngressGateways
operator.istio.io/managed=Reconcile
operator.istio.io/version=1.9.4
release=istio
Annotations: <none>
Selector: app=istio-ingressgateway,istio=ingressgateway
Type: LoadBalancer
IP Families: <none>
IP: 10.99.46.126
IPs: 10.99.46.126
LoadBalancer Ingress: a0b2836186bb34cd4bcc0e3eac2317f1-1808881596.eu-west-1.elb.amazonaws.com
Port: status-port 15021/TCP
TargetPort: 15021/TCP
NodePort: status-port 31084/TCP
Endpoints:
Port: http2 80/TCP
TargetPort: 8080/TCP
NodePort: http2 32300/TCP
Endpoints:
Port: https 443/TCP
TargetPort: 8443/TCP
NodePort: https 30832/TCP
Endpoints:
Port: tcp-istiod 15012/TCP
TargetPort: 15012/TCP
NodePort: tcp-istiod 32261/TCP
Endpoints:
Port: tls 15443/TCP
TargetPort: 15443/TCP
NodePort: tls 30707/TCP
Endpoints:
Session Affinity: None
External Traffic Policy: Cluster
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal EnsuringLoadBalancer 8m44s service-controller Ensuring load balancer
Normal EnsuredLoadBalancer 8m42s service-controller Ensured load balancer
For the record i’ve fixed this. I had to ensure that Source / Destination Check was disabled on the ec2 instance. I forgot to do this when I rebuilt the Worker Node. I think lining up the component versions helped too.