I am trying to install istio on RHEL 7. Earlier on this machine istio used to get installed easily but now I have started facing the issue with new installation. istio-ingressgateway and istio-egressgateway are not starting below are some more details.
istio version: istio-1.19.1
minikube version: v1.31.2
docker ce: 24.0.6
Logs:
kubectl logs istio-ingressgateway-fcd9bc44-c6hzp -n istio-system
vailable desc = connection error: desc = “transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.0.4:39653->10.96.0.10:53: i/o timeout”
2023-10-05T06:56:34.050685Z warning envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:190 StreamAggregatedResources gRPC config stream to xds-grpc closed since 45483s ago: 14, connection error: desc = “transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout” thread=18
2023-10-05T06:57:05.949571Z warning envoy config external/envoy/source/extensions/config_subscription/grpc/grpc_stream.h:190 StreamAggregatedResources gRPC config stream to xds-grpc closed since 45515s ago: 14, connection error: desc = “transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc: i/o timeout” thread=18
2023-10-05T06:57:30.603529Z warn ca ca request failed, starting attempt 1 in 105.49937ms
2023-10-05T06:57:30.709971Z warn ca ca request failed, starting attempt 2 in 186.212321ms
2023-10-05T06:57:30.896372Z warn ca ca request failed, starting attempt 3 in 384.182969ms
2023-10-05T06:57:31.280821Z warn ca ca request failed, starting attempt 4 in 793.933477ms
2023-10-05T06:57:32.075938Z warn sds failed to warm certificate: failed to generate workload certificate: create certificate: rpc error: code = Unavailable desc = connection error: desc = “transport: Error while dialing: dial tcp: lookup istiod.istio-system.svc on 10.96.0.10:53: read udp 10.244.0.4:39653->10.96.0.10:53: i/o timeout”
kubectl logs istiod-759bb458bc-nh6gs -n istio-system
2023-10-04T18:17:57.514481Z info validationController Not ready to switch validation to fail-closed: dummy invalid config not rejected
2023-10-04T18:17:57.525393Z info validationController Successfully updated validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Ignore,resourceVersion=581)
2023-10-04T18:17:57.525431Z error controllers error handling istio-validator-istio-system, retrying (retry count: 1): webhook is not ready, retry controller=validation
2023-10-04T18:17:58.530169Z info validationController Not ready to switch validation to fail-closed: dummy invalid config not rejected
2023-10-04T18:17:58.530205Z info validationController validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Ignore, resourceVersion=581) is up-to-date. No change required.
2023-10-04T18:17:58.530216Z error controllers error handling istio-validator-istio-system, retrying (retry count: 2): webhook is not ready, retry controller=validation
2023-10-04T18:17:58.568895Z info ads Full push, new service istio-system/istiod.istio-system.svc.cluster.local
2023-10-04T18:17:58.669236Z info ads Push debounce stable[2] 1 for config ServiceEntry/istio-system/istiod.istio-system.svc.cluster.local: 100.276745ms since last change, 100.276466ms since last push, full=true
2023-10-04T18:17:58.669455Z info ads XDS: Pushing Services:3 ConnectedEndpoints:0 Version:2023-10-04T18:17:58Z/2
2023-10-04T18:17:59.537827Z info validationServer configuration is invalid: gateway must have at least one server (dry run)
2023-10-04T18:17:59.539358Z info validationController Endpoint successfully rejected invalid config. Switching to fail-close.
2023-10-04T18:17:59.544555Z info validationController Successfully updated validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Fail,resourceVersion=590)
2023-10-04T18:17:59.544628Z info validationController validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Fail, resourceVersion=590) is up-to-date. No change required.
2023-10-04T18:17:59.544658Z info validationController validatingwebhookconfiguration istio-validator-istio-system (failurePolicy=Fail, resourceVersion=590) is up-to-date. No change required.
2023-10-04T18:18:00.445893Z info ads Incremental push, service istio-egressgateway.istio-system.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-10-04T18:18:00.539721Z info ads Incremental push, service istio-ingressgateway.istio-system.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-10-04T18:18:00.640189Z info ads Push debounce stable[3] 4 for config ServiceEntry/istio-system/istio-egressgateway.istio-system.svc.cluster.local and 1 more configs: 100.410848ms since last change, 202.995786ms since last push, full=true
2023-10-04T18:18:00.640391Z info ads XDS: Pushing Services:5 ConnectedEndpoints:0 Version:2023-10-04T18:18:00Z/3
2023-10-04T18:18:08.546547Z info ads Incremental push, service istio-ingressgateway.istio-system.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-10-04T18:18:08.646897Z info ads Push debounce stable[4] 1 for config ServiceEntry/istio-system/istio-ingressgateway.istio-system.svc.cluster.local: 100.284524ms since last change, 100.284214ms since last push, full=false
2023-10-04T18:18:08.647033Z info ads XDS: Incremental Pushing ConnectedEndpoints:0 Version:2023-10-04T18:18:00Z/3
2023-10-04T18:18:10.558409Z info ads Incremental push, service istio-egressgateway.istio-system.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-10-04T18:18:10.658703Z info ads Push debounce stable[5] 1 for config ServiceEntry/istio-system/istio-egressgateway.istio-system.svc.cluster.local: 100.23181ms since last change, 100.231508ms since last push, full=false
2023-10-04T18:18:10.658779Z info ads XDS: Incremental Pushing ConnectedEndpoints:0 Version:2023-10-04T18:18:00Z/3
2023-10-04T18:36:47.214333Z info Sidecar injection request for /
2023-10-04T18:36:48.053591Z info ads Incremental push, service sleep.foo.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-10-04T18:36:48.154500Z info ads Push debounce stable[6] 2 for config ServiceEntry/foo/sleep.foo.svc.cluster.local: 100.837076ms since last change, 108.870362ms since last push, full=true
2023-10-04T18:36:48.154731Z info ads XDS: Pushing Services:6 ConnectedEndpoints:0 Version:2023-10-04T18:36:48Z/4
2023-10-04T18:36:49.819221Z info ads Incremental push, service sleep.foo.svc.cluster.local at shard Kubernetes/Kubernetes has no endpoints
2023-10-04T18:36:49.919583Z info ads Push debounce stable[7] 1 for config ServiceEntry/foo/sleep.foo.svc.cluster.local: 100.276264ms since last change, 100.276118ms since last push, full=false
2023-10-04T18:36:49.919677Z info ads XDS: Incremental Pushing ConnectedEndpoints:0 Version:2023-10-04T18:36:48Z/4
2023-10-04T19:02:35.701975Z info rootcertrotator Jitter complete, start rotator.
2023-10-04T20:02:35.702520Z info rootcertrotator Check and rotate root cert.
2023-10-04T20:02:35.705683Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
2023-10-04T21:02:35.702515Z info rootcertrotator Check and rotate root cert.
2023-10-04T21:02:35.705217Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
2023-10-04T22:02:35.702083Z info rootcertrotator Check and rotate root cert.
2023-10-04T22:02:35.705221Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
2023-10-04T23:02:35.702494Z info rootcertrotator Check and rotate root cert.
2023-10-04T23:02:35.705162Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
2023-10-05T00:02:35.702841Z info rootcertrotator Check and rotate root cert.
2023-10-05T00:02:35.706603Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
2023-10-05T01:02:35.702747Z info rootcertrotator Check and rotate root cert.
2023-10-05T01:02:35.705492Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
2023-10-05T02:02:35.702333Z info rootcertrotator Check and rotate root cert.
2023-10-05T02:02:35.705090Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
2023-10-05T03:02:35.702908Z info rootcertrotator Check and rotate root cert.
2023-10-05T03:02:35.705532Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
2023-10-05T04:02:35.702838Z info rootcertrotator Check and rotate root cert.
2023-10-05T04:02:35.706034Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
2023-10-05T05:02:35.702955Z info rootcertrotator Check and rotate root cert.
2023-10-05T05:02:35.706261Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
2023-10-05T06:02:35.702500Z info rootcertrotator Check and rotate root cert.
2023-10-05T06:02:35.705073Z info rootcertrotator Root cert is not about to expire, skipping root cert rotation.
minikube ssh
sudo iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
KUBE-PROXY-FIREWALL all – anywhere anywhere ctstate NEW /* kubernetes load balancer firewall /
KUBE-NODEPORTS all – anywhere anywhere / kubernetes health check service ports /
KUBE-EXTERNAL-SERVICES all – anywhere anywhere ctstate NEW / kubernetes externally-visible service portals */
KUBE-FIREWALL all – anywhere anywhere
Chain FORWARD (policy ACCEPT)
target prot opt source destination
KUBE-PROXY-FIREWALL all – anywhere anywhere ctstate NEW /* kubernetes load balancer firewall /
KUBE-FORWARD all – anywhere anywhere / kubernetes forwarding rules /
KUBE-SERVICES all – anywhere anywhere ctstate NEW / kubernetes service portals /
KUBE-EXTERNAL-SERVICES all – anywhere anywhere ctstate NEW / kubernetes externally-visible service portals */
DOCKER-USER all – anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all – anywhere anywhere
ACCEPT all – anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all – anywhere anywhere
ACCEPT all – anywhere anywhere
ACCEPT all – anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
KUBE-PROXY-FIREWALL all – anywhere anywhere ctstate NEW /* kubernetes load balancer firewall /
KUBE-SERVICES all – anywhere anywhere ctstate NEW / kubernetes service portals */
KUBE-FIREWALL all – anywhere anywhere
Chain DOCKER (1 references)
target prot opt source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all – anywhere anywhere
RETURN all – anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (1 references)
target prot opt source destination
DROP all – anywhere anywhere
RETURN all – anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all – anywhere anywhere
Chain KUBE-EXTERNAL-SERVICES (2 references)
target prot opt source destination
REJECT tcp – anywhere anywhere /* istio-system/istio-ingressgateway:tls has no endpoints / ADDRTYPE match dst-type LOCAL tcp dpt:31520 reject-with icmp-port-unreachable
REJECT tcp – anywhere anywhere / istio-system/istio-ingressgateway:status-port has no endpoints / ADDRTYPE match dst-type LOCAL tcp dpt:30167 reject-with icmp-port-unreachable
REJECT tcp – anywhere anywhere / istio-system/istio-ingressgateway:http2 has no endpoints / ADDRTYPE match dst-type LOCAL tcp dpt:32633 reject-with icmp-port-unreachable
REJECT tcp – anywhere anywhere / istio-system/istio-ingressgateway:https has no endpoints / ADDRTYPE match dst-type LOCAL tcp dpt:30683 reject-with icmp-port-unreachable
REJECT tcp – anywhere anywhere / istio-system/istio-ingressgateway:tcp has no endpoints */ ADDRTYPE match dst-type LOCAL tcp dpt:31675 reject-with icmp-port-unreachable
Chain KUBE-FIREWALL (2 references)
target prot opt source destination
DROP all – !127.0.0.0/8 127.0.0.0/8 /* block incoming localnet connections */ ! ctstate RELATED,ESTABLISHED,DNAT
Chain KUBE-FORWARD (1 references)
target prot opt source destination
DROP all – anywhere anywhere ctstate INVALID
ACCEPT all – anywhere anywhere /* kubernetes forwarding rules / mark match 0x4000/0x4000
ACCEPT all – anywhere anywhere / kubernetes forwarding conntrack rule */ ctstate RELATED,ESTABLISHED
Chain KUBE-KUBELET-CANARY (0 references)
target prot opt source destination
Chain KUBE-NODEPORTS (1 references)
target prot opt source destination
Chain KUBE-PROXY-CANARY (0 references)
target prot opt source destination
Chain KUBE-PROXY-FIREWALL (3 references)
target prot opt source destination
Chain KUBE-SERVICES (2 references)
target prot opt source destination
REJECT tcp – anywhere 10.110.193.193 /* istio-system/istio-ingressgateway:tls has no endpoints / tcp dpt:15443 reject-with icmp-port-unreachable
REJECT tcp – anywhere 10.97.205.180 / istio-system/istio-egressgateway:https has no endpoints / tcp dpt:https reject-with icmp-port-unreachable
REJECT tcp – anywhere 10.110.193.193 / istio-system/istio-ingressgateway:status-port has no endpoints / tcp dpt:15021 reject-with icmp-port-unreachable
REJECT tcp – anywhere 10.110.193.193 / istio-system/istio-ingressgateway:http2 has no endpoints / tcp dpt:http reject-with icmp-port-unreachable
REJECT tcp – anywhere 10.110.193.193 / istio-system/istio-ingressgateway:https has no endpoints / tcp dpt:https reject-with icmp-port-unreachable
REJECT tcp – anywhere 10.110.193.193 / istio-system/istio-ingressgateway:tcp has no endpoints / tcp dpt:31400 reject-with icmp-port-unreachable
REJECT tcp – anywhere 10.101.91.162 / foo/sleep:http has no endpoints / tcp dpt:http reject-with icmp-port-unreachable
REJECT tcp – anywhere 10.97.205.180 / istio-system/istio-egressgateway:http2 has no endpoints */ tcp dpt:http reject-with icmp-port-unreachable
kubectl get pods -n kube-system
coredns-5d78c9869d-pcn29 1/1 Running 0 12h
etcd-minikube 1/1 Running 0 12h
kube-apiserver-minikube 1/1 Running 0 12h
kube-controller-manager-minikube 1/1 Running 0 12h
kube-proxy-z9m7n 1/1 Running 0 12h
kube-scheduler-minikube 1/1 Running 0 12h
storage-provisioner 1/1 Running 1 (12h ago) 12h
kubectl logs coredns-5d78c9869d-pcn29 -n kube-system
[INFO] 10.244.0.5:52375 - 5020 “A IN istiod.istio-system.svc.svc.cluster.local. udp 70 false 1232” NXDOMAIN qr,aa,rd 152 0.000232524s
[INFO] 10.244.0.5:55069 - 31164 “AAAA IN istiod.istio-system.svc.istio-system.svc.cluster.local. udp 83 false 1232” NXDOMAIN qr,aa,rd 165 0.00011688s
[INFO] 10.244.0.5:48420 - 55545 “A IN istiod.istio-system.svc.istio-system.svc.cluster.local. udp 83 false 1232” NXDOMAIN qr,aa,rd 165 0.000180462s
[INFO] 10.244.0.6:46070 - 47871 “A IN istiod.istio-system.svc.foo.svc.cluster.local. udp 74 false 1232” NXDOMAIN qr,aa,rd 156 0.000249638s
[INFO] 10.244.0.6:45216 - 53713 “AAAA IN istiod.istio-system.svc.foo.svc.cluster.local. udp 74 false 1232” NXDOMAIN qr,aa,rd 156 0.000324934s
[INFO] 10.244.0.5:41544 - 21595 “AAAA IN istiod.istio-system.svc.istio-system.svc.cluster.local. udp 83 false 1232” NXDOMAIN qr,aa,rd 165 0.000112856s
[INFO] 10.244.0.5:38333 - 12636 “A IN istiod.istio-system.svc.istio-system.svc.cluster.local. udp 83 false 1232” NXDOMAIN qr,aa,rd 165 0.000190188s
[INFO] 10.244.0.6:44973 - 52485 “A IN istiod.istio-system.svc.foo.svc.cluster.local. udp 74 false 1232” NXDOMAIN qr,aa,rd 156 0.000116301s
[INFO] 10.244.0.6:52858 - 54888 “AAAA IN istiod.istio-system.svc.foo.svc.cluster.local. udp 74 false 1232” NXDOMAIN qr,aa,rd 156 0.000176134s
[INFO] 10.244.0.5:33074 - 48159 “A IN istiod.istio-system.svc.svc.cluster.local. udp 70 false 1232” NXDOMAIN qr,aa,rd 152 0.000123383s
[INFO] 10.244.0.5:34489 - 45704 “AAAA IN istiod.istio-system.svc.svc.cluster.local. udp 70 false 1232” NXDOMAIN qr,aa,rd 152 0.000187342s
[INFO] 10.244.0.4:42884 - 17409 “AAAA IN istiod.istio-system.svc.istio-system.svc.cluster.local. udp 83 false 1232” NXDOMAIN qr,aa,rd 165 0.000236942s
[INFO] 10.244.0.4:58737 - 30693 “A IN istiod.istio-system.svc.istio-system.svc.cluster.local. udp 83 false 1232” NXDOMAIN qr,aa,rd 165 0.000299228s
[INFO] 10.244.0.6:38372 - 38901 “AAAA IN istiod.istio-system.svc.svc.cluster.local. udp 70 false 1232” NXDOMAIN qr,aa,rd 152 0.000236102s
[INFO] 10.244.0.6:36144 - 36113 “A IN istiod.istio-system.svc.svc.cluster.local. udp 70 false 1232” NXDOMAIN qr,aa,rd 152 0.000297888s