Istiod installation fails

:heavy_check_mark: Istio core installed
✘ Istiod encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition
Deployment/istio-system/istiod
Processing resources for Addons, Egress gateways, Ingress gateways. Waiting for Deployment/istio-system/istio-egressgateway, Deployment/istio-system/ist…

NAME READY STATUS RESTARTS AGE
grafana-74dc798895-8s25t 1/1 Running 0 2m6s
istio-egressgateway-6b9fff7b95-wcvl8 0/1 ContainerCreating 0 7m56s
istio-ingressgateway-5f5b59696c-ng8rv 0/1 ContainerCreating 0 7m56s
istio-tracing-8584b4d7f9-xldbz 1/1 Running 0 2m5s
istiod-69bbcb65c-bl52k 0/1 Running 0 12m
kiali-6f457f5964-5xcmk 1/1 Running 0 7m55s
prometheus-77f78d599d-ngz4m 0/2 ContainerCreating 0 2m5s

What the logs on the istiod pod ?

What give kubectl describe -n istio-system xxx with xxx all the pod in ContainerCreating state ?

Hey - Please find the logs. AWS App mesh is running fine but istio shows this error -
We are running this on EKS1.16 with Additional CIDR Blocks of CIDR- 100.64.0.0/10

Events:
Type Reason Age From Message


Normal Scheduled 6m48s default-scheduler Successfully assigned istio-system/istiod-69bbcb65c-qlhdh to ip-10-0-0-86.us-west-2.compute.internal
Normal Pulled 6m47s kubelet, ip-10-0-0-86.us-west-2.compute.internal Container image “docker.io/istio/pilot:1.6.2” already present on machine
Normal Created 6m47s kubelet, ip-10-0-0-86.us-west-2.compute.internal Created container discovery
Normal Started 6m46s kubelet, ip-10-0-0-86.us-west-2.compute.internal Started container discovery
Warning Unhealthy 105s (x101 over 6m45s) kubelet, ip-10-0-0-86.us-west-2.compute.internal Readiness probe failed: Get http://100.64.38.9:8080/ready: dial tcp 100.64.38.9:8080: connect: connection refused


NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default my-nginx-f97c96f6d-jjfrh 1/1 Running 0 31m 100.64.61.210 ip-10-0-0-86.us-west-2.compute.internal
default my-nginx-f97c96f6d-n8lpp 1/1 Running 0 31m 100.64.38.138 ip-10-0-0-86.us-west-2.compute.internal
default redis-master-hbcj9 1/1 Running 0 16m 100.64.54.108 ip-10-0-0-86.us-west-2.compute.internal
istio-system istiod-69bbcb65c-qlhdh 0/1 Running 0 6m23s 100.64.38.9 ip-10-0-0-86.us-west-2.compute.internal
kube-system aws-node-2m2z2 1/1 Running 1 34m 10.0.0.86 ip-10-0-0-86.us-west-2.compute.internal <n

The desccribe show your istiod is not starting properly because readuness is not ok.

You will need to show logs of your istiod pod to try to understand why.

Hi @Gregoire

i have a same issue. i setup a k8s multi-node cluster (using vagrant). when i installed istio on that it show me:

vagrant@k8smaster:~/istio-1.6.1$ istioctl install --set profile=demo
Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.
✔ Istio core installed                                                                                                                         
- Processing resources for Istiod. Waiting for Deployment/istio-system/istiod                                                                  
✘ Istiod encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition                
Deployment/istio-system/istiod
✘ Egress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the conditiont/is...
Deployment/istio-system/istio-egressgateway
✘ Ingress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the conditionome...
Deployment/istio-system/istio-ingressgateway
✘ Addons encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition                
Deployment/istio-system/prometheus
- Pruning removed resources                                                                                                                    Error: failed to apply manifests: errors occurred during operation

when i run kubectl get pod -n istio-system

> vagrant@k8smaster:~/istio-1.6.1$ kubectl get pod -n istio-system
> NAME                                    READY   STATUS              RESTARTS   AGE
> grafana-54b54568fc-mnx5b                1/1     Running             0          4h22m
> istio-egressgateway-77f5cd5497-nfjt7    0/1     ContainerCreating   0          4h22m
> istio-ingressgateway-6956964b69-xjqzq   0/1     ContainerCreating   0          4h22m
> istio-tracing-9dd6c4f7c-z67zx           1/1     Running             0          4h22m
> istiod-74f67f75d5-gt9kq                 0/1     Running             0          4h27m
> kiali-d45468dc4-fgmm6                   1/1     Running             0          4h22m
> prometheus-5767f54db5-nxm9p             0/2     ContainerCreating   0          4h22m

output of kubectl describe istiod-74f67f75d5-gt9kq -n istio-system

> Events:
>   Type     Reason     Age                     From               Message
>   ----     ------     ----                    ----               -------
>   Warning  Unhealthy  58s (x6200 over 5h10m)  kubelet, k8snode1  Readiness probe failed: Get http://192.168.249.20:8080/ready: dial tcp 192.168.249.20:8080: connect: connection refused

What is the logs on the istiod container ?

output of kubectl logs istiod-74f67f75d5-gt9kq -n istio-system

vagrant@k8smaster:~$ kubectl logs  istiod-74f67f75d5-gt9kq -n istio-system
gc 1 @0.007s 11%: 0.027+1.5+1.0 ms clock, 0.054+0.19/0.033/1.0+2.0 ms cpu, 4->4->2 MB, 5 MB goal, 2 P
gc 2 @0.013s 8%: 0.009+2.8+0.015 ms clock, 0.018+0.25/0.061/1.8+0.031 ms cpu, 4->5->3 MB, 5 MB goal, 2 P
2020-06-16T07:05:07.405787Z	info	FLAG: --appNamespace=""
2020-06-16T07:05:07.405810Z	info	FLAG: --caCertFile=""
2020-06-16T07:05:07.405813Z	info	FLAG: --clusterID="Kubernetes"
2020-06-16T07:05:07.405815Z	info	FLAG: --clusterRegistriesNamespace="istio-system"
2020-06-16T07:05:07.405816Z	info	FLAG: --configDir=""
2020-06-16T07:05:07.405818Z	info	FLAG: --consulserverURL=""
2020-06-16T07:05:07.405820Z	info	FLAG: --ctrlz_address="localhost"
2020-06-16T07:05:07.405822Z	info	FLAG: --ctrlz_port="9876"
2020-06-16T07:05:07.405825Z	info	FLAG: --disable-install-crds="true"
2020-06-16T07:05:07.405826Z	info	FLAG: --domain="cluster.local"
2020-06-16T07:05:07.405828Z	info	FLAG: --grpcAddr=":15010"
2020-06-16T07:05:07.405830Z	info	FLAG: --help="false"
2020-06-16T07:05:07.405831Z	info	FLAG: --httpAddr=":8080"
2020-06-16T07:05:07.405833Z	info	FLAG: --httpsAddr=":15017"
2020-06-16T07:05:07.405836Z	info	FLAG: --keepaliveInterval="30s"
2020-06-16T07:05:07.405838Z	info	FLAG: --keepaliveMaxServerConnectionAge="30m0s"
2020-06-16T07:05:07.405840Z	info	FLAG: --keepaliveTimeout="10s"
2020-06-16T07:05:07.405841Z	info	FLAG: --kubeconfig=""
2020-06-16T07:05:07.405843Z	info	FLAG: --log_as_json="false"
2020-06-16T07:05:07.405844Z	info	FLAG: --log_caller=""
2020-06-16T07:05:07.405846Z	info	FLAG: --log_output_level="default:info"
2020-06-16T07:05:07.405847Z	info	FLAG: --log_rotate=""
2020-06-16T07:05:07.405850Z	info	FLAG: --log_rotate_max_age="30"
2020-06-16T07:05:07.405852Z	info	FLAG: --log_rotate_max_backups="1000"
2020-06-16T07:05:07.405853Z	info	FLAG: --log_rotate_max_size="104857600"
2020-06-16T07:05:07.405855Z	info	FLAG: --log_stacktrace_level="default:none"
2020-06-16T07:05:07.405858Z	info	FLAG: --log_target="[stdout]"
2020-06-16T07:05:07.405860Z	info	FLAG: --mcpInitialConnWindowSize="1048576"
2020-06-16T07:05:07.405862Z	info	FLAG: --mcpInitialWindowSize="1048576"
2020-06-16T07:05:07.405864Z	info	FLAG: --mcpMaxMsgSize="4194304"
2020-06-16T07:05:07.405865Z	info	FLAG: --meshConfig="/etc/istio/config/mesh"
2020-06-16T07:05:07.405867Z	info	FLAG: --monitoringAddr=":15014"
2020-06-16T07:05:07.405869Z	info	FLAG: --namespace="istio-system"
2020-06-16T07:05:07.405871Z	info	FLAG: --networksConfig="/etc/istio/config/meshNetworks"
2020-06-16T07:05:07.405876Z	info	FLAG: --plugins="[authn,authz,health,mixer]"
2020-06-16T07:05:07.405878Z	info	FLAG: --profile="true"
2020-06-16T07:05:07.405880Z	info	FLAG: --registries="[Kubernetes]"
2020-06-16T07:05:07.405882Z	info	FLAG: --resync="1m0s"
2020-06-16T07:05:07.405884Z	info	FLAG: --tlsCertFile=""
2020-06-16T07:05:07.405885Z	info	FLAG: --tlsKeyFile=""
2020-06-16T07:05:07.405887Z	info	FLAG: --trust-domain="cluster.local"
2020-06-16T07:05:07.408393Z	info	mesh configuration: {
    "disableMixerHttpReports": true,
    "proxyListenPort": 15001,
    "connectTimeout": "10s",
    "protocolDetectionTimeout": "0.100s",
    "ingressClass": "istio",
    "ingressService": "istio-ingressgateway",
    "ingressControllerMode": "STRICT",
    "enableTracing": true,
    "accessLogFile": "/dev/stdout",
    "defaultConfig": {
        "configPath": "./etc/istio/proxy",
        "binaryPath": "/usr/local/bin/envoy",
        "serviceCluster": "istio-proxy",
        "drainDuration": "45s",
        "parentShutdownDuration": "60s",
        "discoveryAddress": "istiod.istio-system.svc:15012",
        "proxyAdminPort": 15000,
        "statNameLength": 189,
        "concurrency": 2,
        "tracing": {
            "zipkin": {
                "address": "zipkin.istio-system:9411"
            }
        },
        "envoyAccessLogService": {

        },
        "envoyMetricsService": {

        },
        "proxyMetadata": {
            "DNS_AGENT": ""
        },
        "statusPort": 15020
    },
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "sdsUdsPath": "unix:/etc/istio/proxy/SDS",
    "enableAutoMtls": true,
    "trustDomain": "cluster.local",
    "trustDomainAliases": [
    ],
    "defaultServiceExportTo": [
        "*"
    ],
    "defaultVirtualServiceExportTo": [
        "*"
    ],
    "defaultDestinationRuleExportTo": [
        "*"
    ],
    "rootNamespace": "istio-system",
    "localityLbSetting": {
        "enabled": true
    },
    "dnsRefreshRate": "5s",
    "reportBatchMaxEntries": 100,
    "reportBatchMaxTime": "1s",
    "certificates": [
    ],
    "thriftConfig": {

    },
    "serviceSettings": [
    ],
    "enablePrometheusMerge": false
}
2020-06-16T07:05:07.408407Z	info	version: 1.6.1-f07efb91db1c29b7d6f1ee036ac98e48458cd139-dirty-Modified
gc 3 @0.023s 5%: 0.005+3.8+0.011 ms clock, 0.011+0.27/0.044/2.7+0.022 ms cpu, 7->7->5 MB, 8 MB goal, 2 P
2020-06-16T07:05:07.408773Z	info	flags: 
2020-06-16T07:05:07.408860Z	info	mesh networks configuration: {
   "networks": {
   }
}
2020-06-16T07:05:07.408887Z	info	No certificates specified, skipping K8S DNS certificate controller
2020-06-16T07:05:07.409424Z	info	CRD controller watching namespaces ""
2020-06-16T07:05:37.409713Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
2020-06-16T07:06:08.410675Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
2020-06-16T07:06:40.413062Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 4 @120.036s 0%: 0.011+7.9+0.008 ms clock, 0.023+0/0.35/15+0.017 ms cpu, 9->9->5 MB, 10 MB goal, 2 P
2020-06-16T07:07:14.414137Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
2020-06-16T07:07:52.415752Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
2020-06-16T07:08:38.436295Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 5 @241.036s 0%: 0.012+12+0.009 ms clock, 0.024+0/0.28/18+0.019 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:09:40.437312Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 6 @362.036s 0%: 0.012+9.1+0.010 ms clock, 0.024+0/0.68/16+0.020 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:11:10.441826Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
2020-06-16T07:12:40.443390Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 7 @483.036s 0%: 0.005+2.0+0.003 ms clock, 0.010+0/0.13/3.8+0.007 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:14:10.449698Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 8 @603.085s 0%: 0.004+2.0+0.003 ms clock, 0.009+0/0.072/3.9+0.007 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:15:40.464921Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
2020-06-16T07:17:10.470815Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
gc 9 @723.095s 0%: 0.31+8.3+0.008 ms clock, 0.62+0.42/1.4/14+0.017 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:18:40.472415Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 10 @844.036s 0%: 0.076+8.3+0.008 ms clock, 0.15+0/0.71/14+0.016 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:20:10.473672Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 11 @965.036s 0%: 0.086+4.8+0.006 ms clock, 0.17+0/0.18/9.3+0.012 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:21:40.474545Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
2020-06-16T07:23:10.477538Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 12 @1086.039s 0%: 0.014+13+0.030 ms clock, 0.028+0/3.4/23+0.060 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:24:40.479295Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 13 @1207.036s 0%: 0.090+6.3+0.006 ms clock, 0.18+0/0.24/10+0.013 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:26:10.485287Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 14 @1328.036s 0%: 0.15+10+0.009 ms clock, 0.30+0/0.32/18+0.019 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:27:40.487234Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
.
.
.
.

2020-06-16T12:18:41.438827Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 159 @18855.038s 0%: 0.008+6.5+0.006 ms clock, 0.016+0/0.19/8.8+0.012 ms cpu, 6->6->5 MB, 11 MB goal, 2 P

2020-06-16T07:05:37.409713Z error failed to list CRDs: Get “https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions”: dial tcp 10.96.0.1:443: i/o timeout this is which is repeated in the logs. AFAIK it is unable to access the kube api-server.

any help would be appreciated…

Im having the same issue, could be related to a prerequisites platform? I facing this installation in a lab with a K8s Cluster, master and a node virtualized with “Virtual Machine Manager by Redhat”, maybe i didn’t find the correct sheets :confused:. Any suggests? Thanks

Hi @Gregoire

Any update on this. Still stuck here.

It can be some errors and they may be hard to track.

Do you have a global network policy that prevent inter-namespace request?
if you create a pod on default namespace, is a “curl -k https://kubernetes.default” return something?
If you put the pod in the istio-system what happen to the curl ?

Sorry @Gregoire for late response.

I tried the both of these condition i.e pod in default ns & istio-system namespace. it give the same error i.e.(unable to connect to the api-server)

curl: (7) Failed to connect to kubernetes.default port 443: Connection timed out

and also don’t have any global network policy. i think there is a problem in multi-node cluster. will investigate it more and tell you.

I have the same problem, anyone can help? Thanks.