Istiod installation fails

1

:heavy_check_mark: Istio core installed
✘ Istiod encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition
Deployment/istio-system/istiod
Processing resources for Addons, Egress gateways, Ingress gateways. Waiting for Deployment/istio-system/istio-egressgateway, Deployment/istio-system/ist…

NAME READY STATUS RESTARTS AGE
grafana-74dc798895-8s25t 1/1 Running 0 2m6s
istio-egressgateway-6b9fff7b95-wcvl8 0/1 ContainerCreating 0 7m56s
istio-ingressgateway-5f5b59696c-ng8rv 0/1 ContainerCreating 0 7m56s
istio-tracing-8584b4d7f9-xldbz 1/1 Running 0 2m5s
istiod-69bbcb65c-bl52k 0/1 Running 0 12m
kiali-6f457f5964-5xcmk 1/1 Running 0 7m55s
prometheus-77f78d599d-ngz4m 0/2 ContainerCreating 0 2m5s

1 Like
2

What the logs on the istiod pod ?

What give kubectl describe -n istio-system xxx with xxx all the pod in ContainerCreating state ?

3

Hey - Please find the logs. AWS App mesh is running fine but istio shows this error -
We are running this on EKS1.16 with Additional CIDR Blocks of CIDR- 100.64.0.0/10

Events:
Type Reason Age From Message

Normal Scheduled 6m48s default-scheduler Successfully assigned istio-system/istiod-69bbcb65c-qlhdh to ip-10-0-0-86.us-west-2.compute.internal
Normal Pulled 6m47s kubelet, ip-10-0-0-86.us-west-2.compute.internal Container image “docker.io/istio/pilot:1.6.2” already present on machine
Normal Created 6m47s kubelet, ip-10-0-0-86.us-west-2.compute.internal Created container discovery
Normal Started 6m46s kubelet, ip-10-0-0-86.us-west-2.compute.internal Started container discovery
Warning Unhealthy 105s (x101 over 6m45s) kubelet, ip-10-0-0-86.us-west-2.compute.internal Readiness probe failed: Get http://100.64.38.9:8080/ready: dial tcp 100.64.38.9:8080: connect: connection refused

NAMESPACE NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
default my-nginx-f97c96f6d-jjfrh 1/1 Running 0 31m 100.64.61.210 ip-10-0-0-86.us-west-2.compute.internal
default my-nginx-f97c96f6d-n8lpp 1/1 Running 0 31m 100.64.38.138 ip-10-0-0-86.us-west-2.compute.internal
default redis-master-hbcj9 1/1 Running 0 16m 100.64.54.108 ip-10-0-0-86.us-west-2.compute.internal
istio-system istiod-69bbcb65c-qlhdh 0/1 Running 0 6m23s 100.64.38.9 ip-10-0-0-86.us-west-2.compute.internal
kube-system aws-node-2m2z2 1/1 Running 1 34m 10.0.0.86 ip-10-0-0-86.us-west-2.compute.internal <n

4

The desccribe show your istiod is not starting properly because readuness is not ok.

You will need to show logs of your istiod pod to try to understand why.

5

Hi @Gregoire

i have a same issue. i setup a k8s multi-node cluster (using vagrant). when i installed istio on that it show me:

vagrant@k8smaster:~/istio-1.6.1$ istioctl install --set profile=demo
Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.
✔ Istio core installed                                                                                                                         
- Processing resources for Istiod. Waiting for Deployment/istio-system/istiod                                                                  
✘ Istiod encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition                
Deployment/istio-system/istiod
✘ Egress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the conditiont/is...
Deployment/istio-system/istio-egressgateway
✘ Ingress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the conditionome...
Deployment/istio-system/istio-ingressgateway
✘ Addons encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition                
Deployment/istio-system/prometheus
- Pruning removed resources                                                                                                                    Error: failed to apply manifests: errors occurred during operation

when i run kubectl get pod -n istio-system

> vagrant@k8smaster:~/istio-1.6.1$ kubectl get pod -n istio-system
> NAME                                    READY   STATUS              RESTARTS   AGE
> grafana-54b54568fc-mnx5b                1/1     Running             0          4h22m
> istio-egressgateway-77f5cd5497-nfjt7    0/1     ContainerCreating   0          4h22m
> istio-ingressgateway-6956964b69-xjqzq   0/1     ContainerCreating   0          4h22m
> istio-tracing-9dd6c4f7c-z67zx           1/1     Running             0          4h22m
> istiod-74f67f75d5-gt9kq                 0/1     Running             0          4h27m
> kiali-d45468dc4-fgmm6                   1/1     Running             0          4h22m
> prometheus-5767f54db5-nxm9p             0/2     ContainerCreating   0          4h22m

output of kubectl describe istiod-74f67f75d5-gt9kq -n istio-system

> Events:
>   Type     Reason     Age                     From               Message
>   ----     ------     ----                    ----               -------
>   Warning  Unhealthy  58s (x6200 over 5h10m)  kubelet, k8snode1  Readiness probe failed: Get http://192.168.249.20:8080/ready: dial tcp 192.168.249.20:8080: connect: connection refused
6

What is the logs on the istiod container ?

7

output of kubectl logs istiod-74f67f75d5-gt9kq -n istio-system

vagrant@k8smaster:~$ kubectl logs  istiod-74f67f75d5-gt9kq -n istio-system
gc 1 @0.007s 11%: 0.027+1.5+1.0 ms clock, 0.054+0.19/0.033/1.0+2.0 ms cpu, 4->4->2 MB, 5 MB goal, 2 P
gc 2 @0.013s 8%: 0.009+2.8+0.015 ms clock, 0.018+0.25/0.061/1.8+0.031 ms cpu, 4->5->3 MB, 5 MB goal, 2 P
2020-06-16T07:05:07.405787Z	info	FLAG: --appNamespace=""
2020-06-16T07:05:07.405810Z	info	FLAG: --caCertFile=""
2020-06-16T07:05:07.405813Z	info	FLAG: --clusterID="Kubernetes"
2020-06-16T07:05:07.405815Z	info	FLAG: --clusterRegistriesNamespace="istio-system"
2020-06-16T07:05:07.405816Z	info	FLAG: --configDir=""
2020-06-16T07:05:07.405818Z	info	FLAG: --consulserverURL=""
2020-06-16T07:05:07.405820Z	info	FLAG: --ctrlz_address="localhost"
2020-06-16T07:05:07.405822Z	info	FLAG: --ctrlz_port="9876"
2020-06-16T07:05:07.405825Z	info	FLAG: --disable-install-crds="true"
2020-06-16T07:05:07.405826Z	info	FLAG: --domain="cluster.local"
2020-06-16T07:05:07.405828Z	info	FLAG: --grpcAddr=":15010"
2020-06-16T07:05:07.405830Z	info	FLAG: --help="false"
2020-06-16T07:05:07.405831Z	info	FLAG: --httpAddr=":8080"
2020-06-16T07:05:07.405833Z	info	FLAG: --httpsAddr=":15017"
2020-06-16T07:05:07.405836Z	info	FLAG: --keepaliveInterval="30s"
2020-06-16T07:05:07.405838Z	info	FLAG: --keepaliveMaxServerConnectionAge="30m0s"
2020-06-16T07:05:07.405840Z	info	FLAG: --keepaliveTimeout="10s"
2020-06-16T07:05:07.405841Z	info	FLAG: --kubeconfig=""
2020-06-16T07:05:07.405843Z	info	FLAG: --log_as_json="false"
2020-06-16T07:05:07.405844Z	info	FLAG: --log_caller=""
2020-06-16T07:05:07.405846Z	info	FLAG: --log_output_level="default:info"
2020-06-16T07:05:07.405847Z	info	FLAG: --log_rotate=""
2020-06-16T07:05:07.405850Z	info	FLAG: --log_rotate_max_age="30"
2020-06-16T07:05:07.405852Z	info	FLAG: --log_rotate_max_backups="1000"
2020-06-16T07:05:07.405853Z	info	FLAG: --log_rotate_max_size="104857600"
2020-06-16T07:05:07.405855Z	info	FLAG: --log_stacktrace_level="default:none"
2020-06-16T07:05:07.405858Z	info	FLAG: --log_target="[stdout]"
2020-06-16T07:05:07.405860Z	info	FLAG: --mcpInitialConnWindowSize="1048576"
2020-06-16T07:05:07.405862Z	info	FLAG: --mcpInitialWindowSize="1048576"
2020-06-16T07:05:07.405864Z	info	FLAG: --mcpMaxMsgSize="4194304"
2020-06-16T07:05:07.405865Z	info	FLAG: --meshConfig="/etc/istio/config/mesh"
2020-06-16T07:05:07.405867Z	info	FLAG: --monitoringAddr=":15014"
2020-06-16T07:05:07.405869Z	info	FLAG: --namespace="istio-system"
2020-06-16T07:05:07.405871Z	info	FLAG: --networksConfig="/etc/istio/config/meshNetworks"
2020-06-16T07:05:07.405876Z	info	FLAG: --plugins="[authn,authz,health,mixer]"
2020-06-16T07:05:07.405878Z	info	FLAG: --profile="true"
2020-06-16T07:05:07.405880Z	info	FLAG: --registries="[Kubernetes]"
2020-06-16T07:05:07.405882Z	info	FLAG: --resync="1m0s"
2020-06-16T07:05:07.405884Z	info	FLAG: --tlsCertFile=""
2020-06-16T07:05:07.405885Z	info	FLAG: --tlsKeyFile=""
2020-06-16T07:05:07.405887Z	info	FLAG: --trust-domain="cluster.local"
2020-06-16T07:05:07.408393Z	info	mesh configuration: {
    "disableMixerHttpReports": true,
    "proxyListenPort": 15001,
    "connectTimeout": "10s",
    "protocolDetectionTimeout": "0.100s",
    "ingressClass": "istio",
    "ingressService": "istio-ingressgateway",
    "ingressControllerMode": "STRICT",
    "enableTracing": true,
    "accessLogFile": "/dev/stdout",
    "defaultConfig": {
        "configPath": "./etc/istio/proxy",
        "binaryPath": "/usr/local/bin/envoy",
        "serviceCluster": "istio-proxy",
        "drainDuration": "45s",
        "parentShutdownDuration": "60s",
        "discoveryAddress": "istiod.istio-system.svc:15012",
        "proxyAdminPort": 15000,
        "statNameLength": 189,
        "concurrency": 2,
        "tracing": {
            "zipkin": {
                "address": "zipkin.istio-system:9411"
            }
        },
        "envoyAccessLogService": {

        },
        "envoyMetricsService": {

        },
        "proxyMetadata": {
            "DNS_AGENT": ""
        },
        "statusPort": 15020
    },
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "sdsUdsPath": "unix:/etc/istio/proxy/SDS",
    "enableAutoMtls": true,
    "trustDomain": "cluster.local",
    "trustDomainAliases": [
    ],
    "defaultServiceExportTo": [
        "*"
    ],
    "defaultVirtualServiceExportTo": [
        "*"
    ],
    "defaultDestinationRuleExportTo": [
        "*"
    ],
    "rootNamespace": "istio-system",
    "localityLbSetting": {
        "enabled": true
    },
    "dnsRefreshRate": "5s",
    "reportBatchMaxEntries": 100,
    "reportBatchMaxTime": "1s",
    "certificates": [
    ],
    "thriftConfig": {

    },
    "serviceSettings": [
    ],
    "enablePrometheusMerge": false
}
2020-06-16T07:05:07.408407Z	info	version: 1.6.1-f07efb91db1c29b7d6f1ee036ac98e48458cd139-dirty-Modified
gc 3 @0.023s 5%: 0.005+3.8+0.011 ms clock, 0.011+0.27/0.044/2.7+0.022 ms cpu, 7->7->5 MB, 8 MB goal, 2 P
2020-06-16T07:05:07.408773Z	info	flags: 
2020-06-16T07:05:07.408860Z	info	mesh networks configuration: {
   "networks": {
   }
}
2020-06-16T07:05:07.408887Z	info	No certificates specified, skipping K8S DNS certificate controller
2020-06-16T07:05:07.409424Z	info	CRD controller watching namespaces ""
2020-06-16T07:05:37.409713Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
2020-06-16T07:06:08.410675Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
2020-06-16T07:06:40.413062Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 4 @120.036s 0%: 0.011+7.9+0.008 ms clock, 0.023+0/0.35/15+0.017 ms cpu, 9->9->5 MB, 10 MB goal, 2 P
2020-06-16T07:07:14.414137Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
2020-06-16T07:07:52.415752Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
2020-06-16T07:08:38.436295Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 5 @241.036s 0%: 0.012+12+0.009 ms clock, 0.024+0/0.28/18+0.019 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:09:40.437312Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 6 @362.036s 0%: 0.012+9.1+0.010 ms clock, 0.024+0/0.68/16+0.020 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:11:10.441826Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
2020-06-16T07:12:40.443390Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 7 @483.036s 0%: 0.005+2.0+0.003 ms clock, 0.010+0/0.13/3.8+0.007 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:14:10.449698Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 8 @603.085s 0%: 0.004+2.0+0.003 ms clock, 0.009+0/0.072/3.9+0.007 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:15:40.464921Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
2020-06-16T07:17:10.470815Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
gc 9 @723.095s 0%: 0.31+8.3+0.008 ms clock, 0.62+0.42/1.4/14+0.017 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:18:40.472415Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 10 @844.036s 0%: 0.076+8.3+0.008 ms clock, 0.15+0/0.71/14+0.016 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:20:10.473672Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 11 @965.036s 0%: 0.086+4.8+0.006 ms clock, 0.17+0/0.18/9.3+0.012 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:21:40.474545Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
2020-06-16T07:23:10.477538Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 12 @1086.039s 0%: 0.014+13+0.030 ms clock, 0.028+0/3.4/23+0.060 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:24:40.479295Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 13 @1207.036s 0%: 0.090+6.3+0.006 ms clock, 0.18+0/0.24/10+0.013 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:26:10.485287Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 14 @1328.036s 0%: 0.15+10+0.009 ms clock, 0.30+0/0.32/18+0.019 ms cpu, 6->6->5 MB, 11 MB goal, 2 P
2020-06-16T07:27:40.487234Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
.
.
.
.

2020-06-16T12:18:41.438827Z	error	failed to list CRDs: Get "https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions": dial tcp 10.96.0.1:443: i/o timeout
GC forced
gc 159 @18855.038s 0%: 0.008+6.5+0.006 ms clock, 0.016+0/0.19/8.8+0.012 ms cpu, 6->6->5 MB, 11 MB goal, 2 P

2020-06-16T07:05:37.409713Z error failed to list CRDs: Get “https://10.96.0.1:443/apis/apiextensions.k8s.io/v1beta1/customresourcedefinitions”: dial tcp 10.96.0.1:443: i/o timeout this is which is repeated in the logs. AFAIK it is unable to access the kube api-server.

any help would be appreciated…

8

Im having the same issue, could be related to a prerequisites platform? I facing this installation in a lab with a K8s Cluster, master and a node virtualized with “Virtual Machine Manager by Redhat”, maybe i didn’t find the correct sheets :confused:. Any suggests? Thanks

9

Hi @Gregoire

Any update on this. Still stuck here.

10

It can be some errors and they may be hard to track.

Do you have a global network policy that prevent inter-namespace request?
if you create a pod on default namespace, is a “curl -k https://kubernetes.default” return something?
If you put the pod in the istio-system what happen to the curl ?

11

Sorry @Gregoire for late response.

I tried the both of these condition i.e pod in default ns & istio-system namespace. it give the same error i.e.(unable to connect to the api-server)

curl: (7) Failed to connect to kubernetes.default port 443: Connection timed out

and also don’t have any global network policy. i think there is a problem in multi-node cluster. will investigate it more and tell you.

1 Like
12

I have the same problem, anyone can help? Thanks.

13

Anyone got the idea how to solve it? I am facing the same error while using EKS with custom networking enabled. Everything on kubernetes side is working but istio is not even getting installed.

$ istioctl version
1 error occurred:
	* error port-forewarding into istiod-5c868d8bdd-9zq4s : the server is currently unable to handle the request (get pods istiod-5c868d8bdd-9zq4s:15014)


1.7.2

$ istioctl install -f istio-cni.yaml

✔ Istio core installed
- Processing resources for Istiod.                                                                   2020-10-14T14:01:57.603166Z	error	installer	failed to create "Service/istio-system/istiod": Internal error occurred: failed to allocate a serviceIP: rpc error: code = Unavailable desc = transport is closing
✘ Istiod encountered an error: failed to create "Service/istio-system/istiod": Internal error occurred: failed to allocate a serviceIP: rpc error: code = Unavailable desc = transport is closing
✔ CNI installed
✘ Ingress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition
Deployment/istio-system/istio-ingressgateway
- Pruning removed resources                                                                          Error: failed to install manifests: errors occurred during operation
14

I have the identical issues on my pods… Istiod is getting connection refused where as egress & ingress gateways remains in containerCreating

kubectl describe gives below error:

Warning FailedMount 31h (x5 over 47h) kubelet Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert], unattached volumes=[gatewaysdsudspath podinfo ingressgateway-certs ingressgateway-ca-certs istio-ingressgateway-service-account-token-shvfv istio-envoy config-volume istiod-ca-cert]: timed out waiting for the condition

kindly suggest

15

The issue occurs if namespaces istio-system is labeled with “istio-injection=enabled”

Just remove the label and try again:
kubectl label ns istio-system istio-injection-

16

Facing the same issue when I am using kind!! Has anyone found a concrete solution for this?

17

I solve this issue by deploying the istiod deployment on master node.

18

Hey folks !

Encountered the same issue on Virtual Box.

Found out it was a lower memory issue. It should be in the official Istio documentation.
Resolved the issue by just updating memory (RAM) for the master node, from 2048 MB to 4096 MB.

Credit goes to this Stackoverflow answer by Dean Schulze.
Unfortunately I am unable to upvote on Stackoverflow because I am a new member there.
Please go ahead to upvote his answer, to help the community.

:+1: :wink:

2 Likes
19

I had the same problem. I solved it with memory adjustment. Thanks!