Istio installation fails on v1.19.4

ancyra · November 17, 2020, 2:08pm

Hello,

I installed Kubernetes v1.19.4. The cluster has 3 nodes and those nodes have Public IP addresses. They working in a web hosting company’s infrastructure.

I managed to Istio before in a few weeks ago. However, I can’t install it anymore.

I’m using Kubernetes v1.19.4, istio-1.7.4 and Calico.

I created the cluster via this command:
kubeadm init --pod-network-cidr=192.168.0.0/16

istioctl can’t install ingress and egress gateways.

istiod logs:

2020-11-17T13:48:00.260228Z	info	FLAG: --appNamespace=""
2020-11-17T13:48:00.260383Z	info	FLAG: --caCertFile=""
2020-11-17T13:48:00.260452Z	info	FLAG: --clusterID="Kubernetes"
2020-11-17T13:48:00.260491Z	info	FLAG: --clusterRegistriesNamespace="istio-system"
2020-11-17T13:48:00.260539Z	info	FLAG: --configDir=""
2020-11-17T13:48:00.260571Z	info	FLAG: --consulserverURL=""
2020-11-17T13:48:00.260614Z	info	FLAG: --ctrlz_address="localhost"
2020-11-17T13:48:00.260651Z	info	FLAG: --ctrlz_port="9876"
2020-11-17T13:48:00.260696Z	info	FLAG: --domain="cluster.local"
2020-11-17T13:48:00.260731Z	info	FLAG: --grpcAddr=":15010"
2020-11-17T13:48:00.260775Z	info	FLAG: --help="false"
2020-11-17T13:48:00.260811Z	info	FLAG: --httpAddr=":8080"
2020-11-17T13:48:00.260855Z	info	FLAG: --httpsAddr=":15017"
2020-11-17T13:48:00.260898Z	info	FLAG: --keepaliveInterval="30s"
2020-11-17T13:48:00.260931Z	info	FLAG: --keepaliveMaxServerConnectionAge="30m0s"
2020-11-17T13:48:00.260974Z	info	FLAG: --keepaliveTimeout="10s"
2020-11-17T13:48:00.261009Z	info	FLAG: --kubeconfig=""
2020-11-17T13:48:00.261052Z	info	FLAG: --log_as_json="false"
2020-11-17T13:48:00.261087Z	info	FLAG: --log_caller=""
2020-11-17T13:48:00.261131Z	info	FLAG: --log_output_level="default:info"
2020-11-17T13:48:00.261172Z	info	FLAG: --log_rotate=""
2020-11-17T13:48:00.261204Z	info	FLAG: --log_rotate_max_age="30"
2020-11-17T13:48:00.261248Z	info	FLAG: --log_rotate_max_backups="1000"
2020-11-17T13:48:00.261284Z	info	FLAG: --log_rotate_max_size="104857600"
2020-11-17T13:48:00.261327Z	info	FLAG: --log_stacktrace_level="default:none"
2020-11-17T13:48:00.261365Z	info	FLAG: --log_target="[stdout]"
2020-11-17T13:48:00.261409Z	info	FLAG: --mcpInitialConnWindowSize="1048576"
2020-11-17T13:48:00.261445Z	info	FLAG: --mcpInitialWindowSize="1048576"
2020-11-17T13:48:00.261490Z	info	FLAG: --mcpMaxMsgSize="4194304"
2020-11-17T13:48:00.261532Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2020-11-17T13:48:00.261577Z	info	FLAG: --monitoringAddr=":15014"
2020-11-17T13:48:00.261609Z	info	FLAG: --namespace="istio-system"
2020-11-17T13:48:00.261653Z	info	FLAG: --networksConfig="/etc/istio/config/meshNetworks"
2020-11-17T13:48:00.261694Z	info	FLAG: --plugins="[authn,authz,health,mixer]"
2020-11-17T13:48:00.261741Z	info	FLAG: --profile="true"
2020-11-17T13:48:00.261774Z	info	FLAG: --registries="[Kubernetes]"
2020-11-17T13:48:00.261829Z	info	FLAG: --resync="1m0s"
2020-11-17T13:48:00.261873Z	info	FLAG: --secureGRPCAddr=":15012"
2020-11-17T13:48:00.261918Z	info	FLAG: --tlsCertFile=""
2020-11-17T13:48:00.261952Z	info	FLAG: --tlsKeyFile=""
2020-11-17T13:48:00.261996Z	info	FLAG: --trust-domain="cluster.local"
2020-11-17T13:48:00.292936Z	info	initializing mesh configuration
2020-11-17T13:48:00.294092Z	info	mesh configuration: {
    "disablePolicyChecks": true,
    "disableMixerHttpReports": true,
    "proxyListenPort": 15001,
    "connectTimeout": "10s",
    "protocolDetectionTimeout": "5s",
    "ingressClass": "istio",
    "ingressService": "istio-ingressgateway",
    "ingressControllerMode": "STRICT",
    "enableTracing": true,
    "accessLogFile": "/dev/stdout",
    "defaultConfig": {
        "configPath": "./etc/istio/proxy",
        "binaryPath": "/usr/local/bin/envoy",
        "serviceCluster": "istio-proxy",
        "drainDuration": "45s",
        "parentShutdownDuration": "60s",
        "discoveryAddress": "istiod.istio-system.svc:15012",
        "proxyAdminPort": 15000,
        "controlPlaneAuthPolicy": "MUTUAL_TLS",
        "statNameLength": 189,
        "concurrency": 2,
        "tracing": {
            "zipkin": {
                "address": "zipkin.istio-system:9411"
            }
        },
        "envoyAccessLogService": {

        },
        "envoyMetricsService": {

        },
        "proxyMetadata": {
            "DNS_AGENT": ""
        },
        "statusPort": 15020,
        "terminationDrainDuration": "5s"
    },
    "outboundTrafficPolicy": {
        "mode": "ALLOW_ANY"
    },
    "sdsUdsPath": "unix:./etc/istio/proxy/SDS",
    "enableAutoMtls": true,
    "trustDomain": "cluster.local",
    "trustDomainAliases": [
    ],
    "defaultServiceExportTo": [
        "*"
    ],
    "defaultVirtualServiceExportTo": [
        "*"
    ],
    "defaultDestinationRuleExportTo": [
        "*"
    ],
    "rootNamespace": "istio-system",
    "localityLbSetting": {
        "enabled": true
    },
    "dnsRefreshRate": "5s",
    "reportBatchMaxEntries": 100,
    "reportBatchMaxTime": "1s",
    "certificates": [
    ],
    "thriftConfig": {

    },
    "serviceSettings": [
    ],
    "enablePrometheusMerge": true
}
2020-11-17T13:48:00.294106Z	info	version: 1.7.4-4ce531ff1823a3abb9f42fa9d35523b0436e2d04-Clean
2020-11-17T13:48:00.294342Z	info	flags:
2020-11-17T13:48:00.294456Z	warn	Config not found: /var/run/secrets/remote/config
2020-11-17T13:48:00.295490Z	info	initializing mesh networks
2020-11-17T13:48:00.295659Z	info	mesh networks configuration: {
   "networks": {
   }
}
2020-11-17T13:48:00.295722Z	info	initializing mesh handlers
2020-11-17T13:48:00.295731Z	info	initializing controllers
2020-11-17T13:48:00.295734Z	info	No certificates specified, skipping K8S DNS certificate controller
2020-11-17T13:48:00.482105Z	warn	kube	Skipping CRD networking.x-k8s.io/v1alpha1/GatewayClass as it is not present
2020-11-17T13:48:00.482127Z	warn	kube	Skipping CRD networking.x-k8s.io/v1alpha1/Gateway as it is not present
2020-11-17T13:48:00.482130Z	warn	kube	Skipping CRD networking.x-k8s.io/v1alpha1/HTTPRoute as it is not present
2020-11-17T13:48:00.482133Z	warn	kube	Skipping CRD networking.x-k8s.io/v1alpha1/TcpRoute as it is not present
2020-11-17T13:48:00.482136Z	warn	kube	Skipping CRD networking.x-k8s.io/v1alpha1/TrafficSplit as it is not present
2020-11-17T13:48:00.482315Z	info	Ingress controller watching namespaces ""
2020-11-17T13:48:00.684435Z	warn	Config Store &{0xc001174510 cluster.local 0xc00164acc0 [] [] 0xc0008e7050 0xc0018d8cf0 0xc001964270 0xc000e5cb80} cannot track distribution in aggregate: this SetLedger operation is not supported by kube ingress controller
2020-11-17T13:48:00.684472Z	info	Adding Kubernetes registry adapter
2020-11-17T13:48:00.684481Z	info	Initializing Kubernetes service registry "Kubernetes"
2020-11-17T13:48:00.684582Z	info	JWT policy is first-party-jwt
2020-11-17T13:48:00.684590Z	info	creating CA and initializing public key
2020-11-17T13:48:00.684623Z	info	Use self-signed certificate as the CA certificate
2020-11-17T13:48:00.758472Z	info	pkica	Failed to get secret (error: secrets "istio-ca-secret" not found), will create one
2020-11-17T13:48:00.946088Z	info	pkica	Using self-generated public key: -----BEGIN CERTIFICATE-----
MIIC3jCCAcagAwIBAgIRAKgS6yTpUKcoIkqT4CvBI6owDQYJKoZIhvcNAQELBQAw
GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yMDExMTcxMzQ4MDBaFw0zMDEx
MTUxMzQ4MDBaMBgxFjAUBgNVBAoTDWNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQCzFT4boDNUaP00FbbTUs4kf+x+T0nTcxoan4Og
HhhWMfRSplZjN9nxb7UOGTwbt5N9iZ/LXVejSncJx69RceOqyrZbU+tBh+z6GS8Z
52YCi+qOtOM66AI1rwXhILqE4Myyo2C9jg5y/eUhd31XR5j7Irib+hmRjT8qmEv2
Wvve6S9gWSa6P68ZqMyqM/F9gT6Od5afJDGKLkyq01wufT6Q2cQ1QV3k+4sSPLDF
+hTW5043pI6UWaW8tFxqtxLG3pSPt0IluWO8DzYSFqVHuNeX87flqT/oF1eQh/1/
4J7M0F9Ipp3y4nwZOY3aLPYYqCXOzEetZaufxGN50e1v99FtAgMBAAGjIzAhMA4G
A1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IB
AQB0CrRwiSEN0s2gM7f+VjoZtXi2RACbY8d76HVh3XxCISHPH3ThuBG3trl8kL0T
Ve+E65r1eFVRY13MuODzDruOMOR6zi88VQr5Ca6E+TpplR34mgOURCPi8569ktnS
eCz9X7BBZZUgIRTOE3yRw0jwrOOwsXukOKNsjPlO3yovbiZ6RdTPUg78LBtgDwDI
MjKCfYhZT6+oZ04i3XurwakajJozoFsk40Tp3dlR9KM5IdJ/jPAY3nbaXhblFyHN
9EYB18bLXAczjIG7NCp+aD5maBGgiAcHYSxRANAbvFsH3v8ieMdPSmn3Khgv5KoX
LzW1dF6ktvfclYk8LmzUb91G
-----END CERTIFICATE-----

2020-11-17T13:48:00.946121Z	info	rootcertrotator	Set up back off time 17m35s to start rotator.
2020-11-17T13:48:00.946276Z	info	initializing Istiod DNS certificates host: istiod.istio-system.svc, custom host:
2020-11-17T13:48:00.946290Z	info	Generating istiod-signed cert for [istiod.istio-system.svc istiod-remote.istio-system.svc istio-pilot.istio-system.svc]
2020-11-17T13:48:00.947023Z	info	rootcertrotator	Jitter is enabled, wait 17m35s before starting root cert rotator.
2020-11-17T13:48:01.082031Z	info	No plugged-in cert at etc/cacerts/ca-key.pem; self-signed cert is used
2020-11-17T13:48:01.082224Z	info	DNS certificates created in ./var/run/secrets/istio-dns
2020-11-17T13:48:01.082393Z	info	adding watcher for certificate var/run/secrets/istio-dns/cert-chain.pem
2020-11-17T13:48:01.082502Z	info	adding watcher for certificate var/run/secrets/istio-dns/key.pem
2020-11-17T13:48:01.082592Z	info	spiffe	Added 1 certs to trust domain cluster.local in peer cert verifier
2020-11-17T13:48:01.082613Z	info	initializing secure discovery service
2020-11-17T13:48:01.083406Z	info	initializing secure webhook server for istiod webhooks
2020-11-17T13:48:01.083426Z	info	initializing sidecar injector
2020-11-17T13:48:01.084178Z	info	initializing config validator
2020-11-17T13:48:01.084213Z	info	initializing Istiod admin server
2020-11-17T13:48:01.084319Z	info	initializing registry event handlers
2020-11-17T13:48:01.084376Z	info	starting discovery service
2020-11-17T13:48:01.084444Z	info	initializing Kubernetes cluster registry
2020-11-17T13:48:01.084483Z	info	Setting up event handlers
2020-11-17T13:48:01.084541Z	info	initializing DNS server
2020-11-17T13:48:01.086992Z	info	Staring Istiod Server with primary cluster Kubernetes
2020-11-17T13:48:01.100938Z	info	Starting Secrets controller
2020-11-17T13:48:01.101024Z	info	Waiting for informer caches to sync
2020-11-17T13:48:01.101151Z	info	ControlZ available at 127.0.0.1:9876
2020-11-17T13:48:01.101291Z	info	status	Starting status follower controller
2020-11-17T13:48:01.101448Z	info	attempting to acquire leader lease  istio-system/istio-leader...
2020-11-17T13:48:01.101645Z	info	kube	Starting Pilot K8S CRD controller
2020-11-17T13:48:01.101728Z	info	kube	controller "networking.istio.io/v1alpha3/ServiceEntry" is syncing...
2020-11-17T13:48:01.112141Z	info	ads	Starting ADS server
2020-11-17T13:48:01.112335Z	info	Started DNS-TLS[::]:15053
2020-11-17T13:48:01.112472Z	info	Started DNS :15053
2020-11-17T13:48:01.112519Z	info	staring CA
2020-11-17T13:48:01.112652Z	info	serverca	added client certificate authenticator
2020-11-17T13:48:01.112748Z	info	serverca	added K8s JWT authenticator
2020-11-17T13:48:01.113334Z	info	attempting to acquire leader lease  istio-system/istio-validation-controller-election...
2020-11-17T13:48:01.113466Z	info	Istiod CA has started
2020-11-17T13:48:01.113644Z	info	attempting to acquire leader lease  istio-system/istio-namespace-controller-election...
2020-11-17T13:48:01.148109Z	info	successfully acquired lease istio-system/istio-leader
2020-11-17T13:48:01.201846Z	info	kube	controller "config.istio.io/v1alpha2/HTTPAPISpec" is syncing...
2020-11-17T13:48:01.306783Z	info	kube	controller "config.istio.io/v1alpha2/HTTPAPISpec" is syncing...
2020-11-17T13:48:01.402037Z	info	kube	controller "networking.istio.io/v1alpha3/WorkloadEntry" is syncing...
2020-11-17T13:48:01.402515Z	info	ads	Full push, new service kube-dns.kube-system.svc.cluster.local
2020-11-17T13:48:01.402750Z	info	ads	Full push, new service calico-typha.calico-system.svc.cluster.local
2020-11-17T13:48:01.403006Z	info	ads	Full push, new service kubernetes.default.svc.cluster.local
2020-11-17T13:48:01.501851Z	info	kube	controller "config.istio.io/v1alpha2/HTTPAPISpec" is syncing...
2020-11-17T13:48:01.503735Z	info	ads	Push debounce stable[1] 20: 100.163984ms since last change, 101.363173ms since last push, full=true
2020-11-17T13:48:01.511968Z	info	ads	XDS: Pushing:2020-11-17T13:48:01Z/0 Services:4 ConnectedEndpoints:0
2020-11-17T13:48:01.587786Z	info	successfully acquired lease istio-system/istio-validation-controller-election
2020-11-17T13:48:01.588377Z	info	Starting validation controller
2020-11-17T13:48:01.601972Z	info	kube	controller "networking.istio.io/v1alpha3/Sidecar" is syncing...
2020-11-17T13:48:01.652377Z	info	successfully acquired lease istio-system/istio-namespace-controller-election
2020-11-17T13:48:01.652583Z	info	Starting namespace controller
2020-11-17T13:48:01.703642Z	info	kube	controller "config.istio.io/v1alpha2/HTTPAPISpec" is syncing...
2020-11-17T13:48:01.801983Z	info	kube	controller "networking.istio.io/v1alpha3/WorkloadEntry" is syncing...
2020-11-17T13:48:01.902028Z	info	kube	controller "networking.istio.io/v1alpha3/WorkloadEntry" is syncing...
2020-11-17T13:48:02.001990Z	info	kube	controller "security.istio.io/v1beta1/AuthorizationPolicy" is syncing...
2020-11-17T13:48:02.089388Z	info	validationController	Reconcile(enter): add event (v1, Kind=Endpoints) istio-system/istiod
2020-11-17T13:48:02.115606Z	info	All caches have been synced up, marking server ready
2020-11-17T13:48:02.116304Z	info	starting Http service at [::]:8080
2020-11-17T13:48:02.116965Z	info	starting secure gRPC discovery service at [::]:15012
2020-11-17T13:48:02.117240Z	info	starting gRPC discovery service at [::]:15010
2020-11-17T13:48:02.116924Z	info	starting webhook service at [::]:8080
2020-11-17T13:48:02.149309Z	info	Starting ingress controller
2020-11-17T13:48:02.149390Z	warn	Missing ingress, skip status updates
2020-11-17T13:48:02.154406Z	info	Namespace controller started
2020-11-17T13:48:02.160359Z	info	ads	Full push, new service istiod.istio-system.svc.cluster.local
2020-11-17T13:48:02.260542Z	info	ads	Push debounce stable[2] 9: 100.140711ms since last change, 158.431962ms since last push, full=true
2020-11-17T13:48:02.265652Z	info	ads	XDS: Pushing:2020-11-17T13:48:02Z/1 Services:4 ConnectedEndpoints:0
2020-11-17T13:48:03.422289Z	info	ads	Push debounce stable[3] 4: 100.159877ms since last change, 284.78587ms since last push, full=true
2020-11-17T13:48:03.422920Z	info	ads	XDS: Pushing:2020-11-17T13:48:03Z/2 Services:6 ConnectedEndpoints:0
2020-11-17T13:48:11.114079Z	info	ads	Push Status: {}
2020-11-17T13:48:17.939556Z	info	ads	Push debounce stable[4] 1: 100.250536ms since last change, 100.250305ms since last push, full=false
2020-11-17T13:48:17.939674Z	info	ads	XDS:EDSInc Pushing:2020-11-17T13:48:03Z/2 Services:map[istio-egressgateway.istio-system.svc.cluster.local:{}] ConnectedEndpoints:0
2020-11-17T13:48:18.944194Z	info	ads	Push debounce stable[5] 1: 100.279373ms since last change, 100.278966ms since last push, full=false
2020-11-17T13:48:18.944864Z	info	ads	XDS:EDSInc Pushing:2020-11-17T13:48:03Z/2 Services:map[istio-ingressgateway.istio-system.svc.cluster.local:{}] ConnectedEndpoints:0
2020-11-17T13:48:32.122822Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:48:32.162912Z	info	validationController	Successfully updated validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore,resourceVersion=1863)
2020-11-17T13:48:32.163145Z	info	validationController	Reconcile(enter): add event (admissionregistration.k8s.io/v1beta1, Kind=ValidatingWebhookConfiguration) istiod-istio-system
2020-11-17T13:48:33.154408Z	info	ads	Full push, service accounts changed, calico-typha.calico-system.svc.cluster.local
2020-11-17T13:48:33.254648Z	info	ads	Push debounce stable[6] 1: 100.179455ms since last change, 100.179243ms since last push, full=true
2020-11-17T13:48:33.255130Z	info	ads	XDS: Pushing:2020-11-17T13:48:33Z/3 Services:6 ConnectedEndpoints:0
2020-11-17T13:48:41.115260Z	info	ads	Push Status: {}
2020-11-17T13:48:41.185759Z	info	ads	Push debounce stable[7] 1: 100.16769ms since last change, 100.167492ms since last push, full=false
2020-11-17T13:48:41.185851Z	info	ads	XDS:EDSInc Pushing:2020-11-17T13:48:33Z/3 Services:map[calico-typha.calico-system.svc.cluster.local:{}] ConnectedEndpoints:0
2020-11-17T13:49:02.170973Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:49:02.171887Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:49:02.171913Z	info	validationController	Reconcile(enter): initial request to kickstart reconciliation
2020-11-17T13:49:32.179342Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:49:32.179896Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:49:32.179919Z	info	validationController	Reconcile(enter): update event (v1, Kind=Endpoints) istio-system/istiod
2020-11-17T13:50:02.186504Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:50:02.187563Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:50:02.187704Z	info	validationController	Reconcile(enter): update event (admissionregistration.k8s.io/v1beta1, Kind=ValidatingWebhookConfiguration) istiod-istio-system
2020-11-17T13:50:32.194764Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:50:32.195988Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:50:32.196017Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:51:02.204187Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:51:02.205110Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:51:02.205165Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:51:32.216104Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:51:32.217967Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:51:32.218235Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:52:02.227766Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:52:02.229296Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:52:02.229337Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:52:32.236812Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:52:32.237497Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:52:32.237585Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:53:02.244546Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:53:02.246271Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:53:02.246596Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:53:32.253628Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:53:32.254623Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:53:32.254784Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:54:02.262234Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:54:02.263592Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:54:02.263878Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:54:32.273496Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:54:32.274551Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:54:32.274589Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:55:02.284361Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:55:02.286149Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:55:02.286415Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:55:32.295245Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:55:32.296798Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:55:32.296823Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:56:02.305904Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:56:02.306295Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:56:02.306307Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:56:32.320853Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:56:32.322094Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:56:32.322128Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:57:02.334234Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:57:02.335082Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:57:02.335128Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:57:32.345600Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:57:32.346524Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:57:32.346632Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:58:02.356978Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:58:02.357823Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:58:02.357853Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:58:32.367263Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:58:32.367958Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:58:32.367986Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:59:02.375286Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:59:02.375685Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:59:02.375739Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T13:59:32.383978Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T13:59:32.384884Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T13:59:32.385075Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T14:00:02.392158Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T14:00:02.393282Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T14:00:02.393432Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T14:00:32.402538Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T14:00:32.403824Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T14:00:32.403880Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T14:01:02.415606Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T14:01:02.416276Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T14:01:02.416296Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T14:01:32.425831Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T14:01:32.426913Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T14:01:32.426968Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T14:02:02.436029Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T14:02:02.437078Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T14:02:02.437113Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
2020-11-17T14:02:32.444293Z	info	validationController	Not ready to switch validation to fail-closed: dummy invalid config not rejected
2020-11-17T14:02:32.445079Z	info	validationController	validatingwebhookconfiguration istiod-istio-system (failurePolicy=Ignore, resourceVersion=1863) is up-to-date. No change required.
2020-11-17T14:02:32.445126Z	info	validationController	Reconcile(enter): retry dry-run creation of invalid config
root@kub1:~#

ancyra · November 17, 2020, 2:09pm

istio-ingressgateway logs:

2020-11-17T13:48:18.205587Z	info	FLAG: --concurrency="0"
2020-11-17T13:48:18.205627Z	info	FLAG: --disableInternalTelemetry="false"
2020-11-17T13:48:18.205635Z	info	FLAG: --domain="istio-system.svc.cluster.local"
2020-11-17T13:48:18.205639Z	info	FLAG: --help="false"
2020-11-17T13:48:18.205650Z	info	FLAG: --id=""
2020-11-17T13:48:18.205857Z	info	FLAG: --ip=""
2020-11-17T13:48:18.205875Z	info	FLAG: --log_as_json="false"
2020-11-17T13:48:18.205878Z	info	FLAG: --log_caller=""
2020-11-17T13:48:18.205891Z	info	FLAG: --log_output_level="default:info"
2020-11-17T13:48:18.205894Z	info	FLAG: --log_rotate=""
2020-11-17T13:48:18.205898Z	info	FLAG: --log_rotate_max_age="30"
2020-11-17T13:48:18.205901Z	info	FLAG: --log_rotate_max_backups="1000"
2020-11-17T13:48:18.205989Z	info	FLAG: --log_rotate_max_size="104857600"
2020-11-17T13:48:18.206099Z	info	FLAG: --log_stacktrace_level="default:none"
2020-11-17T13:48:18.206138Z	info	FLAG: --log_target="[stdout]"
2020-11-17T13:48:18.206252Z	info	FLAG: --meshConfig="./etc/istio/config/mesh"
2020-11-17T13:48:18.206264Z	info	FLAG: --mixerIdentity=""
2020-11-17T13:48:18.206269Z	info	FLAG: --outlierLogPath=""
2020-11-17T13:48:18.206272Z	info	FLAG: --proxyComponentLogLevel="misc:error"
2020-11-17T13:48:18.206291Z	info	FLAG: --proxyLogLevel="warning"
2020-11-17T13:48:18.206296Z	info	FLAG: --serviceCluster="istio-ingressgateway"
2020-11-17T13:48:18.206300Z	info	FLAG: --serviceregistry="Kubernetes"
2020-11-17T13:48:18.206383Z	info	FLAG: --stsPort="0"
2020-11-17T13:48:18.206395Z	info	FLAG: --templateFile=""
2020-11-17T13:48:18.206399Z	info	FLAG: --tokenManagerPlugin="GoogleTokenExchange"
2020-11-17T13:48:18.206403Z	info	FLAG: --trust-domain="cluster.local"
2020-11-17T13:48:18.206479Z	info	Version 1.7.4-4ce531ff1823a3abb9f42fa9d35523b0436e2d04-Clean
2020-11-17T13:48:18.206673Z	info	Obtained private IP [192.168.154.194]
2020-11-17T13:48:18.207276Z	info	Apply mesh config from file accessLogFile: /dev/stdout
defaultConfig:
  discoveryAddress: istiod.istio-system.svc:15012
  proxyMetadata:
    DNS_AGENT: ""
  tracing:
    zipkin:
      address: zipkin.istio-system:9411
disableMixerHttpReports: true
enablePrometheusMerge: true
rootNamespace: istio-system
trustDomain: cluster.local
2020-11-17T13:48:18.208963Z	info	Effective config: binaryPath: /usr/local/bin/envoy
concurrency: 0
configPath: ./etc/istio/proxy
controlPlaneAuthPolicy: MUTUAL_TLS
discoveryAddress: istiod.istio-system.svc:15012
drainDuration: 45s
envoyAccessLogService: {}
envoyMetricsService: {}
parentShutdownDuration: 60s
proxyAdminPort: 15000
proxyMetadata:
  DNS_AGENT: ""
serviceCluster: istio-ingressgateway
statNameLength: 189
statusPort: 15020
terminationDrainDuration: 5s
tracing:
  zipkin:
    address: zipkin.istio-system:9411

2020-11-17T13:48:18.209168Z	info	Proxy role: &model.Proxy{Type:"router", IPAddresses:[]string{"192.168.154.194"}, ID:"istio-ingressgateway-67fc4949df-749c2.istio-system", Locality:(*envoy_config_core_v3.Locality)(nil), DNSDomain:"istio-system.svc.cluster.local", ConfigNamespace:"", Metadata:(*model.NodeMetadata)(nil), SidecarScope:(*model.SidecarScope)(nil), PrevSidecarScope:(*model.SidecarScope)(nil), MergedGateway:(*model.MergedGateway)(nil), ServiceInstances:[]*model.ServiceInstance(nil), IstioVersion:(*model.IstioVersion)(nil), ipv6Support:false, ipv4Support:false, GlobalUnicastIP:"", XdsResourceGenerator:model.XdsResourceGenerator(nil), Active:map[string]*model.WatchedResource(nil), ActiveExperimental:map[string]*model.WatchedResource(nil), RequestedTypes:struct { CDS string; EDS string; RDS string; LDS string }{CDS:"", EDS:"", RDS:"", LDS:""}}
2020-11-17T13:48:18.209184Z	info	JWT policy is first-party-jwt
2020-11-17T13:48:18.209337Z	info	PilotSAN []string{"istiod.istio-system.svc"}
2020-11-17T13:48:18.209350Z	info	MixerSAN []string{"spiffe://cluster.local/ns/istio-system/sa/istio-mixer-service-account"}
2020-11-17T13:48:18.209416Z	info	sa.serverOptions.CAEndpoint == istiod.istio-system.svc:15012
2020-11-17T13:48:18.209429Z	info	Using user-configured CA istiod.istio-system.svc:15012
2020-11-17T13:48:18.209432Z	info	istiod uses self-issued certificate
2020-11-17T13:48:18.209819Z	info	the CA cert of istiod is: -----BEGIN CERTIFICATE-----
MIIC3jCCAcagAwIBAgIRAKgS6yTpUKcoIkqT4CvBI6owDQYJKoZIhvcNAQELBQAw
GDEWMBQGA1UEChMNY2x1c3Rlci5sb2NhbDAeFw0yMDExMTcxMzQ4MDBaFw0zMDEx
MTUxMzQ4MDBaMBgxFjAUBgNVBAoTDWNsdXN0ZXIubG9jYWwwggEiMA0GCSqGSIb3
DQEBAQUAA4IBDwAwggEKAoIBAQCzFT4boDNUaP00FbbTUs4kf+x+T0nTcxoan4Og
HhhWMfRSplZjN9nxb7UOGTwbt5N9iZ/LXVejSncJx69RceOqyrZbU+tBh+z6GS8Z
52YCi+qOtOM66AI1rwXhILqE4Myyo2C9jg5y/eUhd31XR5j7Irib+hmRjT8qmEv2
Wvve6S9gWSa6P68ZqMyqM/F9gT6Od5afJDGKLkyq01wufT6Q2cQ1QV3k+4sSPLDF
+hTW5043pI6UWaW8tFxqtxLG3pSPt0IluWO8DzYSFqVHuNeX87flqT/oF1eQh/1/
4J7M0F9Ipp3y4nwZOY3aLPYYqCXOzEetZaufxGN50e1v99FtAgMBAAGjIzAhMA4G
A1UdDwEB/wQEAwICBDAPBgNVHRMBAf8EBTADAQH/MA0GCSqGSIb3DQEBCwUAA4IB
AQB0CrRwiSEN0s2gM7f+VjoZtXi2RACbY8d76HVh3XxCISHPH3ThuBG3trl8kL0T
Ve+E65r1eFVRY13MuODzDruOMOR6zi88VQr5Ca6E+TpplR34mgOURCPi8569ktnS
eCz9X7BBZZUgIRTOE3yRw0jwrOOwsXukOKNsjPlO3yovbiZ6RdTPUg78LBtgDwDI
MjKCfYhZT6+oZ04i3XurwakajJozoFsk40Tp3dlR9KM5IdJ/jPAY3nbaXhblFyHN
9EYB18bLXAczjIG7NCp+aD5maBGgiAcHYSxRANAbvFsH3v8ieMdPSmn3Khgv5KoX
LzW1dF6ktvfclYk8LmzUb91G
-----END CERTIFICATE-----

2020-11-17T13:48:18.248208Z	info	Starting gateway SDS
2020-11-17T13:48:18.349861Z	info	sds	SDS gRPC server for workload UDS starts, listening on "./etc/istio/proxy/SDS"

2020-11-17T13:48:18.349981Z	info	sds	SDS gRPC server for gateway controller starts, listening on "./var/run/ingress_gateway/sds"

2020-11-17T13:48:18.350038Z	info	sds	Start SDS grpc server
2020-11-17T13:48:18.350081Z	info	Starting proxy agent
2020-11-17T13:48:18.350223Z	info	Opening status port 15020

2020-11-17T13:48:18.350276Z	info	Received new config, creating new Envoy epoch 0
2020-11-17T13:48:18.350333Z	info	Epoch 0 starting
2020-11-17T13:48:18.350724Z	info	sds	Start SDS grpc server for ingress gateway proxy
2020-11-17T13:48:20.353301Z	info	Envoy command: [-c etc/istio/proxy/envoy-rev0.json --restart-epoch 0 --drain-time-s 45 --parent-shutdown-time-s 60 --service-cluster istio-ingressgateway --service-node router~192.168.154.194~istio-ingressgateway-67fc4949df-749c2.istio-system~istio-system.svc.cluster.local --local-address-ip-version v4 --log-format-prefix-with-location 0 --log-format %Y-%m-%dT%T.%fZ	%l	envoy %n	%v -l warning --component-log-level misc:error]
2020-11-17T13:48:20.381930Z	warning	envoy runtime	Unable to use runtime singleton for feature envoy.reloadable_features.activate_fds_next_event_loop
2020-11-17T13:48:20.411039Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:48:20.411079Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:48:21.327918Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:48:21.327953Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:48:21.948533Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:48:21.948574Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:48:30.556033Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:48:30.556066Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:48:52.946589Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:48:52.946621Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:49:15.704523Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:49:15.704554Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:49:23.195807Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:49:23.195852Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:49:35.422209Z	info	sds	resource:default new connection
2020-11-17T13:49:35.422311Z	info	sds	Skipping waiting for gateway secret
2020-11-17T13:49:44.117549Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:44.167701Z	warn	cache	resource:default request:4c725bc2-69b5-4e04-9dcb-a140272ce909 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 50 millisec
2020-11-17T13:49:44.167787Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:44.243881Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:49:44.243905Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:49:44.267901Z	warn	cache	resource:default request:4c725bc2-69b5-4e04-9dcb-a140272ce909 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 100 millisec
2020-11-17T13:49:44.267978Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:44.468119Z	warn	cache	resource:default request:4c725bc2-69b5-4e04-9dcb-a140272ce909 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 200 millisec
2020-11-17T13:49:44.468222Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:44.868361Z	warn	cache	resource:default request:4c725bc2-69b5-4e04-9dcb-a140272ce909 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 400 millisec
2020-11-17T13:49:44.868503Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:45.668640Z	warn	cache	resource:default request:4c725bc2-69b5-4e04-9dcb-a140272ce909 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 800 millisec
2020-11-17T13:49:45.668717Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:45.668730Z	error	cache	resource:default request:4c725bc2-69b5-4e04-9dcb-a140272ce909 CSR retrial timed out: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:45.668739Z	error	cache	resource:default failed to generate secret for proxy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:45.668748Z	error	sds	resource:default Close connection. Failed to get secret for proxy "router~192.168.154.194~istio-ingressgateway-67fc4949df-749c2.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:45.668822Z	info	sds	resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2020-11-17T13:49:45.669029Z	warning	envoy config	StreamSecrets gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:45.846074Z	info	sds	resource:default new connection
2020-11-17T13:49:45.846212Z	info	sds	Skipping waiting for gateway secret
2020-11-17T13:49:46.083701Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:46.133899Z	warn	cache	resource:default request:60e7c5f2-7c60-4035-9a27-163246733603 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 50 millisec
2020-11-17T13:49:46.134079Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:46.234332Z	warn	cache	resource:default request:60e7c5f2-7c60-4035-9a27-163246733603 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 100 millisec
2020-11-17T13:49:46.234414Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:46.434570Z	warn	cache	resource:default request:60e7c5f2-7c60-4035-9a27-163246733603 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 200 millisec
2020-11-17T13:49:46.434734Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:46.834962Z	warn	cache	resource:default request:60e7c5f2-7c60-4035-9a27-163246733603 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 400 millisec
2020-11-17T13:49:46.835040Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:49:47.635185Z	warn	cache	resource:default request:60e7c5f2-7c60-4035-9a27-163246733603 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 800 millisec
2020-11-17T13:49:55.167896Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:49:55.167924Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:49:55.672597Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:49:55.672634Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:50:07.450482Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:07.450526Z	error	cache	resource:default request:60e7c5f2-7c60-4035-9a27-163246733603 CSR retrial timed out: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:07.450754Z	error	cache	resource:default failed to generate secret for proxy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:07.450852Z	error	sds	resource:default Close connection. Failed to get secret for proxy "router~192.168.154.194~istio-ingressgateway-67fc4949df-749c2.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:07.450981Z	info	sds	resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2020-11-17T13:50:07.451169Z	warning	envoy config	StreamSecrets gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:07.530839Z	info	sds	resource:default new connection
2020-11-17T13:50:07.530951Z	info	sds	Skipping waiting for gateway secret
2020-11-17T13:50:08.099545Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:08.149698Z	warn	cache	resource:default request:6a94b18a-312d-4da0-9a95-c7ba1d8999bb CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 50 millisec
2020-11-17T13:50:08.149793Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:08.249984Z	warn	cache	resource:default request:6a94b18a-312d-4da0-9a95-c7ba1d8999bb CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 100 millisec
2020-11-17T13:50:08.250108Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:08.450265Z	warn	cache	resource:default request:6a94b18a-312d-4da0-9a95-c7ba1d8999bb CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 200 millisec
2020-11-17T13:50:08.450559Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:08.850772Z	warn	cache	resource:default request:6a94b18a-312d-4da0-9a95-c7ba1d8999bb CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 400 millisec
2020-11-17T13:50:08.850884Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:09.651060Z	warn	cache	resource:default request:6a94b18a-312d-4da0-9a95-c7ba1d8999bb CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 800 millisec
2020-11-17T13:50:09.651140Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:11.251346Z	warn	cache	resource:default request:6a94b18a-312d-4da0-9a95-c7ba1d8999bb CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 1600 millisec
2020-11-17T13:50:11.251546Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:14.451772Z	warn	cache	resource:default request:6a94b18a-312d-4da0-9a95-c7ba1d8999bb CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 3200 millisec
2020-11-17T13:50:23.555851Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:50:23.555882Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:50:34.175365Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:34.175417Z	error	cache	resource:default request:6a94b18a-312d-4da0-9a95-c7ba1d8999bb CSR retrial timed out: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:34.175448Z	error	cache	resource:default failed to generate secret for proxy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:34.175464Z	error	sds	resource:default Close connection. Failed to get secret for proxy "router~192.168.154.194~istio-ingressgateway-67fc4949df-749c2.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:34.175541Z	info	sds	resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2020-11-17T13:50:34.175817Z	warning	envoy config	StreamSecrets gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:34.936707Z	info	sds	resource:default new connection
2020-11-17T13:50:34.936789Z	info	sds	Skipping waiting for gateway secret
2020-11-17T13:50:35.045375Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:35.095547Z	warn	cache	resource:default request:378b5bc0-eaf2-4d60-b47e-cd9039d26f65 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 50 millisec
2020-11-17T13:50:35.095787Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:35.196015Z	warn	cache	resource:default request:378b5bc0-eaf2-4d60-b47e-cd9039d26f65 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 100 millisec
2020-11-17T13:50:35.196228Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:35.396456Z	warn	cache	resource:default request:378b5bc0-eaf2-4d60-b47e-cd9039d26f65 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 200 millisec
2020-11-17T13:50:35.396643Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:35.796895Z	warn	cache	resource:default request:378b5bc0-eaf2-4d60-b47e-cd9039d26f65 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 400 millisec
2020-11-17T13:50:35.797132Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:36.597330Z	warn	cache	resource:default request:378b5bc0-eaf2-4d60-b47e-cd9039d26f65 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 800 millisec
2020-11-17T13:50:36.597546Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:38.197697Z	warn	cache	resource:default request:378b5bc0-eaf2-4d60-b47e-cd9039d26f65 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 1600 millisec
2020-11-17T13:50:38.197840Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:41.398001Z	warn	cache	resource:default request:378b5bc0-eaf2-4d60-b47e-cd9039d26f65 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 3200 millisec
2020-11-17T13:50:41.398119Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:50:42.448311Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:50:42.448455Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:50:47.798253Z	warn	cache	resource:default request:378b5bc0-eaf2-4d60-b47e-cd9039d26f65 CSR failed with error: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout", retry in 6400 millisec
2020-11-17T13:50:54.390476Z	warning	envoy config	StreamAggregatedResources gRPC config stream closed: 14, no healthy upstream
2020-11-17T13:50:54.390515Z	warning	envoy config	Unable to establish new stream
2020-11-17T13:51:02.942099Z	error	citadelclient	Failed to create certificate: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:51:02.942152Z	error	cache	resource:default request:378b5bc0-eaf2-4d60-b47e-cd9039d26f65 CSR retrial timed out: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:51:02.942179Z	error	cache	resource:default failed to generate secret for proxy: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:51:02.942193Z	error	sds	resource:default Close connection. Failed to get secret for proxy "router~192.168.154.194~istio-ingressgateway-67fc4949df-749c2.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unavailable desc = connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"
2020-11-17T13:51:02.942252Z	info	sds	resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
2020-11-17T13:51:02.942998Z	warning	envoy config	StreamSecrets gRPC config stream closed: 14, connection error: desc = "transport: Error while dialing dial tcp: i/o timeout"

ancyra · November 17, 2020, 8:59pm

I switched from Calico to weave net. It worked.

Topic		Replies	Views
Demo install issue on k8s bare metal cluster with calico and crio	3	2068	July 7, 2021
Trying to install Istio 1.10.0 but istiod fails	0	529	November 18, 2021
Kubernetes v1.16.2 and istio v1.3.4	0	345	November 5, 2019
Istio 1.10.4 ingress and egress pods are not ready Networking release	0	398	September 21, 2021
Istio v1.3+ components/ ingress gateways are not getting installed on a private GKE cluster	0	590	February 17, 2020

Istio installation fails on v1.19.4

Related Topics