Hi Experts,
I have deployed Istio v1.5.2 (default profile) using the helm template option in my local IPv6 k8s cluster. The deployment is fine and I see that all the pods are in Running state. However, when I delete the istio-ingressgateway or istio-egressgateway pod, it takes around 8 minutes for the container to get to Ready (1/1) state.
This happens most of the times. Only very few times, the container comes to Ready state in just few seconds. I have tried this in different IPv6 k8s clusters and I observe the same behavior. Interestingly, I do not see this happening in the IPv4 environment.
Here is the screenshot showing pod status in the istio-system namespace.
(I see Ready containers exactly after 8 minutes)
Here is the info I get when I describe the pod while it is restarting after initiating kubectl delete.
Containers:
istio-proxy:
Container ID: docker://4eb2ee97f00dcec2f8d04209b6b535b2412a05414aa33699573320ff70333a61
Image: docker.io/istio/proxyv2:1.5.2
Image ID: docker-pullable://istio/proxyv2@sha256:b569b3226624b63ca007f3252162390545643433770c79a9aadb1406687d767a
Ports: 80/TCP, 443/TCP, 15443/TCP, 15090/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP, 0/TCP
Args:
proxy
router
--domain
$(POD_NAMESPACE).svc.cluster.local
--log_output_level=default:info
--drainDuration
45s
--parentShutdownDuration
1m0s
--connectTimeout
10s
--serviceCluster
istio-egressgateway
--zipkinAddress
zipkin:9411
--proxyAdminPort
15000
--statusPort
15020
--controlPlaneAuthPolicy
NONE
--discoveryAddress
istio-pilot:15010
State: Running
Started: Thu, 14 May 2020 23:55:14 +0200
Ready: False
Restart Count: 0
Limits:
cpu: 2
memory: 1Gi
Requests:
cpu: 10m
memory: 40Mi
Readiness: http-get http://:15020/healthz/ready delay=1s timeout=1s period=2s #success=1 #failure=30
Environment:
NODE_NAME: (v1:spec.nodeName)
POD_NAME: istio-egressgateway-6df7d766d9-2nbfp (v1:metadata.name)
POD_NAMESPACE: istio-system (v1:metadata.namespace)
INSTANCE_IP: (v1:status.podIP)
HOST_IP: (v1:status.hostIP)
SERVICE_ACCOUNT: (v1:spec.serviceAccountName)
ISTIO_AUTO_MTLS_ENABLED: true
ISTIO_META_POD_NAME: istio-egressgateway-6df7d766d9-2nbfp (v1:metadata.name)
ISTIO_META_CONFIG_NAMESPACE: istio-system (v1:metadata.namespace)
ISTIO_METAJSON_LABELS: {"app":"istio-egressgateway","chart":"gateways","heritage":"Tiller","istio":"egressgateway","release":"istio"}
ISTIO_META_CLUSTER_ID: Kubernetes
SDS_ENABLED: false
ISTIO_META_WORKLOAD_NAME: istio-egressgateway
ISTIO_META_OWNER: kubernetes://apis/apps/v1/namespaces/istio-system/deployments/istio-egressgateway
ISTIO_META_ROUTER_MODE: standard
Mounts:
/etc/certs from istio-certs (ro)
/etc/istio/egressgateway-ca-certs from egressgateway-ca-certs (ro)
/etc/istio/egressgateway-certs from egressgateway-certs (ro)
/var/run/secrets/kubernetes.io/serviceaccount from istio-egressgateway-service-account-token-wlb2g (ro)
Conditions:
Type Status
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
istio-certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio.istio-egressgateway-service-account
Optional: true
egressgateway-certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio-egressgateway-certs
Optional: true
egressgateway-ca-certs:
Type: Secret (a volume populated by a Secret)
SecretName: istio-egressgateway-ca-certs
Optional: true
istio-egressgateway-service-account-token-wlb2g:
Type: Secret (a volume populated by a Secret)
SecretName: istio-egressgateway-service-account-token-wlb2g
Optional: false
QoS Class: Burstable
Node-Selectors: <none>
Tolerations: node.kubernetes.io/not-ready:NoExecute for 300s
node.kubernetes.io/unreachable:NoExecute for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled <unknown> default-scheduler Successfully assigned istio-system/istio-egressgateway-6df7d766d9-2nbfp to mastermint
Normal Pulled 77s kubelet, mastermint Container image "docker.io/istio/proxyv2:1.5.2" already present on machine
Normal Created 77s kubelet, mastermint Created container istio-proxy
Normal Started 76s kubelet, mastermint Started container istio-proxy
Warning Unhealthy 32s (x22 over 74s) kubelet, mastermint Readiness probe failed: HTTP probe failed with statuscode: 503
Any idea why this would be happening?
Thanks in advance!
Best Regards,
sha-rath