Failed 1.6 Upgrade from Helm install

I wasn’t sure whether to create a new topic or hijack the other (1.6 upgrade path from Helm chart) …

I tried a similar process using the steps indicated in your link, attempting a canary install of 1.6 alongside a Helm installed 1.5.4.

The install failed with the following output:

KUBE-NONPROD: sdadmin@us-prod-provision-01:~/Kubernetes/Software/Istio/istio-1.6.0$ ./bin/istioctl install --set revision=test -f ../../../Nonprod/IstioUpgrade/nonprod-profile.yaml
Detected that your cluster does not support third party JWT authentication. Falling back to less secure first party JWT. See https://istio.io/docs/ops/best-practices/security/#configure-third-party-service-account-tokens for details.
✔ Istio core installed                                                                                                                                                                                                                                                                             
✘ Istiod encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition                                                                                                                                                                    
Deployment/istio-system/istiod-test                                                                                                                                 
✘ Addons encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition-system/prometheus                                                                                                                                                  
Deployment/istio-system/prometheus                                                                                                                                  
✘ Ingress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition                                                                                                                                                          
Deployment/istio-system/istio-ingressgateway                                                                                                              
- Pruning removed resources
Error: failed to apply manifests: errors occurred during operation

The istiod-test pod becomes stuck in a ContainerCreating state due to a FailedMount due to Unable to attach or mount volumes: unmounted volumes=[istio-token], unattached volumes=[cacerts inject istiod-service-account-token-m28v2 config-volume istio-token local-certs]: timed out waiting for the condition

The same is true of the istio-ingressgateway: Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert istio-token], unattached volumes=[istiod-ca-cert ingressgateway-certs istio-token ingressgatewaysdsudspath podinfo istio-envoy config-volume ingressgateway-ca-certs istio-ingressgateway-service-account-token-r88zd]: timed out waiting for the condition

and prometheus container Unable to attach or mount volumes: unmounted volumes=[istiod-ca-cert istio-token], unattached volumes=[config-volume istio-certs prometheus-token-6jb2j istiod-ca-cert istio-envoy istio-token istio-config-volume]: timed out waiting for the condition

The only container that started successfully was kiali pod, which appears to have no dependency on either of those mounts.

I am unsure if this is due to there being no Istio Operator / istiod present in my 1.5.4 install, however it is unclear to me how to provision it.

I was hoping to understand the correct process for performing this upgrade with as little downtime as possible… zero being ideal.

Regards,

mgh

Hi
i have this same type of error when i was doing in-place upgrade. on my local cluster (minikube) from istio1.5.2 to istio1.6.0.
This may be an api-server issue. see this issue https://github.com/istio/istio/issues/17378#issuecomment-604429711
i followed this step for minikube. there also step mention for kubespray & for kops. see the issue.

May be this will help you .

1 Like

Im also facing same issue. Trying to do a fresh installation of version 1.6 but it is failing
.
Platform: Amazon EKS v 1.15
:heavy_check_mark: Istio core installed
✘ Istiod encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the condition
Deployment/istio-system/istiod
✘ Egress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the conditionem/istio-egressgateway, Deplo…
Deployment/istio-system/istio-egressgateway
✘ Ingress gateways encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the conditionteway, Deployment/istio-syst…
Deployment/istio-system/istio-ingressgateway
✘ Addons encountered an error: failed to wait for resource: resources not ready after 5m0s: timed out waiting for the conditionnt/istio-system/kiali, Deployment/isti…
Deployment/istio-system/grafana
Deployment/istio-system/istio-tracing
Deployment/istio-system/kiali
Deployment/istio-system/prometheus

  • Pruning removed resources Error: failed to apply manifests: errors occurred during operation

Please do let me know how to solve it.

Thanks very much Shubham, this appears to have been the issue, the issue linked led me to https://forums.rancher.com/t/path-to-service-account-key-files/16287 which contains the necessary configurations.

Regards,

mgh