1.6 upgrade path from Helm chart

#1

Hi,

We are using Istio 1.5.2 installed via the “legacy Helm chart”. I’m trying to find an upgrade path to 1.6. I understand this is best done via canary as the existing Helm charts don’t have a direct upgrade path to istioctl/istio-operator.

I’ve tried using istioctl and the Istio operator, but with both they create resources that have naming conflicts with the existing Helm charts. When I try to delete the old Helm chart, those resources get deleted. Specifically, the istio-ingressgateway deployment, and some service accounts.

How am I supposed to do this in a zero-downtime manner? Are there docs that detail how to do this?

5 Likes
Failed 1.6 Upgrade from Helm install
#2

I assume you followed the steps in https://istio.io/docs/setup/upgrade/?
Can you provide more detail on what commands you ran and what happened?

#3

@David_Norton Curious if you made any progress toward Istio 1.6. We’re still on 1.4.9 helm installed and have been using the helm upgrade process since 1.2 without issues, when I’ve made attempts to migrate beyond 1.4.x it did not go well I wasn’t even able to upgrade as far as you have with the legacy helm chart. I am using Helm 3.

#4

I had similar issue while upgrading from 1.4.3 to 1.5.4. But i resolved by creating a manifest file that resembles the existing setup of 1.4.3. By mentioning the desired version to upgrade in manifest file, i managed to upgrade my application using istioctl without any downtime.

You can find the samples for manifest in istio installation directory.

Hope this helps.

#5

@David_Norton any progress there?

I’m in a similar situation, I need to go from 1.5.4 installed using helm charts (so with all the legacy pods, no istiod) to 1.7.1.

I saw some document that explain how to perform a canary upgrade but no one explains how to deal with:

  • configurations: how these are consumed by the new control plane? Are old config compatible with new control planes?
  • what happens to the ingress controller when installing the second control plane? How do I “attach” the ingress controller to the new control plane?

@ostromart can you comment on this?

#6

Istioctl upgrade checks the schema so it will alert you to any incompatible fields. Helm doesn’t do this so you pretty much have to go through manually and check for deprecated fields. An workaround is to pass your Helm config to istioctl manifest generate under the spec.values passthrough API and see if you get any errors.
When you say ingress controller which specific resource are you referring to? istiod is the ingress controller by default.

#7

Hi @ostromart,
thank you for the answer.

Sorry, I meant istio-ingressgateway.

Cheers
Nicola

#8

I actually attempted a Canary on an existing cluster running 1.4.9 with version 1.6.10. I basically rewrote the helm config into an istio-operator config using an identical configuration for the ingress-gateways etc. I then deployed the canary, which replaces the gateways with the 1.6.10 gateway and deploys the canary of istio-d. After that I relabeled all of my namespaces with the canary revision tag and restarted all of my pods. Everything seemed great and tested ok so I manually removed all of my 1.4.9 assets and the helm secrets in the namespace. Everything still seemed great, until a deployment the next day pushed out new virtual-service config the following day, then I lost all ingress to one of my gateways. I realized my gateways didn’t have SDS in the 1.6.10 gateways and I couldn’t find anyway to mirror that configuration in 1.6, so I ended up pulling everything and rolling back and have been stuck on 1.4.9 since. This is the error my gateway that basically was unable to talk to any services under it.

|2020-09-28T17:44:25.996687Z|warning|envoy config|[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamSecrets gRPC config stream closed: 16, request authenticate failure|
|---|---|---|---|
|2020-09-28T17:44:28.115769Z|info|sds|resource:default new connection|
|2020-09-28T17:44:28.115925Z|info|sds|Skipping waiting for ingress gateway secret|
|2020-09-28T17:44:28.426937Z|error|citadelclient|Failed to create certificate: rpc error: code = Unauthenticated desc = request authenticate failure|
|2020-09-28T17:44:28.426986Z|error|cache|resource:default request:4efd50bb-5705-44a2-81e0-4bd71ce0cd13 CSR hit non-retryable error (HTTP code: 0). Error: rpc error: code = Unauthenticated desc = request authenticate failure|
|2020-09-28T17:44:28.427020Z|error|cache|resource:default failed to generate secret for proxy: rpc error: code = Unauthenticated desc = request authenticate failure|
|2020-09-28T17:44:28.427040Z|error|sds|resource:default Close connection. Failed to get secret for proxy "router~172.16.10.235~istio-internal-ingressgateway-5b69f8cb69-xvmd8.istio-system~istio-system.svc.cluster.local" from secret cache: rpc error: code = Unauthenticated desc = request authenticate failure|
|2020-09-28T17:44:28.427155Z|info|sds|resource:default connection is terminated: rpc error: code = Canceled desc = context canceled|
|2020-09-28T17:44:28.427409Z|warning|envoy config|[bazel-out/k8-opt/bin/external/envoy/source/common/config/_virtual_includes/grpc_stream_lib/common/config/grpc_stream.h:92] StreamSecrets gRPC config stream closed: 16, request authenticate failure|