@seeruk, would you also be able to share a scrubbed version of the values.yaml that resulted in an incomplete IstioOperator CR?
Hey all!
I would like to ask @ostromart’s insight on what happens when someone updates Istio Helm installation via Istioctl. The Installation guide only says not to use Istioctl for update in case your deployment is installed by Helm. It doesn’t mention why (or at least I did not find any traces). I am asking because I may have found a way to migrate from Helm -> Istioctl without downtime.
Steps:
- Start with an Istio 1.5.8 Helm deployment
- Create an IstioOperator YAML containing the values you have used for the Helm install (–set variables)
- Run
istioctl upgrade --force -f update-values.yaml(because of the mismatching versions, we have to force the upgrade) (Istioctl is version 1.6) - Delete ValidatingWebhookConfiguration:
kubectl delete ValidatingWebhookConfiguration istio-galley -n istio-system - Now we are free to delete the old istio-telemetry, istio-policy, istio-sidecar-injector, istio-system istio-citadel, istio-galley, istio-pilot deployments / services / HPAs.
- Delete:
kubectl delete crd clusterrbacconfigs.rbac.istio.io -n istio-system
kubectl delete crd -n istio-system meshpolicies.authentication.istio.io policies.authentication.istio.io
kubectl delete crd -n istio-system rbacconfigs.rbac.istio.io servicerolebindings.rbac.istio.io serviceroles.rbac.istio.io - Run
istioctl upgrade --force -f update-values.yamlagain - You have Istio 1.6 installed with Istioctl.
I have finished these steps in a test cluster with zero downtime. I would like to ask for a sanity check that you may point to possible errors and mistakes that I don’t yet see.
The way I see, the only disadvantage using above steps is there can be som leftover and deprecated CRD / configmaps / secrets / etc stay in the cluster. What do you think?
I’m facing what I think are slightly different issues migrating from a 1.5 helm install to a 1.6 istioctl install: Gateways.
We’re trying to follow the canary “upgrade” process for migrating from helm to istioctl. However there’s no canary option for gateways. If you name your gateways in the IstioOperator yaml the same name as the 1.5 gateways, it upgrades them in place (I found an issue in GitHub saying this is expected because we can install multiple gateways). This might be okay except for the fact that (maybe it’s our specific configuration) all inbound requests to the gateway respond with an HTTP 400 when trying to route traffic to the workloads in the cluster running a 1.5 sidecar. I haven’t quite figured out why this is broken yet.
We have 7 different gateways in our cluster and our applications around bound to various gateway labels for different use cases. Installing 7 brand new 1.6 gateways with new labels and then moving all our applications to them will be a TON of work for us. We have around 75 applications we would have to migrate in our staging cluster alone. Additionally, we have production DNS names pointed at AWS ELBs for the existing gateways. Moving apps to a new gateway would cause those apps to stop accepting traffic on the existing DNS name.
Is there any kind of recommendation for this migration process? We have to have a way for zero downtime upgrades.
@peter your process looks sound but some caveats:
- the helm charts have to be the “new” Helm charts that istioctl is also using internally (the ones under manifests/, not the ones under install/kubernetes/helm).
- don’t run with --force the first time until you’re sure the only error is the version. You can use --dry-run to test.
In general, while the canary approach is a bit more complex than in-place, it’s safer because it allows for just a few workloads to use the new version and makes rollback much easier if anything goes wrong.
I tried to migrate our values file via istioctl 1.6.5 but it did not translate the SDS parameters. So, I migrated the values file via istioctl 1.5 to an IstioControlPlane. When I try to use this as input to istioctl manifest migrate (because the help text says I can), I get inconsistent errors (i.e. first it complains about kind, the next run complains about metadata, etc):
~/istio165$ istio-1.6.5/bin/istioctl manifest migrate values-istio15.yaml --dry-run
Error: the input values.yaml fail validation: unknown field "kind" in v1alpha1.Values
check against https://github.com/istio/istio/blob/2511ab8c8c59a203e77bb804846593c3690fcf4a/operator/pkg/apis/istio/v1alpha1/values_types.proto for schema
or run the command with --force flag to ignore the error
~/istio165$ istio-1.6.5/bin/istioctl manifest migrate values-istio15.yaml --dry-run
Error: the input values.yaml fail validation: unknown field "metadata" in v1alpha1.Values
check against https://github.com/istio/istio/blob/2511ab8c8c59a203e77bb804846593c3690fcf4a/operator/pkg/apis/istio/v1alpha1/values_types.proto for schema
or run the command with --force flag to ignore the error
~/istio165$ istio-1.6.5/bin/istioctl manifest migrate values-istio15.yaml --dry-run
Error: the input values.yaml fail validation: unknown field "apiVersion" in v1alpha1.Values
check against https://github.com/istio/istio/blob/2511ab8c8c59a203e77bb804846593c3690fcf4a/operator/pkg/apis/istio/v1alpha1/values_types.proto for schema
or run the command with --force flag to ignore the error
~/istio165$ istio-1.6.5/bin/istioctl manifest migrate values-istio15.yaml --dry-run
Error: the input values.yaml fail validation: unknown field "kind" in v1alpha1.Values
check against https://github.com/istio/istio/blob/2511ab8c8c59a203e77bb804846593c3690fcf4a/operator/pkg/apis/istio/v1alpha1/values_types.proto for schema
or run the command with --force flag to ignore the error
~/istio165$ istio-1.6.5/bin/istioctl manifest migrate values-istio15.yaml --dry-run
Error: the input values.yaml fail validation: unknown field "spec" in v1alpha1.Values
check against https://github.com/istio/istio/blob/2511ab8c8c59a203e77bb804846593c3690fcf4a/operator/pkg/apis/istio/v1alpha1/values_types.proto for schema
or run the command with --force flag to ignore the error
~/istio165$
Also, the --force option did not seem to work:
~/istio165$ istio-1.6.5/bin/istioctl manifest migrate values-istio.yaml --force --dry-run
Error: the input values.yaml fail validation: unknown field "namespace" in v1alpha1.IngressGatewayConfig
check against https://github.com/istio/istio/blob/2511ab8c8c59a203e77bb804846593c3690fcf4a/operator/pkg/apis/istio/v1alpha1/values_types.proto for schema
or run the command with --force flag to ignore the error
~/istio165$
Suggestions?
@tshort istioctl manifest migrate did not worked for me. If you have Helm’s values.yaml, I would recommand to convert them to IstioOprator manually. Here’s a blog which describes the steps I took to upgrade my installation. You will find an example manifest there to assist you in conversion.
I think my original migration was done using istioctl 1.4, which created an IstioControlPlane. Trying to migrate that did not work. I subsequently tried migrating the values file via istioctl 1.5.7 which gave me an IstioOperator, and it mostly works, except we use certificates from Vault for SDS, and the Istio documentation deleted the integration examples for Vault.
@tshort, could you please create an issue? Please put as much detail as you can, including versions and values.yaml you used as inputs.