"service "istiod" not found" during canary upgrade 1.5 -> 1.6 -> 1.7

After upgrading istio from 1.5 to 1.6 and 1.7 with canary mode, I have the 3 istio version in my cluster.
But then I had the following error when trying to execute the istioctl 1.7 install again :

And I see that I have now 3 services in the cluster :
istiod , istiod-1-6-x and istiod-1-7-x
To bypass the above error, I tried to delete the “legacy” istiod from the cluster (to keep only the 1.6 and 1.7 revisions) but then the istioctl 1.7 install gave me this error :

2020-12-04T14:07:42.892106Z error installer Internal error occurred: failed calling webhook “validation.istio.io”: Post https://istiod.istio-system.svc:443/validate?timeout=30s: service “istiod” not found

Can you explain me why this “legacy” istiod service is still needed when installing the new revision please ?

i believe you are getting blocked by the istio validatingwebhookconfiguration. take a look at them here and you might need to delete the old ones as well if there is multiple kubectl get validatingwebhookconfiguration -o yaml

Hello @nick_tetrate and thank you for your help.
Yes it is linked to the validatingwebhook.
Here is the output :

$ kubectl get validatingwebhookconfiguration -o yaml istiod-istio-system
apiVersion: admissionregistration.k8s.io/v1
kind: ValidatingWebhookConfiguration
    app: istiod
    install.operator.istio.io/owning-resource: installed-state-1-7-4
    install.operator.istio.io/owning-resource-namespace: istio-system
    istio: istiod
    istio.io/rev: 1-7-4
    operator.istio.io/component: Base
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.7.4
    release: istio
  name: istiod-istio-system
- admissionReviewVersions:
  - v1beta1
  - v1
    caBundle: LS0tL...Cg==
      name: istiod
      namespace: istio-system
      path: /validate
      port: 443
  failurePolicy: Fail
  matchPolicy: Exact
  name: validation.istio.io
  - apiGroups:
    - config.istio.io
    - security.istio.io
    - authentication.istio.io
    - networking.istio.io
    - '*'
    - CREATE
    - UPDATE
    - '*'
    scope: '*'
  sideEffects: None
  timeoutSeconds: 30

FYI : I manage to workaroud the issue by editing the istiod service selector to point to istio.io/rev: 1-7-4 revision instead of istio: pilot.

$ kubectl get svc -o yaml istiod
apiVersion: v1
kind: Service
    app: istiod
    operator.istio.io/component: Pilot
    operator.istio.io/managed: Reconcile
    operator.istio.io/version: 1.5.4
    release: istio
  name: istiod
  namespace: istio-system
  - name: https-dns
    port: 15012
  - name: https-webhook
    port: 443
    targetPort: 15017
    app: istiod
    istio.io/rev: 1-7-4

Seems to be an issue with this legacy istiod service when deploying canaries revisions no ?

Since both are just validating the CRDs i would recommend deleting the older one and only validate configuration with istio 1.7 istiod