Canary upgrade Istio 1.6.13 to 1.7.8 problem

1

Hello, I am working on upgrading Istio from 1.6.13 to 1.7.8 using canary release, helm, istio operator.
I deployed a new revision of operator and istiod (1.7.8), but when I try to switch workloads to the new revision of Istio sidecar - injection doesn’t work. Deployment error:

Error creating: Internal error occurred: failed calling webhook “rev.namespace.sidecar-injector.istio.io”: Post https://istiod-1-7-8.istio-system.svc:443/inject?timeout=10s: x509: certificate is valid for istiod.istio-system.svc, istiod-remote.istio-system.svc, istio-pilot.istio-system.svc, not istiod-1-7-8.istio-system.svc

istiod-1.7.8 log error:

TLS handshake error from 10.255.220.98:60506: remote error: tls: bad certificate

following Istio / Sidecar Injection Problems if I run

kubectl get mutatingwebhookconfiguration istio-sidecar-injector-1-7-8 -o yaml -o jsonpath='{.webhooks[0].clientConfig.caBundle}' | md5sum

it doesn’t have caBundle parameter at all

Please help to debug this somehow

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  name: istio-operator-1.7.8
  namespace: kube-system
spec:
  addonComponents:
    grafana:
      enabled: false
    kiali:
      enabled: false
      k8s:
        resources: {}
    prometheus:
      enabled: false
    tracing:
      enabled: false
  components:
    egressGateways:
      - enabled: false
        name: istio-egressgateway
    ingressGateways:
      - enabled: false
        k8s:
          hpaSpec:
            minReplicas: 2
          resources:
            limits:
              cpu: 1000m
              memory: 512Mi
            requests:
              cpu: 100m
              memory: 128Mi
          serviceAnnotations:
            service.beta.kubernetes.io/aws-load-balancer-backend-protocol: http
            service.beta.kubernetes.io/aws-load-balancer-ssl-cert: >-
              arn:aws:acm:xxxxxxxxxxxxxx:certificate/xxxxxxxxxxxxxxxxxxxxxxxxxxxxx
            service.beta.kubernetes.io/aws-load-balancer-ssl-ports: https
          strategy:
            rollingUpdate:
              maxSurge: 100%
              maxUnavailable: 25%
        name: istio-ingressgateway
    pilot:
      enabled: true
      k8s:
        hpaSpec:
          minReplicas: 2
        resources:
          limits:
            cpu: 500m
            memory: 1G
          requests:
            cpu: 250m
            memory: 512Mi
  hub: docker.io/istio
  meshConfig:
    accessLogFile: /dev/stdout
    enableTracing: false
    outboundTrafficPolicy:
      mode: REGISTRY_ONLY
  revision: 1-7-8
  tag: 1.7.8
  values:
    gateways:
      istio-ingressgateway:
        externalTrafficPolicy: Local
        podAntiAffinityTermLabelSelector:
          - key: app
            operator: In
            topologyKey: failure-domain.beta.kubernetes.io/zone
            values: istio-ingressgateway
    global:
      logging:
        level: 'default:info'
      proxy:
        excludeIPRanges: 169.254.169.254/32
        logLevel: warning
        resources:
          limits:
            cpu: 250m
            memory: 128Mi
          requests:
            cpu: 10m
            memory: 32Mi
    pilot:
      traceSampling: 100