Ingress Gateway fails to listen on node port and refuses connection after upgrade to 1.10

Networking
1

Hello,

new users if Istio here. We started evaluation Istio with the 1.9.4 release. Installed using the Operator on our Kubernetes 1.18.17 cluster.

This is the spec:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: control-plane
spec:
  profile: demo
  components:
    cni:
      enabled: true
    pilot:
      enabled: true
      k8s:
        priorityClassName: tier-1-service
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        priorityClassName: tier-1-service
  values:
    cni:
      excludeNamespaces:
      - istio-system
      - kube-system
    gateways:
      istio-ingressgateway:
        type: NodePort

Then deploying the demo app. Using a modified Gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: bookinfo-gateway
  namespace: default
spec:
  selector:
    istio: ingressgateway
  servers:
    - hosts:
        - default/*.istio.ourdomain.com
        - istio-system/*.istio.ourdomain.com
      port:
        name: http
        number: 80
        protocol: HTTP
    - hosts:
        - default/*.istio.ourdomain.com
        - istio-system/*.istio.ourdomain.com
      port:
        name: https
        number: 443
        protocol: HTTPS
      tls:
        credentialName: cert-wild.istio.ourdomain.com
        mode: SIMPLE

and virstual service

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: bookinfo
  namespace: default
spec:
  gateways:
    - bookinfo-gateway
  hosts:
    - bookinfo.istio.ourdomain.com
  http:
    - match:
        - uri:
            exact: /productpage
        - uri:
            prefix: /static
        - uri:
            exact: /login
        - uri:
            exact: /logout
        - uri:
            prefix: /api/v1/products
      route:
        - destination:
            host: productpage
            port:
              number: 9080

This worked fine. We then performed an inplace upgrade. First upgrading the operator using istioctl 1.10.0. Then applying the operator resource above with both the demo and default profiles. Now all would keep working except the ingress gateway. It looks all OK in Kubernetes, pods green, no logs of interest. The applications are also working (port forwarding to both them and Kiali). But the ingress gateway would refuse all connections, execing to its containers we find it is not listening on its defined ports. Out of desperation we decide to downgrade. Simply applying this:

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
metadata:
  namespace: istio-system
  name: control-plane
spec:
  profile: default
  tag: 1.9.4
  meshConfig:
    discoverySelectors:
    - matchLabels:
        istio-discovery: enabled
  components:
    cni:
      enabled: true
    pilot:
      enabled: true
      k8s:
        priorityClassName: tier-1-service
    ingressGateways:
    - name: istio-ingressgateway
      enabled: true
      k8s:
        priorityClassName: tier-1-service
  values:
    cni:
      excludeNamespaces:
      - istio-system
      - kube-system
    gateways:
      istio-ingressgateway:
        type: NodePort

And once the gateway pods with docker.io/istio/proxyv2:1.9.4 images are up, it all starts working again.

Can anyone help with what is the problem here. I can’t find anything in the documentation about or upgrade notes for 1.10