Upgrade istio from 1.13.0 to 1.14.0 with zero downtime

Networking
1

I currently have istio-1-13-0 installed and it is setup in this way

  • controlplane in istio-system ns
  • Ingress gateway in istio-ingress ns
  • Create an ingress (uses aws ALB) in istio-ingress ns
  • create a gateway in istio-ingress ns
  • Virtual Service is created in the ns of application needing them

This is working fine and all traffic are getting through

To upgrade this using Canary setup

  • Created new istio controlplane (1-14-0)
  • Created new Ingress gateway 1-14-0 in istio-ingress ns
  • Create an ingress and gateway 1-14-0 in istio-ingress ns
  • created New VS for the applications to use 1-14-0

For my Ingress, I added alb.ingress.kubernetes.io/group.name but I am having problem routing traffic to the new VS in 1-14-0 as when I delete VS of old version, I lose connection.

I have tried alb.ingress.kubernetes.io/actions.servic-name but when VS is deleted I still get HTTP ERROR 404

here is my ingress, gateway and virtual service config

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingressgateway-aws-ingress
  namespace: public-istio-ingress
  annotations:
    alb.ingress.kubernetes.io/scheme: internet-facing
    alb.ingress.kubernetes.io/target-type: ip
    kubernetes.io/ingress.class: alb
    alb.ingress.kubernetes.io/group.name: sre.public-ingress
    alb.ingress.kubernetes.io/target-group-attributes: deregistration_delay.timeout_seconds=30
    alb.ingress.kubernetes.io/actions.public-ingressgateway: |
      {
        "type":"forward",
        "forwardConfig":{
          "targetGroups":[
            {
              "serviceName":"public-ingressgateway",
              "servicePort":"80",
              "weight":100
            }
          ]
        }
      }
spec:
  rules:
    - http:
        paths:
          - path: /*
            backend:
              serviceName: public-ingressgateway
              servicePort: http2

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: public-https-gateway
  namespace: public-istio-ingress
spec:
  selector:
    istio: public-ingressgateway
    service.istio.io/canonical-name: public-ingressgateway
  servers:
    - port:
        number: 80
        name: ingest-http
        protocol: HTTP
      hosts:
        - "*"

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: httpbin-vs
spec:
  hosts:
    - "*"
  gateways:
    - public-istio-ingress/public-https-gateway
  http:
    - match:
      - uri:
          prefix: /
      route:
      - destination:
          host: httpbin
          port:
            number: 8000

For the upgrade one, I created same configuration but added number to the names e.g public-https-gateway was changed to public-https-gateway-02

Can someone help with what I am doing wrong?