Gateway+VirtualService stops working after about 2 minute of working fine if ingress root route gives 404

Networking
1

Been working on adding istio via the helm chart.

Setup is
Ingress = Azure AGIC.
AGIC points to istio’s ingress where all internal traffic happens on http

Gateway:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: our-gateway
spec:
  selector:
    istio: "dev-istio"
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "dev.ourdomain.com"

VirtualService

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: our-gateway
spec:
  hosts:
    - "dev.ourdomain.com"
  gateways:
    - our-gateway
  http:
    - match:
        - uri:
            prefix: "/kiali"
      route:
        - destination:
            host: "kiali.istio-system.svc.cluster.local"
            port:
              number: 20001
    - match:
        - uri:
            prefix: "/jaeger"
      route:
        - destination:
            host: "dev-jaeger-query.jaeger.svc.cluster.local"
            port:
              number: 80
    - route:
        - destination:
            host: "our-custom-ingress-svc.backend.svc.cluster.local"
            port:
              number: 80

This works ok for around 2 minutes, then it gives 503.

“our-custom-ingress-svc.backend.svc.cluster.local” gives 404 as response when you call it on route “/”, but route “/product1” will route to product1.

Even the other routes like /kiali and /jaeger gives 503. It’s completely broken.

If I add a 200 response code and some text on root “/” route on “our-custom-ingress-svc.backend.svc.cluster.local” it works just fine, and never changes to 503. I guess this is a workaround.

This is istio 1.17.1. Our ingress always gave 404 on “/” path, but now suddenly it’s a problem. Worked fine in istio 1.16.1 (I think).

Why does it behave like that?
Is there some configuration that can has to be set to allow “/” route to return 404 without istio signaling it as a problem?