Istio returns 404 for 1 of 3 services, with identical istio config

Networking
1

I have 3 services with identical config i.e. 3 Gateways, 3 Virtual services. Let’s call the services

  • service-1
  • service-2
  • service-3

So the Gateways use the same host/port combination.

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  labels:
    app: service-N
    app.kubernetes.io/name: svc-service-N
  name: gateway-service-N
  namespace: myns
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - serviceendpoint.intra
    port:
      name: http
      number: 80
      protocol: HTTP

And the respective VirtualService

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  labels:
    app: service-N
    app.kubernetes.io/name: service-N
    app.kubernetes.io/part-of: issuerapi
  name: service-N
  namespace: myns
spec:
  gateways:
  - gateway-service-N
  hosts:
  - serviceendpoint.intra
  http:
  - match:
    - name: service-N-route
      uri:
        prefix: /svcN/
    route:
    - destination:
        host: service-N
        port:
          number: 80
        subset: service-N
      weight: 100

And this is the DestinationRules

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  labels:
    app: service-N
    app.kubernetes.io/name: service-N
  name: service-N
  namespace: myns
spec:
  host: service-N
  subsets:
  - labels:
      app.kubernetes.io/instance: service-N
    name: service-N
  trafficPolicy:
    loadBalancer:
      consistentHash:
        httpHeaderName: calling-user
    tls:
      mode: ISTIO_MUTUAL

So in details, all services point to the same host but eventually have a different prefix

  • service-1 → serviceendpoint.intra/svc1
  • service-2 → serviceendpoint.intra/svc2
  • service-3 → serviceendpoint.intra/svc3

So while accessing service-1 and service-2 work fine, for service-3 I get a 404 (NOT default backend - 404)

So I navigate to https://serviceendpoint.intra/svc3/persons/111111/ I get

404 Not Found

nginx/1.23.0

The log shows this

message: 10.120.46.105 - - [07/Oct/2022:07:51:10 +0000] 
"GET /svc3/persons/111111/ HTTP/2.0" 404 555 "-" 
"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 
(KHTML, like Gecko) Chrome/106.0.0.0 Safari/537.36" 473 0.007 
[istio-system-istio-ingressgateway-80] [] 10.42.5.27:8080
555 0.006 404 cf1da9fc910abbc4fa6816ce3eac2d55

Interestingly, if I change the host name for service-3 to something different - e.g. demo.intra - accessing the url https://demo.intra/svc3/persons/111111/ works fine.

I went trough all the suggestions under common-problems, which did not help.