Istio 404 error suddenly although virtual services are created

Networking
1

TLDR;
Can’t see routing paths even though virtual services are created.
istioctl proxy-config routes istio-ingressgateway-54967ddd58-djvnx -n istio-system
NOTE: This output only contains routes loaded via RDS.
NAME DOMAINS MATCH VIRTUAL SERVICE
* /healthz/ready*
* /stats/prometheus*

A healthy cluster gives all the routing paths configured via virtual service, but this cluster gives empty results.

Hi
Our dev team has a cluster that has GLB which sends traffic to istio-ingressgateway and then istio gateway sends traffic to services based on the routing paths configured via virtual services. This setup was working perfectly since several months, but all of a sudden postman calls from outside of the cluster to GLB as well as internal curl requests to istio gateway endpoint end up as 404 error outputs.

Inorder to debug, i sent a curl request directly to the pod IP and it worked fine, so the problem is with the virtual service routing paths, they dont seem to work some how.
Following are our istio objects:

Ingress gateway in istio-system namespace:


apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"networking.istio.io/v1alpha3","kind":"Gateway","metadata":{"annotations":{},"name":"istio-gateway","namespace":"istio-system"},"spec":{"selector":{"i
stio":"ingressgateway"},"servers":[{"hosts":["*"],"port":{"name":"http","number":80,"protocol":"HTTP"}}]}}
  creationTimestamp: "2022-06-15T20:21:15Z"
  generation: 1
  name: istio-gateway
  namespace: istio-system
  resourceVersion: "208348108"
  uid: 81015cd5-d654-45ae-af59-8adc76907ed7
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP

virtual service:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:  
  labels:
    app: tx-hogan-simulator    
    deployment-type: application
    release: tx-hogan-simulator    
  name: tx-hogan-simulator
  namespace: sigma  
spec:
  gateways:
  - istio-system/istio-gateway
  hosts:
  - '*'
  http:
  - match:
    - uri:
        prefix: /sigma/tx-hogan-simulator/
    - uri:
        prefix: /sigma/tx-hogan-simulator
    rewrite:
      uri: /
    route:
    - destination:
        host: tx-hogan-simulator
        port:
          number: 8104

A curl request to microservice via gateway–>virtualservice–>service–>pod:

  • About to connect() to 172.23.100.169 port 8080 (#0)
  • Trying 172.23.100.169…
  • Connected to 172.23.100.169 (172.23.100.169) port 8080 (#0)

GET /sigma/hogansimulator/liveness HTTP/1.1
User-Agent: curl/7.29.0
Host: 172.23.100.169:8080
Accept: /

< HTTP/1.1 404 Not Found
< date: Thu, 16 Jun 2022 01:36:43 GMT
< server: istio-envoy
< content-length: 0
<

  • Connection #0 to host 172.23.100.169 left intact
1 Like
2

Did you every find a reason for the break, or a solution?