How to troubleshoot routing from Istio Ingress Gateway to the Kubernetes Service

1

I would say most of the hours I spend on Istio configuration relates to routing. I must say I dont find the basic logs on the Istio Ingress Gateway to be particularly helpful. The errors are all expressed in HTTP terms like service not available or 404. Yet here is where the routing is occurring. I dont want to see end user messages at this point but messages like route not found. No matching route etc. In addition to that I think I should be able to trace end to end exactly what happens to a request and what routes or virtual services got applied during the request. This feature might be there but after 3 months of working with Istio its not clear to me exactly how to access these features. If they are not available I think a request for this feature should be added.

The main point is that the Istio Ingress Gateway should not be a black hole where data goes in and HTTP status codes are returned. If you have admin access to the Istio Ingress Gateway the trace should be much more about the route matching etc. What routes were examined during the request and why it matched or did not match should be readily available…

So for example I am trying reach a service that I already know exists and is running ok from port forwarding. On the client side this is what I see:
coffee:test Steven$ curl -v http://35.187.24.323:80/v1/list-invoices

  • Trying 35.187.24.323…
  • TCP_NODELAY set
  • Connected to35.187.24.323 (35.187.24.323) port 80 (#0)

GET /v1/list-invoices HTTP/1.1
Host: 35.187.226.216
User-Agent: curl/7.54.0
Accept: /

< HTTP/1.1 503 Service Unavailable
< date: Thu, 29 Aug 2019 08:17:15 GMT
< server: istio-envoy
< content-length: 0
<

  • Connection #0 to host 35.187.24.323 left intact

Not really a lot of information but its for end user client. But inside Istio Ingress Gateway I expect to see lots more useful information inside I see:
[2019-08-29T08:17:16.054Z] “GET /v1/list-invoices HTTP/1.1” 503 NR “-” “-” 0 0 1 - “10.36.0.1” “curl/7.54.0” “f689a211-38e0-9b58-8ca9-81ef3316111d” “35.187.224.124” “-” - - 10.36.0.30:80 10.36.0.1:52770 -

So first of all I dont not what NR stands for ? No Route?
What about the curl/7.74.0 whats that about?
And why do I see “" "” 0 0 1.

I dont see this as ideal messages when what I to know is what routes where compared to and did it pass it not.

1 Like
2
  • NR stands for No Route, more refer to this doc
  • curl/7.74.0 stands for User Agent for client
  • for other field you can refer to Envoy access log

And for troubleshooting you can go through this doc

1 Like