I am new to Istio and Tracing. I followed the examples to play with Distributed Tracing over Jaeger. Istio Documentation says, “Although Istio proxies are able to automatically send spans, they need some hints to tie together the entire trace. Applications need to propagate the appropriate HTTP headers so that when the proxies send span information, the spans can be correlated correctly into a single trace.”
I have a few questions on “Applications need to propagate the appropriate HTTP headers”:
Can’t the side-car proxy inject/extract hint (such as x-request-id) rather than “Applications or Client Services” needing to propagate the appropriate HTTP headers for Span correlation in a given Trace? Can this Span Correlation be supported in Istio without Applications injecting in the HTTP Headers?
If not supported, can’t Istio support a “Tracing Route Map” feature so side-car proxy or Mixer can take advantage of this “Tracing Route Map” to auto inject or extract the info based on configuration rather than expecting the “Applications” to propagate? Currently, I have Applications to annotate or decorate this functionality but it does involve change to the codebase and testing.
I think the Span “Correlation” support should be the works of the Service Mesh and not the Client. Client can tailor to their needs but “Span Correlation” support must be extended by Istio as a ‘default’.
The short answer is that some code modification is required to support distributed tracing. There is not a shortcut around that requirement.
There is a FAQ devoted to this line of questioning. Please see the answers there for more detail.
From the FAQ:
Although an Istio sidecar will process both inbound and outbound requests for an associated application instance, it has no implicit way of correlating the outbound requests to the inbound request that caused them. The only way this correlation can be achieved is if the application propagates relevant information (i.e. headers) from the inbound request to the outbound requests.
Envoy provides the capability for reporting tracing information regarding communications between services in the mesh. However, to be able to correlate the pieces of tracing information generated by the various proxies within a call flow, the services must propagate certain trace context between the inbound and outbound requests.
Whichever tracing provider is being used, the service should propagate the x-request-id to enable logging across the invoked services to be correlated.
For #1:x-request-id still needs to be forwarded for any scheme around it to work. Applications need to send that out on their outgoing requests. Without that bit of modification, there can be no correlation.
For #2: how would such a map work? what would be in the map? one incoming request can result in 0…N outgoing requests. and outgoing requests can be made from applications that have no associated incoming request.
It would indeed be useful if there was someway to generically correlate all requests without any context being forwarded. We have yet to develop such a scheme.
“someway to generically correlate all requests without any context being forwarded. We have yet to develop such a scheme”
I was alluding to above such feature/scheme where side-car proxy can generically correlate all requests by default. It would be a helpful feature rather than having the client route requests over another proxy/decorator/annotation etc., to add the hints in the headers. Such a scheme should be part of the Service Mesh foundation and will value-add to Istio. Thank you for the reply.
You would need to outline in detail how your proposal would work.
As I point out in my response, there are lots of edge cases around request correlation that need to be addressed – of which I’ve only mentioned a few. Consider, too, that a single proxy may be handling 1000s of requests (inbound and outbound) at any given instant.
At a minimum, some context must be passed between application and proxy. There is no way around that.
I think the problem here is: how would you even theoretically accomplish that?
An example to make it more concrete. A pod currently has 30 active inbound http requests, then it makes an outbound request to another service. How does Envoy decide which inbound request to associate this outbound request with?
