Can't get distributed traces from istio-proxy

#1

Hello everyone,

I’m trying to get tracing spans from istio-proxy sidecar whenever my container receives a request. I get spans from every other component of Istio (ingress gateway, mixer itself, etc.), but not from istio-proxy. I see in istio-proxy’s logs that it registers incoming requests to my container and that it’s also configured with the correct zipkin endpoint. I have the most standard Istio installation imaginable (Azure Kubernetes + istio-demo.yaml with a small modification to have mixer log out tracing spans that it sends), and I can’t seem to figure this out.
Note: my application does not propagate distributed tracing headers, but I would still expect to receive a span from istio-proxy once a request hits it. I’m using Jaeger UI to view the spans, and I see spans from every component except the sidecar. I also don’t see sidecar’s span in mixer logs (I’ve enabled span logging there), only the spans that mixer generates itself.
How do I troubleshoot? Is there something I’m missing? Does the sidecar use Envoy’s functionality to send spans to mixer or is it something built on top of that? What can I further check?

Thank you.

#2

Alex,

Are you expecting to see spans for the time spent exclusively in the proxy itself?

Right now, the proxies generate traces on behalf of the applications they front – with the operation names matching the hostname and port (iirc). Examples of this can be seen in the Jaeger Distributed Tracing Task images, for example.

If your application itself generates spans, you will see those spans as child spans of the spans generated by the proxy.

You see gateway spans as the gateway isn’t a proxy for any single service. You see mixer spans, because mixer is an application that participates in the traces like any other application may when it serves traffic for the istio-policy calls.

As a background, I recommend reading the new Distributed Tracing FAQ. Hopefully that will help clarify at least some of the details of how tracing works.

I hope that helps,
Doug.

#3

Thank you Douglas,

I’m expecting to see any spans generated by the proxy, I’m not even thinking about what those spans actually measure yet. I do understand that the span will have the name of the application as the operation name (and I’m completely fine with it). Since “proxies generate traces on behalf of the applications they front” - that means proxies do generate traces, right? My application generates nothing, but I should still be seeing spans generated by the proxy. Is that a reasonable expectation?
I went through the sources you’ve listed, and that’s the assumptions I hold after reading through them :slight_smile:
The problem is that my proxies seem to not generate any spans at all. That’s not expected, correct? Is there any way to troubleshoot?

#4

That is a reasonable expectation – and should be happening.

What may be happening (depending on the version you’ve installed) is that the default sampling rate is 1%. So, you may need to send ~100 requests through each proxy to see trace data (especially if you aren’t forwarding headers).

Do you know how your sampling rate is currently configured ?

#5

Should I be looking at Mixer for that configuration or at Pilot? Or directly at Envoy config? Also, do proxies send spans to the tracing endpoint directly or through Mixer? I’m seeing contradicting information on that.

#6

Please take a look at the FAQ entry: Why are my request not being traced?

It links to an overview of updating trace sampling rates, as well as providing further guidance on troubleshooting.

#7

Ok, it was the unnamed port. I saw that section previously, but in that older version of the docs it only mentioned named ports for proper traffic routing support, so I never considered it as a possible root cause. Thank you very much!