Apply zipkin adapter w/ tracespan template not working

Policies and Telemetry
1

I was trying to apply a zipkin mixer adapter to enable mixer tracing, but it looks it doesn’t work. Here’s the YAML:

# Configuration for tracespan instances
apiVersion: config.istio.io/v1alpha2
kind: instance
metadata:
  name: mytrace
  namespace: istio-system
spec:
  compiledTemplate: tracespan
  params:
    traceId: request.headers["x-b3-traceid"]
    spanId: request.headers["x-b3-spanid"] | ""
    parentSpanId: request.headers["x-b3-parentspanid"] | ""
    spanName: request.path | "/"
    startTime: request.time
    endTime: response.time
    clientSpan: (context.reporter.kind | "inbound") == "outbound"
    rewriteClientSpanId: false
    spanTags:
      http.method: request.method | ""
      http.status_code: response.code | 200
      http.url: request.path | ""
      request.size: request.size | 0
      response.size: response.size | 0
      source.principal: source.principal | ""
      source.version: source.labels["version"] | ""
---
# Configuration for a Zipkin handler
apiVersion: config.istio.io/v1alpha2
kind: handler
metadata:
  name: zipkin
  namespace: istio-system
spec:
  compiledAdapter: zipkin
  params:
    url: "http://zipkin.istio-system:9412/api/v1/spans"
    sample_probability: 1
---
# Rule to send tracespan instances to the Zipkin handler
apiVersion: config.istio.io/v1alpha2
kind: rule
metadata:
  name: mytracerule
  namespace: istio-system
spec:
  match: "true" # match for all requests
  actions:
   - handler: zipkin
     instances:
     - mytrace
---

Since I haven’t had zipkin installed yet, I just use zipkin.istio-system w/ port 9412 that is not really existed and expect there can be something error thrown in logs. I applied the configure as below:

kubectl apply -f zipkin-istio.yaml

However, after refresh the bookinfo web page, I couldn’t find any error in logs. By enabling the trace:

  • --log_output_level=all:debug in mixer container in istio-telemetry deployment
  • --proxyLogLevel=trace and --proxyComponentLogLevel=tracing:trace,misc:error, in istio-proxy container in ingressgateway

I can see the envoy in ingressgateway was calling /istio.mixer.v1.Mixer/Check and /istio.mixer.v1.Mixer/Report, but on the mixer side, I saw this:

2019-07-11T09:48:53.622779Z	debug	config.Snapshot creation error=4 errors occurred:
	* instance='mytrace.instance.istio-system': json: cannot unmarshal bool into Go value of type string
	* action='mytracerule.rule.istio-system[0]': Instance not found: instance='mytrace'
	* action='mytracerule.rule.istio-system[0]': No valid instances found
	* rule=mytracerule.rule.istio-system: No valid actions found in rule

Guess that could be the cause, but have no idea why… Did I miss something?

2

That is indeed the issue. Try putting quotes around false (rewriteClientSpanId: "false").

3

Also, just curious, but if all you want is to send traces to a Zipkin API, you don’t need Mixer (unless you want to shape the trace spans directly). Istio already can configure Envoy to generate traces and export to Zipkin API endpoints without the need for Mixer config.

4

Thanks @douglas-reid for your prompt help so much… I’ve verified it does work!

I think the sample snippet provided by tracespan.proto is misleading, because I just simply copy and paste, then took me a few hours to dive into the flow by enabling "all:debug" logs. If that makes sense, I could submit a simple PR to fix the typo, so that could save others’ time when they get the same problem just like me:-)

To answer your question, yeah… to use Zipkin endpoint is just a quick test to prove if it’s working. My real requirement is to have a second tracing backend to allow traces being filled in w/o breaking the default Jager backend. Here’s my understand:

  • Envoy generates traces to Jaeger by default w/o going through Mixer
  • To redirect Envoy to my own tracing backend will stop it sending traces to Jaeger
  • I could forward traces to Jaeger in my own tracing backend after persist them locally
  • Or, I could configure Mixer that uses Zipkin adapter to listen traces sent from Envoy, then post to my own tracing backend.
  • Also, that seems to be more flexible by using Mixer because there can be multiple adapters being used to consume and generate telemetry to different multiple backends simultaneously. (Although I’m not sure how that’s performant)

Does that make sense? Or, if there’s a better option that I missed?

5

A PR to fix the example would be very welcome, if you have the chance!

Your understanding is correct. Envoy is generating tracespans and sending them to a Zipkin API endpoint (in your case, jaeger). Envoy is also sending request metadata to Mixer which, via config, can be distilled down into tracespans and sent to a number of backends.

If you aren’t interested in editing the tracespans or customizing them, you could also consider using something like the OpenCensus Service (https://opencensus.io/service/). You would only need to deploy the Collector and point the istio zipkin service to it. Then you could configure it with multiple exporters. I haven’t tried configuring it to distribute to multiple backends, but I can provide details on the other configuration bits, if you would like.

6

@douglas-reid Here’s the PR: https://github.com/istio/istio/pull/15511

And, OpenCensus Service seems to be a nice option. So, envoy will send trace spans to OpenCensus Service, right? What is the "details on the other configuration bits" you mean, or where can I find it…

Thanks!

7

@morningspace I just created a PR for adding alpha support for the OpenCensus service to the new installer. It shows how something like this could be done. You would need to extend the exporters section of the config (based on the OC config), but this should cover what you need.

1 Like