Sending mesh traces to an OpenTelemetry endpoint

1

Hi,

Does anybody have experience sending mesh traces to an OpenTelemetry endpoint? The service can listen on both http and grpc endpoints, but I’m not sure which tracing integration to use (Jaeger/Lightstep/Zipkin) and if there’s any intermediary layer I need to use to format the traces so that the OpenTelemetry endpoint can accept it.

2

I’m about to try heading down this route, too, and have been trying to research it. There seem to be a couple of things at play that I’m having to figure out:

Using W3C trace context: It’s not well documented, but I found a StackOverflow answer that shows how to get the system to propagate W3C standard trace context headers instead of Jaeger or Zipkin headers. The key is in some Envoy configuration settings. That’s not necessarily required for sending the traces to an OpenTelemetry endpoint, but it does seem the OpenTelemetry SDK libraries and documentation all prefer W3C trace context.

Sending traces in OpenTelemetry format to a collector: Jaeger, Lightstep, and Zipkin are actually protocols where the trace packets take a different form and the content encoding is different. Like Jaeger generally uses Apache Thrift over UDP. OpenTelemetry, from what I’m gathering so far, uses the same protocol as OpenCensus did. That StackOverflow answer shows how to point the OpenCensus stuff at your OpenTelemetry collector.

Now, I haven’t had a chance yet to actually try any of this because I just started doing the research on it, but this is the direction I’m going to head down to start.

For completeness, I’ll post the Istio config YAML from that StackOverflow answer.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  components:
    pilot:
      enabled: true
      k8s:
        env:
        - name: PILOT_TRACE_SAMPLING
          value: "100"
  values:
    global:
      proxy:
        tracer: openCensusAgent 
  meshConfig:
    enableTracing: true
    defaultConfig:
      tracing:
        openCensusAgent:
          address: "dns:opentelemetry-collector.istio-system.svc:55678"
          context: [W3C_TRACE_CONTEXT]
1 Like
3

Thanks for sharing @tillig.fisv

Here’s a convo I had with somebody on the Istio Slack the other day about this if you’d like some more data points for your research Slack

4

Doesn’t look like I have access to the Istio Slack. Can you summarize the basic points?

5

The main idea is that if you have a need to use the B3 spec for now (like I do - I’m coming from Lightstep and would prefer to change 1 thing at a time) you can do so by using the zipkin tracer and pointing it at Tempo:

  meshConfig:
    enableTracing: true
    defaultConfig:
      tracing:
        sampling: 100
        zipkin:
          address: tempo-headless.monitoring.svc.cluster.local:9411

I haven’t investigated putting the OTel collector in between these two things but I’m assuming that can forward B3 traces (AKA zipkin, in this case), too. Nonetheless, of course, we will want to get on the W3C standard as soon as we can.

6

faied with ocagent send traces to otel.
istio version:1.14.6
meshconfig

defaultConfig:
      discoveryAddress: istiod.istio-system.svc:15012
      tracing:
        openCensusAgent:
          address: "dns:otel-collector.istio-system.svc.cluster.local:4317"
    extensionProviders:
    - name: otel
      opencensus:
        port: 4317
        service: otel-collector.istio-system.svc.cluster.local

error msg:
OcAgent trace exporter: Export() failed: UNIMPLEMENTED: unknown service opencensus.proto.agent.trace.v1.TraceService