Lightstep broken in Istio 1.8.2

Hi folks,

We’ve updated our Lightstep config for our mesh to consist of the following profile settings:

  meshConfig:
      defaultConfig:
        tracing:
          tlsSettings:
            mode: "SIMPLE"
            caCertificates: "/etc/lightstep/cacert.pem"

values:
    global:
      tracer:
        lightstep:
          address: [redacted]:9292
          accessToken: [access token]

      proxy:
        tracer: lightstep

    pilot:
      traceSampling: 100

… but our services report missing traces at the mesh level. This only occurred with the upgrade / change in config to using this meshConfig field. Our other cluster on Istio 1.5.1 reports Lightstep traces for the same calls just fine.

There’s a lot of this in the ingress-gateway pod logs, but no issues in the application pod logs . We find this unusual because the ingress-gateway typically is unaware of the Lightstep configs.

2021-03-03T18:30:53.679547Z error sds resource:file-root:/etc/lightstep/cacert.pem Close connection. Failed to get secret for proxy “router~10.1.10.164~istio-ingressgateway-5b7475bbb6-547xf.istio-system~istio-system.svc.cluster.local” from secret cache: open /etc/lightstep/cacert.pem: no such file or directory
2021-03-03T18:30:53.679632Z info sds resource:file-root:/etc/lightstep/cacert.pem connection is terminated: rpc error: code = Canceled desc = context canceled
2021-03-03T18:30:53.679801Z warning envoy config StreamSecrets gRPC config stream closed: 2, open /etc/lightstep/cacert.pem: no such file or directory
2021-03-03T18:31:16.104033Z info sds resource:file-root:/etc/lightstep/cacert.pem new connection

Has anybody encountered this before? This is a big blocker for us in this upgrade.

To any future lurkers who are looking for the solution: Lightstep tracing with caCertificates defined report missing spans · Issue #31190 · istio/istio · GitHub