Lightstep broken in Istio 1.8.2

Hi folks,

We’ve updated our Lightstep config for our mesh to consist of the following profile settings:

            mode: "SIMPLE"
            caCertificates: "/etc/lightstep/cacert.pem"

          address: [redacted]:9292
          accessToken: [access token]

        tracer: lightstep

      traceSampling: 100

… but our services report missing traces at the mesh level. This only occurred with the upgrade / change in config to using this meshConfig field. Our other cluster on Istio 1.5.1 reports Lightstep traces for the same calls just fine.

There’s a lot of this in the ingress-gateway pod logs, but no issues in the application pod logs . We find this unusual because the ingress-gateway typically is unaware of the Lightstep configs.

2021-03-03T18:30:53.679547Z error sds resource:file-root:/etc/lightstep/cacert.pem Close connection. Failed to get secret for proxy “router~” from secret cache: open /etc/lightstep/cacert.pem: no such file or directory
2021-03-03T18:30:53.679632Z info sds resource:file-root:/etc/lightstep/cacert.pem connection is terminated: rpc error: code = Canceled desc = context canceled
2021-03-03T18:30:53.679801Z warning envoy config StreamSecrets gRPC config stream closed: 2, open /etc/lightstep/cacert.pem: no such file or directory
2021-03-03T18:31:16.104033Z info sds resource:file-root:/etc/lightstep/cacert.pem new connection

Has anybody encountered this before? This is a big blocker for us in this upgrade.

To any future lurkers who are looking for the solution: Lightstep tracing with caCertificates defined report missing spans · Issue #31190 · istio/istio · GitHub