In my kubernetes cluster, all of my microservices log messages in JSON format and include metadata fields like “traceId” in order to correlate logs.
I would love to be able to similarly correlate the logs from my Istio sidecar containers, the istio-proxy.
Is there any way to achieve this? Can we customize the log output format of the istio-proxy? Ideally, I’d love to be able to tell it to write in JSON format and construct metadata fields by telling it, for example, to write a field called traceId, derived by looking at the request’s x-b3-traceid header. But I’ll take anything I can get, including simply adding such a header to the string output.
Figured it out after a lot of digging around. Here’s an example of what I added to my IstioOperator config to get this working.
Note that in this case, what I really wanted to do was make two changes. 1) Enable JSON formatted access logging and 2) add a custom traceId field to the logged objects pulling data from the x-b3-traceid header of the req.
We’re employing the same workaround now. We had to go from the much cleaner config from my answer above to the ugly escaped JSON string when we upgraded Istio.
One note regarding the sample config that you mentioned above, which otherwise works (with Istio 1.6.8 too): you want to have %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% instead of %REQ(X-ENVOY-UPSTREAM-SERVICE-TIME)%, because the value won’t be known at the request time.