Include traceId in istio-proxy logs?

#1

In my kubernetes cluster, all of my microservices log messages in JSON format and include metadata fields like “traceId” in order to correlate logs.

I would love to be able to similarly correlate the logs from my Istio sidecar containers, the istio-proxy.

Is there any way to achieve this? Can we customize the log output format of the istio-proxy? Ideally, I’d love to be able to tell it to write in JSON format and construct metadata fields by telling it, for example, to write a field called traceId, derived by looking at the request’s x-b3-traceid header. But I’ll take anything I can get, including simply adding such a header to the string output.

Istio operator meshconfig access log format
#2

I think this link can help you achieve what you want with some tweaking.

I would look specially here: https://raw.githubusercontent.com/istio/istio/release-1.6/samples/bookinfo/telemetry/fluentd-istio.yaml

1 Like
#3

Thanks, @feitnomore. This looks like it’s sending me down a good path.

One bit of confusion, though: When I try to find the available options / syntax for an “instance”, I end up in a section of the documentation that’s flagged as deprecated. https://istio.io/latest/docs/reference/config/policy-and-telemetry/attribute-vocabulary/

What is the non-deprecated equivalent in Istio 1.6?

#4

Sorry, I have no clue :frowning:
I’ve used this with older versions of Istio… I hope someone can chime in here.

#5

Figured it out after a lot of digging around. Here’s an example of what I added to my IstioOperator config to get this working.

Note that in this case, what I really wanted to do was make two changes. 1) Enable JSON formatted access logging and 2) add a custom traceId field to the logged objects pulling data from the x-b3-traceid header of the req.

I first enabled access logging with JSON and observed all of the default fields. In my custom format I then manually re-configured all of those fields based on the envoy docs (https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage) and finally added my extra traceId field.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      logAsJson: true
      proxy:
        accessLogFile: '/dev/stdout'
        accessLogEncoding: 'JSON'
        # istioctl warns that this is deprecated and should go under meshConfig, but that doesn't work: https://github.com/istio/istio/issues/24318
        accessLogFormat: |
          {
            "traceId": "%REQ(x-b3-traceid)%",
            "protocol": "%PROTOCOL%",
            "upstream_service_time": "%REQ(x-envoy-upstream-service-time)%",
            "upstream_local_address": "%UPSTREAM_LOCAL_ADDRESS%",
            "duration": "%DURATION%",
            "upstream_transport_failure_reason": "%UPSTREAM_TRANSPORT_FAILURE_REASON%",
            "route_name": "%ROUTE_NAME%",
            "downstream_local_address": "%DOWNSTREAM_LOCAL_ADDRESS%",
            "user_agent": "%REQ(USER-AGENT)%",
            "response_code": "%RESPONSE_CODE%",
            "response_flags": "%RESPONSE_FLAGS%",
            "start_time": "%START_TIME%",
            "method": "%REQ(:METHOD)%",
            "request_id": "%REQ(X-REQUEST-ID)%",
            "upstream_host": "%UPSTREAM_HOST%",
            "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%",
            "requested_server_name": "%REQUESTED_SERVER_NAME%",
            "bytes_received": "%BYTES_RECEIVED%",
            "istio_policy_status": "-",
            "bytes_sent": "%BYTES_SENT%",
            "upstream_cluster": "%UPSTREAM_CLUSTER%",
            "downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%",
            "authority": "%REQ(:AUTHORITY)%",
            "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
          }
2 Likes
#6

This is what worked for me actually. https://github.com/istio/istio/issues/24318#issuecomment-688672926

meshConfig:
  accessLogFile: /dev/stdout
  accessLogEncoding: JSON
  accessLogFormat: |
    {
      "accessLogFormat": "{\"traceID\": \"%REQ(x-b3-traceid)%\",\"protocol\": \"%PROTOCOL%\",\"upstream_service_time\": \"%REQ(x-envoy-upstream-service-time)%\",\"upstream_local_address\": \"%UPSTREAM_LOCAL_ADDRESS%\",\"duration\": \"%DURATION%\",\"upstream_transport_failure_reason\": \"%UPSTREAM_TRANSPORT_FAILURE_REASON%\",\"route_name\": \"%ROUTE_NAME%\",\"downstream_local_address\": \"%DOWNSTREAM_LOCAL_ADDRESS%\",\"user_agent\": \"%REQ(USER-AGENT)%\",\"response_code\": \"%RESPONSE_CODE%\",\"response_flags\": \"%RESPONSE_FLAGS%\",\"start_time\": \"%START_TIME%\",\"method\": \"%REQ(:METHOD)%\",\"request_id\": \"%REQ(X-REQUEST-ID)%\",\"upstream_host\": \"%UPSTREAM_HOST%\",\"x_forwarded_for\": \"%REQ(X-FORWARDED-FOR)%\",\"requested_server_name\": \"%REQUESTED_SERVER_NAME%\",\"bytes_received\": \"%BYTES_RECEIVED%\",\"istio_policy_status\": \"-\",\"bytes_sent\": \"%BYTES_SENT%\",\"upstream_cluster\": \"%UPSTREAM_CLUSTER%\",\"downstream_remote_address\": \"%DOWNSTREAM_REMOTE_ADDRESS%\",\"authority\": \"%REQ(:AUTHORITY)%\",\"path\": \"%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%\"}"
    }
1 Like
#7

We’re employing the same workaround now. We had to go from the much cleaner config from my answer above to the ugly escaped JSON string when we upgraded Istio.

1 Like
#8

One note regarding the sample config that you mentioned above, which otherwise works (with Istio 1.6.8 too): you want to have %RESP(X-ENVOY-UPSTREAM-SERVICE-TIME)% instead of %REQ(X-ENVOY-UPSTREAM-SERVICE-TIME)%, because the value won’t be known at the request time.

1 Like