Include traceId in istio-proxy logs?

#1

In my kubernetes cluster, all of my microservices log messages in JSON format and include metadata fields like “traceId” in order to correlate logs.

I would love to be able to similarly correlate the logs from my Istio sidecar containers, the istio-proxy.

Is there any way to achieve this? Can we customize the log output format of the istio-proxy? Ideally, I’d love to be able to tell it to write in JSON format and construct metadata fields by telling it, for example, to write a field called traceId, derived by looking at the request’s x-b3-traceid header. But I’ll take anything I can get, including simply adding such a header to the string output.

#2

I think this link can help you achieve what you want with some tweaking.

I would look specially here: https://raw.githubusercontent.com/istio/istio/release-1.6/samples/bookinfo/telemetry/fluentd-istio.yaml

1 Like
#3

Thanks, @feitnomore. This looks like it’s sending me down a good path.

One bit of confusion, though: When I try to find the available options / syntax for an “instance”, I end up in a section of the documentation that’s flagged as deprecated. https://istio.io/latest/docs/reference/config/policy-and-telemetry/attribute-vocabulary/

What is the non-deprecated equivalent in Istio 1.6?

#4

Sorry, I have no clue :frowning:
I’ve used this with older versions of Istio… I hope someone can chime in here.

#5

Figured it out after a lot of digging around. Here’s an example of what I added to my IstioOperator config to get this working.

Note that in this case, what I really wanted to do was make two changes. 1) Enable JSON formatted access logging and 2) add a custom traceId field to the logged objects pulling data from the x-b3-traceid header of the req.

I first enabled access logging with JSON and observed all of the default fields. In my custom format I then manually re-configured all of those fields based on the envoy docs (https://www.envoyproxy.io/docs/envoy/latest/configuration/observability/access_log/usage) and finally added my extra traceId field.

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  values:
    global:
      logAsJson: true
      proxy:
        accessLogFile: '/dev/stdout'
        accessLogEncoding: 'JSON'
        # istioctl warns that this is deprecated and should go under meshConfig, but that doesn't work: https://github.com/istio/istio/issues/24318
        accessLogFormat: |
          {
            "traceId": "%REQ(x-b3-traceid)%",
            "protocol": "%PROTOCOL%",
            "upstream_service_time": "%REQ(x-envoy-upstream-service-time)%",
            "upstream_local_address": "%UPSTREAM_LOCAL_ADDRESS%",
            "duration": "%DURATION%",
            "upstream_transport_failure_reason": "%UPSTREAM_TRANSPORT_FAILURE_REASON%",
            "route_name": "%ROUTE_NAME%",
            "downstream_local_address": "%DOWNSTREAM_LOCAL_ADDRESS%",
            "user_agent": "%REQ(USER-AGENT)%",
            "response_code": "%RESPONSE_CODE%",
            "response_flags": "%RESPONSE_FLAGS%",
            "start_time": "%START_TIME%",
            "method": "%REQ(:METHOD)%",
            "request_id": "%REQ(X-REQUEST-ID)%",
            "upstream_host": "%UPSTREAM_HOST%",
            "x_forwarded_for": "%REQ(X-FORWARDED-FOR)%",
            "requested_server_name": "%REQUESTED_SERVER_NAME%",
            "bytes_received": "%BYTES_RECEIVED%",
            "istio_policy_status": "-",
            "bytes_sent": "%BYTES_SENT%",
            "upstream_cluster": "%UPSTREAM_CLUSTER%",
            "downstream_remote_address": "%DOWNSTREAM_REMOTE_ADDRESS%",
            "authority": "%REQ(:AUTHORITY)%",
            "path": "%REQ(X-ENVOY-ORIGINAL-PATH?:PATH)%"
          }
1 Like