Log Correlation ID Injection

Policies and Telemetry
1

Hi,

Currently I have some microservices running along side a stack that uses Filebeat (in Daemonset mode) to collect logs, along with the ELK stack to collect and store structured json logs.

As well as using trace / span ids in distributed traces from Istio, I am hoping to find a mechanism to inject a correlation ID into logs that are output from each service on stdout before they are collected by Filebeat, without changing each service and without having to try to inject such an ID in places like an API Gateway.

Is there any mechanism that I could use to inject a correlation id strings into my structured json logs that are being output from each of my microservices, before they get collected by filebeat and sent to Logstash / Elasticsearch? It is not clear to me if this is possible from looking at the current set of documentation on the logentry or stdio objects.

Any advice on this topic would be greatly appreciated.

Regards,
Mark

2

Have you looked at using the x-request-id header by any chance? The idea behind that header is to provide that sort of information.

3

Thanks for your reply douglas. Thats something I will look at. We have debated many ways to possibly create some sort of correlation id (eg in the). Would you say this particular header would be the typically recommended one to use for such logs as opposed to some of the other trace/span ids from Istio / Distributed Tracing? Will the x-request-id always be there for every request if using Istio Gateway / VirtualService?

I have not seen any documentation anywhere yet around this area of logging to recommend any particular header to use or means to inject that header for each call that comes into the system either from the outside through the gateway / virtual service, or generated from within the cluster (eg a Kubernetes Job).