System clock not synchronized cause odd trace span timeline

we are using template tracespan to generate bookinfo tracing:

currently, there are 2 k8s nodes, something odd from the timeline:

  1. client span is finished before server span.
  2. server span is started before client span.

after some debug, I believe these are because the system time of the 2 nodes are not the exactly the same, maybe 1 or 2 ms time difference.
I think this is not istio issue, because envoy(or opentracing/jaeger) generate start/end time from local time, so, is this acceptable? or is there some solution?

Does your cluster have NTP? Can you sync the clocks and try again to confirm?

1 Like

Unfortunately I could not find equivalent upstream Kubernetes documentation indicating NTP is a best practice or requirement of installing Kubernetes, OpenShift seems to think so. I also believe NTP is mandatory - and not just for accurate tracing :grin:

https://docs.openshift.com/container-platform/3.6/install_config/install/prerequisites.html#prereq-NTP