Need source telemetry to capture 'the first hop' when mirroring traffic


#1

This question involves mirrored traffic telemetry. It is going to be hard to describe - I’ll try to be brief but yet make sense :slight_smile:

I have traffic mirroring set up in bookinfo (see this yaml if you are interested) where productpage-v1 sends its requests to reviews-v1 but mirrors to reviews-v2.

Visualizing the resulting telemetry in Kiali looks like this:

missing-telemetry

Notice that “first hop” in the mirrored traffic is missing - that is to say, the request going from productpage-v1 to reviews-v2 is missing. This is because there is no reporter=“source” metric. However, as that mirrored traffic flows “downstream”, there is source telemetry for the rest of the request as it flows from service to service (which is why you see edges from reviews-v2 to ratings-v1 and -v2 as well as ratings-v2 to mongodb-v1).

The question is: Can the Istio implementation be changed so the full mirrored traffic (starting at that “first hop”) can be represented by source telemetry? As it is now, the source telemetry has a “hole” in it as you see when the telemetry is visualized.

Side note: there is reporter=“destination” telemetry for that first hop - but that is from the point of view of the reviews-v2 workload (thus reporter=“destination”). The Kiali graph is visualizing reporter=“source” telemetry because that is the only side that provides information about client-side errors (like injected faults, network errors, etc). Here are some attributes from the only metric available for that edge between productpage-v1 and reviews-v2 (this is found by executing the prometheus query istio_requests_total{destination_workload=“reviews-v2”}) - notice there is no reporter=“source” metric; only reporter=“destination” is found in telemetry:

reporter=“destination”
destination_app=“reviews”
destination_version=“v2”
destination_workload=“reviews-v2”
source_app=“productpage”
source_version=“v1”
source_workload=“productpage-v1”
… and others…


#2

Hi!

I think this is indeed a bug in that we are not supplying istio attributes for mirrored traffic.
Do you mind associating a github issue? This might require some envoy expertise to understand traffic mirroring interaction with filters.

–kuat


#3

OK - here is the github issue I just wrote on this: https://github.com/istio/istio/issues/11093