I am trying to figure out if it is possible to configure a different trace sampling value for ingress-gateway vs that used for the sidecar proxies. The use-case is fairly common where we want set the sampling at the for all incoming traffic through ingress-gateway to say 10% whereas all the sidecar istio-proxies should only add spans to the trace if and only if ingress-gateway already decided to trace a given request. I see some new upcoming feature related to tracing configuration here: https://istio.io/docs/tasks/observability/distributed-tracing/configurability/ but wondering if this ingress-gateway sampling scenario is possible in some way.
I believe ingress and egress gateways are handled differently when they are deployed, as they do not have sidecars injected. There is capability to change, per pod, tracing settings, but I’m not sure how that would work in conjunction with gateways. Unfortunately, changing the per pod tracing policy will be done in 1.6.1 (see https://github.com/istio/istio.io/pull/7348/files for an example).
There may be a way to hack around the istio-ingressgateway deployment and add that annotation when 1.6.1 ships, but I would say such a hack is not supported and you’d have to assume your own level of risk as you are modifying default istio resources.
Thanks @Jacob. Yes this could work given the resource annotation mechanism in general works for gateways similar to how they work for sidecar-proxies. So as I understand, starting 1.6.1, tracing sampling will be one of the configurations that will be changeable via resource annotations.