Distributing tracing doesn't work when clients send traces to node local jaeger agent IP

Networking
1

Hello. We are using Istio 1.6.5 and we have deployed a separate Jaeger installation using Helm. We have configured the istio proxies to send traces to the jaeger collector and we are able to see those traces reach Jaeger. We have configured our applications to send tracers to the Jaeger agent and we are not able to see those traces.

Unlike the jaeger collector, the jaeger agent is deployed by default as a daemonset. Per the documentation our applications have been configured to resolve the jaeger agent host IP address from the local node.

env:
- name: JAEGER_AGENT_HOST
  valueFrom:
    fieldRef:
      fieldPath: status.hostIP

This does not seem to work and I suspect it has something to do with how the istio proxy routes traffic. During testing I instead configured the JAEGER_AGENT_HOST environment variable to point to the jaeger agent service (jaeger-agent.observability) and this fixed the issue. The downside of this fix is that I believe traces will be load balanced to all the jaeger agents, which defeats the purpose of using a daemonset in the first place.

Does anyone know what might be going on?