Cross-Cluster DNS Resolution in Split-horizon EDS Not Working

We have an installation of split-horizon EDS (cluster-based routing) with one master and two remotes enmeshed. External traffic enters via the master’s ingressgateway and is successfully routed via cluster-local DNS extensions to a remote (say to someservice.remote1.svc.cluster.local). This all works as expected and documented. However, when a service in a remote needs to reach a service in another remote (say someservice.remote1.svc.cluster.local needs to reach out to otherservice.remote2.svc.cluster.local`) the application is unable to resolve that host. Am I missing something about the intended function of this Istio pattern? Does the cross-cluster KubeDNS configuration only apply to the master?

Could really use some support on this as it is not at all clear from the documentation exactly how resolution between services within the split-horizon EDS installation is intended to work. If we have an installation much like the one depicted in the diagram at the top of the help article:

image

Exactly how would helloworld.sample in cluster 1 resolve some other service (lets say goodbyeworld.sample in cluster 2? From my understanding, when the application attempts to resolve the host, it bubbles up to the kubedns handler for the *.local domain which, thanks to some configuration installed when the Istio chart was applied, is hooked into the EDS and is aware that the service exists in a remote cluster.

It would then respond to the DNS query with the IP of the gateway for cluster 2, no? Please correct me if my understanding is off, we have been trying desperately for many weeks now to get a solid multi-cluster service mesh deployed and have had great difficulty getting clarification on whether some of the more basic use cases are even supported by current releases of Istio.

I am in a similar boat (istio-version: 1.16.2)

istioctl pc endpoints sleep-784d69f945-q2l6z -n test-ns| select-string helloworld

192.168.59.20:15443                                     HEALTHY     OK                outbound|5000||helloworld.sample.svc.cluster.local

So my access request with FQDN works, but it does not work it ExernalEntry:

kubectl exec -n test-ns -c sleep "$(kubectl get pod -n test-ns -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -- curl -sS helloworld.sample.svc.cluster.local:5000/hello
Hello version: v2, instance: helloworld-v2-54df5f84b-cvn8x

kubectl exec -n test-ns -c sleep "$(kubectl get pod -n test-ns -l app=sleep -o jsonpath='{.items[0].metadata.name}')" -- curl -sS **helloworld:5000/hello**
curl: (6) Could not resolve host: helloworld
command terminated with exit code 6