Failover for ServiceEntries


I am trying to set up a HA Multicluster Application using Istio. We have 2 seperate Clusters, where the same application is running mirrored. The problem is now, that the application is not entirely stateless, so I only want to root to the second cluster, when the one one on my own cluster is crashing.
For that I created a serviceEntry pointing to both the internal and the external Application and a Destination rule with an Outlier Detection, but I cant find a way to make the ServiceEntry sticky and only change when the Outlier Detection goes off.
I tried setting the weights to 100 and 0 respectively, but then the Endpoint with weight 0 is ignored and no switch happenes at all.
Is there some Parameter or Function I missed in the documentation?
Thanks for the help in advance