What to do about VirtualServices with Destinations that have no healthy upstreams

Given a VirtualService with one route and two destinations, where one destination/service has a selector which returns no pods, how can we configure Istio to only send traffic to the destination that has healthy upstreams?