How do I use the istio retry policies?

Networking
1

I’m having problem getting istio retries to work. Currently I have a service running in my cluster called test-svc in a namespace test-ns. I have a VirtualService pointing to it that is configured as follows -

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: test-svc-8080-vs
  namespace: test-ns
spec:
  gateways:
  - istio-system/gateway
  hosts:
  - test-svc.example.com
  http:
  - retries:
      attempts: 1000000
      perTryTimeout: 3s
      retryOn: 5xx,reset
    route:
    - destination:
        host: test-svc
        port:
          number: 8080

The service is properly accessible when I try https://test-svc.example.com .
I reduced the number of replicas for this service to 0 to see if istio retries before returning 503 but it does not and I immediately get a response with no healthy upstream . Is their something I’m missing here?