Unable to set retries for rate limiting applied at ingress gateway

We’ve configured rate limiting using guideline from here - Istio / Enabling Rate Limits using Envoy.

We applied it to an ingress gateway, so that every request coming to our k8s cluster can be rate limited. We wanted to apply retries when the rate limit service is called from ingress gateway, so we decided to apply VirtualService resource:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: rate-limit-service-retry
  namespace: abc
spec:
  hosts:
    - rate-limit-service
  http:
    - route:
        - destination:
            host: rate-limit-service
            port:
              number: 9090
      retries:
        attempts: 3
        perTryTimeout: 2s

When I fetch configuration from one of our workloads:

istioctl proxy-config routes -n echoserver-tls deploy/echoserver-tls --output=yaml

It contains retries configuration:

route:
  cluster: outbound|9090||rate-limit-service.abc.svc.cluster.local
  maxGrpcTimeout: 0s
  retryPolicy:
    hostSelectionRetryMaxAttempts: "5"
    numRetries: 3
    perTryTimeout: 2s
    retriableStatusCodes:
      - 503
    retryHostPredicate:
      - name: envoy.retry_host_predicates.previous_hosts
        typedConfig:
          '@type': type.googleapis.com/envoy.extensions.retry.host.previous_hosts.v3.PreviousHostsPredicate
    retryOn: connect-failure,refused-stream,unavailable,cancelled,retriable-status-codes
  timeout: 0s

However, ingress gateway proxy does not contain this route, therefore requests made by an ingress gateway to the rate limit service are not retried. How we can fix this?