Swagger Unauthorized Virtual Service-Help needed!

Config
1

Hi All,
I have the below virtual service. When i try to hit https://mydomain.com/customapi/swagger , I am getting unauthorized error(401) from istio-envoy whereas when I try to hit the same api throught the loadbalancer IP (http://x.x.x.x/custom/swagger) I am able to access the service successfully. However, I am able to access the other route https://mydomain.com/organizations/id/custom successfully from istio-envoy. Need help on finding what is the mistake I am doing here.
Istio 1.4.3
Dotnet core 3.0 application

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  namespace: default
  name: mycustomservice-vs
spec:
  hosts:
  - mydomain.com
  gateways:
  - ngm-gateway.default.svc.cluster.local
  http:  
  - match:
    - uri:
        prefix: /customapi
    - uri:
        prefix: /customapi/
    rewrite:
      uri: /custom/swagger
    route:
    - destination:
        host: mycustomservice.default.svc.cluster.local
        port:
          number: 5000        
  - match:
    - headers:
        referer:
          prefix: https://mydomain.com/customapi/index.html
    - uri: 
        prefix: /custom/swagger
    route:
    - destination:
        host: mycustomservice.default.svc.cluster.local
        port:
          number: 5000
  - name: "mycustomservice-vs"
    match:
    - uri:
        regex: .*\/organizations\/.*\/custom.*
      ignoreUriCase: true
    route:
    - destination:
        host: mycustomservice.default.svc.cluster.local
        port:
          number: 5000
    corsPolicy:
      allowOrigin:
      - "*"
      allowMethods:
      - POST
      - GET
      - PUT
      - PATCH
      - DELETE
      - OPTIONS
      allowCredentials: true
      allowHeaders:
      - authorization
      - content-type
      - etag
      - x-requested-with            
      - api-version