Istio virtual service subset not able to send request to specific pods

Hi,

I am facing problem while routing the service using DestinationRules. My scenario is following
User → FastAPI → Users

  • FastAPI (API Gateway)
  • Users (gRPC service)

below are the yaml deployment manifests

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: users
  labels:
    app: users
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: users
      version: v1
  template:
    metadata:
      labels:
        app: users
        version: v1
    spec:
      containers:
        - image: users:v0.0.1
          imagePullPolicy: Always
          name: svc
          ports:
            - containerPort: 9090
---
kind: Service
apiVersion: v1
metadata:
  name: users
  labels:
    app: users
spec:
  selector:
    app: users
  ports:
  - name: grpc
    protocol: TCP
    port: 9090

---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: fastapi
  labels:
    app: fastapi
    version: v1
spec:
  replicas: 1
  selector:
    matchLabels:
      app: fastapi
      version: v1
  template:
    metadata:
      labels:
        app: fastapi
        version: v1
    spec:
      containers:
        - image: fastapi:latest
          imagePullPolicy: Always
          name: web
          ports:
            - containerPort: 8080
          env:
            - name: USERS_SVC
              value: 'users:9090'
---
kind: Service
apiVersion: v1
metadata:
  name: fastapi
  labels:
    app: fastapi
spec:
  selector:
    app: fastapi
  ports:
    - port: 8080
      name: http

i deployed version v2 for users service using below manifest

apiVersion: apps/v1
kind: Deployment
metadata:
  name: users-v2
  labels:
    app: users
    version: v2
spec:
  replicas: 1
  selector:
    matchLabels:
      app: users
      version: v2
  template:
    metadata:
      labels:
        app: users
        version: v2
    spec:
      containers:
        - image: users:v0.0.1
          imagePullPolicy: Always
          name: svc
          ports:
            - containerPort: 9090

After this I tried to test virtual service and route to users service (version: v2) when https header is passed. below is the codes for virtual service and destination rules

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: users-service-destination-rule
spec:
  host: users
  subsets:
  - name: v1
    labels:
      version: v1
  - name: v2
    labels:
      version: v2

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: users-virtual-service
spec:
  hosts:
    - users
  http:
  - match:
    - headers:
        x-user-testing:
          exact: tester
    route:
    - destination:
        host: users
        subset: v2
  - route:
    - destination:
        host: users
        subset: v1

When i tested with curl -H "x-user-testing: tester" localhost/users the request always goes to version v1. somehow the header value of x-user-testing is not getting recognised and always goes to default route.

Please help me.
Pankaj