Subset rules don't behave the same when inside the mesh

Networking
1

Description

I’m trying to mark different traffic so that I can route them to different subset. But I found that when I request from outside the mesh ,everything works as expected . But inside the mesh ,the result seems still round-robin.

If I try to curl outside the mesh the response will always from base, and gray if changed to internal IP

curl -H "Host: demo.com" pubIP/
curl -H "Host: demo.com" intIP/

However if I try to curl inside the mesh , the response seems round-robin. ( I have a httpbin running the same namespace and I’ve confirm my outbound envoyFilter works)

curl httpbin:8000/headers
{
  "headers": {
    "Accept": "*/*", 
    "Host": "httpbin:8000", 
    "Origin": "escalade-server", 
    "User-Agent": "curl/7.52.1", 
    "X-B3-Parentspanid": "8a1422dbebcc4d8d", 
    "X-B3-Sampled": "0", 
    "X-B3-Spanid": "415a2760d7e59676", 
    "X-B3-Traceid": "6e52811a952883178a1422dbebcc4d8d", 
    "X-Envoy-Attempt-Count": "1", 
    "X-Forwarded-Client-Cert": "By=spiffe://cluster.local/ns/demo/sa/httpbin;Hash=256d2ba2f8cb851cdd6e6c7cc395026f44a6267456a762254d3a9a9884395d00;Subject=\"\";URI=spiffe://cluster.local/ns/demo/sa/default", 
    "X-Gandalf-Mode": "gray"
  }
}

Calling httpbin just want to make sure my outbound envoyFilter is working so that the traffic can be routed to base/ gray when I request api.demo.

And even if I specify the header like , it’s still round-robin.

curl -H "X-Gandalf-Mode: gray" api.demo.svc.cluster.local:8080/headers

Version

client version: 1.12.2
control plane version: 1.12.2
data plane version: 1.12.2 (7 proxies), 1.12.1 (7 proxies)

Additional Information

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: http-request-labelling-according-source
  namespace: istio-system
spec:
#  workloadSelector:
#    labels:
#      app: istio-ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
#      context: ANY
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
            subFilter:
              name: "envoy.filters.http.router"
    patch:
      operation: INSERT_BEFORE
      value:
       name: envoy.lua
       typed_config:
         "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
         inlineCode: |
           function envoy_on_request(request_handle)
             local interHeaderName = "X-Envoy-Internal"
             local exterHeaderName = "X-Envoy-External-Address"
             headers = request_handle:headers()
             interFlag = headers:get(interHeaderName)
             exterFlag = headers:get(exterHeaderName)
             if (interFlag ~= nil)  then
                 headers:add("X-Gandalf-Mode","gray")
             elseif (exterFlag ~= nil) then
                 headers:add("X-Gandalf-Mode","base")
             else
                 headers:add("X-Gandalf-Mode","notmatch")
             end
           end

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: mark-outbound-request
  namespace: istio-system
spec:
  configPatches:
  - applyTo: HTTP_FILTER
    match:
#      context: GATEWAY
      context: SIDECAR_OUTBOUND
      listener:
        filterChain:
          filter:
            name: "envoy.filters.network.http_connection_manager"
            subFilter:
              name: "envoy.filters.http.router"
    patch:
      operation: INSERT_BEFORE
      value:
       name: envoy.lua
       typed_config:
         "@type": "type.googleapis.com/envoy.extensions.filters.http.lua.v3.Lua"
         inlineCode: |
           function envoy_on_request(request_handle)
              request_handle:logWarn("Hello World")
              request_handle:headers():add("origin", os.getenv("ISTIO_META_WORKLOAD_NAME"))
              request_handle:headers():add("X-Gandalf-Mode", "gray")
           end

---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: demo-ext-vs
  namespace: demo
spec:
  hosts:
    - demo.com
    - api.demo.svc.cluster.local
  gateways:
    - mesh
    - istio-system/public-gateway
    - istio-system/internal-gateway
  http:
    - match:
        - headers:
            X-Gandalf-Mode:
              exact: gray
      route:
        - destination:
            host: api.demo.svc.cluster.local
            subset: version-gray
    - route:
        - destination:
            host: api.demo.svc.cluster.local
            subset: version-base
---
apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: traffic-marker-dr
  namespace: demo
spec:
  host: api.demo.svc.cluster.local
  trafficPolicy:
    tls:
      mode: DISABLE
  subsets:
    - name: version-gray
      labels:
        version: gray
    - name: version-base
      labels:
        version: base

istioctl x describe po demo-server-gray-6f6d94c94-5kz94 -n demo                                                                                                                                         
Pod: demo-server-gray-6f6d94c94-5kz94
   Pod Ports: 8080 (demo-server), 15090 (istio-proxy)
--------------------
Service: api
   Port: tcp 8080/TCP targets pod port 8080
DestinationRule: traffic-marker-dr for "api.demo.svc.cluster.local"
   Matching subsets: version-gray
      (Non-matching subsets version-base)
   Traffic Policy TLS Mode: DISABLE


Exposed on Ingress Gateway http://172.19.146.186
VirtualService: demo-ext-vs
   when headers are X-Gandalf-Mode=gray
   1 additional destination(s) that will not reach this pod