Outbound http request not being proxied by isito-proxy

Networking
1

We’re seeing outbound http traffic not being proxied by istio-proxy, and this only happened to some pods in the same deployment

we have a client that request to a suggestion service, and for this suggestion service we have destination rule as:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: suggestion-destinationrule
  namespace: suggestion-service
spec:
  host: suggestion
  subsets:
  - labels:
      env: prod
      model: one
    name: one
  - labels:
      env: prod
      model: two
    name: two
  - labels:
      env: prod
      model: three
    name: three
  - labels:
      env: prod
      model: four
    name: four

and virtual service:

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: suggestion-virtualservice
  namespace: suggestion-service
spec:
  hosts:
  - suggestion.suggestion-service.svc.cluster.local
  http:
  - match:
    - headers:
        version:
          exact: four
    route:
    - destination:
        host: suggestion
        subset: four
  - match:
    - headers:
        version:
          exact: three
    route:
    - destination:
        host: suggestion
        subset: three
  - match:
    - headers:
        version:
          exact: two
    route:
    - destination:
        host: suggestion
        subset: two
  - route:
    - destination:
        host: suggestion
        subset: one

and service:

apiVersion: v1
kind: Service
metadata:
  name: suggestion
  namespace: suggestion-service
spec:
  clusterIP: 10.47.253.57
  ports:
  - name: http
    port: 80
    protocol: TCP
    targetPort: 8080
  selector:
    app: suggestion
  sessionAffinity: None
  type: ClusterIP

We’ve checked the clusters/listeners/endpoints/routes, as well as the experiment command shows:

$ istioctl x des pod suggestion-two-fdc5c96cc-h49zk
Pod: suggestion-twp-fdc5c96cc-h49zk
   Pod Ports: 8080 (suggestion), 15090 (istio-proxy)
Suggestion: add 'version' label to pod for Istio telemetry.
--------------------
Service: suggestion
   Port: http 80/HTTP targets pod port 8080
DestinationRule: suggestion-destinationrule for "suggestion"
   Matching subsets: two
      (Non-matching subsets one,three,four)
   No Traffic Policy
VirtualService: suggestion-virtualservice
   when headers are version=two
   3 additional destination(s) that will not reach this pod

we enabled istio-proxy debug logs for the client pods, and found some pods (same deployment, and it’s the client) we can see the outbound logs like

2021-02-25T13:15:13.790764Z debug envoy router  [external/envoy/source/common/router/router.cc:477] [C12458][S5370940600346970392] 
cluster 'outbound|80|two|suggestion.suggestion-service.svc.cluster.local' match for URL '/v1/predict'

but some client pods they dont have such logs, however when we exec into the pod and run tcpdump without such logs, we can still seeing the requests:

13:36:42.603821 IP 10.44.176.25.34790 > 10.47.253.57.80: Flags [.], ack 403, win 330, options [nop,nop,TS val 515030915 ecr 751500812], length 0
13:36:43.016592 IP 10.44.176.25.34790 > 10.47.253.57.80: Flags [P.], seq 393:791, ack 403, win 331, options [nop,nop,TS val 515031328 ecr 751500812], length 398: HTTP: POST /v1/predict HTTP/1.1
13:36:43.051505 IP 10.47.253.57.80 > 10.44.176.25.34790: Flags [P.], seq 403:811, ack 791, win 331, options [nop,nop,TS val 751501259 ecr 515031328], length 408: HTTP: HTTP/1.1 200 OK
13:36:43.051525 IP 10.44.176.25.34790 > 10.47.253.57.80: Flags [.], ack 811, win 330, options [nop,nop,TS val 515031363 ecr 751501259], length 0

(the suggesiton service is running with cluster IP 10.47.253.57 and port 80)
I’m not sure what went wrong, it seems to us that the requests are not being proxied to istio-proxy,
but this only happen to some pods in the same deployment,
If anyone has any insights/thoughts it’ll be much appreciated.

We’re using istio 1.6.14 (istiod), and this started to happen after we upgraded from 1.5.8 (istiod)