Sidecars in service consumer's namespace seems not honoring destination rules in service provider's namespace

Networking
1

I’m having a try on failoverPriority released in 1.12 to avoid cross-zone traffic in EKS, and somehow it doesn’t work as expected.

The following destination rule is configured in the provider’s namespace, and both provider and consumer pods are auto injected with sidecar.

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: svc-name
  namespace: provider-ns
spec:
  host: svc-name.provider-ns.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
        failoverPriority:
        - topology.kubernetes.io/region
        - topology.kubernetes.io/zone
    outlierDetection: {}

Things work perfectly in the provider pods, exactly as described. However, the outbound traffic from consumer pods in their own namespace doesn’t really honor this destination rule, as it hits all provider pods regardless of locality.

The traffic is simply http traffic, and no virtual service or gateway is configured for this service. It’s just the k8s service and this destination rule only. I tried to delete istiod pod and redeploy both providers and consumers, and the issue remains there.

I tried to update the destination rule to use locality-based distribute, instead of new failover, in the following way. It remains not working, so I guess I’m missing something fundamental :upside_down_face:

apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
  name: svc-name
  namespace: provider-ns
spec:
  host: svc-name.provider-ns.svc.cluster.local
  trafficPolicy:
    loadBalancer:
      localityLbSetting:
        enabled: true
        distribute:
        - from: us-east-1/us-east-1a/*
          to:
            us-east-1/us-east-1a/*: 100
        - from: us-east-1/us-east-1b/*
          to:
            us-east-1/us-east-1b/*: 100
    outlierDetection: {}

Here are the versions of the EKS environment, and please feel free to let me know if something further is required. Looking forward to some help!

$ istioctl version
client version: 1.14.3
control plane version: 1.12.5
data plane version: 1.12.5 (9 proxies)
$ kubectl version --short
Client Version: v1.22.2
Server Version: v1.22.15-eks-fb459a0
2

Problem solved –

The consumer pods must serve some k8s service before they can really honor the destination rules. I haven’t really checked the code base why it is what it is, but at least it’s working now.

That’s something quite fundamental, I have to say :joy: