mTLS uses wrong SNI in TLS Client Hello

Hi,

we are using RedHat Service Mesh 1.1.2. I am aware that the actual upstream project is Maistra but we are observing mTLS problems where i am not sure if we are not understanding the intended architecture/configuration or if it’s a major bug which possibly affects Istio also.
In our setup we have deployed two namespaces, first x2 and afterwards x3. Both are from a configuration and deployment perspective identical (of course the namespace specific config within the yamls differ accordingly), both have mTLS enabled and a headless service.
In our setup we have one Istio control plane (istio-system) and are trying to do mTLS within the namespaces. Just in case, we are not trying to do mTLS between multiple namespaces.

---
apiVersion: v1
kind: Service
metadata:
  name: headless
spec:
  clusterIP: None
  selector:
    galera.v1beta2.sql.databases/galera-name: galera-cluster
  ports:
    - name: s3306
      protocol: TCP
      port: 3306
      targetPort: 3306
    - name: s4444
      protocol: TCP
      port: 4444
      targetPort: 4444
    - name: s4567
      protocol: TCP
      port: 4567
      targetPort: 4567
    - name: s4568
      protocol: TCP
      port: 4568
      targetPort: 4568
---
apiVersion: "authentication.istio.io/v1alpha1"
kind: "Policy"
metadata:
  name: default
spec:
  peers:
    - mtls: {}
---
apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: default
spec:
  host: "*.x2.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
---

In the first namespace, x2, mTLS is working as expected.

istioctl authn tls-check galera-cluster-bb55l -n x2 | grep x2.svc
headless.x2.svc.cluster.local:3306                                 OK         STRICT         ISTIO_MUTUAL     x2/default                                     x2/default
headless.x2.svc.cluster.local:4444                                 OK         STRICT         ISTIO_MUTUAL     x2/default                                     x2/default
headless.x2.svc.cluster.local:4567                                 OK         STRICT         ISTIO_MUTUAL     x2/default                                     x2/default
headless.x2.svc.cluster.local:4568                                 OK         STRICT         ISTIO_MUTUAL     x2/default                                     x2/default

When we deploy x3 with the same configuration as x2, the x3 pods are not able to communicate with each other.

istioctl authn tls-check galera-cluster-24z99 -n x3 | grep x3.svc
headless.x3.svc.cluster.local:3306                                 OK         STRICT         ISTIO_MUTUAL     x3/default                                     x3/default
headless.x3.svc.cluster.local:4444                                 OK         STRICT         ISTIO_MUTUAL     x3/default                                     x3/default
headless.x3.svc.cluster.local:4567                                 OK         STRICT         ISTIO_MUTUAL     x3/default                                     x3/default
headless.x3.svc.cluster.local:4568                                 OK         STRICT         ISTIO_MUTUAL     x3/default                                     x3/default

A tcpdump revealed that the TLS handshake between the envoy proxies fails with “Certificate Unknown (46)”. The reason for this is that in the TLS Client Hello the SNI for x2 is used (outbound_.4567_._.headless.x2.svc.cluster.local), which is obviously wrong. It seems that the mesh (i use this term on purpose because i don’t know which component of it is responsible for this behaviour) uses the first service fqdn that is created for this tcp port. When we delete the x2 namespace the mTLS communication in x3 starts working as expected.
If needed i can provide further configuration and tcpdumps.
We did not find a way to change this behaviour by configuration (different ServiceEntries, DestinationRules etc.) nor did we find a hint in the documentation that this should or should not work.

From an architectural or configuration point of view is this behaviour expected?

Thank you for you support!

Best Regards,

This seems like a bug, is this using Istio 1.1? That is a very old version and is no longer supported, I’m wondering if you can try this with a newer Istio version (e.g. 1.5 or 1.6).

Just a sanity check, what if you delete the DestinationRule in the x2 namespace? Does it help the x3 namespace to become normal?

Actually it is istio 1.4.6, i will try removing the Destination Rule.

Removing the DestinationRule does not make any difference. x3 uses still the x2 TLS client cert.