Filter chain not found when using TCP proxy filter with strict mTLS enabled

Hi,

I am using Istio 1.16 release.

I have a use case to support mTLS communication between 2 pods (belonging to different namespaces) with the client pod talking to server pod using the server pod IP address directly. For this I have created headless service with the port name set to tcp as documented in step 2 here:

The server headless service looks like below:

kc describe svc server-headless -n fed-smf
Name: server-headless
Namespace: server-ns
Type: ClusterIP
IP Family Policy: SingleStack
IP Families: IPv4
IP: None
IPs: None
Port: tcp-server 9090/TCP
TargetPort: 9090/TCP

With the above configuration, I`m seeing an error in the server istio-proxy sidecar where it is not able to forward the packer to the server application. I have enabled the istio-proxy sidecar logs and I see the below error:

2023-04-27T20:22:03.748077Z debug envoy conn_handler closing connection from 192.168.211.44:34688: no matching filter chain found
[2023-04-27T20:22:03.747Z] “- - -” 0 NR filter_chain_not_found - “-” 0 0 0 - “-” “-” “-” “-” “-” - - 192.168.211.27:9090 192.168.211.44:34688 - -

Am I missing something? Appreciate your help!

Thanks.