Istio | Envoy Proxy Problem: 0 NR filter_chain_not_found | TCP - Python Socket Client and Socket Server in one cluster (MESH_INTERNAL)

Networking
1

Dear all,
i have a minor problem with Istio and the EnvoyProxy: NR filter_chain_not_found

The socket client and the socket server run within the same cluster (seperated docker-container) and send each other plaintext messages at intervals. The socket server runs on port 50000, the socket client on port 50001. Without mTLS (PERMISSIVE), the communication works without problems. If I activate mTLS (STRICT), the error listed below occurs. I have already tried writing EnvoyFilters, but I can’t imagine that this is the right way.

Log from the envoy-proxy:

On the Socket Server side:

[2023-01-16T19:52:55.941Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 5000 - "-" "-" "-" "-" "-" - - 10.1.2.142:50000 10.1.2.146:50001 - -

[2023-01-16T19:58:05.909Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 5001 - "-" "-" "-" "-" "-" - - 10.1.2.142:50000 10.1.2.146:50001 - -

On the Socket Client side:

Connect to SocketServer...  server-c-socket-server-service.server-c-socket-server.svc.cluster.local
SERVER_NAME as string => server-c-socket-server-service.server-c-socket-server.svc.cluster.local
Traceback (most recent call last):
File "/service/server-c-socket-client.py", line 94, in <module>
main()
File "/service/server-c-socket-client.py", line 91, in main
ConnectToSocketServer(SERVER_NAME)
File "/service/server-c-socket-client.py", line 60, in ConnectToSocketServer
answer = con.recv(1024)
^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer

Some More Informations:

istio-strict-meshpolicy.yaml

apiVersion: security.istio.io/v1beta1 
kind: PeerAuthentication
 metadata:
   name: "default"
   namespace: "istio-system"
 spec:
   mtls:
     mode: STRICT

istio-virtualservice-socket-client.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: server-c-socket-client-virtualservice
  namespace: server-c-socket-client
spec:
  hosts:
  - server-c-socket-client-service.server-c-socket-client.svc.cluster.local
  tcp:
  - match:
    - port: 50001
    route:
    - destination:
      host: server-c-socket-client-service.server-c-socket-client.svc.cluster.local
      port:
        number: 50001
    weight: 100

istio-virtualservice-socket-server.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: server-c-socket-server-virtualservice
  namespace: server-c-socket-server
spec:
  hosts: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
  tcp:
  route:
  - destination:
    host: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
    port:
      number: 50000
  weight: 100

istio-destinationrule-socket-client.yaml

apiVersion: networking.istio.io/v1alpha3 
kind: DestinationRule 
metadata:
   name: server-c-socket-client-destinationrule
   namespace: server-c-socket-client 
spec:
   host: server-c-socket-client-service.server-c-socket-client.svc.cluster.local
   trafficPolicy:
     tls:
       mode: MUTUAL
       credentialName: cacerts
       sni: server-c-socket-client-service.server-c-socket-client.svc.cluster.local

istio-destinationrule-socket-server.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule 
metadata:
   name: server-c-socket-server-destinationrule
   namespace: server-c-socket-server
 spec:
   host: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
   trafficPolicy:
     tls:
       mode: MUTUAL
       credentialName: cacerts
       sni: server-c-socket-server-service.server-c-socket-server.svc.cluster.local

istio-peerauthentication-socket-client.yaml

apiVersion: security.istio.io/v1beta1 
kind: PeerAuthentication 
metadata:
   name: server-c-socket-client-peerauthentication
   namespace: server-c-socket-client
 spec:
   mtls:
     mode: STRICT

istio-peerauthentication-socket-server.yaml

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
   name: server-c-socket-server-peerauthentication
   namespace: server-c-socket-server
 spec:
   mtls:
     mode: STRICT

System

Kubernetes: MicroK8s v1.25.5 revision 4418

kubectl version:
Client Version: v1.25.5
Kustomize Version: v4.5.7
Server Version: v1.25.5

OS: Ubuntu 22.04.1

istioctl proxy-status

NAME                                                                      CLUSTER        CDS        LDS        EDS        RDS          ECDS         ISTIOD                     VERSION
istio-ingressgateway-78f69b5b89-w24fx.istio-system                        Kubernetes     SYNCED     SYNCED     SYNCED     NOT SENT     NOT SENT     istiod-d887c9b84-xk9tn     1.14.4
server-c-nginx-deploy-7cb9cc7574-57tdw.server-c-nginx                     Kubernetes     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-d887c9b84-xk9tn     1.14.4
server-c-socket-client-deploy-7469697f89-ndf89.server-c-socket-client     Kubernetes     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-d887c9b84-xk9tn     1.14.4
server-c-socket-server-deploy-5d47669d86-fk8kh.server-c-socket-server     Kubernetes     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-d887c9b84-xk9tn     1.14.4

I have already tried many properties in istio and have struggled with creating an EnvoyFilter, which unfortunately does not give the desired results.
As an example: “NR filter_chain_not_found” #30819 TCP mTLS through ingressgateway "NR filter_chain_not_found" · Issue #30819 · istio/istio · GitHub or Istio MTLS Smartness Explained - RandomBytes

In the end, the plain text messages (TCP) should be encrypted, which does not work in STRICT mode.

If you have any ideas or need more information, please let me know.

Best regards.

(I posted the same discussion on stackoverflow, maybe somebody can help me here)