Istio | Envoy Proxy Problem: 0 NR filter_chain_not_found | TCP - Python Socket Client and Socket Server in one cluster (MESH_INTERNAL)

Dear all,
i have a minor problem with Istio and the EnvoyProxy: NR filter_chain_not_found

The socket client and the socket server run within the same cluster (seperated docker-container) and send each other plaintext messages at intervals. The socket server runs on port 50000, the socket client on port 50001. Without mTLS (PERMISSIVE), the communication works without problems. If I activate mTLS (STRICT), the error listed below occurs. I have already tried writing EnvoyFilters, but I can’t imagine that this is the right way.

Log from the envoy-proxy:

On the Socket Server side:

[2023-01-16T19:52:55.941Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 5000 - "-" "-" "-" "-" "-" - - 10.1.2.142:50000 10.1.2.146:50001 - -

[2023-01-16T19:58:05.909Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 5001 - "-" "-" "-" "-" "-" - - 10.1.2.142:50000 10.1.2.146:50001 - -

On the Socket Client side:

Connect to SocketServer...  server-c-socket-server-service.server-c-socket-server.svc.cluster.local
SERVER_NAME as string => server-c-socket-server-service.server-c-socket-server.svc.cluster.local
Traceback (most recent call last):
File "/service/server-c-socket-client.py", line 94, in <module>
main()
File "/service/server-c-socket-client.py", line 91, in main
ConnectToSocketServer(SERVER_NAME)
File "/service/server-c-socket-client.py", line 60, in ConnectToSocketServer
answer = con.recv(1024)
^^^^^^^^^^^^^^
ConnectionResetError: [Errno 104] Connection reset by peer

Some More Informations:

istio-strict-meshpolicy.yaml

apiVersion: security.istio.io/v1beta1 
kind: PeerAuthentication
 metadata:
   name: "default"
   namespace: "istio-system"
 spec:
   mtls:
     mode: STRICT

istio-virtualservice-socket-client.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: server-c-socket-client-virtualservice
  namespace: server-c-socket-client
spec:
  hosts:
  - server-c-socket-client-service.server-c-socket-client.svc.cluster.local
  tcp:
  - match:
    - port: 50001
    route:
    - destination:
      host: server-c-socket-client-service.server-c-socket-client.svc.cluster.local
      port:
        number: 50001
    weight: 100

istio-virtualservice-socket-server.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: server-c-socket-server-virtualservice
  namespace: server-c-socket-server
spec:
  hosts: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
  tcp:
  route:
  - destination:
    host: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
    port:
      number: 50000
  weight: 100

istio-destinationrule-socket-client.yaml

apiVersion: networking.istio.io/v1alpha3 
kind: DestinationRule 
metadata:
   name: server-c-socket-client-destinationrule
   namespace: server-c-socket-client 
spec:
   host: server-c-socket-client-service.server-c-socket-client.svc.cluster.local
   trafficPolicy:
     tls:
       mode: MUTUAL
       credentialName: cacerts
       sni: server-c-socket-client-service.server-c-socket-client.svc.cluster.local

istio-destinationrule-socket-server.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule 
metadata:
   name: server-c-socket-server-destinationrule
   namespace: server-c-socket-server
 spec:
   host: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
   trafficPolicy:
     tls:
       mode: MUTUAL
       credentialName: cacerts
       sni: server-c-socket-server-service.server-c-socket-server.svc.cluster.local

istio-peerauthentication-socket-client.yaml

apiVersion: security.istio.io/v1beta1 
kind: PeerAuthentication 
metadata:
   name: server-c-socket-client-peerauthentication
   namespace: server-c-socket-client
 spec:
   mtls:
     mode: STRICT

istio-peerauthentication-socket-server.yaml

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
   name: server-c-socket-server-peerauthentication
   namespace: server-c-socket-server
 spec:
   mtls:
     mode: STRICT

System

Kubernetes: MicroK8s v1.25.5 revision 4418

kubectl version:
Client Version: v1.25.5
Kustomize Version: v4.5.7
Server Version: v1.25.5

OS: Ubuntu 22.04.1

istioctl proxy-status

NAME                                                                      CLUSTER        CDS        LDS        EDS        RDS          ECDS         ISTIOD                     VERSION
istio-ingressgateway-78f69b5b89-w24fx.istio-system                        Kubernetes     SYNCED     SYNCED     SYNCED     NOT SENT     NOT SENT     istiod-d887c9b84-xk9tn     1.14.4
server-c-nginx-deploy-7cb9cc7574-57tdw.server-c-nginx                     Kubernetes     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-d887c9b84-xk9tn     1.14.4
server-c-socket-client-deploy-7469697f89-ndf89.server-c-socket-client     Kubernetes     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-d887c9b84-xk9tn     1.14.4
server-c-socket-server-deploy-5d47669d86-fk8kh.server-c-socket-server     Kubernetes     SYNCED     SYNCED     SYNCED     SYNCED       NOT SENT     istiod-d887c9b84-xk9tn     1.14.4

I have already tried many properties in istio and have struggled with creating an EnvoyFilter, which unfortunately does not give the desired results.
As an example: “NR filter_chain_not_found” #30819 TCP mTLS through ingressgateway "NR filter_chain_not_found" · Issue #30819 · istio/istio · GitHub or Istio MTLS Smartness Explained - RandomBytes

In the end, the plain text messages (TCP) should be encrypted, which does not work in STRICT mode.

If you have any ideas or need more information, please let me know.

Best regards.

(I posted the same discussion on stackoverflow, maybe somebody can help me here)

(post deleted by author)

Small Update, i just fixed the DestinationRule, but still got the same problem:

server-c@server-c:~$ microk8s istioctl experimental describe pod server-c-socket-server-deploy-5d47669d86-s9wzj.server-c-socket-server
Pod: server-c-socket-server-deploy-5d47669d86-s9wzj.server-c-socket-server
   Pod Revision: default
   Pod Ports: 50000 (server-c-socket-server-app), 15090 (istio-proxy)
--------------------
Service: server-c-socket-server-service.server-c-socket-server
   Port: tcp 50000/TCP targets pod port 50000
DestinationRule: server-c-socket-server-destinationrule.server-c-socket-server for "server-c-socket-server-service.server-c-socket-server.svc.cluster.local"
   Matching subsets: v1
   Traffic Policy TLS Mode: MUTUAL
--------------------
Effective PeerAuthentication:
   Workload mTLS mode: STRICT
Applied PeerAuthentication:
   default.istio-system, server-c-socket-server-peerauthentication.server-c-socket-server
server-c@server-c:~$ microk8s istioctl experimental describe pod server-c-socket-client-deploy-7469697f89-ngktr.server-c-socket-client
Pod: server-c-socket-client-deploy-7469697f89-ngktr.server-c-socket-client
   Pod Revision: default
   Pod Ports: 50001 (server-c-socket-client-app), 15090 (istio-proxy)
WARNING: Pod server-c-socket-client-deploy-7469697f89-ngktr.server-c-socket-client Container server-c-socket-client-app NOT READY
--------------------
Service: server-c-socket-client-service.server-c-socket-client
   Port: tcp 50001/TCP targets pod port 50001
DestinationRule: server-c-socket-client-destinationrule.server-c-socket-client for "server-c-socket-client-service.server-c-socket-client.svc.cluster.local"
   Matching subsets: v1
   Traffic Policy TLS Mode: MUTUAL
--------------------
Effective PeerAuthentication:
   Workload mTLS mode: STRICT
Applied PeerAuthentication:
   default.istio-system, server-c-socket-client-peerauthentication.server-c-socket-client

Some Updated files | 01/20/2023:

  • the communication is in one cluster
  • no outgoing / ingoing external clustertraffic (eg. no ingress or egress gateway is configured)
  • the Socket Server is in the namespace: server-c-socket-server
  • the Socket Client is in the namespace: server-c-socket-client
  • if i edit the PeerAuthentication from the Socket Server to PERMISSIVE its works immediately, but not encrypted… :frowning:
  • I also added a sleep command to the socket client Python script (about 3 minutes), as I suspected a timing problem between deployment and envoy-sidecar
  • What I noticed with the error with the Envoy “10.1.2.142:50000 10.1.2.146:50001” the first IP-address is the Socket Server and the second one is the Socket Client, its looks like the Server does not know how to reply the Socket-connection request…

istio-destinationrule-socket-client.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: server-c-socket-client-destinationrule
  namespace: server-c-socket-client
spec:
  host: server-c-socket-client-service.server-c-socket-client.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
      sni: server-c-socket-client-service.server-c-socket-client.svc.cluster.local

istio-destinationrule-socket-server.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: server-c-socket-server-destinationrule
  namespace: server-c-socket-server
spec:
  host: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
  subsets:
  - name: v1
    labels:
      version: v1
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL
      sni: server-c-socket-server-service.server-c-socket-server.svc.cluster.local

istio-peerauthentication-socket-client.yaml

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: server-c-socket-client-peerauthentication
  namespace: server-c-socket-client
spec:
  mtls:
    mode: STRICT

istio-peerauthentication-socket-server.yaml

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: server-c-socket-server-peerauthentication
  namespace: server-c-socket-server
spec:
  mtls:
    mode: STRICT

istio-strict-meshpolicy.yaml

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: default
  namespace: istio-system
spec:
  mtls:
    mode: STRICT

istio-virtualservice-socket-client.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: server-c-socket-client-virtualservice
  namespace: server-c-socket-client
spec:
  hosts:
  - server-c-socket-client-service.server-c-socket-client.svc.cluster.local
  tcp:
  - match:
    - port: 50001
    route:
    - destination:
        host: server-c-socket-client-service.server-c-socket-client.svc.cluster.local
        subset: v1
        port:
          number: 50001
      weight: 100

istio-virtualservice-socket-server.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: server-c-socket-server-virtualservice
  namespace: server-c-socket-server
spec:
  hosts:
  - server-c-socket-server-service.server-c-socket-server.svc.cluster.local
  tcp:
  - match:
    - port: 50000
    route:
    - destination:
        host: server-c-socket-server-service.server-c-socket-server.svc.cluster.local
        subset: v1
        port:
          number: 50000
      weight: 100

istio-protocolversion.yaml

apiVersion: install.istio.io/v1alpha1
kind: IstioOperator
spec:
  meshConfig:
    enableTracing: true
    accessLogFile: "/dev/stdout"
    meshMTLS:
      minProtocolVersion: TLSV1_3
server-c@server-c:~$ microk8s istioctl experimental describe pod server-c-socket-client-deploy-7469697f89-ngktr.server-c-socket-client
Pod: server-c-socket-client-deploy-7469697f89-ngktr.server-c-socket-client
   Pod Revision: default
   Pod Ports: 50001 (server-c-socket-client-app), 15090 (istio-proxy)
   WARNING: User ID (UID) 1337 is reserved for the sidecar proxy.
--------------------
Service: server-c-socket-client-service.server-c-socket-client
   Port: tcp 50001/TCP targets pod port 50001
DestinationRule: server-c-socket-client-destinationrule.server-c-socket-client for "server-c-socket-client-service.server-c-socket-client.svc.cluster.local"
   Matching subsets: v1
   Traffic Policy TLS Mode: ISTIO_MUTUAL
--------------------
Effective PeerAuthentication:
   Workload mTLS mode: STRICT
Applied PeerAuthentication:
   default.istio-system, server-c-socket-client-peerauthentication.server-c-socket-client
server-c@server-c:~$ microk8s istioctl experimental describe pod server-c-socket-server-deploy-5d47669d86-s9wzj.server-c-socket-server
Pod: server-c-socket-server-deploy-5d47669d86-s9wzj.server-c-socket-server
   Pod Revision: default
   Pod Ports: 50000 (server-c-socket-server-app), 15090 (istio-proxy)
   WARNING: User ID (UID) 1337 is reserved for the sidecar proxy.
--------------------
Service: server-c-socket-server-service.server-c-socket-server
   Port: tcp 50000/TCP targets pod port 50000
DestinationRule: server-c-socket-server-destinationrule.server-c-socket-server for "server-c-socket-server-service.server-c-socket-server.svc.cluster.local"
   Matching subsets: v1
   Traffic Policy TLS Mode: ISTIO_MUTUAL
--------------------
Effective PeerAuthentication:
   Workload mTLS mode: STRICT
Applied PeerAuthentication:
   default.istio-system, server-c-socket-server-peerauthentication.server-c-socket-server

mtls: STRICT

server-c@server-c:~$ microk8s istioctl pc listeners deploy/server-c-socket-server-deploy -n server-c-socket-server --port 15006
ADDRESS 		PORT  	MATCH                                                                    					DESTINATION
0.0.0.0 		15006 	Addr: *:15006                                                            					Non-HTTP/Non-TCP
0.0.0.0 		15006 	Trans: tls; App: istio-http/1.0,istio-http/1.1,istio-h2; Addr: 0.0.0.0/0 					InboundPassthroughClusterIpv4
0.0.0.0 		15006 	Trans: tls; Addr: 0.0.0.0/0                                           						InboundPassthroughClusterIpv4
0.0.0.0 		15006 	Trans: tls; Addr: *:50000                                                					Cluster: inbound|50000||

mtls: PERMISSIVE

server-c@server-c:~$ microk8s istioctl pc listeners deploy/server-c-socket-server-deploy -n server-c-socket-server --port 15006
ADDRESS 		PORT  MATCH                                                                                            				DESTINATION
0.0.0.0 		15006 	Addr: *:15006                                                                                    			Non-HTTP/Non-TCP
0.0.0.0 		15006 	Trans: tls; App: istio-http/1.0,istio-http/1.1,istio-h2; Addr: 0.0.0.0/0       								InboundPassthroughClusterIpv4
0.0.0.0 		15006 	Trans: raw_buffer; App: http/1.1,h2c; Addr: 0.0.0.0/0         	    	                  					InboundPassthroughClusterIpv4
0.0.0.0 		15006 	Trans: tls; App: TCP TLS; Addr: 0.0.0.0/0                                          							InboundPassthroughClusterIpv4
0.0.0.0 		15006 	Trans: raw_buffer; Addr: 0.0.0.0/0                                                              			InboundPassthroughClusterIpv4
0.0.0.0 		15006 	Trans: tls; Addr: 0.0.0.0/0                                                                     			InboundPassthroughClusterIpv4
0.0.0.0 		15006 	Trans: tls; App: istio,istio-peer-exchange,istio-http/1.0,istio-http/1.1,istio-h2; Addr: *:50000 			Cluster: inbound|50000||
0.0.0.0 		15006 	Trans: tls; Addr: *:50000                                                                        			Cluster: inbound|50000||
0.0.0.0 		15006 	Trans: raw_buffer; Addr: *:50000                                                                 			Cluster: inbound|50000||