Istio mTLS strict mode with custom TLS

Networking
1

Hi.

I have Istio mTLS with STRICT mode enabled on my cluster.
I have two pods, spicedb server and spicedb client, which are communicating over GRPC with custom self-signed TLS (communication without custom TLS is not supported).

The issue I m facing is that the client is not able to communicate with server, when Istio mTLS STRICT mode is enabled.

grpc._channel._MultiThreadedRendezvous: <_MultiThreadedRendezvous of RPC that terminated with:
        status = StatusCode.UNAVAILABLE
        details = "failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.208.154:50051: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED"
        debug_error_string = "UNKNOWN:failed to connect to all addresses; last error: UNKNOWN: ipv4:10.43.208.154:50051: Ssl handshake failed: SSL_ERROR_SSL: error:1000007d:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED {created_time:"2023-09-21T13:45:25.73670136+00:00", grpc_status:14}"
>

STRICT mode is set at global level. I have tried various destination rules combination, but without success (SIMPLE mode, DISABLE mode, MUTUAL mode with TLS certs mounted).

Is anyone able to help understanding what am I doing wrong here?

2

A known issue with grpc.
Make sure the service port has name like grpc-xxx.
But imho the only way for those two pods to communicate is with exclusion from the mtls.
So you got two options:
Edit the mtls PA file and add there the excluded ports via special line. See peer authentication docs.
Or

Alternatively: add custom pod annotation to each pod to bypass istio mesh on that port. The annotation docs of istio will show you how to exclude a specific port, ports , range of ports.
.

3

thx @tomhawk , I have already excluded it via annotation, as separate line in mTLS PA did not work for me.

4

may be similar to the issue i reported ALPN filter incorrectly applies to non-Istio TLS traffic · Issue #40680 · istio/istio · GitHub
mine was with https, not grpc, see if yours is same issue