we got a similar requirement to use canary deployment in single mesh multi cluster environment similar to this issue
Background: we follow multi primary on different network using this guide, it works with AUTO_PASSTHROUGH set in Gateway tls mode in cluster2 however, after change it to use ISTIO_MUTUAL same as the requirement, request to helloworld starts to reporting
upstream connect error or disconnect/reset before headers. retried and the latest reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED
we cay this error from client, from this reply
the client proxy is expecting the peer’s identity to be spiffe://cluster.local/ns/sample/sa/default (i.e. mTLS should be terminated by the sample app proxy). With you configuration, the ingress is terminating mTLS, with identity spiffe://cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account, so client side cert verification fails. Typically the gateway should be configured as a AUTO_PASSTHROUGH, instead of terminating TLS for cross cluster traffics.
we guess this is related to trust domain validating, however there’s no option to disable this behaviour from client side, there’s option PILOT_SKIP_VALIDATE_TRUST_DOMAIN we tried that, seems this setting only validates from server side.
here’s the debug log from client
023-05-26T22:37:25.479674Z debug envoy http [C25708] new stream 2023-05-26T22:37:25.479748Z debug envoy http [C25708][S10242475731578652906] request headers complete (end_stream=true): ‘:authority’, ‘helloworld.sample.svc.cluster.local:5000’ ‘:path’, ‘/hello’ ‘:method’, ‘GET’ ‘user-agent’, ‘curl/8.1.1-DEV’ ‘accept’, ‘/’ 2023-05-26T22:37:25.479763Z debug envoy http [C25708][S10242475731578652906] request end stream 2023-05-26T22:37:25.479791Z debug envoy connection [C25708] current connecting state: false 2023-05-26T22:37:25.479943Z debug envoy router [C25708][S10242475731578652906] cluster ‘outbound|5000||helloworld.sample.svc.cluster.local’ match for URL ‘/hello’ 2023-05-26T22:37:25.480024Z debug envoy router [C25708][S10242475731578652906] router decoding headers: ‘:authority’, ‘helloworld.sample.svc.cluster.local:5000’ ‘:path’, ‘/hello’ ‘:method’, ‘GET’ ‘:scheme’, ‘http’ ‘user-agent’, ‘curl/8.1.1-DEV’ ‘accept’, ‘/’ ‘x-forwarded-proto’, ‘http’ ‘x-request-id’, ‘a6524713-e460-9e0d-8f64-e9ca5edc74ec’ ‘x-envoy-decorator-operation’, ‘helloworld.sample.svc.cluster.local:5000/*’ ‘x-envoy-peer-metadata’, ‘ChkKDkFQUF9DT05UQUlORVJTEgcaBXNsZWVwChkKCkNMVVNURVJfSUQSCxoJZngtZGV2LXVzCh8KDElOU1RBTkNFX0lQUxIPGg0xMC42MS4xMzkuMTE4ChkKDUlTVElPX1ZFUlNJT04SCBoGMS4xNi4wCt8BCgZMQUJFTFMS1AEq0QEKDgoDYXBwEgcaBXNsZWVwChIKBWlzdGlvEgkaB2VuYWJsZWQKJAoZc2VjdXJpdHkuaXN0aW8uaW8vdGxzTW9kZRIHGgVpc3RpbwoqCh9zZXJ2aWNlLmlzdGlvLmlvL2Nhbm9uaWNhbC1uYW1lEgcaBXNsZWVwCi8KI3NlcnZpY2UuaXN0aW8uaW8vY2Fub25pY2FsLXJldmlzaW9uEggaBmxhdGVzdAooChl0b3BvbG9neS5pc3Rpby5pby9uZXR3b3JrEgsaCWZ4LWRldi11cwoWCgdNRVNIX0lEEgsaCWZ4LWRldi11cwogCgROQU1FEhgaFnNsZWVwLTY1NWI3NzY5NTQtN2Y2cmgKFQoJTkFNRVNQQUNFEggaBnNhbXBsZQpICgVPV05FUhI/Gj1rdWJlcm5ldGVzOi8vYXBpcy9hcHBzL3YxL25hbWVzcGFjZXMvc2FtcGxlL2RlcGxveW1lbnRzL3NsZWVwCtkCChFQTEFURk9STV9NRVRBREFUQRLDAirAAgosChNnY3BfZ2NlX2luc3RhbmNlX2lkEhUaEzM0MDA2NjI2MDczOTYwNjAzNDkKIwoUZ2NwX2drZV9jbHVzdGVyX25hbWUSCxoJZngtZGV2LXVzCoEBChNnY3BfZ2tlX2NsdXN0ZXJfdXJsEmoaaGh0dHBzOi8vY29udGFpbmVyLmdvb2dsZWFwaXMuY29tL3YxL3Byb2plY3RzL2Z4LW5vbnByb2QtNzE2ZjI3MjMvbG9jYXRpb25zL3VzLWVhc3Q0LWEvY2x1c3RlcnMvZngtZGV2LXVzChwKDGdjcF9sb2NhdGlvbhIMGgp1cy1lYXN0NC1hCiQKC2djcF9wcm9qZWN0EhUaE2Z4LW5vbnByb2QtNzE2ZjI3MjMKIwoSZ2NwX3Byb2plY3RfbnVtYmVyEg0aCzkwMjA1NDkzMjUxChgKDVdPUktMT0FEX05BTUUSBxoFc2xlZXA=’ ‘x-envoy-peer-metadata-id’, ‘sidecar~10.61.139.118~sleep-655b776954-7f6rh.sample~sample.svc.cluster.local’ ‘x-envoy-attempt-count’, ‘1’ 2023-05-26T22:37:25.480052Z debug envoy pool queueing stream due to no available connections (ready=0 busy=0 connecting=0) 2023-05-26T22:37:25.480056Z debug envoy pool trying to create new connection 2023-05-26T22:37:25.480061Z debug envoy pool creating a new connection (connecting=0) 2023-05-26T22:37:25.480140Z debug envoy connection [C25709] current connecting state: true 2023-05-26T22:37:25.480153Z debug envoy connection [C25709] connecting to 10.60.8.54:15443 2023-05-26T22:37:25.480275Z debug envoy connection [C25709] connection in progress 2023-05-26T22:37:25.481848Z debug envoy connection [C25709] connected 2023-05-26T22:37:25.484905Z debug envoy connection verify cert failed: SAN matcher 2023-05-26T22:37:25.484979Z debug envoy connection [C25709] remote address:10.60.8.54:15443,TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED 2023-05-26T22:37:25.484987Z debug envoy connection [C25709] closing socket: 0 2023-05-26T22:37:25.485008Z debug envoy connection [C25709] remote address:10.60.8.54:15443,TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED 2023-05-26T22:37:25.485054Z debug envoy pool [C25709] client disconnected, failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED 2023-05-26T22:37:25.485084Z debug envoy router [C25708][S10242475731578652906] upstream reset: reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED 2023-05-26T22:37:25.485126Z debug envoy pool invoking idle callbacks - is_draining_for_deletion_=false 2023-05-26T22:37:25.491458Z debug envoy router [C25708][S10242475731578652906] performing retry 2023-05-26T22:37:25.491591Z debug envoy pool queueing stream due to no available connections (ready=0 busy=0 connecting=0) 2023-05-26T22:37:25.491603Z debug envoy pool trying to create new connection 2023-05-26T22:37:25.491606Z debug envoy pool creating a new connection (connecting=0) 2023-05-26T22:37:25.491726Z debug envoy connection [C25710] current connecting state: true 2023-05-26T22:37:25.491774Z debug envoy connection [C25710] connecting to 10.60.8.54:15443 2023-05-26T22:37:25.491892Z debug envoy connection [C25710] connection in progress 2023-05-26T22:37:25.496882Z debug envoy connection [C25710] connected 2023-05-26T22:37:25.499309Z debug envoy connection verify cert failed: SAN matcher 2023-05-26T22:37:25.499387Z debug envoy connection [C25710] remote address:10.60.8.54:15443,TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED 2023-05-26T22:37:25.499394Z debug envoy connection [C25710] closing socket: 0 2023-05-26T22:37:25.499417Z debug envoy connection [C25710] remote address:10.60.8.54:15443,TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED 2023-05-26T22:37:25.499459Z debug envoy pool [C25710] client disconnected, failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED 2023-05-26T22:37:25.499492Z debug envoy router [C25708][S10242475731578652906] upstream reset: reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED 2023-05-26T22:37:25.499540Z debug envoy pool invoking idle callbacks - is_draining_for_deletion_=false 2023-05-26T22:37:25.532795Z debug envoy router [C25708][S10242475731578652906] performing retry 2023-05-26T22:37:25.532949Z debug envoy pool queueing stream due to no available conne
it’s reporting verify cert failed: SAN matcher