Hi,
I have multicluster multinetwork setup and I am trying to connect to HTTPS service in second cluster.
I have HTTPS services for legacy reasons and I am unable to switch to HTTP in a timely manner.
I am in PERMISSIVE mode, so as long as I am within cluster, my “simple” HTTPS is working just fine as it is used instead of Istio’s mTLS.
Problem arises when I attempt to connect to second cluster via eastwestgateway:
Connecting from within cluster with curl:
If I originate TLS with curl, the request gets proxied all the way to HTTPS endpoint in cluster2. This is great.
Destination rule is empty in this case, I control TLS origination with CURL. It is my understanding this request gets proxied using ISTIO_MUTUAL.
POD:
2022-01-26T19:46:19.351661Z debug envoy filter [C11928[] new tcp proxy session
2022-01-26T19:46:19.351688Z debug envoy filter [C11928[] Creating connection to cluster outbound|443||ipg-vt-blue-svc.ipg-test3.svc.cluster.local
2022-01-26T19:46:19.351880Z debug envoy conn_handler [C11928[] new connection from 172.27.7.100:47764
2022-01-26T19:46:19.352482Z debug envoy filter [C11928[] Creating connection to cluster outbound|443||ipg-vt-blue-svc.ipg-test3.svc.cluster.local
2022-01-26T19:46:19.352590Z trace envoy conn_handler [C11928[] connection on event 1
EASTWESTGATEWAY:
2022-01-26T19:46:32.331771Z debug envoy filter tls inspector: new connection accepted
2022-01-26T19:46:32.331802Z trace envoy filter tls inspector: recv: 1398
2022-01-26T19:46:32.331845Z trace envoy filter tls inspector: recv: 2796
2022-01-26T19:46:32.331854Z trace envoy filter tls inspector: recv: 2796
2022-01-26T19:46:32.331903Z trace envoy filter tls inspector: recv: 3272
2022-01-26T19:46:32.331921Z trace envoy filter tls:onALPN(), ALPN: istio-peer-exchange,istio
2022-01-26T19:46:32.331935Z debug envoy filter tls:onServerName(), requestedServerName: outbound_.443_._.ipg-vt-blue-svc.ipg-test3.svc.cluster.local
2022-01-26T19:46:32.331942Z trace envoy filter tls inspector: done: true
2022-01-26T19:46:32.331985Z debug envoy filter [C13794[] new tcp proxy session
2022-01-26T19:46:32.332005Z debug envoy filter [C13794[] Creating connection to cluster outbound_.443_._.ipg-vt-blue-svc.ipg-test3.svc.cluster.local
2022-01-26T19:46:32.332179Z debug envoy conn_handler [C13794[] new connection from 172.27.9.0:53785
2022-01-26T19:46:32.332434Z debug envoy filter [C13794[] TCP:onUpstreamEvent(), requestedServerName: outbound_.443_._.ipg-vt-blue-svc.ipg-test3.svc.cluster.local
Connecting using ingressgateway from browser
If I originate request from browser via ingressroute, I have two options:
Option 1: DestinationRule is empty. Istio successfully connects to eastwestgateway and request fails with “Bad Request This combination of host and port requires TLS.” This is to be expected as ingressgateway has no idea that it should be using HTTPS insted of HTTP.
Option2: I specify TLS: SIMPLE for port in question in destination rule. While this potentially fixes the issue with protocol, it also replaces ISTIO_MUTUAL with SIMPLE and it completely breaks connectivity with eastwestgateway:
INGRESSGATEWAY:
2022-01-26T20:02:32.503066Z debug envoy pool creating a new connection
2022-01-26T20:02:32.503169Z debug envoy client [C9188[] connecting
2022-01-26T20:02:32.503176Z debug envoy connection [C9188[] connecting to 10.77.6.57:15443
2022-01-26T20:02:32.503271Z debug envoy connection [C9188[] connection in progress
2022-01-26T20:02:32.504194Z debug envoy connection [C9188[] connected
2022-01-26T20:02:32.504933Z debug envoy connection [C9188[] TLS error: 33554536:system library:OPENSSL_internal:Connection reset by peer
2022-01-26T20:02:32.504950Z debug envoy connection [C9188[] closing socket: 0
2022-01-26T20:02:32.504960Z debug envoy connection [C9188[] TLS error: 33554536:system library:OPENSSL_internal:Connection reset by peer
2022-01-26T20:02:32.504984Z debug envoy client [C9188[] disconnect. resetting 0 pending requests
2022-01-26T20:02:32.504991Z debug envoy pool [C9188[] client disconnected, failure reason: TLS error: 33554536:system library:OPENSSL_internal:Connection reset by peer
2022-01-26T20:02:32.505002Z debug envoy router [C4279[][S15010137060977388042[] upstream reset: reset reason: connection failure, transport failure reason: TLS error: 33554536:system library:OPENSSL_internal:Connection reset by peer
2022-01-26T20:02:32.505022Z debug envoy pool invoking idle callbacks - is_draining_for_deletion_=false
EASTWESTGATEWAY:
2022-01-26T20:02:32.504128Z debug envoy filter tls inspector: new connection accepted
2022-01-26T20:02:32.504171Z trace envoy filter tls inspector: recv: 130
2022-01-26T20:02:32.504191Z trace envoy filter tls:onALPN(), ALPN: http/1.1
2022-01-26T20:02:32.504216Z debug envoy conn_handler closing connection from 172.27.9.0:24906: no matching filter chain found
2022-01-26T20:02:32.525173Z debug envoy conn_handler [C14627[] new connection from 10.77.7.89:1495
2022-01-26T20:02:32.526008Z trace envoy conn_handler [C14627[] connection on event 0
2022-01-26T20:02:32.526028Z debug envoy conn_handler [C14627[] adding to cleanup list
Is there anyway around this? If I understand it correctly, setting TLS: SIMPLE does two things: tells ingressgateway to originate TLS + also replaces TLS mode with eastwestgateway. I would need to somehow decouple this configuration and only do the first without the second. I know this is technically possible as it works with curl, I am just unable to configure such behavior.