How To: Ingress Gateway + TLS Termination and Routing -> HTTPS Service

Networking
1

First of all, thank you very much for this great piece of techonology. We love Istio :slight_smile:

After reading and experimenting with various ingress configurations the following question popped up in our team.

What is the best configuration if wanting to combine the nice features given by a Gateway + VirtualService which does TLS termination and provides the possibility to define routing rules (e.g. redirection, rewrite, avoiding public access to some of api paths, etc.) while a service can handle only HTTPS, i.e. no HTTP.

What have we done so far?

A) We have setup Metallb and Istio ingress properly. Our service is exposed to external world.

B) We have experimented with “Ingress Gateway without TLS Termination”: Istio / Ingress Gateway without TLS Termination

  • While we could successfully communicate with our service over HTTPS, we could not apply any uri-based routing rules, as this approach is basing on SNI.

C) We have experimented with “Secure Gateways”: Istio / Secure Gateways

  • While we could apply uri-based routing rules, we could not communicate with our service, as the communcation between Gateway and our service was over HTTP (despite the mTLS in between), again our service needs HTTPS.

D) We have experimented with “Ingress Sidecar TLS Termination”: Istio / Ingress Sidecar TLS Termination

  • We had some trouble to get this working, and we feel a central termination in a gateway would be our favorite way to go (what do you mean about this idea?).

I think technically we need a combination of B) and C) and possibly also D). Are there any best practices / examples for our use case? I feel our use case is not that special, there should be some simple setup for it, but we could not find it in documentation pages.

Any help is highly appreciated.

2

After some experiments, following setup was succesfull.

  1. Create a gateway with TLS termination
  2. Create a virtual service defining your routes and destinating your upstream service (using https port)
  3. Create a destination rule with TLS origination in SIMPLE mode
  4. Create a peer authentication for disabling it for your upstream service app

Point 4 took days to get figured out. Missing point 4 leads to TLS handshake failure between istio-proxy and your upstream service app, which results in an error message like below (which you usually see in your browser).

upstream connect error or disconnect/reset before headers. reset reason: connection termination

This problem seems to have popped up with version 1.14. See also here for some datails about this problem: ALPN filter incorrectly applies to non-Istio TLS traffic · Issue #40680 · istio/istio · GitHub

Hint: You can set the log level to debug in istio-ingressgateway deployment and take a look on the logs of its pod. The logs unveil details about the TLS handshake problem, if you don’t disable peer authentication for your service app as described in point 4 above. You will see some logs similar to this:

2023-08-11T07:34:36.401345Z debug envoy connection [C32170] remote address:10.1.69.219:9002,TLS error: 268436572:SSL routines:OPENSSL_internal:TLSV1_ALERT_CERTIFICATE_REQUIRED
2023-08-11T07:34:36.401386Z debug envoy connection [C32170] remote close
2023-08-11T07:34:36.401396Z debug envoy connection [C32170] closing socket: 0
2023-08-11T07:34:36.401439Z debug envoy connection [C32170] SSL shutdown: rc=-1
2023-08-11T07:34:36.401484Z debug envoy connection [C32170] remote address:10.1.69.219:9002,TLS error: 268436572:SSL routines:OPENSSL_internal:TLSV1_ALERT_CERTIFICATE_REQUIRED 33554464:system library:OPENSSL_internal:Broken pipe
2023-08-11T07:34:36.401565Z debug envoy client [C32170] disconnect. resetting 1 pending requests
2023-08-11T07:34:36.401580Z debug envoy client [C32170] request reset
2023-08-11T07:34:36.401597Z debug envoy router [C32150][S4870470944224312237] upstream reset: reset reason: connection termination, transport failure reason: 
2023-08-11T07:34:36.401671Z debug envoy http [C32150][S4870470944224312237] Sending local reply with details upstream_reset_before_response_started{connection_termination}
2023-08-11T07:34:36.401807Z debug envoy http [C32150][S4870470944224312237] encoding headers via codec (end_stream=false):
1 Like