Egress-gateway with TLS origination gives 503 TLS WRONG_VERSION_NUMBER

Networking
1

Hi all,

Hoping to get some help or insight into the right direction. We have a service which is hosted behind Cloudfront and only responds on 443. Our k8s cluster has been whitelisted by cloud front, so we want to use Istio as a proxy, however TLS termination is done on the load balancer and therefore we can’t do a simple passthrough. So, the end goal is:

GCP LB (443/80) => Istio-ingress (80) => SE, VS, DR => istio-egress (80->443) => Cloudfront (443).

This was failing so I broke it down to simpler parts to troubleshoot and even doing the TLS origination via egress-gw gives me a 503. If I do the CNN example from the documentation, then everything works. If I take that example verbatim and swap out the urls for the cloudfront service, then I start getting 503 errors with TLS ERROR WRONG_VERSION_NUMBER.

Any idea what could be causing this and what further I can attempt to solve the issue. Pulling at straws already as it seems forcing TLS version does nothing and I have no decent logs in Cloudfront to go on.

Thanks in advance

2

We had a similar issue and it was root caused to the Cloudfront instance requiring SNI on the TLS handshake for any connecting clients.

In this case, our client was the istio-proxy sidecar that was originating TLS for egress traffic on behalf of the application/workload.

So not via an istio-egress gateway as in your case, but this should be pretty similar.

To resolve, our choices were:

  1. to ensure that the Cloudfront instance’s configuration did not result in requiring SNI OR
  2. add the SNI in the TLS settings for the DestinationRule

We went with the 1st approach, but the 2nd was tried out as well and it did work.

1 Like
3

Thank you Naresh!!

I had implemented a workaround at the time, however you comment was super timely as today an issue arose that “broke” my workaround. Started digging into this and in fact issue was SNI. Since I didn’t have much access to the service I was routing to, troubleshooting was tough. Got it sorted out now adding SNI to tls settings in the DR.