Issue with Egress Gateway with TLS Origination


I am encountering a strange issue and wanted to see if someone else has stumbled across something similar. So, here it goes…

I am trying to route requests from internal services to an external service. However, we plan to migrate the external service into our K8s cluster which is why we want to use Istio MTLS from the get-go.
So, we wanted to setup an Egress Gateway with TLS origination for the external service.
I followed these instructions step-by-step: Istio / Egress Gateways with TLS Origination

For this specific service, everything works perfectly fine, i.e. my service inside K8s can reach the external service (CNN in this case).

However, when I change everything to use our own external service (currently running on a different cloud, I am getting a strange error message:
upstream connect error or disconnect/reset before headers. reset reason: connection failure, transport failure reason: TLS error: 268435581:SSL routines:OPENSSL_internal:CERTIFICATE_VERIFY_FAILED

I was literally starting to pull my hairs out after having tried a gazillion different permutations of various config changes. While the error message changed some times, the effect was the same: no TLS connection possible. When I changed the config back to a different external service it started working again.

Today I decided to inspect the different certificates and noticed something that I suspect to be a possible root cause:
Our server certificate has been issued by Let’s Encrypt while all the other certs are signed by different CA’s.
So, is it possible that Istio does not have Let’s Encrypt in its certificate chain?

Any idea that could get me to solve this would be highly appreciated!

P.S. There is no point in copying my config files here since they are identical to the one in the article on the site, except that I have replaced with my external server name, e.g. (of course the actual DNS name is different, but that should not make any difference).

UPDATE: I turned up the log level on Envoy to DEBUG and found the following error message:
2022-03-22T23:56:16.247595Z debug envoy connection [C11395] TLS error: 268436501:SSL routines:OPENSSL_internal:SSLV3_ALERT_CERTIFICATE_EXPIRED
After further inspection, I noticed that some certs on my egress-gateway seem to be expired. I just don’t know yet how that happened but I suspect the self-signed certs bundled with the latest release seem to have expired. I will check more…

UPDATE 2: this seems to be an issue in how Envoy calculates the expiration time of certificates which is then used by istiod to possibly renew certificates.
I reported this here: istiod failed to renew certificates in sidecars and gateways · Issue #38081 · istio/istio · GitHub

However, this seems to be an edge case and may not be too much of a concern for general Istio deployments.