We faced interesting issue today:
Our HAProxy host was not able to establish connection with ingress-gateway.
We checked as usually with curl and it was OK, we set up headers for health-check - still no luck
At some point we find out that curl requests with IP address are failing with:
LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to X.Y.W.Z:443
curl -vv https://X.Y.W.Z/api/myapp/v1.0/health -H "Host: my.application.k8s.host.com"
* Trying X.Y.W.Z...
* TCP_NODELAY set
* Connected to X.Y.W.Z (X.Y.W.Z) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* LibreSSL SSL_connect: SSL_ERROR_SYSCALL in connection to X.Y.W.Z:443
We did a check with openssl:
openssl s_client -connect my.application.k8s.host.com:443 -showcerts -msg
CONNECTED(00000006)
>>> TLS 1.2 Handshake [length 00bf], ClientHello
01 00 00 bb 03 03 05 69 90 b5 0d 2d 17 c2 6b b5
37 e5 40 ff fd 87 67 7b 5e 28 f1 e7 20 ca 89 fb
2a 65 aa 80 ef cc 00 00 5c aa 30 c0 2c c0 28 c0
24 c0 14 c0 0a 00 ff 00 6b 00 39 cc a9 cc a8 cc
aa ff aa 00 c4 00 88 00 81 00 9d 00 3d 00 35 00
c0 00 84 ff 2f c0 2b c0 27 c0 23 c0 13 c0 09 00
9e 00 67 00 33 00 be 00 45 00 aa 00 3c 00 2f 00
ba 00 41 c0 11 c0 07 ff 05 00 04 c0 12 c0 08 00
16 00 0a 00 ff 01 00 00 36 00 0b 00 02 01 00 00
0d 00 1c 00 1a 06 01 06 03 ff ef 05 01 05 03 04
ff 04 03 ee ee ed ed ff ff 03 03 02 01 ff 03
4409703916:error:140040E5:SSL routines:CONNECT_CR_SRVR_HELLO:ssl handshake failure:/BuildRoot/Library/Caches/com.apple.xbs/Sources/libressl/libressl-47.11.1/libressl-2.8/ssl/ssl_pkt.c:585:
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 0 bytes
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : 0000
Session-ID:
Session-ID-ctx:
Master-Key:
Start Time: 1570552589
Timeout : 7200 (sec)
Verify return code: 0 (ok)
---
And it looks like connection fails after HELLO.
Our setup has several gateway objects, each configured with appropriate hosts to serve like:
my.application.k8s.host.com
my2.application.k8s.host.com
my3.application.k8s.host.com
my4.application.k8s.host.com
Once we add a gateway with a * host match SSL connection start working.
According to RFC 5246 Client hello is
struct {
ProtocolVersion client_version;
Random random;
SessionID session_id;
CipherSuite cipher_suites<2..2^16-2>;
CompressionMethod compression_methods<1..2^8-1>;
select (extensions_present) {
case false:
struct {};
case true:
Extension extensions<0..2^16-1>;
};
} ClientHello;
Where Extension could contain server_name / Server Name Indication (SNI)
We did a test against traditional nginx ingress and it works just fine for IP address calls with curl and openssl.
Does Envoy handles SSL setup not like Nginx?
What we are missing in this puzzle?