I’m encountering a vexing issue with my Spring Boot microservices (with sidecar injection) connecting to a Consul service in a separate namespace without sidecar injection. The uServices are configured to talk with Consul via vanilla HTTP. If I deploy the uService without a sidecar, the service is able to connect without issue. It’s only when deploying with Istio that I get the following errors:
This is Istio 1.6.3
[2020-06-25T17:58:23.685Z] “GET /v1/kv/config/authserver-istio/?recurse&token= HTTP/1.1” 503 UF,URX “-” “TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER” 0 91 51 - “-” “Apache-HttpClient/4.5.8 (Java/1.8.0_191)” “d32771fe-2660-4b72-9671-24d979ac0182” “consul-server.core-qa-infra:8500” “10.80.3.130:8500” outbound|8500||consul-server.core-qa-infra.svc.cluster.local - 10.80.3.130:8500 10.80.2.36:45722 - default
It’s obviously trying to talk over HTTPS, but why? From the istio-proxy debug logs, I see the following:
2020-06-25T18:48:39.736442Z debug envoy http [external/envoy/source/common/http/conn_manager_impl.cc:268] [C2188] new stream
2020-06-25T18:48:39.736481Z debug envoy http [external/envoy/source/common/http/conn_manager_impl.cc:781] [C2188][S11123454195951414239] request headers complete (end_stream=true):
':authority', 'consul-server.core-qa-infra:8500'
':path', '/v1/kv/config/authserver-istio/?recurse&token='
':method', 'GET'
'connection', 'Keep-Alive'
'user-agent', 'Apache-HttpClient/4.5.8 (Java/1.8.0_191)'
'accept-encoding', 'gzip,deflate'
2020-06-25T18:48:39.736489Z debug envoy http [external/envoy/source/common/http/conn_manager_impl.cc:1333] [C2188][S11123454195951414239] request end stream
2020-06-25T18:48:39.736557Z debug envoy filter [src/envoy/http/alpn/alpn_filter.cc:81] override with 2 ALPNs
2020-06-25T18:48:39.736586Z debug envoy router [external/envoy/source/common/router/router.cc:477] [C2188][S11123454195951414239] cluster 'outbound|8500||consul-server.core-qa-infra.svc.cluster.local' match for URL '/v1/kv/config/authserver-istio/?recurse&token='
2020-06-25T18:48:39.736598Z debug envoy upstream [external/envoy/source/common/upstream/original_dst_cluster.cc:45] Using existing host 10.80.2.119:8500.
2020-06-25T18:48:39.736635Z debug envoy router [external/envoy/source/common/router/router.cc:634] [C2188][S11123454195951414239] router decoding headers:
':authority', 'consul-server.core-qa-infra:8500'
':path', '/v1/kv/config/authserver-istio/?recurse&token='
':method', 'GET'
':scheme', 'https'
'user-agent', 'Apache-HttpClient/4.5.8 (Java/1.8.0_191)'
'accept-encoding', 'gzip,deflate'
'x-forwarded-proto', 'http'
'x-request-id', '30e13e06-930a-47b6-bc43-99cd13c64b6f'
'x-envoy-decorator-operation', 'consul-server.core-qa-infra.svc.cluster.local:8500/*'
'x-envoy-peer-metadata', 'snip'
'x-envoy-peer-metadata-id', 'sidecar~10.80.2.36~authserver-69f96b58d5-s87v8.core-qa~core-qa.svc.cluster.local'
'x-envoy-attempt-count', '1'
'x-b3-traceid', 'f28aff7b7480bc2d752849ce408449af'
'x-b3-spanid', '752849ce408449af'
'x-b3-sampled', '0'
2020-06-25T18:48:39.736656Z debug envoy pool [external/envoy/source/common/http/conn_pool_base.cc:337] queueing request due to no available connections
2020-06-25T18:48:39.736664Z debug envoy pool [external/envoy/source/common/http/conn_pool_base.cc:47] creating a new connection
2020-06-25T18:48:39.736713Z debug envoy client [external/envoy/source/common/http/codec_client.cc:34] [C2203] connecting
2020-06-25T18:48:39.736721Z debug envoy connection [external/envoy/source/common/network/connection_impl.cc:727] [C2203] connecting to 10.80.2.119:8500
2020-06-25T18:48:39.736810Z debug envoy connection [external/envoy/source/common/network/connection_impl.cc:736] [C2203] connection in progress
2020-06-25T18:48:39.736870Z debug envoy connection [external/envoy/source/common/network/connection_impl.cc:592] [C2203] connected
2020-06-25T18:48:39.736935Z debug envoy connection [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:191] [C2203] handshake expecting read
2020-06-25T18:48:39.737060Z debug envoy connection [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:198] [C2203] handshake error: 1
2020-06-25T18:48:39.737071Z debug envoy connection [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:226] [C2203] TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
2020-06-25T18:48:39.737077Z debug envoy connection [external/envoy/source/common/network/connection_impl.cc:200] [C2203] closing socket: 0
2020-06-25T18:48:39.737093Z debug envoy client [external/envoy/source/common/http/codec_client.cc:91] [C2203] disconnect. resetting 0 pending requests
2020-06-25T18:48:39.737099Z debug envoy pool [external/envoy/source/common/http/conn_pool_base.cc:265] [C2203] client disconnected, failure reason: TLS error: 268435703:SSL routines:OPENSSL_internal:WRONG_VERSION_NUMBER
2020-06-25T18:48:39.737108Z debug envoy router [external/envoy/source/common/router/router.cc:1018] [C2188][S11123454195951414239] upstream reset: reset reason connection failure
2020-06-25T18:48:39.737129Z debug envoy router [external/envoy/source/common/router/router.cc:1390] [C2188][S11123454195951414239] performing retry
Interesting enough, when deployed in the mesh these uServices connect to Vault (another Hashicorp product) over HTTP without issue.
Any help debugging this would be greatly appreciated.