Hi. Hoping to get some clarity.
We are doing some performance testing to try and pin down a cause of random disconnects with our setup.
We have an elasticsearch running in GKE and are loading data via an Ingress GW from a Dataproc Cluster.
The direction is:
Dataproc > Loadbalancer > Ingress GW > istio-proxy sidecar > Elasticsearch
What we see is that randomly the requests get dropped and our loading tasks would fail.
We’ve done a lot of tests to try and pinpoint the cause, but no luck so far.
Some of the tests we tried were
- Direct to Pod access (no istio sidecar) without SSL. Result: No drops
- Direct to Pod access (no istio sidecar) with SSL. Result: No drops
- Via the Ingress GW, but mTLS and RBAC Disabled between Ingress GW and Sidecar. Result: No drops
- Via the Ingress GW, mTLS on by default. Result: Multiple drops
We don’t want to disable mTLS, but still interesting as to why it is more stable without mTLS/RBAC in place.
Everytime there is a failure, in the sidecar logs I get pretty much the same output:
[Envoy (Epoch 0)] [2020-11-30 17:24:58.611][28][debug][pool] [external/envoy/source/common/http/http1/conn_pool.cc:201] [C99] response complete
[Envoy (Epoch 0)] [2020-11-30 17:24:58.611][28][debug][pool] [external/envoy/source/common/http/http1/conn_pool.cc:239] [C99] moving to ready
[Envoy (Epoch 0)] [2020-11-30 17:24:58.611][28][debug][filter] [src/envoy/http/mixer/filter.cc:222] Called Mixer::Filter : log
[Envoy (Epoch 0)] [2020-11-30 17:24:58.611][28][debug][filter] [./src/envoy/http/mixer/report_data.h:144] No dynamic_metadata found for filter envoy.filters.http.rbac
[Envoy (Epoch 0)] [2020-11-30 17:24:58.612][28][debug][filter] [src/istio/control/client_context_base.cc:139] Report attributes: attributes {
key: "connection.mtls"
value {
bool_value: true
}
}
attributes {
key: "connection.requested_server_name"
value {
string_value: "outbound_.9200_._.elasticsearc
[Envoy (Epoch 0)] [2020-11-30 17:24:58.613][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:455] [C98] onBelowWriteBufferLowWatermark
[Envoy (Epoch 0)] [2020-11-30 17:24:58.872][28][debug][filter] [src/envoy/http/mixer/filter.cc:96] Called Mixer::Filter : decodeData (740, false)
[Envoy (Epoch 0)] [2020-11-30 17:24:58.877][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1276] [C79][S14371904417471549645] request end stream
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][filter] [src/envoy/http/mixer/filter.cc:96] Called Mixer::Filter : decodeData (0, true)
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:538] [C79] remote close
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:192] [C79] closing socket: 0
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:298] [C79] SSL shutdown: rc=1
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][connection] [external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:226] [C79]
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:1855] [C79][S14371904417471549645] stream reset
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][filter] [src/envoy/http/authn/http_filter.cc:49] Called AuthenticationFilter : onDestroy
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][filter] [src/envoy/http/mixer/filter.cc:208] Called Mixer::Filter : onDestroy state: 2
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][router] [external/envoy/source/common/router/router.cc:1657] [C79][S14371904417471549645] resetting pool request
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][client] [external/envoy/source/common/http/codec_client.cc:114] [C103] request reset
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:101] [C103] closing data_to_write=740 type=1
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][connection] [external/envoy/source/common/network/connection_impl.cc:192] [C103] closing socket: 1
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][client] [external/envoy/source/common/http/codec_client.cc:91] [C103] disconnect. resetting 0 pending requests
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][pool] [external/envoy/source/common/http/http1/conn_pool.cc:136] [C103] client disconnected, failure reason:
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][conn_handler] [external/envoy/source/server/connection_handler_impl.cc:86] [C79] adding to cleanup list
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][filter] [src/envoy/http/mixer/filter.cc:222] Called Mixer::Filter : log
[Envoy (Epoch 0)] [2020-11-30 17:24:58.879][28][debug][filter] [./src/envoy/http/mixer/report_data.h:144] No dynamic_metadata found for filter envoy.filters.http.rbac
[Envoy (Epoch 0)] [2020-11-30 17:24:58.880][28][debug][filter] [src/istio/control/client_context_base.cc:139] Report attributes: attributes {
key: "connection.mtls"
value {
bool_value: true
}
}
attributes {
key: "connection.requested_server_name"
value {
string_value: "outbound_.9200_._.elasticsearc
[Envoy (Epoch 0)] [2020-11-30 17:24:58.882][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:263] [C98] new stream
[Envoy (Epoch 0)] [2020-11-30 17:24:58.882][28][debug][http] [external/envoy/source/common/http/conn_manager_impl.cc:731] [C98][S1048916221831723008] request headers complete (end_stream=false):
Any suggestions on how to pin point what the cause is?