Istio 1.5 Ingress Gateway Proxy Crash

Hi,

We’re having a problem with the IngressGateway Envoy Proxy crashing regularly on incoming requests when our custom lua filter is applied in a particular order (running Istio 1.5.1). The lua filter (which does redirect and handles authn callback) must be after both of the envoy.filters.http.wasm filters (envoy.wasm.metadata_exchange, and envoy.wasm.stats) or the ingress envoy proxy will crash every few requests.

Not really sure what’s causing the crash, but it’s very consistent and goes away when removing and reapplying the Envoy Filter, which causes it to be applied in a different order.

Unfortunately, since Envoy doesn’t give us much control over the ordering (and we’re doing declarative deployments via Flux) we’re having trouble getting around this without being very hacky. Even when using the subfilter and INSERT_AFTER, the lua filter ends up between the two wasm filters (presumably since they’re named the same).

It seems there’s an unhandled exception here somewhere, but it’s really hard to pinpoint… See backtrace below.

Has anyone seen something similar? Any advice on narrowing the problem down more?

Also, it would be really nice to be able to make the filter order deterministic. If I’m right about the WASM filter naming getting in the way, do we need (or already have?) a different mechanism to specify filter order rather than name?

Here’s the backtrace from the crash:

2020-05-07T09:59:48.979281Z info No more active epochs, terminating
2020-05-07T09:59:48.979266Z error Epoch 0 exited with error: signal: aborted (core dumped)
2020-05-07T09:59:48.978645Z error sds Remote side closed connection
2020-05-07T09:59:48.978538Z info sds node:router~10.1.0.115~istio-ingressgateway-598df4fd84-hc6gb.istio-system~istio-system.svc.cluster.local-1 resource:ROOTCA connection is terminated: rpc error: code = Canceled desc = context canceled
2020-05-07T09:59:48.978508Z info transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-05-07T09:59:48.978498Z error sds Remote side closed connection
2020-05-07T09:59:48.978467Z info sds node:router~10.1.0.115~istio-ingressgateway-598df4fd84-hc6gb.istio-system~istio-system.svc.cluster.local-2 resource:default connection is terminated: rpc error: code = Canceled desc = context canceled
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #23: start_thread [0x7f4b31c346db]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #22: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::$_0::__invoke() [0x56048ac43f53]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #21: Envoy::Server::WorkerImpl::threadRoutine() [0x56048a73d818]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #20: event_base_loop [0x56048aa7781e]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #19: event_process_active_single_queue [0x56048aa78f8b]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #18: Envoy::Event::FileEventImpl::assignEvents()::$_0::__invoke() [0x56048a747b90]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #17: Envoy::Network::ConnectionImpl::onFileEvent() [0x56048a74d0bd]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #16: Envoy::Network::ConnectionImpl::onReadReady() [0x56048a74d5ea]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #15: Envoy::Network::FilterManagerImpl::onContinueReading() [0x56048a750b7b]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #14: Envoy::Http::ConnectionManagerImpl::onData() [0x56048a992451]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #13: Envoy::Http::Http2::ServerConnectionImpl::dispatch() [0x56048a9b4730]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #12: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x56048a9b0ad9]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #11: nghttp2_session_mem_recv [0x56048aa019fa]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #10: nghttp2_session_on_rst_stream_received [0x56048a9fe771]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #9: nghttp2_session_close_stream [0x56048a9fba67]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #8: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_12::__invoke() [0x56048a9b598c]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #7: Envoy::Http::Http2::ConnectionImpl::onStreamClose() [0x56048a9b250e]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #6: Envoy::Http::ConnectionManagerImpl::ActiveStream::onResetStream() [0x56048a99b919]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #5: Envoy::Http::ConnectionManagerImpl::doDeferredStreamDestroy() [0x56048a991c3e]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #4: Envoy::Extensions::Common::Wasm::Context::onDone() [0x560489a0fff7]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #3: std::__1::__function::__func<>::operator()() [0x56048931b628]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #2: std::__terminate() [0x56048b171073]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.823][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:77] #1: [0x56048ac0e2b9]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.808][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #0: __restore_rt [0x7f4b31c3f890]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.808][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:71] Envoy version: 6cdd2c56746ef582c8a851ed234c71719f9e3742/1.13.1-dev/Clean/RELEASE/BoringSSL
[Envoy (Epoch 0)] [2020-05-07 09:59:48.808][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:70] Backtrace (use tools/stack_decode.py to get line numbers):
[Envoy (Epoch 0)] [2020-05-07 09:59:48.808][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:83] Caught Aborted, suspect faulting address 0x10
[Envoy (Epoch 0)] [2020-05-07 09:59:48.808][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #23: start_thread [0x7f4b31c346db]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.808][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #22: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::$_0::__invoke() [0x56048ac43f53]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.792][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #21: Envoy::Server::WorkerImpl::threadRoutine() [0x56048a73d818]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.773][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #20: event_base_loop [0x56048aa7781e]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.755][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #19: event_process_active_single_queue [0x56048aa78f8b]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.738][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #18: Envoy::Event::FileEventImpl::assignEvents()::$_0::__invoke() [0x56048a747b90]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.727][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #17: Envoy::Network::ConnectionImpl::onFileEvent() [0x56048a74d0bd]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.713][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #16: Envoy::Network::ConnectionImpl::onReadReady() [0x56048a74d5ea]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.701][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #15: Envoy::Network::FilterManagerImpl::onContinueReading() [0x56048a750b7b]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.688][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #14: Envoy::Http::ConnectionManagerImpl::onData() [0x56048a992451]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.676][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #13: Envoy::Http::Http2::ServerConnectionImpl::dispatch() [0x56048a9b4730]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.663][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #12: Envoy::Http::Http2::ConnectionImpl::dispatch() [0x56048a9b0ad9]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.650][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #11: nghttp2_session_mem_recv [0x56048aa019fa]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.638][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #10: nghttp2_session_on_rst_stream_received [0x56048a9fe771]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.627][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #9: nghttp2_session_close_stream [0x56048a9fba67]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.614][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #8: Envoy::Http::Http2::ConnectionImpl::Http2Callbacks::Http2Callbacks()::$_12::__invoke() [0x56048a9b598c]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.601][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #7: Envoy::Http::Http2::ConnectionImpl::onStreamClose() [0x56048a9b250e]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.587][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #6: Envoy::Http::ConnectionManagerImpl::ActiveStream::onResetStream() [0x56048a99b919]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.575][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #5: Envoy::Http::ConnectionManagerImpl::doDeferredStreamDestroy() [0x56048a991c3e]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.564][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #4: Envoy::Extensions::Common::Wasm::Context::onDone() [0x560489a0fff7]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.553][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #3: std::__1::__function::__func<>::operator()() [0x56048931b628]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.543][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #2: std::__terminate() [0x56048b171073]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.533][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:77] #1: [0x56048ac0e2b9]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.517][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #0: Envoy::TerminateHandler::logOnTerminate()::$_0::operator()() [0x56048ac0e3ae]
[Envoy (Epoch 0)] [2020-05-07 09:59:48.502][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:71] Envoy version: 6cdd2c56746ef582c8a851ed234c71719f9e3742/1.13.1-dev/Clean/RELEASE/BoringSSL
[Envoy (Epoch 0)] [2020-05-07 09:59:48.502][26][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:70] Backtrace (use tools/stack_decode.py to get line numbers):
[Envoy (Epoch 0)] [2020-05-07 09:59:48.502][26][critical][main] [external/envoy/source/exe/terminate_handler.cc:13] std::terminate called! (possible uncaught exception, see trace)
[2020-05-07T09:59:48.011Z] "GET /Apis HTTP/2" 200 - "-" "-" 0 4924 126 - "10.1.1.250" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:76.0) Gecko/20100101 Firefox/76.0" ...
2020-05-07T09:59:37.301075Z info Envoy proxy is ready

Thanks.

This seems to be more widespread than I originally thought. Even after temporarily working around the problem at the ingressgateway level, we’re having very similar problems with pods with opa-istio applied (which also creates an envoy.ext_authz filter). The crash is less frequent - and maybe only be with multiple concurrent requests, but it’s still very common. In this case it doesn’t seem to make a difference what order the filters are applied in though. Once again, from the backtrace, this appears to be in the wasm onDone. We have confirmed the same behavior in Istio 1.5.2.

[Envoy (Epoch 0)] [2020-05-11 14:53:05.559][23][debug][pool] [external/envoy/source/common/http/http1/conn_pool.cc:104] [C1101] using existing connection
[Envoy (Epoch 0)] [2020-05-11 14:53:05.559][23][debug][router] [external/envoy/source/common/router/router.cc:1711] [C1159][S1228942189900616082] pool ready
[Envoy (Epoch 0)] [2020-05-11 14:53:05.560][23][debug][grpc] [external/envoy/source/common/grpc/google_async_client_impl.cc:379] Stream cleanup with 0 in-flight tags
[Envoy (Epoch 0)] [2020-05-11 14:53:05.560][23][debug][grpc] [external/envoy/source/common/grpc/google_async_client_impl.cc:368] Deferred delete
[Envoy (Epoch 0)] [2020-05-11 14:53:05.561][23][debug][grpc] [external/envoy/source/common/grpc/google_async_client_impl.cc:145] GoogleAsyncStreamImpl destruct
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:77] #1: [0x55a419f8d2b9]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #2: std::__terminate() [0x55a41a4f0073]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #3: std::__1::__function::__func<>::operator()() [0x55a41869a628]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #4: Envoy::Extensions::Common::Wasm::Context::onDone() [0x55a418d8eff7]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #5: Envoy::Http::ConnectionManagerImpl::doDeferredStreamDestroy() [0x55a419d10c3e]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #6: Envoy::Http::ConnectionManagerImpl::ActiveStream::onResetStream() [0x55a419d1a919]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #7: Envoy::Http::ConnectionManagerImpl::resetAllStreams() [0x55a419d11223]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #8: Envoy::Network::ConnectionImplBase::raiseConnectionEvent() [0x55a419acf02d]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #9: Envoy::Network::ConnectionImpl::raiseEvent() [0x55a419acb212]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #10: Envoy::Network::ConnectionImpl::closeSocket() [0x55a419acacd8]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #11: Envoy::Network::ConnectionImpl::onReadReady() [0x55a419acc6b2]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #12: Envoy::Network::ConnectionImpl::onFileEvent() [0x55a419acc0bd]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #13: Envoy::Event::FileEventImpl::assignEvents()::$_0::__invoke() [0x55a419ac6b90]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #14: event_process_active_single_queue [0x55a419df7f8b]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #15: event_base_loop [0x55a419df681e]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.574][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #16: Envoy::Server::WorkerImpl::threadRoutine() [0x55a419abc818]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.575][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #17: Envoy::Thread::ThreadImplPosix::ThreadImplPosix()::$_0::__invoke() [0x55a419fc2f53]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.575][24][critical][backtrace] [bazel-out/k8-opt/bin/external/envoy/source/server/_virtual_includes/backtrace_lib/server/backtrace.h:75] #18: start_thread [0x7f109b0ec6db]
[Envoy (Epoch 0)] [2020-05-11 14:53:05.575][23][debug][filter] [external/envoy/source/extensions/filters/listener/original_dst/original_dst.cc:18] original_dst: New connection accepted
2020-05-11T14:53:05.792862Z	info	sds	node:sidecar~10.1.1.187~eco-portal-prod-54d98ccbdb-bhtjx.ecosystem-prod~ecosystem-prod.svc.cluster.local-1 resource:ROOTCA connection is terminated: rpc error: code = Canceled desc = context canceled
2020-05-11T14:53:05.792895Z	error	sds	Remote side closed connection
2020-05-11T14:53:05.792941Z	info	transport: loopyWriter.run returning. connection error: desc = "transport is closing"
2020-05-11T14:53:05.801615Z	error	Epoch 0 exited with error: signal: aborted (core dumped)
2020-05-11T14:53:05.802136Z	info	No more active epochs, terminating
1 Like

I have same problem,

I haven’t been able to replicate this (either case describe above) after updating to 1.5.4. Are you able to try that?

Great! thanks for your information.
istio gw is good with1.5.4