Hello @lambdai i’m experience a similar issue some envoy proxy are hitting container cpu limit (2 core)
Here the attached cpu profiler when this is happening:
ile: envoy
Type: cpu
Showing nodes accounting for 16.21s, 100% of 16.21s total
----------------------------------------------------------±------------
flat flat% sum% cum cum% calls calls% + context
----------------------------------------------------------±------------
3.25s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
3.25s 20.05% 20.05% 3.25s 20.05% | external/envoy/source/common/network/connection_impl.cc
----------------------------------------------------------±------------
2.54s 100% | external/envoy/source/common/network/connection_impl.cc:295
2.54s 15.67% 35.72% 2.54s 15.67% | external/envoy/source/common/network/io_socket_handle_impl.cc:36
----------------------------------------------------------±------------
2.51s 100% | external/envoy/source/common/network/connection_impl.cc:295
2.51s 15.48% 51.20% 2.51s 15.48% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/connection_lib/common/network/connection_impl.h:111
----------------------------------------------------------±------------
1.87s 100% | external/envoy/source/common/network/connection_impl.cc:295
1.87s 11.54% 62.74% 1.87s 11.54% | /usr/lib/llvm-9/include/c++/v1/memory:2558
----------------------------------------------------------±------------
1.78s 100% | external/envoy/source/common/http/http1/codec_impl.cc:899
1.78s 10.98% 73.72% 1.78s 10.98% | external/envoy/source/common/network/connection_impl.cc:172
----------------------------------------------------------±------------
0.93s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.93s 5.74% 79.46% 0.93s 5.74% | /usr/lib/llvm-9/include/c++/v1/memory:2562
----------------------------------------------------------±------------
0.81s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.81s 5.00% 84.45% 0.81s 5.00% | external/envoy/source/common/http/http1/codec_impl.cc:898
----------------------------------------------------------±------------
0.65s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.65s 4.01% 88.46% 0.65s 4.01% | external/envoy/source/common/network/connection_impl.cc:173
----------------------------------------------------------±------------
0.51s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.51s 3.15% 91.61% 0.51s 3.15% | external/envoy/source/common/network/connection_impl.cc:180
----------------------------------------------------------±------------
0.50s 100% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.50s 3.08% 94.69% 0.50s 3.08% | external/envoy/source/common/network/connection_impl.cc:293
----------------------------------------------------------±------------
0.33s 94.29% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.02s 5.71% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.35s 2.16% 96.85% 0.35s 2.16% | external/envoy/source/common/network/connection_impl.cc:342
----------------------------------------------------------±------------
9.29s 100% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.20s 1.23% 98.09% 9.29s 57.31% | external/envoy/source/common/network/connection_impl.cc:295
2.54s 27.34% | external/envoy/source/common/network/io_socket_handle_impl.cc:36
2.51s 27.02% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/connection_lib/common/network/connection_impl.h:111
1.87s 20.13% | /usr/lib/llvm-9/include/c++/v1/memory:2558
0.93s 10.01% | /usr/lib/llvm-9/include/c++/v1/memory:2562
0.65s 7.00% | external/envoy/source/common/network/connection_impl.cc:173
0.51s 5.49% | external/envoy/source/common/network/connection_impl.cc:180
0.08s 0.86% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/listen_socket_lib/common/network/listen_socket_impl.h:27
----------------------------------------------------------±------------
12.09s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.19s 1.17% 99.26% 12.09s 74.58% | external/envoy/source/common/http/http1/codec_impl.cc:899
9.29s 76.84% | external/envoy/source/common/network/connection_impl.cc:295
1.78s 14.72% | external/envoy/source/common/network/connection_impl.cc:172
0.50s 4.14% | external/envoy/source/common/network/connection_impl.cc:293
0.33s 2.73% | external/envoy/source/common/network/connection_impl.cc:342
----------------------------------------------------------±------------
0.08s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.08s 0.49% 99.75% 0.08s 0.49% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/listen_socket_lib/common/network/listen_socket_impl.h:27
----------------------------------------------------------±------------
0.03s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.03s 0.19% 99.94% 0.03s 0.19% | external/envoy/source/common/network/connection_impl.cc:361
----------------------------------------------------------±------------
0.01s 0.062% 100% 0.01s 0.062% | [libc-2.27.so]
----------------------------------------------------------±------------
16.20s 100% | [envoy]
16.20s 100% | [libpthread-2.27.so]
0 0% 100% 16.20s 99.94% | /usr/lib/llvm-9/include/c++/v1/functional:1860
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:502
16.20s 100% | external/envoy/source/server/worker_impl.cc:110
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:474
16.20s 100% | external/envoy/source/server/worker_impl.cc:110
0 0% 100% 16.20s 99.94% | [envoy]
16.20s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:376
----------------------------------------------------------±------------
0 0% 100% 16.20s 99.94% | [libpthread-2.27.so]
16.20s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/codec_client.cc:82
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/codec_client.cc:122
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:462
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:348
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/codec_client.cc:82
16.20s 100% | external/envoy/source/common/http/codec_client.cc:122
----------------------------------------------------------±------------
16.20s 100% | [envoy]
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/http1/codec_impl.cc:376
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/codec_client.cc:122
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/http1/codec_impl.cc:462
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:474
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:462
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/http1/codec_impl.cc:474
16.20s 100% | [envoy]
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:376
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/http1/codec_impl.cc:592
12.09s 74.63% | external/envoy/source/common/http/http1/codec_impl.cc:899
3.25s 20.06% | external/envoy/source/common/network/connection_impl.cc
0.81s 5.00% | external/envoy/source/common/http/http1/codec_impl.cc:898
0.03s 0.19% | external/envoy/source/common/network/connection_impl.cc:361
0.02s 0.12% | external/envoy/source/common/network/connection_impl.cc:342
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:535
0 0% 100% 16.20s 99.94% | external/envoy/source/common/network/connection_impl.cc:211
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:348
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:211
0 0% 100% 16.20s 99.94% | external/envoy/source/common/network/connection_impl.cc:348
16.20s 100% | external/envoy/source/common/http/codec_client.cc:82
----------------------------------------------------------±------------
16.20s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
0 0% 100% 16.20s 99.94% | external/envoy/source/common/network/connection_impl.cc:502
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:535
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:502
0 0% 100% 16.20s 99.94% | external/envoy/source/common/network/connection_impl.cc:535
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:211
----------------------------------------------------------±------------
16.20s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
0 0% 100% 16.20s 99.94% | external/envoy/source/server/worker_impl.cc:110
16.20s 100% | [envoy]
----------------------------------------------------------±------------