jeff_r
March 6, 2020, 6:05pm
1
I’m having an issue with the ingress-gateways(v1.4.6) getting stuck at 100% vcpu. I cannot seem to find any correlation in the logs, nor is it consistent when it happens, it could happen 15 minutes after restarting the deployment or 10 hours, but generally within the 15-2 hour range the cpu will start climbing over a 5 minute period and then plateau until killed.
the traffic is not really substantial and the ingress gateway(s) handle it up until the point one of them starts peaking,
Platform: AWS EKS 1.4.9
client version: 1.4.5
control plane version: 1.4.6
data plane version: 1.4.6 (59 proxies)
Could anyone give me any pointers on troubleshooting this?
the following is our helm values file
galley:
enabled: true
replicaCount: 2
gateway_name: central-gateway
gateways:
enabled: true
istio-ingressgateway:
autoscaleMin: 2
secretVolumes:
- mountPath: /etc/istio/ingressgateway-cert
name: ingressgateway-cert
secretName: istio-ingressgateway-tls
type: NodePort
global:
enableTracing: true
k8sIngressSelector: ingressgateway
mtls:
enabled: false
proxy:
accessLogEncoding: JSON
accessLogFile: /dev/stdout
autoInject: disabled
protocolDetectionTimeout: 0s
grafana:
enabled: true
istio_namespace: istio-system
istio_version: 1.4.6
kiali:
enabled: true
mixer:
policy:
autoscaleMin: 2
enabled: true
telemetry:
autoscaleMin: 2
enabled: true
pilot:
autoscaleMin: 2
enableProtocolSniffingForOutbound: false
enabled: true
prometheus:
enabled: true
security:
enabled: true
replicaCount: 2
sidecarInjectorWebhook:
enabled: true
replicaCount: 2
tracing:
enabled: true
Have you experienced this on previous releases? Or is this new after upgrading to 1.4.6?
jeff_r
March 6, 2020, 6:13pm
3
I should have clarified that, we just started moving to istio in 1.4.2, We ran into this issue, upgrade to 1.4.4 and have since then upgraded to 1.4.5 -> 1.4.6 with the same experience.
One question i do have that I could not find an answer for,
We are currently running 1 central gateway + ingress gateway(s) for all services (10 or so). should each service have its own gateway + ingress gateway?
Could you have a quick diagnose on the cpu utilization?
Compared with normal ingressgateway, is the user cpu rising or sys cpu rising?
jeff_r
March 6, 2020, 6:49pm
5
currently 100% user @lambdai
pidstat -u -p 20 30
Linux 4.14.154-128.181.amzn2.x86_64 (istio-ingressgateway-cddb7ff6b-7grsl) 03/06/20 x86_64 (8 CPU)
18:47:20 UID PID %usr %system %guest %wait %CPU CPU Command
18:47:50 0 20 100.00 0.70 0.00 0.40 100.00 6 envoy
18:48:20 0 20 100.00 0.53 0.00 0.17 100.00 5 envoy
jeff_r:
EDIT: fix the profiler uri
Great! Could you run below? Updated to istio wiki
export POD=istio-ingressgateway-XXXX
export NS=istio-system
kubectl -n ${NS} exec "$POD" -c istio-proxy -- sh -c 'sudo mkdir -p /var/log/envoy && sudo chmod 777 /var/log/envoy && curl -X POST -s "http://localhost:15000/cpuprofiler?enable=y"'
sleep 15
kubectl -n ${NS} exec "$POD" -c istio-proxy -- sh -c 'curl -X POST -s "http://localhost:15000/cpuprofiler?enable=n"'
rm -rf /tmp/envoy
kubectl -n ${NS} cp "$POD":/var/log/envoy/ /tmp/envoy -c istio-proxy
kubectl -n ${NS} cp "$POD":/lib/x86_64-linux-gnu /tmp/envoy/lib -c istio-proxy
kubectl -n ${NS} cp "$POD":/usr/local/bin/envoy /tmp/envoy/lib/envoy -c istio-proxy
jeff_r
March 6, 2020, 7:27pm
7
is this what youre looking for @lambdai ?
Getting close!
I pasted forgetting to change the heap profile to cpu profile … one moment.
Super! Only one step away: debug symbol
I don’t know where we hide them
jeff_r
March 6, 2020, 8:42pm
12
@lambdai could you explain how to use that? I assume i build a new container and run that envoy?
thanks for the help!
I am trying teaching pprof to read from that envoy with debug symbol.
But if you can create a new proxyv2 container overriding the envoy binary in istio-proxyv2:1.4.6 by the one in https://gcsweb.istio.io/gcs/istio-build/proxy/envoy-symbol-f777dd13e28edc4ff394e98cd82c82ca4a2bcc71.tar.gz , redeploy istio-ingressgateway with the new image, wait until cpu climbing, rerun the profiling script. The visualize profiled should zoom in at the ‘[envoy]’
jeff_r
March 6, 2020, 9:09pm
14
File: envoy
Type: cpu
Showing nodes accounting for 33.01s, 100% of 33.01s total
----------------------------------------------------------+-------------
flat flat% sum% cum cum% calls calls% + context
----------------------------------------------------------+-------------
6.30s 99.84% | external/envoy/source/common/network/connection_impl.cc:295
0.01s 0.16% | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:182
6.31s 19.12% 19.12% 6.31s 19.12% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/connection_lib/common/network/connection_impl.h:111
----------------------------------------------------------+-------------
5s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
5s 15.15% 34.26% 5s 15.15% | external/envoy/source/common/network/connection_impl.cc
----------------------------------------------------------+-------------
4.61s 100% | external/envoy/source/common/network/connection_impl.cc:295
4.61s 13.97% 48.23% 4.61s 13.97% | external/envoy/source/common/network/io_socket_handle_impl.cc:36
----------------------------------------------------------+-------------
3.81s 100% | external/envoy/source/common/network/connection_impl.cc:295
3.81s 11.54% 59.77% 3.81s 11.54% | /usr/lib/llvm-9/include/c++/v1/memory:2558
----------------------------------------------------------+-------------
1.56s 71.89% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.61s 28.11% | external/envoy/source/common/network/connection_impl.cc:295
2.17s 6.57% 66.34% 2.17s 6.57% | external/envoy/source/common/network/connection_impl.cc:172
----------------------------------------------------------+-------------
1.36s 71.96% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.53s 28.04% | external/envoy/source/common/http/http1/codec_impl.cc:592
1.89s 5.73% 72.07% 1.89s 5.73% | external/envoy/source/common/network/connection_impl.cc:342
----------------------------------------------------------+-------------
1.13s 61.41% | external/envoy/source/common/network/connection_impl.cc:295
0.71s 38.59% | external/envoy/source/common/http/http1/codec_impl.cc:899
1.84s 5.57% 77.64% 1.84s 5.57% | external/envoy/source/common/network/connection_impl.cc:180
----------------------------------------------------------+-------------
1.45s 100% | external/envoy/source/common/network/connection_impl.cc:295
1.45s 4.39% 82.04% 1.45s 4.39% | external/envoy/source/common/network/connection_impl.cc:173
----------------------------------------------------------+-------------
1.27s 97.69% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.03s 2.31% | external/envoy/source/common/http/http1/codec_impl.cc:592
1.30s 3.94% 85.97% 1.30s 3.94% | external/envoy/source/common/network/connection_impl.cc:293
----------------------------------------------------------+-------------
1.26s 100% | external/envoy/source/common/network/connection_impl.cc:295
1.26s 3.82% 89.79% 1.26s 3.82% | /usr/lib/llvm-9/include/c++/v1/memory:2562
----------------------------------------------------------+-------------
20.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.95s 2.88% 92.67% 20.20s 61.19% | external/envoy/source/common/network/connection_impl.cc:295
6.30s 31.19% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/connection_lib/common/network/connection_impl.h:111
4.61s 22.82% | external/envoy/source/common/network/io_socket_handle_impl.cc:36
3.81s 18.86% | /usr/lib/llvm-9/include/c++/v1/memory:2558
1.45s 7.18% | external/envoy/source/common/network/connection_impl.cc:173
1.26s 6.24% | /usr/lib/llvm-9/include/c++/v1/memory:2562
1.13s 5.59% | external/envoy/source/common/network/connection_impl.cc:180
0.61s 3.02% | external/envoy/source/common/network/connection_impl.cc:172
0.08s 0.4% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/listen_socket_lib/common/network/listen_socket_impl.h:27
----------------------------------------------------------+-------------
0.74s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.74s 2.24% 94.91% 0.74s 2.24% | external/envoy/source/common/http/http1/codec_impl.cc:898
----------------------------------------------------------+-------------
25.78s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.68s 2.06% 96.97% 25.78s 78.10% | external/envoy/source/common/http/http1/codec_impl.cc:899
20.20s 78.36% | external/envoy/source/common/network/connection_impl.cc:295
1.56s 6.05% | external/envoy/source/common/network/connection_impl.cc:172
1.36s 5.28% | external/envoy/source/common/network/connection_impl.cc:342
1.27s 4.93% | external/envoy/source/common/network/connection_impl.cc:293
0.71s 2.75% | external/envoy/source/common/network/connection_impl.cc:180
----------------------------------------------------------+-------------
0.66s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.66s 2.00% 98.97% 0.66s 2.00% | external/envoy/source/common/network/connection_impl.cc:361
----------------------------------------------------------+-------------
32.82s 99.58% | external/envoy/source/server/worker_impl.cc:110
32.74s 99.33% | external/envoy/source/common/http/http1/codec_impl.cc:474
0.07s 0.21% | <unknown>
0.02s 0.061% | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:174
0.02s 0.061% | external/envoy/source/server/server.cc:579
0.01s 0.03% | /usr/lib/llvm-9/include/c++/v1/functional:1795
0.01s 0.03% | /usr/lib/llvm-9/include/c++/v1/functional:1860
0.01s 0.03% | src/envoy/http/mixer/filter.cc:247
0.17s 0.51% 99.49% 32.96s 99.85% | [envoy]
32.77s 99.42% | /usr/lib/llvm-9/include/c++/v1/functional:1860
32.74s 99.33% | external/envoy/source/common/http/http1/codec_impl.cc:376
0.05s 0.15% | /usr/lib/llvm-9/include/c++/v1/memory:2338
0.01s 0.03% | external/envoy/source/common/http/date_provider_impl.cc:21
0.01s 0.03% | src/envoy/http/mixer/report_data.h:89
----------------------------------------------------------+-------------
0.08s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.08s 0.24% 99.73% 0.08s 0.24% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/listen_socket_lib/common/network/listen_socket_impl.h:27
----------------------------------------------------------+-------------
0.01s 20.00% | <unknown>
0.03s 0.091% 99.82% 0.05s 0.15% | [libc-2.27.so]
0.02s 40.00% | main bazel-out/k8-opt/bin/external/envoy/source/exe/_virtual_includes/envoy_main_common_lib/exe/main_common.h:97
----------------------------------------------------------+-------------
0.02s 0.061% 99.88% 32.84s 99.49% | [libpthread-2.27.so]
32.82s 99.94% | /usr/lib/llvm-9/include/c++/v1/functional:1860
----------------------------------------------------------+-------------
0.01s 100% | /usr/lib/llvm-9/include/c++/v1/map:1519
0.01s 0.03% 99.91% 0.01s 0.03% | /usr/lib/llvm-9/include/c++/v1/__tree:2113
----------------------------------------------------------+-------------
0.01s 100% | bazel-out/k8-opt/bin/external/envoy/source/common/http/http2/_virtual_includes/codec_lib/common/http/http2/codec_impl.h:238
0.01s 0.03% 99.94% 0.01s 0.03% | /usr/lib/llvm-9/include/c++/v1/functional:1690
----------------------------------------------------------+-------------
0.01s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
0.01s 0.03% 100% 0.01s 0.03% | bazel-out/k8-opt/bin/external/envoy/source/common/upstream/_virtual_includes/upstream_includes/common/upstream/upstream_impl.h:121
----------------------------------------------------------+-------------
0.01s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
0.01s 0.03% 100% 0.01s 0.03% | getProtocolString external/envoy/source/common/http/utility.cc:390
----------------------------------------------------------+-------------
0.01s 100% | external/envoy/source/common/thread_local/thread_local_impl.cc:214
0 0% 100% 0.01s 0.03% | /usr/lib/llvm-9/include/c++/v1/functional:1795
0.01s 100% | [envoy]
----------------------------------------------------------+-------------
32.82s 100% | [libpthread-2.27.so]
32.77s 99.85% | [envoy]
0.03s 0.091% | external/envoy/source/common/access_log/access_log_formatter.cc:112
0 0% 100% 32.82s 99.42% | /usr/lib/llvm-9/include/c++/v1/functional:1860
32.82s 100% | external/envoy/source/server/worker_impl.cc:110
32.74s 99.76% | external/envoy/source/common/network/connection_impl.cc:502
0.03s 0.091% | external/envoy/source/common/network/connection_impl.cc:496
0.01s 0.03% | [envoy]
0.01s 0.03% | bazel-out/k8-opt/bin/external/envoy/source/common/upstream/_virtual_includes/upstream_includes/common/upstream/upstream_impl.h:121
0.01s 0.03% | getProtocolString external/envoy/source/common/http/utility.cc:390
----------------------------------------------------------+-------------
0.01s 100% | external/envoy/source/common/http/header_map_impl.cc:521
0 0% 100% 0.01s 0.03% | /usr/lib/llvm-9/include/c++/v1/map:1519
0.01s 100% | /usr/lib/llvm-9/include/c++/v1/__tree:2113
----------------------------------------------------------+-------------
0.05s 100% | [envoy]
0 0% 100% 0.05s 0.15% | /usr/lib/llvm-9/include/c++/v1/memory:2338
0.04s 80.00% | external/envoy/source/common/http/conn_manager_impl.cc:542
0.01s 20.00% | bazel-out/k8-opt/bin/external/envoy/source/common/http/http2/_virtual_includes/codec_lib/common/http/http2/codec_impl.h:238
----------------------------------------------------------+-------------
0 0% 100% 0.08s 0.24% | <unknown>
0.07s 87.50% | [envoy]
0.01s 12.50% | [libc-2.27.so]
----------------------------------------------------------+-------------
0.01s 100% | src/envoy/http/mixer/report_data.h:89
0 0% 100% 0.01s 0.03% | ExtractHeaders src/envoy/utils/utils.cc:73
0.01s 100% | external/envoy/source/common/http/header_map_impl.cc:521
----------------------------------------------------------+-------------
0.01s 100% | /usr/lib/llvm-9/include/c++/v1/memory:2338
0 0% 100% 0.01s 0.03% | bazel-out/k8-opt/bin/external/envoy/source/common/http/http2/_virtual_includes/codec_lib/common/http/http2/codec_impl.h:238
0.01s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1690
----------------------------------------------------------+-------------
0.03s 100% | external/envoy/source/common/access_log/access_log_formatter.cc:153
0 0% 100% 0.03s 0.091% | external/envoy/source/common/access_log/access_log_formatter.cc:112
0.03s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
----------------------------------------------------------+-------------
0.03s 100% | external/envoy/source/extensions/access_loggers/file/file_access_log_impl.cc:20
0 0% 100% 0.03s 0.091% | external/envoy/source/common/access_log/access_log_formatter.cc:129
0.03s 100% | external/envoy/source/common/access_log/access_log_formatter.cc:153
----------------------------------------------------------+-------------
0.03s 100% | external/envoy/source/common/access_log/access_log_formatter.cc:129
0 0% 100% 0.03s 0.091% | external/envoy/source/common/access_log/access_log_formatter.cc:153
0.03s 100% | external/envoy/source/common/access_log/access_log_formatter.cc:112
----------------------------------------------------------+-------------
32.74s 100% | external/envoy/source/common/http/codec_client.cc:82
0 0% 100% 32.74s 99.18% | external/envoy/source/common/http/codec_client.cc:122
32.74s 100% | external/envoy/source/common/http/http1/codec_impl.cc:462
----------------------------------------------------------+-------------
32.74s 100% | external/envoy/source/common/network/connection_impl.cc:348
0 0% 100% 32.74s 99.18% | external/envoy/source/common/http/codec_client.cc:82
32.74s 100% | external/envoy/source/common/http/codec_client.cc:122
----------------------------------------------------------+-------------
0.04s 100% | /usr/lib/llvm-9/include/c++/v1/memory:2338
0 0% 100% 0.04s 0.12% | external/envoy/source/common/http/conn_manager_impl.cc:542
0.03s 75.00% | external/envoy/source/common/http/conn_manager_impl.cc:565
0.01s 25.00% | external/envoy/source/common/http/conn_manager_impl.cc:569
----------------------------------------------------------+-------------
0.03s 100% | external/envoy/source/common/http/conn_manager_impl.cc:542
0 0% 100% 0.03s 0.091% | external/envoy/source/common/http/conn_manager_impl.cc:565
0.03s 100% | external/envoy/source/extensions/access_loggers/file/file_access_log_impl.cc:20
----------------------------------------------------------+-------------
0.01s 100% | external/envoy/source/common/http/conn_manager_impl.cc:542
0 0% 100% 0.01s 0.03% | external/envoy/source/common/http/conn_manager_impl.cc:569
0.01s 100% | src/envoy/http/mixer/filter.cc:247
----------------------------------------------------------+-------------
0.01s 100% | [envoy]
0 0% 100% 0.01s 0.03% | external/envoy/source/common/http/date_provider_impl.cc:21
0.01s 100% | external/envoy/source/common/thread_local/thread_local_impl.cc:100
----------------------------------------------------------+-------------
0.01s 100% | ExtractHeaders src/envoy/utils/utils.cc:73
0 0% 100% 0.01s 0.03% | external/envoy/source/common/http/header_map_impl.cc:521
0.01s 100% | /usr/lib/llvm-9/include/c++/v1/map:1519
----------------------------------------------------------+-------------
32.74s 100% | [envoy]
0 0% 100% 32.74s 99.18% | external/envoy/source/common/http/http1/codec_impl.cc:376
32.74s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
----------------------------------------------------------+-------------
32.74s 100% | external/envoy/source/common/http/codec_client.cc:122
0 0% 100% 32.74s 99.18% | external/envoy/source/common/http/http1/codec_impl.cc:462
32.74s 100% | external/envoy/source/common/http/http1/codec_impl.cc:474
----------------------------------------------------------+-------------
32.74s 100% | external/envoy/source/common/http/http1/codec_impl.cc:462
0 0% 100% 32.74s 99.18% | external/envoy/source/common/http/http1/codec_impl.cc:474
32.74s 100% | [envoy]
----------------------------------------------------------+-------------
32.74s 100% | external/envoy/source/common/http/http1/codec_impl.cc:376
0 0% 100% 32.74s 99.18% | external/envoy/source/common/http/http1/codec_impl.cc:592
25.78s 78.74% | external/envoy/source/common/http/http1/codec_impl.cc:899
5s 15.27% | external/envoy/source/common/network/connection_impl.cc
0.74s 2.26% | external/envoy/source/common/http/http1/codec_impl.cc:898
0.66s 2.02% | external/envoy/source/common/network/connection_impl.cc:361
0.53s 1.62% | external/envoy/source/common/network/connection_impl.cc:342
0.03s 0.092% | external/envoy/source/common/network/connection_impl.cc:293
----------------------------------------------------------+-------------
32.74s 100% | external/envoy/source/common/network/connection_impl.cc:535
0 0% 100% 32.74s 99.18% | external/envoy/source/common/network/connection_impl.cc:211
32.74s 100% | external/envoy/source/common/network/connection_impl.cc:348
----------------------------------------------------------+-------------
32.74s 100% | external/envoy/source/common/network/connection_impl.cc:211
0 0% 100% 32.74s 99.18% | external/envoy/source/common/network/connection_impl.cc:348
32.74s 100% | external/envoy/source/common/http/codec_client.cc:82
----------------------------------------------------------+-------------
0.03s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
0 0% 100% 0.03s 0.091% | external/envoy/source/common/network/connection_impl.cc:496
0.03s 100% | external/envoy/source/common/network/connection_impl.cc:581
----------------------------------------------------------+-------------
32.74s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
0 0% 100% 32.74s 99.18% | external/envoy/source/common/network/connection_impl.cc:502
32.74s 100% | external/envoy/source/common/network/connection_impl.cc:535
----------------------------------------------------------+-------------
32.74s 100% | external/envoy/source/common/network/connection_impl.cc:502
0 0% 100% 32.74s 99.18% | external/envoy/source/common/network/connection_impl.cc:535
32.74s 100% | external/envoy/source/common/network/connection_impl.cc:211
----------------------------------------------------------+-------------
0.03s 100% | external/envoy/source/common/network/connection_impl.cc:496
0 0% 100% 0.03s 0.091% | external/envoy/source/common/network/connection_impl.cc:581
0.03s 100% | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:235
----------------------------------------------------------+-------------
0.01s 100% | external/envoy/source/common/http/date_provider_impl.cc:21
0 0% 100% 0.01s 0.03% | external/envoy/source/common/thread_local/thread_local_impl.cc:100
0.01s 100% | external/envoy/source/common/thread_local/thread_local_impl.cc:214
----------------------------------------------------------+-------------
0.01s 100% | external/envoy/source/common/thread_local/thread_local_impl.cc:100
0 0% 100% 0.01s 0.03% | external/envoy/source/common/thread_local/thread_local_impl.cc:214
0.01s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1795
----------------------------------------------------------+-------------
0.02s 100% | main bazel-out/k8-opt/bin/external/envoy/source/exe/_virtual_includes/envoy_main_common_lib/exe/main_common.h:97
0 0% 100% 0.02s 0.061% | external/envoy/source/exe/main_common.cc:105
0.02s 100% | external/envoy/source/server/server.cc:579
----------------------------------------------------------+-------------
0.03s 100% | external/envoy/source/common/http/conn_manager_impl.cc:565
0 0% 100% 0.03s 0.091% | external/envoy/source/extensions/access_loggers/file/file_access_log_impl.cc:20
0.03s 100% | external/envoy/source/common/access_log/access_log_formatter.cc:129
----------------------------------------------------------+-------------
0.02s 100% | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:235
0 0% 100% 0.02s 0.061% | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:174
0.02s 100% | [envoy]
----------------------------------------------------------+-------------
0.01s 100% | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:235
0 0% 100% 0.01s 0.03% | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:182
0.01s 100% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/connection_lib/common/network/connection_impl.h:111
----------------------------------------------------------+-------------
0.03s 100% | external/envoy/source/common/network/connection_impl.cc:581
0 0% 100% 0.03s 0.091% | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:235
0.02s 66.67% | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:174
0.01s 33.33% | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:182
----------------------------------------------------------+-------------
0.02s 100% | external/envoy/source/exe/main_common.cc:105
0 0% 100% 0.02s 0.061% | external/envoy/source/server/server.cc:579
0.02s 100% | [envoy]
----------------------------------------------------------+-------------
32.82s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
0 0% 100% 32.82s 99.42% | external/envoy/source/server/worker_impl.cc:110
32.82s 100% | [envoy]
----------------------------------------------------------+-------------
0.02s 100% | [libc-2.27.so]
0 0% 100% 0.02s 0.061% | main bazel-out/k8-opt/bin/external/envoy/source/exe/_virtual_includes/envoy_main_common_lib/exe/main_common.h:97
0.02s 100% | external/envoy/source/exe/main_common.cc:105
----------------------------------------------------------+-------------
0.01s 100% | external/envoy/source/common/http/conn_manager_impl.cc:569
0 0% 100% 0.01s 0.03% | src/envoy/http/mixer/filter.cc:247
0.01s 100% | [envoy]
----------------------------------------------------------+-------------
0.01s 100% | [envoy]
0 0% 100% 0.01s 0.03% | src/envoy/http/mixer/report_data.h:89
0.01s 100% | ExtractHeaders src/envoy/utils/utils.cc:73
----------------------------------------------------------+-------------
@jeff_r
Thank you for the patience. I am sorry I haven’t figured out the root cause.
The perf output reveal the fact that the thread is trapped in the loop of
https://github.com/istio/envoy/blame/release-1.4.6-patch/source/common/http/http1/codec_impl.cc#L898-L900
I strongly believe it’s a infinite loop
The root cause is not clear to me yet.
Hello @lambdai i’m experience a similar issue some envoy proxy are hitting container cpu limit (2 core)
Here the attached cpu profiler when this is happening:
ile: envoy
Type: cpu
Showing nodes accounting for 16.21s, 100% of 16.21s total
----------------------------------------------------------±------------
flat flat% sum% cum cum% calls calls% + context
----------------------------------------------------------±------------
3.25s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
3.25s 20.05% 20.05% 3.25s 20.05% | external/envoy/source/common/network/connection_impl.cc
----------------------------------------------------------±------------
2.54s 100% | external/envoy/source/common/network/connection_impl.cc:295
2.54s 15.67% 35.72% 2.54s 15.67% | external/envoy/source/common/network/io_socket_handle_impl.cc:36
----------------------------------------------------------±------------
2.51s 100% | external/envoy/source/common/network/connection_impl.cc:295
2.51s 15.48% 51.20% 2.51s 15.48% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/connection_lib/common/network/connection_impl.h:111
----------------------------------------------------------±------------
1.87s 100% | external/envoy/source/common/network/connection_impl.cc:295
1.87s 11.54% 62.74% 1.87s 11.54% | /usr/lib/llvm-9/include/c++/v1/memory:2558
----------------------------------------------------------±------------
1.78s 100% | external/envoy/source/common/http/http1/codec_impl.cc:899
1.78s 10.98% 73.72% 1.78s 10.98% | external/envoy/source/common/network/connection_impl.cc:172
----------------------------------------------------------±------------
0.93s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.93s 5.74% 79.46% 0.93s 5.74% | /usr/lib/llvm-9/include/c++/v1/memory:2562
----------------------------------------------------------±------------
0.81s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.81s 5.00% 84.45% 0.81s 5.00% | external/envoy/source/common/http/http1/codec_impl.cc:898
----------------------------------------------------------±------------
0.65s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.65s 4.01% 88.46% 0.65s 4.01% | external/envoy/source/common/network/connection_impl.cc:173
----------------------------------------------------------±------------
0.51s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.51s 3.15% 91.61% 0.51s 3.15% | external/envoy/source/common/network/connection_impl.cc:180
----------------------------------------------------------±------------
0.50s 100% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.50s 3.08% 94.69% 0.50s 3.08% | external/envoy/source/common/network/connection_impl.cc:293
----------------------------------------------------------±------------
0.33s 94.29% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.02s 5.71% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.35s 2.16% 96.85% 0.35s 2.16% | external/envoy/source/common/network/connection_impl.cc:342
----------------------------------------------------------±------------
9.29s 100% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.20s 1.23% 98.09% 9.29s 57.31% | external/envoy/source/common/network/connection_impl.cc:295
2.54s 27.34% | external/envoy/source/common/network/io_socket_handle_impl.cc:36
2.51s 27.02% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/connection_lib/common/network/connection_impl.h:111
1.87s 20.13% | /usr/lib/llvm-9/include/c++/v1/memory:2558
0.93s 10.01% | /usr/lib/llvm-9/include/c++/v1/memory:2562
0.65s 7.00% | external/envoy/source/common/network/connection_impl.cc:173
0.51s 5.49% | external/envoy/source/common/network/connection_impl.cc:180
0.08s 0.86% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/listen_socket_lib/common/network/listen_socket_impl.h:27
----------------------------------------------------------±------------
12.09s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.19s 1.17% 99.26% 12.09s 74.58% | external/envoy/source/common/http/http1/codec_impl.cc:899
9.29s 76.84% | external/envoy/source/common/network/connection_impl.cc:295
1.78s 14.72% | external/envoy/source/common/network/connection_impl.cc:172
0.50s 4.14% | external/envoy/source/common/network/connection_impl.cc:293
0.33s 2.73% | external/envoy/source/common/network/connection_impl.cc:342
----------------------------------------------------------±------------
0.08s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.08s 0.49% 99.75% 0.08s 0.49% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/listen_socket_lib/common/network/listen_socket_impl.h:27
----------------------------------------------------------±------------
0.03s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.03s 0.19% 99.94% 0.03s 0.19% | external/envoy/source/common/network/connection_impl.cc:361
----------------------------------------------------------±------------
0.01s 0.062% 100% 0.01s 0.062% | [libc-2.27.so]
----------------------------------------------------------±------------
16.20s 100% | [envoy]
16.20s 100% | [libpthread-2.27.so]
0 0% 100% 16.20s 99.94% | /usr/lib/llvm-9/include/c++/v1/functional:1860
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:502
16.20s 100% | external/envoy/source/server/worker_impl.cc:110
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:474
16.20s 100% | external/envoy/source/server/worker_impl.cc:110
0 0% 100% 16.20s 99.94% | [envoy]
16.20s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:376
----------------------------------------------------------±------------
0 0% 100% 16.20s 99.94% | [libpthread-2.27.so]
16.20s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/codec_client.cc:82
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/codec_client.cc:122
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:462
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:348
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/codec_client.cc:82
16.20s 100% | external/envoy/source/common/http/codec_client.cc:122
----------------------------------------------------------±------------
16.20s 100% | [envoy]
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/http1/codec_impl.cc:376
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/codec_client.cc:122
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/http1/codec_impl.cc:462
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:474
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:462
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/http1/codec_impl.cc:474
16.20s 100% | [envoy]
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:376
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/http1/codec_impl.cc:592
12.09s 74.63% | external/envoy/source/common/http/http1/codec_impl.cc:899
3.25s 20.06% | external/envoy/source/common/network/connection_impl.cc
0.81s 5.00% | external/envoy/source/common/http/http1/codec_impl.cc:898
0.03s 0.19% | external/envoy/source/common/network/connection_impl.cc:361
0.02s 0.12% | external/envoy/source/common/network/connection_impl.cc:342
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:535
0 0% 100% 16.20s 99.94% | external/envoy/source/common/network/connection_impl.cc:211
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:348
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:211
0 0% 100% 16.20s 99.94% | external/envoy/source/common/network/connection_impl.cc:348
16.20s 100% | external/envoy/source/common/http/codec_client.cc:82
----------------------------------------------------------±------------
16.20s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
0 0% 100% 16.20s 99.94% | external/envoy/source/common/network/connection_impl.cc:502
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:535
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:502
0 0% 100% 16.20s 99.94% | external/envoy/source/common/network/connection_impl.cc:535
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:211
----------------------------------------------------------±------------
16.20s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
0 0% 100% 16.20s 99.94% | external/envoy/source/server/worker_impl.cc:110
16.20s 100% | [envoy]
----------------------------------------------------------±------------
jeff_r
March 25, 2020, 5:25pm
17
Upgrading to 1.5 has addressed my issue.
There was an issue in envoyproxy where it would busy wait on closed connections, see https://github.com/envoyproxy/envoy/pull/9509 for details.