Ingress-gateway cpu usage stuck at 100% vcpu allocated 1.4.6

jeff_r · March 6, 2020, 6:05pm

I’m having an issue with the ingress-gateways(v1.4.6) getting stuck at 100% vcpu. I cannot seem to find any correlation in the logs, nor is it consistent when it happens, it could happen 15 minutes after restarting the deployment or 10 hours, but generally within the 15-2 hour range the cpu will start climbing over a 5 minute period and then plateau until killed.

the traffic is not really substantial and the ingress gateway(s) handle it up until the point one of them starts peaking,

Platform: AWS EKS 1.4.9

client version: 1.4.5
control plane version: 1.4.6
data plane version: 1.4.6 (59 proxies)

Could anyone give me any pointers on troubleshooting this?

the following is our helm values file

galley:
  enabled: true
  replicaCount: 2
gateway_name: central-gateway
gateways:
  enabled: true
  istio-ingressgateway:
    autoscaleMin: 2
    secretVolumes:
    - mountPath: /etc/istio/ingressgateway-cert
      name: ingressgateway-cert
      secretName: istio-ingressgateway-tls
    type: NodePort
global:
  enableTracing: true
  k8sIngressSelector: ingressgateway
  mtls:
    enabled: false
  proxy:
    accessLogEncoding: JSON
    accessLogFile: /dev/stdout
    autoInject: disabled
    protocolDetectionTimeout: 0s
grafana:
  enabled: true
istio_namespace: istio-system
istio_version: 1.4.6
kiali:
  enabled: true
mixer:
  policy:
    autoscaleMin: 2
    enabled: true
  telemetry:
    autoscaleMin: 2
    enabled: true
pilot:
  autoscaleMin: 2
  enableProtocolSniffingForOutbound: false
  enabled: true
prometheus:
  enabled: true
security:
  enabled: true
  replicaCount: 2
sidecarInjectorWebhook:
  enabled: true
  replicaCount: 2
tracing:
  enabled: true

Screen Shot 2020-03-06 at 9.55.39 AM

howardjohn · March 6, 2020, 6:12pm

Have you experienced this on previous releases? Or is this new after upgrading to 1.4.6?

jeff_r · March 6, 2020, 6:13pm

I should have clarified that, we just started moving to istio in 1.4.2, We ran into this issue, upgrade to 1.4.4 and have since then upgraded to 1.4.5 -> 1.4.6 with the same experience.

One question i do have that I could not find an answer for,
We are currently running 1 central gateway + ingress gateway(s) for all services (10 or so). should each service have its own gateway + ingress gateway?

lambdai · March 6, 2020, 6:30pm

Could you have a quick diagnose on the cpu utilization?
Compared with normal ingressgateway, is the user cpu rising or sys cpu rising?

jeff_r · March 6, 2020, 6:49pm

currently 100% user @lambdai

pidstat -u -p 20 30
Linux 4.14.154-128.181.amzn2.x86_64 (istio-ingressgateway-cddb7ff6b-7grsl) 03/06/20 x86_64 (8 CPU)
18:47:20 UID PID %usr %system %guest %wait %CPU CPU Command
18:47:50 0 20 100.00 0.70 0.00 0.40 100.00 6 envoy
18:48:20 0 20 100.00 0.53 0.00 0.17 100.00 5 envoy

lambdai · March 6, 2020, 7:03pm

EDIT: fix the profiler uri

Great! Could you run below? Updated to istio wiki

export POD=istio-ingressgateway-XXXX
export NS=istio-system
kubectl -n ${NS} exec "$POD" -c istio-proxy -- sh -c 'sudo mkdir -p /var/log/envoy && sudo chmod 777 /var/log/envoy && curl -X POST -s "http://localhost:15000/cpuprofiler?enable=y"'
sleep 15
kubectl -n ${NS} exec "$POD" -c istio-proxy -- sh -c 'curl -X POST -s "http://localhost:15000/cpuprofiler?enable=n"'
rm -rf /tmp/envoy
kubectl -n ${NS} cp "$POD":/var/log/envoy/ /tmp/envoy -c istio-proxy
kubectl -n ${NS} cp "$POD":/lib/x86_64-linux-gnu /tmp/envoy/lib -c istio-proxy
kubectl -n ${NS} cp "$POD":/usr/local/bin/envoy /tmp/envoy/lib/envoy -c istio-proxy

jeff_r · March 6, 2020, 7:27pm

is this what youre looking for @lambdai?

lambdai · March 6, 2020, 7:47pm

Getting close!
I pasted forgetting to change the heap profile to cpu profile … one moment.

jeff_r · March 6, 2020, 7:58pm

lambdai · March 6, 2020, 8:09pm

Super! Only one step away: debug symbol
I don’t know where we hide them

lambdai · March 6, 2020, 8:25pm

1.4.6: https://gcsweb.istio.io/gcs/istio-build/proxy/envoy-symbol-f777dd13e28edc4ff394e98cd82c82ca4a2bcc71.tar.gz

jeff_r · March 6, 2020, 8:42pm

@lambdai could you explain how to use that? I assume i build a new container and run that envoy?
thanks for the help!

lambdai · March 6, 2020, 8:50pm

I am trying teaching pprof to read from that envoy with debug symbol.

But if you can create a new proxyv2 container overriding the envoy binary in istio-proxyv2:1.4.6 by the one in https://gcsweb.istio.io/gcs/istio-build/proxy/envoy-symbol-f777dd13e28edc4ff394e98cd82c82ca4a2bcc71.tar.gz , redeploy istio-ingressgateway with the new image, wait until cpu climbing, rerun the profiling script. The visualize profiled should zoom in at the ‘[envoy]’

jeff_r · March 6, 2020, 9:09pm

      File: envoy
Type: cpu
Showing nodes accounting for 33.01s, 100% of 33.01s total
----------------------------------------------------------+-------------
      flat  flat%   sum%        cum   cum%   calls calls% + context 	 	 
----------------------------------------------------------+-------------
                                             6.30s 99.84% |   external/envoy/source/common/network/connection_impl.cc:295
                                             0.01s  0.16% |   external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:182
     6.31s 19.12% 19.12%      6.31s 19.12%                | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/connection_lib/common/network/connection_impl.h:111
----------------------------------------------------------+-------------
                                                5s   100% |   external/envoy/source/common/http/http1/codec_impl.cc:592
        5s 15.15% 34.26%         5s 15.15%                | external/envoy/source/common/network/connection_impl.cc
----------------------------------------------------------+-------------
                                             4.61s   100% |   external/envoy/source/common/network/connection_impl.cc:295
     4.61s 13.97% 48.23%      4.61s 13.97%                | external/envoy/source/common/network/io_socket_handle_impl.cc:36
----------------------------------------------------------+-------------
                                             3.81s   100% |   external/envoy/source/common/network/connection_impl.cc:295
     3.81s 11.54% 59.77%      3.81s 11.54%                | /usr/lib/llvm-9/include/c++/v1/memory:2558
----------------------------------------------------------+-------------
                                             1.56s 71.89% |   external/envoy/source/common/http/http1/codec_impl.cc:899
                                             0.61s 28.11% |   external/envoy/source/common/network/connection_impl.cc:295
     2.17s  6.57% 66.34%      2.17s  6.57%                | external/envoy/source/common/network/connection_impl.cc:172
----------------------------------------------------------+-------------
                                             1.36s 71.96% |   external/envoy/source/common/http/http1/codec_impl.cc:899
                                             0.53s 28.04% |   external/envoy/source/common/http/http1/codec_impl.cc:592
     1.89s  5.73% 72.07%      1.89s  5.73%                | external/envoy/source/common/network/connection_impl.cc:342
----------------------------------------------------------+-------------
                                             1.13s 61.41% |   external/envoy/source/common/network/connection_impl.cc:295
                                             0.71s 38.59% |   external/envoy/source/common/http/http1/codec_impl.cc:899
     1.84s  5.57% 77.64%      1.84s  5.57%                | external/envoy/source/common/network/connection_impl.cc:180
----------------------------------------------------------+-------------
                                             1.45s   100% |   external/envoy/source/common/network/connection_impl.cc:295
     1.45s  4.39% 82.04%      1.45s  4.39%                | external/envoy/source/common/network/connection_impl.cc:173
----------------------------------------------------------+-------------
                                             1.27s 97.69% |   external/envoy/source/common/http/http1/codec_impl.cc:899
                                             0.03s  2.31% |   external/envoy/source/common/http/http1/codec_impl.cc:592
     1.30s  3.94% 85.97%      1.30s  3.94%                | external/envoy/source/common/network/connection_impl.cc:293
----------------------------------------------------------+-------------
                                             1.26s   100% |   external/envoy/source/common/network/connection_impl.cc:295
     1.26s  3.82% 89.79%      1.26s  3.82%                | /usr/lib/llvm-9/include/c++/v1/memory:2562
----------------------------------------------------------+-------------
                                            20.20s   100% |   external/envoy/source/common/http/http1/codec_impl.cc:899
     0.95s  2.88% 92.67%     20.20s 61.19%                | external/envoy/source/common/network/connection_impl.cc:295
                                             6.30s 31.19% |   bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/connection_lib/common/network/connection_impl.h:111
                                             4.61s 22.82% |   external/envoy/source/common/network/io_socket_handle_impl.cc:36
                                             3.81s 18.86% |   /usr/lib/llvm-9/include/c++/v1/memory:2558
                                             1.45s  7.18% |   external/envoy/source/common/network/connection_impl.cc:173
                                             1.26s  6.24% |   /usr/lib/llvm-9/include/c++/v1/memory:2562
                                             1.13s  5.59% |   external/envoy/source/common/network/connection_impl.cc:180
                                             0.61s  3.02% |   external/envoy/source/common/network/connection_impl.cc:172
                                             0.08s   0.4% |   bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/listen_socket_lib/common/network/listen_socket_impl.h:27
----------------------------------------------------------+-------------
                                             0.74s   100% |   external/envoy/source/common/http/http1/codec_impl.cc:592
     0.74s  2.24% 94.91%      0.74s  2.24%                | external/envoy/source/common/http/http1/codec_impl.cc:898
----------------------------------------------------------+-------------
                                            25.78s   100% |   external/envoy/source/common/http/http1/codec_impl.cc:592
     0.68s  2.06% 96.97%     25.78s 78.10%                | external/envoy/source/common/http/http1/codec_impl.cc:899
                                            20.20s 78.36% |   external/envoy/source/common/network/connection_impl.cc:295
                                             1.56s  6.05% |   external/envoy/source/common/network/connection_impl.cc:172
                                             1.36s  5.28% |   external/envoy/source/common/network/connection_impl.cc:342
                                             1.27s  4.93% |   external/envoy/source/common/network/connection_impl.cc:293
                                             0.71s  2.75% |   external/envoy/source/common/network/connection_impl.cc:180
----------------------------------------------------------+-------------
                                             0.66s   100% |   external/envoy/source/common/http/http1/codec_impl.cc:592
     0.66s  2.00% 98.97%      0.66s  2.00%                | external/envoy/source/common/network/connection_impl.cc:361
----------------------------------------------------------+-------------
                                            32.82s 99.58% |   external/envoy/source/server/worker_impl.cc:110
                                            32.74s 99.33% |   external/envoy/source/common/http/http1/codec_impl.cc:474
                                             0.07s  0.21% |   <unknown>
                                             0.02s 0.061% |   external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:174
                                             0.02s 0.061% |   external/envoy/source/server/server.cc:579
                                             0.01s  0.03% |   /usr/lib/llvm-9/include/c++/v1/functional:1795
                                             0.01s  0.03% |   /usr/lib/llvm-9/include/c++/v1/functional:1860
                                             0.01s  0.03% |   src/envoy/http/mixer/filter.cc:247
     0.17s  0.51% 99.49%     32.96s 99.85%                | [envoy]
                                            32.77s 99.42% |   /usr/lib/llvm-9/include/c++/v1/functional:1860
                                            32.74s 99.33% |   external/envoy/source/common/http/http1/codec_impl.cc:376
                                             0.05s  0.15% |   /usr/lib/llvm-9/include/c++/v1/memory:2338
                                             0.01s  0.03% |   external/envoy/source/common/http/date_provider_impl.cc:21
                                             0.01s  0.03% |   src/envoy/http/mixer/report_data.h:89
----------------------------------------------------------+-------------
                                             0.08s   100% |   external/envoy/source/common/network/connection_impl.cc:295
     0.08s  0.24% 99.73%      0.08s  0.24%                | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/listen_socket_lib/common/network/listen_socket_impl.h:27
----------------------------------------------------------+-------------
                                             0.01s 20.00% |   <unknown>
     0.03s 0.091% 99.82%      0.05s  0.15%                | [libc-2.27.so]
                                             0.02s 40.00% |   main bazel-out/k8-opt/bin/external/envoy/source/exe/_virtual_includes/envoy_main_common_lib/exe/main_common.h:97
----------------------------------------------------------+-------------
     0.02s 0.061% 99.88%     32.84s 99.49%                | [libpthread-2.27.so]
                                            32.82s 99.94% |   /usr/lib/llvm-9/include/c++/v1/functional:1860
----------------------------------------------------------+-------------
                                             0.01s   100% |   /usr/lib/llvm-9/include/c++/v1/map:1519
     0.01s  0.03% 99.91%      0.01s  0.03%                | /usr/lib/llvm-9/include/c++/v1/__tree:2113
----------------------------------------------------------+-------------
                                             0.01s   100% |   bazel-out/k8-opt/bin/external/envoy/source/common/http/http2/_virtual_includes/codec_lib/common/http/http2/codec_impl.h:238
     0.01s  0.03% 99.94%      0.01s  0.03%                | /usr/lib/llvm-9/include/c++/v1/functional:1690
----------------------------------------------------------+-------------
                                             0.01s   100% |   /usr/lib/llvm-9/include/c++/v1/functional:1860
     0.01s  0.03%   100%      0.01s  0.03%                | bazel-out/k8-opt/bin/external/envoy/source/common/upstream/_virtual_includes/upstream_includes/common/upstream/upstream_impl.h:121
----------------------------------------------------------+-------------
                                             0.01s   100% |   /usr/lib/llvm-9/include/c++/v1/functional:1860
     0.01s  0.03%   100%      0.01s  0.03%                | getProtocolString external/envoy/source/common/http/utility.cc:390
----------------------------------------------------------+-------------
                                             0.01s   100% |   external/envoy/source/common/thread_local/thread_local_impl.cc:214
         0     0%   100%      0.01s  0.03%                | /usr/lib/llvm-9/include/c++/v1/functional:1795
                                             0.01s   100% |   [envoy]
----------------------------------------------------------+-------------
                                            32.82s   100% |   [libpthread-2.27.so]
                                            32.77s 99.85% |   [envoy]
                                             0.03s 0.091% |   external/envoy/source/common/access_log/access_log_formatter.cc:112
         0     0%   100%     32.82s 99.42%                | /usr/lib/llvm-9/include/c++/v1/functional:1860
                                            32.82s   100% |   external/envoy/source/server/worker_impl.cc:110
                                            32.74s 99.76% |   external/envoy/source/common/network/connection_impl.cc:502
                                             0.03s 0.091% |   external/envoy/source/common/network/connection_impl.cc:496
                                             0.01s  0.03% |   [envoy]
                                             0.01s  0.03% |   bazel-out/k8-opt/bin/external/envoy/source/common/upstream/_virtual_includes/upstream_includes/common/upstream/upstream_impl.h:121
                                             0.01s  0.03% |   getProtocolString external/envoy/source/common/http/utility.cc:390
----------------------------------------------------------+-------------
                                             0.01s   100% |   external/envoy/source/common/http/header_map_impl.cc:521
         0     0%   100%      0.01s  0.03%                | /usr/lib/llvm-9/include/c++/v1/map:1519
                                             0.01s   100% |   /usr/lib/llvm-9/include/c++/v1/__tree:2113
----------------------------------------------------------+-------------
                                             0.05s   100% |   [envoy]
         0     0%   100%      0.05s  0.15%                | /usr/lib/llvm-9/include/c++/v1/memory:2338
                                             0.04s 80.00% |   external/envoy/source/common/http/conn_manager_impl.cc:542
                                             0.01s 20.00% |   bazel-out/k8-opt/bin/external/envoy/source/common/http/http2/_virtual_includes/codec_lib/common/http/http2/codec_impl.h:238
----------------------------------------------------------+-------------
         0     0%   100%      0.08s  0.24%                | <unknown>
                                             0.07s 87.50% |   [envoy]
                                             0.01s 12.50% |   [libc-2.27.so]
----------------------------------------------------------+-------------
                                             0.01s   100% |   src/envoy/http/mixer/report_data.h:89
         0     0%   100%      0.01s  0.03%                | ExtractHeaders src/envoy/utils/utils.cc:73
                                             0.01s   100% |   external/envoy/source/common/http/header_map_impl.cc:521
----------------------------------------------------------+-------------
                                             0.01s   100% |   /usr/lib/llvm-9/include/c++/v1/memory:2338
         0     0%   100%      0.01s  0.03%                | bazel-out/k8-opt/bin/external/envoy/source/common/http/http2/_virtual_includes/codec_lib/common/http/http2/codec_impl.h:238
                                             0.01s   100% |   /usr/lib/llvm-9/include/c++/v1/functional:1690
----------------------------------------------------------+-------------
                                             0.03s   100% |   external/envoy/source/common/access_log/access_log_formatter.cc:153
         0     0%   100%      0.03s 0.091%                | external/envoy/source/common/access_log/access_log_formatter.cc:112
                                             0.03s   100% |   /usr/lib/llvm-9/include/c++/v1/functional:1860
----------------------------------------------------------+-------------
                                             0.03s   100% |   external/envoy/source/extensions/access_loggers/file/file_access_log_impl.cc:20
         0     0%   100%      0.03s 0.091%                | external/envoy/source/common/access_log/access_log_formatter.cc:129
                                             0.03s   100% |   external/envoy/source/common/access_log/access_log_formatter.cc:153
----------------------------------------------------------+-------------
                                             0.03s   100% |   external/envoy/source/common/access_log/access_log_formatter.cc:129
         0     0%   100%      0.03s 0.091%                | external/envoy/source/common/access_log/access_log_formatter.cc:153
                                             0.03s   100% |   external/envoy/source/common/access_log/access_log_formatter.cc:112
----------------------------------------------------------+-------------
                                            32.74s   100% |   external/envoy/source/common/http/codec_client.cc:82
         0     0%   100%     32.74s 99.18%                | external/envoy/source/common/http/codec_client.cc:122
                                            32.74s   100% |   external/envoy/source/common/http/http1/codec_impl.cc:462
----------------------------------------------------------+-------------
                                            32.74s   100% |   external/envoy/source/common/network/connection_impl.cc:348
         0     0%   100%     32.74s 99.18%                | external/envoy/source/common/http/codec_client.cc:82
                                            32.74s   100% |   external/envoy/source/common/http/codec_client.cc:122
----------------------------------------------------------+-------------
                                             0.04s   100% |   /usr/lib/llvm-9/include/c++/v1/memory:2338
         0     0%   100%      0.04s  0.12%                | external/envoy/source/common/http/conn_manager_impl.cc:542
                                             0.03s 75.00% |   external/envoy/source/common/http/conn_manager_impl.cc:565
                                             0.01s 25.00% |   external/envoy/source/common/http/conn_manager_impl.cc:569
----------------------------------------------------------+-------------
                                             0.03s   100% |   external/envoy/source/common/http/conn_manager_impl.cc:542
         0     0%   100%      0.03s 0.091%                | external/envoy/source/common/http/conn_manager_impl.cc:565
                                             0.03s   100% |   external/envoy/source/extensions/access_loggers/file/file_access_log_impl.cc:20
----------------------------------------------------------+-------------
                                             0.01s   100% |   external/envoy/source/common/http/conn_manager_impl.cc:542
         0     0%   100%      0.01s  0.03%                | external/envoy/source/common/http/conn_manager_impl.cc:569
                                             0.01s   100% |   src/envoy/http/mixer/filter.cc:247
----------------------------------------------------------+-------------
                                             0.01s   100% |   [envoy]
         0     0%   100%      0.01s  0.03%                | external/envoy/source/common/http/date_provider_impl.cc:21
                                             0.01s   100% |   external/envoy/source/common/thread_local/thread_local_impl.cc:100
----------------------------------------------------------+-------------
                                             0.01s   100% |   ExtractHeaders src/envoy/utils/utils.cc:73
         0     0%   100%      0.01s  0.03%                | external/envoy/source/common/http/header_map_impl.cc:521
                                             0.01s   100% |   /usr/lib/llvm-9/include/c++/v1/map:1519
----------------------------------------------------------+-------------
                                            32.74s   100% |   [envoy]
         0     0%   100%     32.74s 99.18%                | external/envoy/source/common/http/http1/codec_impl.cc:376
                                            32.74s   100% |   external/envoy/source/common/http/http1/codec_impl.cc:592
----------------------------------------------------------+-------------
                                            32.74s   100% |   external/envoy/source/common/http/codec_client.cc:122
         0     0%   100%     32.74s 99.18%                | external/envoy/source/common/http/http1/codec_impl.cc:462
                                            32.74s   100% |   external/envoy/source/common/http/http1/codec_impl.cc:474
----------------------------------------------------------+-------------
                                            32.74s   100% |   external/envoy/source/common/http/http1/codec_impl.cc:462
         0     0%   100%     32.74s 99.18%                | external/envoy/source/common/http/http1/codec_impl.cc:474
                                            32.74s   100% |   [envoy]
----------------------------------------------------------+-------------
                                            32.74s   100% |   external/envoy/source/common/http/http1/codec_impl.cc:376
         0     0%   100%     32.74s 99.18%                | external/envoy/source/common/http/http1/codec_impl.cc:592
                                            25.78s 78.74% |   external/envoy/source/common/http/http1/codec_impl.cc:899
                                                5s 15.27% |   external/envoy/source/common/network/connection_impl.cc
                                             0.74s  2.26% |   external/envoy/source/common/http/http1/codec_impl.cc:898
                                             0.66s  2.02% |   external/envoy/source/common/network/connection_impl.cc:361
                                             0.53s  1.62% |   external/envoy/source/common/network/connection_impl.cc:342
                                             0.03s 0.092% |   external/envoy/source/common/network/connection_impl.cc:293
----------------------------------------------------------+-------------
                                            32.74s   100% |   external/envoy/source/common/network/connection_impl.cc:535
         0     0%   100%     32.74s 99.18%                | external/envoy/source/common/network/connection_impl.cc:211
                                            32.74s   100% |   external/envoy/source/common/network/connection_impl.cc:348
----------------------------------------------------------+-------------
                                            32.74s   100% |   external/envoy/source/common/network/connection_impl.cc:211
         0     0%   100%     32.74s 99.18%                | external/envoy/source/common/network/connection_impl.cc:348
                                            32.74s   100% |   external/envoy/source/common/http/codec_client.cc:82
----------------------------------------------------------+-------------
                                             0.03s   100% |   /usr/lib/llvm-9/include/c++/v1/functional:1860
         0     0%   100%      0.03s 0.091%                | external/envoy/source/common/network/connection_impl.cc:496
                                             0.03s   100% |   external/envoy/source/common/network/connection_impl.cc:581
----------------------------------------------------------+-------------
                                            32.74s   100% |   /usr/lib/llvm-9/include/c++/v1/functional:1860
         0     0%   100%     32.74s 99.18%                | external/envoy/source/common/network/connection_impl.cc:502
                                            32.74s   100% |   external/envoy/source/common/network/connection_impl.cc:535
----------------------------------------------------------+-------------
                                            32.74s   100% |   external/envoy/source/common/network/connection_impl.cc:502
         0     0%   100%     32.74s 99.18%                | external/envoy/source/common/network/connection_impl.cc:535
                                            32.74s   100% |   external/envoy/source/common/network/connection_impl.cc:211
----------------------------------------------------------+-------------
                                             0.03s   100% |   external/envoy/source/common/network/connection_impl.cc:496
         0     0%   100%      0.03s 0.091%                | external/envoy/source/common/network/connection_impl.cc:581
                                             0.03s   100% |   external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:235
----------------------------------------------------------+-------------
                                             0.01s   100% |   external/envoy/source/common/http/date_provider_impl.cc:21
         0     0%   100%      0.01s  0.03%                | external/envoy/source/common/thread_local/thread_local_impl.cc:100
                                             0.01s   100% |   external/envoy/source/common/thread_local/thread_local_impl.cc:214
----------------------------------------------------------+-------------
                                             0.01s   100% |   external/envoy/source/common/thread_local/thread_local_impl.cc:100
         0     0%   100%      0.01s  0.03%                | external/envoy/source/common/thread_local/thread_local_impl.cc:214
                                             0.01s   100% |   /usr/lib/llvm-9/include/c++/v1/functional:1795
----------------------------------------------------------+-------------
                                             0.02s   100% |   main bazel-out/k8-opt/bin/external/envoy/source/exe/_virtual_includes/envoy_main_common_lib/exe/main_common.h:97
         0     0%   100%      0.02s 0.061%                | external/envoy/source/exe/main_common.cc:105
                                             0.02s   100% |   external/envoy/source/server/server.cc:579
----------------------------------------------------------+-------------
                                             0.03s   100% |   external/envoy/source/common/http/conn_manager_impl.cc:565
         0     0%   100%      0.03s 0.091%                | external/envoy/source/extensions/access_loggers/file/file_access_log_impl.cc:20
                                             0.03s   100% |   external/envoy/source/common/access_log/access_log_formatter.cc:129
----------------------------------------------------------+-------------
                                             0.02s   100% |   external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:235
         0     0%   100%      0.02s 0.061%                | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:174
                                             0.02s   100% |   [envoy]
----------------------------------------------------------+-------------
                                             0.01s   100% |   external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:235
         0     0%   100%      0.01s  0.03%                | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:182
                                             0.01s   100% |   bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/connection_lib/common/network/connection_impl.h:111
----------------------------------------------------------+-------------
                                             0.03s   100% |   external/envoy/source/common/network/connection_impl.cc:581
         0     0%   100%      0.03s 0.091%                | external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:235
                                             0.02s 66.67% |   external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:174
                                             0.01s 33.33% |   external/envoy/source/extensions/transport_sockets/tls/ssl_socket.cc:182
----------------------------------------------------------+-------------
                                             0.02s   100% |   external/envoy/source/exe/main_common.cc:105
         0     0%   100%      0.02s 0.061%                | external/envoy/source/server/server.cc:579
                                             0.02s   100% |   [envoy]
----------------------------------------------------------+-------------
                                            32.82s   100% |   /usr/lib/llvm-9/include/c++/v1/functional:1860
         0     0%   100%     32.82s 99.42%                | external/envoy/source/server/worker_impl.cc:110
                                            32.82s   100% |   [envoy]
----------------------------------------------------------+-------------
                                             0.02s   100% |   [libc-2.27.so]
         0     0%   100%      0.02s 0.061%                | main bazel-out/k8-opt/bin/external/envoy/source/exe/_virtual_includes/envoy_main_common_lib/exe/main_common.h:97
                                             0.02s   100% |   external/envoy/source/exe/main_common.cc:105
----------------------------------------------------------+-------------
                                             0.01s   100% |   external/envoy/source/common/http/conn_manager_impl.cc:569
         0     0%   100%      0.01s  0.03%                | src/envoy/http/mixer/filter.cc:247
                                             0.01s   100% |   [envoy]
----------------------------------------------------------+-------------
                                             0.01s   100% |   [envoy]
         0     0%   100%      0.01s  0.03%                | src/envoy/http/mixer/report_data.h:89
                                             0.01s   100% |   ExtractHeaders src/envoy/utils/utils.cc:73
----------------------------------------------------------+-------------

lambdai · March 7, 2020, 2:58am

@jeff_r

Thank you for the patience. I am sorry I haven’t figured out the root cause.

The perf output reveal the fact that the thread is trapped in the loop of

https://github.com/istio/envoy/blame/release-1.4.6-patch/source/common/http/http1/codec_impl.cc#L898-L900

I strongly believe it’s a infinite loop

The root cause is not clear to me yet.

ElfoLiNk · March 20, 2020, 4:49pm

Hello @lambdai i’m experience a similar issue some envoy proxy are hitting container cpu limit (2 core)

Here the attached cpu profiler when this is happening:

ile: envoy

Type: cpu
Showing nodes accounting for 16.21s, 100% of 16.21s total
----------------------------------------------------------±------------
flat flat% sum% cum cum% calls calls% + context
----------------------------------------------------------±------------
3.25s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
3.25s 20.05% 20.05% 3.25s 20.05% | external/envoy/source/common/network/connection_impl.cc
----------------------------------------------------------±------------
2.54s 100% | external/envoy/source/common/network/connection_impl.cc:295
2.54s 15.67% 35.72% 2.54s 15.67% | external/envoy/source/common/network/io_socket_handle_impl.cc:36
----------------------------------------------------------±------------
2.51s 100% | external/envoy/source/common/network/connection_impl.cc:295
2.51s 15.48% 51.20% 2.51s 15.48% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/connection_lib/common/network/connection_impl.h:111
----------------------------------------------------------±------------
1.87s 100% | external/envoy/source/common/network/connection_impl.cc:295
1.87s 11.54% 62.74% 1.87s 11.54% | /usr/lib/llvm-9/include/c++/v1/memory:2558
----------------------------------------------------------±------------
1.78s 100% | external/envoy/source/common/http/http1/codec_impl.cc:899
1.78s 10.98% 73.72% 1.78s 10.98% | external/envoy/source/common/network/connection_impl.cc:172
----------------------------------------------------------±------------
0.93s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.93s 5.74% 79.46% 0.93s 5.74% | /usr/lib/llvm-9/include/c++/v1/memory:2562
----------------------------------------------------------±------------
0.81s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.81s 5.00% 84.45% 0.81s 5.00% | external/envoy/source/common/http/http1/codec_impl.cc:898
----------------------------------------------------------±------------
0.65s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.65s 4.01% 88.46% 0.65s 4.01% | external/envoy/source/common/network/connection_impl.cc:173
----------------------------------------------------------±------------
0.51s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.51s 3.15% 91.61% 0.51s 3.15% | external/envoy/source/common/network/connection_impl.cc:180
----------------------------------------------------------±------------
0.50s 100% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.50s 3.08% 94.69% 0.50s 3.08% | external/envoy/source/common/network/connection_impl.cc:293
----------------------------------------------------------±------------
0.33s 94.29% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.02s 5.71% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.35s 2.16% 96.85% 0.35s 2.16% | external/envoy/source/common/network/connection_impl.cc:342
----------------------------------------------------------±------------
9.29s 100% | external/envoy/source/common/http/http1/codec_impl.cc:899
0.20s 1.23% 98.09% 9.29s 57.31% | external/envoy/source/common/network/connection_impl.cc:295
2.54s 27.34% | external/envoy/source/common/network/io_socket_handle_impl.cc:36
2.51s 27.02% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/connection_lib/common/network/connection_impl.h:111
1.87s 20.13% | /usr/lib/llvm-9/include/c++/v1/memory:2558
0.93s 10.01% | /usr/lib/llvm-9/include/c++/v1/memory:2562
0.65s 7.00% | external/envoy/source/common/network/connection_impl.cc:173
0.51s 5.49% | external/envoy/source/common/network/connection_impl.cc:180
0.08s 0.86% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/listen_socket_lib/common/network/listen_socket_impl.h:27
----------------------------------------------------------±------------
12.09s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.19s 1.17% 99.26% 12.09s 74.58% | external/envoy/source/common/http/http1/codec_impl.cc:899
9.29s 76.84% | external/envoy/source/common/network/connection_impl.cc:295
1.78s 14.72% | external/envoy/source/common/network/connection_impl.cc:172
0.50s 4.14% | external/envoy/source/common/network/connection_impl.cc:293
0.33s 2.73% | external/envoy/source/common/network/connection_impl.cc:342
----------------------------------------------------------±------------
0.08s 100% | external/envoy/source/common/network/connection_impl.cc:295
0.08s 0.49% 99.75% 0.08s 0.49% | bazel-out/k8-opt/bin/external/envoy/source/common/network/_virtual_includes/listen_socket_lib/common/network/listen_socket_impl.h:27
----------------------------------------------------------±------------
0.03s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
0.03s 0.19% 99.94% 0.03s 0.19% | external/envoy/source/common/network/connection_impl.cc:361
----------------------------------------------------------±------------
0.01s 0.062% 100% 0.01s 0.062% | [libc-2.27.so]
----------------------------------------------------------±------------
16.20s 100% | [envoy]
16.20s 100% | [libpthread-2.27.so]
0 0% 100% 16.20s 99.94% | /usr/lib/llvm-9/include/c++/v1/functional:1860
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:502
16.20s 100% | external/envoy/source/server/worker_impl.cc:110
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:474
16.20s 100% | external/envoy/source/server/worker_impl.cc:110
0 0% 100% 16.20s 99.94% | [envoy]
16.20s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:376
----------------------------------------------------------±------------
0 0% 100% 16.20s 99.94% | [libpthread-2.27.so]
16.20s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/codec_client.cc:82
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/codec_client.cc:122
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:462
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:348
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/codec_client.cc:82
16.20s 100% | external/envoy/source/common/http/codec_client.cc:122
----------------------------------------------------------±------------
16.20s 100% | [envoy]
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/http1/codec_impl.cc:376
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:592
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/codec_client.cc:122
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/http1/codec_impl.cc:462
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:474
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:462
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/http1/codec_impl.cc:474
16.20s 100% | [envoy]
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/http/http1/codec_impl.cc:376
0 0% 100% 16.20s 99.94% | external/envoy/source/common/http/http1/codec_impl.cc:592
12.09s 74.63% | external/envoy/source/common/http/http1/codec_impl.cc:899
3.25s 20.06% | external/envoy/source/common/network/connection_impl.cc
0.81s 5.00% | external/envoy/source/common/http/http1/codec_impl.cc:898
0.03s 0.19% | external/envoy/source/common/network/connection_impl.cc:361
0.02s 0.12% | external/envoy/source/common/network/connection_impl.cc:342
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:535
0 0% 100% 16.20s 99.94% | external/envoy/source/common/network/connection_impl.cc:211
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:348
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:211
0 0% 100% 16.20s 99.94% | external/envoy/source/common/network/connection_impl.cc:348
16.20s 100% | external/envoy/source/common/http/codec_client.cc:82
----------------------------------------------------------±------------
16.20s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
0 0% 100% 16.20s 99.94% | external/envoy/source/common/network/connection_impl.cc:502
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:535
----------------------------------------------------------±------------
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:502
0 0% 100% 16.20s 99.94% | external/envoy/source/common/network/connection_impl.cc:535
16.20s 100% | external/envoy/source/common/network/connection_impl.cc:211
----------------------------------------------------------±------------
16.20s 100% | /usr/lib/llvm-9/include/c++/v1/functional:1860
0 0% 100% 16.20s 99.94% | external/envoy/source/server/worker_impl.cc:110
16.20s 100% | [envoy]
----------------------------------------------------------±------------

jeff_r · March 25, 2020, 5:25pm

Upgrading to 1.5 has addressed my issue.

kconner · March 25, 2020, 6:56pm

There was an issue in envoyproxy where it would busy wait on closed connections, see https://github.com/envoyproxy/envoy/pull/9509 for details.

Topic		Replies	Views
Ingress Gateway pod consume more CPU during load Performance and Scalability	1	615	March 12, 2020
Istio Ingress Gatewaying Queuing Requests for 10+ seconds?	2	2308	July 18, 2020
Ingress gateway pods takes ages to forward traffic after upgrading to v1.1.x Networking	1	661	May 16, 2019
Istio 1.6 strange behavior after upgrade (traffic disruption) Networking	0	457	June 11, 2020
Limit istio ingress gateway increasing CPU usage Config	1	1365	May 20, 2021

Ingress-gateway cpu usage stuck at 100% vcpu allocated 1.4.6

Related Topics