Waiting for headers takes ~25 seconds

Networking
1

Hello,

I’m using istio 1.2.0(but I had the same problem on 1.1.7) and when my pod have istio-proxy sidecar container, “waiting for headers” step during ‘apt update’ takes 20-30 seconds. This problem doesn’t occur when pod doesn’t have istio-proxy sidecar.

Maybe someone saw this behavior before? I’ll be very grateful for any help.

1 Like
2

Hello,

I still cannot resolve this problem. I’ve gathered some further important informations:

  • i’ve simplified my enviroment as much as i could. I’m deploying single master cluster(without any worker nodes on ubuntu 18.04 with kubeadm. There are all commands used to deploy k8s+istio: https://pastebin.com/raw/pYvxCiAQ)
  • problem occurs on both tested kubernetes versions: 1.14.3 and 1.15.0
  • problem occurs on flannel and calico CNIs
  • it seems that istio dropping TCP connections not only on apt update, I’ve similar problem when trying to download big file via http.
# wget http://www.ovh.net/files/10Gb.dat
--2019-06-27 14:43:57--  http://www.ovh.net/files/10Gb.dat
Resolving www.ovh.net (www.ovh.net)... 213.186.33.6
Connecting to www.ovh.net (www.ovh.net)|213.186.33.6|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1250000000 (1.2G) [application/octet-stream]
Saving to: '10Gb.dat.1'

10Gb.dat.1                     26%[===========>                                    ] 318.39M  17.8MB/s    in 16s

2019-06-27 14:44:13 (20.0 MB/s) - Connection closed at byte 333851919. Retrying.

--2019-06-27 14:44:14--  (try: 2)  http://www.ovh.net/files/10Gb.dat
Connecting to www.ovh.net (www.ovh.net)|213.186.33.6|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 1250000000 (1.2G), 916148081 (874M) remaining [application/octet-stream]
Saving to: '10Gb.dat.1'

10Gb.dat.1                     55%[++++++++++++=============>                      ] 663.15M  18.0MB/s    in 16s

2019-06-27 14:44:30 (21.7 MB/s) - Connection closed at byte 695365566. Retrying.

--2019-06-27 14:44:32--  (try: 3)  http://www.ovh.net/files/10Gb.dat
Connecting to www.ovh.net (www.ovh.net)|213.186.33.6|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 1250000000 (1.2G), 554634434 (529M) remaining [application/octet-stream]
Saving to: '10Gb.dat.1'

10Gb.dat.1                     81%[++++++++++++++++++++++++++============>         ] 974.24M  14.8MB/s    in 16s

2019-06-27 14:44:48 (19.5 MB/s) - Connection closed at byte 1021566165. Retrying.

--2019-06-27 14:44:51--  (try: 4)  http://www.ovh.net/files/10Gb.dat
Connecting to www.ovh.net (www.ovh.net)|213.186.33.6|:80... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 1250000000 (1.2G), 228433835 (218M) remaining [application/octet-stream]
Saving to: '10Gb.dat.1'

10Gb.dat.1                    100%[+++++++++++++++++++++++++++++++++++++++========>]   1.16G  23.3MB/s    in 9.5s

2019-06-27 14:45:01 (22.9 MB/s) - '10Gb.dat.1' saved [1250000000/1250000000]
3

I just wanted to add to the discussion: i’m seeing similar issues. I’ve spent quite some time trying to debug it. My context: EKS with Kubernetes version 1.13, Istio 1.2.2. I’m attaching what i consider to be an interesting Wireshark PCAP file — the apt UI hangs for exactly 30s (this is Ubuntu — on Debian it is 120 seconds) at timestamp 2019-07-09 10:31:37, but the PCAP seems to suggest that the package repository server replied with HTTP 200 very promptly. At about 10:32:06 the UI happily continues. Is the istio-proxy sidecar perhaps holding on to the response for unduly long? Any suggestions on how to debug?

The PCAP: https://random-pcap.s3-ap-southeast-2.amazonaws.com/apt.pcap

Thanks!

4

Observing the same - any progress ?

5

No, no progress unfortunately!