Effect of deleting pilot on istio-proxy containers

1

How does deleting pilot pods effect an already running istio-proxy container?
I was under the impression that if pilot delivers the configuration once, istio-proxy should be able to run with it even when pilot is not available.
I am running fortio in an istio enabled namespace hitting my service (ping) that returns 200OK in response.

Here’s the fortio spec:

 spec:
  containers:
    - name: fortio
      image: fortio/fortio
      command: ["fortio"]
      args: ["load", "-t", "3000s", "-qps", "100", "http://ping:8080/ping"]

The test runs well until I delete the pilot pods. I observed that after about a minute or 2 of deleting the pilot pods, fortio starts giving these errors:

06:40:00 E http_client.go:677> Read error &{{0xc0003d2380}} 46.19.209.189:8080 0 : read tcp 100.122.111.88:43086->46.19.209.189:8080: i/o timeout
06:40:45 E http_client.go:558> Unable to connect to 46.19.209.189:8080 : dial tcp 46.19.209.189:8080: connect: connection refused

The corresponding logs on istio-proxy on ping service say Envoy NOT ready:

* failed checking application ports. listeners="0.0.0.0:15090"
* envoy missing listener for inbound application port: 8080
2019-06-26T06:39:00.246040Z	info	Envoy proxy is NOT ready: 2 errors occurred:

What am I missing?

1 Like
2

This is very weird. I’ve got similar issues when creating load on a container (eg with lots of concurrent requests – via ab). The weirdest thing is that the Envoy proxy of the pod i’m load testing spawns an endpoint for the same IP: 46.19.209.189

I’m maximally confused at this point :<

Screenshot_2019-12-19_01_46_52_1919x707_scrot
Screenshot_2019-12-19_01_46_52_1919x707_scrot1919×707 259 KB

Where did that IP come from cause that’s totally outside my cluster, on the internet. I’m thinking, can it be some memory overflowing an IP somehow?

PS: i didn’t touch the pilot pods. only ran this in a grafana pod, to create some load through Envoy. I acually found this issue seeing that grafana dashboards with lots of panels are failing… i’m beyond astonished

ab -n 9999 -c 100 http://localhost:8080/api/datasources/proxy/2/api/v1/query_range
3

This is the configdump for the endpoint that appears only when Envoy is under load.

Screenshot_2019-12-19_01_54_05_1058x872_scrot
Screenshot_2019-12-19_01_54_05_1058x872_scrot1058×872 61.9 KB

4

Did you ever get to the bottom of this? I’m also seeing the PassthroughCluster entry being created on the fly with that 46.19.x.x address and I’m very confused.

5

Figured it out: