Effect of deleting pilot on istio-proxy containers

How does deleting pilot pods effect an already running istio-proxy container?
I was under the impression that if pilot delivers the configuration once, istio-proxy should be able to run with it even when pilot is not available.
I am running fortio in an istio enabled namespace hitting my service (ping) that returns 200OK in response.

Here’s the fortio spec:

 spec:
  containers:
    - name: fortio
      image: fortio/fortio
      command: ["fortio"]
      args: ["load", "-t", "3000s", "-qps", "100", "http://ping:8080/ping"]

The test runs well until I delete the pilot pods. I observed that after about a minute or 2 of deleting the pilot pods, fortio starts giving these errors:

06:40:00 E http_client.go:677> Read error &{{0xc0003d2380}} 46.19.209.189:8080 0 : read tcp 100.122.111.88:43086->46.19.209.189:8080: i/o timeout
06:40:45 E http_client.go:558> Unable to connect to 46.19.209.189:8080 : dial tcp 46.19.209.189:8080: connect: connection refused

The corresponding logs on istio-proxy on ping service say Envoy NOT ready:

* failed checking application ports. listeners="0.0.0.0:15090"
* envoy missing listener for inbound application port: 8080
2019-06-26T06:39:00.246040Z	info	Envoy proxy is NOT ready: 2 errors occurred:

What am I missing?

1 Like

This is very weird. I’ve got similar issues when creating load on a container (eg with lots of concurrent requests – via ab). The weirdest thing is that the Envoy proxy of the pod i’m load testing spawns an endpoint for the same IP: 46.19.209.189

I’m maximally confused at this point :<

Where did that IP come from cause that’s totally outside my cluster, on the internet. I’m thinking, can it be some memory overflowing an IP somehow?

PS: i didn’t touch the pilot pods. only ran this in a grafana pod, to create some load through Envoy. I acually found this issue seeing that grafana dashboards with lots of panels are failing… i’m beyond astonished

ab -n 9999 -c 100 http://localhost:8080/api/datasources/proxy/2/api/v1/query_range

This is the configdump for the endpoint that appears only when Envoy is under load.

Did you ever get to the bottom of this? I’m also seeing the PassthroughCluster entry being created on the fly with that 46.19.x.x address and I’m very confused.

Figured it out: