Access to k8s API server is lost after ~90 minutes


I’m having a really strange problem on my K8s cluster running istio 1.1.0. My workload is able to access the Kubernetes server for around 90 minutes or so and then it stops being able to reach the API endpoint.

I dumped all proxy-config options (listeners, endpoints, routes, clusters) at the beginning and once the problem started. There are no material diffs between the before and after state - the only observable difference is that access to the API server is no longer possible.

I verified basic network connectivity by curl-ing the API server endpoint from the istio-proxy container. This works. When I try the command under sudo (so as to change the UID/ GID such that proxying will be in effect), it fails.

All other services are accessible, just not the API server.

This one is a real head-scratcher and I’m looking for some pointers on how to troubleshoot something like this. Thanks in advance.