Trouble talking to kubernetes.default.svc usig Istio 1.9.4

Networking
1

Hello,

We have an existing K8s offering on AKS that involves several deployments within the K8s cluster that require an ingress engine. In our application, we need to add routes underneath our root based on user configurations (eg our app is at /my/app and based on user config we needed to add /my/app/userendpoint to point at a new deployment in a different namespace). We had an existing solution with Contour but that required creating our own proxy service to route requests. To eliminate that proxy service, in Contour we would have to modify the top level HttpProxy. We wanted to avoid modifying top level configs based on user configurations, so we did a POC and verified Istio would allow us to insert new routes by just adding a VirtualService and not modifying anything else.

Our app consists of several Play apps built with Scala that talk to the Kubernetes REST API using the hostname kubernetes.default.svc (using https) to gather information about other services launched based on user configurations. The code communicating with kubernetes.default.svc was running without a problem with Contour as the Ingress.

Since switching to Istio and the accompanying service mesh in our dev environments, our apps can no longer consistently talk to the Kubernetes REST API. We get a lot of connection refused and connection reset errors in our logs when querying something like https://kubernetes.default.svc/api/v1/namespaces. We found this old issue in Github Internal Kubernetes API Calls Blocked by Istio · Issue #8696 · istio/istio · GitHub and tried specifying the .spec.values.global.proxy.excludeIPRanges entry as suggested in our IstioOperator definition. We also tried adding the ServiceEntry like the one listed in the issue but the problem persists. Another site recommended connecting to the external hostname of the cluster instead of kubernetes.default.svc, but that yields similar results.

The pods in question that are having issues all have the istio sidecar, so that is likely playing a role in what’s going on.

What suggestions does the community have for troubleshooting this? I’m not sure what else to try.

-Javier

2

BTW, we’re running on AKS 1.19.7 with Istio 1.9.4

3

BTW, got around this issue with the following DestRule:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: k8s-default-destrule
  namespace: my-service-namespace
spec:
  host: "kubernetes.default.svc" #Disabling it for Kube API Server communication
  trafficPolicy:
    tls:
      mode: DISABLE

Thanks to this post: https://stackoverflow.com/a/63603897 for the idea.

-Javier

4

the problem is back today. Doing some more research…