How does kubectl port-forward bypass envoy mTLS enforcement?

Networking
1

I have an architectural question for which I haven’t been able to find an answer in the Googles.

If I’ve enabled mandatory mTLS, and I deploy an app to the mesh, then I cannot connect to it via a Kubernetes LoadBalancer service. This is expected, as I’m not doing mTLS in my connection.

However, if I then do a kubectl port-forward, then I can connect to the app through the locally forwarded port. I’m not sure how that works since the port-forward is sending my connection to the exposed port on the Pod in the same way that the service was. Does Istio/Envoy explicitly not implement policy against ports forwarded this way? If so, is there documentation explaining that (or how it works)?

1 Like
2

I’m not 100% on the details of how port-forward works, but it definitely is not “the same way that the service was.”

Port-forward involves the Kubernetes API server opening a SPDY connection to Kubelet, and Kubelet opening the stream to your pod. My suspicion is that the kubelet to pod stream is done in such a way as it bypasses the iptables rules that normally redirect IP packets thru the proxy. It also depends on the container runtime (e.g. Docker, CRI-O, rkt).

I’ll ask around the office and see if I can get a more detailed answer from Kube experts here.

2 Likes
3

Much appreciated, Spike. I expect that understanding how this works will help me understand Istio/K8s networking in general.

4

Port forwarding works by essentially running a ‘docker exec’ into the container and running socat against the port on localhost. Since this is over localhost it is not captured by envoy.

Kev

5

The code you are looking for is below, it is entering the containers network namespace (nsenter) and within that running socat against TCP4:localhost:%d

Kev

1 Like
6

Perfect! Thanks @kconner!

Would you accept a PR explaining this in the Istio documentation? If so, which page would you suggest editing?

7

I don’t know of a good existing place to include this information. What kind of thing are you wanting to say? Maybe we can find a good home for it.

cc @geeknoid

8

For this kind of content, I’d shoot for either a FAQ page (/help/faq), or something in the operations guide (/help/ops)

9

I have a problem is quite different.
When I execute the command
kubectl port-forward -n monitoring prometheus-prometheus-operator-prometheus-0 9090 on my cluster Kubernetes (AKS) it gives me this result
Forwarding from 127.0.0.1.19090 -> 9090
Forwarding from [:: 1]: 9090 -> 9090
and when I put the URL http://127.0.0.1:9090/targets on my browser, I have no result.

$ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-prometheus-operator-alertmanager-0 2/2 Running 0 11h
prometheus-operator-grafana-8549568bc7-zntgz 2/2 Running 0 11h
prometheus-operator-kube-state-metrics-5fc876d8f6-v5zh7 1/1 Running 0 11h
prometheus-operator-operator-75d88ccc6-t4lt5 1/1 Running 0 11h
prometheus-operator-prometheus-node-export-zqk7t 1/1 Running 0 11h
prometheus-prometheus-operator-prometheus-0 3/3 Running 1 11h

What should be done ? or how to forwarding prometheus from my public cloud to my local machine.
Thank you