How does kubectl port-forward bypass envoy mTLS enforcement?


#1

I have an architectural question for which I haven’t been able to find an answer in the Googles.

If I’ve enabled mandatory mTLS, and I deploy an app to the mesh, then I cannot connect to it via a Kubernetes LoadBalancer service. This is expected, as I’m not doing mTLS in my connection.

However, if I then do a kubectl port-forward, then I can connect to the app through the locally forwarded port. I’m not sure how that works since the port-forward is sending my connection to the exposed port on the Pod in the same way that the service was. Does Istio/Envoy explicitly not implement policy against ports forwarded this way? If so, is there documentation explaining that (or how it works)?


#2

I’m not 100% on the details of how port-forward works, but it definitely is not “the same way that the service was.”

Port-forward involves the Kubernetes API server opening a SPDY connection to Kubelet, and Kubelet opening the stream to your pod. My suspicion is that the kubelet to pod stream is done in such a way as it bypasses the iptables rules that normally redirect IP packets thru the proxy. It also depends on the container runtime (e.g. Docker, CRI-O, rkt).

I’ll ask around the office and see if I can get a more detailed answer from Kube experts here.


#3

Much appreciated, Spike. I expect that understanding how this works will help me understand Istio/K8s networking in general.


#4

Port forwarding works by essentially running a ‘docker exec’ into the container and running socat against the port on localhost. Since this is over localhost it is not captured by envoy.

Kev


#5

The code you are looking for is below, it is entering the containers network namespace (nsenter) and within that running socat against TCP4:localhost:%d

Kev


#6

Perfect! Thanks @kconner!

Would you accept a PR explaining this in the Istio documentation? If so, which page would you suggest editing?


#7

I don’t know of a good existing place to include this information. What kind of thing are you wanting to say? Maybe we can find a good home for it.

cc @geeknoid


#8

For this kind of content, I’d shoot for either a FAQ page (/help/faq), or something in the operations guide (/help/ops)