How does kubectl port-forward bypass envoy mTLS enforcement?

tammersaleh · February 6, 2019, 8:36pm

I have an architectural question for which I haven’t been able to find an answer in the Googles.

If I’ve enabled mandatory mTLS, and I deploy an app to the mesh, then I cannot connect to it via a Kubernetes LoadBalancer service. This is expected, as I’m not doing mTLS in my connection.

However, if I then do a kubectl port-forward, then I can connect to the app through the locally forwarded port. I’m not sure how that works since the port-forward is sending my connection to the exposed port on the Pod in the same way that the service was. Does Istio/Envoy explicitly not implement policy against ports forwarded this way? If so, is there documentation explaining that (or how it works)?

spikecurtis · February 6, 2019, 10:57pm

I’m not 100% on the details of how port-forward works, but it definitely is not “the same way that the service was.”

Port-forward involves the Kubernetes API server opening a SPDY connection to Kubelet, and Kubelet opening the stream to your pod. My suspicion is that the kubelet to pod stream is done in such a way as it bypasses the iptables rules that normally redirect IP packets thru the proxy. It also depends on the container runtime (e.g. Docker, CRI-O, rkt).

I’ll ask around the office and see if I can get a more detailed answer from Kube experts here.

tammersaleh · February 6, 2019, 11:21pm

Much appreciated, Spike. I expect that understanding how this works will help me understand Istio/K8s networking in general.

kconner · February 7, 2019, 12:48am

Port forwarding works by essentially running a ‘docker exec’ into the container and running socat against the port on localhost. Since this is over localhost it is not captured by envoy.

Kev

kconner · February 7, 2019, 12:56am

The code you are looking for is below, it is entering the containers network namespace (nsenter) and within that running socat against TCP4:localhost:%d

Kev

github.com

kubernetes/kubernetes/blob/0daa1219f64d0773c77b9f573c0a920f34f2536c/pkg/kubelet/dockershim/docker_streaming.go#L178-L233


func portForward(client libdocker.Interface, podSandboxID string, port int32, stream io.ReadWriteCloser) error {
	container, err := client.InspectContainer(podSandboxID)
	if err != nil {
		return err
	}


	if !container.State.Running {
		return fmt.Errorf("container not running (%s)", container.ID)
	}


	containerPid := container.State.Pid
	socatPath, lookupErr := exec.LookPath("socat")
	if lookupErr != nil {
		return fmt.Errorf("unable to do port forwarding: socat not found.")
	}


	args := []string{"-t", fmt.Sprintf("%d", containerPid), "-n", socatPath, "-", fmt.Sprintf("TCP4:localhost:%d", port)}


	nsenterPath, lookupErr := exec.LookPath("nsenter")
	if lookupErr != nil {

This file has been truncated. show original

tammersaleh · February 7, 2019, 1:02am

Perfect! Thanks @kconner!

Would you accept a PR explaining this in the Istio documentation? If so, which page would you suggest editing?

spikecurtis · February 8, 2019, 9:32pm

I don’t know of a good existing place to include this information. What kind of thing are you wanting to say? Maybe we can find a good home for it.

cc @geeknoid

geeknoid · February 9, 2019, 12:17am

For this kind of content, I’d shoot for either a FAQ page (/help/faq), or something in the operations guide (/help/ops)

Farid_BADAROU · February 22, 2019, 8:59am

I have a problem is quite different.
When I execute the command
kubectl port-forward -n monitoring prometheus-prometheus-operator-prometheus-0 9090 on my cluster Kubernetes (AKS) it gives me this result
Forwarding from 127.0.0.1.19090 -> 9090
Forwarding from [:: 1]: 9090 -> 9090
and when I put the URL http://127.0.0.1:9090/targets on my browser, I have no result.

$ kubectl get pods -n monitoring
NAME READY STATUS RESTARTS AGE
alertmanager-prometheus-operator-alertmanager-0 2/2 Running 0 11h
prometheus-operator-grafana-8549568bc7-zntgz 2/2 Running 0 11h
prometheus-operator-kube-state-metrics-5fc876d8f6-v5zh7 1/1 Running 0 11h
prometheus-operator-operator-75d88ccc6-t4lt5 1/1 Running 0 11h
prometheus-operator-prometheus-node-export-zqk7t 1/1 Running 0 11h
prometheus-prometheus-operator-prometheus-0 3/3 Running 1 11h

What should be done ? or how to forwarding prometheus from my public cloud to my local machine.
Thank you

Topic		Replies	Views
Does kubectl port-forward svc bypass Istio's proxy? Networking	1	1021	November 14, 2022
Bypassing envoy when calling localhost inside a pod Networking	1	2638	September 17, 2019
Envoy's traffic hijacking(port forward) rule is stored at the pod level or host level? Networking	2	2115	April 9, 2019
Getting false tcp connection from withing container Networking	6	2489	July 20, 2022
Istio enabled pods and egress connections Networking	4	819	July 1, 2019

How does kubectl port-forward bypass envoy mTLS enforcement?

Related topics