How do i take istio mTLS tcpdump?

1

Below i have 2 deployment app1(nginx) and app2(httpd) as below

image
image904×482 37.4 KB

[root@lp-kmaster-1 istio-1.9.0]# kubectl get deploy -n istio -o wide
NAME   READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES   SELECTOR
app1   1/1     1            1           5d17h   app1         nginx    run=app1
app2   1/1     1            1           5d17h   app2         httpd    run=app2


[root@lp-kmaster-1 istio-1.9.0]# kubectl get svc -n istio -o wide
NAME   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE    SELECTOR
app1   NodePort   10.99.55.73     <none>        80:31601/TCP   7d5h   run=app1
app2   NodePort   10.97.232.183   <none>        80:30125/TCP   7d5h   run=app2


[root@lp-kmaster-1 istio-1.9.0]# kubectl get pod -n istio -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP          NODE              NOMINATED NODE   READINESS GATES
app1-b6c8f6c99-x5g7f    2/2     Running   0          20m   10.36.0.7   lp-knode-1.home   <none>           <none>
app2-6b9b9d7b87-kb9js   2/2     Running   0          29m   10.42.0.2   lp-knode-2.home   <none>           <none>

I have also set peerauthentication as STRICT for namespace istio

[root@lp-kmaster-1 istio-1.9.0]# kubectl get peerauthentication -n istio
NAME      MODE     AGE
default   STRICT   3h17m

1. I have also verified mTLS is working with curl from istio sidecar proxy
2. I can see mTLS lock on kiali dashboard as well

My security team still need some proof if it’s actaully TLS :slight_smile:

I have tried below steps but it’s not showing any TLS trafiic when i opened the traffic file in wireshark.

  1. Ran below command on lp-knode-2

tcpdump -vvvv -A -i weave '((dst port 80) and (net 10.42.0.2))' -w app-2.cap

  1. From app1 pod curl http://app2 5 times
  2. From app1-istio-proxy curl -kv https://app2:80 2 times

and i opened the app-2.cap file in Wireshark and i don’t see the TLS in protocol

image
image1271×462 20.1 KB

I received 25 packets and i have added 2 packet below

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       10.36.0.7             10.42.0.2             TCP      1242   42028 → 80 [PSH, ACK] Seq=1 Ack=1 Win=411 Len=1176 TSval=200322778 TSecr=199934808

Frame 1: 1242 bytes on wire (9936 bits), 1242 bytes captured (9936 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: Feb 19, 2021 19:25:25.515949000 India Standard Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1613742925.515949000 seconds
    [Time delta from previous captured frame: 0.000000000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.000000000 seconds]
    Frame Number: 1
    Frame Length: 1242 bytes (9936 bits)
    Capture Length: 1242 bytes (9936 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:tcp]
    [Coloring Rule Name: HTTP]
    [Coloring Rule String: http || tcp.port == 80 || http2]
Ethernet II, Src: a2:8d:af:12:cb:a1 (a2:8d:af:12:cb:a1), Dst: f2:3a:42:5e:bf:0b (f2:3a:42:5e:bf:0b)
Internet Protocol Version 4, Src: 10.36.0.7, Dst: 10.42.0.2
Transmission Control Protocol, Src Port: 42028, Dst Port: 80, Seq: 1, Ack: 1, Len: 1176
    Source Port: 42028
    Destination Port: 80
    [Stream index: 0]
    [TCP Segment Len: 1176]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 1024589860
    [Next Sequence Number: 1177    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 2623239331
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 411
    [Calculated window size: 411]
    [Window size scaling factor: -1 (unknown)]
    Checksum: 0xc422 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
    [Timestamps]
    TCP payload (1176 bytes)

No.     Time           Source                Destination           Protocol Length Info
      2 0.002503       10.36.0.7             10.42.0.2             TCP      66     42028 → 80 [ACK] Seq=1177 Ack=1127 Win=429 Len=0 TSval=200322780 TSecr=200317785

Frame 2: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: Feb 19, 2021 19:25:25.518452000 India Standard Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1613742925.518452000 seconds
    [Time delta from previous captured frame: 0.002503000 seconds]
    [Time delta from previous displayed frame: 0.002503000 seconds]
    [Time since reference or first frame: 0.002503000 seconds]
    Frame Number: 2
    Frame Length: 66 bytes (528 bits)
    Capture Length: 66 bytes (528 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:tcp]
    [Coloring Rule Name: HTTP]
    [Coloring Rule String: http || tcp.port == 80 || http2]
Ethernet II, Src: a2:8d:af:12:cb:a1 (a2:8d:af:12:cb:a1), Dst: f2:3a:42:5e:bf:0b (f2:3a:42:5e:bf:0b)
Internet Protocol Version 4, Src: 10.36.0.7, Dst: 10.42.0.2
Transmission Control Protocol, Src Port: 42028, Dst Port: 80, Seq: 1177, Ack: 1127, Len: 0
    Source Port: 42028
    Destination Port: 80
    [Stream index: 0]
    [TCP Segment Len: 0]
    Sequence Number: 1177    (relative sequence number)
    Sequence Number (raw): 1024591036
    [Next Sequence Number: 1177    (relative sequence number)]
    Acknowledgment Number: 1127    (relative ack number)
    Acknowledgment number (raw): 2623240457
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x010 (ACK)
    Window: 429
    [Calculated window size: 429]
    [Window size scaling factor: -1 (unknown)]
    Checksum: 0x02f4 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]

am i missing something?

is there any way to verify the mTLS traffic with tcpdump command?

2

Tried using tcpdump(as a root) command provided by istio-proxy but it says permission issue.

tcpdump: eth0: You don't have permission to capture on that device

raised bug request : can't use tcpdump in istio-proxy - eth0: You don't have permission to capture on that device · Issue #30982 · istio/istio · GitHub

3

After running istioctl install --set values.global.proxy.privileged=true i was able to run sudo tcpdump command.

Faced 2 issues:

  1. The istio-proxy is mounted as read-only and it was not letting me genrate app1.pcap file sudo tcpdump -vv -ni eth0 -w app2.cap

workaround : cd /dev then run tcpdump -vv -ni eth0 -w app2.cap

  1. couldn’t copy app2.cap from istio-proxy container to host with docker cp 370d55b9ca09:/dev/app2.cap /tmp as it was giving Error: No such container:path: 370d55b9ca09:/dev/app2.cap also scp, rsync was not available for obvious reasons :slight_smile:

workaround : used curl command to send app2.cap file

Opening file app2.cap in still shows me TCP connection :frowning: . I might be missing something.

However i have looked at tcpdump example mention here(Istio / Mutual TLS Migration) which shows the encrypted data when i run curl app2 from pod app1. I think this will be good enough proof to show security team :slight_smile:

4

I was looking at wrong value from start. I found out it’s only displaying TCP in protocol but in background the data is encrypted via mTLS.

To verify it i have first set MODE=PERMISSIVE and sent some curl request too app2.istio service within istio meshed and non-meshed namespace.

From non-istio-meshed namespace:

curl app2.istio/asd

image
image745×672 40.4 KB

From istio-meshed namespace:

curl app2.istio/asd

image
image974×713 51.9 KB

5

Hi @koolwithk , I am able to see the same results as your last comment, for mtls traffic the data are encrypted in TCP payload area. I am wondering if you know how can we see the “normal” SSL handshake(client hello, client certificate, server certificate exchange) processes for istio MTLS communication?
This encrypted payload area doesn’t give much info

6

Because the traffic is not on port 443 wireshark doesn’t know to automatically dissect the traffic as TLS. You have to force it to do so. Right click on the stream you want to decode, click “Decode As…”, in the box that pops up under the rightmost column called “Current” click the word you see there (probably “None”) and chose TLS from the drop down then click OK.