How do i take istio mTLS tcpdump?

Below i have 2 deployment app1(nginx) and app2(httpd) as below

[root@lp-kmaster-1 istio-1.9.0]# kubectl get deploy -n istio -o wide
NAME   READY   UP-TO-DATE   AVAILABLE   AGE     CONTAINERS   IMAGES   SELECTOR
app1   1/1     1            1           5d17h   app1         nginx    run=app1
app2   1/1     1            1           5d17h   app2         httpd    run=app2


[root@lp-kmaster-1 istio-1.9.0]# kubectl get svc -n istio -o wide
NAME   TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)        AGE    SELECTOR
app1   NodePort   10.99.55.73     <none>        80:31601/TCP   7d5h   run=app1
app2   NodePort   10.97.232.183   <none>        80:30125/TCP   7d5h   run=app2


[root@lp-kmaster-1 istio-1.9.0]# kubectl get pod -n istio -o wide
NAME                    READY   STATUS    RESTARTS   AGE   IP          NODE              NOMINATED NODE   READINESS GATES
app1-b6c8f6c99-x5g7f    2/2     Running   0          20m   10.36.0.7   lp-knode-1.home   <none>           <none>
app2-6b9b9d7b87-kb9js   2/2     Running   0          29m   10.42.0.2   lp-knode-2.home   <none>           <none>

I have also set peerauthentication as STRICT for namespace istio

[root@lp-kmaster-1 istio-1.9.0]# kubectl get peerauthentication -n istio
NAME      MODE     AGE
default   STRICT   3h17m

1. I have also verified mTLS is working with curl from istio sidecar proxy
2. I can see mTLS lock on kiali dashboard as well

My security team still need some proof if it’s actaully TLS :slight_smile:

I have tried below steps but it’s not showing any TLS trafiic when i opened the traffic file in wireshark.

  1. Ran below command on lp-knode-2

tcpdump -vvvv -A -i weave '((dst port 80) and (net 10.42.0.2))' -w app-2.cap

  1. From app1 pod curl http://app2 5 times
  2. From app1-istio-proxy curl -kv https://app2:80 2 times

and i opened the app-2.cap file in Wireshark and i don’t see the TLS in protocol

I received 25 packets and i have added 2 packet below

No.     Time           Source                Destination           Protocol Length Info
      1 0.000000       10.36.0.7             10.42.0.2             TCP      1242   42028 → 80 [PSH, ACK] Seq=1 Ack=1 Win=411 Len=1176 TSval=200322778 TSecr=199934808

Frame 1: 1242 bytes on wire (9936 bits), 1242 bytes captured (9936 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: Feb 19, 2021 19:25:25.515949000 India Standard Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1613742925.515949000 seconds
    [Time delta from previous captured frame: 0.000000000 seconds]
    [Time delta from previous displayed frame: 0.000000000 seconds]
    [Time since reference or first frame: 0.000000000 seconds]
    Frame Number: 1
    Frame Length: 1242 bytes (9936 bits)
    Capture Length: 1242 bytes (9936 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:tcp]
    [Coloring Rule Name: HTTP]
    [Coloring Rule String: http || tcp.port == 80 || http2]
Ethernet II, Src: a2:8d:af:12:cb:a1 (a2:8d:af:12:cb:a1), Dst: f2:3a:42:5e:bf:0b (f2:3a:42:5e:bf:0b)
Internet Protocol Version 4, Src: 10.36.0.7, Dst: 10.42.0.2
Transmission Control Protocol, Src Port: 42028, Dst Port: 80, Seq: 1, Ack: 1, Len: 1176
    Source Port: 42028
    Destination Port: 80
    [Stream index: 0]
    [TCP Segment Len: 1176]
    Sequence Number: 1    (relative sequence number)
    Sequence Number (raw): 1024589860
    [Next Sequence Number: 1177    (relative sequence number)]
    Acknowledgment Number: 1    (relative ack number)
    Acknowledgment number (raw): 2623239331
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x018 (PSH, ACK)
    Window: 411
    [Calculated window size: 411]
    [Window size scaling factor: -1 (unknown)]
    Checksum: 0xc422 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [SEQ/ACK analysis]
    [Timestamps]
    TCP payload (1176 bytes)

No.     Time           Source                Destination           Protocol Length Info
      2 0.002503       10.36.0.7             10.42.0.2             TCP      66     42028 → 80 [ACK] Seq=1177 Ack=1127 Win=429 Len=0 TSval=200322780 TSecr=200317785

Frame 2: 66 bytes on wire (528 bits), 66 bytes captured (528 bits)
    Encapsulation type: Ethernet (1)
    Arrival Time: Feb 19, 2021 19:25:25.518452000 India Standard Time
    [Time shift for this packet: 0.000000000 seconds]
    Epoch Time: 1613742925.518452000 seconds
    [Time delta from previous captured frame: 0.002503000 seconds]
    [Time delta from previous displayed frame: 0.002503000 seconds]
    [Time since reference or first frame: 0.002503000 seconds]
    Frame Number: 2
    Frame Length: 66 bytes (528 bits)
    Capture Length: 66 bytes (528 bits)
    [Frame is marked: False]
    [Frame is ignored: False]
    [Protocols in frame: eth:ethertype:ip:tcp]
    [Coloring Rule Name: HTTP]
    [Coloring Rule String: http || tcp.port == 80 || http2]
Ethernet II, Src: a2:8d:af:12:cb:a1 (a2:8d:af:12:cb:a1), Dst: f2:3a:42:5e:bf:0b (f2:3a:42:5e:bf:0b)
Internet Protocol Version 4, Src: 10.36.0.7, Dst: 10.42.0.2
Transmission Control Protocol, Src Port: 42028, Dst Port: 80, Seq: 1177, Ack: 1127, Len: 0
    Source Port: 42028
    Destination Port: 80
    [Stream index: 0]
    [TCP Segment Len: 0]
    Sequence Number: 1177    (relative sequence number)
    Sequence Number (raw): 1024591036
    [Next Sequence Number: 1177    (relative sequence number)]
    Acknowledgment Number: 1127    (relative ack number)
    Acknowledgment number (raw): 2623240457
    1000 .... = Header Length: 32 bytes (8)
    Flags: 0x010 (ACK)
    Window: 429
    [Calculated window size: 429]
    [Window size scaling factor: -1 (unknown)]
    Checksum: 0x02f4 [unverified]
    [Checksum Status: Unverified]
    Urgent Pointer: 0
    Options: (12 bytes), No-Operation (NOP), No-Operation (NOP), Timestamps
    [Timestamps]

am i missing something?

is there any way to verify the mTLS traffic with tcpdump command?

Tried using tcpdump(as a root) command provided by istio-proxy but it says permission issue.

tcpdump: eth0: You don't have permission to capture on that device

raised bug request : can't use tcpdump in istio-proxy - eth0: You don't have permission to capture on that device · Issue #30982 · istio/istio · GitHub

After running istioctl install --set values.global.proxy.privileged=true i was able to run sudo tcpdump command.

Faced 2 issues:

  1. The istio-proxy is mounted as read-only and it was not letting me genrate app1.pcap file sudo tcpdump -vv -ni eth0 -w app2.cap

workaround : cd /dev then run tcpdump -vv -ni eth0 -w app2.cap

  1. couldn’t copy app2.cap from istio-proxy container to host with docker cp 370d55b9ca09:/dev/app2.cap /tmp as it was giving Error: No such container:path: 370d55b9ca09:/dev/app2.cap also scp, rsync was not available for obvious reasons :slight_smile:

workaround : used curl command to send app2.cap file

Opening file app2.cap in still shows me TCP connection :frowning: . I might be missing something.

However i have looked at tcpdump example mention here(Istio / Mutual TLS Migration) which shows the encrypted data when i run curl app2 from pod app1. I think this will be good enough proof to show security team :slight_smile:

I was looking at wrong value from start. I found out it’s only displaying TCP in protocol but in background the data is encrypted via mTLS.

To verify it i have first set MODE=PERMISSIVE and sent some curl request too app2.istio service within istio meshed and non-meshed namespace.

From non-istio-meshed namespace:

curl app2.istio/asd

From istio-meshed namespace:

curl app2.istio/asd