tshark can’t process macOS’s pcapng file well

Wireshark‘s tshark program can’t process macOS‘s pcapng file well. E.g.:

$ sudo tcpdump -w foo.pcapng
Password:
tcpdump: data link type PKTAP
tcpdump: listening on pktap, link-type PKTAP (Apple DLT_PKTAP), capture size 262144 bytes
^C24 packets captured
27 packets received by filter
0 packets dropped by kernel

Use tshark to read and write the generated foo.pcapng:

$ tshark -r foo.pcapng -w bar.pcapng
tshark: An error occurred while writing to the file "bar.pcapng": Internal error.

I also met following error before:

$ tshark -r apsd-107.pcapng -w foo.pcapng
tshark: The capture file being read can't be written as a "pcapng" file.

macOS has its own bespoke libpcap and tcpdump, so if the pcapng file is generated by tcpdump, using tcpdump itself to process pcapng file seems the only choice.

A workaround is if you don’t care about losing information, you can use wireshark to convert the pcapng file to pcap first:

Compare error prompts of different awk flavours

Awk requires the opening brace of an action must be on the same line as the pattern it accompanies. I compared error prompts of different awk flavours, and my OS is NetBSD.

(1) awk shipped in NetBSD:

# awk 'BEGIN
> {print "Hello Awk!"}'
awk: syntax error at source line 2
 context is
        BEGIN >>>
 <<<
awk: bailing out at source line 2

(2) nawk:

# nawk 'BEGIN
{print "Hello Awk!"}'
nawk: syntax error at source line 2
 context is
        BEGIN >>>
 <<<
nawk: bailing out at source line 2

(3) mawk:

# mawk 'BEGIN
{print "Hello Awk!"}'
mawk: line 1: syntax error at or near end of line

(4) gawk:

# gawk 'BEGIN
{print "Hello Awk!"}'
gawk: cmd. line:2: BEGIN blocks must have an action part

IMHO, gawk gives more clear clues for this error.

Process large pcap file

To process large pcap file, usually it is better to split it into small chunks first, then process every chunk in parallel. I implement a simple shell script to do it:

#!/bin/sh

input_pcap=input.pcap
output_pcap=./pcap/frag.pcap
spilt_size=1000
output_index=1
loop_count=10
exit_flag=0

command() {
    echo "$1" "$2" > log"$2"
}

tcpdump -r ${input_pcap} -w ${output_pcap} -C ${spilt_size}

command ${output_pcap}

while :
do
    loop_index=0
    while test ${loop_index} -lt ${loop_count}
    do
        if test -e ${output_pcap}${output_index}
        then
            command ${output_pcap} ${output_index} &
            output_index=$((output_index + 1))
            loop_index=$((loop_index + 1))
        else
            exit_flag=1
            break
        fi
    done
    wait

    if test ${exit_flag} -eq 1
    then
        exit 0
    fi
done

First of all, split input pcap file into 1GB chunks. Then launch 10 processes to crunch data (in above example, just simple output). Definitely you can customize it.

BTW, the code can be downloaded here.

The IO stream’s state when EOF occurs

Check following simple C++ program:

#include <iostream>
using namespace std;

int
main()
{
    char ch;

    while (cin >> ch)
    {
        cout << ch << '\n';
    }

    cout << "bad: " << cin.bad() << ", eof: " << cin.eof() << ", fail: " << cin.fail() << '\n';

    return 0;
}

Compile and run it, press Ctrl + D:

$ c++ foo.cpp -o foo
$ ./foo
bad: 0, eof: 1, fail: 1

We can see both fail and eof bits are set to 1. From this table, we can see when both fail and eof bits are set to 1, the operator bool of stream will return false.

pcap_next_ex() blocks in Void Linux forever

My Void Linux is a virtual machine. I implemented a simple capturing packets program using libpcap, but found the program is blocked in pcap_next_ex():

(gdb) bt
#0  0x00007ffff7ad7763 in __GI___poll (fds=fds@entry=0x7fffffffe160, nfds=nfds@entry=1, timeout=-1)
    at ../sysdeps/unix/sysv/linux/poll.c:29
#1  0x00007ffff7f8309d in poll (__timeout=<optimized out>, __nfds=1, __fds=0x7fffffffe160) at /usr/include/bits/poll2.h:46
#2  pcap_wait_for_frames_mmap (handle=handle@entry=0x55555556eba0) at ./pcap-linux.c:5018
#3  0x00007ffff7f886f4 in pcap_read_linux_mmap_v3 (handle=0x55555556eba0, max_packets=1, callback=0x7ffff7f82bf0 <pcap_oneshot_mmap>,
    user=0x7fffffffe1f0 "\260\355VUUU") at ./pcap-linux.c:5577
#4  0x00007ffff7f8c6a2 in pcap_next_ex (p=<optimized out>, pkt_header=<optimized out>, pkt_data=<optimized out>) at ./pcap.c:505
......

The libpcap version in Void Linux is 1.9.1. After checking code, I found although I set timeout in pcap_open_live(), this value doesn’t take effect in set_poll_timeout:

    ......
    if (handlep->tp_version == TPACKET_V3 && !broken_tpacket_v3)
        handlep->poll_timeout = -1; /* block forever, let TPACKET_V3 wake us up */
    ......

I don’t have physical machine, so not sure this issue only happens in virtual machine or not.