Extracting Packets From Large Captures
Sometimes we have to work with very large packet captures, captures that can be several gigabytes in size. These are cumbersome (if even possible) to analyze in an application like Wireshark because the entire capture file must be loaded into running memory at once. And manual analysis typically means we're only interested in a small portion of the capture file anyway. How can we break a huge capture into smaller, more manageable chunks?
One approach is to cut the file into slices, with each slice a containing a constant number of packets or bytes or covering a given length of time. Practically speaking, this is how huge traffic captures should be performed in the first place, using a ring buffer. But we can similarly chop up large capture files after the fact using editcap (part of the Wireshark family).
Take for instance this capture file of over a quarter million packets weighing in at 256 MB:
$ capinfos lotsapackets.cap File name: lotsapackets.cap File type: Wireshark/tcpdump/... - libpcap File encapsulation: Ethernet Number of packets: 260778 File size: 267802612 bytes Data size: 263630140 bytes Capture duration: 204 seconds Start time: Mon Apr 4 22:31:52 2011 End time: Mon Apr 4 22:35:16 2011 Data byte rate: 1290166.92 bytes/sec Data bit rate: 10321335.37 bits/sec Average packet size: 1010.94 bytes Average packet rate: 1276.21 packets/sec
256 MB certainly isn't an insurmountable file size, but it's large enough that we may want something more flexible. We can split it into a few files of, say, 50,000 packets each and load each one into Wireshark individually.
$ editcap -c 50000 lotsapackets.cap fewerpackets.cap $ capinfos -c fewerpackets*.cap File name: fewerpackets_00000_20110404223152.cap Number of packets: 50000 File name: fewerpackets_00001_20110404223310.cap Number of packets: 50000 File name: fewerpackets_00002_20110404223340.cap Number of packets: 50000 File name: fewerpackets_00003_20110404223410.cap Number of packets: 50000 File name: fewerpackets_00004_20110404223440.cap Number of packets: 50000 File name: fewerpackets_00005_20110404223510.cap Number of packets: 10778
That's a pretty good solution, but as mentioned earlier, we're probably only interested in specific traffic; for example, only traffic destined for a particular host, or a particular UDP or TCP port number. Searching each of these smaller files by hand would be a tedious waste of your time.
Luckily, we can use tshark (another Wireshark tool) to extract interesting traffic from a capture file. We just need to define a display filter to match the traffic we want. For example, if we wanted to extract all DNS traffic from our large capture file, we could do this:
$ tshark -r lotsapackets.cap -R dns -w dns.cap $ capinfos dns.cap File name: dns.cap File type: Wireshark/tcpdump/... - libpcap File encapsulation: Ethernet Number of packets: 220 File size: 24664 bytes Data size: 21120 bytes Capture duration: 32 seconds Start time: Mon Apr 4 22:31:58 2011 End time: Mon Apr 4 22:32:30 2011 Data byte rate: 656.35 bytes/sec Data bit rate: 5250.76 bits/sec Average packet size: 96.00 bytes Average packet rate: 6.84 packets/sec
After just a few seconds, tshark copies every DNS packet from our original capture into the much smaller file dns.cap, which we can easily examine in Wireshark at our leisure.
Filters can be mixed and matched just like in Wireshark. The example below matches all DNS traffic and all traffic sent to or from TCP port 80.
$ tshark -r lotsapackets.cap -R "dns or tcp.port==80" -w web.cap $ capinfos web.cap File name: web.cap File type: Wireshark/tcpdump/... - libpcap File encapsulation: Ethernet Number of packets: 1559 File size: 644994 bytes Data size: 620026 bytes Capture duration: 165 seconds Start time: Mon Apr 4 22:31:55 2011 End time: Mon Apr 4 22:34:40 2011 Data byte rate: 3757.36 bytes/sec Data bit rate: 30058.86 bits/sec Average packet size: 397.71 bytes Average packet rate: 9.45 packets/sec
The Wireshark display filter cheat sheet offers an idea of the variety of filters available. With a little practice you should be able to automatically extract just about any type of traffic.
About the Author
Jeremy Stretch is a networking engineer and the maintainer of PacketLife.net. He currently lives in the Raleigh-Durham area of North Carolina. Although employed full-time out of necessity, his true passion lies in improving the field of network engineering around the world. You can contact him by email or follow him on Twitter.
Comments
You can also use Perl to parse pcap files and extract out the packets which interest you: http://hype-free.blogspot.com/2010/03/parsing-pcap-files-with-perl.html
There is a great command line tool for Windows called "SplitCap" that can split a pcap into multiple pcap files, one for each unique TCP/UDP session. SplitCap can also split a large PCAP file based on IP addresses, so that each IP host on the network gets its packets in a separate file (with the "-s host" switch)
SplitCap is really fast and outperforms all other pcap splitting tools I know.
SplitCap is available from SourceForge:
http://splitcap.sourceforge.net/
Nice article Stretch.
I usually go the tshark route, since I'm usually dealing with large pcaps with VoIP traffic in it. Snag my signaling w/SDP with the first tshark run and print it so I can grab the UDP ports from the SDP, then run it a second time with the same signaling filter and add the UDP ports from my SDP. Works slick.
Great article thanks Stretch. Wish i'd saw it a few days earlier :)
Just droppin a note - this was helpful :) Didn't know these utils came with tshark/wireshark. Handy.
Hi Stretch,
Nice article - I saw a similar tool on honeynet.org a tool named 'streams' which was announced recently. To quote their blog:
"If you ever needed to process large pcap files on a session level, you will love this tool"
See:
http://www.honeynet.org/node/633
and
http://src.carnivore.it/streams/about
for more info.
Chris