Introducing on-the-fly 10 Gbit pcap compression on n2disk

Posted · Add Comment

Compressing pcap produced by n2disk is a good idea for a few reasons:

  1. It allows disk space to be saved as compressed data takes less space on disk.
  2. It enables the creation of cheaper packet recorder appliances as with the same hardware you can save more data onto disk and thus in some problem domain you can double the capacity of your existing box.
  3. As n2disk leaves some space in CPU cycles (in particular when used on top of Napatech adapters) we have room to compress and index packets on-the-fly at 10 Gbit, without dropping a byte.

As many of you know, lossless data compression works based on the assumption that on data there are sequences of repeating characters that can be replaced with only one sequence. This means that in order to compress data n2disk has to buffer some packets (~ 1 MB) and then compress them in chunks (i.e. we don’t compress packet-by-packet). Due to compression, file is not anymore in pcap format (we use the .npcap file extension), the internal format is still pcap but it is compressed, similar to what you can do with gzip when you compress a pcap file. The advantage with .npcap files with respect to .pcap.gz is that it is compressed in 1 MB chunks and thus if we want to extract (e.g. using the index generated by n2disk) a few packets, we can can decompress only the chunks that contain them instead of decompressing the whole file that is pretty time consuming (remember that at 10 Gbit, every second you produce 1.25 GB of data).

In order to support compression, we have revamped all the n2disk companion tools so that they can search, extract, decompress n2disk compressed files. On the nBox interface you will not see any difference between a compressed pcap or an uncompressed one, as everything is handled transparently. The only difference is that you save space on disk.

We want to repeat that pcap compression is an activity performed by n2disk:

  1. During packet capture (not not in post-processing).
  2. Packet compression and indexing happen simultaneously during capture.
  3. Pcap indexes are always saved in compressed format, whereas you can choose if you want to dump uncompressed (plain .pcap files) or compressed (.npcap) files.
  4. Packet compression does not degrade the performance as n2disk can still dump to disk traffic at line rate 10 Gbit (64 bytes packets) or at 2 x 10 Gbit if your storage subsystem is adequate.

Said that pcap compression will not affect your capture performance, let’s now see what is the compression ratio you can expect. For our tests we have used the following setup:

  1. Intel(R) Xeon(R) CPU E5-2643 v2 @ 3.50GHz
  2. RAM 4 x 8GB 1600Mhz
  3. 8 x 10K RPM HDD with hardware RAID controller.
Traffic Type Speed
(Gbps)
Packets
(Mpps)
Space
Save
Command Line Used
(Compression + Indexing)
Synthetic
(pfsend)
10 Gbit 14.88 Mpps 95% n2disk10g -i dna0 -o /storage/n2disk/dna0/ -b 4096 -C 4096 -p 1024 -g -s 1518 -M -I -Z -S 0 -c 1 -z 2 -w 3 -m 100 -n 50 -H -A /storage/n2disk/dna0/timeline
High Frequency Trading 10 Gbit 9.62 Mpps 82% n2disk10g -i dna0 -o /storage/n2disk/dna0/ -b 4096 -C 4096 -p 1024 -g -s 1518 -M -I -Z -S 0 -c 1 -z 2,3 -w 4 -m 100 -n 50 -H -A /storage/n2disk/dna0/timeline
Mixed Mobile
(GTP-encapsulated)
10 Gbit 2.38 Mpps 6% n2disk10g -i dna0 -o /storage/n2disk/dna0/ -b 4096 -C 4096 -p 1024 -g -s 1518 -M -I -Z -S 0 -c 1 -z 2,3 -w 4 -m 100 -n 50 -H -A /storage/n2disk/dna0/timeline
Internet Worst Case
(Youtube and Torrent)
10 Gbit 1.36 Mpps 6-10% n2disk10g -i dna0 -o /storage/n2disk/dna0/ -b 4096 -C 4096 -p 1024 -g -s 1518 -M -I -Z -S 0 -c 1 -z 2,3 -w 4 -m 100 -n 50 -H -A /storage/n2disk/dna0/timeline

 

As you can see the compression rate strongly depends on the traffic type. We can save from as little as 6% when compressing YouTube streams, up to 95% when compressing synthetic traffic generated by pfsend. In our tests, using mixed Internet traffic (email, web, moderate download) we can save in average 25-40%. So in essence compression ratio depends on the traffic you are going to compress. In some specific environments such as high frequency trading you can squeeze a 100 MB pcap into a 18 MB compressed pcap file, that means that your 10 Gbit storage subsystem can be adequate for saving 2×10 Gbit thanks to the compression.

In conclusion, you need to test on your environment what is your average compression rate based on your typical traffic pattern. The good news are:

  • Compression takes place on-the-fly without affecting capture performance (so why not to use it if it basically comes for free).
  • n2disk companion tools seamlessly handle compressed and uncompressed pcaps.
  • pcap compression is supported on all n2disk versions, running on top of PF_RING, PF_RING ZC/DNA, and Napatech.
  • There is no extra fee for n2disk compression for both existing and new users: give it a try!