Building a (Cheap) 2×10 Gbit (Continuous) Packet Recorder using n2disk and PF_RING

Posted · Add Comment

Continuous packet recorders are devices that capture network traffic and save it to disk. The term continuous means that this activity is performed “continuously” until the device is active and not just for a few minutes. At ntop we have developed two companion applications to be used on a packet recorder:

  1. n2disk is a software application that captures network at line rate (multi 10 Gbit) and dumps it to disk on pcap format. During packet capture, n2disk can also:
    1. Create a pcap index to be used for searching specific packets matching a BPF filter out of the captured traffic. In essence it speeds up an operation that without an index would required to read the full pcap from the beginning to the end.
    2. Compress captured traffic during capture to save disk space and thus decrease search time as the applications have to manipulate smaller pcap files. If you compile pcap-based applications on top of PF_RING-aware libpcap, all apps (e.g. tcpdump and wireshark) can read compressed pcap files seamlessly.
  2. disk2n is a software application of reproducing pcap files either at line rate or at the same capture speed, so that you can reproduce on your lab the same traffic conditions that happened when n2disk captured the traffic. Note that disk2n can reproduce any pcap file (not just those captured by n2disk) and that the amount of traffic to be reproduced can exceed the available memory (i.e. you can reproduce multiple pcap files in sequence that can be even more than a Terabyte in size).

In order to simplify the operations, we have created the free nBox web GUI that allows users to graphically start/stop/replay/filter/download traffic in a matter of clicks. All the above applications operate at multi-10 Gbit on top of PF_RING ZC that features not just high speed packet-capture and replay but also comes with free applications such as zero-copy packet balancers and fan-out to manipulate traffic prior to dump it to disk (e.g. send the same ingress packet to both n2disk and nProbe for generating traffic traces). All applications can operate on top of Intel network adapters and specialised NICs such as those manufactured by Napatech.

Traditionally packet recorders are expensive devices because they need a fast storage system, and also because manufacturers sometimes have charged an “extra” for high-end customers. At ntop we believe instead in simplicity and on the fact that we should give everyone the best technology at affordable prices, adding a low price tag enough for us to continue innovate though research. On this blog post we will explain how to build a packet recorder using n2disk and commodity hardware so that you can build it yourself.


Question 1: Intel or Napatech NICs?

Our readers know that we have pioneered packet capture on commodity hardware since many years, but at the same time PF_RING ZC also supports specialised NICs such as those manufactured by Napatech. If with PF_RING ZC on top of Intel adapters we can achieve 10G packet capture with 64 byte packets, why bother with Napatech NICs that sport many nice features (e.g. traffic balancing/filtering in hardware) for an extra cost? The answer to this question isn’t as simple as a yes/no, so we’ll try to clarify it in detail.

  • Intel Adapters
    • [+] Cheap network adapters, available from the shop around the corner.
    • [+] Natively supported by PF_RING ZC both in RX and TX line rate multi-10G.
    • [-] Packet timestamps computed in software (unless you use specialised NICs that will limit the capture performance as the packet payload is extended with the hardware timestamp).
    • [-] All non capture-related activities (e.g. filtering or balancing) happen on the CPU on PF_RING ZC. In order to do this you need extra CPU cores devoted to this activity and thus a much more expensive CPU.
    • [-] With small packets, the NIC transfers packets one-by-one putting pressure on the PCIe bus and thus increasing system utilisation with respect to Napatech.
    • [-] When capturing from multiple network adapters (e.g. from 2 x 10G ports), packet merging happens on n2disk at a cost of an extra CPU load. Due to this it is not possible to merge and index/compress packets at line rate on 20G. There is a workaround explained later on this post.
  • Napatech Adapters
    • [+] High-precision hardware timestamps, in-hardware packet filtering/slicing/balancing, large in-card memory buffers for virtually 0-packet loss even in worst cases.
    • [+] Very efficient packet transfer from NIC-to-CPU and special “capture” mode that significantly reduces CPU utilisation with respect to Intel. This means that the CPU you can use with Napatech NIC can be much cheaper and with fewer cores than the one you need to use with Intel.
    • [+] Merging of multiple 10G ports happens in hardware with high-precision timestamps, offloading the CPU for this task.
    • [-] Napatech NICs have an extra cost with respect to Intel NICs even though you can save money with CPU and storage as explained later on this post.

The good news is that PF_RING ZC masks all these differences, so for an end user operating a packet recorder on top of Intel or Napatech NICs is basically the same.


Question 2: What Storage System Do I Need?

At ntop we use 10k RPM SATA drives. You can use faster 15K RPM SAS drives or SSDs, but in our experience the speed increase you will have has a price tag in terms of higher price and smaller storage. So the user will decide but high-quality 10k RPM SATA drives are good. For 10 Gbit to disk you need at least 8 x drives, for 20G you need at least 16 disks. If you decide to use Napatech NICs, you need at least 10 or 20 drives instead, as Napatech NICs also captures the ethernet CRC slightly increasing the data volume (i.e. your NIC will send to the host more that 10 Gbit due to the ethernet header). In general these are the minimum you can buy. As 10K RPM SATA drives are usually 1 TB in size, (remember that at 20G you capture 2.5 GB/sec) you will probably want to use at least 16 drives for 10G and 24 drives for 20G in order to have a storage system adequate with your needs.

As previously explained, Napatech NICs do 2x10G merging in hardware, whereas with Intel we need to merge packets on the host. As you will probably want to index and (maybe also) compress the captured traffic, using Intel NICs you cannot achieve all this on top of packet merging. For this reason if you decide to use a Napatech NIC with n2disk, you can use one single RAID subsystem where you can store 20G. With Intel NICs you need two RAID subsystem: one for one NIC and one for the other. When you extract/filter packets, the nBox will transparently merge both NICs honouring packet timestamps (so in a way the result is the same). The drawback of this Intel-based solution is that we cannot do all this on a CPU node at 20G as the number of cores we will need will be too high. So at 20G for packet capture+index+compression you need respectively:

  • Intel
    • Dual CPU (2 NUMA nodes): each CPU will take care of one 10G network adapter.
    • As you need many cores (at least 6) you need to use for instance Intel E5-2677 or (better) E5-2690. In essence be prepared to spend 3k/4k USD just for the CPUs.
    • 2 x single-port 10G Intel adapters: you need to install one adapter per NUMA node (remember this caveat).
    • 2 x RAID controller, each driving 12 drives: you need to install one controller per NUMA node.
  • Napatech
    • Single node system. A 4-cores Intel E5-2643 is enough or (better) an E5-2677.
    • Single RAID controller.

For RAID controller you can use various products (e.g. this or this) depending on the number of drives and expandability.


Final Remarks

As described above on this article, ntop products support 2 x 10G to disk using both Intel and Napatech NICs. Goal of this article is to tell you what are the hardware components you need to buy for building yourself a continuous packet capture device. We have described the pros and cons of both platforms, and explained that the BOM (Bill Of Materials) for Intel and Napatech is different, but money-wise pretty close. Choosing one NIC or the other has impact on the server you need to buy and on its architecture. In essence is the user that will decide which solution best first its requirements, said that ntop support seamlessly both platforms using the same unified n2disk/nBox web interface.

Now it’s time to build yourself your first packet recorder device !



Q. How can I build a 40 Gbit packet recorder?
A. Using Intel NICs you need a 4 nodes NUMA system, 4 NICs, 4 x 10G single-port adapters. Every node will save a portion of the traffic and during extraction ntop tools will merge directions. With Napatech you need a single node 8-core (fast) CPU or a dual-node NUMA system. Of course you need double the disks for sustaining speed.

Q. How can I build a system with hundred of TBs?
A. The cheapest solution is to use a RAID controller able to driver a SAS expander. The controller listed earlier on this post can drive up to 256 disks.

Q. What hardware system is adequate for building a 2 x 10G packet recorder?
A. There are many options available including Dell R720 (CPU E5-2690 v2, RAID controller Perc H710P) or Supermicro 2027R.