Continuous packet recorders are devices that capture raw traffic to disk, providing a window into network history, that allows you to go back in time when a network event occurs, and analyse traffic up to the packet level to find the exact network activity that caused the problem.
n2disk is a software application part of the ntop suite able to capture traffic at high speed (it relies on the PF_RING packet capture framework, able to deliver line-rate packet capture up to 100 Gbit/s) and dump traffic to disk using the standard PCAP format (which is used by packet analysis tools like Wireshark and ntopng). Network traffic is recorded permanently and the oldest data is overwritten as disk space fills up, in order to provide continuous recording and the best data retention time.
Besides storing network data to disk, n2disk can also:
- Index and organize data in a timeline, to be able to retrieve traffic searching for packets matching a specific BPF filter in a selected time interval.
- Compress data to save disk space (if you compile pcap-based applications on top of PF_RING-aware libpcap, any application compatible with the PCAP format can read compressed pcap files seamlessly).
- Filter traffic, up to L7: you can discard traffic matching selected application protocols.
- Shunt traffic: you can save disk space by recording only a few packets from the beginning of each communication for selected application protocols (e.g. encrypted or multimedia elephant flows).
- Slice packets: the ability to reduce packet size by cutting them (e.g. up to the IP or TCP/UDP header).
In a previous post (part 1) we described how to build a 2×10 Gbit continuous packet recorder using n2disk and PF_RING, however it’s time to update it as a few years have past, new features have been added, and new capture and storage technologies are available.
Network Adapter: Intel vs FPGAs
All ntop applications, including n2disk, are based on PF_RING and can operate on top of commodity adapters (accelerated Zero-Copy drivers are available for Intel) as well as specialised FPGAs adapters like Napatech, Fiberblaze and others (the full list is available in the PF_RING documentation).
In order to choose the best adapter we need to take into account a few factors, including capture speed, features and price. Intel adapters are cheap and can deliver 10+ Gbps packet capture with 64 byte packets using PF_RING ZC accelerated drivers. FPGA adapters are (more or less, depending on the manufacturer) expensive, but provide, in addition to higher capture speed with small packet sizes, support for useful features like ports aggregation, nanosecond timestamping, traffic filtering. We can summarize it in a small table:
| Link Speed / Features Required | Recommended Adapter |
|---|---|
| 1 Gbit | Any adapter |
| 10 Gbit | Intel (e.g. 82599/X520) |
| 2 x 10 Gbit Aggregation / Nanosecond Timestamp | FPGA (Napatech, Silicom) |
| 40/100 Gbit | FPGA (Napatech, Silicom) |
What Storage System Do I Need?
If you need to record 1 Gbps, even a single (fast) HDD is enough to keep up with the traffic throughput. If you need to record 10+ Gbps, you need to increase the I/O throughput by using a RAID system with many drives. At ntop we usually use 2.5″ 10K RPM SAS HDD drives for compact systems (e.g. 2U form factor with up to 24 disks), or 3.5″ 7.2 KRPM SAS HDD for modular systems when rack space is not a problem and many units are required to increase data retention (in this case you need to use a RAID controller able to drive a SAS expander, which is able to handle hundreds of disks). More space in the storage system translates in a higher retention time and thus the ability to go further back in time to find old data.
The number of drives, combined with the I/O throughput for each drive, and the RAID configuration, determine the final I/O throughput you are able to achieve. The drive speed depends on the drive type and model, they can be summarized in the table below:
| Drive Type | Sustained Sequential I/O |
|---|---|
| SAS/SATA HDD | 100-250 MB/s |
| SSD | 100-500 MB/s |
| NVMe (PCIe SSD) | 500-3000 MB/s |
In order to record traffic at 10 Gbps for instance, you need 8-10 SAS HDDs in RAID 0, 10-12 disks in RAID 50. The RAID controller should have at least 1-2 GB of buffer onboard in order to keep up with 10+ Gbps. Alternatively you can use 3-5 SSDs, or 1-2 NVMe (PCIe SSD) drives. SSDs are usually used when concurrent write and read are required under intensive workload, to avoid that HDD’s seek time jeopardize the performance. Please make sure that you select write-intensive flash disks that guarantee great endurance over time.
At 20 Gbps at ntop we usually use 16-24 HDDs. At 40-100 Gbps you probably also need to use multiple controllers as most controllers are able to handle up to 35-40 Gbps sustained and you need to distribute the load across a few of them. In fact, since version 3.2, n2disk implements multithreaded dump, that means it is able to write to multiple volumes in parallel. This is also useful when using NVMe disks, as they are directly attached to the PCIe bus and lightfast, but they cannot be driven by a standard controller, thus you can use n2disk to write to many NVMe disks in parallel: we have been able to achieve 140 Gbps of sustained throughput using 8 write-intensive NVMe disks!
What CPU Do I Need?
Choosing the right CPU depends on a few factors.
First of all the adapter model. Intel adapters transfers packets one-by-one putting pressure on the PCIe bus and thus increasing the overall system utilisation with respect to FPGA adapter like Napatech or Silicom that are able to work in “chunk” mode (other NIC vendors such as Accolade for instance do not support it yet). FPGA adapters are also able to aggregate traffic in hardware at line-rate, whereas with Intel we need to merge packets on the host, and it is hard to scale above 20-25 Mpps in this configuration. A CPU with high frequency (3+ GHz) is required with Intel.
The second factor is definitely traffic indexing. You probably want to index traffic to accelerate traffic extraction and this requires a few CPU cores. In order to index traffic on the fly at 10 Gbps, 1-2 dedicated cores/threads are required (in addition to the capture and writer threads. At 40 Gbps you probably need 4-6 indexing threads. At 100 Gbps at least 8-10 threads.
In short, if you need to record 10 Gbps, a cheap Intel Xeon E3 with 4 cores and 3+ GHz is usually enough even with Intel adapters. If you need to record and index 20+ Gbps, you should probably go with something more expensive like an Intel Xeon Scalable (e.g. Xeon Gold 6136) with 12+ cores and 3+ GHz. Pay attention to the core affinity and NUMA as already discussed in the past.
How Much Does It Cost?
Continuous packet recorders on the market are expensive devices because they need fast/expensive storage systems and they are usually part of enterprise-grade solutions designed for high-end customers. At ntop we want to deliver everyone the best technology at affordable prices, and we recently updated our price list lowering down prices for the n2disk product (please check the shop for more info). Education and no-profit can use our commercial tools at no cost.
For further information about the n2disk configuration and tuning, please refer to the n2disk documentation.