Introducing n2disk 3.0

Posted · Add Comment

This is to announce n2disk 3.0 that is more than a maintenance release, as it:

  • Consolidates pre-existing functionalities
  • Adds extraction security features that pave the way to GDPR support.
  • Adds flow offload support
  • Simplifies storage management to avoid headaches during the n2disk configuration

During our last meeting at Sharkfest EU we talked about Hardware Flow Offload. In essence, applications running on top of PF_RING and (supported) FPGA adapters are now able to offload flow processing to the network card that be programmed to:

  1. Keep flow state, doing (basic) flow classification in hw.
  2. Periodically provide informations like hash, packets, bytes, first/last packet timestamp, tcp flags, to the application.
  3. Drop/bypass/prioritize flow packets.

This technology dramatically reduces CPU utilization in applications like our nProbe Cento Netflow generator, or IDSs like Suricata. With this new release we added support for Flow Offload also in n2disk, for a better/faster integration with Netflow applications like nProbe Cento. Thanks to this integration, n2disk is able to record raw data while feeding nProbe Cento with flow updates. Optionally, when nDPI is enabled in nProbe Cento, for L7 protocol detection, n2disk can be instructed to forward also raw traffic, using a feedback queue for shunting flow packets as soon as the nDPI engine detects the protocol. All this allows you to do traffic recording and Netflow generation at high speed on the same box with a really low CPU utilization!

 

Those familiar with the n2disk configuration have likely spent some time finding the right dump-set sizing configuration. Until the previous n2disk version, in order to configure the maximum space on disk to be used for PCAP files (n2disk overwrites old files when the maximum data retention is reached), the user was required to set

  • (A) The maximum file size
  • (B) The maximum number of folders containing PCAP files (this is needed to improve the filesystem performance)
  • (C) The maximum number of files per folder.

As a result, the maximum amount of disk space that n2disk could use was A x B x C. This is not really user friendly, and there is also another limitation: A is the *maximum* disk space that n2disk is able to use, in fact it happens to create PCAP files smaller than the configured file size (e.g. when enabling the index timeline that cuts PCAP files in time slots) with the result that the dump set usually contains less data than the maximum specified.


With this new n2disk release, it is now possible to simply specify the disk space to use, either as absolute value (MBytes) or disk size percentage. n2disk will track disk usage and dynamically computes the number of folders and files to keep on disk in order to match exactly the configured disk utilisation.

n2disk 3.0 also introduces PAM support, this allows you to integrate multiple authentication schemes (including LDAP for instance) for granting traffic retrieval capabilities to selected users/groups. This is yet another step forward implementing those measures which meet the principles of data protection as defined by the GDPR regulation.

This is the complete changelog of the 3.0 release:

  • n2disk
    • Dynamic disk management: new –disk-limit option to specify the max amount of disk space to use (MByte or %), instead of using -m (max number of files) and -n (max number of directories) which is less flexible.
    • Raw packets and flow updates export based on the new PF_RING 7 flow offload support. This also includes a feedback queue for raw packets shunting (when used in combination with nProbe Cento and with DPI enabled for instance).
    • Support for kill -USR1 to close and flush the current pcap in order to make live traffic immediately available
    • Microburst detection now works also in multithreaded capture mode (ZC) and segment mode (FPGA capture)
    • New –reader-threads-queue-len option to configure the queue len in multithreaded capture
    • Printing microseconds in timeline file names always with 6 digits now
    • Fixed drop percentage stats
    • Fixed threads synchronisation
  • disk2n
    • Fixed nanosecond pcap files replay
  • npcapextract
    • Support for PAM authentication for running extractions
    • New -o – option to write to stdout
    • Fixed extraction of huge packets (e.g. captured with gro)
    • Fixed npcap open mode from ‘read/write’ to ‘read only’
    • Fixed extraction on compressed pcaps
  • Tools
    • New npcapdecompress -o – option to write to stdout
    • New npcapprintindex option -c to check index sanity
    • New npcaprepair tool for reparing indexes and timeline
    • npcapmode now creates relative paths
  • Misc
    • Fixed a few corner cases in the init.d scripts
    • Fixed systemd dependencies