Introducing n2disk 3.6: full L7 support, fast flow export, replay rate control

Posted · Add Comment

This is to announce a new n2disk release 3.6.

This release adds full support for indexing and retrieving traffic based on the Layer-7 application protocol. This can now be enabled even when flow export is disabled, and it is possible to use the extraction tool to extract selected application traffic using the Layer-7 protocol as part of the nBPF filter.

n2disk is now also able to use the main storage as a cache, and in the meantime archive pcap files moving them from the fast to a slower storage, even when the new “disk-limit” file schema. This is useful to handle peak hours with high throughput using a fast/small NVMe storage, moving data to a slower/larger/cheaper storage off peak, slowly or overnight.

Flow export has been optimized, to be able to handle a high number of flows to be exported to ntopng while dumping traffic to disk. More settings have also been added to provide full control on flow termination and export.

The disk2n tool has also been improved: it is now possible to control the transmission rate by specifying the number of packets per second (e.g. –transmission-rate 100 – 100 packets/s), or bit rate (e.g. –transmission-rate 1.25Gbps – 1.25 Gigabit/sec), or the relative speed (e.g. –transmission-rate 50% – 50% of the original traffic rate).

Below you can find the full changelog.

Enjoy!

Changelog

  • n2disk (dump)
    • Add support for Metawatch Metamako packet trailer (timestamp is added to packet header and index, device and port ID are exported as flow metadata using INPUT_SNMP/OUTPUT_SNMP/OBSERVATION_POINT_ID IEs)
    • Add support for Arista 7150 Series packet trailer and keyframes (timestamp is added to packet header and index)
    • New -E 2 option to enable application protocol (L7) indexing when ZMQ export is disabled
    • Add support for archiving to a slower storage (-O) when the –disk-limit dump schema is used
    • Set a default disk limit (auto computing 80% of free space + space already in use) when not configured
    • Increse maximum number of interfaces (up to 32)
    • Exporting FirstDumpedEpoch only when available
    • Fix access to latest deleted PCAP file epoch
    • Fix drop stats in PCAP mode (do not account drop in recv)
    • Fix index root folder (when a folder different from the dump folder is specified)
    • Fix -I<index path> with –disk-limit
    • Support for Ubuntu 20
  • Flow export
    • Add –lifetime-timeout and –idle-timeout options to control flow expiration
    • Optimize flow export with batch mode
    • Fix ZMQ message ID
  • npcapextract
    • Add support for L7 filtering using nBPF
  •  disk2n (replay)
    • Add new –transmission-rate option to set the replay speed in bps, pps or % (relative to the original traffic speed)
  • Misc
    • Add -a option to npcapmove to generate absolute paths
    • Fix npcapmanage in case of relative paths
    • Fix logrotate configuration file permission