Continuous network traffic recorders are applications (or appliances) that write network traffic on disk. In case of issues (e.g. security breach or network outage) they enable network and security analysts to go back in time and see how a problem originated. The main limitation of this practice is that a lot of data it is written to disk even when there is nothing special happening on the network. Similar to the evolution of surveillance cameras that implemented “motion detection” to trigger recording when some meaningful even happen, this is to introduce a similar concept for network packet recording. Namely, we now enable (this is a configuration option) the ability to record network traffic only if relevant activities are detected (call it “network motion detection” if you wish): less disk space used, more data recorded, faster data access as non relevant data is discarded. IMPORTANT: contrary to “motion detection” where recording starts the a motion is detected, in smart packet recording we have implemented a mechanism (through a technique called “extract on recording“) that allows us to start recording before the security event is triggered, and so give people the ability to see how everything started and not when the problem is already in place. Below we show how nDPI/ntopng (and Suricata via ntopng) trigger smart recording, but soon we want to extend this facility to other third party security applications that can trigger recording in n2disk.
Continue reading this article if you want to know how smart recording works in detail.
The Whole Story
In one of our Webinars last year we introduced the idea of implementing a smart way of recording PCAP data. This was based on the necessity of going back in time and get the raw traffic matching an event that our system (e.g. ntopng) detected. In fact, the recording window (data retention) for PCAP data, that depends on the size of the storage system, may not be large enough to contain (and let us download) the data for the event we are analysing, especially at high traffic rates (a 10 Gbit link can produce more than 100 Terabytes per day!).
Those who are running ntopng for security or troubleshooting, in most cases are using it in combination with n2disk for enabling Continuous Traffic Recording. When ntopng triggers an alert, in case of a security event or a Network issue, this allows them to drill down and download the raw packets, providing full evidence of the event. But it happens that at the time the manual analysis is conducted, the PCAP data is no longer on disk, as n2disk already deleted it to make room for new data.
n2disk implements several techniques for improving the data retention and enlarge the recording window. This includes:
- Packet slicing, to strip the packet payload
- Layer 2/3/4 filtering based on optimized BPF filters, to discard traffic based on packet headers
- Layer-7 filtering, to discard or shunt unwanted traffic based on the application protocol
- Traffic compression
However all the filtering techniques described above have some limitation: they are not able to predict which traffic is really important (i.e. packets matching Network events) and keep keep using disk space for traffic which is not really relevant for our analysis.
This is why we introduced the concept of Smart Recording, which is a new technique that allows us prioritise selected traffic (e.g. based on security events generated by ntopng or third party tools), by deleting the rest of the traffic first, when it’s time to make room for new data on disk.
The Smart Recording has been implemented as a new service which is running on the box, which is processing events published by the analysis tools, and it is archiving PCAP data matching those events to a secondary storage (this can be on a dedicated or the same volume). In short n2disk is using:
- a 1st level storage to implement Continuous Recording with a short data retention
- a 2nd level storage to archive traffic for Network events with a longer data retention
This architecture is already supported by ntopng 5.7 (available in ntopng Enterprise XL) and can be enabled from the Traffic Recording page in the ntopng GUI. This requires n2disk to be installed as described in the documentation, and can be enabled in the same page used to manage Traffic Recording:
- Toggle Smart Traffic Recording
- Configure the desired Max Smart Disk Space which is the disk space to be used for the second level storage (events traffic)
- Save the configuration and enable Smart Recording with the Save Settings button
At this point everything is up and running: n2disk is continuously recording PCAP traffic up to the configured Max Disk Space, and the Smart Recording service is processing events and moving event’s data to the secondary storage up to Max Smart Disk Space. When traffic extraction is requested, from one of the charts in the ntopng GUI, or when downloading PCAP traffic for a specific alert or flow, traffic is extracted from the first-level storage if it falls inside the first-level recording window, or from the second-level storage otherwise (only traffic matching network events will be available in the latter case).