Most ntopng users make extensive use of ClickHouse support for storing historical flow data and running analysis on it. ClickHouse is highly optimized and offers a high compression rate (estimated at an average of 60 bytes per flow), allowing for long data retention even with limited storage.
However, to comply with regulations such as GDPR, SOX, HIPAA, and PCI DSS, it is often necessary to retain data for extended periods. This is manageable when flow rates are low to moderate, but can require significant disk space when flow rates are high.
To support long-term data retention on limited storage, we have introduced the ability to export flow data from the database. This feature allows administrators to back up data to external long-term storage systems before it is deleted due to TTL (time to live), ensuring regulatory compliance while freeing up local disk space.
Key Benefits
- Regulatory Compliance: Meet data retention requirements for GDPR, SOX, HIPAA, PCI DSS, and other regulations.
- Storage Optimization: Export data before TTL deletion to free up space.
- Audit Trail: Maintain historical network flow records for forensic analysis.
- Data Recovery: Re-import archived flows for compliance investigations.
Configuration
Enabling this feature is as simple as enabling the preference from the UI. From the Left navbar in ntopng: Settings → Preferences (Expert View) → Clickhouse, and toggle the option as shown in the picture below. With this configured, at each midnight ntopng will export flow records deleted due to TTL.
By default, flows are exported to /var/lib/ntopng, one file per each day is created before deleting data from the clickhouse database due to TTL. It is possible to configure a custom export path from the configuration file or from command line by setting the –db-archive-dir <path> parameter.
Import Archived Data
To restore exported flows for analysis, on a machine with clickhouse-client installed, run:
clickhouse-client –query=”INSERT INTO ntopng.flows FORMAT Native” < FILE_PATH
After importing the archived flows using the command above, it is possible to inspect them from the UI:
- Navigate to Flows → Historical in the ntopng web interface
- Select the time range of your imported data
Alternatively then can be imported in any ClickHouse database and inspect them manually with clickhouse-client.
Important Notes
- Imported data contains 1 day of data with UTC timestamp begin and end, so flows for date X 00:00:00 UTC +24h are stored in a single file.
- Retention rules will delete imported data at midnight if outside the TTL window.
- Plan imports accordingly for compliance analysis.
Related Documentation
- How to start ntopng from command line or config file ntopng Startup Configuration
- ClickHouse Integration Guide
- ClickHouse Cloud Setup
- Storage optimization for clickhouse and ntopng ClickHouse Storage Optimization
Conclusion
The new ClickHouse flow export feature enhances ntopng’s Enterprise capabilities by providing automated compliance archival and storage management. Network administrators can now maintain regulatory compliance while optimizing storage resources through efficient data lifecycle management.
Enjoy!