ntopng 3.6 release is paving the way to metrics-based traffic analysis. We have finally put ntopng on top of a timeseries-independent layer that allowed us to currently RRD and InfluxDB and in the future other backends. This means that you can now also (you can for instance use ntopng as a flow exporter and as a Grafana data source) use ntopng as a time series datasource (see the timeseries API for further information) or you can analyse data through the ntop web interface that has been greatly enhanced.
As you can see from the above chart you can now, for each time period, compare the current traffic (green) with past traffic on the same period (dotted line). This allows you to see how your traffic has changed, and soon we will improve ntopng to trigger alerts whenever (as in the above picture) the traffic has changed significantly with respect to the past. In addition to that, we have introduced the trend line (in blue) that summarises the metric curve trend in a simple way and that it can be used as baseline for comparisons. Zooming has also been modified and now you do that with a mouse on the chart dragging an area as you are used to do with other tools. In the following release we will further enhance ntopng to dump more accurate metrics by lowering the dump cycle time (e.g. for host nDPI we dump counters every 5 minutes that is reasonable on RRD but that could be lowered when InfluxDB is used) so that graphs can be even smoother and traffic comparison will be more accurate. We suggest to use RRD on small hosts that do not have much traffic/hosts to analyse, and use InfluxDB on deployments with many metrics where you need scalability that RRD cannot offer. Remember that you can select the timeseries backend from the ntopng “Timeseries” preference pane.
In addition to timeseries, we have ease debugging an troubleshooting by introducing configuration backup/restore so that you can clone your ntopng hosts with ease in case of crash. In case you need to do a quick packet capture, instead of leaving the ntopng interface for using the command line, you can now do that from within ntopng. Under the each interface and host, in addition to the existing JSON link, you can now find a new element for streaming a quick capture to your browser.
if you click on the button from a host, only the traffic from/to the host will be dumped, instead if you select the same feature from the network interface all the traffic will be considered. You can also set an optional BPF filter for further refining the traffic you are interested in (e.g. “port 53” for DNS traffic only) and thus avoid downloading too much data.
In this release we have also greatly reworked SNMP, added ubuntu 18.04 support, moved from GeoIP to libmaxmind(*) for geolocation, improved Slack alerting, and made several changes that should have greatly hardened ntopng. As usual for all details, please refer to the complete changelog you can find below.
Enjoy!
(*) Ubuntu 14 users: run add-apt-repository ppa:maxmind/ppa && apt-get update && apt-get install libmaxminddb0 to install the new geolocation library before upgrading ntopng.
ntopng 3.6 Changelog
New features
- New pro charts
- Ability to compare data with the past (time shift)
- Trend lines based on ASAP
- Average and percentile lines overlayed on the graph and animated
- New color scheme that uses pastel colors for better visualization
- https://www.ntop.org/ntopng/ntopng-and-time-series-from-rrd-to-influxdb-new-charts-with-time-shift/
- New timeseries API with support for RRD and InfluxDB
- Abstracts and handles multiple sources transparently
- https://www.ntop.org/guides/ntopng/api/lua/timeseries/index.html
- Streaming pcap captures with BPF support
- Download live packet captures right from the browser
- New SNMP devices caching
- Periodically cache information of all the SNMP device configured
- Calculate and visualize interfaces throughput
Improvements
- Security
- Access to the web user interface is controlled with ACLs
- Secure ntopng cookies with SameSite and HttpOnly
- HTTP cookie authentication
- Improved random session id generation
- Various SNMP improvements
- Caching
- Interfaces status change alerts
- Device interfaces page
- Devices and interfaces added to flows
- Fixed several library memory leaks
- Improved device and interface charts
- Interfaces throughput calculation and visualization
- Ability to delete all SNMP devices at once
- Improved active devices discovery
- OS detection via HTTP User-Agent
- Alerts
- Crypto miners alerts toggle
- Detection and alerting of anomalous terminations
- Module for sending telegram.org alerts
- Slack
- Configurable Slack channel names
- Added Slack test button
- Charts
- Active flows vs local hosts chart
- Active flows vs interface traffic chart
- Ubuntu 18.04 support
- Support for ElasticSearch 6 export
- Added support for custom categories lists
- Added ability to use the non-JIT Lua interpreter
- Improved ntopng startup and shutdown time
- Support for capturing from interface pairs with PF_RING ZC
- Support for variable PPP header lenght
- Migrated geolocation to GeoLite2 and libmaxminddb
- Configuration backup and restore
- Improved IE browser support
- Using client SSL certificate for protocol detection
- Optimized host/flows purging
nEdge
- Netfilter queue fill level monitoring
- Bridging support with VLANs
- Added user members management page
- Added systemd service alias to ntopng
- Captive portal fixes
- Informative captive portal (no login)
- Improved captive portal support with WISPr XML
- Disabled global DNS forging by default
- Added netfilter stats RRDs
- Fixed bad MAC traffic increment
- Fixed slow shutdown/reboot
- Fixed invalid banned site redirection
- Fixed bad gateway status
- Fixed gateway network unreacheable when gateway is down
- Fixed SSL traffic not blocked when captive portal is active
- Fixed invalid read during local DNS lookup
- Workaround for dhclient bug stuck while a lease already exists
Fixes
- SNMP
- Fixed SNMP devices deletion
- Fixed format for odd SNMP interfaces speed
- Fixed SNMP community selection
- Fixed MDNS decoding
- Fixed login redirection
- Fixed MAC manufacturers escaping
- Fixed host validation errors
- Fixed traffic throughput burst when loading a serialized host
- Allowing multiple consecutive dots in password fields
- Reworked shutdown to allow graceful periodic activities termimation
- Fixed validation error in profiles with spaces in names
- Fixed old top talkers stats deletion
- Fixed 32-bit integers pushed to Lua
- Fixed service dependency from pfring
- Fixes for enabling broken SSL certificate mismatch alerts
- Fixed allowed interfaces users access
- Fixes for crashes on Windows
- Fixed lua platform dependent execution
- Fixed subnet search in hist data explorer
- Fixed flow devices and sflow mappings with SNMP
- Fixed invalid login page encoding
- LDAP fixes (overflow, invalid LDAP fields length)
- Fixed encoding for local/LDAP UTF-8 passwords
- Added POST timeout to prevent housekeeping from blocking indefinitely
- Windows resize fixes
- Fixed invalid uPnP URL
- Fixed wrong hosts retrv by pool id, OS, network, and country
- Fixed JS errors with IE browser
- Fixed custom categories matching