Introducing ntopng 5.6: New Reports and Cybersecurity Indicators, Kafka, Lua/Python API, Flow Collection Clustering

Posted · Add Comment

This is to announce the availability of ntopng 5.6 stable release that brings several additions and improvements:

  • We have started to introduce responsiveness in ntopng GUI by means of VueJS. All timeseries and historical pages are now rewritten to take advantage of modern web technologies. You can now compare timeseries across hosts, devices, or anything that is a timeseries created by ntopng.
  • In addition to the traditional/efficient C++ alerting subsystems, we have introduced a Lua API for developing new checks in seconds. This is a simple way to quickly prototype custom checks that can eventually be converted in C++ or stay in Lua as the overall performance is very good as we have coded micro-calls.
  • We have introduced a new Python API for extracting data from ntopng and using it as a live data lake. Please check out our examples including live PDF reports generated using ntopng live/historical data. Please attend this session at FOSDEM this week-end for details on Lua and Python APIs.
  • Cybersecurity features, most of which leveraging on the new nDPI 4.6, have been extended with new flow risks and checks.
  • ntopng can now speak Kafka when receiving data from nProbe, and export flows to Kafka consumers.
  • We have improved the application performance by simplifying code and recoding selected components more efficiently.
  • We have made various packaging changes in the OPNsense build due to changes in the latest version of the popular security platform.
  • Support of ClickHouse cluster for scaling up in distributed and large deployments.
  • Historical reports have been improved both in features and look.
  • Fully multitenancy support, including historical data.
  • Live pcap analysis (without ntopng restart) for using ntopng in traffic analysis.
  • We have introduced several new reports on both alerts and live data some of which shown below.
  • nEdge (ntopng inline) finally supports VLAN setup and integration.

In essence with this release we have made various changes that will enable us to plan for great new features in the next release. We will soon announce a webinar that describes all new features in detail. Stay tuned.

This slideshow requires JavaScript.

Below you can find the complete changelog.

Enjoy !

 

 

ntopng 5.6 Changelog

Breakthroughs

  • Python API
  • Add support Rocky9
  • Add support to Kafka
  • Increased max num of exporters
  • Introduce nTap support
  • Introduce support to ClickHouse Cluster
  • Rework Historical Chart Page
  • Rework pages using VueJS and moving towards responsive client
  • Add XL license

Improvements

  • Handle allowed networks for unprivileged users
  • Improve multitenancy support
  • Improve thread names
  • Improve mac formatting
  • Improve top host sites adding reset method
  • Improve pcap upload
  • Improve ports formatting
  • Improve handling for Cisco NBAR collection
  • Improve source style
  • Improve Linux OS detection
  • Improve Engaged Time Report in Chart
  • Improve passive DNS host resolution
  • Improve alerts reports
  • Improve OPNsense installation instruction
  • Improve host report
  • Improve support to NDPI_TCP_ISSUES flow risk
  • Improve layout
  • Improve ICMP flow handling
  • Lowered memory consumption due to alert score
  • Rework pro code directories
  • Rework lua code
  • Rework flow aggregation
  • Rework capabilities support
  • Socket code cleanup
  • Use API to build interface report
  • Update rrd calculations
  • Update JP localization (courtesy of Yoshihiro Ishikawa)

Changes

  • Add logo to package
  • Add missing deps
  • Add link to host
  • Add options to send report by email
  • Add Report class and example
  • Add internal server error on health/interfaces doc api
  • Add support for external (REST) host alerts
  • Add various help and parameters
  • Add script to create a pdf report from historical API data
  • Add NXLOG/Active Directory documentation
  • Add reload button in various pages
  • Add third party resources
  • Add flow exporter ips to observation points
  • Add support for the python API documentation
  • Add forced offline variable to mantain the –offline option
  • Add support for Lua host engaged alerts using timeout
  • Add observation points ts
  • Add HTTP server in flow details
  • Add token-based authentication
  • Add Flow Risk (Bitmap) Filter in alerts
  • Add make targets for pip package Updated package classes
  • Add L7 information in flow object adding
  • Add CodeQL workflow for GitHub code scanning
  • Add modal-download-file component and add export timeseries png picture button
  • Add critical and emergency status to alerts
  • Add oneway TCP flows counters
  • Add support for nDPI network handling in flows
  • Add -n 4 for name resolution
  • Add IMAP/POP stats
  • Add Stratosphere Labs Blacklist support
  • Add support d3v7
  • Add Requires for RH9 (redhat-lsb-core is deprecated)
  • Add interfaces stats api and refactor the others health api
  • Add support to application protocol and master protocol
  • Add CIDR support in Historical Flows
  • Add new Aggregated Flows page
  • Add new Alerts Analysis page
  • Add support for estimating the number of TCP contacted servers with no reply
  • Add new Ports Analysis page
  • Add detection of periodic flows and exported it as flow risk in both flows and alerts
  • Add REST API to get DB columns and info
  • Add ability to query alerts from Python
  • Add Zoom streams handling
  • Add various checks
  • Add IP-in-IP decapsulation
  • Add Host Rules page (possibility to trigger alerts based on timeseries)
  • Add the ability to analyze a pcap without creating a new interface
  • Add Windows timezone handling
  • Change table definition
  • Cleanup file names
  • Disabled host serialization
  • Enlarged the number of local networks to 1024
  • Increased upload size to 25 MB
  • Implement custom script check
  • Implement support of host filtering with TX traffic sent
  • Implement unresponsive peers host report
  • Implement count of incoming tx peers with TCP flows unanswered
  • Move ts business logic in ts_rest_utils.lua
  • Patch for handling nicely clock drift at startup
  • Remove obsolete autogen commands On Linux stay with g++ unless a sanitizer is used
  • Remove REST API v0 (discontinued since ntopng 4.2)
  • Remove no more used severity
  • Refactor range-picker query_presets
  • Rework host packets page and removed dscp page
  • Rework host ports implementation
  • Rework Historical class
  • Rework OPNsense plugin package build
  • Self test fixes and improvements
  • Update documentation
  • Update REST API
  • Update bootstrap table css
  • Update various pages to vuejs
  • Update counter scaling (no gauge)
  • Update response in service disabled case

nEdge

  • Add support to multi LAN and fixes DHCP service error
  • Add VLAN and multi WAN support to nedge
  • Add routing_policy to nedge configuration callback
  • Fix netplan configuration error
  • Update VLAN trunk doc

Fix

  • Df columns error management, table export formatted with % and column reordering now working
  • Fix missing openssl dependency from MacOS
  • Fix clang
  • Fix host sankey minor issues
  • Fix hyperlinks to historical charts not working
  • Fix hyperlinks not working correctly
  • Fix Regex escape
  • Fix application name resolution on aggregated views
  • Fix RRD driver for step calaculation
  • Fix visual bugs with master and app proto
  • Fix various interface page minor bugs
  • Fix shortened labels
  • Fix default sort not working
  • Fix influxdb retention not updated
  • Fix name and size of charts
  • Fix vlan label not mapped
  • Fix for FreeBSD configure
  • Fix ip resolution not updating the name
  • Fix discrepancy in Traffic Calculation (Interface Chart)
  • Fix measurement units not uniform
  • Fix crash swap
  • Fix bug that reported wrong DNS information
  • Fix build process with opnsense/plugins
  • Fix validators regexps
  • Fix ICMP emtropy report Improved HTTP flows report
  • Fix Telegram Reported alerts contain HTML
  • Fix multi-series Charts are Unreadable in Dark Mode
  • Fix invalid reverse host resolution that caused hosts to be labelled with wrong symbolic name
  • Fix delete obsoleted code from page-stats
  • Fix for circular dependency js
  • Fix overlay not working
  • Fix due to changes to nDPI ALPN handling
  • Fix CSS Inconsistency Across Browsers
  • Fix Deep copy also for array of objects
  • Fix missing modules
  • Fix NAT handling with nprobe
  • Fix initialization crash
  • Removed multiple load from tables
  • ZMQ encryption key is now reported in hex to avoid escape problems