Welcome to ntopng 5.4: Enhanced Traffic Analysis and Cybersecurity

Posted · Add Comment

The previous stable release introduced a new persistency layer based on ClickHouse, paving the way for a more flexible yet fast historical data analysis, with its ability to store billion of records (alerts and flows) with limited disk space and very low query time. This new 5.4 release introduces many enhancements in the historical data analysis with more comprehensive information and additional analysis pages to provide clear insights about Network issues. In order to further easy the analysis, the search bar has also been reworked, to let you find what you are looking for at a glance.

Behavioural analysis has also been extended, with the new Similarity and Centrality maps, and by taking further advantage of the cyber score. New traffic checks and flow risks have been added as well, to cover more use cases and detect additional Network threats and security issues. ntopng now delivers more control on traffic checks, by providing additional ways of configuring exceptions and avoiding falso positives to let you focus on the events requiring real attention.

This release also takes advantage of the new nProbe 10 agent mode to monitor processes on Linux and Windows and provide visibility at the process and user level, merging Network with System visibility.

For the full list of changes please take a look at the changelog below.

Enjoy !

Breakthroughs

  • New search bar, with more results, information, links
  • New listening ports page when collecting process information from nProbe (agent mode)
  • New support for ELK version 8 and standardized ELK export format
  • New packages for Ubuntu 22.04
  • New Centrality Map in service map
  • New Similarity Map
  • Major performance improvements for periodic scripts
  • New alert exclusion management (for checks and nDPI flow risks)
  • Introduce Vue.js in the frontend
  • Expose Chart Vue components for external websites

Improvements

  • Add new alerts (DHCP Storm, DNS Fragmented, Scan Detection, …)
  • Add Top Dropdown menu (Top Clients, Top Servers, …) to the alert explorer
  • Add ability to set historical flow permission to users
  • Rework and Improve Maps (Service/Periodicity/Host)
  • Improve buttons look and feel using latest Bootstrap version
  • Improve Historical Flow and Alerts information (add many new fields for better analysis)
  • Improve IEC support (e.g. iec_invalid_transition)
  • Add various mapping (DNS answers, DNS query types, ICMP answers, …)
  • Improve documentation, added all the available checks description
  • Improve Exporter IP Flow Layout
  • Improve ClickHouse queries performance with a better use of indexes
  • Improve ZMQ flow idle timeout handling
  • Updated ECS to 8.1 version
  • Add various SNMP checks
  • Add npm and Webpack support
  • Add new alert exclusions fields (Domain and IssuerDN)
  • Add DGA domain handling received via ZMQ
  • Add Network matrix for view interfaces
  • Add VLAN field support to alert exclusions
  • Add Top Sites for flows collected from nProbe
  • Add ELK dump frequency to Settings
  • Implement Network/FQDN exclusion for alerts
  • Add ‘dpi’ and ‘guessed’ badge to flow list and details
  • Add support for L7 confidence
  • Add ClickHouse search in JSON fields
  • Add filters to Service/Periodicity maps
  • Add –offline option to force offline mode in case of limited connectivity
  • Add support for Active Monitoring selection in recipients
  • Add copy button for all external link
  • Allow download of PCAP in Historical Flows Explorer
  • Add Flow Exporter to view interfaces
  • Add ECS support to ELK flow dump
  • Add MAC Address to View Interfaces
  • Add Similarity check

Changes

  • Remove Telemetry
  • Move UDP unidirection to nDPI alerts
  • Disable flow dump to syslog on MacOS due to broken openlog API on Sierra and later
  • Rework MAC/IP Reassociation alert used to detect spoofing and MITM (Man In The Middle) Attacks
  • Separate data retention into Flow/Alerts data retention and Timeseries/Top data retention
  • Reduce number of (unnecessary) threads

nEdge

  • Add alert when a Gateway is unreachable
  • Improve the Captive Portal

Fixes

  • Fix cookie attributes to the user and password cookies on the 302 redirect response
  • Fix various GUI incorrect/undefined names
  • Fix datatables incorrect data visualization
  • Fix RRD timeseries implementation
  • Fix log spam in case of endpoint not working
  • Fix modals not hiding
  • Fix alert/historical page filters not working correctly
  • Fix bugs with flows informations while using View Interface
  • Fix time format, shown as local instead of server time in some pages
  • Fix format validations not correctly working
  • Fix nProbe template flow mapping
  • Fix access to uninitialized obj leading to segfault
  • Fix idle time too low
  • Fix invalid risk set from nDPI to ntopng’s Flow class
  • Fix dns large packets alert incorrectly triggered
  • Fix network discovery
  • Fix CSV download
  • Fix bug that prevented flows to be dumped on ClickHouse
  • Fix external URLs not correctly working
  • Fix database initialization
  • Fix IEC continuous dissection
  • Fix NetBIOS name should not be used for hostnames
  • Fix various CSS bugs
  • Fix filter operators
  • Fix name lookup
  • Fix for detecting ZMQ drops
  • Fix Historical Filters lost when switching windows
  • Fix traffic directions with mirrored traffic
  • Fix various API not correctly working
  • Fix range picker not correctly working
  • Fix crash when using interfaces with no database
  • Fix various nil description
  • Fix SIGABRT on shutdown with Views
  • Fix for SNMP bridge alerting
  • Fix external links not working
  • Fix flow drilldown not correctly working