Introducing nDPI 4.2: More Protocols and Robustness with -80% Memory

Posted · Add Comment

This is to announce the availability of nDPI 4.2 stable that brings several improvements and a reduced per-flow memory footprint (about -80% with respect to 4.0). We have continued to improve the DPI engine adding richer protocol metadata, as well as adding support for many platforms. The continuous integration toolchain along with fuzzy-testing allowed us to improve the overall library robustness and reliability which is a key feature when analyzing traffic, in particular for cybersecurity. In our vision, nDPI should be a traffic analysis layer sitting on top of packet capture toolkits such as PF_RING or DPDK that simplifies the design of applications that can finally focus on what rather than on how. This said nDPI is not just a toolkit for deep-packet inspection, but also a comprehensive toolkit for data analysis. You can hear more about this during this presentation at FOSDEM 2022.

Below you can find the complete changelog.

Enjoy !

 

nDPI 4.2 (Feb 2022)

New Features

  • Add a “confidence” field indicating the reliability of the classification
  • Add risk exceptions for services and domain names via ndpi_add_domain_risk_exceptions()
  • Add ability to report whether a protocol is encrypted

New Supported Protocols and Services

  • Add protocol detection for:
    • Badoo
    • Cassandra
    • EthernetIP

Improvements

  • Significantly reduced memory footprint from 2.94 KB to 688 B per flow
  • Improve protocol detection for:
    • BitTorrent
    • ICloud Private Relay
    • IMAP, POP3, SMTP
    • Log4J/Log4Shell
    • Microsoft Azure
    • Pandora TV
    • RTP
    • RTSP
    • Salesforce
    • STUN
    • Whatsapp
    • QUICv2
    • Zoom
  • Add flow risk:
    • NDPI_CLEAR_TEXT_CREDENTIALS
    • NDPI_POSSIBLE_EXPLOIT (Log4J)
    • NDPI_TLS_FATAL_ALERT
    • NDPI_TLS_CERTIFICATE_ABOUT_TO_EXPIRE
  • Update WhatsAPP and Instagram addresses
  • Update the list of default ports for QUIC
  • Update WindowsUpdate URLs
  • Add support for the .goog Google TLD
  • Add googletagmanager.com
  • Add bitmaps and API for handling compressed bitmaps
  • Add JA3 in risk exceptions
  • Add entropy calculation to check for suspicious (encrypted) payload
  • Add extraction of hostname in SMTP
  • Add RDP over UDP dissection
  • Add support for TLS over IPV6 in Subject Alt Names field
  • Improve JSON and CSV serialization
  • Improve IPv6 support for almost all dissectors
  • Improve CI and unit tests, add arm64, armhf and s390x as part of CI
  • Improve WHOIS detection, reduce false positives
  • Improve DGA detection for skipping potential DGAs of known/popular domain names
  • Improve user agent analysis
  • Reworked HTTP protocol dissection including HTTP proxy and HTTP connect

Changes

  • TLS obsolete protocol is set when TLS < 1.2 (used to be 1.1)
  • Numeric IPs are not considered for DGA checks
  • Differentiate between standard Amazon stuff (i.e market) and AWS
  • Remove Playstation VUE protocol
  • Remove pandora.tv from Pandora protocol
  • Remove outdated SoulSeek dissector

Fixes

  • Fix race conditions
  • Fix dissectors to be big-endian friendly
  • Fix heap overflow in realloc wrapper
  • Fix errors in Kerberos, TLS, H323, Netbios, CSGO, Bittorrent
  • Fix wrong tuple comparison
  • Fix ndpi_serialize_string_int64
  • Fix Grease values parsing
  • Fix certificate mismatch check
  • Fix null-dereference read for Zattoo with IPv6
  • Fix dissectors initialization for XBox, Diameter
  • Fix confidence for STUN classifications
  • Fix FreeBSD support
  • Fix old GQUIC versions on big-endian machines
  • Fix aho-corasick on big-endian machines
  • Fix DGA false positive
  • Fix integer overflow for QUIC
  • Fix HTTP false positives
  • Fix SonarCloud-CI support
  • Fix clashes setting the hostname on similar protocols (FTP, SMTP)
  • Fix some invalid TLS guesses
  • Fix crash on ARM (Raspberry)
  • Fix DNS (including fragmented DNS) dissection
  • Fix parsing of IPv6 packets with extension headers
  • Fix extraction of Realm attribute in STUN
  • Fix support for START-TLS sessions in FTP
  • Fix TCP retransmissions for multiple dissectors
  • Fix DES initialisation
  • Fix Git protocol dissection
  • Fix certificate mismatch for TLS flows with no client hello observed
  • Fix old versions of GQUIC on big-endian machines

Misc

  • Add tool for generating automatically the Azure IP list