Towards Traffic Behaviour Analysis: Introducing nDPI 3.2

Posted · Add Comment

This is to announce the new stable release of nDPI 3.2. The main trend of nDPI is to move from “simple” application protocol detection towards behavioral traffic interpretation. This has been implemented with the integration of modules for detecting attacks (e.g. SQL injections and XSS in HTTP) and behavioral indications on packet length/time/entropy as well indicators used for creating simple indicators typical of IDS systems. In essence nDPI is moving from protocol reporting to comprehensive traffic interpretation.

nDPI now includes functions for efficiently serialising data in both JSON and binary format, that are used for instance by ntopng/nprobe to communicate more efficiently than before using less bandwidth and being faster.

In addition thanks to the work of Philippe Antoine, nDPI is now continuously analysed by ClusterFuzz for searching memory leaks, invalid memory accesses all based on libfuzz. In essence nDPI is a better product now.

Enjoy!

This is the complete changelog:

  • New Features
    • New API calls
      • Protocol detection: ndpi_is_protocol_detected
      • Categories: ndpi_load_categories_file / ndpi_load_category
      • JSON/TLV serialization: ndpi_serialize_string_boolean / ndpi_serialize_uint32_boolean
      • Patricia tree: ndpi_load_ipv4_ptree
      • Module initialization: ndpi_init_detection_module / ndpi_finalize_initalization
      • Base64 encoding: ndpi_base64_encode
      • JSON export: ndpi_flow2json
      • Print protocol: ndpi_get_l4_proto_name / ndpi_get_l4_proto_info
    • Libfuzz integration
    • Implemented Community ID hash (API call ndpi_flowv6_flow_hash and ndpi_flowv4_flow_hash)
    • Detection of RCE in HTTP GET requests via PCRE
    • Integration of the libinjection library to detect SQL injections and XSS type attacks in HTTP requests
  • New Supported Protocols and Services
    • TLS: new decoder
    • Added ALPN support
    • Added export of supported version in TLS header
    • Added Telnet dissector with metadata extraction
    • Added Zabbix dissector
    • Added POP3/IMAP metadata extraction
    • Added FTP user/password extraction
    • Added NetBIOS metadata extraction
    • Added Kerberos metadata extraction
    • Implemented SQL Injection and XSS attack detection
    • Host-based detection improvements and changes
    • Added Microsoft range
    • Added twitch.tv website
    • Added brasilbandalarga.com.br and .eaqbr.com.br as EAQ
    • Added 20.180.0.0/14, 20.184.0.0/13 range as Skype
    • Added 52.84.0.0/14 range as Amazon
    • Added ^pastebin.com
    • Changed 13.64.0.0/11 range from Skype to Microsoft
    • Refreshed Whatsapp server list, added *whatsapp-*.fbcdn.net IPs
    • Added public DNSoverHTTPS servers
  • Improvements
    • Reworked and improved the TLS dissector
    • Reworked Kerberos dissector
    • Improved DNS response decoding
    • Support for DNS continuous flow dissection
    • Improved Python bindings
    • Improved Ethereum support
    • Improved categories detection with streaming and HTTP
    • Support for IP-based detection to compute the application protocol
    • Renamed protocol 104 to IEC60870 (more meaningful)
    • Added failed authentication support with FTP
    • Renamed DNSoverHTTPS to handle bot DoH and DoT
    • Implemented stacked DPI decoding
    • Improvements for CapWAP and Bloomberg
    • Improved SMB dissection
    • Improved SSH dissection
    • Added capwap support
    • Modified API signatures for ndpi_ssl_version2str / ndpi_detection_giveup
    • Removed ndpi_pref_http_dont_dissect_response / ndpi_pref_dns_dont_dissect_response (replaced by ndpi_extra_dissection_possible)
  •  Fixes
    • Fixed memory invalid access in SMTP and leaks in TLS
    • Fixed a few memory leaks
    • Fixed invalid memory access in a few protocol dissectors (HTTP, memcached, Citrix, STUN, DNS, Amazon Video, TLS, Viber)
    • Fixed IPv6 address format across the various platforms/distributions
    • Fixed infinite loop in ndpi_workflow_process_packet
    • Fixed SHA1 certificate detection
    • Fixed custom protocol detection
    • Fixed SMTP dissection (including email)
    • Fixed Telnet dissection and invalid password report
    • Fixed invalid category matching in HTTP
    • Fixed Skype and STUN false positives
    • Fixed SQL Injection detection
    • Fixed invalid SMBv1 detection
    • Fixed SSH dissection
    • Fixed ndpi_ssl_version2str
    • Fixed ndpi_extra_dissection_possible
    • Fixed out of bounds read in ndpi_match_custom_category
  • Misc
    • ndpiReader
      • CSV output enhancements
      • Added tunnelling decapsulation
      • Improved HTTP reporting
      • Added scan and HTTP attacks (XSS, SQL Injection) detection