Introducing nDPI v3: Encrypted/Malware Traffic Analysis with Ease

Posted · Add Comment

Those who though that DPI died with the advent of traffic encryption should play with nDPI v3 that we’re introducing today. As already discussed, the pervasive use of encrypted traffic requires a new mindset when analysing network traffic. We decided to enhance nDPI adding the best traffic analysis techniques available today, in particular Cisco Joy, and facilities for calculating metrics such as entropy, standard deviation etc. that can be used to identify hidden traffic properties otherwise invisible. Thanks to all this, nDPI is now able to report if a SSH connection is an interactive connection or a file transfer, or if a TLS connection hides a malware activity. In essence we have tried to turn encryption from a problem to a new opportunity. This is for creating a baseline for developing traffic analysis applications without having to deal with low-level traffic details (nDPI will take care of them) and let developers focus on using the metrics rather than computing them.

Thanks to SSH and TLS fingerprint additions, you can now create security oriented applications in a a few lines of codes lightweight (read it as efficient/slim vs slow/fat) IDS/IPS applications using nDPI and PF_RING FT. You can now create your mini-Zeek or mini-Suricata over the week-end as we provide you all the necessary ingredients. In order to demonstrate the new nDPI capabilities  we have enhanced the ndpiReader application to export all the new data we compute, as well added the ability to export data in CSV so you can use your favourite data analysis libraries to analyse traffic data with new and richer metrics. For our Python users, Python bindings have been developed, so that they can start playing with DPI using their favourite language.

As people is using more and more nDPI for inline traffic analysis (i.e. read it as IPS or traffic block) we have changed the library design to report immediately the application protocol in case you just need to block (eg. via JA3 blacklisting)/prioritise traffic ASAP, or wait until all metadata has been dissected (see the new API call ndpi_extra_dissection_possible()) whenever you need full metadata visibility (for instance the TLS certificate fingerprint requires more than 10 packets usually as the certificate is exchanged after initial negotiation and this might be too late for an IPS, so that you now have the choice to do what you want).

This is all for v3. You can read more on the changelog below what we did in detail. However this is not yet the time for resting as we need to look forward. There are many things on the pipeline including for instance nDPI OVS integration. Stay tuned as the best is not yet here.

Enjoy!

Changelog

 

New Features

  • nDPI now reports the protocol ASAP even when specific fields have not yet been dissected because such packets have not yet been observed. This is important for inline applications that can immediately act on traffic. Applications that need full dissection need to call the new API function ndpi_extra_dissection_possible() to check if metadata dissection has been completely performed or if there is more to read before declaring it completed.
  • TLS (formerly identified as SSL in nDPI v2.x) is now dissected more deeply, certificate validity is extracted as well certificate SHA-1.
  • nDPIreader can now export data in CSV format with option -C
  • Implemented Sequence of Packet Length and Time (SPLT) and Byte Distribution (BD) as specified by Cisco Joy (https://github.com/cisco/joy). This allows malware activities on encrypted TLS streams. Read more at https://blogs.cisco.com/security/detecting-encrypted-malware-traffic-without-decryption
    • Available as library and in ndpiReader with option -J
  • Promoted usage of protocol categories rather than protocol identifiers in order to classify protocols. This allows application protocols to be clustered in families and thus better managed by users/developers rather than using hundred of protocols unknown to most of the people.
  • Added Inter-Arrival Time (IAT) calculation used to detect protocol misbehaviour (e.g. slow-DoS detection)
  • Added data analysis features for computign metrics such as entropy, average, stddev, variance on a single and consistent place that will prevent when possible. This should ease traffic analysis on monitoring/security applications. New API calls have been implemented such as ndpi_data_XXX() to handle these calculations.
  • Initial release of Python bindings available under nDPI/python.
  • Implemented search of human readable strings for promoting data exfiltration detection
    • Available as library and in ndpiReader with option -e
  • Fingerprints
  • Implemented a library to serialize/deserialize data in both Type-Length-Value (TLV) and JSON format
    • Used by nProbe/ntopng to exchange data via ZMQ

New Supported Protocols and Services

  • DTLS (i.e. TLS over UDP)
  • Hulu
  • TikTok/Musical.ly
  • WhatsApp Video
  • DNSoverHTTPS
  • Datasaver
  • Line protocol
  • Google Duo and Hangout merged
  • WireGuard VPN
  • IMO
  • Zoom.us

Improvements

  • TLS
    • Organizations
    • Ciphers
    • Certificate analysis
  • Added PUBLISH/SUBSCRIBE methods to SIP
  • Implemented STUN cache to enhance matching of STUN-based protocols
  • Dissection improvements
    • Viber
    • WhatsApp
    • AmazonVideo
    • SnapChat
    • FTP
    • QUIC
    • OpenVPN support for UDP-based VPNs
    • Facebook Messenger mobile
    • Various improvements for STUN, Hangout and Duo
  • Added new categories: CUSTOM_CATEGORY_ANTIMALWARE, NDPI_PROTOCOL_CATEGORY_MUSIC, NDPI_PROTOCOL_CATEGORY_VIDEO, NDPI_PROTOCOL_CATEGORY_SHOPPING, NDPI_PROTOCOL_CATEGORY_PRODUCTIVITY and NDPI_PROTOCOL_CATEGORY_FILE_SHARING
  • Added NDPI_PROTOCOL_DANGEROUS classification

Fixes

  • Fixed the dissection of certain invalid DNS responses
  • Fixed Spotify dissection
  • Fixed false positives with FTP and FTP_DATA
  • Fix to discard STUN over TCP flows
  • Fixed MySQL dissector
  • Fix category detection due to missing initialization
  • Fix DNS rsp_addr missing in some tiny responses
  • Various hardening fixes