Introducing nProbe 9.2: Collection Pass-Through and Reforge, OpenWRT support, Flexible JSON-export

Posted · Add Comment

This is to announce the release of nProbe 9.2. The main new features of this release are focused on flow collection speed and flexibility in particular for modern JSON-based flow consumers. This is to enable applications relying on nProbe, e.g. ntopng, to scale up when collecting flows:

  • The new –collector-passthrough option allows the flow cache to be bypassed when flows are collected. This mean that flows are forwarded to remote collectors unmodified (i.e. -T is not used) without placing them into the flow cache (i.e. flows are not merged by nProbe but forwarded unmodified) for maximum speed. In our tests this new feature allows flow collection and forwarding to be greatly enhanced (~ 5x speedup) with flows begin collected and exported at about 100k flows/sec per instance (and of course you can start multiple nProbe instances).
  • The new –collector-nf-reforge allows incoming flows to be filtered according to the NetFlow/IPFIX interfaceId and reforged in terms of collector IP address. Thanks to this new option, it is possible to reconcile flows with real IP sender in case of NAT, or ignore flows created on network interfaces that instead need to be discarded.
  • Better template handling when collecting from multiple routers: you can now simultaneously collect flows, per nProbe instance, coming from 128+ routers. This is very much needed when collecting from many small IoT devices sending flows to the same nProbe for conversion.

This new release also

  • Supports the latest nDPI and thus it is possible to interpret flows and export the nDPI flow risk value that interprets flow information and reports security-oriented information that is very valuable for identifying cybersecurity issues.
  • Greatly enhanced GTP support and VoLTE support.
  • We have improved OpenWRT support and optimised the code for running on embedded environments such as the new Nokia Beacon 6 home Wi-Fi.

Below you can find the complete changelog.

Enjoy!

ChangeLog

New Features and Command Line Options

  • Added Kafka and Syslog export when –collector-passthrough is used
  • Changed -p format to <Outer VLAN Id>.<Inner VLAN Id/<proto>/<IP>/<port>/<TOS>/<SCTP StreamId>/<exporter IP>
  • Added the ability to specify a binding IPv4 address in collector mode (e.g. -3 127.0.0.1:1234)
  • Implemented –collector-nf-reforge for filtering and reforging collected netflow flows
  • Flow cache is now disabled by default in collection mode: replaced –disable-cache with –enable-collection-cache
  • Added –gtpv1-track-non-gtp-u-traffic and –gtpv2-track-non-gtp-u-traffic for non GTP-encapsulated user export in IE %FLOW_USER_NAME

Extensions

  • Added the ability to sniff from stdin by using -i –
  • Added %L7_PROTO_RISK %L7_PROTO_RISK_NAME
  • Added %TCP_WIN_MAX_IN %TCP_WIN_MAX_OUT IEs to @NTOPNG@
  • Added DNS/HTTP IEs to @NTOPNG@ in probe mode
  • Added collected flow lifetime export via ZMQ
  • Added IP-in-IP (IPv4 encapsulated in IPv6) support
  • Improved DNS plugin with additional records and NAPTR query type
  • Exporting %SEQ_PLEN as 8 bit element
  • Added TOS export via ZMQ
  • GTP traffic analysis improvements
  • Improved IMSI/APN traffic accounting and aggregation when using –imsi-apn-aggregation
  • Support for SIP over TCP (VoLTE)
  • Added IPv6 support in GTPv1
  • Added IPv4+IPv6 GTP-C v2 dissection
  • Improvement on GTP-C v1 dissection
  • Added support for %BGP_PREV_ADJACENT_ASN %BGP_NEXT_ADJACENT_ASN when collecting sFlow and Netflow
  • Added IPv6 PAA export
  • Support for overwriting element names with aliases provided by the user (case sensitive)

Bug Fixes

  • Fixed detection of multiple connections on the same port (RST) exporting multiple flows
  • Fixed EXPORTER_IPV6_ADDRESS
  • Fixed UNTUNNELED_IPV6_SRC_ADDR / UNTUNNELED_IPV6_DST_ADDR
  • Fixed dump of IPv6 flows to MySQL
  • Fixed shutdown crashes
  • Fixed kafka stats number overflow
  • Fixed multiple –collection-filter options
  • Fixed accounting of bidirectional flows in stats
  • Fixed export of empty data
  • Fixed invalid flow idle computation
  • Fixed CSV export (always print all columns)
  • Fixed AS lookup/calculation support for .mmdb files part of the ntopng-data package
  • Fixed bug that caused FLOW_USER_NAME to be empty
  • Fixed custom template elements support
  • Fixed SIP decoding with malformed packets
  • Fixed IPv6 dissection when encapsulated in GTP
  • Fixed application protocol detection with GTP
  • Fixed GTPv1 GTPV1_END_USER_IP field
  • Fixed drop count

Miscellaneous

  • Moved all binaries and libraries from /usr/local/ to /usr/
  • Plugins are now loaded from ./plugins, /usr/lib/nprobe/plugins, /usr/local/lib/nprobe/plugins
  • Added Ubuntu 20.04 support
  • Improved OpenWRT support
  • Windows fixes
  • Improved plugins SDK