Introducing nProbe 8.6: Per-Second Measurements and Collection of Proprietary Flows

Posted · Add Comment

We are glad to announce the release of nProbe 8.6 stable release. Among the main new features, this release brings:

  • Per-second measurements of flows traffic
  • Ability to collect proprietary (i.e. using non standard information elements) flows

These new features come along with a wide range of new extensions and improvements to the currently existing features and, least but not last, security and stability fixes.

Let’s have a brief look at the two main new features mentioned above.

Per-second Traffic Measurements

Getting cumulative measurements with respect to the flow lifetime not always provide enough information to really understand certain traffic patterns. Traditional flow-monitoring technologies such a la NetFlow, that just report cumulative measurements at the end of the flow (i.e. average traffic values), create a blind spot across the whole flow lifetime. Receiving a 2 GB flow with a 2-minute lifetime doesn’t tell anything about the actual pattern of the traffic. Have those 2 GB been sent in a bunch of seconds right before the end of the flow? Or have them been sent at a constant rate for 2 minutes?

Well, this release of nProbe offers extra visibility into the traffic by providing second-by-second flow byte counters. This means that you’ll able to get, for every monitored flow, a timeseries with a point every second telling the exact volume of traffic done by that flow.

Cool, isn’t it? Check out this blog post for a detailed description of this feature.

Collection of Proprietary Flows

Until the previous release, nProbe was able to collect some selected proprietary information elements in addition to the standard NetFlow and IPFIX ones. As our user community has demanded us to add further support to proprietary fields, we have decided to change the nProbe engine in order to be open to extensions by means of a configuration file instead of modifying every time the application engine. As specified in the nProbe user’s guide that covers al details, nProbe is now able to collect proprietary flows from selected manufactures including:

  • Alcatel-Lucent
  • Cisco
  • Gigamon
  • Ixia
  • Palo Alto
  • Procera
  • SonicWall

just defining the proprietary fields on a text file. You can refer to this page, for details about the above manufacturers configuration files.

Changelog

The complete list of changes, including enhancements and fixes, is available below

Main New Features

  • Added second-by-second client-to-server and server-to-client flow bytes
    • https://www.ntop.org/nprobe/introducing-per-second-measurements-in-nprobe-flow-exports/
  • Implemented an embedded web server that can be optionally enabled to
    • Force a flush of all the active flows
    • Receive a response monitored traffic traffic statistics
    • Query the nProbe version
  • Seamless support of ElasticSearch 5 and 6 and automatic template push
    • ElasticSearch version is automatically determined upon nProbe startup
    • A proper template is pushed to ElasticSearch on the basis of its version
  • Implemented modbus plugin

Extensions

  • Added support for the collection of NetFlow/IPFIX vendor-proprietary information elements through simple configuration files
  • Supported vendors include Sonicwall, Cisco, IXIA, an others
  • Configuration files published at https://github.com/ntop/nProbe/tree/master/custom_fields
  • The default NetFlow version is now V9
  • Plugins are disabled in collector mode
  • Improved support for Ubuntu18
  • Implements SIP user agents dissections (Client and Server)
  • Implements TCP/UDP/Other min flow size to discard flows below a certain minimum size
  • nProbe runs with user ‘nprobe’ by default, falling back to nobody if user ‘nprobe’ does not exist
  • New NetFlow information elements %NAT_ORIGINATING_ADDRESS_REALM and %NAT_EVENT
  • L7_PROTO exports now the protocol in format <master>.<application> protocol
  • Added fields %SRC_TO_DST_SECOND_BYTES and %DST_TO_SRC_SECOND_BYTES to export second-by-second flow bytes
  • Migrates geolocation support to GeoLite2 and libmaxminddb
  • Migration of nProbe manual to HTML
  • Manual available at https://www.ntop.org/guides/nProbe/

New Options

  • --http-server to enable the embedded web server
  • --help-neflow to dump a long help including plugin and template information

Fixes

  • Checks for hardening comparisons with partial strings
  • Further Checks to avoid crossing certain memory bundaries
  • Checks to avoid loops with malformed sctp packets
  • Fixes for flow start/end times and timestamp calculation in proxy mode
  • Fixes issues with SIP call id in RTP flows
  • Fixes length calculation in IPFIX variable-length fields
  • Fixed ZMQ buffer termination when flushing ZMQ buffers
  • Fixed wrong %EXPORTER_IPV4_ADDRESS exported over ZMQ export in case on Netflow != v5
  • Fixed a race condition that was preventing all flows to be dumped on file
  • Fix to avoid dumped files to be overwritten when -P is used with -F < 60
  • Adds missing librdkafka support on Centos7