nProbe™

An Extensible NetFlow v5/v9/IPFIX Probe for IPv4/v6


In commercial environments, NetFlow is probably the de-facto standard for network traffic accounting. nProbe includes both a NetFlow v5/v9/IPFIX probe and collector that can be used to play with NetFlow flows. This means nProbe™ can be used:

  • To collect and export NetFlow flows generated by border gateways/switches/routers or any other device that can export in NetFlow v5/v9
  • As a drop-in replacement of embedded, low-speed, NetFlow probes that may already been deployed
  • To analyze multi-Gbit networks at full speed with no (or very moderate) packet loss
  • To send monitored flows towards a collector such as the open-source ntopng or a commercial one (e.g. Cisco NetFlow Collector or Plixer)

Currently nProbe™ is a software application available stand-alone or as an embedded system named nBox .

Main nProbe™ Features

  • Available for Linux, Windows, and embedded environments ARM and MIPS/MIPSEL.
  • Layer-7 application visibility (250+ applications including Skype, BitTorrent and Citrix).
  • Layer-7 application propagation in exported flows to enable accurate accounting.
  • NetFlow v5/v9/IPFIX support for efficient flow handling.
  • Cisco NetFlow-Lite support.
  • Full IPFIX support: PEN (Private Enterprise Numbers) and Variable length encoding.
  • Complete support for IPv4 and IPv6.
  • Limited memory footprint (less that 2 MB of memory regardless of the network size) and CPU savvy.
  • Ability to natively export flows to Apache™ , Syslog, MySQL/MariaDB, Splunk (via TCP streaming).
  • Ability to natively export flows to Kafka and ElasticSearch (using the Export Plugin).
  • Ability to dump flows in format ready for import in columnar databases.
  • Native support for technologies PF_RING and the newest kernel-bypass PF_RING Zero Copy (ZC) for ultra-high speed packet capture.
  • Ability to act as flow collector and proxy. All combinations are supported.
  • Ability to collect sFlow flows and transparently translate them into NetFlow v5/v9/IPFIX.
  • Ability to forge NetFlow interface identificators based on MAC/IP addresses.
  • Collection of Cisco ASA flows and conversion into NetFlow v5/v9/IPFIX.
  • Multi-threaded architecture for the exploitation of multi-processor, multi-core elaboration systems.
  • Support of tunneled (including GRE, PPP and GTP) traffic and ability to export inner/outer envelope/packet information.
  • Support of both flow and packet sampling.
  • Support of Flexible Netflow for the creation of custom NetFlow templates, with optional PEN support.
  • VoIP (SIP and RTP) traffic analysis including voice quality and (pseudo-)MOS.
  • HTTP, MySQL/Oracle, DNS protocol analysis: ability to generate logs of web, MySQL/Oracle and DNS activities in addition to flow export.
  • BGP Plugin for establishing a BGP session with a router and generate flows with AS and AS path information.
  • Plugin architecture for easy extensibility via custom V9/IPFIX tags.
  • Fully interoperable with commercial collectors such as IsarFlow, Fluke, Cisco, Dartware, Arbor Networks, Plixer, NetFlow Auditor, SolarWinds Orion NTA, Andrisoft.
  • Designed for running on environments with limited resources (the nProbe™ binary < 100 Kb) and embedded systems (e.g. ARM- and  MIPSEL-based appliances).
  • It can be used to build cheap NetFlow probes using commodity hardware.
  • Ability to save flows on disk for later analysis or integration into an existing monitoring application.
  • Fully user-configurable.
  • High-performance probe: commercial probes included those embedded on routers and switches are often not able to keep up with high-speeds.
  • Can be used in conjunction with ntopng to visualize and analyze monitored traffic.

Using nProbe™

The current nProbe™ version is much more that a simple netflow probe.

Probe mode

nprobe -i eth0 --collector 127.0.0.1:2055

 

nprobe_probe

Collector mode

nprobe --collector-port 2055

nprobe_flow_collector

Proxy mode

nprobe --collector-port 2055 --collector 127.0.0.1:2055 -V 9
 nprobe_proxy

 

It can be a probe, probe+collector, collector, or a proxy. In proxy mode it is possible to convert from/to IPFIX/NetFlow v5/v9 in order to smoothly upgrade to newer NetFlow protocol versions while capitalizing on previous protocol versions. So you can for instance convert flows coming from your v5 router into IPFIX and vice-versa. Note that with some combinations (e.g. from v9 to v5) you might lose some flow information.

Performance

Many people are aware that not all the available NetFlow probes are scalable. nProbe™ has been designed to keep up with multi-Gbit speeds on commodity hardware. Using a dual core CPU, nProbe™ can be used for capturing packets at 1 Gbit with no/very little (< 1%) packet loss using vanilla PF_RING (no ZC). With the PF_RING ZC kernel-bypass technologies packet capture is eve faster as is it possible to read below. Please note that performance figures are per-core. This means that, for example, by leveraging on PF_RING ZC, it is possible to achieve a 4x performance improvement on a quad-core CPU if compared to a single-core one.

Packet Size (Bytes) Per-core nProbe™ Sustained Throughtput with no packet loss
PF_RING ZC
fixed 64 3.32 Mpps, 2.15 Gb/secs
fixed 512 Wire rate
fixed 1500
random 64-1500

The table above shows the result of a worst-case performance test using

  • nProbe™ 7.2 Pro/Plugins (native PF_RING support)
  • Ubuntu Linux 14.10
  • PF_RING 6.1.X
  • Dell R220
  • CPU Intel E3-1241 v3 @ 3.50GHz
  • Intel 82599-based 10 Gbit card
  • Traffic Generator: pfsend -i zc:ethX -a -g 1 -b 250000
  • 250K rotating IP addresses
  • Generation of 250K flows/minute
  • Command used: nprobe -i zc:eth1 –cpu-affinity 1 -t 60 -b 1 -w 500000 -V 9
  • No flow storage on DB or disk, just forwarding to a collector

For the latest news about nProbe, please read the ntop blog.

Usage

nProbe™ is distributed in binary format. Once installed, nProbe™ is ready be used and does not require any additional configuration. In order to function properly in probe mode, nProbe™ needs to see/capture the traffic of interest. For this reason, in case of switched networks, it is necessary to either mirror traffic (VLAN or port mirror) or place the probe in a location (e.g. by the border gateway) that is traversed by the most part of the traffic. Under normal operating conditions nProbe™ will collect traffic data and emit NetFlow v5/v9/IPFIX flows towards the specified collector. Any standard NetFlow collector can be used to analyze the flows generated by nProbe™ — although not all the commercial collectors support v9. nProbe™ can also be used in conjunction with ntopng. In the latter case an optimized, optionally compressed and encrypted format will be used for data exchange, leading to a lightweight monitoring architecture that decouples the monitoring part from the visualization and analysis part.

FAQ

  1. Q: Do your release nProbe™ source code?
    A: We have decided not to release the source to everyone as in the past some people made some buses. Requests will be evaluated on a case-by-case basis
  2. Q: Is nProbe™ able to operate on Gbit networks at full speed?
    A: Yes. Note that for exploiting the Gbit packet capture you need a 64-bit PCI Gigabit Ethernet interface
  3. Q: Can I redistribute the nProbe source or build a derivative product?
    A: Source code cannot be redistributed. If you plan to build a product or sell nProbe-based solutions you need to contact us.
  4. Q: What do you do with the money you get charging for nProbe™?
    A: This money is invested for doing research and product development.

Documentation

nProbe User’s Guide

Credits

NetFlow is copyright by Cisco Systems.
nProbe™ is a trademark registered in the USA and the European Union.

nProbe Plugins

nProbe™ is extensible by means of optional plugins, supported only by the nProbe Pro version. Below you can find the list of currently available plugins all available in binary format only.

Available Plugins:

  • HTTP
    Decode HTTP traffic and HTTPS certificates. It can generate a comprehensive log of HTTP traffic, including page download and network/server delay.
  • DHCP
    Decode DHCP traffic and export DHCP information in flows or file dump.
  • Export
    Export to ElasticSearchPlugin that can natively export flow information into ElasticSearch without third party converters such as Logstash.
  • DNS
    Decodes DNS traffic, and produce a log of main domain name resolution activities. Microcloud friendly. Available only in binary format.
  • flow-to-MySQL
    Dumps exported flows into a MySQL database. This plugin is part of nProbe Pro and it does not require a license.
  • MySQL
    Decodes (unencrypted) MySQL traffic, and produce a log of SQL requests/responses along with performance indicators.
  • Oracle
    Similar to MySQL plugin, just for Oracle databases.
  • BGP
    Fills nProbe with AS path information. The BGP decoding is performed by a Perl-script provided with the plugin that acts as a BGP server. This plugin is part of nProbe Pro and it does not require a license.
  • IMAP, POP3, SMTP
    Email plugins for decoding (unencrypted) email traffic and generate flows and logs of email activities.
  • Voice: SIP, RTP
    Plugins for decoding VoIP (Voice over IP) traffic and producing call log, and voice information (jitter and packet loss, pseudo-MOS/R-Factor).
  • Radius
    Plugin decoding Radius traffic including 3GPP extensions for mobile networks.
  • Diameter
    Plugin decoding Diameter traffic for both wired and mobile networks.
  • GTPv0
    Same as GTPv1 plugin, just for v0 protocol version.
  • GTPv1
    Plugin for decoding GTPv1-C (2G and 3G networks) signalling and producing comprehensive mobile user and traffic tracking.
  • GTPv2
    Same as GTPv1 plugin, just for v2 protocol version used in LTE (Long Term Evolution) mobile networks.
  • S1AP
    Plugin decoding S1AP traffic used on mobile networks.
  • NetFlow-Lite
    Plugin for collecting NetFlow-Lite traffic sent by some Cisco switches.
  • Process
    Linux plugin that allows local processes to be monitored (CPU, memory, I/O) and be associated with the network traffic they produce.

Binary packages are available selected platforms from http://packages.ntop.org

License

nProbe is distributed under the EULA and requires a license per system.

Get It

nProbe™ is available in two flavours

Version Unix Windows (x64)
Standard Probe with no plugins and basic libpcap-based packet capture. Same as Unix.
Pro with Plugins Same as Pro version with native PF_RING, and support for plugins.
It also includes the following plugins: flow dump into MySQL database (flow-to-MySQL) and BGP plugin.
Same as Unix.

nProbe™ is available for a little fee, that’s used for running the project and funding the new developments. You can purchase online your copy of nProbe™ at the ntop e-shop site, that includes one year support. After the transaction is completed you can download your nProbe™ copy immediately.

If you want to test drive nProbe™ you can use our pre-build binary packages.

If you are a no profit institution or a university, you can have nProbe™ at no cost (even if your donations are welcome): please drop us a mail from your university account where you explain why you qualify (emails originating from non-university account including hotmail, gmail and yahoo will be ignored).

Note that for nProbe™ OEM, reselling, repackaging (including device embed) you need a written commercial licence that’s available on request from its author.