nProbe™ v7

An Extensible NetFlow v5/v9/IPFIX GPL Probe for IPv4/v6


In commercial environments, NetFlow is probably the de-facto standard for network traffic accounting. ntop includes both a NetFlow v5/v9/IPFIX probe and collector that can be used to play with NetFlow flows. This means that you can use ntop:

  • for analysing NetFlow flows generated by your border gateway
  • replacing the embedded, low-speed, NetFlow probe available on your gateway
  • analyzing Gbit networks at full speed with no (or very moderate) packet loss exploiting nProbe™
  • as a NetFlow probe that sends flows towards a collector either ntop or a commercial one (e.g. Cisco NetFlow Collector or HP-OV)
  • both as a probe and collector.

Nevertheless, due to the original ntop design, it cannot be easily deployed as a pure NetFlow collector in environments such as a diskless embedded system with limited resources or a corporate firewall.

In addition, in some environments it would be nice to distribute light network probes on the network that send traffic information towards a central traffic analysis console such as ntop.

In order to satisfy the above requirements nProbe™ has been designed. Currently nProbe™ is a software application available stand-alone or as an embedded system named nBox .

Main nProbe™ Features


  • Available for Unix (including MacOS X and Solaris), Windows, and embedded environments.
  • Added layer 7 application visibility (including Skype, BitTorrent, Citrix….) using nDPI.
  • NetFlow v9/IPFIX support for efficient flow handling.
  • Added Cisco NetFlow-Lite support (as of version 6.5).
  • Full IPFIX support: PEN (Private Enterprise Numbers) and Variable length encoding.
  • Support for IPv4 and v6.
  • Limited memory footprint (less that 2 MB of memory regardless of the network size) and CPU savvy.
  • Ability to natively save flows into MySQL and SQLite, as well as text and binary.
  • Ability to natively export flows in Splunk and ElasticSearch
  • Ability to dump flows in format ready for import in columnar databases such as InfiniDB.
  • Native PF_RING support for high speed flow generation (nProbe™ Pro Unix and above).
  • Ability to act as flow collector and proxy. All combinations are supported.
  • Ability to collect sFlow flows and turn them into flows (v5/v9/IPFIX).
  • Support of detect protocols via DPI (deep packet inspection) and report protocol name in flows for precise collector protocol accounting.
  • Ability to forge NetFlow interfaceIds based on MAC/IP addresses.
  • Collection of Cisco ASA flows and conversion in ‘standard’ flows.
  • New nprobe architecture for better performance and exploitation of multicore architectures.
  • Support of tunneled (including GRE, PPP and GTP) traffic and ability to export in flows inner/outer envelope/packet information.
  • Support of both flow and packet sampling.
  • Support of Flexible Netflow: create your netflow templates, now with PEN support.
  • VoIP (SIP and RTP) traffic analysis including voice quality.
  • HTTP and MySQL/Oracle, DNS protocol analysis: ability to generate logs of web, MySQL/Oracle and DNS activities in addition to flow export.
  • BGP Plugin for establishing a BGP session with a router and generate flows with AS and AS path information.
  • Plugin architecture for easy extensibility via custom V9/IPFIX tags.
  • Fully interoperable with commercial collectors such as IsarFlow, Fluke, Cisco, Dartware, AdventNet, Arbor Networks, Plixer, NetFlow Auditor, SolarWinds Orion NTA.
  • Designed for running on environments with limited resources (the nProbe™ binary < 100 Kb) and embedded systems (e.g. ARM-based appliaces).
  • It can be used to build cheap NetFlow probes using commodity hardware.
  • Able to save flows on disk for later analysis or integration into an existing monitoring application.
  • Fully user configurable.
  • High-performance probe: commercial probes included those embedded on routers and switches are often not able to keep up with high-speeds.
  • Ntop can be used as collector and analyser for NetFlow v5/v9/IPFIX flows such as those generated by nProbe™ and commercial routers.

Using nProbe™


The current nProbe™ version is much more that a simple netflow probe.

Probe mode


Command: nprobe -i eth0 -n collector_ip:2055

Collector mode


Command: nprobe –nf-collector-port 2055

Proxy mode


Command: nprobe –nf-collector-port 2055 -n collector_ip:2055 -V 9

It can be a probe, probe+collector, collector, or a proxy. In proxy mode you can convert from/to IPFIX/NetFlow v5/v9 in order to smoothly upgrade to newer netflow protocol versions while capitalizing on previous protocol versions. So you can for instance convert flows coming from your v5 router into IPFIX and vice-versa. Note that with some combinations (e.g. from v9 to v5) you might loose some flow information.

Performance


Many people are aware that not all the available NetFlow probes are scalable. nProbe™ has been designed to keep up with Gigabit speeds on commodity hardware. Using a dual core CPU, nProbe™ can be used for capturing packets at full speed with no/very little (< 1%) packet loss using PF_RING. Better results can be achieved using packet/flow sampling (i.e. the probe does not receive all the packets but just a sample), or using an accelerated packet capture card.

Packet Size (Bytes) nProbe™ Sustained Throughtput with no packet loss
PF_RING PF_RING DNA
fixed 64 462 Kpps [~237 Mbit] Wire rate
fixed 512 Wire rate
fixed 1500
random 64-1500

The table above shows the result of a worst-case performance test using

  • nProbe™ 6.9.x Pro/Plugins (native PF_RING support)
  • Ubuntu Linux 11.10
  • PF_RING 5.3.x
  • Supermicro PDSM4+ board
  • Intel(R) Core(TM)2 CPU 6320 [1.86GHz]
  • Intel PCIe Gbit card
  • IXIA 400 Traffic Generator
  • 100K rotating IP addresses
  • Generation of 6’500 flows/minute
  • Command used: nprobe -i eth4 -b 1 -w 512000
  • No flow storage on DB or disk, just forwarding to a collector

For the latest news about nProbe, please read the ntop blog.

Running nProbe™ at 10 Gbit


Today commodity hardware cannot provide full 10 Gbit traffic analysis unless some special drivers are used. Using PF_RING DNA is designed to offer wire-speed packet capture performance. nProbe on top of DNA and multi RX-queue can process about 11 Mpps as described on this paper.

nProbe™, FastBit, and InfiniDB


In early 6.x nProbe releases we supported natively the FastBit database. Later on we decided to drop this feature in favour of import in InfiniDB, that features a performance similar to FastBit but it offers full SQL support and integration with MySQL. In InfiniDB data import is performed with the cpimport utility that acceps as input the flow files dumped by nProbe with -P command line flag.

Usage


nProbe™ is distributed in both source and binary format. Once installed, nProbe™ is available for use with no further configuration. Similar to ntop, nProbe™ will be activated on a PC from which it is possible to see/capture the traffic you’re interested in. For this reason, in case of switched networks, it is necessary to either mirror traffic (VLAN or port mirror) or place the probe on a location (e.g. by the border gateway) where most of the traffic flows.

Once activated, nProbe™ will collect traffic data (see below) and emit NetFlow v5/v9/IPFIX flows towards the specified collector. A set of packets with the same (src ip & port, dst ip & port, protocol #) is called flow (note that some protocols such as ICMP have no concept of ports). Every flow, even a very long standing ISO CD image download, has a limited lifetime; this is because the flow collector should periodically receive flow chunks for accounting traffic precisely.

Any standard NetFlow collector including ntop can be used to analyse the flows generated by nProbe™ (please note that not all the commercial collecotrs support v9).
When used with ntop, the nProbe™ can act as a remote and light traffic probe, and ntop as a central network monitoring console for IPFIX/v5/v9.

FAQ


  1. Q: Do your release nProbe™ source code?
    A: Yes of both nProbe™ standard and Pro. We have decided not to release the plugin source to everyone: requests will be evaluate on a case-by-case value if the requestor qualify (e.g. research institution)
  2. Q: Is nProbe™ able to operate on Gbit networks at full speed?
    A: Yes. Note that for exploiting the Gbit packet capture you need a 64-bit PCI Gigabit Ethernet interface.
  3. Q: Is the nProbe™ source code available?
    A: Yes of course, it’s GPL.
  4. Q: Why do you charge for nProbe™ although it’s GPL?
    A: GPL has nothing to do with price ([1] [2] [3]) but with freedom. Many open source companies ask a fee for their software.
  5. Q: What do you do with the money you get charging for nProbe™?
    A: This money is invested for doing research in ntop, nBox and nProbe™ projects.

Documentation


nProbe User’s Guide

Credits


NetFlow is copyright Cisco Systems.
nProbe™ is a trademark registered in USA and the European Union.

nProbe Plugins


nProbe™ is extensible by means of optional plugins. Depending on the platform you can get them in source (Unix) or pre-compiled binary (Win32). Plugins can be used to dissect specific traffic or to provide other features (e.g. traffic collection). Below you can find the list of currently available plugins.

Plugin Description
HTTP Decode HTTP traffic and HTTPS certificates. It can generate a comprehensive log of HTTP traffic, including page
download and network/server delay. Microcloud friendly Available only in binary format..
DNS Decodes DNS traffic, and produce a log of main domain name resolution activities. Microcloud friendly. Available only in binary format.
flow-to-MySQL Dumps exported flows into a MySQL database.
MySQL Decodes (unencrypted) MySQL traffic, and produce a log of SQL requests/responses along with performance indicators. Available only in binary format.
Oracle Similar to MySQL plugin, just for Oracle databases.Available only in binary format.
BGP Fills nProbe with AS path information. The BGP decoding is performed by a Perl-script provided with the plugin that acts
as a BGP server. This plugin is part of nProbe Pro/Plugins.Available only in binary format.
IMAP, POP3, SMTP Email plugins for decoding (unencrypted) email traffic and generate flows and logs of email activities. Available only in binary format.
Voice: SIP, RTP Plugins for decoding VoIP (Voice over IP) traffic and producing call log, and voice information (jitter and packet loss).
There are two version available with/without voice quality (pseudo-MOS/R-Factor). Available only in binary format.
Radius Plugin decoding Radius traffic including 3GPP extensions for mobile networks. Microcloud friendly.
Available only in binary format.
GTPv1 Plugin for decoding GTPv1-C (2G and 3G networks) signalling and producing comprehensive mobile user and traffic
tracking. Microcloud friendly. Available only in binary format.
GTPv2 Same as GTPv1 plugin, just for v2 protocol version used in LTE (Long Term Evolution) mobile networks.
Available only in binary format.
Process Linux plugin that allows local processes to be monitored (CPU, memory, I/O) and be associated with the
network traffic they produce. Available only in binary format.
NetFlow-Lite Plugin Plugin for collecting NetFlow-Lite traffic sent by some Cisco switches.

Note that the Windows version (x64 only) of nProbe is available as binary version (we prebuilt it for you), whereas the Unix version is available as source (you need to compile it yourself). Binary packages are available for selected platform from http://packages.ntop.org

Get It


nProbe™ is available in two flavours

Version Unix Windows (x64)
Standard Probe with no plugins and basic libpcap-based packet capture. Same as Unix.
Pro with Plugins Same as Pro version with native PF_RING, and support for plugins.
It also includes the following plugins: flow dump into MySQL database (flow-to-MySQL) and BGP plugin.
Same as Unix.

nProbe™ is available under the GPLv2 licence for a little fee, that’s used for running the project and funding the new developments. You can purchase online your copy of nProbe™ at the ntop e-shop site, that includes one year support. After the transaction is completed you can download your nProbe™ copy immediately.

If you are an existing nProbe™ owner, you can get the standard version and we’ll give you a free upgrade to the pro/plugins version (do not forget to send us a mail after completing your transaction).

If you want to test drive nProbe™ you can use our pre-build binary packages.

If you are a no profit institution or a university, you can have nProbe™ at no cost (even if your donations are welcome): please drop us a mail from your university account where you explain why you qualify (emails originating from non-university account including hotmail, gmail and yahoo will be ignored).

Note that for nProbe™ OEM, reselling, repackaging (including device embed) you need a written commercial licence that’s available on request from its author. This because this is considered as derived work as specified in the GPL license.