An Extensible NetFlow v5/v9/IPFIX Open Source Probe for IPv4/v6
In commercial environments, NetFlow is probably the de-facto standard for network traffic accounting. ntop includes both a NetFlow v5/v9/IPFIX probe and collector that can be used to play with NetFlow flows. This means that you can use ntop:
- for analysing NetFlow flows generated by your border gateway
- replacing the embedded, low-speed, NetFlow probe available on your gateway
- analyzing Gbit networks at full speed with no (or very moderate) packet loss exploiting nProbe™
- as a NetFlow probe that sends flows towards a collector either ntop or a commercial one (e.g. Cisco NetFlow Collector or HP-OV)
- both as a probe and collector.
Nevertheless, due to the original ntop design, it cannot be easily deployed as a pure NetFlow collector in environments such as a diskless embedded system with limited resources or a corporate firewall.
In addition, in some environments it would be nice to distribute light network probes on the network that send traffic information towards a central traffic analysis console such as ntop.
In order to satisfy the above requirements nProbe™ has been designed. Currently nProbe™ is a software application available stand-alone or as an embedded system named nBox .
Main nProbe™ Features
- Available for Unix (including MacOS X and Solaris), Windows, and embedded environments.
- Added layer 7 application visibility (including Skype, BitTorrent, Citrix….) using nDPI.
- NetFlow v9/IPFIX support for efficient flow handling.
- Added Cisco NetFlow-Lite support (as of version 6.5).
- Full IPFIX support: PEN (Private Enterprise Numbers) and Variable length encoding.
- Support for IPv4 and v6.
- Limited memory footprint (less that 2 MB of memory regardless of the network size) and CPU savvy.
- Ability to natively save flows into MySQL and SQLite, as well as text and binary.
- Ability to natively export flows in Splunk and ElasticSearch
- Ability to dump flows in format ready for import in columnar databases such as InfiniDB.
- Native PF_RING support for high speed flow generation (nProbe™ Pro Unix and above).
- Ability to act as flow collector and proxy. All combinations are supported.
- Ability to collect sFlow flows and turn them into flows (v5/v9/IPFIX).
- Support of detect protocols via DPI (deep packet inspection) and report protocol name in flows for precise collector protocol accounting.
- Ability to forge NetFlow interfaceIds based on MAC/IP addresses.
- Collection of Cisco ASA flows and conversion in ‘standard’ flows.
- New nprobe architecture for better performance and exploitation of multicore architectures.
- Support of tunneled (including GRE, PPP and GTP) traffic and ability to export in flows inner/outer envelope/packet information.
- Support of both flow and packet sampling.
- Support of Flexible Netflow: create your netflow templates, now with PEN support.
- VoIP (SIP and RTP) traffic analysis including voice quality.
- HTTP and MySQL/Oracle, DNS protocol analysis: ability to generate logs of web, MySQL/Oracle and DNS activities in addition to flow export.
- BGP Plugin for establishing a BGP session with a router and generate flows with AS and AS path information.
- Plugin architecture for easy extensibility via custom V9/IPFIX tags.
- Fully interoperable with commercial collectors such as IsarFlow, Fluke, Cisco, Dartware, AdventNet, Arbor Networks, Plixer, NetFlow Auditor, SolarWinds Orion NTA.
- Designed for running on environments with limited resources (the nProbe™ binary < 100 Kb) and embedded systems (e.g. ARM-based appliaces).
- It can be used to build cheap NetFlow probes using commodity hardware.
- Able to save flows on disk for later analysis or integration into an existing monitoring application.
- Fully user configurable.
- High-performance probe: commercial probes included those embedded on routers and switches are often not able to keep up with high-speeds.
- Ntop can be used as collector and analyser for NetFlow v5/v9/IPFIX flows such as those generated by nProbe™ and commercial routers.
The current nProbe™ version is much more that a simple netflow probe.
nprobe -i eth0 -n collector_ip:2055
nprobe –nf-collector-port 2055
nprobe –nf-collector-port 2055 -n collector_ip:2055 -V 9
It can be a probe, probe+collector, collector, or a proxy. In proxy mode you can convert from/to IPFIX/NetFlow v5/v9 in order to smoothly upgrade to newer netflow protocol versions while capitalizing on previous protocol versions. So you can for instance convert flows coming from your v5 router into IPFIX and vice-versa. Note that with some combinations (e.g. from v9 to v5) you might loose some flow information.
Many people are aware that not all the available NetFlow probes are scalable. nProbe™ has been designed to keep up with Gigabit speeds on commodity hardware. Using a dual core CPU, nProbe™ can be used for capturing packets at full speed with no/very little (< 1%) packet loss using PF_RING. Better results can be achieved using packet/flow sampling (i.e. the probe does not receive all the packets but just a sample), or using an accelerated packet capture card.
|Packet Size (Bytes)||nProbe™ Sustained Throughtput with no packet loss|
|fixed 64||462 Kpps [~237 Mbit]||Wire rate|
|fixed 512||Wire rate|
The table above shows the result of a worst-case performance test using
- nProbe™ 6.9.x Pro/Plugins (native PF_RING support)
- Ubuntu Linux 11.10
- PF_RING 5.3.x
- Supermicro PDSM4+ board
- Intel(R) Core(TM)2 CPU 6320 [1.86GHz]
- Intel PCIe Gbit card
- IXIA 400 Traffic Generator
- 100K rotating IP addresses
- Generation of 6’500 flows/minute
- Command used: nprobe -i eth4 -b 1 -w 512000
- No flow storage on DB or disk, just forwarding to a collector
For the latest news about nProbe, please read the ntop blog.
Running nProbe™ at 10 Gbit
Today commodity hardware cannot provide full 10 Gbit traffic analysis unless some special drivers are used. Using PF_RING ZC is designed to offer wire-speed packet capture performance. nProbe on top of ZC/DNA and multi RX-queue can process about 11 Mpps as described on this paper.
nProbe™, FastBit, and InfiniDB
In early 6.x nProbe releases we supported natively the FastBit database. Later on we decided to drop this feature in favour of import in InfiniDB, that features a performance similar to FastBit but it offers full SQL support and integration with MySQL. In InfiniDB data import is performed with the cpimport utility that acceps as input the flow files dumped by nProbe with -P command line flag.
nProbe™ is distributed in both source and binary format. Once installed, nProbe™ is available for use with no further configuration. Similar to ntop, nProbe™ will be activated on a PC from which it is possible to see/capture the traffic you’re interested in. For this reason, in case of switched networks, it is necessary to either mirror traffic (VLAN or port mirror) or place the probe on a location (e.g. by the border gateway) where most of the traffic flows.
Once activated, nProbe™ will collect traffic data (see below) and emit NetFlow v5/v9/IPFIX flows towards the specified collector. A set of packets with the same (src ip & port, dst ip & port, protocol #) is called flow (note that some protocols such as ICMP have no concept of ports). Every flow, even a very long standing ISO CD image download, has a limited lifetime; this is because the flow collector should periodically receive flow chunks for accounting traffic precisely.
Any standard NetFlow collector including ntop can be used to analyse the flows generated by nProbe™ (please note that not all the commercial collecotrs support v9).
When used with ntop, the nProbe™ can act as a remote and light traffic probe, and ntop as a central network monitoring console for IPFIX/v5/v9.
- Q: Do your release nProbe™ source code?
A: Yes of both nProbe™ standard and Pro. We have decided not to release the plugin source to everyone: requests will be evaluate on a case-by-case value if the requestor qualify (e.g. research institution).
- Q: Is nProbe™ able to operate on Gbit networks at full speed?
A: Yes. Note that for exploiting the Gbit packet capture you need a 64-bit PCI Gigabit Ethernet interface.
- Q: Can I redistribute the nProbe source or build a derivative product?
A: No source code cannot be redistributed and it is only for educational purposes and private use. If you plan to build a product or sell nProbe-based solutions you need to contact us.
- Q: Why do you charge for nProbe™ although it’s open source?
A: Open source has nothing to do with price but with freedom. Many open source companies ask a fee for their software.
- Q: What do you do with the money you get charging for nProbe™?
A: This money is invested for doing research in ntop, nBox and nProbe™ projects.
NetFlow is copyright Cisco Systems.
nProbe™ is a trademark registered in USA and the European Union.
nProbe™ is extensible by means of optional plugins. Depending on the platform you can get them in source (Unix) or pre-compiled binary (Win32). Plugins can be used to dissect specific traffic or to provide other features (e.g. traffic collection). Below you can find the list of currently available plugins all available in binary format only.
|HTTP||Decode HTTP traffic and HTTPS certificates. It can generate a comprehensive log of HTTP traffic, including page
download and network/server delay. Microcloud friendly.
|DNS||Decodes DNS traffic, and produce a log of main domain name resolution activities. Microcloud friendly. Available only in binary format.|
|flow-to-MySQL||Dumps exported flows into a MySQL database.|
|MySQL||Decodes (unencrypted) MySQL traffic, and produce a log of SQL requests/responses along with performance indicators.|
|Oracle||Similar to MySQL plugin, just for Oracle databases.|
|BGP||Fills nProbe with AS path information. The BGP decoding is performed by a Perl-script provided with the plugin that acts
as a BGP server. This plugin is part of nProbe Pro/Plugins.
|IMAP, POP3, SMTP||Email plugins for decoding (unencrypted) email traffic and generate flows and logs of email activities.|
|Voice: SIP, RTP||Plugins for decoding VoIP (Voice over IP) traffic and producing call log, and voice information (jitter and packet loss).
There are two version available with/without voice quality (pseudo-MOS/R-Factor).
|Radius||Plugin decoding Radius traffic including 3GPP extensions for mobile networks. Microcloud friendly.
Available only in binary format.
|Diameter||Plugin decoding Diameter traffic for both wired and mobile networks.
Available only in binary format.
|GTPv0||Same as GTPv1 plugin, just for v0 protocol version. Available only in binary format.|
|GTPv1||Plugin for decoding GTPv1-C (2G and 3G networks) signalling and producing comprehensive mobile user and traffic tracking. Microcloud friendly.|
|GTPv2||Same as GTPv1 plugin, just for v2 protocol version used in LTE (Long Term Evolution) mobile networks.|
|S1AP||Plugin decoding S1AP traffic used on mobile networks.|
|NetFlow-Lite Plugin||Plugin for collecting NetFlow-Lite traffic sent by some Cisco switches.|
|Process||Linux plugin that allows local processes to be monitored (CPU, memory, I/O) and be associated with the network traffic they produce.|
Note that the Windows version (x64 only) of nProbe is available as binary version (we prebuilt it for you), whereas the Unix version is available as source (you need to compile it yourself). Binary packages are available for selected platform from http://packages.ntop.org
nProbe is distributed under the EULA and requires a license per system.
nProbe™ is available in two flavours
|Standard||Probe with no plugins and basic libpcap-based packet capture.||Same as Unix.|
|Pro with Plugins||Same as Pro version with native PF_RING, and support for plugins.
It also includes the following plugins: flow dump into MySQL database (flow-to-MySQL) and BGP plugin.
|Same as Unix.|
nProbe™ is available for a little fee, that’s used for running the project and funding the new developments. You can purchase online your copy of nProbe™ at the ntop e-shop site, that includes one year support. After the transaction is completed you can download your nProbe™ copy immediately.
If you are an existing nProbe™ owner, you can get the standard version and we’ll give you a free upgrade to the pro/plugins version (do not forget to send us a mail after completing your transaction).
If you want to test drive nProbe™ you can use our pre-build binary packages.
If you are a no profit institution or a university, you can have nProbe™ at no cost (even if your donations are welcome): please drop us a mail from your university account where you explain why you qualify (emails originating from non-university account including hotmail, gmail and yahoo will be ignored).
Note that for nProbe™ OEM, reselling, repackaging (including device embed) you need a written commercial licence that’s available on request from its author.