nProbe™
An Extensible NetFlow v5/v9/IPFIX Probe for IPv4/v6
In commercial environments, NetFlow is probably the de-facto standard for network traffic accounting. nProbe includes both a NetFlow v5/v9/IPFIX probe and collector that can be used to play with NetFlow flows. This means nProbe™ can be used:
- To collect and export NetFlow flows generated by border gateways/switches/routers or any other device that can export in NetFlow v5/v9
- As a drop-in replacement of embedded, low-speed, NetFlow probes that may already been deployed
- To analyze multi-Gbit networks at full speed with no (or very moderate) packet loss
- To send monitored flows towards a collector such as the open-source ntopng or a commercial one (e.g. Cisco NetFlow Collector or Plixer)
Currently nProbe™ is a software application available stand-alone or as an embedded system named nBox 
.
Main nProbe™ Features
- Available for Linux, FreeBSD (including OPNsense and pfSense) Windows, and embedded environments ARM and MIPS/MIPSEL.
- Layer-7 application visibility (250+ applications including Skype, BitTorrent and Citrix).
- Layer-7 application propagation in exported flows to enable accurate accounting.
- NetFlow v5/v9/IPFIX support for efficient flow handling.
- Cisco NetFlow-Lite support.
- IPS Mode: ability to block and shape traffic using nDPI.
- Full IPFIX support: PEN (Private Enterprise Numbers) and Variable length encoding.
- Complete support for IPv4 and IPv6.
- Limited memory footprint (less that 2 MB of memory regardless of the network size) and CPU savvy.
- Ability to natively export flows to Apache™ , Syslog, MySQL/MariaDB, Splunk (via TCP streaming).
- Ability to natively export flows to Kafka and ElasticSearch (using the Export Plugin).
- Ability to dump flows in format ready for import in columnar databases.
- Native support for technologies PF_RING and the newest kernel-bypass PF_RING Zero Copy (ZC) for ultra-high speed packet capture.
- Ability to act as flow collector and proxy. All combinations are supported.
- Ability to collect sFlow flows and transparently translate them into NetFlow v5/v9/IPFIX.
- Ability to forge NetFlow interface identificators based on MAC/IP addresses.
- Collection of Cisco ASA flows and conversion into NetFlow v5/v9/IPFIX.
- Multi-threaded architecture for the exploitation of multi-processor, multi-core elaboration systems.
- Support of tunnelled (including GRE, PPP, VXLAN, and GTP) traffic and ability to export inner/outer envelope/packet information.
- Support of both flow and packet sampling.
- Support of Flexible Netflow for the creation of custom NetFlow templates, with optional PEN support.
- VoIP (SIP and RTP) traffic analysis including voice quality and (pseudo-)MOS.
- HTTP, MySQL/Oracle, DNS protocol analysis: ability to generate logs of web, MySQL/Oracle and DNS activities in addition to flow export.
- Plugin architecture for easy extensibility via custom V9/IPFIX tags.
- Fully interoperable with commercial collectors such as IsarFlow, Fluke, Cisco, Dartware, Arbor Networks, Plixer, NetFlow Auditor, SolarWinds Orion NTA, Andrisoft.
- Designed for running on environments with limited resources (the nProbe™ binary < 100 Kb) and embedded systems (e.g. ARM- and MIPSEL-based appliances).
- It can be used to build cheap NetFlow probes using commodity hardware.
- Ability to save flows on disk for later analysis or integration into an existing monitoring application.
- Fully user-configurable.
- High-performance probe: commercial probes included those embedded on routers and switches are often not able to keep up with high-speeds.
- Can be used with ntopng to visualize, collect, and analyze monitored traffic.
- Native nTap support for collecting traffic from cloud, VMs, containers and physical hosts.
- Agent mode support on Windows and Linux systems (eBPF based) to augment network metadata with users and processes.
nProbe™ Versions
nProbe is available in three versions whose main differences are listed in the table below.
| Pro | Enterprise S | Enterprise M | Enterprise L† | |
|---|---|---|---|---|
| nDPI Traffic Inspection | ✓ | ✓ | ✓ | ✓ |
| Flow Collection | ✓ | ✓ | ✓ | ✓ |
| PF_RING Acceleration | ✓ | ✓ | ✓ | ✓ |
| HTTP Plugin | ✓ | ✓ | ✓ | |
| DNS Plugin | ✓ | ✓ | ✓ | |
| DHCP Plugin | ✓ | ✓ | ||
| Diameter Plugin | ✓ | ✓ | ||
| Elastic/JSON/Kafka Plugin | ✓ | ✓ | ||
| FTP Plugin | ✓ | ✓ | ||
| IMAP/SMTP/POP Plugin | ✓ | ✓ | ||
| NetFlow-Lite Plugin | ✓ | ✓ | ✓ | |
| SIP/RTP Plugin | ✓ | ✓ | ||
| GTP V0/V1/V2 Plugins | ✓ | |||
| Modbus Plugin | ✓ | ✓ | ✓ | ✓ |
| Radius Plugin | ✓ | ✓ | ||
| IPv4 Packet Deduplication | ✓ | ✓ | ✓ | ✓ |
| Agent Mode | ✓ | ✓ | ||
| Native nTap Support | ✓ | ✓ | ||
| Flow Collection Deduplication | ✓ | ✓ | ||
| Max Number of Monitored Hosts | Unlimited (limited by available CPU/memory) | |||
| Max Number of Instances per Host/License | Unlimited (limited by available CPU/memory) | |||
| Max Flow Collection Devices†† | 4 | 8 | 16 | 128 |
| Max number of ZMQ exporters (–zmq) | 4 | 8 | 16 | 32 |
| Max number of Rules/Pools in IPS Mode (–ips-mode) | 4 | 8 | 32 | 256 |
| License | Shop | |||
| ††This is the number of flow devices (e.g. a NetFlow router or sFlow switch) from which a single nProbe instance can collect flows from.
†Enterprise L is offered with customised features and personalised maximums for flow collection devices and ZMQ exporters. |
||||
Contrary to past nProbe versions, plugins are no longer available individually but are bundled with application versions. You you own a old nProbe version, we advise to read this technical note.
Using nProbe™
The current nProbe™ version is much more than a simple netflow probe.
Probe mode
nprobe -i eth0 --collector 127.0.0.1:2055
Collector mode
nprobe --collector-port 2055
Proxy mode
nprobe --collector-port 2055 --collector 127.0.0.1:2055 -V 9
IPS mode
nprobe -i nf:0 --ips-mode ips-rules.conf --collector 127.0.0.1:2055
This configuration is the same as probe mode with the difference that nprobe in essence acts like a bridge device by applying IPS policies to the bridged traffic.
It can be a probe/probe+IPS, probe+collector, collector, or a proxy. In proxy mode it is possible to convert from/to IPFIX/NetFlow v5/v9 in order to smoothly upgrade to newer NetFlow protocol versions while capitalizing on previous protocol versions. So you can for instance convert flows coming from your v5 router into IPFIX and vice-versa. Note that with some combinations (e.g. from v9 to v5) you might lose some flow information.
Performance
Probe mode
Many people are aware that not all the available NetFlow probes are scalable. nProbe™ has been designed to keep up with multi-Gbit speeds on commodity hardware. Using a dual core CPU, nProbe™ can be used for capturing packets at 1 Gbit with no/very little (< 1%) packet loss using vanilla PF_RING (no ZC). With the PF_RING ZC kernel-bypass technologies packet capture is eve faster as is it possible to read below. Please note that performance figures are per-core. This means that, for example, by leveraging on PF_RING ZC, it is possible to achieve a 4x performance improvement on a quad-core CPU if compared to a single-core one.
| Packet Size (Bytes) | Per-core nProbe™ Sustained Throughput with no packet loss | |
|---|---|---|
| PF_RING ZC | ||
| fixed 64 | 3.32 Mpps, 2.15 Gb/secs | |
| fixed 512 | Wire rate | |
| fixed 1500 | ||
| random 64-1500 | ||
The table above shows the result of a worst-case performance test using
- nProbe™ 7.2 Pro/Plugins (native PF_RING support)
- Ubuntu Linux 14.10
- PF_RING 6.1.X
- Dell R220
- CPU Intel E3-1241 v3 @ 3.50GHz
- Intel 82599-based 10 Gbit card
- Traffic Generator: pfsend -i zc:ethX -a -g 1 -b 250000
- 250K rotating IP addresses
- Generation of 250K flows/minute
- Command used: nprobe -i zc:eth1 –cpu-affinity 1 -t 60 -b 1 -w 500000 -V 9
- No flow storage on DB or disk, just forwarding to a collector
Collector mode
This mode can be used to collect flows in NetFlow v5/v9/IPFIX format and deliver flows to ntopng. Please find below the performance of nProbe collecting NetFlow and exporting flows over ZMQ.
| Template | Ingress Rate (NetFlow) | Export Rate (ZMQ) |
|---|---|---|
| Default | 12’000 packets/second (avg 19 records/packet) | 230’000 flows/second |
| @NTOPNG@ | 8’500 packets/second (avg 19 records/packet) | 160’000 flows/second |
The table above shows the result of an export performance test using:
- nProbe™ 8.7 Pro
- CentOS Linux 7.6
- CPU Intel E3-1230 v5 @ 3.40GHz
- Incoming flows in NetFlow v9 format
- ZMQ export (TLV)
- Internal cache disabled (this guarantees that nProbe transparently forwards incoming flows)
- Command used: nprobe -i none -n none –collector-port 2055 –zmq tcp://192.168.1.1:5556 –disable-cache [-T @NTOPNG@]
- No flow storage on DB or disk, just forwarding to ntopng as collector via ZMQ
The bandwidth required for exporting flow information over ZMQ is <30 Mbit/sec for 10’000 flows/second, this number can be scaled based on the actual number of flows. For example, when processing 1 Gbit/s average internet traffic, the bandwidth required is <5 Mbit/s as peak (note: this highly depends on traffic type).
IPS mode
Below you can find the expected performance of the IPS Mode on both Linux and FreeBSD using a low-end computer or mid-range PC.
| Device | Vanilla Linux Bridge Only | Linux nProbe IPS | Vanilla FreeBSD Bridge Only | FreeBSD nProbe IPS |
|---|---|---|---|---|
| PC Engines APU2 | 550 Mbps | 600 Mbps | 1 Gbps | 120 Mbps |
| Intel E3 | 10 Gbps / 1.8 Mpps | 10 Gbps / 2.4 Mpps |
Tests have been performed on the following conditions:
- Linux IPS mode has been tested with 4 queues using a netfiter configuration named “kernel marker bypass”. This explains why nprobe in IPS mode is faster on a low-end box that the vanilla bridge.
- Under FreeBSD there is no kernel marker bypass mode, hence the performance decrease is severe when using nProbe as bridging is performed in user-space.
- Average packet size 1000 byte.
Usage
nProbe™ is distributed in binary format. Once installed, nProbe™ is ready be used and does not require any additional configuration. In order to function properly in probe mode, nProbe™ needs to see/capture the traffic of interest. For this reason, in case of switched networks, it is necessary to either mirror traffic (VLAN or port mirror) or place the probe in a location (e.g. by the border gateway) that is traversed by the most part of the traffic. Under normal operating conditions nProbe™ will collect traffic data and emit NetFlow v5/v9/IPFIX flows towards the specified collector. Any standard NetFlow collector can be used to analyze the flows generated by nProbe™ — although not all the commercial collectors support v9. nProbe™ can also be used in conjunction with ntopng. In the latter case an optimized, optionally compressed and encrypted format will be used for data exchange, leading to a lightweight monitoring architecture that decouples the monitoring part from the visualization and analysis part.
nProbe™ vs nProbe™ Cento
If you are wondering that are the differences between nProbe and nProbe Cento, you can read this page for the details
FAQ
- Q: Do your release nProbe™ source code?
A: We have decided not to release the source to everyone as in the past some people misused it, and thus we want to avoid this to happen again. - Q: Is nProbe™ able to operate on Gbit networks at full speed?
A: Yes. Note that for exploiting the Gbit packet capture you need a 64-bit PCI Gigabit Ethernet interface - Q: What do you do with the money you get charging for nProbe™?
A: This money is invested for doing research and product development.
Documentation
Credits
NetFlow is copyright by Cisco Systems.
nProbe™ is a trademark registered in the USA and the European Union.
nProbe Plugins
nProbe™ is extensible and includes several plugins which are unlocked based on the license version. For instance nProbe Enterprise S includes HTTP, DNS and a few more. Please refer to the comparison table above to check the compatibility. Below you can find the list of currently available plugins.
- HTTP
Decode HTTP traffic and HTTPS certificates. It can generate a comprehensive log of HTTP traffic, including page download and network/server delay. - DHCP
Decode DHCP traffic and export DHCP information in flows or file dump. - Export
Export to ElasticSearchPlugin/Kafka that can natively export flow information into ElasticSearch without third party converters such as Logstash. - DNS
Decodes DNS traffic, and produce a log of main domain name resolution activities. Microcloud friendly. Available only in binary format. - flow-to-MySQL
Dumps exported flows into a MySQL database. This plugin is part of nProbe Pro and it does not require a license. - MySQL
Decodes (unencrypted) MySQL traffic, and produce a log of SQL requests/responses along with performance indicators. - Email: IMAP, POP3, SMTP
Email plugins for decoding (unencrypted) email traffic and generate flows and logs of email activities. - Voice: SIP, RTP
Plugins for decoding VoIP (Voice over IP) traffic and producing call log, and voice information (jitter and packet loss, pseudo-MOS/R-Factor). - Radius
Plugin decoding Radius traffic including 3GPP extensions for mobile networks. - Diameter
Plugin decoding Diameter traffic for both wired and mobile networks. - GTPv0
Same as GTPv1 plugin, just for v0 protocol version. - GTPv1
Plugin for decoding GTPv1-C (2G and 3G networks) signalling and producing comprehensive mobile user and traffic tracking. - GTPv2
Same as GTPv1 plugin, just for v2 protocol version used in LTE (Long Term Evolution) mobile networks. - SSDP
Plugin decoding SSDP (Simple Service Discovery Protocol) traffic used on networks to discover network devices and services. - NetFlow-Lite
Plugin for collecting NetFlow-Lite traffic sent by some Cisco switches. - NetBIOS
Plugin decoding NetBIOS traffic used in Windows networks.
Operating Systems
License
nProbe is distributed under the EULA and requires a license per system.
Get It
nProbe™ is available for a little fee, that’s used for running the project and funding the new developments. You can purchase online your copy of nProbe™ at the ntop e-shop site, that includes one year support. After the transaction is completed you can download your nProbe™ copy immediately.
If you want to test drive nProbe™ you can use our pre-build binary packages.
If you are a no profit institution or a university, you can have nProbe™ at no cost (even if your donations are welcome): please drop us a mail from your university account where you explain why you qualify (emails originating from non-university account including hotmail, gmail and yahoo will be ignored).
Note that for nProbe™ OEM, reselling, repackaging (including device embed) you need a written commercial licence that’s available on request from its author.