nProbe™ v6

An Extensible NetFlow v5/v9/IPFIX GPL Probe for IPv4/v6

In commercial environments, NetFlow is probably the de-facto standard for network traffic accounting. ntop includes both a NetFlow v5/v9/IPFIX probe and collector that can be used to play with NetFlow flows. This means that you can use ntop:

  • for analysing NetFlow flows generated by your border gateway
  • replacing the embedded, low-speed, NetFlow probe available on your gateway
  • analyzing Gbit networks at full speed with no (or very moderate) packet loss exploiting nProbe™
  • as a NetFlow probe that sends flows towards a collector either ntop or a commercial one (e.g. Cisco NetFlow Collector or HP-OV)
  • both as a probe and collector.

Nevertheless, due to the original ntop design, it cannot be easily deployed as a pure NetFlow collector in environments such as a diskless embedded system with limited resources or a corporate firewall.

In addition, in some environments it would be nice to distribute light network probes on the network that send traffic information towards a central traffic analysis console such as ntop.

In order to satisfy the above requirements nProbe™ has been designed. Currently nProbe™ is a software
application available stand-alone or as an embedded system named nBox .

 

Main nProbe™ Features

  • Available for Unix (including MacOS X and Solaris), Windows, and embedded environments.
  • Added layer 7 application visibility (including Skype, BitTorrent, Citrix….).
  • NetFlow v9/IPFIX support for efficient flow handling.
  • Added Cisco NetFlow-Lite support (as of version 6.5).
  • Full IPFIX support: PEN (Private Enterprise Numbers) and Variable length encoding.
  • Support for IPv4 and v6.
  • Limited memory footprint (less that 2 MB of memory regardless of the network size) and CPU savvy.
  • Ability to natively save flows into MySQL and SQLite, as well as text and binary.
  • Ability to natively dump flows in FastBit format.
  • Native PF_RING support for high speed flow generation (nProbe™ Pro Unix and above).
  • Ability to act as flow collector and proxy. All combinations are supported.
  • Ability to collect sFlow flows and turn them into flows (v5/v9/IPFIX).
  • Support of detect protocols via DPI (deep packet inspection) and report protocol name in flows for precise collector protocol accounting.
  • Ability to forge NetFlow interfaceIds based on MAC/IP addresses.
  • Collection of Cisco ASA flows and conversion in ‘standard’ flows.
  • New nprobe architecture for better performance and exploitation of multicore architectures.
  • Support of tunneled (including GRE, PPP and GTP) traffic and ability to export in flows inner/outer envelope/packet information.
  • Support of both flow and packet sampling.
  • Support of Flexible Netflow: create your netflow templates, now with PEN support.
  • VoIP (SIP and RTP) traffic analysis.
  • HTTP and MySQL protocol analysis: ability to generate logs of web and mysql activities in addition to flow export.
  • BGP Plugin for establishing a BGP session with a router and generate flows with AS and AS path information.
  • Plugin architecture for easy extensibility via custom V9/IPFIX tags.
  • Fully interoperable with commercial collectors such as IsarFlow, Fluke, Cisco, Dartware, AdventNet, Arbor Networks, Plixer, NetFlow Auditor, SolarWinds Orion NTA.
  • Designed for running on environments with limited resources (the nProbe™ binary < 100 Kb) and embedded systems (e.g. ARM-based appliaces).
  • It can be used to build cheap NetFlow probes using commodity hardware.
  • Able to save flows on disk for later analysis or integration into an existing monitoring application.
  • Fully user configurable.
  • High-performance probe: commercial probes included those embedded on routers and switches are often not able to keep up with high-speeds.
  • Ntop can be used as collector and analyser for NetFlow v5/v9/IPFIX flows such as those generated by nProbe™ and commercial routers.

 

Using nProbe™

The current nProbe™ version is much more that a simple netflow probe.

 

Probe mode


Command: nprobe -i eth0 -n collector_ip:2055

Collector mode


Command: nprobe –nf-collector-port 2055

Proxy mode


Command: nprobe –nf-collector-port 2055 -n collector_ip:2055 -V 9

It can be a probe, probe+collector, collector, or a proxy. In proxy mode you can convert from/to IPFIX/NetFlow v5/v9 in order to smoothly upgrade to newer netflow protocol versions while capitalizing on previous protocol versions. So you can for instance convert flows coming from your v5 router into IPFIX and vice-versa. Note that with some combinations (e.g. from v9 to v5) you might loose some flow information.

 

Performance

Many people are aware that not all the available NetFlow probes are scalable. nProbe™ has been designed to keep up with Gigabit speeds on commodity hardware. Using a dual core CPU, nProbe™ can be used for capturing packets at full speed with no/very little (< 1%) packet loss using PF_RING. Better results can be achieved using packet/flow sampling (i.e. the probe does not receive all the packets but just a sample), or using an accelerated packet capture card.

Packet Size (Bytes) nProbe™ Sustained Throughtput with no packet loss
PF_RING PF_RING DNA
fixed 64 462 Kpps [~237 Mbit] Wire rate
fixed 512 Wire rate
fixed 1500
random 64-1500

The table above shows the result of a worst-case performance test using

  • nProbe™ 6.9.x Professional (native PF_RING support)
  • Ubuntu Linux 11.10
  • PF_RING 5.3.x
  • Supermicro PDSM4+ board
  • Intel(R) Core(TM)2 CPU 6320 [1.86GHz]
  • Intel PCIe Gbit card
  • IXIA 400 Traffic Generator
  • 100K rotating IP addresses
  • Generation of 6’500 flows/minute
  • Command used: nprobe -i eth4 -b 1 -w 512000
  • No flow storage on DB or disk, just forwarding to a collector

For the latest news about nProbe, please read the ntop blog.

 

Running nProbe™ at 10 Gbit

Today commodity hardware cannot provide full 10 Gbit traffic analysis unless some special drivers are used. Using PF_RING DNA is designed to offer wire-speed packet capture performance. nProbe on top of DNA and multi RX-queue can process almost 11 Mpps as described on this paper.

 

nProbe™ and FastBit

Relational databases are used by many flow collectors for storing data into a database. They are very popular and well knows by the network community, but they sacrifice flow collection speed and query response time. Another approch is to yse raw disk archived that are very efficient in terms of flow-to-fisk collection but have limited query facilities. Fastbit is a column-oriented database that implements indexes using bitmaps. They are very efficient for both dumping and querying data, yet using a subset of SQL. Using FastBit it’s possible to have sub-second search time when performing cardinality searches as needed for exploring data in real-time and implementing interactive drill-down data search.

 

Query MySQLnProbe™ + FastBit
Q1 22.6 sec 10 sec
Q2 69 sec 1.5 sec
Q3 971 sec 32.9 sec
Q4 1341 sec 55.7 sec
Q5 2257 sec 47.3 sec

nProbe™ FastBit vs MySQL

 

 

nProbe™+FastBit 45 sec
nfdump 1500 sec

nProbe™ with FastBit vs nfdump

 

For more information about nProve vs similar solutions please refer to Collection and Exploration of Large Data Monitoring Sets Using Bitmap Databases.

 

Usage

nProbe™ is distributed in both source and binary format. Once installed, nProbe™ is available for use with no further configuration. Similar to ntop, nProbe™ will be activated on a PC from which it is possible to see/capture the traffic you’re interested in. For this reason, in case of switched networks, it is necessary to either mirror traffic (VLAN or port mirror) or place the probe on a location (e.g. by the border gateway) where most of the traffic flows.

Once activated, nProbe™ will collect traffic data (see below) and emit NetFlow v5/v9/IPFIX flows towards the specified collector. A set of packets with the same (src ip & port, dst ip & port, protocol #) is called flow (note that some protocols such as ICMP have no concept of ports). Every flow, even a very long standing ISO CD image download, has a limited lifetime; this is because the flow collector should periodically receive flow chunks for accounting traffic precisely.

Welcome to nprobe v.6.0.0 ($Revision: 1713 $) for i386-apple-darwin10.4.1Built on 09/30/10 08:36:02 PM
Copyright 2002-10 by Luca Deri <deri@ntop.org>

Usage:
nprobe -n <host:port|none> [-i <interface|dump file>] [-t <lifetime timeout>]
[-d <idle timeout>] [-l <queue timeout>] [-s <scan cycle>] [-N]
[-p <aggregation>] [-f <filter>] [-a] [-b <level>] [-G] [-O <# threads>]
[-P <path>] [-F <dump timeout>] [-D <format>]
[-u <in dev idx>] [-Q <out dev idx>]
[-I <probe name>] [-v] [-w <hash size>] [-e <flow delay>] [-B <packet count>]
[-z <min flow size>] [-M <max num flows>][-R <payload Len>]
[-x <payload policy>] [-E <engine>] [-C <flow lock file>]
[-m <min # flows>][-q <host:port>]
[-S <sample rate>] [-A <AS list>] [-g <PID file>]
[-T <flow template>] [-U <flow template id>]
[-o <v9 templ. export policy>] [-L <local nets>] [-c] [-r]
[-1 <MAC>@<ifIdx>][-3 <port>] [-4] [-5 <port>] [-6]
[-9 <path>] [--black-list <networks>] [--pcap-file-list <filename>]
[--fastbit <dir>] [--fastbit-rotation <mins>]
[--fastbit-template <flow template>] [--fastbit-index <flow template>]
[--fastbit-exec <cmd>]

[--collector|-n] <host:port|none> | Address of the NetFlow collector(s).
| Multiple collectors can be defined using
| multiple -n flags. In this case flows
| will be sent in round robin mode to
| all defined collectors if the -a flag
| is used. Note that you can specify
| both IPv4 and IPv6 addresses.
| If you specify none as value,
| no flow will be export; in this case
| the -P parameter is mandatory.
[--interface|-i] <iface|pcap> | Interface name from which packets are
| captured, or .pcap file (debug only)
[--lifetime-timeout|-t] <timeout> | It specifies the maximum (seconds) flow
| lifetime [default=120]
[--idle-timeout|-d] <timeout> | It specifies the maximum (seconds) flow
| idle lifetime [default=30]
[--queue-timeout|-l] <timeout> | It specifies how long expired flows
| (queued before delivery) are emitted
| [default=30]
[--scan-cycle|-s <scan cycle>] | It specifies how often (seconds) expired
| flows are emitted [default=30].
| If -P is used, the scan cycle will be
| set to the value of the -F parameter
[--rebuild-hash|N] | Rebuild the hash at each scan. Useful for
| producing flows that last as the scan
| cycle as netflow collectors do. This
| option is ignored when -P is not used.
[--aggregation|-p] <aggregation> | It specifies the flow aggiregation level:
| <VLAN Id>/<proto>/<IP>/<port>/<TOS>/<AS>
| where each element can be set to 0=ignore
| or 1=take care. Example ‘-p 1/0/1/1/1/1′
| ignores the protocol, whereas
| ‘-p 0/0/1/0/0/0′ ignores everything
| but the IP
[--bpf-filter|-f] <BPF filter> | BPF filter for captured packets
| [default=no filter]
[--all-collectors|-a] | If several collectors are defined, this
| option gives the ability to send all
| collectors all the flows. If the flag is
| omitted collectors are selected in
| round robin.
[--verbose|-b] <level> | Verbose output:
| 0 – No verbose logging
| 1 – Limited logging (traffic statistics)
| 2 – Full verbose logging
[--daemon-mode|-G] | Start as daemon.
[--num-threads|-O] <# threads> | Number of packet fetcher threads
| [default=2]. Use 1 unless you know
| what you’re doing.
[--dump-path|-P] <path> | Directory where dump files will
| be stored.
[--dump-frequency|-F] <dump timeout>| Dump files dump frequencey (sec).
| Default: 60
[--dump-format|-D] <format> | <format>: flows are saved as:
| b : raw/uncompressed flows
| t : text flows
| d : SQLite
| Example: -D b. Note: this flag has no
| effect without -P.
[--in-iface-idx|-u] <in dev idx> | Index of the input device used in the
| emitted flows (incoming traffic). The default
| value is 0. Use -1 as value to dynamically
| set to the last two bytes of
| the MAC address of the flow sender.
[--out-iface-idx|-Q] <out dev idx> | Index of the output device used in the
| emitted flows (outgoing traffic). The default
| value is 0. Use -1 as value to dynamically
| set to the last two bytes of
| the MAC address of the flow receiver.
[--vlanid-as-iface-idx] | Use vlanId (or 0 if the traffic isn’t tagged)
| as interface index. Note that this option
| superseedes the –in/out-iface-idx options
[--nprobe-version|-v] | Prints the program version.
[--flow-lock|-C] <flow lock> | If the flow lock file is present no flows
| are emitted. This facility is useful to
| implement high availability by means of
| a daemon that can create a lock file
| when this instance is in standby.
[--help|-h] | Prints this help.
[--syslog|-I] <probe name> | Log to syslog as <probe name>
| [default=stdout]
[--hash-size|-w] <hash size> | Flows hash size [default=32768]
[--flow-delay|-e] <flow delay> | Delay (in ms) between two flow
| exports [default=1]
[--count-delay|-B] <packet count> | Send this many packets before
| the -e delay [default=1]
[--min-flow-size|-z] <min flow size>| Minimum TCP flow size (in bytes).
| If a TCP flow is shorter than the
| specified size the flow is not
| emitted [default=unlimited]
[--max-num-flows|-M] <max num flows>| Limit the number of active flows. This is
| useful if you want to limit the memory
| or CPU allocated to nProbe™ in case of non
| well-behaved applications such as
| worms or DoS. [default=4294967295]
[--payload-length|-R] <payload Len> | Specify the max payload length
| [default: 0 bytes]
[--payload-policy|-x] <policy> | Specify the max payload export policy.
| Format: TCP:UDP:ICMP:OTHER where all
| parameters can se set to:
| 0: no payload for the selected protocol
| 1: payload for the selected protocol
| 2: payload for TCP sessions with SYN flag
| Example -x 2:0:0:0 [default=2:0:0:0]
[--netflow-engine|-E] <engine> | Specify the engine type and id.
| The format is engineType:engineId.
| [default=0:146] where engineId is a
| random number.
[--min-num-flows|-m] <min # flows> | Minimum number of flows per packet
| unless an expired flow is queued
| for too long (see -l) [default=30
| for v5, dynamic for v9]
[--sender-address|-q] <host:port> | Specifies the address:port of the flow
| sender. This optionis useful for hosts
| with multiple interfaces or if flows
| must be emitted from a static port
[--sample-rate|-S] <pkt rate>:<flow rate>
| Packet capture sampling rate and flow
| sampling rate. If <pkt rate> starts with ‘@’
| it means that nprobe will report the specified
| sampling rate but will not sample itself
| as incoming packets are already sampled
| on the specified capture device at the
| specified rate. Default: 1:1 [no sampling]
[--as-list|-A] <AS list> | GeoIP file containing the list of known ASs.
| Example: GeoIPASNum.dat
[--city-list] <City list> | GeoIP file containing the city/IP mapping.
| Example: GeoLiteCity.dat
[--pid-file|-g] <PID file> | Put the PID in the specified file
[--flow-templ|-T] <flow template> | Specify the NFv9 template (see below).
[--flow-templ-id|-U] <templ. id> | Specify the NFv9 template identifier
| [default: 257]
[--flow-version|-V] <version> | NetFlow Version: 5=v5, 9=v9, 10=IPFIX
[--flows-intra-templ|-o] <num> | Specify how many flow pkts are exported
| between template exports [default: 10]
[--local-networks|-L] <networks> | Specify the local networks (see -c
| and -r options)
[--local-hosts-only|-c] | All the IPv4 hosts outside the local
| network lists will be set to 0.0.0.0
| (-L must be specified before -c).
| This reduces the load on the probe
| instead of discarding flows on the
| collector side.
[--local-traffic-direction|-r] | All the traffic going towards
| the local networks (-L must also be
| specified before -r) is assumed incoming
| traffic all the rest is assumed outgoing
| (see also -u and -Q).
[--src-mac-address|-1] <MAC>@<ifIdx>| Flow source MAC address (see below)
[--count|-2] <number> | Capture a specified number of packets
| and quit (debug only)
[--collector-port|-3] <port> | NetFlow/sFlow collector port for incoming flows
[--tunnel|-5] | Compute flows on tunneled traffic rather than
| on the external envelope
[--no-promisc|-6] | Capture packets in non-promiscuous mode
[--smart-udp-frags|-7] | Ignore UDP fragmented packets with fragment offset
| greater than zero, and compute the fragmented
| packet length on the initial fragment header.
[--ipsec-auth-data-len|-8] <len> | Length of the authentication data of IPSec
| in tunnel mode. If not set, IPSec will not be decoded
[--dump-stats|-9] <path> | Periodically dump traffic stats into the
| specified file
–black-list <networks> | All the IPv4 hosts inside the networks
| black-list will be discarded.
| This reduces the load on the probe
| instead of discarding flows on the
| collector side.
–pcap-file-list <filename> | Specify a filename containing a list
| of pcap files.
| If you use this flag the -i option will be
| ignored.
–csv-separator <separator> | Specify the separator for text files (see -P)
| Default is ‘|’ (pipe)
–fastbit <dir> | Base directory where FastBit files will be created.
–fastbit-rotation <mins> | Every <mins> minutes a new FastBit sub-directory is created
| so that each directory contains at most <mins> minutes.
| Default 5 min(s).
–fastbit-template <flow template> | Fields that will be dumped on FastBit partition. Its syntax
| is the same as the -T flag. If this flag is not specified,
| all the specified flow elements (-T) will be dumped.
–fastbit-index <flow template> | Index each directory containing FastBit files as soon as
| the directory has been dumped. The flow template specifies
| which columns will be indexed. Its syntax is the same as
| the -T flag. This option requires that fbindex application
| is installed or built. If this flag is not specified, all
| columns will be indexed.
–fastbit-exec <cmd> | Execute the specified command after a directory has been
| dumped (and optionally indexed). The command must take an
| argument that is the path to the directory just dumped.
–bi-directional | Force flows to be bi-directional. This option
| is not supported by NetFlow V5 that by nature
| supports only mono-directional flows
–account-l2 | NetFlow accounts IP traffic only, not counting
| L2 headers. Using this option the L2 headers
| are also accounted
–dump-metadata <file> | Dump flow metadata into the specified file
| and quit
–event-log <file> | Dump relevant activities into the specified log file

Further plugin available command line options
—————————————————
30/Sep/2010 21:18:42 [plugin.c:145] Loading plugins [.so] from ./plugins
30/Sep/2010 21:18:42 [dbPlugin.c:72] Initializing DB plugin
[BGP Update Listener]
–bgp-port <port> | TCP port on which BGP updates will be sent

[MySQL DB]
–mysql=<host>:<dbname>:<table_prefix>:<user>:<pw> | Enable MySQL database support configuration
–mysql-skip-db-creation | Skip database schema creation

[DNS Protocol Dissector]
–dns-dump-dir <dump dir> | Directory where DNS logs will be dumped

[HTTP Protocol Dissector]
–http-dump-dir <dump dir> | Directory where HTTP logs will be dumped
–http-exec-cmd <cmd> | Command executed whenever a directory has been dumped
–dont-hash-cookies | Dump cookie string instead of cookie hash
–dont-nest-dump-dirs | Don’t create subdirs on the dump directory
–max-http-log-lines | Max number of lines per log file (default 10000)

[MySQL Plugin]
–mysql-dump-dir <dump dir> | Directory where MySQL logs will be dumped
–mysql-exec-cmd <cmd> | Command executed whenever a directory has been dumped
–max-mysql-log-lines | Max number of lines per log file (default 10000)

Note on interface indexes and (router) MAC addresses
—————————————————
When -u and -Q are specified, it is possible to also specify -1 (even multiple
times) for simulating a router running nProbe™. In this case nProbe™ works
as follows:

[Use Case] -u 1 -Q 2 -1 AA:BB:CC:DD:EE:FF@3 -1 11:22:33:44:55:66@4
All the flows have direction 1->2 except those who are originated
from MAC AA:BB:CC:DD:EE:FF that have 3 as source interface id
and those who are originated from 11:22:33:44:55:66 that have
4 as source interface (direction = flow interface index)

NetFlow v9/IPFIX format [-T]
—————-
The following options can be used to specify the format:

ID Flow Label Description
————————————————
[ 1] %IN_BYTES Incoming flow bytes (src->dst)
[ 1] %SYSTEM_ID
[ 2] %IN_PKTS Incoming flow packets (src->dst)
[ 2] %INTERFACE_ID
[ 3] %FLOWS Number of flows
[ 3] %LINE_CARD
[ 4] %PROTOCOL IP protocol byte
[164] %PROTOCOL_MAP IP protocol name
[ 4] %NETFLOW_CACHE
[ 5] %SRC_TOS Type of service byte
[ 5] %TEMPLATE_ID
[ 6] %TCP_FLAGS Cumulative of all flow TCP flags
[ 7] %L4_SRC_PORT IPv4 source port
[167] %L4_SRC_PORT_MAP IPv4 source port symbolic name
[ 8] %IPV4_SRC_ADDR IPv4 source address
[ 9] %IPV4_SRC_MASK IPv4 source subnet mask (/<bits>)
[ 10] %INPUT_SNMP Input interface SNMP idx
[ 11] %L4_DST_PORT IPv4 destination port
[171] %L4_DST_PORT_MAP IPv4 destination port symbolic name
[ 12] %IPV4_DST_ADDR IPv4 destination address
[ 13] %IPV4_DST_MASK IPv4 dest subnet mask (/<bits>)
[ 14] %OUTPUT_SNMP Output interface SNMP idx
[ 15] %IPV4_NEXT_HOP IPv4 next hop address
[ 16] %SRC_AS Source BGP AS
[ 17] %DST_AS Destination BGP AS
[ 21] %LAST_SWITCHED SysUptime (msec) of the last flow pkt
[ 22] %FIRST_SWITCHED SysUptime (msec) of the first flow pkt
[ 23] %OUT_BYTES Outgoing flow bytes (dst->src)
[ 24] %OUT_PKTS Outgoing flow packets (dst->src)
[ 27] %IPV6_SRC_ADDR IPv6 source address
[ 28] %IPV6_DST_ADDR IPv6 destination address
[ 29] %IPV6_SRC_MASK IPv6 source mask
[ 30] %IPV6_DST_MASK IPv6 destination mask
[ 32] %ICMP_TYPE ICMP Type * 256 + ICMP code
[ 34] %SAMPLING_INTERVAL Sampling rate
[ 35] %SAMPLING_ALGORITHM Sampling type (deterministic/random)
[ 36] %FLOW_ACTIVE_TIMEOUT Activity timeout of flow cache entries
[ 37] %FLOW_INACTIVE_TIMEOUT Inactivity timeout of flow cache entries
[ 38] %ENGINE_TYPE Flow switching engine
[ 39] %ENGINE_ID Id of the flow switching engine
[ 40] %TOTAL_BYTES_EXP Total bytes exported
[ 41] %TOTAL_PKTS_EXP Total flow packets exported
[ 42] %TOTAL_FLOWS_EXP Total number of exported flows
[ 56] %IN_SRC_MAC Source MAC Address
[ 57] %OUT_DST_MAC Destination MAC Address
[ 58] %SRC_VLAN Source VLAN
[ 59] %DST_VLAN Destination VLAN
[ 60] %IP_PROTOCOL_VERSION [4=IPv4][6=IPv6]
[ 61] %DIRECTION [0=ingress][1=egress] flow
[ 62] %IPV6_NEXT_HOP IPv6 next hop address
[ 70] %MPLS_LABEL_1 MPLS label at position 1
[ 71] %MPLS_LABEL_2 MPLS label at position 2
[ 72] %MPLS_LABEL_3 MPLS label at position 3
[ 73] %MPLS_LABEL_4 MPLS label at position 4
[ 74] %MPLS_LABEL_5 MPLS label at position 5
[ 75] %MPLS_LABEL_6 MPLS label at position 6
[ 76] %MPLS_LABEL_7 MPLS label at position 7
[ 77] %MPLS_LABEL_8 MPLS label at position 8
[ 78] %MPLS_LABEL_9 MPLS label at position 9
[ 79] %MPLS_LABEL_10 MPLS label at position 10
[148] %FLOW_ID Serial Flow Identifier
[NFv9 57552][IPFIX 35632.80] %FRAGMENTS Number of fragmented flow packets
[NFv9 57554][IPFIX 35632.82] %CLIENT_NW_DELAY_SEC Network latency client <-> nprobe (sec)
[NFv9 57555][IPFIX 35632.83] %CLIENT_NW_DELAY_USEC Network latency client <-> nprobe (usec)
[NFv9 57556][IPFIX 35632.84] %SERVER_NW_DELAY_SEC Network latency nprobe <-> server (sec)
[NFv9 57557][IPFIX 35632.85] %SERVER_NW_DELAY_USEC Network latency nprobe <-> server (usec)
[NFv9 57558][IPFIX 35632.86] %APPL_LATENCY_SEC Application latency (sec)
[NFv9 57559][IPFIX 35632.87] %APPL_LATENCY_USEC Application latency (usec)
[NFv9 57570][IPFIX 35632.98] %ICMP_FLAGS Cumulative of all flow ICMP types
[NFv9 57573][IPFIX 35632.101] %SRC_IP_COUNTRY Country where the src IP is located
[NFv9 57574][IPFIX 35632.102] %SRC_IP_CITY City where the src IP is located
[NFv9 57575][IPFIX 35632.103] %DST_IP_COUNTRY Country where the dst IP is located
[NFv9 57576][IPFIX 35632.104] %DST_IP_CITY City where the dst IP is located
[NFv9 57577][IPFIX 35632.105] %FLOW_PROTO_PORT L7 port that identifies the flow protocol or 0 if unknown
[NFv9 57578][IPFIX 35632.106] %TUNNEL_ID Tunnel identifier (e.g. GTP tunnel Id) or 0 if unknown
[NFv9 57579][IPFIX 35632.107] %LONGEST_FLOW_PKT Longest packet (bytes) of the flow
[NFv9 57580][IPFIX 35632.108] %SHORTEST_FLOW_PKT Shortest packet (bytes) of the flow
[NFv9 57581][IPFIX 35632.109] %RETRANSMITTED_IN_PKTS Number of retransmitted TCP flow packets (src->dst)
[NFv9 57582][IPFIX 35632.110] %RETRANSMITTED_OUT_PKTS Number of retransmitted TCP flow packets (dst->src)
[NFv9 57583][IPFIX 35632.111] %OOORDER_IN_PKTS Number of out of order TCP flow packets (dst->src)
[NFv9 57584][IPFIX 35632.112] %OOORDER_OUT_PKTS Number of out of order TCP flow packets (dst->src)
[NFv9 57585][IPFIX 35632.113] %UNTUNNELED_PROTOCOL Untunneled IP protocol byte
[NFv9 57586][IPFIX 35632.114] %UNTUNNELED_IPV4_SRC_ADDR Untunneled IPv4 source address
[NFv9 57587][IPFIX 35632.115] %UNTUNNELED_L4_SRC_PORT Untunneled IPv4 source port
[NFv9 57588][IPFIX 35632.116] %UNTUNNELED_IPV4_DST_ADDR Untunneled IPv4 destination address
[NFv9 57589][IPFIX 35632.117] %UNTUNNELED_L4_DST_PORT Untunneled IPv4 destination port

Plugin BGP Update Listener templates:
[NFv9 57762][IPFIX 35632.290] %SRC_AS_PATH_1 Src AS path position 1
[NFv9 57763][IPFIX 35632.291] %SRC_AS_PATH_2 Src AS path position 2
[NFv9 57764][IPFIX 35632.292] %SRC_AS_PATH_3 Src AS path position 3
[NFv9 57765][IPFIX 35632.293] %SRC_AS_PATH_4 Src AS path position 4
[NFv9 57766][IPFIX 35632.294] %SRC_AS_PATH_5 Src AS path position 5
[NFv9 57767][IPFIX 35632.295] %SRC_AS_PATH_6 Src AS path position 6
[NFv9 57768][IPFIX 35632.296] %SRC_AS_PATH_7 Src AS path position 7
[NFv9 57769][IPFIX 35632.297] %SRC_AS_PATH_8 Src AS path position 8
[NFv9 57770][IPFIX 35632.298] %SRC_AS_PATH_9 Src AS path position 9
[NFv9 57771][IPFIX 35632.299] %SRC_AS_PATH_10 Src AS path position 10
[NFv9 57772][IPFIX 35632.300] %DST_AS_PATH_1 Dest AS path position 1
[NFv9 57773][IPFIX 35632.301] %DST_AS_PATH_2 Dest AS path position 2
[NFv9 57774][IPFIX 35632.302] %DST_AS_PATH_3 Dest AS path position 3
[NFv9 57775][IPFIX 35632.303] %DST_AS_PATH_4 Dest AS path position 4
[NFv9 57776][IPFIX 35632.304] %DST_AS_PATH_5 Dest AS path position 5
[NFv9 57777][IPFIX 35632.305] %DST_AS_PATH_6 Dest AS path position 6
[NFv9 57778][IPFIX 35632.306] %DST_AS_PATH_7 Dest AS path position 7
[NFv9 57779][IPFIX 35632.307] %DST_AS_PATH_8 Dest AS path position 8
[NFv9 57780][IPFIX 35632.308] %DST_AS_PATH_9 Dest AS path position 9
[NFv9 57781][IPFIX 35632.309] %DST_AS_PATH_10 Dest AS path position 10

Plugin DNS Protocol Dissector templates:
[NFv9 57677][IPFIX 35632.205] %DNS_QUERY DNS QUERY
[NFv9 57678][IPFIX 35632.206] %DNS_QUERY_ID DNS query transaction Id
[NFv9 57679][IPFIX 35632.207] %DNS_QUERY_TYPE DNS query type (e.g. 1=A, 2=NS..)
[NFv9 57680][IPFIX 35632.208] %DNS_RET_CODE DNS return code (e.g. 0=no error)
[NFv9 57681][IPFIX 35632.209] %DNS_NUM_ANSWER DNS # of returned answers

Plugin dump templates:
[NFv9 57592][IPFIX 35632.120] %DUMP_PATH Path where dumps will be saved

Plugin HTTP Protocol Dissector templates:
[NFv9 57652][IPFIX 35632.180] %HTTP_URL HTTP URL
[NFv9 57653][IPFIX 35632.181] %HTTP_RET_CODE HTTP return code (e.g. 200, 304…)
[NFv9 57654][IPFIX 35632.182] %HTTP_REFERER HTTP Referer
[NFv9 57655][IPFIX 35632.183] %HTTP_UA HTTP User Agent
[NFv9 57656][IPFIX 35632.184] %HTTP_MIME HTTP Mime Type

Plugin L7 Protocol Recognition templates:
[NFv9 57637][IPFIX 35632.165] %L7_PROTO Symbolic layer 7 protocol description

Plugin MySQL Plugin templates:
[NFv9 57667][IPFIX 35632.195] %MYSQL_SERVER_VERSION MySQL server version
[NFv9 57668][IPFIX 35632.196] %MYSQL_USERNAME MySQL username
[NFv9 57669][IPFIX 35632.197] %MYSQL_DB MySQL database in use
[NFv9 57670][IPFIX 35632.198] %MYSQL_QUERY MySQL Query
[NFv9 57671][IPFIX 35632.199] %MYSQL_RESPONSE MySQL server response

Plugin RTP templates:
[NFv9 57622][IPFIX 35632.150] %RTP_FIRST_SSRC First flow RTP Sync Source ID
[NFv9 57623][IPFIX 35632.151] %RTP_FIRST_TS First flow RTP timestamp
[NFv9 57624][IPFIX 35632.152] %RTP_LAST_SSRC Last flow RTP Sync Source ID
[NFv9 57625][IPFIX 35632.153] %RTP_LAST_TS Last flow RTP timestamp
[NFv9 57626][IPFIX 35632.154] %RTP_IN_JITTER RTP Jitter (ms * 1000)
[NFv9 57627][IPFIX 35632.155] %RTP_OUT_JITTER RTP Jitter (ms * 1000)
[NFv9 57628][IPFIX 35632.156] %RTP_IN_PKT_LOST Packet lost in stream
[NFv9 57629][IPFIX 35632.157] %RTP_OUT_PKT_LOST Packet lost in stream
[NFv9 57630][IPFIX 35632.158] %RTP_OUT_PAYLOAD_TYPE RTP payload type
[NFv9 57631][IPFIX 35632.159] %RTP_IN_MAX_DELTA Max delta (ms*100) between consecutive pkts
[NFv9 57632][IPFIX 35632.160] %RTP_OUT_MAX_DELTA Max delta (ms*100) between consecutive pkts

Plugin SIP templates:
[NFv9 57602][IPFIX 35632.130] %SIP_CALL_ID SIP call-id
[NFv9 57603][IPFIX 35632.131] %SIP_CALLING_PARTY SIP Call initiator
[NFv9 57604][IPFIX 35632.132] %SIP_CALLED_PARTY SIP Called party
[NFv9 57605][IPFIX 35632.133] %SIP_RTP_CODECS SIP RTP codecs
[NFv9 57606][IPFIX 35632.134] %SIP_INVITE_TIME SIP SysUptime (msec) of INVITE
[NFv9 57607][IPFIX 35632.135] %SIP_TRYING_TIME SIP SysUptime (msec) of Trying
[NFv9 57608][IPFIX 35632.136] %SIP_RINGING_TIME SIP SysUptime (msec) of RINGING
[NFv9 57609][IPFIX 35632.137] %SIP_OK_TIME SIP SysUptime (msec) of OK
[NFv9 57610][IPFIX 35632.138] %SIP_BYE_TIME SIP SysUptime (msec) of BYE
[NFv9 57611][IPFIX 35632.139] %SIP_RTP_SRC_IP SIP RTP stream source IP
[NFv9 57612][IPFIX 35632.140] %SIP_RTP_SRC_PORT SIP RTP stream source port
[NFv9 57613][IPFIX 35632.141] %SIP_RTP_DST_IP SIP RTP stream dest IP
[NFv9 57614][IPFIX 35632.142] %SIP_RTP_DST_PORT SIP RTP stream dest port

Plugin SMTP Protocol Dissector templates:
[NFv9 57657][IPFIX 35632.185] %SMTP_MAIL_FROM Mail sender
[NFv9 57658][IPFIX 35632.186] %SMTP_RCPT_TO Mail recipient

Any standard NetFlow collector including ntop can be used to analyse the flows generated by nProbe™ (please note that not all the commercial collecotrs support v9).
When used with ntop, the nProbe™ can act as a remote and light traffic probe, and ntop as a central network monitoring console for IPFIX/v5/v9.

 

FAQ

  1. Q: Is nProbe™ able to operate on Gbit networks at full speed?
    A: Yes. Note that for exploiting the Gbit packet capture you need a 64-bit PCI Gigabit Ethernet interface.
  2. Q: Is the nProbe™ source code available?
    A: Yes of course, it’s GPL.
  3. Q: Why do you charge for nProbe™ although it’s GPL?
    A: GPL has nothing to do with price ([1] [2] [3]) but with freedom. Many open source companies ask a fee for their software.
  4. Q: What do you do with the money you get charging for nProbe™?
    A: This money is invested for doing research in ntop, nBox and nProbe™ projects.

 

Credits

NetFlow is copyright Cisco Systems.
nProbe™ is a trademark registered in USA and the European Union.

 

Get It

nProbe™ is available in three flavours.

Version Unix Win32
Standard Probe with no plugins
Pro Same as standard version with native PF_RING support (i.e. full packet capture acceleration), Fastbit support and some plugins (database, dump and layer-7 inspection plugin). Not available
Pro with Plugins Same as standard version with native PF_RING, Fastbit support and all plugins; SIP, RTP, BGP, MySQL, SMTP. Same of Unix with no BGP plugin

In addition to the above versions you can also get extra plugins.

Plugin Unix Win32
HTTP Plugin Available
as add-on		 Integrated in the nProbe Pro version.
NetFlow-Lite Plugin Available
as add-on		 The nprobe HTTP plugin is integrated in the nProbe Pro version. The kernel PF_RING plugin for accelerationg NetFlow-Lite is not available for Win32.

Note that the Win32 version of nProbe is available as binary version, whereas the Unix version is available as source.

nProbe™ is available under the GPLv2 licence for a little fee, that’s used for running the project and funding the new developments. You can purchase online your copy of nProbe™ at the ntop e-shop site, that includes one year support. After the transaction is completed you can download your nProbe™ copy immediately.

If you are an existing nProbe™ owner, you can get the standard version and we’ll give you a free upgrade to the professional version (do not forget to send us a mail after completing your transaction).

If you want to test drive nProbe™ you can fetch a demo copy limited to 2,000 flows export.

If you are a no profit institution or a university, you can have nProbe™ at no cost (even if your donations are welcome): please drop us a mail from your university account where you explain why you qualify (emails originating from non-university account including hotmail, gmail and yahoo will be ignored).

Note that for nProbe™ OEM, reselling, repackaging (including device embed) you need a written commercial licence that’s available on request from its author. This because this is considered as derived work as specified in the GPL license.