Using nProbe in combination with ntopng is a common practice. The benefits of this combination are manyfold and include:
- A complete decoupling of monitoring activities (taking place on the nProbe) from visualization tasks (taking place on ntopng);
- The capability of building distributed deployments where multiple (remote) nProbe instances send monitored data towards one or more ntopng instances for visualization;
- A comprehensive support for the collection, harmonization and visualization of heterogeneous flow export protocols and technologies, including NetFlow V5/v9/V10 IPFIX and sFlow;
- Full support for any proprietary technology that sends custom fields over NetFlow V5/v9/V10 with visualization of data;
- Harmonization of diverse physical network interfaces and flow export protocols and technologies into a single, clear, JSON format sent over ZMQ to ntopng.
ntopng and nProbe communicate each other via a publish-subscribe mechanism implemented over ZMQ. Exchanged data contains both interface updates (e.g., the number of bytes and packets monitored) as well as network flows, obtained by monitoring physical interfaces (NIC cards) or by processing flow export technologies such as NetFlow.
Since flows sent over ZMQ are only those that active or recently expired, one has to store and archive them systematically for later accesses and analyses. Presently, ntopng offer rich historical flow exploration features when it is instructed to archive flows to MySQL (see part 1 and part 2 of the tutorial “Exploring Historical Data Using ntopng”). However, there are cases where MySQL flow export must be done directly on nProbe. Such cases include, but are not limited to:
- The capability of creating a database column for each nProbe template field — ntopng creates a fixed set of database columns;
- A MySQL database that is closer or more effectively reached from nProbe rather than from ntopng;
- A low-end ntopng device that couldn’t deal with massive/batched database insertions.
In the cases above, it is still desirable to have full access to the ntopng historical flow exploration features. Therefore ntopng must work seamlessly even when operating on top of a database created and used by nProbe for flow export.
Fortunately, this interoperability is accomplished transparently by mean of database table views. Just a couple of things are required. The first thing is to instruct ntopng to connect to the nprobe database using a special mysql-nprobe prefix in the -F option. The second thing is to ensure nProbe will create a minimum set of database columns as required by ntopng by specifying the macro @NTOPNG@ inside nProbe template. This macro will expand to the following set of fields:
%L7_PROTO %IPV4_SRC_ADDR %IPV4_DST_ADDR %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IP_PROTOCOL_VERSION %PROTOCOL %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %FIRST_SWITCHED %LAST_SWITCHED %SRC_VLAN
Following is a working example that illustrates ntopng and nProbe configurations. Clearly, database connection parameters (host, user and password, schema name, and table name) must be the same on both sides.
./nprobe -i eno1 -T "@NTOPNG@" --mysql="localhost:ntopng:nf:root:root" --zmq tcp://127.0.0.1:5556 --zmq-probe-mode ./ntopng -i tcp://*:5556c -F "mysql-nprobe;localhost;ntopng;nf;root;root"
Note that when ntopng operates in this mode, it won’t export flows (no one want the same flows stored twice in the database). It will just visualize them.
Happy flow hunting!