ntopng can be used to visualize traffic data that has been generated or collected by nProbe. Using ntopng with nProbe is convenient in several scenarios, including:
- The visualization of NetFlow/sFlow data originated by routers, switches, and network devices in general. In this scenario, nProbe collects and parse NetFlow/sFlow traffic from the devices, and send the resulting flows to ntopng for the visualization.
- The monitoring of physical network interfaces that are attached to remote systems. In this scenario, ntopng cannot directly monitor network interfaces nor it can see their packets. One or multiple nProbe can be used to capture remote network interfaces traffic and send the resulting flows towards a central ntopng for the analysis and visualization.
The following picture summarizes the two scenarios highlighted above and demonstrates that they can also be combined together.
This post complements the extensive documentation already available at https://www.ntop.org/guides/ntopng/case_study/using_with_nprobe.html and wants to serve as a quick memorandum to effectively deploy ntopng and nProbe for the collection of flows.
Multiple nProbe to One ntopng
Collecting flows from multiple nProbe using a single ntopng can be useful to have a single place that is in charge of visualizing and archiving traffic data.
To collect flows from multiple nProbe, ntopng has to be started with an extra c (that sands for collector) at the end of the ZMQ endpoint, whereas every nProbe needs option --zmq-probe-mode. In this configuration, the nProbes initiate the connection towards ntopng that acts as a server, and not the vice versa. Therefore, you must ensure ntopng is listening on the ANY address (that is, the wildcard * in the ZMQ endpoint address) or on another address that is reachable by the various nProbe.
An example of such configuration is the following
ntopng -i tcp://*:5556c nprobe --zmq "tcp://<ip address of ntopng>:5556" --zmq-probe-mode -i eth1 -n none -T "@NTOPNG@" nprobe --zmq "tcp://<ip address of ntopng>:5556" --zmq-probe-mode -i none -n none --collector-port 2055 -T "@NTOPNG@" nprobe --zmq "tcp://<ip address of ntopng>:5556" --zmq-probe-mode -i none -n none --collector-port 6343 -T "@NTOPNG@"
NAT
IP reachability of nProbe and ntopng cannot always be taken for granted. Sometimes, it could be necessary for an ntopng to collect flows from an nProbe in a separate network, possibly behind a NAT or even shielded with a firewall. Similarly, it could be necessary for an ntopng behind a NAT to collect flows from an nProbe in another network. Luckily, to handle these scenarios, ntopng (and nProbe) can be configured to act as either the client or the server of the JSON-Over-ZMQ communication, interchangeably. This avoids the insertion of lengthy, time-consuming, and possibly insecure rules in network devices, as it is enough to ensure the client can reach the server, while NATs will automatically handle the returning server-to-client part of the communication.
When both nProbe and ntopng are on the same network, or when ntopng is in another network but can reach nProbe, the following configuration should be used
ntopng -i tcp://<ip address of nProbe>:5556 nprobe --zmq "tcp://*:5556" -i eth1 -n none -T "@NTOPNG@"
When ntopng cannot reach nProbe, but nProbe can reach ntopng the configuration that should be used is
ntopng -i tcp://*:5556c nprobe --zmq "tcp://<ip address of ntopng>:5556" --zmq-probe-mode -i eth1 -n none -T "@NTOPNG@"
Note that changing the client/server roles of ntopng and nProbe does not affect the subsequent flow collection so both configurations can be used interchangeably.
Templates
The set of flow fields that are sent from nProbe to ntopng is controlled using option -T. All fields listed there will be sent to ntopng. The macro @NTOPNG@ is be used as an alias for a minimum set of fields necessary for ntopng to operate correctly. The macro can be combined with additional fields as shown in this example
nprobe --zmq "tcp://*:5556" -i eth1 -n none -T "@NTOPNG@ %IN_SRC_MAC %OUT_DST_MAC"
Symmetric-Key Encryption
A symmetric key can be used to encrypt the stream of flows that go from nProbe to ntopng. The key needs to be specified both at ntopng and nProbe using option --zmq-encrypt-pwd
ntopng -i "tcp://127.0.0.1:1234" --zmq-encrypt-pwd myencryptionkey nprobe -i eth0 -n none --zmq "tcp://127.0.0.1:1234" --zmq-encrypt-pwd myencryptionkey
Data Compression
The stream of flows is automatically compressed by nProbe before being sent to ntopng. For debugging purposes, it may be useful to turn off compression. Compression can be turned off using option --zmq-disable-compression
nprobe -i eth0 -n none --zmq "tcp://127.0.0.1:1234" --zmq-disable-compression
Data Buffering
To optimize performance at high speeds, nProbe automatically buffers multiple flows into a single message. This makes it possible to achieve higher throughputs at the expense of a slightly higher latency and larger messages traveling on the network. Disabling buffering may be a good choice in low speed environments. Buffering can be disabled using option --zmq-disable-buffering
nprobe -i eth0 -n none --zmq "tcp://127.0.0.1:1234" --zmq-disable-buffering