In flow-based monitoring there are two main components: the probe (a.k.a. flow exporter) and the flow collector/analyser. Usually NetFlow/sFlow is a push mode paradigm as network devices have almost no memory/storage and thus they send out data as soon as possible towards a collector. This architecture is suboptimal as the probe is pushing the same data to all collectors (i.e. collector X cannot tell the probe that it is interested only to HTTP-based flows, but it has to collect everything and discard un-needed information) and also because in case a new collector has to be added, the probe has to be reconfigured (i.e. no dynamic attach/detach). Another issue is that data exchanged is in clear, meaning that anyone intercepting flows sent by the probe, can find out what happens in the monitored network; we are aware that you can setup a dedicated VLAN/VPN to avoid this but this practice adds complexity.
ntopng has reverted this paradigm using a poll-mode architecture.
Via ZMQ ntopng dynamically subscribes to the probe, tells the probe what type of flow data it is interested in, and the probe sends ntopng only this information, without sending all flows to ntopng as probes do. This practice optimises network traffic and limits the CPU cycles to those really necessary to carry on to collect flows.
This architecture however is unable to operate in case of a NAT.
In fact, in case you run the ntopng collector on a public IP (e.g. on a cheap VPS host) and nProbe on a private network, ntopng is unable to connect to the probe and thus flow collection won’t work.
In the latest development versions of ntopng and nProbe, we have introduced several enhancements to address these issues. In particular:
- nProbe/ntopng can now operate both in pull/push mode.
- Flow information is now compressed and (optionally) encrypted: your privacy is preserved even when sending traffic over the Internet.
- ntopng now subscribes to nProbe for second-based throughput statistics, so that you will now see in ntopng realtime throughput statistics even if you configure the probe to aggregate flows to 1 min or more. In this case you need to add a ‘c’ on the ntopng side to declare it as collector endpoint (in ntopng 2.4 we used –zmq-collector-mode that has been removed as it was global to ntopng whereas with the trailing ‘c’ you can selectively decide which endpoint act as collector).
Let’s see how to use flow collection. Suppose that you run ntopng on host X and nProbe on host Y.
Poll Mode
host X> ntopng -i "tcp://Y:1234" --zmq-encrypt-pwd myencryptionkey host Y> nprobe -n none --zmq "tcp://*:1234" --zmq-encrypt-pwd myencryptionkey
Push Mode
host X> ntopng -i "tcp://*:1234c" --zmq-encrypt-pwd myencryptionkey host Y> nprobe -n none --zmq "tcp://X:1234" --zmq-probe-mode --zmq-encrypt-pwd myencryptionkey
Notes:
- All the zmq options have a double dash “-“. Example –zmq .. etc.
- The –zmq-encrypt-pwd is optional: if you set it data is encrypted with the specified symmetric key. In ntopng, in case you have configured multiple probes, the same encryption key is used for all probes (i.e. you cannot set a per-probe encryption key).
- Flows are always sent in compressed format. Space savings can range from -30/-40% up to -90%.
- ntopng now automatically subscribes to nProbe for 1 second traffic updates.
In ntopng you will now see an enhanced view of your probes, knowing not just traffic stats but also additional information such as remote probe IP (even if behind a NAT), the public IP from which flows are collected, and the interface speed being monitored by the probe.
In essence you can now see in realtime both your flows, traffic statistics and remote probe information. All using as little bandwidth as possible, protecting your flow information with encryption.
We believe that with these enhancements, we have created a very advanced flow-collection architecture, that addressed concerns of using the flow paradigm over the Internet and that gives users maximum flexibility now available with traditional probes/collectors.