nProbe (and ntopng) is a traditional packet-based application, whose lifecycle is
- Capture a packet and dissect/decode it
- Update the representation in memory of the network traffic (e.g. the flow table)
- Export the information
Using packets for traffic analysis has several positive things including:
- Ability to analyse traffic using a port mirror/TAP without installing and agent on every monitored host, thing that might be a nightmare if your network is heterogeneous.
- Scalability issues have been solved (e.g. see PF_RING ZC) years ago, so monitoring a 40/100G network is no longer a problem.
However while this model worked for decades and it’s still a good one, there are now some new challenges:
- As packets do not carry metadata, nProbe is unable for instance to report the user/application that produced that traffic (information that might be relevant when doing security analysis).
- Limited visibility on virtualised environments (both VM and container based) and many traffic flows are internal to the system, and inspecting packets we only see a portion of the picture being unable to monitor the traffic flows internal to the system.
One of the reasons why protocols such as SNMP failed to become popular for host-based traffic monitoring and stay limited to device (e.g. router/switch) management is the need to install an agent per system. This creates problems in particular on proprietary systems (e.g. if you have purchased a costly database system, the manufacturer will prevent you from installing software on it, or will allow it but your support/guarantee will be void) or on embedded systems such as a NAS. On the other hand people demand tools able to provide advanced network visibility, not for all systems but in particular for some of them (those that are relevant for an organisation) where monitoring data should be rich and not just packets/bytes-based.
This is the reason why we have developed nProbe agent that is a lightweight, event based (read it as non-packet based) agent, that you install on the host you want to monitor. This tool will provide you very detailed monitoring metrics including latencies, user/process/container information that can be used to complement your traditional packet-based traffic analysis provided by nProbe; this is in order to increase visibility of key company assets while avoid installing the agent on all systems, thing that might be unfeasible or simply not practical.
In the picture above you can see a typical deployment where nProbe monitors all network traffic and on key assets nProbe Agent is installed to provide detailed network visibility. Of course if you want to monitor just one important assets, you can use just the agent that can export monitoring data of the asset without having to also reply nProbe, this unless you want to monitor also additional hosts and not just such asset. The main advantages of the agent with respect to nProbe include:
- Visibility of intra-process and intra-container communications. In case of containerised environments you can install an additional container with the nProbe agent to monitor the whole system.
- Low-memory and CPU utilisation (typical CPU usage is 1-3%) that is independent from the traffic rate/volume.
- Detailed latency and environmental information (e.g. process producing a certain traffic flow) read directly from the kernel.
- You can use the nProbe Agent to provide detailed monitoring information without having to deal with packets, that is a practice that falls short on virtualised environments.
- nProbe-exported information can be complemented with metrics exported by the agent, this to implement different levels of visibility based on the importance of some specific assets.
- One nProbe Agent instance can monitor all the containers/processes running on a system using low CPU/memory resources.
- Using a hybrid nProbe/nProbe Agent approach you can selectively decide the level of visibility you need to monitor your assets.