ntop tools are able to collect various type of flows NetFlow/IPFIX (including dialects such as J-Flow, NetStream) and sFlow/NetFlowLite, this in addition to packet capture/processing. We have decided to seamlessly handle all these formats so that the user does not have to know the inner details of them. so what you do is the usual pipeline
where nProbe collects flow from devices (i.e. router or switch) or turns packets into flows. In both cases nProbe will deliver this information to ntopng by enriching the exported flows with additional data (e.g. nDPI) with respect to the original flow and removing differences introduce by the various flow formats used by some vendors.
In the latest (dev) version of nProbe/ntopng, we have made some enhancements for users who collect flows. In the flow details we have added a new badge that should you whether the flow displayed originates from NetFlow/IPFIX or sFlow. In case the flow is computed by packet capture (either on nProbe or ntopng side) no badge is displayed.
This way users who collect traffic from multiple exporters on the same interface, can figure out the nature of the flow.
Packet sampling is optional in NetFlow/IPFIX whereas is compulsory on sFlow. This means that when a flows is calculated on the flow device (i.e. router or switch), in case of packet sampling only a subset of the packets is use to account traffic. Packet sampling is a smart technique for reducing load on the monitoring device but it is handled differently based on the protocols:
- NetFlow/IPFIX exporters usually do not store in exported flows the packet sampling rate (if used). If the router has a configured sampling rate of 1:x, it means that all the flows accounted on the device have the same sampling rate. As the used sampling rate is not always present in exported flows, you need to add “-S 1:x:1”. In case of unsampled NetFlow/IPFIX flows you do not need to specify -S at all.
- sFlow is a bit more complex, and it has a variable sampling rate per exported flow. In this case, -S is not necessary as nProbe will automatically upscale traffic using the information part of the sFlow export.
In summary the good news is that nProbe can seamlessly take care care of flow sampling. The only caveat is that if you collect sampled NetFlow/IPFIX flows generated by multiple exporters with a single nProbe instance, as -S is a global option for all exporters, you need to make sure that all the exporters use the same sampling rate. If this is not the case, please start multiple nProbe instances, one per flow sampling rate as in the picture below.
As already stated, ntopng can seamless handle all the above flows as nProbe automatically upscales traffic to the correct sampling rate. This means that bytes/packets are correctly reported and multiplied transparently. Instead, when ntopng calculates the flow throughput it has to know more about the flow origin as explained below.
In the above flow view you see the actual throughput, and inside the flow details you can see the current, peak and average (for the whole flow duration) throughput.
Actual throughout is computed as follows:
- Packet interfaces: periodically (typically every 5 seconds) ntopng computes a bytes/packet delta with respect to the previous period and calculates the throughput.
- sFlow Flows: in this case a sFlow flow is basically a single packet upscaled with the specified dynamic flow sampling rate. For sFlow, the throughput is computed as for the packet interface because each received flows is basically a single packet and sFlow does not export any timing information except the time when the packet was sampled.. This is not super accurate as in sFlow we do not have the guarantee to receive a constant flow data stream for a given flow, hence it is more accurate for flows that receive more samples with respect to flows that have received only a single sFlow sample.
- NetFlow/IPFIX flows: the throughput is computed whenever a (sub-)flow is received by ntopng. As flows contain the flow start/end, NetFlow/IPFIX throughout is computed as the average throughput of the flow. In other words is is computed as bytes/(flow_end – flow_start + 1). For this reason, the throughout of NetFlow/IPFIX flows is updated whenever a new (sub)flow is received and not periodically.
If you read until here, we hope that all the above concepts are now clear. This post applies to nProbe/ntopng part of the dev branch. The above enhancements will be included in the next stable release.
Enjoy !