Precise Interface Merging Without Hardware Timestamps

Posted · Add Comment
In network monitoring it is very common to use taps for duplicating network traffic (RX and TX directions).

Taps are important as they allow network probes to operate passively without interfering with network operations. The two traffic directions (A to B and B to A) are plugged into two network ports of the probe. Having the two directions separated has advantages (e.g. packets are not mixed across directions) and disadvantages. The main disadvantage is that when reading packets from the two interfaces, it is not possible to know which packet out of the two directions arrived first. For many applications (e.g. NetFlow traffic monitoring) this is not a problem, but for some others it is necessary to know the exact order of the packets. This is because if two hosts, one on the A side and the other on the B side, communicate over  a protocol that has some kind of state, the probe must see messages in the exact order they hit the wire. If this does not happen, it might be confused and not operate as expected. We call this precise interface merging (PIM).

One of the myths about PIM is that it is necessary to use costly hardware timestamp-based NICs that are able to know the exact time (usually with a precision of 50 or 100 nsec) when incoming packets hit the probe interface. While hardware timestamps can definitively be used for PIM, we claim that they in practice are not necessary as commodity adapters with DNA and without hardware timestamps are enough for this task. We’re now going to explain why this statement holds.

With the advent of PF_RING DNA, it is possible to capture packets on multiple interfaces at line rate 1/10 Gbit with no loss. At 10 Gbit line rate, the adapter receives a packet every 67.2 nsec (1 sec /14.880.000 pps). Modern adapters as those based on the Intel 82599 chip can buffer up to 4096 packets in memory that is 275.2 usec (4096 * 67.2). As DNA operates at line rate, it means that it is able to consume packets as fast as the network (otherwise we wouldn’t be able to operate at line rate). With DNA we have written an example application named pfcount_aggregator that is able to read packets from two interfaces simulatenously. Using this application it is possible to read packets in order as two peers (one on the A and the other on the B side) that exchange data over a state-machine based protocol (i.e. A sends a request, and B replies with a response to the previous request) such as GTP can safely assume that packets are read in order by the aggregator. This is because the time it takes for a request (as seen on the first tap-ed interface) to go from A to B, and the response from B to A (as seen on the second tap-ed interface) is definitively more (in particular for a WAN) than 275.2 usec. This means that by the time the aggregator has read the request, it has also read all the following packets (up to the buffer length) and thus it’s impossible to read the response packet before the request. Basically it’s impossible for the probe to see the response before the request when an application such as pfcount_aggregator is used. Note that this is as such, because of DNA as with the NAPI-based polling model where interfaces can be polled sequentially, this property might not be true.

On a nutshell hardware timestamps and costly NICs are not necessary for PIM. This contrary to the common belief that without them PIM wouldn’t happen. Juvenal said 2000 years ago: Quis custodiet ipsos custodes? Have fun with DNA and commodity adapters.