PF_RING has been designed for enhancing packet capture performance. This means that the RX path must be accelerated, and in particular a way to accelerate this is by reducing the journey of the packet from the adapter to userland. This is obtained by allowing the driver to push the packet from the NIC to PF_RING directly and not through the usual kernel path. For this reason PF_RING has introduced an option named “transparent mode” whose goal is to tune how packets are moved from the NIC to PF_RING. This option (that can be specified when inserting the PF_RING module via insmod) can have three values:

1. insmod pf_ring.ko transparent_mode=0
   This is the default and it means that packets are sent to PF_RING via the standard kernel mechanisms. In this setup the packets are both sent to PF_RING but to all other kernel components. All NIC drivers support this mode.
2. insmod pf_ring.ko transparent_mode=1
   In this mode, packets are sent directly by the NIC driver to PF_RING, packets are still propagated to other kernel components. In this mode packet capture is accelerated because packets are copied by the NIC driver without passing through the usual kernel path. Please note that in order to enable this mode, you must use a NIC driver that supports PF_RING. Available PF_RING-enabled drivers can be found in the drivers/ directory of PF_RING.
3. insmod pf_ring.ko transparent_mode=2
   In this mode, packets are sent directly by the NIC driver to PF_RING, packets are not propagated to other kernel components as this slows down packet capture. Please note that:
   • in order to enable this mode, you must use a NIC driver that supports PF_RING.
   • Packets are not sent to the kernel after they have been delivered to PF_RING. This means that you won’t have connectivity from NICs driven by PF_RING-aware drivers.
   • This mode is the fastest one as packets are quickly copied to PF_RING and immediately discarded after they have been processed.

This is to announce the availability of PF_RING DNA (Direct NIC Access) that significantly increments performance (up to 80%) when compared with Linux packet capture and PF_RING (non DNA).

PF_RING is polling packets from NICs by means of Linux NAPI. This means that NAPI copies packets from the NIC to the PF_RING circular buffer, and then the userland application reads packets from ring. In this scenario, there are two pollers, both the application and NAPI and this results in CPU cycles used for this polling; the advantage is that PF_RING can distribute incoming packets to multiple rings (hence multiple applications) simultaneously.

PF_RING DNA (Direct NIC Access) is a way to map NIC memory and registers to userland so that packet copy from the NIC to the DMA ring is done by the NIC NPU (Network Process Unit) and not by NAPI. This results in better performance as CPU cycles are used uniquely for consuming packets and not for moving them off the adapter. The drawback is that only one application at time can open the DMA ring, or in other words that applications in userland need to talk each other in order to distribute packets.

In a nutshell if you like flexibility you should use PF_RING, if you want pure speed PF_RING DNA is the solution. Please note that in DNA mode NAPI polling does not take place, hence PF_RING features such as reflection and packet filtering are not supported.

For more information, please have a look at the PF_RING home page.

Suricata is the next generation open source IDS/IPS developed byt the Open Information Security Foundation. It is a pleasure to announce that ntop has joined the core development team as the Linux version of Suricata is based on acceleration provided by PF_RING. In the near future PF_RING will be extended so that it can also accelerate packet transmission in order to move the Suricata IPS performance to the next level.

More information can be found here.

We’re proud to announce that ntop.org has been accredited as Endace technology partner as recognition for ntop contribution in the open-source world and also as guarantee for Endace customers that products such as ntop and nProbe run smoothly (and faster) on Endace DAG cards.

This article Improvement of libpcap for lossless packet capturing in Linux using PF_RING kernel patch positions PF_RING (3.x, so some changes are needed when using version 4) against the Linux standard PF_PACKET packet capture facility. In PF_RING v4, due to popular demand, I have decided to move some of the PF_RING accelerations into the NIC driver with the advantage of being now able to compile PF_RING against an unpatched kernel. The PF_RING distribution has now a drivers/ directory that contains accelerated drivers for popular 1 and 10 Gbit adapter. This means that using PF_RING on top of Linux without any accelerated drivers, gives you a little speed advantage when compared with standard Linux. However if you use a PF_RING-aware driver or even better TNAPI, your speed bump will be much better. I summarize some lessons learnt on this field on this research paper named Exploiting Commodity Multi-core Systems for Network Traffic Analysis.

ntop supports NetFlow since many years including the latest v9/IPFIX versions. In 2005 Cisco ha releases a new line of  x86 based security devices named ASA that unfortunately do not fully comply with NetFlow specs. In particular it seems that they do not periodically export templates as the standard requires. This means that ntop is unable to decode all flow records, not because ntop is bugged, but because of this lack of standard compliance. As ASA are pretty used in the industry, future ntop versions will support them with ad-hoc hacks, this until Cisco does not update its NetFlow implementation.

References

• ASA White Paper
• NetFlow ASA Implementation

In order to analyze network traffic, it’s necessary to feed ntop/nProbe with network packets. There are two solutions to the problem:

• port mirror (also called SPAN in Cisco parlance)
• network tap

Prior to explain the differences between these two solutions, it’s important to understand how ethernet works. In 100 Mbit and above, hosts usually speak in full duplex meaning that a hosts can both send and receive simultaneously. This means that on a 100 Mbit cable connected to a host, the total amount of traffic that a host can send/receive is 2 x 100 Mbit = 200 Mbit.

A port mirror is active packet duplication, meaning that a network device physically has the duty to copy packets onto a mirror port.

This means that the device has to carry on this task by using some resources (e.g. CPU) and that both traffic directions will be copied into the same port. As explained before, in full duplex links this means that the total of A->B and B->A can’t exceed the network speed before packet loss occurs. This because there’s physically no space to copy packets. The consequence is that a port mirror is a great technique as it can be performed by many switches (but not all) with the drawback of packet loss if you monitor a link with over 50% load, or mirror the ports onto a faster port (e.g. mirror 100 Mbit ports onto a 1 Gbit port). Not to mention that packet mirror might require switch resources that can load the unit and lead to reduced switching performance. Note that you can 1 port onto a port, or 1 VLAN onto 1 port, but usually you can’t copy many ports to 1. Please note that in this architecture, the switch is a point of failure as if it breaks the connectivity (and packet mirror) is lost.

A network tap instead is a fully passive device.

Electrically or optically (e.g. using a prism) packets are copied onto the tap ports. Unless you use an aggregation tap, a tap has one tap port per direction. This means that in order to monitor 1 link, you need 2 NICs, one for the first and the other for the second direction. In this case you won’t ever loose a single packet as in the mirror case, because each direction is using a link, instead of merging both directions onto the same link. If you want to merge both directions into one single port, you can use a network switch as depicted below:

As tap ports do not receive but transmit only, the switch has no clue who’s sitting behind the ports. The consequence is that it broadcast the packets to all ports. So if you connect your monitoring device to the switch, such device will receive all packets. Note that this mechanism works if the monitoring device does not send any packet to the switch, otherwise the switch will assume that the tapped packets are not for such device. In order to achieve that, you can either use a network cable on which you have not connected the TX wires, or use an IP-less (and DHCP-less) network interface that does not transmit packets at all. Finally note that if you want to use a tap for not loosing packets, then either don’t merge directions or use a switch where tapped directions are slower (e.g. 100 Mbit) that the merge port (e.g. 1 Gbit).

Bottom line. Packet mirror and tap have both pros and cons. You can select the right solution based on your network (and wallet).

If you want to know more about this topic, I suggest you to read this document named Multi-Tap Network Packet Capturing. It describes more in detail how to play with network taps and it highlights how to use NST (Network Security Toolkit, a Linux distribution that also includes ntop) in common network monitoring scenarios, where very likely your setup is also included.

Multi-Tap Network Packet Capturing

On Linux, interrupts are handled automatically by the kernel. In particular there’s a process named irqbalancer that is responsible for balancing interrupts across processors. Unfortunately the default is to let all processors handle interrupts with the result that the overall performance is not optimal in particular on multi-core systems. This is because modern NICs can have multiple RX queues that work perfectly if cache coherency is respected. This means that interrupts for ethX-RX queue Y must be sent to 1 (one) core or at most to a core and its Hyper-Threaded (HT) companion. If multiple processors handle the same RX queue, this invalidates the cache and performance gets worse. For this reason IRQ balancing is the key for performance. In particular what I suggest is to have 1/2 (in case of HT) core(s) handle the same interrupt. For this reason on Linux interrupts are usually send to all processors hence /prox/irq/X/smp_affinity is set to ffffffff that means all processors. Instead as I have just stated above it’s better to avoid that all processors handle all interrupts. Example

Read the rest of this entry »