Those of you who are familiar with kernel-bypass drivers like PF_RING ZC know that it is not possible to run multiple applications on top of the same Network interface and capture the same traffic twice. This is the case of Intel and most FPGA adapters. In fact, since the application takes full control of the adapter and configures it to copy packets directly to the application’s memory in hardware, access to the device must be exclusive. This unless the adapter natively support multiple consumers: this is the case of Mellanox/NVIDIA and Fiberblaze/Silicom adapters.
Last autumn we have introduced the iavf ZC driver, which adds support for a new range of (virtual) adapters from Intel: i40e (X710/XL710) and ice (E810) SR-IOV Virtual Functions. As we announced in the latest PF_RING changelog, this new driver paves the way for new packet capture architectures as it enables traffic duplication on Intel!
You are probable wondering what Virtual Functions have to do with traffic duplication. Well, SR-IOV on X710/XL710 adapters supports a working mode called trust mode which has been introduced to enable promiscuous capture on Virtual Functions, to allow multiple virtual interfaces to see all traffic hitting the physical interface. This means that it is now possible to duplicate (i.e. capture the packet once and resent it multiple times) and capture all traffic hitting a physical (Intel) interface from multiple Virtual Functions, in promiscuous mode, at high speed, using a ZC driver. And, in addition to this, it is possible to optionally filter traffic in hardware based on VLAN or MAC address.
A typical use case for traffic duplication is traffic analysis combined with traffic recording at high speed. In this case you would setup two virtual interfaces (Virtual Functions), each receiving a full copy of the traffic, and you would run nProbe or ntopng on one of them for traffic analysis, and n2disk on the other for traffic recording.
In order to configure the adapter to achieve this, you need to:
1. Enable SR-IOV support
Change the kernel parameters in /etc/default/grub as below.
GRUB_CMDLINE_LINUX_DEFAULT="iommu=1 msi=1 pci=assign-busses intel_iommu=on"
Note: pci=assign-busses is required to fix failures reported in dmesg as below.
i40e 0000:02:00.0 enp2s0f0: SR-IOV enabled with 1 VFs i40e 0000:02:00.0: can't enable 1 VFs (bus 03 out of range of [bus 02]) i40e 0000:02:00.0: Failed to enable PCI sriov: -12
Apply the changes and reboot.
update-grub && reboot
2. Configure the iavf ZC driver as reported in the guide, similar to any other PF_RING ZC driver.
3. Enable 2 Virtual Functions per interface.
echo '2' > /sys/bus/pci/devices/$(ethtool -i $IF | grep bus-info | cut -d ' ' -f2)/sriov_numvfs
4. Set the trust mode to enable promiscuous capture (and optionally filter traffic based on VLAN ID)
ip link set $IF vf $VF_ID vlan $VLAN_ID ip link set dev $IF vf $VF_ID trust on
Note: 3. and 4. can be automated by using a pre script as described in the guide
At this point you are ready to run two applications and capture the same traffic from both virtual interfaces using ZC without having to purchase a costly packet broker.
nprobe -i zc:eno0v0 [...] n2disk -i zc:eno0v1 [...]
Enjoy!