10 Gbit Hardware Packet Filtering Using Commodity Network Adapters

Prerequisites


All the concepts and code described in this page requires modern networking hardware, an Intel 82599-based (e.g. Intel X520) or Silicom Director 10 Gbit NIC and a recent Linux kernel (>= 2.6.31).

Developing efficient networking applications in Linux is not a straight forward task and requires a fair amount of understanding of the hardware and the network stack. We strongly advise you to read this white paper in order to see some pointers that are important in developing networking applications.

Introduction: About Hardware Packet Filtering


Using filters in hardware gives two great performance advantages:

  1. Allows the user to use software to set filters that direct specific flows into specific CPU cores allowing better cache utilization. This can also be achieved with software, but HW filtering in PF_RING™ allows for better performance than software filtering.
  2. Allows the dropping of packets with zero CPU impact as these packets are now dropped on the wire and never hit any software stack, once the filter is in place.

PF_RING™ supports on the fly hardware filtering for Intel© 82599-based (in detail, it supports Flow Director and Five Tuple Queue Filtering (FTQF) as described in the 82599 datasheet) and Silicom Director network controllers. The figure below shows the queue assignment based in the hardware filtering of 82599 and PF_RING™.

How to Enable Hw Packet Filtering on Intel 82599-based NICs


Assuming the hardware is installed properly, the PF_RING™ enabled Intel© 10GbE driver (ixgbe) needs to be insmoded with flow director enabled in “perfect filter mode” for the desired while root (please see README of the driver for more details):

ixgbe-3.6.7-* # insmod ./src/ixgbe.ko FdirMode=2,2,2,2 FdirPballoc=3,3,3,3

or you can use the load_*_driver.sh script file that you need to modify changing the interface name according to your system

ixgbe-3.6.7-* # ./src/load_*_driver.sh

Now make sure that the flow director in perfect filtering is indeed enabled.

# dmesg

Intel(R) 10 Gigabit PCI Express Network Driver – version 3.6.7-DNA
Copyright (c) 1999-2011 Intel Corporation.
ixgbe 0000:02:00.0: PCI INT A -> GSI 16 (level, low) -> IRQ 16
ixgbe 0000:02:00.0: setting latency timer to 64
ixgbe: Receive-Side Scaling (RSS) set to 0
ixgbe: Flow Director filtering mode set to 2
ixgbe: 0000:02:00.0: ixgbe_check_options: Flow Director perfect filtering enabled
ixgbe: Flow Director packet buffer allocation set to 3
ixgbe: 0000:02:00.0: ixgbe_check_options: Flow Director will be allocated 256kB of packet buffer

The MSI-X interrupts in Linux as of the time this document is written are assigned randomly which means that each queue will deliver packets to a random core. Through SMP affinity the user can define re-assign the interrupts and choose themselves which queue to bind to each core. For more details how to set the SMP affinity please see this white paper mentioned before. Alternatively, one can use a script provided in the ixgbe driver that will map queue 0 to core 0, queue 1 to core 1 etc.

ixgbe-3.6.7-* # ./scripts/set_irq_affinity.sh

The output, depending on your configuration, will look somehow like this:

no rx vectors found on dna0
no tx vectors found on dna0
eth2 mask=1 for /proc/irq/81/smp_affinity
eth2 mask=2 for /proc/irq/82/smp_affinity
eth2 mask=4 for /proc/irq/83/smp_affinity
eth2 mask=8 for /proc/irq/84/smp_affinity
eth2 mask=10 for /proc/irq/85/smp_affinity
eth2 mask=20 for /proc/irq/86/smp_affinity

Perfect filters comprise of a tuple of [VLAN, protocol, source IP Address/mask, source port, destination IP Address/mask, destination port]. In order to drop a flow rather than redirecting to a particular queue we are using the convention of queue_id=-1.
In the perfect filtering there is a one global mask for all perfect filters. That means that if a /32 mask is used for example for the source IP address and later on another filter gets added with a /16 mask, the global mask will change from /32 to /16 for ALL already added filters, which may not be what the user intended, so we are always using a /32 mask.
Five Tuple Queue Filters instead comprise of a tuple of [protocol, source IP Address, source port, destination IP Address, destination port].
In order to add/remove hardware filters the PF_RING™ API provides two explicit functions:

int pfring_add_hw_rule(pfring *ring, hw_filtering_rule *rule);
int pfring_remove_hw_rule(pfring *ring, u_int16_t rule_id);

or it is possible to use the same functions used for software filters by setting a filtering mode other than software_only via pfring_set_filtering_mode().

/* *********************************************************
* ********************* DEPRECATED ************************

Enabling PF_RING™


Next step is to insmod the PF_RING™:

# insmod PF_RING/kernel/pf_ring.ko transparent_mode=0

Make sure the PF_RING™ has identified the hardware filtering capabilities of the right interface. In this example the 82599 based network controllers are eth2 and eth3:

# dmesg

[PF_RING] Welcome to PF_RING 4.4.0 ($Revision: 4353M$)
(C) 2004-10 L.Deri <deri@ntop.org>
[PF_RING] registered /proc/net/pf_ring/
NET: Registered protocol family 27
[PF_RING] Device eth6 does NOT support hardware packet filtering [1]
[PF_RING] Device eth4 does NOT support hardware packet filtering [1]

[PF_RING] Device eth2 DOES support hardware packet filtering
[PF_RING] Device eth3 DOES support hardware packet filtering

[PF_RING] Ring slots 4096
[PF_RING] Slot version 12
[PF_RING] Capture TX Yes [RX+TX]
[PF_RING] Transparent Mode 0
[PF_RING] IP Defragment No
[PF_RING] Initialized correctly

Adding and Removing Hardware Filters


Now in the /proc pseudo file system there will be added as normal an entry for the PF_RING™ in /proc/net/pf_ring and (again as normal) for the PF_RING™ aware interfaces there will be an entry in /proc/net/pf_ring/dev/ethX where X is the enumeration of the Ethernet interface. For the hardware filtering enabled interfaces one more pseudo file will be added: /proc/net/pf_ring/dev/ethX/rules. One can read this file to get help as to how to add filters on that interface:

# cat /proc/net/pf_ring/dev/eth2/rules
Name: eth2
# Filters: 0Filtering Rules:
[perfect rule] +|-(rule_id,queue_id,vlan,tcp|udp|any,src_ip/mask,src_port,dst_ip/mask,dst_port)
Example: +(1,-1,0,tcp,192.168.0.10/32,25,10.6.0.0/16,0) (queue_id = -1 => drop)

[5 tuple rule] +|-(rule_id,queue_id,tcp|udp|any,src_ip,src_port,dst_ip,dst_port)
Example: +(1,-1,tcp,192.168.0.10,25,0.0.0.0,0)

Note:
- queue_id = -1 => drop
- 0 = ignore value

Adding and Removing Perfect Filters


Perfect filters comprise of a tuple of [VLAN, protocol, source IP Address/mask, source port, destination IP Address/mask, destination port]. As described in the previous section, perfect filters can be applied by echoing the appropriate rule id, queue id and tuple information into the desired /proc/net/pf_ring/dev/ethX/rules file. In the following example we add a rule with id number 1, directing to queue 1 all the TCP packets, with VLAN ID 1, any source IP address, source port 25, destination IP 10.6.any.any and any destination port:

# echo “+(1,1,1,tcp,192.168.0.10/32,25,10.6.0.0/16,0)” > proc/net/pf_ring/dev/eth2/rules

NOTE: For dropping we are using the convention of queue_id=-1 and for ignoring a value, the value of 0 can be used.

Based on the above if the user desires some flow to be dropped then redirecting packets to queue -1 will perform a drop, so in order to drop rather redirect to queue 1 the above TCP packets one should echo the following:

# echo “+(1,-1,1,tcp,192.168.0.10/32,25,10.6.0.0/16,0)” > proc/net/pf_ring/dev/eth2/rules

NOTE: In the perfect filtering there is a one global mask for all perfect filters. That means that if a /32 mask is used for example for the source IP address and later on another filter gets added with a /16 mask, the global mask will change from /32 to /16 for ALL already added filters, which may not be what the user intended.

In order to remove a filter, the user needs to echo the exact same string into the same interface related pseudo file, but with a minus (-) at the beginning rather than plus (+). For our previous example the corresponding removal command would be:

# echo “-(1,-1,1,tcp,192.168.0.10/32,25,10.6.0.0/16,0)” > proc/net/pf_ring/dev/eth2/rules

Adding and Removing Five Tuple Queue Filters


Perfect filters comprise of a tuple of [protocol, source IP Address, source port, destination IP Address, destination port]. Similarly to the perfect filters one needs to echo the appropriate rule id, queue id and tuple information into the desired /proc/net/pf_ring/dev/ethX/rules file. In the following example we add a rule with id number 2, directing to queue 2 all the TCP packets, from any source IP address, source port 25, destination IP 10.6.any.any and any destination port:

# echo “+(2,2,tcp,0.0.0.0,25,10.6.0.0,0)” > proc/net/pf_ring/dev/eth2/rules

Again similarly to the perfect filters the minus (-) sign must precede the string rather than the plus sign (+):

# echo “-(2,2,tcp,0.0.0.0,25,10.6.0.0,0)” > proc/net/pf_ring/dev/eth2/rules

******************************************************** */
?>

Final Notes


Affinitizing the queues of a multiqueue interface in 10 GbE is crucial in the performance of any networking application. With the above user guide the PF_RING™ can be used to take advantage of in-hardware on the fly queue to core assignment allowing the software to have full control over to which flow goes to which CPU core. The result will be a much more efficient application that has more control over how it handles the network packets.

FAQ


  • Q. How many filters a 82599-based card typically supports?
    A. You can have up to 32K hardware filters.
  • Q. Do I need to reconfigure the NIC whenever I add/remove a filter?
    A. No. Filter manipulation is immediate and it does not interfere with other ongoing activities.
  • Q. How long does it take to add/remove a filter?
    A. Contrary to many FPGA-based NICs, this operation is instantenous as filters are configured by means of registers.
  • Q. How much a 82599-based card typically costs?
    A. As of today the cost per-port is well below 1000$ per 10 Gbit port.
  • Q. Beside FlowDirector and FTQF, can 82599 provide other type of filters?
    A. Yes 82599 allows you for instance to filter VLAN and ethernet packets, not to mention payload content. See the 82599 datasheet for more information.
  • Q. What Linux kernel version do I need in order to exploit this code?
    A. You need Linux 2.6.31 or above.
  • Q. Do I get the driver source code?
    A. Absolutely. The code is released under GPL and you get the full source code.
  • Q. What hardware PC do I need for wire-speed 1G packet capture at any packet size?
    A. A single quad-core Xeon is more than enough.
  • Q. My computer has DCA disabled and the BIOS do not allow me to enable it. What shall I do?
    A. Have a look at this link.
  • Q. Is this code useful for general-purpose networking?
    A. Yes, hardware filtering can be applied to all networking tasks.

References


Get It


Hardware packet filtering is available with DNA drivers.