7. Mellanox Support

PF_RING (8.1 or newer) includes native support for Mellanox ConnectX-4, ConnectX-5 and ConnectX-6 adapters. Both packet capture (including RSS, packet hash, hardware timestamp support) and transmission are supported.

7.1. Prerequisite

1. Install the pfring package by configuring one of our repositories at http://packages.ntop.org. PF_RING can also be downloaded in source format from GIT at https://github.com/ntop/PF_RING/

2. Install Mellanox OFED/EN from https://www.mellanox.com/products/infiniband-drivers/linux/mlnx_ofed This package also installs the libibverbs library, which is required in order to use Mellanox adapters with PF_RING, as well as other dependencies. Please note that a reduced toolset can be selected for the OFED SDK to be used by capture frameworks (the –dpdk option is available for that) as in the below example.

cd MLNX_OFED_LINUX-5.4-1.0.3.0-ubuntu20.04-x86_64
./mlnxofedinstall --upstream-libs --dpdk

7.2. Compatibility

Check the adapter firmware:

ibv_devinfo

Recommended firmware versions are:

  • ConnectX-5: >= 16.21.1000
  • ConnectX-6: >= 20.27.0090
  • ConnectX-6 Dx: >= 22.27.0090

7.3. Configuration

Load the required modules:

modprobe -a ib_uverbs mlx5_core mlx5_ib

Check that the adapter is recognised and listed by PF_RING:

pfcount -L -v 1
Name       SystemName Module  MAC               BusID         NumaNode  Status  License Expiration
enp1s0f0       enp1s0f0       pf_ring B8:CE:F6:8E:DD:5A      0000:01:00.0  -1              Up             Valid       1662797500
enp1s0f1       enp1s0f1       pf_ring B8:CE:F6:8E:DD:5B      0000:01:00.1  -1              Up             Valid       1662797500
mlx:mlx5_0 enp1s0f0   mlx     B8:CE:F6:8E:DD:5A 0000:00:00.0  -1        Down    Valid   1662797500
mlx:mlx5_1 enp1s0f1   mlx     B8:CE:F6:8E:DD:5B 0000:00:00.0  -1        Down    Valid   1662797500

Please note that the same interfaces appear twice, as they can se used both using the kernel driver (creating a socket on enp1s0f0 for instance) or the mlx module in zero-copy mode (creating a socket on the corresponding mlx:mlx5_0 interface).

7.4. Capturing Traffic

In order to capture traffic from a Mellanox adapter using the native mlx module mlx:<device> should be specified as interface name, as reported by pfcount -L. Example:

pfcount -i mlx:mlx5_0

7.4.1. RSS / Multi Queue

Multi-queue support (RSS) is available on Mellanox with the constraint that all queues should be used in the same process (supporting multiple capture threads). The number of RSS queues can be set using the standard ethtool command on the kernel interface. Example for 4 queues:

ethtool -L enp1s0f0 combined 4

In order to capture traffic from a queue, mlx:<device>@<queue> should be used as interface name. Example with queue 0:

pfcount -i mlx:mlx5_0@0

Or to open all queues:

pfcount_multichannel -i mlx:mlx5_0

7.5. Traffic Transmission

Packet transmission is also supported on Mellanox (including multiqueue/multithread transmission when RSS is enabled). The same syntax used for capturing traffic can be used to select the TX interface. Example:

pfsend -i mlx:mlx5_0

7.6. Hw Filtering

Mellanox adapters support packet filtering in hw. Up to 64K rules are supported. In order to set an hw filter the pfring_add_hw_rule API should be used.

Sample code for filtering traffic with Mellanox (as well as with other adapters) is available in the pfcount.c sample application (look for sample_filtering_rules).

Filtering rules can be defined as drop or pass. The default behaviour for packets, is defined by the promiscuous mode set using the pfring_open flag PF_RING_PROMISC. With the promisc set, all traffic is received by default, no traffic otherwise.

In order to set a filtering rule, a rule ID (0..65534) should be assigned to the rule. This is a unique identifier that can be used to remove the rule later on. The ID can be automatically assigned by the library by using FILTERING_RULE_AUTO_RULE_ID as rule ID.

A priority can also be assigned to the rule, in the range 0..2. Two applications capturing traffic from the same interface, and setting a pass rule which is matching the same traffic and with the same priority, will both receive the same traffic. Instead, only the application which is setting the higher priority on the rule, would receive the traffic otherwise.

Example setting a filtering rule to drop UDP traffic matching a src IP and destination port:

hw_filtering_rule r = { 0 };

r.rule_id = FILTERING_RULE_AUTO_RULE_ID;
r.priority = 0;
r.rule_family_type = generic_flow_tuple_rule;

r.rule_family.flow_tuple_rule.action = flow_drop_rule;

r.rule_family.flow_tuple_rule.ip_version = 4;
r.rule_family.flow_tuple_rule.src_ip.v4 = src_ip_rule;
r.rule_family.flow_tuple_rule.protocol = IPPROTO_UDP;
r.rule_family.flow_tuple_rule.dst_port = 3000;

pfring_add_hw_rule(socket, &r);

Please note that:

  • all fields are in host byte order.
  • when FILTERING_RULE_AUTO_RULE_ID is used, the rule ID assigned by pfring_add_hw_rule is returned in r.rule_id.

For a full list of supported fields please take a look at the generic_flow_tuple_hw_rule struct.

Example of removing a filtering rule by ID:

pfring_remove_hw_rule(socket, RULE_ID);