Installing PF_RING and nProbe on Fedora Core 4 to enhance packet capture performance.

Updates:

2006-04-10

• Add information for building on x86_64 architecture.

2006-01-31

• Add step enabling 64GB RAM support.
• Remove step to add ring to /etc/modprobe.conf

2006-08-08

• changed /src/fedora to /src/redhat
• Expanded some directions
• Added Step to configure kernel versions in /user/src/linux/Makefile
• Fixed incorrect location where pfring.h is located in kernel source directory

Preface:

I have found that installing PF_RING into the kernel and compiling nProbe to use it isn't very well documented. There are all sorts of little pieces of code that need to be compiled and linked together. I'm sending this documentation to the ntop-misc mailing list in the hope it will be of benefit to others. Any comments or corrections are appreciated. I hope this helps, and good luck.

Instructions

Setup System

• Install FC4.
• Install FC4 updates.

Build kernel with PF_RING patches

• Download Source for FC kernel. Download source RPM for kernel running on system ("uname -r" will give you current version. RPM contains vanilla kernel and FC patches).
• Create FC 4 kernel source. (Instructions below are taken the from Fedora Core Release Notes about kernel compiling, check there for definitive information (NMJ).)
• Install Source code for kernel rpm -Uvh kernel-<version>.src.rpm(where version is the version from "uname -r")
• Add Fedora patches to vanilla kernel cd /usr/src/redhat/SPECS rpmbuild -bp --target $(arch) kernel-2.6.spec
• Move kernel to /usr/src mv /usr/src/redhat/BUILD/kernel-<version>/kernel-<version> /usr/src

(Note: there is a two directories with the same name "kernel-<version>" directories. Make sure you are moving the lower level directory).

• Create link to from kernel to standard /usr/src/linux. cd /usr/src ln -s ./linux-<version> linux cd /usr/src/linux
• Install PF_RING patches into kernel source
• Download PF_RING using CVS: CVSROOT=:pserver:anonymous@cvs.ntop.org:/export/home/ntop;export CVSROOT mkdir <directory to store PF_RING files> cd <directory to store PF_RING files> cvs login (password is "ntop") cvs checkout PF_RING cd PF_RING
• Edit mkpatch.sh so Kernel version variables equal FC4 kernel version created above. For instance if "uname -r" returns "2.6.14-1.1656_FC4", then you would change the variables in mkpatch.sh to: VERSION=2 PATCHLEVEL=6 SUBLEVEL=14 EXTRAVERSION="-1.1656_FC4-PF_RING"
• Run "mkpatch.sh"
• Copy the compressed patch file created by mkpatch.sh in the directory "workspace" (i.e. linux-2.6.14-1.1656_FC4-PF_RING) to /usr/src
• Run "zcat <patch file> | patch --dry-run -p0" to test for errors
• Run "zcat <patch file> | patch -p0" to apply patches
• Build the kernel (with the exception of enabling PF_RING and High Memory Support, the instructions below are taken from the Fedora Core Release Notes section on how to compile the kernel. Check there for definitive information). cd /usr/src/linux

Edit Makefile to set VERSION, PATCHLEVEL, and EXTRAVERSION

make menuconfig

Enable PF_RING under "Networking->Networking options". Enable 64 GB memory support under "Processor type and features->High Memory Support"

make make modules_install make install

("make install" installs the kernel in /boot and makes a compressed ram image (unique to FC))

• Edit /boot/grub/grub.conf, set default time out to 10 seconds and change the default boot to the new kernel.
• Reboot and test that system boots new kernel successfully. You can always reboot and select the old kernel from the grub loader menu if the new kernel doesn't work.

Build PF_RING enabled libpcap

• Copy /usr/src/linux/include/linux/ring.h to /usr/include/linux (For some reason this doesn't get done when compiling the kernel).
• Compile libpfring (This is one of those new steps that was not documented). cd PF_RING/userland/libpfring make cp libpfring.a /usr/local/lib cp pfring.h /usr/local/include *Get version of libpcap specified version of PF_RING (Do an directory listing of PF_RING/userland to find version of libpcap to get. It's currently 0.9.4, get it at  http://www.tcpdump.org/ ).
• Untar libpcap source in PF_RING/userland
• If building for the x86_64 architecture, change directories to the PF_RING/userland/libpfring and add -fPIC to the CFLAGS line and rebuild libpfring.a in 64bit mode.
• Copy pcap-int.h and pcap-linux.c from PF_RING/userland/libpcap<VERSION>-ring to directory PF_RING/userland/libpcap<VERSION>.
• Use the following configure command to make sure libpcap can find the ring libraries ./configure CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib"
• If you are building for x86_64 architecture, to prevent errors when trying to link your new libpcap with programs, you will need to edit the Makefile to add -fPIC to the CFLAGS and change the build process for the shared library from ld -shared -o $@.cat VERSION $(OBJ)

to

gcc -shared -Wl,-soname -Wl,libpcap.so.cat VERSION -o libpcap.so.cat VERSION $(OBJ) -pfring -lc

• Build libpcap make make install

(default install is into /usr/local/include and /usr/local/lib)

• cd back to PF_RING/userland/pcount. Make and run pcount to test to test if libpcap is working. You won't see any output until you stop the program with <CTRL-C>.

  [root@nmj-test libpfring]# ./pcount
   Capturing from eth0
   (Wait a bit, and hit <CTRL-C>. You should see the following):
   =========================
   Absolute Stats: [19 pkts rcvd][0 pkts dropped]
   Total Pkts=19/Dropped=0.0 %
   19 pkts [7.8 pkt/sec] - 5159 bytes [0.02 Mbit/sec]
   =========================
   Actual Stats: 19 pkts [-1402903.2 ms][-0.0 pkt/sec]
   =========================
   [root@nmj-test pcount]#
  

Compile nProbe to use libpcap.

• Untar nProbe
• Run "autogen.sh" (generates configure stuff, don't know why)
• Run configure ./configure CPPFLAGS="-I/usr/local/include" LDFLAGS="-L/usr/local/lib -lpfring"

(again adding -lpfring is NOT documented).

• Test running nprobe

Watch the syslog file by running "tail -f /var/log/messages". Run nprobe. Look for the following messages in to appear in /var/log/messages:

Jan 27 14:15:47 nmj-test kernel: RING: successfully allocated 1024 KB [tot_mem=598076][order=8] Jan 27 14:15:47 nmj-test kernel: RING: allocated 7181 slots [slot_len=146][tot_mem=1048576] Jan 27 14:15:47 nmj-test kernel: device eth0 entered promiscuous mode

Misc.

Other things you may want to consider to maximize performance.

• Disable X (you can always start it by running the "startx" command).
  • Edit /etc/inittab and comment out inittab entry that starts.
• Disable graphical boot status
  • Edit /boot/grub/grub.conf and remove "rhgb" option from grub.conf entries
• Don't forget security.
  • Shutdown all unnecessary services
  • Review Firewall settings

Check the archives for ntop-misc mailing list, there is an e-mail about on how to make libpcap a shared library so it can be used with other applications that use libpcap (ethereal, snort). Hopefully some one will add it to this wiki.

This is the e-mail from the ntop-misc list that is referenced above, on how to compile libpcap as a shared object:

Date: Fri Feb 17 14:53:57 2006 Subject: [Ntop-misc] Aditional instruction for : Installing PF_RING and nProbe on FC4

The "Installing PF_RING and nProbe on FC4" from Johnson, Neil M give people a big chance to successfully install and deploy magic PF_RING on Linux box.

But there are some problem related to libpcap that are not well documented even in tcpdump.org and that is installing libpcap as shared object. If you compile libpcap (both standard version and patched version) it just create static library(libpcap.a) and you need to recompile your application that relay on libpcap to capture packetsinclude NTOP , SNORT , IPAUDIT and too many app. other problem is when you run NTOP or SNORT from RPM . when they run , they will look for libpcap.so.0.x.x (shared library) to run and if they don't find it they never work. you need to compile them from source.tcpdump.org don't provide shared object by on it's standard configure script and you need to do that by hand. To get this work so that too many app (run from RPM) can use it without recompile form source :

• after patch standard libpcap with userland/libpcap-version-ring :

./configure , make , make install. in this step static library (libpcap.a) created

• last step create some object files (*.o) that you need to creating shared object. you also need to copy all *.o file from PF_RING/userland/libpfring to source dir of patched libpcap.
• use gcc to create shared library of libpcap as follow : [bash/]# gcc -shared -Wl,-soname,libpcap.so.0 -o libpcap.so.0.9.4 *.o
• then copy libpcap.so.0.9.4 to /usr/lib and run ldconfig to cache new lib file.
• if you run NTOP / SNORT from RPM-FC4 don't forget to create a soft link as libpcap.so.0.8.3

Now you are ready to capture packets at wire speed.i run SNORT/NTOP/IPAUDIT on SMP machine with two INTEL Gigabit NIC (device polling aware) that capture pachet from two SPAN port (mirror port) of two Cisco Catalyst 6509 with average traffic amount 400 Mbps on each campus network with 0% packet drop. Thanks to Luca Deri for his magic work , and sorry for my bad English.

Regards, Adam