ntop

IRQ Balancing

lucaderi — Sun, 13 Dec 2009 11:42:31 +0000

On Linux, interrupts are handled automatically by the kernel. In particular there's a process named irqbalancer that is responsible for balancing interrupts across processors. Unfortunately the default is to let all processors handle interrupts with the result that the overall performance is not optimal in particular on multi-core systems. This is because modern NICs can have multiple RX queues that work perfectly if cache coherency is respected. This means that interrupts for ethX-RX queue Y must be sent to 1 (one) core or at most to a core and its Hyper-Threaded (HT) companion. If multiple processors handle the same RX queue, this invalidates the cache and performance gets worse. For this reason IRQ balancing is the key for performance. In particular what I suggest is to have 1/2 (in case of HT) core(s) handle the same interrupt. For this reason on Linux interrupts are usually send to all processors hence /prox/irq/X/smp_affinity is set to ffffffff that means all processors. Instead as I have just stated above it's better to avoid that all processors handle all interrupts. Example

~# grep eth /proc/interrupts
191:          0          0          3          0          0          0          2  310630615        454          0          0          0          0          0          2          0   PCI-MSI-edge      eth5-rx-3
192:          0          3          0          0          0          2          0  314774529          0          0          0          0          0          2          0          0   PCI-MSI-edge      eth5-rx-2
193:          3          0          0          0          2  309832652        454          0          0          0          0          0          2          0          0          0   PCI-MSI-edge      eth5-rx-1
194:          0          0          0          2          0  314283930          0          0          0          0          0          2          0          0          0          3   PCI-MSI-edge      eth5-TxRx-0
195:          0          0          1          0          1          0          0          0          0          0          1          0          0          0          1          0   PCI-MSI-edge      eth5
196:          0          3          0  311226806          0          0          0          0          0          2          0          0          0

where


# cat /proc/irq/19[12345]/smp_affinity
00008080
00008080
00002020
00002020
ffffffff

This setup allows to maximize performance, in particular when using PF_RING and TNAPI. Please disable irqbalancer when manually tweaking interrupts, as irqbalancer will restore interrupts the way it likes jeopardizing all your work. Further reading:

Port Mirror vs Network Tap

lucaderi — Fri, 01 Jan 2010 20:09:12 +0000

In order to analyze network traffic, it's necessary to feed ntop/nProbe with network packets. There are two solutions to the problem:

port mirror (also called SPAN in Cisco parlance)
network tap

Prior to explain the differences between these two solutions, it's important to understand how ethernet works. In 100 Mbit and above, hosts usually speak in full duplex meaning that a hosts can both send and receive simultaneously. This means that on a 100 Mbit cable connected to a host, the total amount of traffic that a host can send/receive is 2 x 100 Mbit = 200 Mbit. A port mirror is active packet duplication, meaning that a network device physically has the duty to copy packets onto a mirror port.

This means that the device has to carry on this task by using some resources (e.g. CPU) and that both traffic directions will be copied into the same port. As explained before, in full duplex links this means that the total of A->B and B->A can't exceed the network speed before packet loss occurs. This because there's physically no space to copy packets. The consequence is that a port mirror is a great technique as it can be performed by many switches (but not all) with the drawback of packet loss if you monitor a link with over 50% load, or mirror the ports onto a faster port (e.g. mirror 100 Mbit ports onto a 1 Gbit port). Not to mention that packet mirror might require switch resources that can load the unit and lead to reduced switching performance. Note that you can 1 port onto a port, or 1 VLAN onto 1 port, but usually you can't copy many ports to 1. Please note that in this architecture, the switch is a point of failure as if it breaks the connectivity (and packet mirror) is lost. A network tap instead is a fully passive device.

Electrically or optically (e.g. using a prism) packets are copied onto the tap ports. Unless you use an aggregation tap, a tap has one tap port per direction. This means that in order to monitor 1 link, you need 2 NICs, one for the first and the other for the second direction. In this case you won't ever loose a single packet as in the mirror case, because each direction is using a link, instead of merging both directions onto the same link. If you want to merge both directions into one single port, you can use a network switch as depicted below:

As tap ports do not receive but transmit only, the switch has no clue who's sitting behind the ports. The consequence is that it broadcast the packets to all ports. So if you connect your monitoring device to the switch, such device will receive all packets. Note that this mechanism works if the monitoring device does not send any packet to the switch, otherwise the switch will assume that the tapped packets are not for such device. In order to achieve that, you can either use a network cable on which you have not connected the TX wires, or use an IP-less (and DHCP-less) network interface that does not transmit packets at all. Finally note that if you want to use a tap for not loosing packets, then either don't merge directions or use a switch where tapped directions are slower (e.g. 100 Mbit) that the merge port (e.g. 1 Gbit).

Bottom line. Packet mirror and tap have both pros and cons. You can select the right solution based on your network (and wallet).

If you want to know more about this topic, I suggest you to read this document named Multi-Tap Network Packet Capturing. It describes more in detail how to play with network taps and it highlights how to use NST (Network Security Toolkit, a Linux distribution that also includes ntop) in common network monitoring scenarios, where very likely your setup is also included.

Multi-Tap Network Packet Capturing

ntop ASA Support

lucaderi — Sat, 09 Jan 2010 08:24:21 +0000

ntop supports NetFlow since many years including the latest v9/IPFIX versions. In 2005 Cisco ha releases a new line of x86 based security devices named ASA that unfortunately have not been supported by ntop/nProbe for a long time. As of today (June 15th 2010, SVN revision 4299) ntop/nProbe finally supports ASA. Please note that as ASA units do not export templates too often, ntop might need some time to start decoding flows (this until the template is received). Furthermore as the nature of ASA flows (e.g. notify when a new connection inside the unit has been created/deleted) is not the same of NetFlow, some information is missing and need to be emulated by ntop/nProbe (e.g. the number of flow packets, so ntop computes them assuming that their size is 512 bytes). Many thanks to David Bowman for his help during ASA support implementation. References

Exploiting Commodity Multi-core Systems for Network Traffic Analysis

lucaderi — Sat, 30 Jan 2010 10:22:40 +0000

This article Improvement of libpcap for lossless packet capturing in Linux using PF_RING kernel patch positions PF_RING (3.x, so some changes are needed when using version 4) against the Linux standard PF_PACKET packet capture facility. In PF_RING v4, due to popular demand, I have decided to move some of the PF_RING accelerations into the NIC driver with the advantage of being now able to compile PF_RING against an unpatched kernel. The PF_RING distribution has now a drivers/ directory that contains accelerated drivers for popular 1 and 10 Gbit adapter. This means that using PF_RING on top of Linux without any accelerated drivers, gives you a little speed advantage when compared with standard Linux. However if you use a PF_RING-aware driver or even better TNAPI, your speed bump will be much better. I summarize some lessons learnt on this field on this research paper named Exploiting Commodity Multi-core Systems for Network Traffic Analysis.

ntop.org Accredited as Endace Technology Partner

lucaderi — Thu, 04 Feb 2010 09:25:31 +0000

We're proud to announce that ntop.org has been accredited as Endace technology partner as recognition for ntop contribution in the open-source world and also as guarantee for Endace customers that products such as ntop and nProbe run smoothly (and faster) on Endace DAG cards.

ntop.org Joins the Open Information Security Foundation

lucaderi — Thu, 18 Feb 2010 08:49:00 +0000

Suricata is the next generation open source IDS/IPS developed byt the Open Information Security Foundation. It is a pleasure to announce that ntop has joined the core development team as the Linux version of Suricata is based on acceleration provided by PF_RING. In the near future PF_RING will be extended so that it can also accelerate packet transmission in order to move the Suricata IPS performance to the next level. More information can be found here.

Introducing PF_RING DNA (Direct NIC Access)

lucaderi — Sun, 21 Feb 2010 19:29:40 +0000

This is to announce the availability of PF_RING DNA (Direct NIC Access) that significantly increments performance (up to 80%) when compared with Linux packet capture and PF_RING (non DNA).

PF_RING is polling packets from NICs by means of Linux NAPI. This means that NAPI copies packets from the NIC to the PF_RING circular buffer, and then the userland application reads packets from ring. In this scenario, there are two pollers, both the application and NAPI and this results in CPU cycles used for this polling; the advantage is that PF_RING can distribute incoming packets to multiple rings (hence multiple applications) simultaneously. PF_RING DNA (Direct NIC Access) is a way to map NIC memory and registers to userland so that packet copy from the NIC to the DMA ring is done by the NIC NPU (Network Process Unit) and not by NAPI. This results in better performance as CPU cycles are used uniquely for consuming packets and not for moving them off the adapter. The drawback is that only one application at time can open the DMA ring, or in other words that applications in userland need to talk each other in order to distribute packets. In a nutshell if you like flexibility you should use PF_RING, if you want pure speed PF_RING DNA is the solution. Please note that in DNA mode NAPI polling does not take place, hence PF_RING features such as reflection and packet filtering are not supported. For more information, please have a look at the PF_RING home page.

PF_RING and Transparent Mode

lucaderi — Sun, 07 Mar 2010 09:52:40 +0000

PF_RING has been designed for enhancing packet capture performance. This means that the RX path must be accelerated, and in particular a way to accelerate this is by reducing the journey of the packet from the adapter to userland. This is obtained by allowing the driver to push the packet from the NIC to PF_RING directly and not through the usual kernel path. For this reason PF_RING has introduced an option named "transparent mode" whose goal is to tune how packets are moved from the NIC to PF_RING. This option (that can be specified when inserting the PF_RING module via insmod) can have three values:

insmod pf_ring.ko transparent_mode=0 This is the default and it means that packets are sent to PF_RING via the standard kernel mechanisms. In this setup the packets are both sent to PF_RING but to all other kernel components. All NIC drivers support this mode.
insmod pf_ring.ko transparent_mode=1 In this mode, packets are sent directly by the NIC driver to PF_RING, packets are still propagated to other kernel components. In this mode packet capture is accelerated because packets are copied by the NIC driver without passing through the usual kernel path. Please note that in order to enable this mode, you must use a NIC driver that supports PF_RING. Available PF_RING-enabled drivers can be found in the drivers/ directory of PF_RING.
insmod pf_ring.ko transparent_mode=2 In this mode, packets are sent directly by the NIC driver to PF_RING, packets are not propagated to other kernel components as this slows down packet capture. Please note that:
- in order to enable this mode, you must use a NIC driver that supports PF_RING.
- Packets are not sent to the kernel after they have been delivered to PF_RING. This means that you won't have connectivity from NICs driven by PF_RING-aware drivers.
- This mode is the fastest one as packets are quickly copied to PF_RING and immediately discarded after they have been processed.

ZDNet.de: ntop: ein bedeutsames Tool fuer die Netzwerkanalyse

lucaderi — Fri, 26 Mar 2010 15:09:58 +0000

ZDNet.de speaks about ntop.

Collection and Exploration of Large Data Monitoring Sets Using nProbe

lucaderi — Wed, 07 Apr 2010 13:08:38 +0000

Collecting and exploring monitoring data is becoming increasingly challenging as networks become larger and faster. Solutions based on both SQL-databases and specialized binary formats do not scale well as the amount of monitoring information increases. This paper presents a novel approach to the problem by using a bitmap database that allowed the authors to implement an efficient solution for both data collection and retrieval. The validation process on production networks has demonstrated the advantage of the proposed solution over traditional approaches. This makes it suitable for efficiently handling and interactively exploring large data monitoring sets. Read full paper or see a live presentation (or even a picture) of the work.

IANA Assigned a PEN to ntop

lucaderi — Thu, 15 Apr 2010 17:24:52 +0000

Internet Assigned Numbers Authority (IANA) has assigned to ntop the 35632 PEN (Private Enterprise Number) number. This means for instance that nProbe extensions (e.g. HTTP and VoIP traffic monitoring) will be exported using IPFIX using a valid template that will be recognized by all flow collectors available in the market. A side effect is that whoever will use ntop/nProbe to monitor its own network or code monitoring extensions will be able to export them using a uniform template that will be handled by all applications. This is a major step for pushing open source applications even into commercial environments.

Meet ntop @ Bolzano (May 20th): Conference on Nagios, NTOP @ OSS Monitoring featuring Ethan Galstad an Luca Deri

lucaderi — Sun, 18 Apr 2010 19:00:17 +0000

Following the great interest in 2009, the successful series of an international Conferences on Nagios, NTOP and OSS Monitoring will continue also in 2010. Therefore the organization team of Würth Phoenix spared no efforts to top last year’s agenda and bring international Nagios and OSS Monitoring experts to Bolzano/Italy. This way, next to Nagios founder Ethan Galstad also Michael Medin, Cacti Europe leader Reinhard Scheck, ntop founder Luca Deri as well as the worldwide experienced Swedish Nagios service provider of op5 will be among the key speakers.

The presented sessions will go deep into the actual business trends and best practices without missing out the technical and strategic hard facts. The attendees will have the opportunity to hear about highly current topics like the strongly discussed Nagios Community development as well as the strategic positioning and the future plans of the namable monitoring standard.

Beside the keynote speeches also a panel discussion will be part of the agenda to highlight the changing market relevance of Nagios and other relevant OSS projects such as ntop.

Who Should Attend

We welcome everyone passionate about Open Source, from sys admins to developers, and programmers as well as IT managers and entrepreneurs. The entry, including networking lunch, is free. For all details including registration go to following link: www.wuerth-phoenix.com/events

Why You Should Attend

You will hardly find a similar Conference with such a high profile representatives of Nagios and OSS Monitoring in Southern Europe, and all this without any registration fee. You will be able to share experiences, strategies and hear some of the most current Nagios developments from the founder himself. And not least you will meet a lot of nice, friendly and mind liked people.

Beside the keynote speeches also a panel discussion will be part of the agenda to highlight the changing market relevance of Nagios and other relevant OSS projects such as ntop.

Who Should Attend

Why You Should Attend

See the video announcement of Ethan Galstad

See the video announcement of Michael Medin

10 Gbit PF_RING-based Hardware Packet Filtering and Balancing Previewed at the Intel Europe Conference

lucaderi — Wed, 05 May 2010 18:29:59 +0000

Intel Research Europe Conference, Bruxelles, May 4th 2010 Luca Deri and Joseph Gasparakis, senior Intel engineer, have previewed a new PF_RING-based technology they have co-developed that allows Linux users to fully exploit the hardware capabilities of the newest Intel X520 10 Gbit adapter (based on Intel 82599 controller). This technology that is close to public availability (at no cost), will enable PF_RING users to program the X520 card with (over 32'000) rules that allow to both balance and filter traffic in hardware with no CPU intervention. Linux users will be able to prioritize traffic, move specific traffic to selected CPU cores, and implement 10Gbit efficient hardware-based firewalls using a low cost commodity adapter. A specific TNAPI version for 82599-based NICs will enable developers to even further accelerate their monitoring and security applications. Luca and Joseph thank Peter Waskiewicz Jr for his support during the development of this driver.

Intel X520 Card

Intel Conference - Demo Setup

Intel Fellow and "Mother of the Internet" Radia Perlman visit Luca and Joseph

Joseph and Luca during the conference

Meet ntop @ Zürich (June 23rd): Large-scale Flow Monitoring Through Open Source Software

lucaderi — Wed, 05 May 2010 19:32:01 +0000

AIMS 2010 Conference Tutorial Zürich, June 21-25, 2010 Large and high-speed networks produce a large number of flows that need to be collected and analyzed. Most collectors are unable to keep up with the flow export rate, and also have severe speed limitations when creating reports, due to the amount of data that needs to be analyzed. This tutorial shows how recent innovation in databases, combined with existing open-source software applications, allow flow collection and exploration of large-scale flows to become feasible. Furthermore the use of web 2.0 technologies enable network administrators to analyze collected data in real-time and explore it interactively by means of a web browser

Meet ntop @ Florence (May 9th): Opening-up ntop using Python

lucaderi — Wed, 05 May 2010 19:34:20 +0000

Pycon Conference Florence, May 7-9 2010 ntop (http://www.ntop.org) is an open-source project aimed at monitoring network traffic. Recently, in order to make the tool even more flexible than before this for letting people adapt it to their needs, the python interpreter has been embedded into ntop. The result is that users can now use python for scripting ntop or building monitoring applications in python leveraging the ntop monitoring engine. This talk presents the ntop/python integration and describes some real network monitoring problems that have been effectively solved using this solution.

Extending ntop using Python

lucaderi — Sat, 08 May 2010 15:40:47 +0000

ntop was designed to be self-contained in order to avoid people configuration and usage headache. Unfortunately the drawback is that extending ntop has always been a difficult activity as users had to code in C and know ntop's internals. Recently thanks to the integration of ntop with Python, it is possible to script the application and add new features on the fly with minimal effort and no knowledge of how ntop works internally. This tutorial shows how the ntop+Python integration works, and it describes what users can do with it.

ntop and Plixer Partnered for Advanced Flow-based Monitoring

lucaderi — Mon, 17 May 2010 20:37:20 +0000

May 17th 2010 Press Release Plixer International, Inc., a leading global provider of network traffic monitoring and analysis tools, today announced that it has partnered with NTOP of Italy to launch Scrutinizer 7.7 with nProbe™ support for advanced flow-based monitoring to analyze client, server and application latency. If the flow involves HTTP, the URL information can also be exported. With its unique software-based nProbe™ support, Scrutinizer 7.7 is the first-of-its-kind NetFlow analyzer to enable affordable remote probe deployment on individual PCs or servers to track and pinpoint traffic and application issues. While traditional NetFlow reporting involves displaying top IP addresses, ports and data volume, Scrutinizer v7.7 when receiving IPFIX from the nProbe, can deliver details generally only available via packet analysis. For example, if the flow exported by the nProbe involves HTTP, the exact URL and corresponding client, server and application latency experienced by the end systems is displayed. More detail means more awareness and shorter network troubleshoots. “The ability to deploy this unique open source nProbe™ monitor makes this a powerful combination that is extensible and customizable for other applications, such as monitoring for VoIP and e-mail traffic,” said Michael Patterson, Scrutinizer product manager with Plixer. “This integration takes application performance analysis with NetFlow or IPFIX to a whole new level.” By expanding flow monitoring deeper into the actual packets, Scrutinizer 7.7 with nProbe™ can provide greater detail on database performance, latency, e-mail and URL activities. The nProbe collects the data and transfers it to Scrutinizer via NetFlow v9 or IPFIX for reporting and archiving. The system also stores historical data for baseline trend analysis. The combined solution allows customers to drill down on conversations to determine client round-trip time and server processing latency. If the communication involves HTTP, the complete URL is provided, as well as the ability to click and actually view the page accessed by the client. E-mail details include mail sender and recipient. The result is much more detailed network traffic data with a primary focus on ensuring optimum application availability, troubleshooting and analysis far beyond traditional NetFlow analysis. “Without Scrutinizer 7.7 with nProbe™ support, the only other way to get this kind of data is to use packet analyzer technology,” said Luca Deri, founder of NTOP and developer of the nProbe™ technology. “But if the problem is in an off-site location, this can be costly and time consuming. With remote deployment of our nProbe technology, you can analyze traffic at any location and have it report back to Scrutinizer for more efficient, network-wide troubleshooting and analysis.” The nProbe™ integration for Scrutinizer 7.7 is remarkably affordable at just $195 for an unlimited server site license. Existing Scrutinizer customers under maintenance will receive a free upgrade to version 7.7, with just $195 required for the nProbe™ functionality. Plixer and NTOP have partnered with Ravica for U.S. distribution of nBox, the nProbe™-powered hardware appliance. For more information on Scrutinizer 7.7 with nProbe™ integration, visit www.plixer.com. Jon Mills Marketing & Public Relations Manager

Installation Guide For PF_RING

lucaderi — Tue, 15 Jun 2010 21:57:20 +0000

Below you can find an installation guide for PF_RING written by Gunjan Bansal.

The original blog entry can be found at this URL.

-------------

Hi,

This is my first guide so please bear with me for any disrespencies.

These steps were tested on Intel Core 2 Duo machine with 4 GB Ram and Intel(R) PRO/1000 Network Card , with Ubuntu 9.10 installed.This guide explains the installation procedure for Version 4.3.1

PF_RING implementation by Luca Deri is a great method for efficient Packet Capture on Commodity Hardware.It can be found on http://www.ntop.org/PF_RING.html

I made a clean install,no other packages were installed other than mentioned.

Uninstall libpcap and other dependent applications/library using syanptic or apt-get
Install subversion(For fetching latest source codes),flex and bison(Required to recompile pf_ring aware pcap),ethtool(if not preinstalled,required for some basic Nic info of your computer)
Use Subversion to fetch source codes svn co https://svn.ntop.org/svn/ntop/trunk/PF_RING/ PF_RING
Check your current network card/driver using ethtool ethtool -i eth0 #change eth0 to your ethernet card Sample driver: e1000e version: 1.0.2-k2 firmware-version: 0.4-3 bus-info: 0000:00:19.0 use man ethtool for more information on how to use ethtool As of version 4.3.1 PF_RING can be fully exploited only on the PF_RING aware device drivers.The modified device drivers for some of the popular network cards can be found in PF_RING/drivers . These are modified version of the drivers supplied from respective companies.(Might be a bit older sometimes but will work)
Unload the ethernet card driver (this is shown in the first line of output of above command sudo rmmod e1000e
Change current working directory to kernel cd PF_RING/kernel
Make the source codes make Here i had observed that many a times people(even I on first attempt)use sudo make or sudo -s , make . The former is not a correct method and will give you errors such as bounds.c missing , etc. Latter is also wrong but will do the job . The reason for such a thing is better explained by the difference in the command "sudo" and "sudo -s" . This can be found at http://ubuntuforums.org/showthread.php?t=983645 , or searching for "difference between sudo and sudo -s" on our favorite google
Now install the newly build source sudo make install
Change the working directory to PF_RING/userland/lib cd ../userland/lib
Again build the source codes make
Install the library (This include pfring.h) sudo make install
One bizarre thing that I observed is that the make install copies pfring.h to /usr/local/include but leaves the other dependent files these are :-
- pfring_e1000e_dna.c
- pfring_e1000e_dna.h
Although the function in these files are not required in much of the program, they are include in pfring.h and i don't want to mess up with that.So we copy this to /usr/local/include . Please check this in later versions . cp pfring_e1000e_dna.c /usr/local/include cp pfring_e1000e_dna.h /usr/local/include
Now we have to compile PF_RING aware pcap library . Change the working directory to userland/libpcap-1.0.0-ring cd ../libpcap-1.0.0-ring/
Configure ./configure
Build the sources make
Install pf_ring aware libpcap sudo make install
Now we need to install the device driver(pf_ring aware).Change the working directory to drivers///src In my case it is "drivers/intel/e1000e-1.0.15/src" cd ../../drivers/intel/e1000e-1.0.15/src
Build the source make
Install the driver sudo make install
Now we need to activate PF_RING if its not already activated . You can use lsmod to check if pf_ring is started or not.Change the working diectory to /lib/modules//kernel/net/pf_ring . cd /lib/modules/2.6.31-14-generic/kernel/net/pf_ring Use uname -r to get the kernel version
Enable PF_RING(if already enabled you can disable it using sudo rmmod pf_ring) sudo insmod pf_ring.ko transparent_mode=1 More on transparent mode can be found at http://www.ntop.org/blog/?p=56
Now enable to enable your driver go to /lib/modules//kernel/drivers/net/e100e cd /lib/modules/2.6.31-14-generic/kernel/drivers/net/e1000e
Enable the driver sudo insmod e1000e.ko
Now you can start working on your PF_RING application.You will have to recompile many applications such as tcpdump(modified included),network manager etc. Google for doing so :)

nProbe with FastBit database: an innovative flows storage solution

lorenzetti — Sat, 19 Jun 2010 16:18:39 +0000

nProbe, acronym for NetFlow probe, is an open-source probe that supports both NetFlow and sFlow collection. It has been designed to keep up with Gigabit speeds on commodity hardware and it can be used for capturing packets and analyzing networks at full speed with no (or very moderate) packet loss using PF_RING.

Each captured packet is analyzed, associated to a flow record, and periodically, the expired flows are emitted and exported to the specified collectors. nProbe is fully inter-operable with commercial collectors and open source tools such as ntop.

The new version of nProbe (that will be released soon) has been extended and now contains a new storage system designed primarily to answer queries efficiently.

The new storage system

When nProbe is used as probe and collector, it supports flow collection and storage, both on raw files and relational databases such as MySQL and SQLite.

Support of relational databases has always been controversial as nProbe users appreciated the ability to query flow records using SQL, but at the same time flow dump to database could lead to flow records loss due to the database-processing overhead. On the contrary, the speed advantage of dumping flow records in raw format is paid at each search operation in terms of amount of data to read. Furthermore, the query language that can be used is limited when compared to SQL facilities.

In order to overcome the limitations of existing flow-management systems, an extension of nProbe has been developed. The new version of nProbe allows flow records to be stored on disk, using an innovative column-oriented database with an efficient compressed bitmap indexing technology named FastBit.

Conceptually FastBit is a database that stores its content by column, rather than by row (this structure is known as "vertical organization"). Data is represented as tables with rows and columns. A large table may be partitioned into many data partitions and each of them is stored on a distinct directory, with each column stored as a separated file in raw binary form. Users can configure partition duration (in minutes) at runtime and when a partition reaches its maximum duration, a new one is automatically created.

Furthermore, for tasks that demand the fastest possible query processing speed, bitmap indexes perform extremely well. These because the intersection between the search results on each variable is a simple AND operation over the resulting bitmaps. The consequence of this major speed improvement is that it is now possible to query data in real-time.

Additional details

The new extended nProbe creates FastBit partitions depending on the flow templates being configured (in probe mode) or read from incoming flows (in collector mode). Below there is a simple example where nProbe is configured to dump flow records using a temporary directory with a rotation period of 10 minutes:

nprobe -n none -i eth0 --fastbit /tmp/fastbit/ --fastbit-rotation 10 --fastbit-template "%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL"

Flow records can be dumped at full speed with no index-build overhead. Thus, not considering flow receive/decoding overhead, it is possible to save on disk more than one million flow records/sec on a standard Serial ATA (SATA) disk.

Additional advantages of this technology are listed below:

Ability to save flow records on disk with minimal overhead allowing no-loss on-the-fly flow-to-disk storage, as it happens with tools based on raw files.
Compact data storage to limit disk usage as this enables users to store months of flow records on a cheap hard-disk with no need to use expensive storage systems.
Simple data archive structure in order to move ancient data on off-line storage systems, without having to use complex data partitioning solutions.
On tens of millions of records: sub-second search time when performing cardinality searches (e.g. count the number or records that satisfy a certain criteria) and sub-minute search time when extracting records matching a certain criteria (e.g. top X hosts and their total traffic on TCP port Y).

If you want to know more about this topic or view the results of comparative tests that were performed, you can read the research paper named "Collection and Exploration of Large Data Monitoring Sets Using Bitmap Databases" (Proceedings of TMA 2010, Zurich - April 2010).

To know the new parameters of next release of nProbe allowing to store flow records in the FastBit database and to see some examples of use, you can read this manual.

If you are interested in nProbe, follow this link to know how to get it!

Interview with Luca Deri

lucaderi — Mon, 21 Jun 2010 06:56:41 +0000

In this video Luca presents the ntop project and gives an outlook of future activities. It was presented during the OSS conference that took place last May in Bolzano. Finally this short interview gives an idea of how ntop can benefit when integrated with commercial applications and vendors such as Würth-Phoenix.

Modern Packet Capture and Analysis: Multi-Core, Multi-Gigabit, and Beyond

lucaderi — Tue, 29 Jun 2010 05:56:13 +0000

Sometimes people ask me a tutorial about PF_RING. Last year I have given a tutorial about it at the IM 2009 conference. I think that everyone interested in using PF_RING for going beyond packet capture acceleration should read this set of slides I used for the tutorial. Today the cost of packet capture is limited with respect to packet analysis. For this reason you should use PF_RING as a framework for creating simple yet powerful traffic monitoring applications.

Creating 3D Maps using ntop

lucaderi — Mon, 19 Jul 2010 14:27:22 +0000

Since some time ntop support geolocation. Now courtesy of Ronald W. Henderson it can also display mercator maps and natively integrate with tools such as Google Earth. These ntop extensions are part of the NST (Network Security Toolkit) toolkit. For more information please visit the NST Wiki page.

Released ntop 4.0

lucaderi — Mon, 19 Jul 2010 21:20:34 +0000

After a few years of work, this is to announce the availability of ntop 4.0. Major changes include:

Partially rewritten ntop processing engine to address reliability and performance
Several bugs and stability issues fixed
Added better support for IPFIX and NetFlow v9, as well as ntop PEN (Private Enterprise Number)
Added support for Cisco ASA firewalls
Added ntop engine scriptability via the python programming language
Added RRDalarm plugin for generating alerts based on thresholds
Improved google maps integration
Enhanced sFlow support

ntop is available for both Unix and Windows platforms. The source code can be downloaded from here. Prebuilt Win32 binaries are available here. Many thanks to all code contributors, testers and all those who spread the word.

Using Genetic Algorithms for Network Intrusion Detection and Integration into nProbe

lucaderi — Wed, 21 Jul 2010 08:59:38 +0000

Conference: OSCON 2010 Presentation Link: Ignite Track Presented by: Brian Lavender SNORT is popular Network Intrusion Detection System (NIDS) tool that currently uses a custom rule based system to identify attacks. This presentation emphasizes on writing the algorithm to write generate the rules through GA and the integration of them into nProbe, a similar network monitoring tool written by Luca Deri with a plug-in architecture. Genetic Algorithms are dependent upon identifying attributes to describe a problem and evolving a desired population. In this case, the problem is an attack through the network and identifying the attack through connection property attributes. Genetic Algorithms depends upon training data. DARPA datasets provide training data, in categorized format (attack vs. normal) along with a corresponding raw network recorded format called tcpdump. nProbe has a plug-in architecture allowing for customization. This presentation explains original code in C to evolve rules. It uses the same chromosome attributes used by Gong. The development verifies and contrasts against the research performed by Gong. It also presents the code for integration into nProbe.

ntop on Ubuntu

lucaderi — Fri, 30 Jul 2010 07:31:13 +0000

The ubuntu community has published a post that explains how to compile/use ntop on Ubuntu. This is the URL of the post.

Twelve years of ntop

lucaderi — Fri, 27 Aug 2010 20:12:05 +0000

The Internet is pretty volatile. As new information become available, the old one disappears. Sometimes we have to look back and see what's happened in the past years. Shall you be interested in seeing how ntop changed in the past twelve years, you can have a look at this URL, that has several snapshots of the ntop web site.

PF_RING/TNAPI-based 10 Gbit Network Monitoring on Multicore Systems

lucaderi — Sat, 04 Sep 2010 06:08:16 +0000

Over the past couple of years, PF_RING has been enhanced to exploit innovations in computer hardware. In particular the availability of multicore systems and efficient controllers such as those introduced by Intel with the i7 family (in particular Nehelem and Sandy Bridge) has allowed applications to spread their load across all available processors (24 cores in dual-CPU Westmere systems). In addition to this, modern 82599-based 10 Gbit network adapters feature hardware-based packet filtering and prioritization across RX queues, have opened up a whole world of opportunities. For this reason in the past months, PF_RING and in particular TNAPI has been greatly enhanced to support all the above listed features. In order to better give an idea of what PF_RING can do for you, and also give an outlook of new possibilities offered combining it with all these hardware innovations, I have written a few papers that will be presented at leading conferences. For all those who cannot attend them, you can read them online.

Towards Monitoring Programmability in Future Internet: challenges and solutions Proceeding of 21st Tyrrhenian Workshop on Digital Communications.

Wire-Speed Hardware-Assisted Traffic Filtering with Mainstream Network Adapters Proceedings of NEMA 2010.

High Speed Network Traffic Analysis with Commodity Multi-core Systems Proceedings of IMC 2010.

Another interesting paper I have not written but that I suggest you to read is Design considerations for efficient network applications with Intel® multi-core processor-based systems on Linux that has been written by Joseph Gasparakis and Peter P Waskiewicz, Jr. who are also working with me on 10 Gbit network monitoring.

10 Gbit Hardware Packet Filtering Using Commodity Network Adapters

lucaderi — Sat, 04 Sep 2010 12:06:39 +0000

The promise of filtering packets in hardware is not new. Unfortunately filtering network adapters are pretty expensive, not to mention if they run at 10 Gbit. Furthermore many commercial FPGA-based NICs feature hardware packet filtering, but often require card reconfiguration whenever flow rules are added/removed and have a limited set of rules that can be configured. The release of Intel X520, the first NIC based on the 82599-controller, has triggered my interest as this controller is much more powerful than what Linux can do with it. Thanks to support from Intel and in particular Joseph Gasparakis of Intel Shannon, I have jointly developed an extension to the ixgbe driver (used to drive 82599-based NICs) for adding hardware packet filtering support. Thanks to this work, users can specify up to 32K (yes thirty-two thousand) filters that can be added on the fly without any hardware reconfiguration. And if you want the cherry on top, the cost per port of X520 is well below 1000$. So you now have no reason for not jumping on the 10 Gbit wagon. The enhanced driver is released free of charge as part of the PF_RING distribution (inside PF_RING/drivers/intel). If you also want packet capture acceleration in addition to hardware filtering you can use TNAPI that now supports hardware packet filtering too. You can find more information about this work at this page.

nProbe Internals

lucaderi — Sat, 11 Sep 2010 17:09:50 +0000

nProbe is an efficient processing engine able to produce flows based on captured packets, converts flow format (e.g. from NetFlow v5 to v9), or from sFlow to NetFlow. Its engine is fully extensible by means of plugins, and it can handle many application-level protocols. This short document gives an overview of the nProbe internals and it describes the nProbe plugins structure.

Introducing nProbe v6

lucaderi — Thu, 30 Sep 2010 20:14:25 +0000

Today the new nProbe v6 has been released. It includes several improvements with respect to the previous version including:

Full IPFIX support: PEN (Private Enterprise Numbers) and Variable length encoding.
Ability to natively dump flows in FastBit format that allows to outperform relational and raw flow-based collectors.
Ability to collect sFlow flows and turn them into flows (v5/v9/IPFIX).
Collection of Cisco ASA flows and conversion in 'standard' flows.
New nprobe architecture for better performance and exploitation of multicore architectures.
Support of tunneled (including GRE, PPP and GTP) traffic and ability to export in flows inner/outer envelope/packet information.
HTTP and MySQL protocol analysis: ability to generate logs of web and mysql activities in addition to flow export.
BGP Plugin for establishing a BGP session with a router and generate flows with AS and AS path information.

nProbe is available in three versions:

Standard version: nProbe™ (without plugins) source code. You can compile this software on Unix, Linux, Solaris, OSX and Win32 (nProbe™ for Win32 is available in both binary and source format).
Professional version: same as standard version with native PF_RING support (i.e. full packet capture acceleration), Fastbit support and some plugins (database, dump and layer-7 inspection plugin). This version is available only for Linux.
Professional version with plugins: same as standard version with native PF_RING, Fastbit support and all plugins (including http and MySQL plugin). This version is available only for both Unix and Win32.

As usual nProbe is released under GPLv2 and it is free of charge for universities and researchers. Existing nProbe owners (who purchased previous versions no more than a year ago) can download v6 free of charge from the URL they have received when registered for the previous version. More information can be found at the nProbe web page.

Monitoring and Solving Network Management Challenges

lucaderi — Fri, 01 Oct 2010 19:44:35 +0000

One of the unique nProbe features is its architecture that's open to extensions. Plixer International is exploiting these nProbe features in their products. If interested you should attend this presentation. 2010 ACUTA Fall Seminar 10/24/2010 - 10/27/2010 Sheraton Premier at Tysons Corner Vienna, VA Presenter: Michael Patterson President/CEO Plixer International 1 Eagle Drive Sanford, ME 04073 Bio sketch: Michael Patterson leverages his 16+ years of experience in network management to oversee the direction of the company's network management solutions. Under Mike's direction, Plixer has worked with more than 100 universities and more than 30 hospitals. He is a seasoned speaker, with presentation experience including engagements at White Hat Security Day, Enterasys Networks, Sharkfest and MTUG, as well as regular appearances at local Cisco user groups.(2010) Title of Session: Monitoring and Solving Network Management Challenges Date and Times: 10/27/2010 8:30 AM to 10/27/2010 9:40 AM Session description: Illegal downloading at universities and increasing network demands are big issues. This presentation will size up the risks of low availability of bandwidth for universities (including those with medical centers). Patterson will touch on the pain points of monitoring heavy traffic and ways network managers can reduce application latency over WAN connections. Attendees will learn how to dig deeper into network traffic to keep high availability for transactions with crucial and latency-sensitive applications.

Using ntop as a flow collector for nProbe

lucaderi — Thu, 07 Oct 2010 15:29:24 +0000

nProbe is an efficient netflow/IPFIX probe that can also act as a collector dumpling flows on disk or onto a database (MySQL, sqlite and Fastbit). As ntop has not been designed to operate on large/fast networks, it's possible to use nProbe as pre-processor. In this configuration, nProbe captures packets from a network interface (or collects flows on a socket), computes flows based on packets, and sends them to ntop. Thus ntop acts as a flow collector. Supposing to:

receive packets to account/analyze on interface eth1 of host X
start ntop on host Y (note that both ntop and nProbe can run on the same host simultaneously)

the configuration to use is the following

nProbe
- Start nProbe as nprobe -i eth1 -n X:2055 In this case nProbe computes flows and sends them to host X on port 2055
ntop
- Start ntop as usual
- Enable the NetFlow plugin (menu Plugins -> NetFlow -> Activate)
- Inside the NetFlow plugin create a new virtual interface configured as follows:
  - NetFlow Device: pick a name you like (e.g. MyNetFlow) and click "Set Interface Name".
  - Local Collector UDP port: 2055 and click "Set Port".
  - ntop automatically detects the flow version and decodes the flows without any further configuration.
- At this point switch the ntop view to the netflow interface you have just created (menu Admin -> Switch NIC -> MyNetFlow)

As soon as nProbe sends flows to ntop, the ntop web interface will show the flows being received. In case you see no flows coming you can:

Check if you have a firewall or similar blocking flows
See if there are decoding problems. You can do this accessing the NetFlow statistics (menu Plugins -> NetFlow -> Statistics).

A safe network for a relaxed life

lucaderi — Sat, 16 Oct 2010 09:42:22 +0000

My friends at Würth-Phoenix (I have to thank them for spreading the word about it) have prepared this presentation. It has not been conceived for professionals, but rather for those wishing to have a clue what's ntop about.

Improving snort performance using PF_RING and multi-queue adapters

lucaderi — Sat, 30 Oct 2010 16:23:31 +0000

As of PF_RING 4.5.x, the user-space tools part of PF_RING have been enhanced with native snort support. As of version 2.9, snort sits on top of a library called DAQ (Data Acquisition library) that creates a transparent layer between snort and the packet capture modules. PF_RING is now a first class citizen in DAQ, as in PF_RING/userland/snort you can find the PF_RING DAQ module. This modules not only allows snort to take advantage of PF_RING acceleration, but it allows to offload to PF_RING some of its processing tasks. For instance when snort marks a specific communication flow as "bad" (i.e. all the packets of the flow must be dropped as they are suspicious), instead of wasting precious CPU cycles moving packets off the adapter to snort (that will drop them), the PF_RING DAQ module stops those packets directly into PF_RING. In fact, this module intercepts the snort "verdict" and if for some reason snort decides that the flow must be dropped, PF_RING transparently adds a filtering rule into the kernel module so that future packets belonging to the same flow will be dropped into the kernel. As you can imagine, this is a major performance improvement as packets marked as bad, will not hit snort at all, nor be moved off PF_RING as they will be dropped immediately. This is not all you can do with PF_RING. In fact when combining multi-queue adapters such as Intel ET (1 Gbit) and X520 (10 Gbit) with PF_RING with TNAPI, it is possible to run multiple snort instances on top of the same ethernet port, where each snort instance will receive a portion of the traffic. The magic is done by PF_RING that avoid to merge NIC RX queues, by preserving the queue information and creating virtual ethernet interfaces (e.g. eth0@3 is the 3rd RX queue of the eth0 adapter). This feature allows users to parallelize snort execution (snort is single threaded so this is a good feature to have) as several instances can run simultaneously sitting on different CPU cores, while fully unleashing the power of multi-core computers as all cores are busy processing packets. If this is seems to be enough, you can do more. As most of you know, 82599-based adapters have the ability to assign traffic to cores by means of a component named flow director (FD). As FD is implemented in hardware, the dirty work is performed inside the NIC and not into the kernel. FD allows packets to be dropped into the NIC (i.e. at wire-speed in hardware) by means of filtering rules (up to 32k for perfect filters or infinite for hashing where you just have to pay in term of false positives). PF_RING allows to take advantage of FD so that via PF_RING you can set a filtering rule and if your nic is FD-aware, packets will be filtered into the NIC without any CPU cycle wasted on this. If you want, you can read all the details in this article. Imagine that snort encounters a bad flow and decides to drop it. Thanks to PF_RING, if you have a 82599-based adapter, you can block those bad packets directly into the NIC. transparently. At 10 Gbit wire speed. All this using a commodity network adapter. Is the time of costly FPGA-based NICs over? Maybe. Have fun with PF_RING for now.

Meet ntop at RIPE 61 Rome (15-19 November)

lucaderi — Wed, 03 Nov 2010 12:41:38 +0000

Those who are interested in hearing about high-speed packet capture and filtering and to monitoring in general, can show up at the next RIPE 61 meeting that till take place in Rome (15-19 November). I will be speaking about hardware packet filtering using commodity adapters and how this work can be used in real life, ranging from ntop/nProbe to snort and network troubleshooting.

Using PF_RING with Snort and Suricata for IDS/IPS Acceleration

lucaderi — Wed, 10 Nov 2010 06:11:17 +0000

Some users are exploiting PF_RING acceleration to improve popular IDS/IPS applications such as Snort and Suricata. Suricata leveraged PF_RING since day one thanks to Will Metcalf, whereas I have added (again together with Will) support in snort using the DAQ library part of the 2.9 version. Acceleration does not mean just improved packet capture, but also the ability to fully exploit multicore architectures by spreading packets across multiple application instances. This is a unique feature that can't be found in pcap-based libraries. This is an excerpt from the snort-users mailing list that describes how to load balance traffic across multiple snort instances using the DAQ/PF_RING module.

Re: [Snort-users] Multiple Snort Instances - One Interface From: Will Metcalf - 2010-10-29 18:40
Ahhh James Thorton you found the marble in the oatmeal your a lucky

lucky lucky little boy because you wanna know why you get to drink

from the IDS FIREHOSE!!!

Butchering quotes for Weird Al Yankovic's masterpiece UHF aside, this

now possible with the version of PF_RING in SVN. It should be noted

that the code is probably still of beta quality. Luca Deri did a lot

of awesome work developing a PF_RING aware DAQ module. I helped a bit

by adding support for load balancing based on flow via PF_RING

clusters and setting per process affinity. It is incomplete at the

moment i.e. last time a checked it didn't have support for filtering

packets. Additionally code should probably added to allow a list of

processes to be added to the cpu set. If you want to check it out you

can follow instructions here on building PF_RING as a dkms module.
http://www.openinfosecfoundation.org/doc/INSTALL.PF_RING.txt
Additionally you will have to build PF_RING aware daq by going into

the daq-0.2 dir and doing the following

./configure --with-libpfring-libraries=/opt/PF_RING/lib

--with-libpfring-includes=/opt/PF_RING/include

--with-libpcap-libraries=/opt/PF_RING/lib

--with-libpcap-includes=/opt/PF_RING/include

LD_RUN_PATH="/opt/PF_RING/lib:/usr/lib:/usr/local/lib"

--prefix=/opt/PF_RING && make && sudo make install

Then download snort 2.9.0 and build with the following params.

PATH="/opt/PF_RING/bin:$PATH" ./configure --enable-perfprofiling

--with-libpfring-libraries=/opt/PF_RING/lib

--with-libpfring-includes=/opt/PF_RING/include

--with-libpcap-libraries=/opt/PF_RING/lib

--with-libpcap-includes=/opt/PF_RING/include

LD_RUN_PATH="/opt/PF_RING/lib:/usr/lib:/usr/local/lib"

--prefix=/opt/PF_RING && make && make install

/opt/PF_RING/bin/snort -c etc/snort.conf --pid-path=./log2 -D --daq

pfring -i eth1 --daq-var clusterid=44 --daq-var bindcpu=1 -l ./log1

/opt/PF_RING/bin/snort -c etc/snort.conf --pid-path=./log3 -D --daq

pfring -i eth1 --daq-var clusterid=44 --daq-var bindcpu=2 -l ./log2

/opt/PF_RING/bin/snort -c etc/snort.conf --pid-path=./log4 -D --daq

pfring -i eth1 --daq-var clusterid=44 --daq-var bindcpu=3 -l ./log3

You will then have traffic load balanced across multiple snort

processes based on flow. Enjoy drinking from the ids firehose ;-)...

Also, you could also always checkout other err ummm open source IDS

projects that support this functionality natively ;-)

Regards,

Will

On Fri, Oct 29, 2010 at 12:48 PM, James Thornton

<
james.f.thornton@gmail.com
> wrote:

> I could be mistaken, but believe you need the TNAPI driver with PF_RING to

> accomplish this. TNAPI driver is roughly $400. That is outside of my

> budget at the moment.

>

> Thanks,

>

> Jim T

>

> On Fri, Oct 29, 2010 at 1:30 PM, Will Metcalf <
william.metcalf@gmail.com
>

> wrote:

>>

>> Whats wrong with using PF_RING to do this? ;-)

>>

>> Regards,

>>

>> Will

>>

>> On Fri, Oct 29, 2010 at 8:38 AM, James Thornton

>> <
james.f.thornton@gmail.com
> wrote:

>> > All -

>> >

>> > On my quad core system, I would like to load-balance traffic from a

>> > single

>> > Ethernet device across two or four Snort processes. Has anyone on the

>> > list

>> > accomplished this in the past? Aside from the PF_RING library, I've had

>> > no

>> > success on Internet searches for load balancing software modules that

>> > provide this capability. Any guidance from the group would be

>> > appreciated.

>> >

>> > Thank You,

>> >

>> > Jim T

>> >

>> >

>> > ------------------------------------------------------------------------------

>> > Nokia and AT&T present the 2010 Calling All Innovators-North America

>> > contest

>> > Create new apps & games for the Nokia N8 for consumers in U.S. and

>> > Canada

>> > $10 million total in prizes - $4M cash, 500 devices, nearly $6M in

>> > marketing

>> > Develop with Nokia Qt SDK, Web Runtime, or Java and Publish to Ovi Store

>> >
http://p.sf.net/sfu/nokia-dev2dev
>> > _______________________________________________

>> > Snort-users mailing list

>> >
Snort-users@lists.sourceforge.net
>> > Go to this URL to change user options or unsubscribe:

>> >
https://lists.sourceforge.net/lists/listinfo/snort-users
>> > Snort-users list archive:

>> >
http://www.geocrawler.com/redir-sf.php3?list=snort-users
>> >

>

>
Re: [Snort-users] Multiple Snort Instances - One Interface From: Jim Hranicky - 2010-11-01 15:52
On Fri, 29 Oct 2010 13:40:08 -0500

Will Metcalf <
william.metcalf@gmail.com
> wrote:

> You will then have traffic load balanced across multiple snort

> processes based on flow. Enjoy drinking from the ids firehose ;-)...

> Also, you could also always checkout other err ummm open source IDS

> projects that support this functionality natively ;-)

Damn:

--- /tmp/snort1.out ---

*** Caught Usr-Signal

Packet I/O Totals:

Received: 2608501

Analyzed: 2608501 (100.000%)

Dropped: 0 ( 0.000%)

Filtered: 0 ( 0.000%)

Outstanding: 0 ( 0.000%)

Injected: 0

--- /tmp/snort2.out ---

*** Caught Usr-Signal

Packet I/O Totals:

Received: 2988261

Analyzed: 2988261 (100.000%)

Dropped: 0 ( 0.000%)

Filtered: 0 ( 0.000%)

Outstanding: 0 ( 0.000%)

Injected: 0

--- /tmp/snort3.out ---

*** Caught Usr-Signal

Packet I/O Totals:

Received: 2417539

Analyzed: 2417539 (100.000%)

Dropped: 0 ( 0.000%)

Filtered: 0 ( 0.000%)

Outstanding: 0 ( 0.000%)

Injected: 0

--- /tmp/snort4.out ---

*** Caught Usr-Signal

Packet I/O Totals:

Received: 2382326

Analyzed: 2382326 (100.000%)

Dropped: 0 ( 0.000%)

Filtered: 0 ( 0.000%)

Outstanding: 0 ( 0.000%)

Injected: 0

--- /tmp/snort5.out ---

*** Caught Usr-Signal

Packet I/O Totals:

Received: 2427689

Analyzed: 2427689 (100.000%)

Dropped: 0 ( 0.000%)

Filtered: 0 ( 0.000%)

Outstanding: 0 ( 0.000%)

Injected: 0

--- /tmp/snort6.out ---

*** Caught Usr-Signal

Packet I/O Totals:

Received: 2577258

Analyzed: 2577258 (100.000%)

Dropped: 0 ( 0.000%)

Filtered: 0 ( 0.000%)

Outstanding: 0 ( 0.000%)

Injected: 0

--- /tmp/snort7.out ---

*** Caught Usr-Signal

Packet I/O Totals:

Received: 2406892

Analyzed: 2406892 (100.000%)

Dropped: 0 ( 0.000%)

Filtered: 0 ( 0.000%)

Outstanding: 0 ( 0.000%)

Injected: 0

--- /tmp/snort8.out ---

*** Caught Usr-Signal

Packet I/O Totals:

Received: 2528434

Analyzed: 2528434 (100.000%)

Dropped: 0 ( 0.000%)

Filtered: 0 ( 0.000%)

Outstanding: 0 ( 0.000%)

Injected: 0

That was 5 minutes ago...I'm now up to ~7M Received/Analyzed per process

without a drop on any.

Wow.

--

Jim Hranicky

IT Security Engineer

Office of Information Security and Compliance

University of Florida

Monitoring Traffic Using ntop: Cisco Traffic Analyzer

lucaderi — Thu, 11 Nov 2010 09:14:35 +0000

Most network administrators use ntop for monitoring ethernet traffic. ntop can do much more than this and also monitor Fibre Channel and SCSI traffic. Cisco Traffic Analyzer is a software product based on ntop whose goal is to give Cisco MDS 9000 users a view of the network traffic. Did you know that ntop can also do this?

How to Configure nProbe to Export URLs and Latency via NetFlow

lucaderi — Wed, 24 Nov 2010 08:02:11 +0000

Our friends at Plixer have written a nice article about how to use nProbe to export HTTP and latency information. Note that you can also use the nProbe http plugin to trace HTTP events and rebuild user sessions. This as netflow is not exactly the best protocol to use for exporting this information. The available options are:


--http-dump-dir 
--http-exec-cmd Command executed whenever a directory has been dumped
--dont-hash-cookies Dump cookie string instead of cookie hash
--dont-nest-dump-dirs Don't create subdirs on the dump directory
--max-http-log-lines Max number of lines per log file (default 10000)

For instance nprobe --http-dump-dir ~/http --http-exec-cmd /home/deri/processHTTP.py --max-http-log-lines 500

dumps files in ~/http of up to 500 lines and once the file has been dumped is is processed using proccessHTTP.py. Dump files have the following format:

#
# Client	Server	Protocol	Method	URL	HTTPReturnCode	Referer	UserAgent	ContentType	Bytes	BeginTime	EndTime	Flow Hash	Cookie
#
65.175.140.3	www.plixer.com	http		/blog/wp-content/plugins/wp-cumulus/tagcloud.swf?r=8093784	200	www.plixer.com/blog/index.php?s=netflowMozilla/5.0 (Windows; U; Windows NT 6.1; en-US) AppleWebKit/532.5 (KHTML, like Gecko) Chrome/4.1.249.1064 Safari/532.5	application/x-shockwave-flash	39720	1273583995	1273583996	1507291460	80462
82.211.65.226	www.plixer.com	http		/includes/AC_RunActiveContent.js	304	www.plixer.com/support/download_request.php	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)		3869	1273584001	1273584002	1794801542	68289
82.211.65.226	www.plixer.com	http		/includes/functions.js	304	www.plixer.com/support/download_request.php	Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.30; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729)		3676	1273584001	1273584002	1794801542	68976

that enable you to do everything with them ranging from web stats to network forensics.

Latency using NetFlow from the nProbe: Part 1

lucaderi — Sat, 18 Dec 2010 08:46:23 +0000

nProbe v6 features both network and application latency measurement. Our partner Ravica has written a nice tutorial on this subject.

ntop in 2011

lucaderi — Thu, 06 Jan 2011 21:34:47 +0000

Most of you know only small pieces of the ntop project. I have decided to prepare a few slides that you can use as tutorial for showing how the various project components can be used to efficiently monitor networks, and what you can expect in 2011 from this project (see for instance vPF_RING and n2disk). Happy new year.

HTTP Traffic Analysis Using nProbe and Scrutinizer

lucaderi — Wed, 26 Jan 2011 22:53:26 +0000

Are you interested in getting URL information from NetFlow? The nProbe NetFlow probe or the nBox can do it. Paul at Plixer International recently wrote a blog on Recommended nProbe Templates. For a foundation on this topic, check out his blog. As an extension of his blog, I’ll explain how to get URLS from the nProbe. Scrutinizer from Plixer is the ideal tool for advanced IPFIX reporting and network traffic analysis. Below is a top domain report. For our company, the first page of this report is usually legitimate sites, so I went to page 3 in the pagination. There I noticed craigslist.org.

I wanted to see a list of the URLs people are hitting on this domain. I clicked on craigslist.com:

Below is a list of the URLs people are viewing on craigslist.com for the timeframe selected.

I copied the URL and pasted it into my browser. After viewing several URLs, I was able to determine that visits to this website were not work related.

Look at the pagination below the table (i.e. 25 pages). This is showing that approximately 250 URLs have been viewed on craigslist for the time frame selected. I selected a URL which brought up a menu of reports I can run for it.

Below we see the filter on the left for the domain craigslist.org and the URL filter. Host 10.1.7.21 is our culprit.

What other URLs is this host visiting? Just click it to find out! nProbe and Scrutinizer make network traffic analysis that simple!

PF_RING and transparent_mode

lucaderi — Wed, 02 Feb 2011 14:18:51 +0000

Many PF_RING users know that for avoid patching the Linux kernel, as of PF_RING 4.x packets are received though NAPI. This means that the packet journey is the same used in standard Linux, thus the performance improvement with respect to vanilla Linux is minimal (< 5%) although PF_RING allows to do many more things than the standard AF_PACKET. In order to boost performance PF_RING supports a parameter named transparent_mode that can be used when the module is loaded into the kernel as follows insmod pf_ring.ko transparent_mode=X where X can either be set to 0, 1 or 2. Its meaning changes depending on the NIC driver used for the interface on which you hook your PF_RING-based application (e.g. pfcount); so it might be that on one interface you have a standard driver, and on another interface you have a PF_RING-aware driver. Currently all supported PF_RING-aware drivers reside in PF_RING/drivers in addition to TNAPI. Note that in the case of DNA drivers, as the kernel is totally bypassed, the transparent_mode parameter has no effect. The following table explains the meaning of this parameter.

Mode	Standard driver	PF_RING-aware driver	Packet Capture Acceleration
0	Packets are received through Linux NAPI		Same as Vanilla Linux
1	Packets are received through Linux NAPI	Packets are passed to NAPI (for sending them to PF_RING-unaware applications) and copied directly to PF_RING for PF_RING-aware applications (i.e. PF_RING does not need NAPI for receiving packets)	Limited
2	The driver sends packets only to PF_RING so PF_RING-unaware applications do not see any packet	The driver copies packets directly to PF_RING only (i.e. NAPI does not receive any packet)	Extreme

Cisco(Live) and ntop

lucaderi — Wed, 02 Feb 2011 15:21:17 +0000

Just like Apple is the computer brand I use since 1985, for me Cisco is the networking company, the one that created the first routers and switches on which the Internet was built. It has been a great surprise when last summer I have been contacted by a Cisco representative, who has asked me whether I was interested in starting a new project on NetFlow. After the initial surprise, of course I have accepted, and now it's a few months I work with (not for) Cisco on this nice and challenging project. I have worked (either employed or as a consultant) for a few companies in the past years, and in general companies are not quite open to new ideas for many reasons: "Who's this guy coming here to tell us what to do?" is an answer I have heard several times. Instead the experience with Cisco has been unique. For the first time in my life, I have managed to work on a large team (sales, training, support, marketing, testing, hardware and software design) who has accepted me as a resource and not as an intruder. This is not very common, and as a non-Cisco employee, I can tell you that what's written here is definitively true. This week, the initial result of this project is previewed at CiscoLive in London. Yesterday I had the chance to fly over and see the new born baby on a live demo sending flows to the Plixer Scrutinizer collector. If you have a chance to be at CiscoLive this week, go to see it live at Unified Fabric - Virtualized Distributed Data Center booth in the show floor: Kedar will show you a sneak preview. Instead if you cannot fly to London, stay tuned on this blog as I will soon publish more about this project. [gallery link="file" columns="2" orderby="title"]

Say hello to NetFlow-Lite (NFLite)

lucaderi — Sun, 06 Feb 2011 11:14:51 +0000

As you all know, NetFlow has been initially designed for routers (or L3 switches if you wish), contrary to sFlow that instead has been deployed mostly on switches. In this view, people use NetFlow just for monitoring internet traffic, as NetFlow is not supported across the product portfolio due to dedicated ASIC required. NetFlow-lite (first introduced with Catalyst 4948E) bridges the gap by providing a lightweight solution that allows capturing of important flow information through packet sampling mechanisms combined with the extensibility of NetFlow version 9 and IPFIX.

What is NetFlow-Lite?

In a nutshell NFLite is a technique that allows to move the NetFlow cache outside of the switch into an external entity that contains a NFLite to NetFlow (v5/v9/IPFIX) converter. nProbe (6.1.6 or above) is the first application able to do this conversion, and it has been closely developed by ntop and Cisco to make sure it complies with the protocol specs. This conversion is necessary as NFLite flows generated by the switch are basically encapsulating packet samples, that need to be converted into "legacy" flows that standard collectors can process. Technically speaking NFLite flows are formatted as standard V9 flows with a custom template, that tells the NFLite collector (nProbe in our case) how to decode the enclosed packet that is cut to the configured snaplen.

NetFlow-Lite Advantages

The NetFlow-lite capability on Cisco Catalyst 4948E aims at providing traffic visibility into data center access layer, including server-to-server user-to-server activities.
With the help of nProbe, netflow-lite can be analyzed and supported by any netflow collector that understand version5, 9 or IPFIX.
Netflow-lite solution can be designed from a tiered approach for large scale data center: by deploying an nProbe per zone to summarize NetFlow-lite data within the zone, it reduces the amount of bandwidth overhead and allows data from multiple zones to be analyzed and gathered by a centralized netflow collector.
nProbe & NetFlow-lite supports NetFlow version 9 and IPFIX.
nProbe act as a collector to netflow-lite switches and as a probe to end collector. It works seamlessly with existing, already deployed NetFlow collector as well as allowing a wide choice of NetFlow collector for new deployment.

NetFlow-Lite and nProbe

As stated above, nProbe is the first application to support NFLite. You might wonder what are the challenges behind this work. I can list some:

NFLite collector does not simply receives flows and dumps them on a DB (as most collectors do). It must also implement the flow cache, that is a typical activity that a netflow probe does, and that is missing in collectors (they usually filter and aggregate flows, but nothing more than that).
NFLite flows contain packet samples, that are basically packets seen on switch ports, along with some metadata information such as the port on which such packet has been seen. This means that those packets need to be decoded (i.e. packet parsing) and consolidated into the flow cache. Dumping them on disk in raw format (as NetFlow collectors usually do) is useless as you can't do much with them.
Depending on the number of NFLite-aware switches, traffic and sampling rate, that send traffic towards the same nProbe, the amount of packets can be as high as 1-2 million flows/sec. Most NetFlow collectors can handle a sustained collection speed of a few tenth/hundred flows/sec, that is not adequate for providing accurate network visibility.

Using nProbe as NetFlow-Lite Probe/Collector

In order to use nProbe with NFLite, you don't have to do much.

Installation

From Source

```
tar xvfz nProbe-xxx.tgz
```
```
cd nProbe-xxx
```
```
./autogen.sh
```
```
make
```
```
sudo make install
```

From Binary Package
- RedHat and Centos
```
rpm -i nprobe-xxx.rpm
```
- Ubuntu/Debian
```
dpkg -i nprobe-xxx.deb
```
- Windows Use the graphical installer that comes with the nProbe package

Usage

As said before, with NFLite nProbe acts as both a collector (i.e. it collects and decodes NFLite flows received by NFLite devices such as Catalyst 4948E) and probe (towards a remote NetFlow collector). Note that if your collector is nProbe itself, you can avoid sending converted flows to yet another nProbe instance, but you can use the same nProbe instance you used for NFLite conversion. On the 4948E side you have to make sure the NFLite is properly configured. Example

netflow-lite exporter check
 cos 0
 dscp 60
 ttl 254
 transport udp 1000
 template data timeout 60
 options sampler-table timeout 60
 options interface-table timeout 60
 source 1.1.1.1
 destination 1.1.1.3
 export-protocol netflow-v9
!

netflow-lite sampler check
 packet-rate 32
 packet-section size 64
 packet-offset 0
!         

interface GigabitEthernet1/1
 no switchport
 ip address 40.40.40.1 255.255.255.0
 netflow-lite monitor 1
   sampler check
   exporter check

A typical command line for starting nProbe is the following:

nprobe --collector-port 3000 -i none -n 5.5.5.10:2055 -b 2  -w 512000

where:

3000 is the local UDP port on which NFLite flows are collected
none means that nProbe does not capture packets from a physical device, but it rather receive flows via UDP. Note that you can start nProbe for both collecting NFLite flows and at the same time creating flows capturing packets from a specific interface.
5.5.5.5:2055 is the IP address and port of the NetFlow collector to which NFLite converted flows will be sent.
512000 is the initial size of the NetFlow cache that will be used for aggregating NFLite flows.

Conclusions

NetfFow-Lite is slated to come out on 4948E and 4948E-F in the next software release. As of today, NFLite support is part of nProbe for both Unix and Windows. If you are interested in testing NFLite, you have no excuse as you can download nProbe today. What I am currently doing these days is the ability to improve collection speed in nProbe. Even if today we can handle more than 500k flows/sec per NFLite device, I am developing a new technique (PF_RING can help) that allows to break these limits. Stay tuned.

Credits

Many thanks again to Cisco and to the Catalyst team for letting ntop to be part of this challenging project!

Using Hardware Timestamps with PF_RING

lucaderi — Thu, 10 Feb 2011 23:44:21 +0000

Up to some years ago, hardware timestamps were available only on costly FPGA-based NICs. Slowly, NIC manufactures started to consider hw timestamps as an important feature, and they started to introduce them in new cards. As of today Silicom PE2Gi80, Intel 1 Gbit Ethernet Server Adapter i340 (1 Gbit) and Neterion X3110/X3120 (10 Gbit) offer off-the-shelf hardware timestamps. These cards do not feature a GPS connector, but support IEEE 1588 for clock synchronization. The accuracy of the hw timestamps of these cards ranges from 3 to 7 ns. PF_RING has been enhanced to support hw timestamps, whatever is their source. PF_RING 4.6.1 introduced hw timestamp support in both the PF_RING kernel module, applications (e.g. pfcount), and even libpcap. The user-space pfring library via ioctl()'s call with SIOCSHWTSTAMP, informs the NIC that received packets must be timestamped. The new PF_RING module does the magic by exploiting and passing to userland all the hw timestamps present in packets. Using them is pretty simple

Intel 1Gbit
- cd PF_RING
- insmod ./drivers/intel/igb/igb-2.4.12/src/igb.ko
Neterion X3110/X3210
- cd PF_RING
- insmod ./drivers/neterion/vxge.ko pf_ring_en=1 lro=0 func_mode=0

Supposing to access you card as ethX, you can access hw timestamps using pfcount. Example:


pfcount -i ethX
00:07:14.230020938 [00:1B:ED:26:A9:4F -> 00:16:9C:6E:FD:80][eth_type=0x0800] [77.95.141.6:1637 -> 62.149.128.203:25] [tos=0][tcp_seq_num=2571580375][caplen=128][len=1514][parsed_header_len=0][eth_offset=0][l3_offset=14][l4_offset=34][payload_offset=54]

generated packet with ns timestamp accuracy. The libpcap library present under PF_RING/userland and it companion product tcpdump, demonstrate hw timestamps


root@ubuntu:/home/deri/PF_RING/userland/tcpdump-4.1.1# ./tcpdump -i eth0
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on eth0, link-type EN10MB (Ethernet), capture size 8192 bytes
23:49:37.155010242 IP 10.211.55.10.ssh > 10.211.55.2.63850: Flags [P.], seq 3954409477:3954409669, ack 1206887258, win 239, options [nop,nop,TS val 34171550 ecr 663605968], length 192
23:49:37.155272286 IP 10.211.55.2.63850 > 10.211.55.10.ssh: Flags [.], ack 192, win 65535, options [nop,nop,TS val 663605974 ecr 34171550], length 0
23:49:37.202867182 ARP, Request who-has 10.211.55.1 tell 10.211.55.10, length 28

Applications based on hw timestamps can be manyfold. You can use it to measure network delay or reorder packets hitting different adapter ports.

The author would like to than Silicom, Exar, and Intel for their support during this project.

Developing Monitoring Applications based on PF_RING

lucaderi — Tue, 22 Feb 2011 09:23:25 +0000

Many people use PF_RING just as a "better" libpcap. PF_RING is much more than that, as it can significantly simplify the design of network monitoring applications as well better exploit modern multi-core architectures and network adapters. For those willing to dive into PF_RING, I have released an updated user's guide that can introduce you to the PF_RING API. Do not forget that there's a detailed PF_RING tutorial available, as well several code examples for showing in practice what PF_RING can offer you.

nProbe IPFIX Interoperability Tests

lucaderi — Sat, 19 Mar 2011 08:00:10 +0000

Over the past month quite a lot of effort has been put on the IPFIX side of nProbe. Recently, nProbe has been successfully verified by Juniper as an IPFIX (in addition to v9) collector for flows generated by Juniper MX routers, and Cisco Catalyst 4948E switches. In order to further guarantee users that nProbe respects the IPFIX standards, nProbe will be tested against other IPFIX implementations at the IPFIX Interoperability Event that will take place next week in Prague. In the following months, ntop will also try to push in the IPFIX standard some information elements supported (as PEN) by nProbe but not yet in the standard, that include:

Application Identification (e.g. SMTP, HTTP, FaceBook, Twitter).
SMTP, DNS and VoIP.
Tunnelled information (e.g. GRE, GTP).

Later this spring, ntop will go under strong code rewrite, and we'll be synchronized with nProbe flow support, so that all the features you can find in nProbe (for both probe and collection) will also be available in ntop (as collector).

Remote nsec TimeStamps using PF_RING and cPacket Devices

lucaderi — Wed, 23 Mar 2011 08:56:41 +0000

PF_RING supports nsec timestamps from some modern NICs, such as those based on the Intel 82580 (e.g. Silicom PE2G4i80). But NIC timestamps require installing and running the application on the machine where the adapter is installed. Furthermore, by the time the traffic gets from the wire to the the NIC, its temporal behavior might have been altered by queuing, buffering, and switching caused by SPAN ports or aggregation devices. cPacket offers products that deliver nanosecond accurate timestamps directly from the wire, before switching, queuing, or bufffering. cPacket inline hardware probes are passive and transparent. They deliver precise nanosecond timestamps, which reflect the actual traffic behavior in real time.

The hardware time stamp is snapped by the cPacket inline probe, directly at the wire as soon as the packet data hits the physical interface. The cPacket probe augments the original packet with the hardware time stamp, by attaching it to to the end of the Ethernet frame (extra 12 bytes) as follows:

<4 bytes epoch time (seconds)><4 bytes fractional time (nanoseconds)><4 bytes reserved>

PF_RING has been extended to recognize cPacket timestamps, strip them off to recover the original packet, and set the kernel data structures (skbuff). This means that these "hardware timestamps directly from the wire" can be seamlessly available for any user space applications. To demonstrate this technology out of the box, we created a software package (cpacket_ts.tgz) that contains:

ixgbe driver (both in source and binary format for Ubuntu 10.10 server and RedHat EL 6 x84)
Enhanced tcpdump application that uses the precise hardware timestamps natively
n2disk, packet-to-disk application that can efficiently dump packets on disk in pcap format, which includes the accurate hardware time stamp in common pcap file, which can be used by other applications like Wireshark.

The source code used to generate the package (beside the unreleased n2disk application) is part of the PF_RING project. For more information please refer to the cPacket press release.

ntop and Silicom Inc join the forces

lucaderi — Mon, 28 Mar 2011 19:45:46 +0000

Since a few months ntop and Silicom have started to work together on various network-related topics. The idea is to enhance PF_RING and TNAPI in order to offer better products and support for both the community and Silicom customers. Furthermore, Silicom produces very advanced products such as the content director card and the packet processor card, that could solve various network-related tasks including:

packet mirroring, tapping, duplication
packet steering
QoS enforcement
packet traffic analysis

As these activities are performed in hardware, they operate at wire-speed (at both 1 and 10 Gbit) without any CPU assistance, turning commodity Linux-based servers into powerful packet processor appliances that can deliver the same performance of costly ad-hoc devices. The main difference between these two approaches is both in price but also in freedom to leverage on open-source software that allows users to adapt the solution to their needs without being locked with costly, proprietary solutions. The first results of this joint development effort, will be released in the next few months. Stay tuned.

nProbe complies with the IPFIX specification

lucaderi — Mon, 28 Mar 2011 13:13:45 +0000

Last week I have participated to an IPFIX interoperability event held in Prague, right before the IETF 80. In the picture below you can see me between Benoit Claise (Cisco, one of the IPFIX/NetFlow fathers) and Jiri Novotni (Invea-Tech).

nProbe 6.3.x has been successfully tested against all the available implementations including Vermont, SiLK, nfdump/IPFIX (Cesnet). nProbe has passed all the IPFIX interoperability tests as both probe (over SCTP, UDP, and TCP) and collector (UDP), dissecting both IPv4 and IPv6 traffic, and also converting NetFlow-Lite flows into IPFIX flows. Most of you are still using nProbe over udp only, but you should know that you can specify other transports as follows: nprobe -n ://:. Example:

nprobe -n sctp://1.2.3.4:2055
nprobe -n udp://fe80::ca2a:14ff:fe18:29ce:2055

PRESS RELEASE Prague, March 25th 2011

INVEA-TECH's FlowMon Probe (NetFlow/IPFIX statistics generator) and ntop's nProbe (NetFlow/IPFIX collector) are mutually compatible and have been successfully tested to be compliant with IPFIX standard. The IPFIX compliance was tested during IPFIX interoperability event held in Prague in March 2011.

The FlowMon Probe and nProbe passed all the IPFIX interoperability tests and were also successfully tested against all available IPFIX implementations (e.g. SiLK, Vermont). INVEA-TECH and Luca Deri mainly focused on deep testing between FlowMon Probe and nTop collector to make them perfectly compatible with each other. The whole process results in a possibility to use professional appliance (FlowMon Probe) for network monitoring and exporting data in IPFIX to nProbe (open-source collector) which supports all available IPFIX features.

About INVEA-TECH:

INVEA-TECH is an European vendor. We develop and market comprehensive network solutions for networks from 10Mbps to 10Gbps and more. The core idea of the company is to provide complete range of innovative products and services for network security, network monitoring, traffic analysis and hardware-accelerated applications development. Our flagship product is FlowMon - complete flow (NetFlow/IPFIX) monitoring and security solution for all networks from 10Mbps to 10Gbps.

About ntop:

ntop develops open-source traffic monitoring solutions running on commodity hardware. In addition to nProbe, Its portfolio includes a Linux packet capture acceleration including virtual machine support, and packet-to-disk applications.

Tuning nProbe 6.4 Scalability and Performance

lucaderi — Sun, 03 Apr 2011 12:36:08 +0000

Release 6.3 of nprobe targeted IPFIX compatibility. In release 6.4.x (just introduced) the main focus has been on scalability and performance. Until 6.3, the nProbe architecture was not really exploiting multicore systems, due to heritage of previous versions. With this release nProbe reaches a new level as you can see from the graph below (traffic was generated using an IXIA 400, flows last 5 seconds, and are emitted in V5 format, PF_RING 4.6.3, Intel e1000e capture adapter with PF_RING-aware driver [no TNAPI]). Both graphs depict the sustained throughput rate (Y axis, expressed in Kpps, thus packets-per-second that is much more challenging than volume, or Gbps) with no packet loss, when compared the number of threads (X axis) used in nProbe (-O ). As you can see the new version on a dual core machine performs much better than the previous version. Nevertheless the main advantage is that with this version you can process at least 1 Mpps with no loss, this on both machines. If you use a PF_RING-aware driver on top of a multi-queue card, you can scale up to 10 Gbit (or multi-gbit) netflow monitoring by:

binding a nProbe (in single thread mode) to each core
binding a each nProbe to a RX queue

Guess what you can achieve when using nProbe on top of TNAPI... Of course you can choose to run nprobe on your server or get a turn-key nBox appliance that includes nProbe (you can look at this video created by our Plixer colleagues for more information about the nBox).

How to Monitor Latency Using nProbe

lucaderi — Sat, 09 Apr 2011 08:17:00 +0000

On May 12th in Bolzano (I) at the Nagios World Conference Europe, I will give a speech about network and application latency monitoring using nProbe. This is an hot topic, in particular for those who think of NetFlow/IPFIX as just a way to count bytes and packets. NetFlow/IPFIX instead is (this is my opinion) an open protocol that can be used to carry monitoring data from observation points to monitoring systems. The fact that many probes export you just bytes 'n packets info, it's not a protocol limitation but a probe limitation. In this respect nProbe supports many extensions such as latency monitoring, information about packets out-of-order, retransmitted, fragmented, average flow packet size and many more. In particular, latency is computed both as network and application latency:

Network Latency (Network Delay on nProbe parlance)

Network delay is computed observing (at the nProbe observation point that can be anywhere between the client and the server) the 3-way-handshake packets and computing the time difference between them. As these packets are processed in the IP protocol stack, we assume that there's little (if any) delay added by client/server, and what we measure is basically the time taken by packets to traverse the network.

Application Latency (Application Delay on nProbe parlance)

The application latency is the time taken by a server to process a request. nProbe computes it as the time from the first packet with payload sent by the client, and the time of the first response packet following client's request. If you want to know the whole processing time (from first to last byte of request plus response) you can see it from the flow duration. In the above figure you can see how this works in the case of HTTP. Please note that for same protocols, application latency computation is meaningful (e.g. HTTP, DNS), whereas for others (e.g. SSH, FTP) is not. Please note that if you enable the HTTP plugin in nProbe, you can get per-URL flow information as nProbe in that more will decode HTTP and follow requests, even if they are pipelined in HTTP/1.1 mode over the same TCP connection. If you compute the difference between application and the total network delay, you can obtain the application latency as shown in the above figure. If you come to the conference, you will hear more about this subject. Hope to see you soon in Bolzano!

ntop at the Nagios World Conference Europe

lucaderi — Sun, 17 Apr 2011 11:57:48 +0000

PRESS RELEASE Bolzano April 13^th 2011 ntop at the “Nagios World Conference Europe™” on May 12th at Bolzano/Italy Luca Deri will be among the keynote speakers at the official European edition dedicated to the well-know Open Source monitoring solution After the American edition in Sào Paolo/Brazil, the European counterpart of the Nagios World Conference™ will be held on May 12^th at Bolzano/Italy. Nagios partner Würth Phoenix, who will host the event has confirmed the participation of namable speakers such as Nagios founder Ethan Galstad, Nagios Plugin coordinator Ton Voon or NS Client creator Michael Medin. Moreover, attendees will expect presentations of the new OTRS version and the interaction of monitoring solutions such as Nagios with the integrated CMDB and the Change Management workflow engine. Martin Loschwitz from Viennese Linbit will refer on real time cluster synchronization with their new developed Linux standard DRBD. Best practices and practical experience with Nagios and Nagios based solutions such as NetEye will come from Fashion brand Diesel or the oldest Business School in the world, the University of Bologna. A special intervention is dedicated to ntop. Founder Luca Deri will perform an introduction on application latency monitoring using nProbe via NetFlow. The event is designed for IT managers, system administrators and developers to network with other IT professionals and to facilitate the exchange of information and ideas with other users. If you already work with Open Source monitoring tools or are considering their use, you´re invited to attend the event and learn actual strategies from technical experts that can improve the way you utilize monitoring in your company or business. The entry is free, therefore an early subscription to ensure your participation is recommended. You can register directly at www.wuerth-phoenix.com/world-conference. Würth Phoenix Group As a software and consulting company of the Würth Group, Würth Phoenix supplies forward-looking ERP and CRM solutions based on Microsoft Dynamics. The company, with bases in Italy, Germany and Hungary draws on worldwide experience in business software as well as many years of expertise in trading, warehouse management and logistics. In IT-System Management, Würth Phoenix relies on Nagios-based monitoring solutions that can be seamlessly integrated into existing IT infrastructures. With WÜRTHPHOENIX NetEye customers can react appropriately to the increasing importance of IT for their business activities by ensuring highest availability and reliability. Recent references include renowned companies such as sports article manufacturer Lotto Sport, Tecnica Group, Thun S.p.A. or automotive supplier Mahle/Glacier Vandervell. Press contact: Gerhard Schenk Würth Phoenix Group Via Kravogl 4 I-39100 Bolzano Tel: +39 0471 564 111 Fax: +39 0471 564 122 press@wuerth-phoenix.com www.wuerth-phoenix.com/press

Power to see all

lucaderi — Wed, 04 May 2011 19:14:09 +0000

Almost a decade ago Dr Ian Graham was in Europe for a series of conferences and I have met him in person along with other people from Politecnico di Torino, that were developing winpcap (one of the key guys of the group, Loris Degioanni, at that time was visiting Endace, thus was not present at the meeting. Loris later become a successful entrepreneur having founded Cace, now Riverbed). For me it was a big pleasure to have such meeting, as Endace was in the early days (and also all of us) and still a sort of University spin-off. The nice thing about Endace was its attitude to be open to the research world, this when they were not yet a world leader. Since that day, I have been in touch with Endace as ntop has been one of the 3rd party apps supported by their platform, and also we've done some projects together a few years ago, not to mention a great lunch with its CTO in a great restaurant on a wine cellar in the Chianti valley a couple of years ago. As all of you know, PF_RING is doing its best to improve packet capture, but competing against a DAG card is quite challenging. Last week Endace donated us a 10 Gbit NIC, for making sure that the ntop apps suite are running nicely on top of their cards. Many thanks to Endace for challenging us again!

Packet Capture Performance at 10 Gbit: PF_RING vs TNAPI

lucaderi — Mon, 09 May 2011 23:21:12 +0000

Many of you are using PF_RING and TNAPI for accelerating packet capture performance, but have probably not tested the code for a while. In the past month we have tuned PF_RING performance and squeezed some extra packets captured implementing the quick_mode in PF_RING. When you do insmod pf_ring.ko quick_mode=1, PF_RING optimizes its operations for multi-queue RX adapters and applications capturing traffic from several RX queues simultaneously. The idea behind quick_mode is that people should use it whenever they are interested just in maximum packet capture performance, and do not need PF_RING features such as packet filtering and plugins. We tested PF_RING vs TNAPI on a low-end server equipped with an entry-level Xeon processor, so that we can test the minimum packet capture performance you can expect. The test result is shown below and it basically says that:

With packets of 512 bytes or more, TNAPI and PF_RING-over-NAPI show a similar performance, and they can operate at wire rate as they capture all available packets.
With 64 bytes packets and a multi-threaded/multi-queue poller application such as pfcount_multichannel (you can find all apps in PF_RING/userland/examples) you can capture 5.1 million packets/sec over NAPI, and 6.7 million packets/sec over TNAPI.
This means that with small packets TNAPI offers a +30% performance increase with respect to NAPI.

As in our experience 10 Gbit links do not have in average more than 2 million packets/sec, this test shows that the performance offered by PF_RING in combination with TNAPI (and in a way even NAPI) is more than adequate as can capture more that 6.5 million packets/sec. Is this enough for you? If not, we'll soon introduce a 10 Gbit DNA driver developed with Silicom, so you can see to which extents we pushed packet capture limits. Stay tuned.

Test Environment

Supermicro server equipped with a single Xeon X3450 running at 2.67 GHz (4 real cores with HT, 8 cores in total)
PF_RING 4.6.4 and ixgbe driver (SVN release 4601)
Intel 82598/82599-based adapter
insmod pf_ring.ko transparent_mode=2 quick_mode=1
Interrupts balanced as follows: PF_RING/drivers/intel/ixgbe/ixgbe-3.1.15-FlowDirector-NoTNAPI/scripts/set_irq_affinity.sh eth4
pfcount -i eth4 -w 5000 -b 99 -l
pfcount_multichannel -i eth4 -w 5000 -b 99 -l 64

Going beyond RSS (Receive-Side Scaling)

lucaderi — Tue, 17 May 2011 08:27:44 +0000

When RSS was introduced some years ago, operating systems had the chance to scale also when handling network packets as RSS allowed incoming packets to be distributed across processor cores. Unfortunately RSS uses a one-way hash, that while distributes packets heavenly across queues, it has some drawbacks. The main one is that if you have a connection A <-> B, packets A->B will go on queue X, and those of B->A on queue Y, where X <> Y. This is a major issue for applications, as you cannot assume that each RX queue is "self-contained", being packets shuffled across RX queues. Applications such as those that analyze traffic, and also IDS/IPS such as Suricata and Snort are affected by this issue. In order to overcome this limitation, PF_RING is now featuring a new API call:

int pfring_enable_rss_rehash(pfring *ring);

If you call it on your applications sitting on top of multi-queue adapters (with single-queue adapters,of course, it has no effect) incoming packets will be rehashed using a bi-directional hash function. In this case, in the above example X == Y, thus you can insulate apps per RX queue. Note that the above call is effective on all driver types (PF_RING-aware and TNAPI). Its cost is basically zero, as PF_RING always parses incoming packets, thus hashing them is a very little cost. It's now time to use multi-queue RX adapters.

Using nProbe as NetFlow-Lite Cache

lucaderi — Mon, 23 May 2011 00:01:31 +0000

As previously stated on this blog, we have worked tightly with Cisco as nProbe has been selected as reference implementation for NetFlow-Lite flow conversion. Although NetFlow-Lite support has been added to nprobe since version 6.1.4 and it's available on all supported platforms (both Unix and Windows), with nProbe 6.5 (just released) we have moved NetFlow-Lite support to the next level. This is because nProbe now features both a

Specialized plugin for NetFlow-lite flow collection that increases of 5x times the collection performance.
PF_RING kernel plugin (Linux only) that can convert NetFlow-Lite flows at over 1 million flows/sec (performance measured on a Xeon server).

For further information about nProbe NetFlow-Lite support, there is a specific web page that describes in detail the plugin features and supported platforms. This is to say that it's now time for Cisco 4948E users (this is the first switch family that support NetFlow-Lite) to upgrade the IOS and start adding visibility to the LAN. Many thanks again to Cisco for having supported this development work.

Invitation to NetFlow-Lite Webinar

lucaderi — Sun, 29 May 2011 06:57:04 +0000

As most of you know, nProbe has recently added NetFlow-Lite support in 6.5 release. NetFlow-Lite is a protocol that brings you visibility into switched networks, similar to what NetFlow "classic" is doing on routed networks. As this technology is pretty new, perhaps you might be interested in hearing more about it right from the source. I would like to invite you to this free webinar that will take place later this week. Shall you be interested please register now.

Cisco NetFlow-Lite: Enabling Traffic Monitoring at Data Center Access

Date: May 31st, 2011 at 10am EST Presenters: Ellie Chou: Product Manager for the Catalyst 4500, Cisco Systems Luca Deri: Founder of ntop.org Michael Patterson: Product Manager of Scrutinizer

Highlights

Gain the visibility into data center server-server traffic & server-user traffic for capacity planning and traffic analysis
Learn about new NetFlow-lite implementation on Cisco Catalyst 4948E: L2 & L3 traffic, IPv4 & IPv6, hardware-based monitoring, and a whole lot more all with standard IPFIX and NetFlow
Integrate NetFlow-lite with NetFlow
Design a scalable data center monitoring solution

NetFlow-Lite and nProbe: a Tutorial

lucaderi — Tue, 31 May 2011 17:28:15 +0000

Today we have held a webinar about NetFlow-Lite with both Cisco and Plixer. Subscribers of this blog should know by now what is NetFlow-Lite and why nProbe is necessary to exploit this technology. Nevertheless you might be interested to know more about NetFlow-Lite, both in terms of features and usage scenarios. Below you can find a could of presentations about this topic that I think are worth reading:

ntop, Implementing a NetFlow Cache for NetFlow-Lite
Cisco Systems, Catalyst 4948E NetFlow-lite
ntop, Using nProbe as NetFlow-Lite Aggregator

In interested, you can also see the video of the webinar: [ YouTube ] [ Vimeo ].

How to send/receive 26Mpps using PF_RING on commodity hardware

lucaderi — Sun, 12 Jun 2011 08:50:21 +0000

Until last month, I have struggled to reach 7 Mpps packet capture using TNAPI. This week I see users still asking questions about how to handle 2 x 1 Gbit wire rate on commodity hardware. I believe it's now time to move to the next level, and achieve full 10Gbit wire rate on both RX and TX, using little CPU cycles so that we can not just capture but also process traffic. Together with Silicom we have developed a 10 Gbit PF_RING DNA driver, that we'll soon introduce to the Linux community. We have been amazed to see how efficient is modern commodity hardware when programming it properly for targeting high-speed networking. The tests below have been made on a Supermicro server PC using a low-end X3450 Xeon processor, and configuring the 82599-based NIC to use just one queue (you can imagine what you can do with multiple queues)

# ./pfsend -i dna:eth5 -g 1 -l 60 -n 0 -r 10
Sending packets on dna:eth5
Using PF_RING v.4.6.5
Estimated CPU freq: 2397347000 Hz
Number of 64-byte Packet Per Second at 10.00 Gbit/s: 14880952.38
TX rate: [current 14754203.44 pps/8104.73 Mbps][average 14752836.88 pps/8103.98 Mbps]

# ./pfcount -i dna:eth5
...
Actual Stats: 14879572 pkts [1000.0 ms][14879259.5 pkt/sec]

RX on two 10 Gbit ports simultaneously
# ./pfcount -i dna:eth5
# ./pfcount -i dna:eth6
Aggregate throughput 25.940 Mpps

Beside the performance figures you can see, it's amazing to see what is the load on the CPU. Let's consider this simple example (RX):

Actual Stats: 7470601 pkts [1'000.04 ms][7'470'242.42 pkt/sec]

PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND

1474 root      20   0  8536  604  496 S   52  0.0   0:39.26 pfcount

So with 52% load on one core (the X3450 has 8 of them) you can capture with pfcount over 7Mpps. It looks that packet capture is no longer a problem, and that as soon as PCIe gen-3 will be out, we can likely support 40 Gbit adapters (as soon as they will become available of course) using DNA.

Introducing the 10 Gbit PF_RING DNA Driver

lucaderi — Sun, 19 Jun 2011 22:42:11 +0000

Today we released PF_RING 4.7.0. It includes 10 Gbit DNA support (RX/TX) for Intel-based 82598/99 ethernet adapters thus you can finally manipulate packets at wire-rate using commodity adapters. With a low-end Core2Duo you can handle more than 11 Mpps per queue, with a Xeon you can have wire rate at any packet size and using limited CPU cycles. We are very grateful to Silicom who has sponsored this developmment work. The source code of the driver is part of PF_RING and it has been placed in the PF_RING SVN. In case you want to try this new DNA driver, you can contact Silicom for getting an evaluation card at no charge.

NetFlow-lite Webcast Invitation

lucaderi — Wed, 22 Jun 2011 14:43:46 +0000

This is to invite you to webcast NetFlow-lite: Enable Data Center-wide Monitoring which is scheduled for Tuesday, 06-28-2011. I will be speaking about NetFlow-lite together with the key Cisco people who worked with me at this project. Hope you will join the workshop!

10 Gbit PF_RING DNA on Virtual Machines (VMware and KVM)

lucaderi — Thu, 30 Jun 2011 14:23:00 +0000

As you know, PF_RING DNA allows you to manipulate packets at 10 Gbit wire speed (any packet size) on low-end Linux servers. As virtualization is becoming pervasive in data-centers, you might wonder whether you can benefit of DNA on virtualized environments. The answer is positive. This post explains you how to use DNA on both VMware and KVM, Linux-native virtualization system. XEN users can also exploit DNA configuring using similar system configurations.

VMware Configuration

In order to use DNA, you must configure the 10G card in passthrough-mode as depicted below.

Once your card has been configured, it will pop-up from your VM. At this point you need to install the DNA driver that is part of the PF_RING distribution.

KVM Configuration

Under KVM you need to make sure you have enabled a few options

Modify the kernel config:

$ make menuconfig

Bus options (PCI etc.)

[*] Support for DMA Remapping Devices

[*] Enable DMA Remapping Devices

[*] Support for Interrupt Remapping

<*> PCI Stub driver

$ make

$ make modules_install

$ make install

Pass "intel_iommu=on" as kernel parameter. For instance, if you are using grub, edit your /boot/grub/menu.lst this way:

title Linux 2.6.36

root (hd0,0)

kernel /boot/kernel-2.6.36 root=/dev/sda3 intel_iommu=on

Unbind the device you want to assign to the VM from the host kernel driver

$ lspci -n

02:00.0 0200: 8086:10fb (rev 01)

$ echo "8086 10fb" > /sys/bus/pci/drivers/pci-stub/new_id

$ echo 0000:02:00.0 > /sys/bus/pci/devices/0000:02:00.0/driver/unbind

$ echo 0000:02:00.0 > /sys/bus/pci/drivers/pci-stub/bind

Load KVM and start the VM

$ modprobe kvm

$ modprobe kvm-intel

$ /usr/local/kvm/bin/qemu-system-x86_64 -m 512 -boot c \

-drive file=virtual_machine.img,if=virtio,boot=on \

-device pci-assign,host=02:00.0

DNA Performance on Virtual Machines

In a previous post, we have tested the DNA performance on bare hardware. Now we're testing the DNA performance on VMs using the same server used in the previous experiment using a Silicom PE10G2SPi-SR 10G card. This allows you to feel the difference in speed. All tests have been performed running a single VM to which we have allocated only one core (single virtual CPU) out of the 8 available on the bare hardware. Both 10G ports have been mapped to a single VM (i.e. the VM is connected to two physical 10G ports), and a fibre is connecting the ports back-to-back.

Test (64 byte packets)	KVM	VMware ESXi 4.1
pfsend/pfcount alone (no both simultaneously)	13'906'747.73 pps/9.35 Gbps	13'689'510.41 pps/9.20 Gbps
pfsend and pfcount (simultaneously on the same VM)	pfsend: 6'688'049.95 pps/4.49 Gbps pfcount: 5'693'580.60 pps/2'732.91 Mbit/sec	pfsend: 6'295'136.14 pps/4.23 Gbps pfcount: 5'614'627.52 pps/2'695.02 Mbit/sec

Final Remarks

On bare hardware you can reach wire rate (14.8 Mpps), whereas on a VM we stop at 13.9 Mpps. This means that using VMs we can reach 94% of the nominal speed using minimal size packets. Considering that our physical box has 8 cores, and we allocated only one core per VM, you can guess what happens when the remaining 7 cores are used...

Tutorial: Monitoring Application and Network Latency with nProbe

lucaderi — Tue, 12 Jul 2011 13:40:40 +0000

Last May I did a tutorial at the Nagios Conference Europe about application and network latency measurement using nProbe. The video of the tutorial has been now released and you are free to watch it if interested in he topic. The video is available here.

Ok, but how much time do I have?

lucaderi — Sun, 31 Jul 2011 15:17:25 +0000

Accelerating packet capture and processing is a constant race. New hardware innovations, modern computing architectures, and improvements in packet capture (e.g. PF_RING) allow applications to reduce the (both CPU and real) time they need for processing packets. But the main question still holds: how much time do I have for processing packets? This is the main point. A common misconception on this field is that hardware-accelerated cards will do the magic and solve all problems. This is a wrong statement. Technologies such as PF_RING, DNA, and those cards reduce the time you need to capture packets to (potentially) zero while reducing the number of CPU cycles your system needs for bringing packets from the wire up to the application. But still how much time do you have? Let's compute it:

1 Gbit/s 1.48 Mpps => 1 sec / 1'488'000 => 0.672 usec / packet
10 Gbit 14.88 Mpps => 1 sec / 14'880'000 => 0.067 usec / packet

So in the worst case, at 1 Gbit you have 0.672 usec/packet. If your application is able to process packets (no matter what) within this time boundary, then you can say your application can handle wire rate at any packet size. At 10 Gbit you have 1/10 of the time you have at 1 Gbit. This is the total time you have, and on that time you have to both capture and process the packet. So you now understand why packet capture is important, because if it exceeds this time available, even if your processing time is zero, you have no chance to handle wire rate traffic.

In general the situation is not that bad, as your application has usually more time available, because usually the bandwidth used is not more that 50-60%, and packet sizes are around 512 bytes. This means that at 1 Gbit you have 235 Kpps, and at 10 Gbit 2.35 Mpps, leaving your apps some more time to relax. Note that this does not mean that your top target is this, but that your average target is close to this. This because network traffic is not constant, but you have peaks and spikes, and thus your application should be prepared to handle them properly.

Buffering can help of course, but only for a few packets. Here you have to think in terms of time and not packets that you can store in memory, as many people do. In fact you should think a buffer as a way to temporary absorb peaks while your application handles traffic. Or in other words as a way to enlarge the time and avoid dropping packets. Unfortunately buffers are pretty small, and making them larger is not a good idea (yet as many people think) as their memory will not stay in cache, and thus the system will spend time handling this buffer. Thus a buffer is a good solution for handling a spike but it does not increase your packet processing speed.

At the 10Gbit (but on a smaller scale also at 1 Gbit) there's another important aspect to consider. As you can expect to process all packets, if and only if your application is faster than (or at least as fast as) the network you want to analyze. If this is the case, it means that from time to time the application has not enough packets to process and thus it needs to wait for them. In order to avoid wasting CPU cycles, your application can call poll(), select() or anything else that would allow your app to wait for packets. These functions are all system calls. If you measure the cost of s system call (you can compute this easily by creating a loop and calling a dummy sys call, then compute the number of sys calls you can issue per second), you will see that it costs around 1 usec. This is the cost of a dummy sys call, just to cross the boundary from user space to kernel; this cost will increase significantly if the sys call has a a buffer associated on which for instance you exchange information. If you go back to the top of this post, you see that 1 usec is enough for receiving 1.5 packets at 1 Gbit or 15 packets at 10 Gbit. This means that calling poll() for waiting for packets at 10 Gbit costs too much in terms of latency, as this sys call is not dummy at all, so on top of the cost of crossing the kernel boundary, it does something that further increases its duration. The consequence is that if your application is much faster than the network it might call poll() often, and when the sys call returns control, the buffer might be mostly full due to the poll() latency. So it's probably better to consider doing active wait and call usleep(1) so that you have the (partial) guarantee that every usec your app can see what happened.

Final remarks:

Think in terms of time and not packets
Be prepared to handle traffic spikes
Large buffers do not increase packet processing speed
Set an upper boundary to your application for processing packets
Packet capture is important, but it is not the only ingredient for handling wire-rate traffic.

Running a load-balanced Snort in a PF_RING cluster

lucaderi — Mon, 01 Aug 2011 17:39:22 +0000

Today I came across this article that I think all PF_RING/snort/suricata users should read.

Released ntop 4.1

lucaderi — Wed, 17 Aug 2011 20:20:12 +0000

Over the week end we released ntop 4.1. We decided to create a smaller version with respect to the previous 4.0.3 in order to remove some legacy code that caused trouble in the past. This release lacks some of the 4.0.3 features but it can benefit in terms of stability and efficiency. The next release will re-incorporate some of the features we cut on 4.1 as we're currently redesigning them. The idea is to make ntop faster and more modern than past versions. In 4.1 for instance we have removed legacy protocols in favor or new (let's call them protocols) ones such as Facebook and Twitter. Enjoy it!

Building a 10 Gbit Traffic Generator using PF_RING and Ostinato

lucaderi — Wed, 17 Aug 2011 23:15:16 +0000

Whoever has developed network applications, soon or later had to buy or rent a traffic generator. Years ago I have purchased my 1 Gbit IXIA 400T on ebay for 2500$, and I wanted to buy a 10 Gbit traffic generator when I started to develop DNA. Unfortunately I could not afford the price of those useful yet costly devices, and I have spent over 10K $ for a 10 Gbit FPGA-based NIC (manufactured by one of the leading companies, guess who, that on my PC can't now keep up with the 10g PF_RING DNA) able to generate synthetic packets after coding an application. I missed so much my IXIA traffic generator that after developing DNA I have said to myself: it's time to create one. PF_RING (and in particular DNA) users know that PF_RING comes with an utility named pfsend that is able to reproduce a pcap file at wire rate (e.g. 14.88 Mpps) using DNA. The tool is great but you need a pcap file to reproduce, so I needed a flexible traffic generator. I recently came across the Ostinato project aimed at creating an open-source traffic generator "inspired" to the IXIA user interface. The nice thing about ostinato is that is a client/server application, so you can run the server (named drone) onto your Linux box running PF_RING DNA, and the GUI on your Mac or Windows box. In order to make things simple, I have enhanced the PF_RING-libpcap and the DNA driver so that in order to use ostinato with DNA you just have to link the drone application with the PF_RING-libpcap. In order to do that you just have to download ostinato from source and edit the file ostinato/server/drone.pro changing the line "LIBS += -lpcap" into "LIBS += ~/PF_RING/userland/libpcap/libpcap.a -lpfring #-lpcap" and then compile ostinato from source. Below you can see how ostinato looks when running on top of PF_RING. [gallery link="file" order="DESC" columns="2" orderby="title"] Like I have said, I limited the changes to libpcap, thus no patches to ostinato are necessary beside pointing it to the PF_RING-libpcap library. The drawback is that on this configuration, ostinato can't generate more than (on my PC) 6.6 Mpps, that is less than half of what pfsend on the same host can do. In order to increase the performance we need to put ostinato directly on top of PF_RING creating a native PF_RING port. This is something left to future activities. For those who need wire speed, can use ostinato to generate a pcap file with the traffic pattern they like, and transmit it using pfcount. You now have a powerful yet inexpensive 10 Gbit traffic generator. No excuse now for not testing your apps at 10 Gbit! Credits: many thanks to Francesco Cocchia for having told me of Ostinato a couple of months ago.

--http-dump-dir
--http-exec-cmd	Command executed whenever a directory has been dumped
--dont-hash-cookies	Dump cookie string instead of cookie hash
--dont-nest-dump-dirs	Don't create subdirs on the dump directory
--max-http-log-lines	Max number of lines per log file (default 10000)