ntop

Say hello to Libzero

admin — Mon, 14 May 2012 14:01:59 +0000

Last year we have introduced PF_RING DNA for implementing 0% CPU receive/transmission on commodity 1/10 Gbit network adapters. We considered DNA as a starting point, as it implemented high-speed RX/TX that was enough for most, but not all of you. This is because commodity adapters do not feature advanced packet balancing techniques as they rely on RSS, that has several limitations such as asymmetric flow balancing (i.e. the two direction of the same flow are spread onto two different cores) and inability to provide users a way to use their balancing function. Another limitation of DNA, again due to its nature that is close to the hardware, is that packets should be processed in sequence (i.e. in FIFO) whereas applications sometimes need to store packets and process them out of sequence (e.g. in case of fragmented packets, a given packet must be rebuilt with all fragments prior to process it).

Although zero-copy is often a buzzword as only a subset of packet management is really performed without copying any packet, at ntop we decided to see whether it was really possible implement zero-copy for all operations, including packet dispatching to threads and applications (including packet fan-out support), packet queuing, and forwarding across interfaces. This is what libzero is for: as its name says, we can do all these operations in zero-copy, with no performance penalty as you will still be able to reach line rate with minimal packet size (14.88 Mpps with 60+4 bytes packets).

Libzero opens up a new world of opportunities, as it enables developers to focus on their application leaving to the library the task of handling packet memory, prefetching memory to let your applications access packet payload at the same speed as counting packets. Now you can finally scale up applications, as you can for instance spawn several snort applications and, without changing a single line of its code, let each instance handle a coherent (across directions) set of packets, all at line rate. In a nutshell, the network is no longer the bottleneck nor the source of complexity.

The ball is on the software side again. You can find all details at the libzero home page.

Getting More Information On Your Network Performance

admin — Tue, 08 May 2012 09:20:01 +0000

This week ntop will be present at the Open Source System Management Conference 2012, that will take place this Thursday in Bolzano, Italy, organized by our partner and sponsor Würth-Phoenix. We’ll give a speech about how to analyze network performance with our nProbe/ntop applications, as well how to characterize the applications generating traffic. In fact it is important not to do generic and aggregate metric monitoring, but to characterize flow-by-flow so that we can generate alerts per-application. During the event we’ll speak about future nProbe extensions that we’ll introduce later this month such as the new Oracle plugin for nProbe, and we’ll preview a new layer-7 plugin that will allow you to combine traffic monitoring with layer-7 bridging/shaping leveraging on nDPI. In fact, many people do not understand the need to deploy network monitoring tools until they have issues, whereas in many cases we believe it’s easier to piggy-back network monitoring by means of tools that for instance implement traffic policies, similar to what costly application firewall do today.

For those who cannot attend the conference, you can

View the slides we’ll use for presentation.
View the live conference streaming

We hope to see you soon!

PF_RING DNA RFC 2544 Benchmark

admin — Wed, 11 Apr 2012 16:00:47 +0000

Over the past couple of weeks we have further improved the DNA performance, and thus we have decided to test its performance. In order to do reproducible measurements we decided to adopt the benchmark specified in RFC 2544. You can find the complete test details and results on this document: DNA_ip_forward_RFC2544.

As you can read we used a low-end single-CPU Supermicro server X9SCM, Linux Fedora Core 15, and a Spirent SRC-2002HS 10 Gbit traffic generator. On a nutshell DNA has not lost a single packet, even with 64 bytes (60 bytes packet + 4 bytes CRC) packets. Thanks to libzero the forward latency across ports is as low as 3.45 usec with minimal size packets. All this is amazing if you consider that these results have been achieved on commodity hardware, matching the performance of costly FPGA-based NICs.

Many thanks to Silicom for supporting the DNA development and benchmarking.

Benchmarking PF_RING DNA

admin — Tue, 20 Mar 2012 18:28:21 +0000

For years networking companies have used the buzzword zero-copy to qualify those hardware/software solutions that allow applications to play with packets without the need to copy them at all. Zero-copy needs DMA (Direct Memory Access) for operating so that applications do not get a (shallow) copy of packets but they actually get the pointer to the packet. As you probably know, PF_RING DNA allows applications to access packets in zero-copy so that in the pfring_recv() call you get a pointer to the packet just receive. Whereas in traditional PF_RING you always get a copy of a packet portion (limited to the snaplen) sitting on a memory ring living on the kernel that is accessed in DMA.

In DNA, zero-copy applies not just to packet capture but also to packet transmission, and bouncing (i.e. you receive a packet on one interface and forward it to another interface). All those who have tested DNA have realized that an old Core2Duo 1.86 GHz is enough for handling TX line rate at 10G, so in general if you own an adequate server, DNA can solve all your RX and TX needs.

Over the past 6 months we have made many changes to DNA, in particular for making it more flexible not just for packet capture but also for packet processing. We have then decided to run some new performance tests in order to position DNA with similar solution such as netmap, and Intel DPDK that is probably the reference software in terms of packet processing on commodity hardware.

For our tests we used a entry-level server Supermicro X9SCL powered by a Xeon E31230 (4 cores + HyperThreading) fitted with two dual 10 Gbit NICs. For traffic generator we used a Spirent (4 x 10 Gbit ports) kindly provided by Silicom. For packet capture we used pfcount, and for packet receive+forwarding we have used pfdnabounce with a pre-release version of the libzero that will be releasing soon and that allows to operate in zero-copy across network interfaces.

Test 1: Packet Capture

We have connected all 4 ports of the Spirent to the four 10 Gbit ports. We injected 64 bytes packets on all ports. pfcount’s were started as follows

# ~/PF_RING/userland/examples/pfcount -i dna0 -a -g 0
# ~/PF_RING/userland/examples/pfcount -i dna1 -a -g 1
# ~/PF_RING/userland/examples/pfcount -i dna2 -a -g 2
# ~/PF_RING/userland/examples/pfcount -i dna3 -a -g 3

where each pfcount is bound to a different core. The ixgbe DNA driver has been loaded with a single RX queue (as you will read later on this article, one queue is enough).

rmmod ixgbe
insmod ./PF_RING/drivers/DNA/ixgbe-3.7.17-DNA/src/ixgbe.ko RSS=0,0,0,0
ifconfig dna0 up
ifconfig dna1 up
ifconfig dna2 up
ifconfig dna3 up
echo "1" > /proc/irq/52/smp_affinity
echo "2" > /proc/irq/52/smp_affinity
echo "4" > /proc/irq/54/smp_affinity
echo "8" > /proc/irq/56/smp_affinity

Each pfcount has reported minimal packet drop, happened when the Spirent started to inject traffic.A generic pfcount stat is the following:

=========================
Absolute Stats: [1041759799 pkts rcvd][14323 pkts dropped]  Total Pkts=1041774122/Dropped=0.0 %
1'041'759'799 pkts - 87'507'823'116 bytes [15'096'354.76 pkt/sec - 10'144.75 Mbit/sec]
=========================
Actual Stats: 14882462 pkts [1'000.10 ms][14'880'973.90 pps/10.00 Gbps]

So we basically have lost (at startup) about 14k packets over a billion packets received. This while 4 pfcount instances were running simultaneously for a total packet capture rate of 4 x 14.880 Mpps, with each pfcount instance loading the CPU at about 90%.

Test 2: Packet Forwarding

We have connected the Spirent as traffic generator, received and then forwarded packets back to the Spirent using pfdnabounce, and then compared the number of received packets with the original number of packets sent. This is a setup similar to RFC 2544.

As you can read we have lost 139 packets out of 1 billion, while forwarding 64 bytes packets at line rate. Also on this case the loss happened when we started the test.

Final Remarks

So while we’re still trying to improve DNA so that we can see zero loss also on these extreme conditions, we are pretty happy of the tests outcome. First of all because when in 2005 we started the PF_RING project, we would have never imagined that it would have been feasible to capture almost 60 million/pps on a machine that costs about 500 Euro (plus NICs). Second because DNA isn’t second to DPDK according to some benchmarks you can find googling a bit. We believe that this is a great result for our open source project, although we are working to continuously improve DNA.

Meet ntop @ Cebit 2012

admin — Thu, 01 Mar 2012 21:18:57 +0000

All those visiting Cebit next week, will have the chance to see what we’re doing at ntop for providing better network monitoring services. We will give a presentation at the Open Source Forum next Wedn at 1.45 PM that is organized by the Linux Magazine.

This would be a good time to speak and meet the ntop community. We hope to see you there.

SFProbe: Embedding nProbe on an SFP

admin — Sun, 19 Feb 2012 18:31:52 +0000

In 2004 my friend Alex Tudor of Agilent involved ntop on a very challenging project. The idea was to monitor the network from the exact place where packets were originated. In fact popular network taps and span ports are not the right tools as they are added to an existing network (i.e. the network does not need them, but probes do need them). The same applies to active monitoring: traffic should be generated from the right place. So if you want to see the router-to-router latency you should let the router ping the other router, and not an external PC connected to such router. To make this long story short, Agilent decided to put the probe on the best place: namely on the SFP. This is because network equipment need SPFs and putting the intelligence there means that the probe is on the equipment side, on a vendor-independent place. You can imagine that we were involved on the software side (unfortunately we can’t build hardware here) and Agilent on the hardware side. You can see how hardware moved from testing boards

to a complete product. After 6 years since the beginning of the project, JDSU (the former Agilent group running the project was incorporated on that company) has finally made the magic.

They managed to squeeze everything into an ASIC chip and what we prototyped so many years ago is finally a product. Obviously this product is ntop-friendly as nProbe is a JDSU PacketPortal-Validated application. This means that using nProbe with PacketPortal you can analyze not just Internet traffic, but also Triple Play and mobile network traffic including LTE (didn’t you know nProbe was able to do that?).

JDSU has funded many nProbe developments including the VNIC integration features (VLAN tags used as netflow interface id), and if nProbe moved off a research project into a more industry-oriented (yet open source) solution it’s also thanks for all the comments we received during this project. If interested, you can visit the PacketPortal page (click on the “Related Products” for jumping to nProbe). Many thanks to Alistair Scott of JSDU for making all this happen.

Packet Monitoring using ntop and Cisco ON100

admin — Wed, 15 Feb 2012 19:52:39 +0000

From time to time, Cisco builds ntop-friendly products. This is the time of the Cisco ON100 network agent.

This tiny device that can fit on your hand, has been integrated with ntop for the purpose of traffic monitoring as you can read on this technical note Enabling ntop Packet Monitoring with Cisco OnPlus Service.

ntop is an optional application watching the second LAN port (Monitor port). The Cisco cloud service provides a web tunnel back to the ON100 to ntop’s web service. No data is interpreted, as ntop does that. This way end users can use ntop application without needing to invest in a PC. The ON100 unit has lots of CPU/RAM and ntop seems to behave very well. Since customers can use either (or both) port mirroring or NetFlow, everyone is happy.

You can find more about ntop and ON100 at this discussion page.

Precise Interface Merging Without Hardware Timestamps

admin — Wed, 08 Feb 2012 23:00:27 +0000

In network monitoring it is very common to use taps for duplicating network traffic (RX and TX directions).

Taps are important as they allow network probes to operate passively without interfering with network operations. The two traffic directions (A to B and B to A) are plugged into two network ports of the probe. Having the two directions separated has advantages (e.g. packets are not mixed across directions) and disadvantages. The main disadvantage is that when reading packets from the two interfaces, it is not possible to know which packet out of the two directions arrived first. For many applications (e.g. NetFlow traffic monitoring) this is not a problem, but for some others it is necessary to know the exact order of the packets. This is because if two hosts, one on the A side and the other on the B side, communicate over a protocol that has some kind of state, the probe must see messages in the exact order they hit the wire. If this does not happen, it might be confused and not operate as expected. We call this precise interface merging (PIM).

One of the myths about PIM is that it is necessary to use costly hardware timestamp-based NICs that are able to know the exact time (usually with a precision of 50 or 100 nsec) when incoming packets hit the probe interface. While hardware timestamps can definitively be used for PIM, we claim that they in practice are not necessary as commodity adapters with DNA and without hardware timestamps are enough for this task. We’re now going to explain why this statement holds.

With the advent of PF_RING DNA, it is possible to capture packets on multiple interfaces at line rate 1/10 Gbit with no loss. At 10 Gbit line rate, the adapter receives a packet every 67.2 nsec (1 sec /14.880.000 pps). Modern adapters as those based on the Intel 82599 chip can buffer up to 4096 packets in memory that is 275.2 usec (4096 * 67.2). As DNA operates at line rate, it means that it is able to consume packets as fast as the network (otherwise we wouldn’t be able to operate at line rate). With DNA we have written an example application named pfcount_aggregator that is able to read packets from two interfaces simulatenously. Using this application it is possible to read packets in order as two peers (one on the A and the other on the B side) that exchange data over a state-machine based protocol (i.e. A sends a request, and B replies with a response to the previous request) such as GTP can safely assume that packets are read in order by the aggregator. This is because the time it takes for a request (as seen on the first tap-ed interface) to go from A to B, and the response from B to A (as seen on the second tap-ed interface) is definitively more (in particular for a WAN) than 275.2 usec. This means that by the time the aggregator has read the request, it has also read all the following packets (up to the buffer length) and thus it’s impossible to read the response packet before the request. Basically it’s impossible for the probe to see the response before the request when an application such as pfcount_aggregator is used. Note that this is as such, because of DNA as with the NAPI-based polling model where interfaces can be polled sequentially, this property might not be true.

On a nutshell hardware timestamps and costly NICs are not necessary for PIM. This contrary to the common belief that without them PIM wouldn’t happen. Juvenal said 2000 years ago: Quis custodiet ipsos custodes? Have fun with DNA and commodity adapters.

Say hello to nDPI (Network DPI)

admin — Fri, 03 Feb 2012 11:57:58 +0000

The equation “port = (application) protocol” no longer holds. DPI (Deep Packet Inspection) is the way to detect known protocols on non-known ports (e.g. http on ports other than 80) and traffic on know port that is not the one we expect (e.g. skype on port 80). On a nutshell, we need to look at packet content and see what’s inside. P2P protocols have been designed from day one with the ability to circumvent network policies in order to reach their peers, and they are good example of places where DPI can help.

Unfortunately there are very few DPI libraries freely available on the Internet, and most of the time they support “common protocols” (e.g. SMTP, DNS) that are not so challenging. On the other hand popular protocols such as HTTP cannot longer be defined a protocol. We believe that Facebook, Twitter, Netflix and many others are not just sub-HTTP protocols (technically they are of course) but first-class protocols. As we have not found any reasonable DPI library freely available, we decided to create our own starting from OpenDPI that is a good starting point but that lacks many interesting protocols (e.g. Skype) as they are available on the commercial library version. This has been the motivation behind nDPI.

nDPI is a ntop-maintained superset of the popular OpenDPI library. Released under the GPL license, its goal is to extend the original library by adding new protocols that are otherwise available only on the paid version of OpenDPI. In addition to Unix platforms, we also support Windows, in order to provide you a cross-platform DPI experience. We have added support for many popular protocols such as Twitter, Skype BitTorrent (major enhancements) and also business protocols such as Citrix.

We plan to maintain this library free of charge and updated as new protocols (versions) come out. But on the other hand we need support from the community for tracking bugs and adding extensions. More information can be found at the nDPI home page.

DNA vs netmap

admin — Fri, 27 Jan 2012 21:10:03 +0000

In the past months I have received a few emails about how to position DNA with respect to netmap. To many people they look like two competing solutions, but in reality they are just two solutions to the same problem. Yesterday I had a nice meeting with Luigi Rizzo, the author of netmap.

I personally know Luigi since almost 15 years as we both live pretty close. The first time I saw him (1999 or so) he was hacking a driver for a CD-ROM drive on FreeBSD while speaking with me. Impressive. He was telling me about projects he previously did, including a PC-based bridge able to run a few university departments. I have to admit that if I have started to code in the kernel, it’s also because he has inspired me showing that the kernel was a radical new (for me) way of looking at things. I decided to go for Linux, he’s a recognized FreeBSD contributor, but our view are not that far.

Even if DNA and netmap have been developed in totally independent ways, they solve the same problem: how to move packets back/forth a network adapter without using too many CPU cycles. And I believe it’s no surprise that the performance of DNA and netmap is basically the same. So yesterday we’ve not discussed about who’s faster, but we talked about where we wanted to drive our technologies.

As you can see from the backboard (pictures are courtesy of our common friend Gianmarco), we have some ideas on the next thing to do. First of all, both DNA and netmap are not the ultimate goal, but instead some enabling technologies for improving networking on operating systems. They are not revolutionary in the sense that the idea of memory mapping is out for years, but companies like Endace and Napatech (that used it for so long time) have been unable to use it properly and thus drive innovation anywhere. If you look at at their technology 15 years ago and today, you will see that nothing changed: just pass me the packet and I will do the rest. In my view (and I think I can also mostly speak for Luigi), we need to go beyond simple packet juggling. Modern network and applications are significantly different from similar applications developed in the last decade. Operating systems changed too, and thus it’s time to rethink a few things, and not just to wait until somebody else would do that. Stay tuned!