The Internet is pretty volatile. As new information become available, the old one disappears. Sometimes we have to look back and see what’s happened in the past years. Shall you be interested in seeing how ntop changed in the past twelve years, you can have a look at this URL, that has several snapshots of the ntop web site.

The ubuntu community has published a post that explains how to compile/use ntop on Ubuntu. This is the URL of the post.

Conference: OSCON 2010
Presentation Link: Ignite Track
Presented by: Brian Lavender

SNORT is popular Network Intrusion Detection System (NIDS) tool that currently uses a custom rule based system to identify attacks. This presentation emphasizes on writing the algorithm to write generate the rules through GA and the integration of them into nProbe, a similar network monitoring tool written by Luca Deri with a plug-in architecture.

Genetic Algorithms are dependent upon identifying attributes to describe a problem and evolving a desired population. In this case, the problem is an attack through the network and identifying the attack through connection property attributes. Genetic Algorithms depends upon training data. DARPA datasets provide training data, in categorized format (attack vs. normal) along with a corresponding raw network recorded format called tcpdump. nProbe has a plug-in architecture allowing for customization.

This presentation explains original code in C to evolve rules. It uses the same chromosome attributes used by Gong. The development verifies and contrasts against the research performed by Gong. It also presents the code for integration into nProbe.

After a few years of work, this is to announce the availability of ntop 4.0. Major changes include:

• Partially rewritten ntop processing engine to address reliability and performance
• Several bugs and stability issues fixed
• Added better support for IPFIX and NetFlow v9, as well as ntop PEN (Private Enterprise Number)
• Added support for Cisco ASA firewalls
• Added ntop engine scriptability via the python programming language
• Added RRDalarm plugin for generating alerts based on thresholds
• Improved google maps integration
• Enhanced sFlow support

ntop is available for both Unix and Windows platforms. The source code can be downloaded from here. Prebuilt Win32 binaries are available here. Many thanks to all code contributors, testers and all those who spread the word.

Since some time ntop support geolocation. Now courtesy of Ronald W. Henderson it can also display mercator maps and natively integrate with tools such  as Google Earth.

These ntop extensions are part of the  NST (Network Security Toolkit) toolkit. For more information please visit the NST Wiki page.

Sometimes people ask me a tutorial about PF_RING. Last year I have given a tutorial about it at the IM 2009 conference. I think that everyone interested in using PF_RING for going beyond packet capture acceleration should read this set of slides I used for the tutorial. Today the cost of packet capture is limited with respect to packet analysis. For this reason you should use PF_RING as a framework for creating simple yet powerful traffic monitoring applications.

In this video Luca presents the ntop project and gives an outlook of future activities. It was presented during the OSS conference that took place last May in Bolzano.

Finally this short interview gives an idea of how ntop can benefit when integrated with commercial applications and vendors such as Würth-Phoenix.

nProbe, acronym for NetFlow probe, is an open-source probe that supports both NetFlow and sFlow collection. It has been designed to keep up with Gigabit speeds on commodity hardware and it can be used for capturing packets and analyzing networks at full speed with no (or very moderate) packet loss using PF_RING.

Each captured packet is analyzed, associated to a flow record, and periodically, the expired flows are emitted and exported to the specified collectors. nProbe is fully inter-operable with commercial collectors and open source tools such as ntop.

The new version of nProbe (that will be released soon) has been extended and now contains a new storage system designed primarily to answer queries efficiently.

The new storage system

When nProbe is used as probe and collector, it supports flow collection and storage, both on raw files and relational databases such as MySQL and SQLite.

Support of relational databases has always been controversial as nProbe users appreciated the ability to query flow records using SQL, but at the same time flow dump to database could lead to flow records loss due to the database-processing overhead. On the contrary, the speed advantage of dumping flow records in raw format is paid at each search operation in terms of amount of data to read. Furthermore, the query language that can be used is limited when compared to SQL facilities.

In order to overcome the limitations of existing flow-management systems, an extension of nProbe has been developed. The new version of nProbe allows flow records to be stored on disk, using an innovative column-oriented database with an efficient compressed bitmap indexing technology named FastBit.

Conceptually FastBit is a database that stores its content by column, rather than by row (this structure is known as “vertical organization”). Data is represented as tables with rows and columns. A large table may be partitioned into many data partitions and each of them is stored on a distinct directory, with each column stored as a separated file in raw binary form. Users can configure partition duration (in minutes) at runtime and when a partition reaches its maximum duration, a new one is automatically created.

Furthermore, for tasks that demand the fastest possible query processing speed, bitmap indexes perform extremely well. These because the intersection between the search results on each variable is a simple AND operation over the resulting bitmaps. The consequence of this major speed improvement is that it is now possible to query data in real-time.

Additional details

The new extended nProbe creates FastBit partitions depending on the flow templates being configured (in probe mode) or read from incoming flows (in collector mode). Below there is a simple example where nProbe is configured to dump flow records using a temporary directory with a rotation period of 10 minutes:

nprobe -n none -i eth0 --fastbit /tmp/fastbit/ --fastbit-rotation 10 --fastbit-template "%IPV4_SRC_ADDR %IPV4_DST_ADDR %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL"


Flow records can be dumped at full speed with no index-build overhead. Thus, not considering flow receive/decoding overhead, it is possible to save on disk more than one million flow records/sec on a standard Serial ATA (SATA) disk.

Additional advantages of this technology are listed below:

• Ability to save flow records on disk with minimal overhead allowing no-loss on-the-fly flow-to-disk storage, as it happens with tools based on raw files.
• Compact data storage to limit disk usage as this enables users to store months of flow records on a cheap hard-disk with no need to use expensive storage systems.
• Simple data archive structure in order to move ancient data on off-line storage systems, without having to use complex data partitioning solutions.
• On tens of millions of records: sub-second search time when performing cardinality searches (e.g. count the number or records that satisfy a certain criteria) and sub-minute search time when extracting records matching a certain criteria (e.g. top X hosts and their total traffic on TCP port Y).

If you want to know more about this topic or view the results of comparative tests that were performed, you can read the research paper named “Collection and Exploration of Large Data Monitoring Sets Using Bitmap Databases” (Proceedings of TMA 2010, Zurich – April 2010).

To know the new parameters of next release of nProbe allowing to store flow records in the FastBit database and to see some examples of use, you can read this manual.

If you are interested in nProbe, follow this link to know how to get it!

May 17th 2010

Press Release

Plixer International, Inc., a leading global provider of network traffic monitoring and analysis tools, today announced that it has partnered with NTOP of Italy to launch Scrutinizer 7.7 with nProbe™ support for advanced flow-based monitoring to analyze client, server and application latency. If the flow involves HTTP, the URL information can also be exported.

With its unique software-based nProbe™ support, Scrutinizer 7.7 is the first-of-its-kind NetFlow analyzer to enable affordable remote probe deployment on individual PCs or servers to track and pinpoint traffic and application issues. While traditional NetFlow reporting involves displaying top IP addresses, ports and data volume, Scrutinizer v7.7 when receiving IPFIX from the nProbe, can deliver details generally only available via packet analysis. For example, if the flow exported by the nProbe involves HTTP, the exact URL and corresponding client, server and application latency experienced by the end systems is displayed. More detail means more awareness and shorter network troubleshoots.

“The ability to deploy this unique open source nProbe™ monitor makes this a powerful combination that is extensible and customizable for other applications, such as monitoring for VoIP and e-mail traffic,” said Michael Patterson, Scrutinizer product manager with Plixer. “This integration takes application performance analysis with NetFlow or IPFIX to a whole new level.”

By expanding flow monitoring deeper into the actual packets, Scrutinizer 7.7 with nProbe™ can provide greater detail on database performance, latency, e-mail and URL activities. The nProbe collects the data and transfers it to Scrutinizer via NetFlow v9 or IPFIX for reporting and archiving. The system also stores historical data for baseline trend analysis.

The combined solution allows customers to drill down on conversations to determine client round-trip time and server processing latency. If the communication involves HTTP, the complete URL is provided, as well as the ability to click and actually view the page accessed by the client. E-mail details include mail sender and recipient.

The result is much more detailed network traffic data with a primary focus on ensuring optimum application availability, troubleshooting and analysis far beyond traditional NetFlow analysis.

“Without Scrutinizer 7.7 with nProbe™ support, the only other way to get this kind of data is to use packet analyzer technology,” said Luca Deri, founder of NTOP and developer of the nProbe™ technology. “But if the problem is in an off-site location, this can be costly and time consuming. With remote deployment of our nProbe technology, you can analyze traffic at any location and have it report back to Scrutinizer for more efficient, network-wide troubleshooting and analysis.”

The nProbe™ integration for Scrutinizer 7.7 is remarkably affordable at just $195 for an unlimited server site license. Existing Scrutinizer customers under maintenance will receive a free upgrade to version 7.7, with just $195 required for the nProbe™ functionality.

Plixer and NTOP have partnered with Ravica for U.S. distribution of nBox, the nProbe™-powered hardware appliance.

For more information on Scrutinizer 7.7 with nProbe™ integration, visit www.plixer.com.

Jon Mills
Marketing & Public Relations Manager