It’s time for a completely new ntop. Say hello to ntopng.

Posted · Add Comment

15 years are past since the first version of ntop. In 1998 network monitoring requirements were very different from today: few protocols (mostly in plain text) to monitor, IP was not yet “the only protocol”, low network speed, very few connected hosts, no iPhones yet, raspberry was still a fruit, Linux was still for geeks. In 2013 the whole picture is very different. One gigabit links are now commodity (10 Gbit is around the corner), (too?) many hosts interconnected and mobile, application protocols (e.g. Spotify or Skype) are “the” protocols (TCP is a generic protocol) so we need nDPI to figure out what is happening on the network.

The way the original ntop was designed was IMHO very advanced for that time, but today is no longer so for many reasons. Today people want to have a flexible network monitoring engine able to scale at multi-Gbit, using limited memory, immune to crashes “no matters what”, scriptable and extensible, able to see what’s happening in realtime with 1-second accuracy, capable of characterising hosts (call it host reputation if you wish) and storing monitoring data on the cloud for (de-)centralised monitoring even of those devices that have no disk space. Over the past years we have tried to address ntop open issues, but the code base was too old, complicated, bug-prone. In essence it was time to start over, preserve the good things of ntop, and learn from mistakes. So basically looking forward by creating a new ntop, able to survive (hopefully) 15 more years and set new monitoring standards.

This has the motivation behind what I temporarily call ntopng (ntop next generation). The work to do is huge but as you can see many things are already working.

ntopng_work_in_progress

The main design principles are:

  • Open source, self-contained with zero configuration, just like the original ntop.
  • ntopng is a cache, just like the original ntop, but contrary to its predecessor we leverage on Redis for implementing multi-level caching:
    • ntopng keeps in memory the current network traffic
    • Redis keep the “recent network history”
    • (Optionally) Persistently dump traffic history on disk for long term traffic analysis.
  • nDPI centric: ports are no longer enough, as we want to identify application protocols even on non standard ports.
  • Ability to leverage on PF_RING for monitoring million packets/second with no drops.
  • Written in C++, with clean code layout. Occasionally some routines from the original ntop will be ported to ntopng, but the idea is to write everything from scratch on a clean room. The ntop code didn’t have a real API and it was so complicated after years of patches, that people were scared of touching me.
  • The web GUI is based on Twitter Bootstrap for modern, consistent, and mobile-friendly GUI.
  • The ntopng engine is scriptable in LuaJIT.
  • Web pages are written in Lua: everyone can write its on pages without having to code in C.
  • ntopng, as well nProbe, leverage on the MicroCloud for creating a comprehensive network view.

This said, the work to do is huge and it will take some time before ntopng will be completed. This means that if you want, we need your help to expedite its development. You can access the ntopng code here:

svn co https://svn.ntop.org/svn/ntop/trunk/ntopng/

The core is stable (we have tested on Linux and OSX, but it will soon be tested/compiled on Windows) although it is still missing some pieces such as IPv6 support, historical charts, NetFlow/sFlow support and more reports. We encourage you to download and test it. Likely you can help us developing it, or at least testing it. As you can imagine, we have no time to support the original ntop, as we are focusing on this new release. We plan to have an initial release by late May June (Note: the porting on Windows is taking more time than expected and we need to fix a few open issues still): time is limited, but we’re confident to include all the core features on this release then refine it through the rest of the year.