ntopng

ntop

Welcome to ntopng 5.2: Historical Data Analysis, Better Performance and Alerting
Initially designed as a maintenance release, 5.2 brings many improvements in its processing engine with over 3’000 code commits. The main goal is to enhance application scalability by optimising memory and CPU usage, while introducing a new persistency layer based on ClickHouse that has replaced nIndex a home-grown high-performance indexing system that we introduced years ago. This layer enables ntopng 5.2 to store billion of flow records and alerts with limited disk space and sub-second response time by providing full visibility in terms of packets, flows and alerts. In essence …
ntopng

ntopng and ClickHouse: Lessons Learnt at California Institute of Technology
Caltech has been experimenting with ntopng on our network for slightly over a year now.  We send a decent amount of traffic to ntopng, bursting up to 20Gbps, utilising Cento to read the wire and forward the data to ntopng via PF_RING ZC.  This configuration has been working pretty well, though we were encountering issues once we reached about 16 – 20 days of data retention, where ntopng would begin to drop data points from that point forward, and I noticed InfluxDB would utilize 60% or more of available memory, …
ntop

Historical Traffic Analysis at Scale: Using ClickHouse with ntopng
Last year we have announced the integration of ClickHouse, an open source high-speed database, with nProbe for high-speed flow collection and storage. Years before we have created nIndex, a columnar data indexing system that we have integrated in ntopng, but that was just an index and not a “real” database. We have selected ClickHouse for a few reasons: It is open source and developed by a vibrant community. It is very efficient in both speed and size, that were the main features for which we created nIndex. This is very …
nDPI

A Gentle Introduction To Timeseries Similarity in nDPI (and ntopng)
Introduction Let’s start from the end. In your organisation you probably have thousand of timeseries of various nature: SNMP interfaces, hosts traffic, protocols etc. You would like to know what timeseries are similar as this is necessary for addressing many different questions: Host A and host B are two different hosts that have nothing in common but have the same traffic behaviour. Host C is under attack: who else is also under attack? SNMP interface X and interface Y are load balancing/sharing the same traffic: is their timeseries alike or …
ntopng

Data Aggregation in ntopng: Host Pools vs Observation Points
ntopng allows users to aggregate data according to various criteria. In networking, IP addressing (network and mask/CIDR) and VLANs are typical solutions to the problem of aggregating homogeneous hosts (e.g. when hosts carry on similar tasks). Sometimes these aggregation facilities are not flexible enough to cluster hosts that have the same operating system, or flows originated by the same router/switch. In addition to typical network-based criteria such as IP, VLAN, ntopng implements two more data aggregation facilities. Hosts Aggregation: Host Pools A host pool is a logical aggregation of hosts, …
ntopng

HowTo Monitor Traffic in SMEs and Home Networks: A Primer
In the first part of this series of articles, we focused on monitoring ISPs and MSP traffic. Today we analyse network traffic in SMEs and home networks. The typical network layout of a home or a small business is depicted below.   The ISP provides a router for connecting to the Internet (e.g. xDSL or fibre) that usually also features an embedded access point used by phones, tablets or laptops to connect to the Internet. In order to monitor LAN traffic, the best solution is to replace the current switch …
nProbe

HowTo Monitor Customer Traffic in Managed Service Providers and ISPs
ISPs have provided Internet access to customers for years and the only goal was to connect their users to the Internet. Managed Service Providers (MSP) and Managed Security Service Providers (MSSP) deliver network, services and infrastructure on customer premises and have become relatively popular in the past few years. Over time customers started to ask new services, including traffic monitoring, security (here MSSP come into the scene) and visibility. So if you as a MSP, MSSP or ISP and you are wondering how to monitor customer traffic using ntop tools, …
Announce

ntopng 5.0 Is Out: Modern Traffic Monitoring for AIOps and Cybersecurity
ntopng was initially designed as a tool for realtime network traffic monitoring. The idea was to create a DPI-based tool able to report traffic statistics. Overtime we have added the ability to implement active monitoring checks, SNMP, and various other features. However there was a fundamental point that was missing: go beyond traffic reporting, moving towards traffic analysis. The current Grafana-like trend of having several large screens full of dashboards is the opposite of what we believe we should do. This approach requires network and security administrators to be trained …
ntopng

Infrastructure Monitoring: Observing The Health and Status of Multiple ntopng Instances
Introduction Quis custodiet ipsos custodes? (Juvenal). In other words: who will guard the guards themselves? If you use ntopng to monitor your network, you also need to make sure ntopng is monitored as in case of failure, ntopng will not report any alert, and the network administrator can interpret that as a sign of good health, instead of interpreting it as lack of monitoring.Recent 4.3+ versions of ntopng have the capability to monitor other ntopng instances, being them in the same local LAN or physically/geographically distributed. This capability, also referred …
nProbe

Collecting Flows from Hundred of Routers Using Observation Points
Collecting flows on large networks with hundred of routers can be challenging. Beside the number of flows to be collected, another key point is to be able to visualize the informations in a simple yet effective way. ntopng allows you to create up to 32 virtual flow collection interfaces that can be used to avoid merging collected flows: unfortunately they are not enough when collecting flows from 100+ routers. In the latest ntopng and nProbe dev versions (soon to become stable), we have implemented the concept of observation point, that …
nProbe

NetFlow/IPFIX At Scale: Comparing nProbe/ClickHouse vs nProbe/ntopng
In our previous post we have analysed the performance of the pipeline nProbe+ntopng for those who need to collect flows and analyse them, trigger alerts, create timeseries, provide a realtime monitoring console, dump them to nIndex and inform remote recipients in case of some problem is detected. This is the main difference between the ntop solution and a NetFlow collector whose main goal is to dump flows on a database with any or little flow analysis. In essence the current state of the art with 4 nProbe instances sending data …
nProbe

NetFlow Collection Performance Using ntopng and nProbe
Introduction ntopng, in combination with nProbe, can be used to collect NetFlow. Their use for NetFlow collection is described in detail here. In this post we measure the performance of nProbe and ntopng when used together to collect, analyze, and dump NetFlow data. The idea is to provide performance figures useful to understand the maximum rate at which NetFlow can be processed without loss of data. Before giving the actual figures, it is worth discussing briefly the most relevant unit of measure that will be used, i.e., the number of …

High-performance network traffic monitoring and analysis tools.

Explore
Newsletter
Subscribe