ntopng

ntopng

HowTo Visualise ntopng Alerts in Kibana

ntopng can export both flows and alerts in Elastic according to the Elastic Common Schema (ECS) format. You can dump flows (not alerts) in Elastic starting ntopng with -F “es;<mapping type>;<idx name>;<es URL>;<http auth>”. For instance you can do ntopng -F "es;ntopng;ntopng-%%Y.%%m.%%d;http://localhost:9200/_bulk;" We do not advise to use Elastic as flow collector, as when the record cardinality increases the database slows down and you are forced to use an Elastic cluster even on mid-size networks. We definitively advise you to enable -F clickhouse instead that is able to handle billion …
nDPI

How to Configure Flow Risk Exclusions in nDPI and ntopng

Flow risks are the mechanism nDPI implements for detecting issues in network traffic whose theoretical design is documented in this paper Using Deep Packet Inspection in CyberTraffic Analysis we have written last year. While we are reworking the definition of risk exceptions in ntopng to make them fully configurable with a matter of clicks, you can easily configure risk exceptions by adding them to a protos.txt file. Such file can be passed to ntopng on the configuration file by adding a line such as --ndpi-protocols=/etc/ntopng/protos.txt and creating the /etc/ntopng/protos.txt file. …
ntopng

How We Simplified Data Search in ntopng

ntopng users are familiar with the search box present at the top of each page. It was originally designed to find hosts and jump to their details page. Over the years we have added a lot of new information in ntopng, and limiting its scope only to hosts was not a good idea. The image below is how we have improved it. In the new search we do not limit our scope to hosts but to everything inside ntopng, as a a mini embedded search engine. The first column shows …
ntopng

Dispatching Alerts: How to Master Notifications in ntopng

Alerts in ntopng are the result of traffic analysis based on checks. Checks detect that specific indicators on traffic require attention: for instance a host whose behavioural score has exceeded a given threshold or a flow that is exfiltrating data. Checks process traffic information with respect to a specific Network element, and for this reason they are divided into families (e.g. host, interface, flow, …). Regardless of the family, they can cover a security aspect, or they can monitor the network performance, for this reason they belong to different categories …
Cybersecurity

Incident Analysis: How to Correlate Alerts with Flows and Packets

In incident analysis it is important to provide evidence of the problem  at various level of details: Alerts Alerts are the result of traffic analysis (in ntopng based on checks) that have detected specific indicators in traffic that triggered the alert. For instance a host whose behavioural score has exceeded a given threshold or a flow that has is exfiltrating data. Flows Are the result of aggregation of packets belonging to the same connection and are used to compute alerts. Packets This is the most granular data that contains evidence …
ntopng

Using ntopng with Checkmk: A Tutorial

Today we’ll discuss the ntopng integration with Checkmk, a popular open source infrastructure monitoring tool to which ntopng adds traffic visibility. If IT infrastructure monitoring and network usage monitoring would see each other on Tinder, they would both for sure swipe right and match. Bringing the big picture perspective of IT infrastructure monitoring together with the in-depth information from network usage monitoring is thus a logical step. That’s why ntop and tribe29, the developers of the IT monitoring solution Checkmk partnered and jointly built a seamless integration of both tools. …
nDPI

You’re invited at FOSDEM 2022 (5 and 6 February) in the ntop stand

As most of our users know, every year we were used to meet the world of open source at FOSDEM in Brussels. Due to pandemic, this yearly event has been moved online so we invite you to attend it wherever you are. You can find more info at this page, but in summary we have two main events On Saturday we plan to show the latest tools we have developed, including ntopng 5.2 that we have just released. The idea is to highlight the main tool features, and discuss about …
ntop

Welcome to ntopng 5.2: Historical Data Analysis, Better Performance and Alerting

Initially designed as a maintenance release, 5.2 brings many improvements in its processing engine with over 3’000 code commits. The main goal is to enhance application scalability by optimising memory and CPU usage, while introducing a new persistency layer based on ClickHouse that has replaced nIndex a home-grown high-performance indexing system that we introduced years ago. This layer enables ntopng 5.2 to store billion of flow records and alerts with limited disk space and sub-second response time by providing full visibility in terms of packets, flows and alerts. In essence …
ntopng

ntopng and ClickHouse: Lessons Learnt at California Institute of Technology

Caltech has been experimenting with ntopng on our network for slightly over a year now.  We send a decent amount of traffic to ntopng, bursting up to 20Gbps, utilising Cento to read the wire and forward the data to ntopng via PF_RING ZC.  This configuration has been working pretty well, though we were encountering issues once we reached about 16 – 20 days of data retention, where ntopng would begin to drop data points from that point forward, and I noticed InfluxDB would utilize 60% or more of available memory, …
ntop

Historical Traffic Analysis at Scale: Using ClickHouse with ntopng

Last year we have announced the integration of ClickHouse, an open source high-speed database, with nProbe for high-speed flow collection and storage. Years before we have created nIndex, a columnar data indexing system that we have integrated in ntopng, but that was just an index and not a “real” database. We have selected ClickHouse for a few reasons: It is open source and developed by a vibrant community. It is very efficient in both speed and size, that were the main features for which we created nIndex. This is very …
nDPI

A Gentle Introduction To Timeseries Similarity in nDPI (and ntopng)

Introduction Let’s start from the end. In your organisation you probably have thousand of timeseries of various nature: SNMP interfaces, hosts traffic, protocols etc. You would like to know what timeseries are similar as this is necessary for addressing many different questions: Host A and host B are two different hosts that have nothing in common but have the same traffic behaviour. Host C is under attack: who else is also under attack? SNMP interface X and interface Y are load balancing/sharing the same traffic: is their timeseries alike or …
ntopng

Data Aggregation in ntopng: Host Pools vs Observation Points

ntopng allows users to aggregate data according to various criteria. In networking, IP addressing (network and mask/CIDR) and VLANs are typical solutions to the problem of aggregating homogeneous hosts (e.g. when hosts carry on similar tasks). Sometimes these aggregation facilities are not flexible enough to cluster hosts that have the same operating system, or flows originated by the same router/switch. In addition to typical network-based criteria such as IP, VLAN, ntopng implements two more data aggregation facilities. Hosts Aggregation: Host Pools A host pool is a logical aggregation of hosts, …