Handling High Flow Rates: Cento and ntopng at Scale

Cento is a high-speed NetFlow probe designed to analyse traffic from high-speed links (100+ Gbit/s) and export flows toward ntopng, third party collectors or big-data systems.

When exporting data to ntopng, Cento uses ZeroMQ (ZMQ) as its primary mechanism for exporting flows in JSON or binary (TLV) format. In short, Cento acts as a ZMQ publisher, sending flow records over a TCP socket. ntopng subscribes to this socket as a ZMQ collector, receiving and processing the flows in real-time. This design allows flexible network deployment, with Cento running on the capture host and ntopng on the same or a remote machine.

At very high traffic rates, however, a single export channel and a single ntopng collector may not be able to keep up with the incoming flows. To handle this, it is possible to scale out using multiple ZMQ channels from Cento and then combine the outputs inside ntopng using an aggregated view interface. In fact, cento can exprt flows through a single ZMQ endpoint or to multiple endpoints for load-balancing or fan-out to multiple collectors. ntopng instances then subscribe to one or more of these endpoints. By combining the outputs in a view, operators can achieve a unified traffic overview while scaling horizontally.

Single ZMQ Channel (N:1)

By default, when a single –zmq option is configured, Cento exports flows via a single ZMQ publisher socket in N:1 mode. For example:

cento -i zc:eth0 --zmq tcp://127.0.0.1:5556

And ntopng subscribes to this endpoint:

ntopng -i tcp://127.0.0.1:5556c

This works well for moderate flow rates, but when hundreds of thousands of flows per second are exported, one pipeline may become saturated.

Scaling with Multiple ZMQ Channels

Cento allows you to create multiple ZMQ publisher sockets. Each publisher can be bound to a different TCP port, and Cento will distribute flows among them. Example:

cento -i zc:eth0 --zmq tcp://127.0.0.1:5556,tcp://127.0.0.1:5557

Now Cento will export flows across two independent ZMQ channels in round-robin.

Note: when a comma-separated list is provided, cento will load-balance flows to the ZMQ sockets in the list. Instead, when multiple –zmq options are provided, cento will copy the same flows (fan-out) to all configured ZMQ sockets.

Each ZMQ publisher can be consumed by a different ntopng interface. For example:

ntopng -i tcp://127.0.0.1:5556 -i tcp://127.0.0.1:5557

This way, the workload is spread across multiple ntopng threads. Each thread only processes the portion of flows delivered to its socket.

While multiple ntopng interfaces solve the scaling problem, operators usually want a single, unified dashboard.
This is achieved through views. A view in ntopng acts as an aggregation layer across multiple interfaces or collectors.
To create a view, all you need to do is to add one more special interface, which is aggregating traffic from all other interfaces:

ntopng -i tcp://127.0.0.1:5556,tcp://127.0.0.1:5557 -i view:all

The view will then expose consolidated traffic statistics, while still allowing you to drill down into each individual ntopng interface if needed.

Scaling with RSS and Multiple ZMQ Channels

In order to handle 100+ Gbit/s, Cento is usually used in combination with RSS, which is load-balancing network traffic coming from the adapter by means of multiple data streams to multiple packet processing threads. In this configuration, the adapter is logically divided into multiple sub-adapters (also known as queues) to which packets are delivered. Example with 4 RSS queues:

cento -i zc:eth0@0 -i zc:eth0@1 -i zc:eth0@2 -i zc:eth0@3 --zmq tcp://127.0.0.1:5556,tcp://127.0.0.1:5557

This is equivalent to:

cento -i zc:eth0@[0-3] --zmq tcp://127.0.0.1:5556,tcp://127.0.0.1:5557

In this configuration, Cento spawns multiple threads, each thread is processing traffic from a data stream. Flows extracted by all threads are still load-balanced to the configured ZMQ endpoints in a Round-Robin fashion.

As an alternative configuration, it is possible to configure Cento to send all flows extracted on a specific data stream to a specific ZMQ endpoint, with a direct 1:1 mapping. This configuration requires as many ZMQ endpoints as the number of configured RSS queues, and the –zmq-direct-mapping option. Example:

cento -i zc:eth0@[0-3] --zmq-direct-mapping --zmq tcp://127.0.0.1:5556,tcp://127.0.0.1:5557,tcp://127.0.0.1:5558,tcp://127.0.0.1:5559

ntopng is still configured in the same way, to collect flows from all endpoints and aggregate all traffic in a view:

ntopng -i tcp://127.0.0.1:5556 -i tcp://127.0.0.1:5557 -i tcp://127.0.0.1:5558 -i tcp://127.0.0.1:5559 -i view:all

Conclusions

On high-speed networks, Cento can easily generate flow rates that exceed what a single ntopng interface can handle. By configuring Cento with multiple ZMQ publishers, deploying multiple ntopng collector interfaces, and aggregating them with views, you can scale horizontally and maintain full visibility without losing data. This architecture ensures that Cento + ntopng can keep up with the flow volumes of modern backbone and datacenter environments.