Howto use Kafka (instead of ZMQ) For Reliable Flow Collection and IPC

Posted · Add Comment

Historically, we have used ZMQ for interconnecting nProbe to ntopng, as this is a fast and simple messaging system. However one of they key advantage of ZMQ of being broker-less is sometime a problem. In case of maintenance, traffic peaks, or unreliable communications, the ZMQ communication between nProbe and ntopng will drop flows with the result that some data will never reach ntopng.

As Apache Kafka is the de-facto standard for messaging communications, we have decided to extend its support from flow egress in nProbe / nProbe Cento, to communication system between them and ntopng. The new architecture will be the following:

In essence all the communications will pass through the Kafka broker to which subscribers, including ntopng, can connect. In case of communication issues, the Kafka broker will store messages until ntopng can reconnect and thus resume consuming messages without any loss, contrary to ZMQ where those messages not immediately handled will be lost forever.

Suppose to run the Kafka broker on host 192.168.2.225, and ntopng/nProbe run on the same host 192.168.2.130, all you need to do is

  • nprobe -i eno1 –ntopng  kafka://192.168.2.225
  • ntopng -i kafka://192.168.2.225

Instead if you want to use ZMQ you need to do

  • nprobe -i eno1 –ntopng  zmq://127.0.0.1:1234
  • ntopng -i zmq://127.0.0.1:1234

if you are using secure connections with you Kafka broker, use kafka-ssl://192.168.2.225 instead.

As you can see we have unified the command line options so that on nProbe side you need to use –ntopng for both Kafka and ZMQ. Please note that the old –zmq and tcp:// are still supported but deprecated. Namely

  • nprobe -i eno1 —zmq://127.0.0.1:1234
  • ntopng -i tcp://127.0.0.1:1234

These new features are supported in the dev (nightly build) branch as preview (we’re still optimising the code and adding new features), and will be included in the next stable versions. They require an enterprise license in order to be used.

Regardless of flow collection, in ntopng it is now possible to deliver data (e.g. export statistics) via Kafka using Lua as follows:

local broker_ip     = "192.168.2.225" --  CHANGEME                                                                                                                                                                                                     
local topic         = "mytopic"
local kafka_options = “”
local message = "Hello World"

ntop.sendKafkaMessage(broker_ip ..";".. topic  ..";".. kafka_options, message)

We now have the best of both worlds: Kafka and ZMQ.

Enjoy !