Introducing Lua-based Host and Flow Behavioural Checks

Posted · Add Comment

With ntopng version 5 we have migrated performance sensitive sections of the ntopng engine from Lua to C++. This has enabled ntopng to scale up nicely while reducing resource needs such as CPU and memory. The drawback is that writing behavioural checks in C++ is not something that everyone can do. For this reason we are introducing two (one for Flows and the other for Hosts) behavioural checks that enable the check logic to be written in Lua. In order not to jeopardise the ntopng v5 performance, these checks are very lightweight and are designed to code checks in a few lines of code.

The idea of this work is to enable end-users with little Lua coding experience, to be able to create custom checks for their own needs. Examples include trigger an alert whenever a:

  • TLS flow uses specific certificate/ciphers.
  • Host is contacting unexpected peers.
  • Flow uses a forbidden protocol (e.g. SMBv1).
  • Host exceeds a specific traffic threshold for a specific protocol (e.g. trigger an alert when host X mae more than X MBytes of DNS traffic)

Below  you can find a couple of examples that should give you an idea of how simple is the API.

Lua Host Check

In order to enable the Lua Host Check you need to enable the “Host User Check Script” in the behavioural checks list and write a the Lua script you can place at /usr/share/ntopng/scripts/callbacks/checks/hosts/custom_host_lua_script.lua

The script is executed periodically on all hosts (typically every minute). In the script ntopng allows you to access a new object named host that points to the current host being checked.

Typically, users check the host for specific conditions (e.g. trigger an alert for all multicast hosts) and trigger an alert that will then appear in the alert page as shown below

For instance the script below triggers an alert for all blacklisted hosts:

if(host.is_blacklisted()) then
   local score   = 100
   local message = "blacklisted host detected"

   host.triggerAlert(score, message)

   -- Tell the ntopng engine to skip this host for future checks as we have already evaluated it
   host.skipVisitedHost()
end

You can find a comprehensive example at this page.

 

Lua Flow Check

In order to enable the Lua Flow Check you need to enable the “Flow User Check Script” in the behavioural checks list and write a the Lua script you can place at /usr/share/ntopng/scripts/callbacks/checks/flows/custom_flow_lua_script.lua

The script is executed once on all flows as soon as the nDPI protocol detection is completed (and thus the L7 protocol has been detected). In the script ntopng allows you to access a new object named flow that points to the flow being checked.

Typically, users check the flow for specific conditions (e.g. a specific host is not using the expected DNS server) and trigger an alert that will then appear in the alert page as shown below

For instance the script below triggers an alert for all flows whose destination port is 53

if(flow.srv_port() == 53) then
   local score   = 102
   local message = "dummy alert message: port 53 detected"

   flow.triggerAlert(score, message)
end

-- IMPORTANT: do not forget this return at the end of the script
return(0)

You can find a comprehensive example at this page.

 

Extending Flow and Host Classes

As these lua scripts are designed to be lightweight because they are executed while traffic is processed, these scripts must be short in size and efficient. For this reason the class methods are simple and designed to return little information in order to minimise the amount of information exchanged between the ntopng engine and these scripts. Currently, the flow and host classes implement various methods for the most popular information used in scripts. However they can be easily extended by adding new methods as follows:

  • The lua flow class is implemented in LuaEngineFlow.cpp and defined in the _ntop_flow_reg table at the bottom of the file.
  • The host flow class is implemented in LuaEngineHost.cpp and defined in the _ntop_host_reg table at the bottom of the file.

Whenever a new method needs to be defined, it can be added to the above tables and the lua script will recognise it immediately. We invite our community to contribute with pull requests in order to implement new methods that can be useful in scripts.

 

Final Remarks

We encourage all ntopng users to learn the Lua Host and Flow Checks API in the ntop API documentation. This feature is present in all ntopng dev versions (from community up) and we hope it will pave the way to our community to develop new checks. Of course we need your feedback as we’re aware that you might need additional features that are not implemented. Please let us know your views using our community channels.

 

Enjoy !