1.2. Custom Timeseries¶
ntopng supports the creation of custom timeseries for:
- Local hosts
- Interfaces
Neither remote hosts nor flows can be used when creating custom timeseries.
Custom timeseries can be created out of metrics.
1.2.1. Interface Metrics¶
Supported metrics for the creation of interface timeseries are:
- Layer-7 applications bytes sent and received
- Layer-4 TCP, UDP, ICMP bytes sent and received
- Total bytes sent and received
- Total alerts
- etc.
An always-updated list of metrics can be determined by inspecting
method NetworkInterface::lua
:
https://github.com/ntop/ntopng/blob/dev/src/NetworkInterface.cpp
Interface metrics are available as a lua table. An excerpt of such table is shown below:
speed number 1000
id number 1
stats table
stats.http_hosts number 0
stats.drops number 0
stats.devices number 2
stats.current_macs number 6
stats.hosts number 23
stats.num_live_captures number 0
stats.bytes number 50559082
stats.flows number 30
stats.local_hosts number 3
stats.packets number 64984
1.2.2. Host Metrics¶
Supported metrics for the creation of host timeseries are:
- Layer-7 applications bytes sent and received
- Layer-4 TCP, UDP, ICMP bytes sent and received
- Total bytes sent and received
- Active flows as client and as server
- Anomalous flows as client and as server
- Total alerts
- Number of hosts contacted as client
- Number of hosts contacts as server
An always-updated list of host metrics can be determined by inspecting this file: https://github.com/ntop/ntopng/blob/dev/src/HostTimeseriesPoint.cpp
Host metrics are available in an handy lua table such as the one exemplified below:
ndpi_categories table
ndpi_categories.Cloud number 2880
anomalous_flows.as_server number 0
active_flows.as_client number 0
bytes.rcvd number 2880
icmp.bytes.rcvd number 0
tcp.bytes.rcvd number 0
total_alerts number 0
udp.bytes.rcvd number 2880
icmp.bytes.sent number 0
other_ip.bytes.rcvd number 0
other_ip.bytes.sent number 0
anomalous_flows.as_client number 0
contacts.as_server number 1
bytes.sent number 0
instant number 1550836500
tcp.bytes.sent number 0
udp.bytes.sent number 0
contacts.as_client number 0
ndpi table
ndpi.Dropbox string 0|2880
active_flows.as_server number 1
Specifically, Layer-7 application protocols are pushed in a table
ndpi
, whose keys are the application names such as
Dropbox
. For every application there are two values separated
by a pipe, namely, bytes sent and bytes received. For example, in the
excerpt above, Dropbox
application had received 0 bytes and
had sent 2880 bytes at the time the excerpt was generated.
The table also contain a field instant
that represents the
time at which metrics have been sampled.
The table above can be accessed and its contents can be read/modified to prepare timeseries points.
1.2.3. Metric Types¶
ntopng provides metrics of two types, namely gauges and counters. Timeseries can be created out of gauges and counters, transparently. The only thing that is necessary is to tell the timeseries engine the actual type, then the rest will be handled automatically.
1.2.3.1. Gauges¶
Gauges are metrics such as the number of active flows
(e.g., active_flows.as_client
, active_flows.as_server
) or active
hosts at a certain point in time.
1.2.3.2. Counters¶
Counters are for continuous incrementing metrics such as the total
number of bytes (e.g., bytes.sent
,
bytes.rcvd
).
1.2.4. Adding Custom Timeseries¶
ntopng handles custom timeseries with updates every:
- 1 minute for interfaces
- 5 minutes for local hosts
This means that custom timeseries with a point every minute and a point every 5 minutes can be generated for interfaces and local hosts, respectively.
ntopng looks for custom timeseries in the following lua files under
scripts/lua/modules/timeseries/custom/
:
ts_minute_custom.lua
for local hosts timeseries with 5-minute updatests_5min_custom.lua
for interface timeseries with 1-minute updates
If file ts_5min_custom.lua
does not exist, ntopng will skip the
creation of custom timeseries with 5-minute updates. Similarly, if
file ts_minute_custom.lua
does not exist, ntopng will skip the
creation of custom timeseries with 1-minute updates.
Sample files ts_5min_custom.lua.sample
and ts_minute_custom.lua.sample
are
created automatically upon ntopng installation with some example
contents. Those files are ignored by ntopng. However, it is safe to
copy them to ts_5min_custom.lua
and
ts_minute_custom.lua
and modify the copies when it is necessary to
add custom timeseries.
1.2.4.1. Structure of Custom Timeseries Files¶
Every custom file must contain a method setup
which defines one or
more schemas. Every custom timeseries needs a schema to function. A
schema defines the timeseries in terms of tags and metrics. The
documentation describes what is a schema in detail. Later in this
section an example schema will be shown.
File ts_5min_custom.lua
must contain a callback
ts_custom.host_update_stats
which is called by ntopng every 5
minutes for every active local host. This callback accepts the
following arguments:
when
The time (expressed as a Unix Epoch) of the callhostname
The IP address of the host, possibly followed by a VLAN taghost
The host metrics in a lua tableifstats
The interface stats of the host interfaceverbose
and extra flag passed when ntopng is working in verbose mode
File ts_minute_custom.lua
must contain a callback
ts_custom.iface_update_stats
which is called by ntopng every
minute for every monitored interface. This callback accepts the
following arguments:
when
The time (expressed as a Unix Epoch) of the call_ifname
The name of the monitored interfaceifstats
The interface stats of the monitored interfaceverbose
and extra flag passed when ntopng is working in verbose mode
Callbacks can be used to append points to the timeseries. Indeed,
once the schema is defined, it is necessary to append
points to
the timeseries. The function used to append points to the timeseries
is the ts_utils.append
documented later in this section of the
documentation.
1.2.4.2. Example¶
Let’s see how to add an interface timseries that counts the number of issues detected when analyzing sequence numbers. The total issues detected when analyzing sequence numbers is considered as the sum of TCP retransmitted, out-of-order and lost packets.
The first thing to do is to add a schema to the setup
function
of ts_minute_custom.lua
. The schema is created as as follows:
schema = ts_utils.newSchema("iface:tcp_seq_errors", {step = 60})
schema:addTag("ifid")
schema:addMetric("packets")
The first argument of newSchema
specifies the timeseries name
"iface:tcp_seq_errors"
. Timeseries interfaces must start
with prefix iface:
. The second argument is a table that must
contain argument step
which tells how frequently the
timeseries will be updated. As we are in the 1-minute local hosts
timeseries lua script, a value of 60
must be specified here.
Then, function addTag
is used to indicate an interface id
ifid
that will be used to uniquely identify the timeseries
when multiple interfaces are monitored. Finally, addMetric
is
called with an argument packets
to indicate the metric
name. Note that both the ifid
and packets
are just
plain strings here, their actual values will be set in the
ts_custom.iface_update_stats
when updating the timeseries with
new points.
The number of issues detected when analyzing sequence numbers is a
counter, that is, is an always-increasing function of
time. By default, schemas consider metrics as counters so there is no
need to specify this type upon schema addition. For gauges, one has
to indicate an extra metrics_type
in the table containing the
step
. So for example, to create a 1-minute timeseries for the number of
active flows of a given host, one can use the following syntax ts_utils.newSchema("host:flows", {step=60, metrics_type=ts_utils.metrics.gauge})
.
To update the timeseries with new points, callback
ts_custom.iface_update_stats
is extended with a
ts_utils.append
call as follows.
ts_utils.append("iface:tcp_seq_errors",
{ifid = ifstats.id,
packets = ifstats.tcpPacketStats.retransmissions
+ ifstats.tcpPacketStats.out_of_order
+ ifstats.tcpPacketStats.lost},
when, verbose)
The first argument of ts_utils.append
is the timeseries name
and must be equal to the one specified when defining the schema. The
second argument is a table which must contain the tag (ifid
)
and the metric (packets
) which must be set to their actual
values. As it can be seen from the example above, the field id
of table ifstats
is used to set tag ifid
, whereas the
sum of ifstats.tcpPacketStats
table fields
retransmissions
, out_of_order
and lost
are used
as value for the metric packets
.
Finally, the third argument when
is the time of the call, and
the latest argument verbose
indicates whether ntopng is
operating in verbose mode.
From that point on, the timeseries will be consistently updated by ntopng.
Multiple schemas and multiple ts_utils.append
can be added in
the same file.
The full example can be seen at: https://github.com/ntop/ntopng/blob/dev/scripts/lua/modules/timeseries/custom/ts_minute_custom.lua.sample
Another example that creates 5-minute timeseries of local hosts total bytes can be seen at https://github.com/ntop/ntopng/blob/dev/scripts/lua/modules/timeseries/custom/ts_5min_custom.lua.sample
1.2.5. Locating Stored Custom Timeseries¶
TODO
1.2.6. Charting Custom Timeseries¶
TODO