Flows Export
############

Using n2disk with other tools like ntopng is often convenient for for extending 
continuous traffic recording with traffic analysis/visualization. For this 
reason n2disk has been extended to be able to export statistics and flows 
metadata to other analysis tools through ZMQ.

For getting started with the integration and for learning more about the 
configuration please read the `ntopng User's Guide <https://www.ntop.org/guides/ntopng/using_with_other_tools/n2disk.html#usingntopngwithn2disk>`_

The :code:`--zmq` option should be used to instruct n2disk to export flows and 
statistics in TLV or JSON format over ZMQ. 

Command line example:

.. code:: bash

   n2disk -i eno1 -o /storage --zmq tcp://127.0.0.1:5556 -I -Z -z 0 --max-file-duration 10

In order to collect flows generated by n2disk using ntopng, please follow the same 
instructions required for `using ntopng with nProbe <https://www.ntop.org/guides/ntopng/using_with_other_tools/nprobe.html>`_
running ntopng in collector mode (n2disk exports flows in probe bode, connecting 
to ntopng).

Example:

.. code:: bash

   ntopng -i "tcp://*:5556c"
   
n2disk implements a basic template (what you usually define in nprobe with the 
:code:`-T` option) that contains the following fields:

.. code:: text

   %SRC_VLAN %IN_SRC_MAC %IN_DST_MAC %IP_PROTOCOL_VERSION %DIRECTION 
   %IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV6_SRC_ADDR %IPV6_DST_ADDR 
   %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %OUT_PKTS %OUT_BYTES 
   %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT 
   %TCP_FLAGS %PROTOCOL %SRC_TOS %L7_PROTO
   %INPUT_SNMP %OUTPUT_SNMP %OBSERVATION_POINT_ID

It is possible to control flow expiration and export using the 
:code:`--lifetime-timeout` and the :code:`--idle-timeout` parameters.

Please note that some fields are populated depending on the configuration:

 - %INPUT_SNMP %OUTPUT_SNMP %OBSERVATION_POINT_ID are available when
   device and port information are added to the packet as metadata by external
   packet brokers. For example Arista Metawatch devices export Metamako
   trailers encoding the device ID and port ID (add *--hw-timestamp metawatch*
   to the n2disk configuration to enable it).
 - %L7_PROTO is available by default (no need to use -E 2 as in previous versions)

Please keep in mind that flows are computed as soon as a PCAP file has
been filled up and ready to be written on disk, this means that if you
have low traffic throughput and bug files, flows are exported with some
delay (up to 10 minutes if you have timeline enabled and little traffic).
In order to reduce this delay you can play with the :code:`--max-file-duration`
option to produce PCAP files more frequently.

Periodic stats are also exported, in JSON format, in addition to flow metadata.
Example:

.. code:: text

        {
               "iface": {
                       "name": "eth1",
                       "speed": 1000,
                       "ip": "10.0.0.2"
               },
               "probe": {
                       "ip": "10.0.0.1"
               },
               "time" : 1568866123.456,
               "bytes": 3582434,
               "packets": 32141,
               "avg": {
                       "bps": 1096415640,
                       "pps": 207410
               },
                 "drops" : {
                       "export_queue_too_long": 0,
                       "too_many_flows": 0
               },
                 "timeout": {
                       "lifetime": 120,
                       "idle": 10
               },
                 "zmq": {
                       "num_flow_exports": 2431,
                       "num_exporters": 1
               }
        }

Where:

 - *iface* reports information about the capture interface
 - *probe* reports information about the machine (e.g. management IP)
 - *time* is the current epoch timestamp
 - *bytes* is the number of captured bytes
 - *packets* is the number of captured packets
 - *avg* is the average capture rate in bps and pps
 - *drops* reports information about flow drops, based on the reason
 - *timeout* reports information about flow export settings
 - *zmq* reports statistics about flows exported on the ZMQ channel