nProbe has been originally designed as an efficient tool able to capture traffic packets and transform them into flows. Call it network probe or sensor. Over the years we have added the ability to collect flows (i.e. nProbe is both a probe and a collector), so that nProbe can now act as probe, collector, also proxy by covering flows across formats. For instance you can collect IPFIX flows and export them in NetFlowV9. All this following the standards as confirmed by the IPFIX interoperability tests.
Until now we focused in collecting standard flow fields (i.e. those defined in the NetFlow/IPFIX RFCs), but as vendors often use custom fields (i.e. on IPFIX those with a non zero PEN) we receive many requests about supporting information element (IE) X and Y. Initially we have handled them as exceptions (e.g. nProbe supports since many years a few Cisco NBAR, PaloAlto and IXIA IEs) but this was not a long term solution.
For this reason we have decided to enhance nProbe Pro (unfortunately the standard nProbe lacks this feature as we rely on mechanisms not present on that version) with the ability for users to define at runtime new fields they want to support during flow export. As nProbe has been based on the concept of template (-T command line option), in essence it is now possible to enhance the list of available IEs with custom ones. These new IEs can be collected and exported, as any other native IE such as IP address and port. For instance if you send nProbe some flows that contain a custom numeric IE that contains the name of the process that generated flow X, nProbe can collect the field and export it for instance in JSON-based formats (e.g. Apache Kafka, and ElasticSearch) but it cannot use it for doing anything more than this. In order to do this. nProbe Pro has a new command line option
--load-custom-fields <file> | Load custom templates from the specified file.
- Field name. This is the string that will be used in the template definition (-T) and when exported to MySQL and JSON.
- STANDARD_ALIAS is a mapping to a standard field. NONE can be specified when no mapping to a standard field is available. Typically, it is useful to specify this mapping to convert
proprietary fields into standard, well-known ones. - PEN. Use 0 if this is a standard field, or a custom enterprise number if this IE is coming from a specific vendor.
- Field Id, is the numeric identifier that is used to identify the field in the flow template.
- Len is the field length (in bytes).
- Format specifies how to represent the field when exporting it (e.g. in JSON) or dumping it on text field (-P).
If you are wondering how you can fill these values, I suggest you to look at the flow specs provided by your flow vendor device, or if you do no have them, to use wireshark to capture the flow template and dissect it.
With this enhancement you can now collect custom IEs with nProbe simply providing a configuration file and specifying with -T what to export. For instance do
nprobe -i none -3 2055 --load-custom-fields custom_fields -P /flows -T "%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_NEXT_HOP %INPUT_SNMP %OUTPUT_SNMP %IN_PKTS %IN_BYTES %FIRST_SWITCHED %LAST_SWITCHED %L4_SRC_PORT %L4_DST_PORT %TCP_FLAGS %PROTOCOL %VENDOR_PROPRIETARY_65"
for collecting flow with a proprietary field Id 65 and dump them in test format under /flows. For the time being, this new feature is present only in the development version of nProbe and it will be included in the next stable version.
Happy flow collection!
PS. If you are wondering if these IEs can be sent to ntopng via ZMQ, the answer is yes.