Amazon Virtual Private Cloud (VPC) flow logs and in essence text-based Netflow-like logs consisting of fields that describe the traffic flow. They are often collected on disk and published to S3 buckets or CloudWatch for an AWS-centric monitoring infrastructure (extra AWS charge is necessary).
Now suppose that you want to use this information to monitor your VPC using ntop tools or turn these logs in industry standard NetFlow/IPFIX flows that can be ingested in any monitoring application unable to understand this proprietary log format. In this case you can use nProbe to process these flow logs and send them to ntopng or your favourite flow collection application, not to mention the ability to deliver them to Elastic or Kafka-based consumers. The setup for VPC logs processing is similar to what you have with flow collection, with the exception that nProbe is not receiving flows via UDP (as it happens with sFlow/NetFlow/IPFIX) but it is monitoring a directory where VPC logs are stored.
For this purpose we have extended the format for the collection port parameter to support these logs. In essence nProbe:
- Watches a directory where flows are stored.
- Uses the VPC flow logs to fill up the flow cache, similar to what happens when collecting flows via UDP.
- Exports them to collectors (e.g. ntopng in the above example) using the standard nProbe features.
- Deletes the VPC log file, after processing.
Below you can find a couple of examples where you collect flows and send them to a local ntopng instance for realtime traffic visualisation and analysis.
- Collect sFlow/NetFlow/IPFIX flows on port 2055:
nprobe -3 2055 --zmq tcp://127.0.0.1:1234 - Collect AWS VPC flows on directory /data/vpc:
nprobe -3 /data/vpc --zmq tcp://127.0.0.1:1234 - Run ntopng to collect flows from nProbe:
ntopng -i tcp://127.0.0.1:1234
Using nProbe you can now transform your proprietary VPC logs into industry-standard NetFlow/IPFX flows and process them with your existing monitoring infrastructure without using proprietary formats or tools. Note that VPC logs do not include information such as DPI, so that nProbe can only guess application protocol with nDPI port/address-based heuristics. Remember that nProbe can run on AWS for full traffic visibility in case you need advanced monitoring information.
This extension is present in the current nProbe nightly build (all versions, Pro and Enterprise) and it will be integrated in the next stable release.
Enjoy!