How to Dump, Index, and Layer-7 Filter Network Traffic at High Speed

Posted · Add Comment

n2disk is an application that many of the ntop community uses to dump traffic up to 100 Gbit. What few people know is that n2disk can index data not just using packet header information (i.e. IP, port. VLAN, MAC…) but also using nDPI to produce an index that contains application protocol information.

This filtering can happen:

  • During packet capture (i.e. instruct n2disk to avoid dumping specific protocols such as Netflix or YouTube that take up a lot of disk space and that are usually harmless).
  • While extracting packets from stored pcap files.
  • With any PF_RING-based application, including those using libpcap such as tcpdump or Suricata.

L7 Capture Filters

Thanks to the integration with PF_RING FT (no additional PF_RING FT license is necessary with n2disk 10/40/100 Gbit), n2disk supports –l7-filter-conf <file> to specify a configuration file where it is possible to define which protocols can be forwarded tot the n2disk engine and which one should be discarded and hence not dumped on pcaps. Example for dropping streaming and dumping the rest, you can specify a filter file name ft.conf as the one below

[global]
default = forward

[filter]
YouTube = discard
Netflix = discard

 

L7 Extraction Filters

During pcap extraction, it is possible to extract selected packets from pcaps using L7 filtering only if an extended (add -I -E 2 to n2disk) index has been created during packet capture. This way the n2disk companion utility named npcapextract can filter packets usign L7 protocol in addition to usual packet header-based filters. For instance for filtering all Instagram traffic made by host 192.168.1.1 do

npcapextract -t /storage -b "2020-09-16 12:05:32" -e "2020-09-16 12:10:32" -o output.pcap -f "ip host 192.168.1.1 and l7proto Instagram"

This technique supports all the nDPI detected protocols, that is continuously updated as new protocol/versions are supported.

 

Using L7 Filtering with PF_RING-based tools (including tcpdump)

In addition to n2disk, PF_RING supports L7 filtering natively. You simply have to compile your application on top of PF_RING or of libpcap-PF_RING. For instance if you use tcpdump compiled on top of PF_RING (that you can find here) you can do

# PF_RING_FT_CONF=ft.conf tcpdump -ni pcap:file.pcap

or

# PF_RING_FT_CONF=ft.conf tcpdump -i eth0

Note that with live traffic, nDPI needs a few packets to detect the application protocol, hence for TCP-based protocols for instance the initial 3WH is not filtered whereas the following packets will be filtered according to L7 rules.

If you want to know more about this technique you can read more in the n2disk user’s guide.

Summary

Thanks for nDPI, via PF_RING, you can now complement existing packet header-based filtering techniques such as BPF, with layer-7 filtering both during packet capture, indexing and extraction. This allows you to save disk space dumping unwanted protocols, and extract only the traffic you care about, that can be complicated to do with the plethora of application protocols present in modern network traffic.