How to use nDPI from CLI to analyse network traffic

Posted · Add Comment

Most people use nDPI indirectly being it part of ntopng and many other non-ntop developed tools. However not many people know that nDPI can also be used from the command line to analyse network traffic. This is useful to create scripts to automate detection of specific issues. ndpiReader is a testing tool used to demonstrate the library features as well run validation tests. With this tool is also possible to generate a report in CSV format that can be analysed with tools such as q.

Below you can find some practical examples of how this technique can be used in real life. Suppose we need to analyse some malware traffic in order to spot anomalies. A good starting point for sample pcap files is the CIC dataset or this website or you can use any pcap you already have collected as those part of the nDPI test set.

From the CIC dataset, support you want to analyse what flows are affected by Slow DoS.

$ ndpiReader -i dos_slow.pcap -C dos_slow.csv

Dumps flow analysis results into CSV file that contains the following fields

$ head -1 ~/Downloads/Ricci/dos_slow.csv
#flow_id,protocol,first_seen,last_seen,duration,src_ip,src_port,dst_ip,dst_port,ndpi_proto_num,ndpi_proto,src2dst_packets,src2dst_bytes,src2dst_goodput_bytes,dst2src_packets,dst2src_bytes,dst2src_goodput_bytes,data_ratio,str_data_ratio,src2dst_goodput_ratio,dst2src_goodput_ratio,iat_flow_min,iat_flow_avg,iat_flow_max,iat_flow_stddev,iat_c_to_s_min,iat_c_to_s_avg,iat_c_to_s_max,iat_c_to_s_stddev,iat_s_to_c_min,iat_s_to_c_avg,iat_s_to_c_max,iat_s_to_c_stddev,pktlen_c_to_s_min,pktlen_c_to_s_avg,pktlen_c_to_s_max,pktlen_c_to_s_stddev,pktlen_s_to_c_min,pktlen_s_to_c_avg,pktlen_s_to_c_max,pktlen_s_to_c_stddev,client_info,server_info,tls_version,ja3c,tls_client_unsafe,ja3s,tls_server_unsafe,ssh_client_hassh,ssh_server_hassh

You can now run this query to find out the top 10 slow DoS flows

q -H -d ',' "select src_ip,src_port,dst_ip,dst_port,ndpi_proto,duration from ./dos_slow.csv where dst2src_goodput_ratio<10 order by duration desc limit 10"
172.16.0.1,54240,192.168.10.50,80,HTTP,1241.436
172.16.0.1,53816,192.168.10.50,80,HTTP,1239.08
172.16.0.1,53824,192.168.10.50,80,HTTP,1239.078
172.16.0.1,53834,192.168.10.50,80,HTTP,1239.077
172.16.0.1,53840,192.168.10.50,80,HTTP,1239.077
172.16.0.1,53846,192.168.10.50,80,HTTP,1239.076
172.16.0.1,53852,192.168.10.50,80,HTTP,1239.075
172.16.0.1,53858,192.168.10.50,80,HTTP,1239.074
172.16.0.1,53866,192.168.10.50,80,HTTP,1239.073
172.16.0.1,53872,192.168.10.50,80,HTTP,1239.072

Suppose you want to know the amount of traffic the top IPe in netfix.pcap have spent watching NetFlix. First run ndpiReader as follows

$ cd nDPI/tests/pcap
$ ../../example/ndpiReader -i netflix.pcap -C /tmp/netflix.csv

then do

$ q -H -d ',' "select src_ip,SUM(src2dst_bytes+dst2src_bytes) from /tmp/netflix.csv where ndpi_proto like '%NetFlix%' group by src_ip"
192.168.1.7,6151821

Possibilites are basically endless.

Enjoy!