Most people use nDPI indirectly being it part of ntopng and many other non-ntop developed tools. However not many people know that nDPI can also be used from the command line to analyse network traffic. This is useful to create scripts to automate detection of specific issues. ndpiReader is a testing tool used to demonstrate the library features as well run validation tests. With this tool is also possible to generate a report in CSV format that can be analysed with tools such as q.
Below you can find some practical examples of how this technique can be used in real life. Suppose we need to analyse some malware traffic in order to spot anomalies. A good starting point for sample pcap files is the CIC dataset or this website or you can use any pcap you already have collected as those part of the nDPI test set.
From the CIC dataset, support you want to analyse what flows are affected by Slow DoS.
$ ndpiReader -i dos_slow.pcap -C dos_slow.csv
Dumps flow analysis results into CSV file that contains the following fields
$ head -1 ~/Downloads/Ricci/dos_slow.csv #flow_id,protocol,first_seen,last_seen,duration,src_ip,src_port,dst_ip,dst_port,ndpi_proto_num,ndpi_proto,src2dst_packets,src2dst_bytes,src2dst_goodput_bytes,dst2src_packets,dst2src_bytes,dst2src_goodput_bytes,data_ratio,str_data_ratio,src2dst_goodput_ratio,dst2src_goodput_ratio,iat_flow_min,iat_flow_avg,iat_flow_max,iat_flow_stddev,iat_c_to_s_min,iat_c_to_s_avg,iat_c_to_s_max,iat_c_to_s_stddev,iat_s_to_c_min,iat_s_to_c_avg,iat_s_to_c_max,iat_s_to_c_stddev,pktlen_c_to_s_min,pktlen_c_to_s_avg,pktlen_c_to_s_max,pktlen_c_to_s_stddev,pktlen_s_to_c_min,pktlen_s_to_c_avg,pktlen_s_to_c_max,pktlen_s_to_c_stddev,client_info,server_info,tls_version,ja3c,tls_client_unsafe,ja3s,tls_server_unsafe,ssh_client_hassh,ssh_server_hassh
You can now run this query to find out the top 10 slow DoS flows
q -H -d ',' "select src_ip,src_port,dst_ip,dst_port,ndpi_proto,duration from ./dos_slow.csv where dst2src_goodput_ratio<10 order by duration desc limit 10" 172.16.0.1,54240,192.168.10.50,80,HTTP,1241.436 172.16.0.1,53816,192.168.10.50,80,HTTP,1239.08 172.16.0.1,53824,192.168.10.50,80,HTTP,1239.078 172.16.0.1,53834,192.168.10.50,80,HTTP,1239.077 172.16.0.1,53840,192.168.10.50,80,HTTP,1239.077 172.16.0.1,53846,192.168.10.50,80,HTTP,1239.076 172.16.0.1,53852,192.168.10.50,80,HTTP,1239.075 172.16.0.1,53858,192.168.10.50,80,HTTP,1239.074 172.16.0.1,53866,192.168.10.50,80,HTTP,1239.073 172.16.0.1,53872,192.168.10.50,80,HTTP,1239.072
Suppose you want to know the amount of traffic the top IPe in netfix.pcap have spent watching NetFlix. First run ndpiReader as follows
$ cd nDPI/tests/pcap $ ../../example/ndpiReader -i netflix.pcap -C /tmp/netflix.csv
$ q -H -d ',' "select src_ip,SUM(src2dst_bytes+dst2src_bytes) from /tmp/netflix.csv where ndpi_proto like '%NetFlix%' group by src_ip" 192.168.1.7,6151821
Possibilites are basically endless.