For years traffic monitoring tools assumed traffic was in clear text. This because when the Internet was created all the main protocols such as DNS, HTTP, SMTP, Telnet, POP were in clear. With this practice it was easy to report let’s say the breakdown of DNS response codes, or detect force brute attacks on HTTP authentication.
With the advent of traffic encryption, the (bad?) practice of inspecting traffic was no longer possible and network developers had several headaches. Those who were unable to see new opportunities with traffic encryption started to ask questions like “is your tool able to decode SSL ?” or “nDPI is useless with encrypted traffic, isn’t it?”. While these questions are legitimate to ask, they indicate that the mindset of those asking these questions was simply wrong. Look at this chart created by ntopng on a WiFi network earlier this summer (note this is just an example, don’t look at % as a number written on stone as on other type of networks numbers will definitively be different. For instance on residential networks, streaming protocols like Netflix will occupy a big portion of the overall picture).
The Internet we used to know is this. DNS and other major protocols are listed under Other, and all the rest of traffic was basically encrypted. At ntop we believe that encryption opens up many opportunities and that it is a very good news for both privacy and network traffic application developers. This is because we can simplify many things. See for instance the IDS market, where tools like Suricata and Zeek are very popular feeds for many cybersecurity companies. One of the core principles of these tools (in particular for Suricata) was to search specific patterns in traffic streams and report an alert when an entry was found. While this practice worked for long time, today is no longer a good idea. See for instance Emerging Threats rules and you will notice that most of the rules are becoming useless.
$ grep ^alert emerging-all.rules | grep -v HTTP_ | grep -v FTP | grep -v POST | grep -v GET | grep -v urilen | grep -v http_uri | wc -l
8499
$ grep #alert emerging-all.rules | wc -l
6923
$ grep ^alert emerging-all.rules | wc -l
17940
$ grep ^alert emerging-all.rules | grep DNS |wc -l
1999
There are about 7k rules commented out as obsolete, out of about 18k rules, ~8k rules are not for HTTP/FTP protocols. Most of these remaining rules are either for obsolete apps (this was last updated in 2010)
alert tcp $HOME_NET any -> $EXTERNAL_NET any (msg:”ET CHAT Yahoo IM conference offer invitation”; flow: to_server,established; content:”YMSG”; nocase; depth: 4; content:”|00|P”; offset: 10; depth: 2; reference:url,doc.emergingthreats.net/2001262; classtype:policy-violation; sid:2001262; rev:5; metadata:created_at 2010_07_30, updated_at 2010_07_30;)
or for protocols such as the DNS that is becoming encrypted (see RFC 8484, DNS Queries over HTTPS). Firefox already uses it, and when operating systems like iOS/Android or browsers like Chrome will use it by default, DNS traffic will become a minor Internet protocol in terms of traffic volume.
So said that traffic encryption is already here and soon it will be pervasive, at ntop we have decided to live with this and make our tools encryption friendly as like it or not, this will be the future of Internet traffic. We have extended nDPI to handle encryption as first citizen, and in the upcoming nDPI v3 version we have already implemented:
- TLS JA3
- TLS Certificate Fingerprint, validity, unsafe ciphers, SNI.
- SSH HASSH and client/server signatures.
this is to allow all nDPI-based applications to use industry-standard techniques to fingerprint encrypted traffic.
In addition to that we have enhanced nDPI to go beyond that and figure out what is happening behind an encrypted connection. A typical question would be: is this SSH connection interactive or it is a SCP file transfer (i.e. read it as data exfiltration)? This TLS connection is safe or a malware is using it? For answering the latter questions, Cisco Joy has been integrated into nDPI (see ndpiReader -J for more info) and we’re planning to go beyond this in post 3.0 release.
In essence due to encryption most software applications will be based on these principles
- As you can’t inspect encrypted traffic anymore, validate it using fingerprints/signatures (abuse.ch is a great one).
- Use IP blacklists to identify traffic coming/going towards malware hosts (iplists.firehol.org for instance).
- Use TLS SNI to categorise encrypted traffic (nDPI does it for a long time).
- Use traffic metrics to characterise the encrypted traffic, and figure out malware activities WITHOUT decoding it.
nDPI is ready for these new challenges, so all ntop tools. It’s time for you to start playing with them and have full 100% traffic visibility in your network with asking the “can you inspect encrypted traffic?” as if you read this article until here, now you know the answer.
Enjoy!
PS. If you are now wondering if many IDSs are obsolete with encrypted traffic, you know this answer too. It’s time to rethink them, make them faster and focused on what matters today, not 15 years ago when they first designed.
PS2. Some fingerprints such as JA3 are nice to have, useful in certain scenarios, but nothing more than that as we observe many false positives when matching them against blacks lists (e.g. those from abuse.ch).