Why Traffic Behaviour Analysis is Good (was Encrypting TLS 1.3 Traffic)

Posted · Add Comment

In the latest nDPI meetup, we have discussed future directions, including extending the current encrypted traffic analysis features. Currently nDPI supports both fingerprint and behaviour encrypted traffic analysis techniques to provide TLS traffic visibility. At ntop we have never liked too much fingerprinting techniques such as JA3 that are used by many popular IDSs and security tools, simply because they often lead to false positives making them a “nice to have” features but nothing more than that.

Recently the IETF is designing a new TLS 1.3 extension named ECHO (Encrypted Client Hello). In TLS 1.3 the only initial handshake message in cleartext is the Client Hello: it is quite important as it contains in cleartext the SNI (Server Name Indication), that is basically the name of the server the client is contacting. ECHO is trying to fix data disclosure, by sending this information encrypted so that a network monitoring tools will be unable to inspect traffic and see if we’re accessing maps.google.com instead of gmail.com. Although ECHO works with cleartext DNS, the idea of Firefox (to date the only browser supporting it) is to avoid any loss of metadata by forcing to use DoH, This is to prevent monitoring tools to see what site has been visited by looking at the DNS traffic. Note that the TLS client encrypts the data in ECHO using a public key that is placed in the DNS of the contacted host (it is stored in a TEXT record named _ensi.domainname). Example:

_esni.hackthetower.it.	1800	IN	TXT	"/wHuOSrNACQAHQAgKO0MmMYpnkF3QL3YhV0AiwWoqqaTEHp7/lKIzBLLeS4AAhMBAQQAAAAAXsvrgAAAAABe09SAAAA="

In nDPI we support detection of encrypted SNI (ESNI).

TCP <-> [proto: 91.220/TLS.Cloudflare][cat: Web/5][197 pkts/17789 bytes <-> 211 pkts/175194 bytes][Goodput ratio: 40/93][6.64 sec][ALPN: h2;http/1.1][TLS Supported Versions: TLSv1.3;TLSv1.2;TLSv1.1;TLSv1][bytes ratio: -0.816 (Download)][IAT c2s/s2c min/avg/max/stddev: 0/0 28/26 1000/1000 139/134][Pkt Len c2s/s2c min/avg/max/stddev: 54/60 90/830 770/1506 77/677][TLSv1.3][JA3C: e5ef852e686954ba9fe060fbfa881e15][JA3S: eb1d94daa7e0344597e756a1fb6e7054][ESNI: 9624CB3C4E230827F78CF5BF640D22DEA33FCC598EA6A32D939905586FBE997B9E68661F8956D4893072E19DE24CD1FB88A9F71FC4CC01BAB5C914FDF96A647D671B5E89859BAEEAB122218688496DF4DF0C328C3D5F940B109CEB2A2743D5CBE3594288A229B8C7E2F88303E3FE1A26A89E5001F2BD936890FEF78F06E05ECC063A68BDB8C18DFAC114CF1FECDB8BE1FC2FEECB2315D27998D682B129FD1E3EB5D7985DCBDC452A1082CCC038E0BF69570FEFAC6BC6FB951F89B6792CADA76403C02CEB5DCE1CE6EDDD16D5F7FB6B85D2B92485448DE0088E421E83F1E28B267FBE3B59AE0496FB845213C271D4C5AC5E9E7E5F6A3072445307FCCEB7306710459991C40CC4DC1FC325154C7974DD780371397805456A19AE23EE88475C1DF07697B666][ESNI Cipher: TLS_AES_128_GCM_SHA256][Cipher: TLS_AES_128_GCM_SHA256]

ESNI is not very useful, and ancillary data such as record keys (used for encryption and places in DNS) are often updated every hour, making the record digest useless to identity the accessed web site (this is because it cannot be used as fingerprint for the contacted site).

You might ask yourself: is this the end of visibility and nDPI? While ECHO has definitively implications on traffic visibility this is not the case, and the reasons are manyfold:

  • ECHO will prevent TLS fingerprints to work, whereas behavior analysis will still work. This is good news for nDPI as we’re working since a few months to enhance behaviour traffic analysis and advertising to avoid using fingerprints as they might lead to false positives.
  • Operators and security teams can decide to block TLS traffic with these extensions as they prevent them from enforcing traffic policies.
  • ECHO is still in draft, so it’s not clear if it will be standardised in the current form.
  • While DoH is not mandatory for ECHO, blocking non-operator provided DoH (that nDPI can detect) servers, can mitigate loss of metadata. This means that traffic monitoring and security applications can still have enough hints on metadata leaking from the DNS queries.
  • Visibility of CDN-based traffic will be partially affected, this unless the IP address/port will disclose the nature of the traffic as most large companies do (e.g. https://docs.microsoft.com/en-us/office365/enterprise/urls-and-ip-address-ranges is used inside nDPI). For non CDNs the IP address is still a good source of information.
  • Malware will probably not use ECHO for a long time (today most malware are TLS 1.1 when not using HTTP). However, just to repeat it one more time, we believe that nDPI should detect malware using behavior analysis techniques rather than fingerprints.
  • To date the only browser supporting ESNI is Firefox with DoH enabled (https://encryptedsni.com).
  • Even if the initial handshake is encrypted, many other things can be observed such as SPLT (Sequence of Packet Length and Time), entropy etc. to complement what will be hidden. So for malware analysis probably ECHO won’t change much, but for traffic visibility traffic monitoring will be affected and some popular open source projects, probably will needs to refocus a bit to be effective.

In summary, expect in the next few years TLS to be completely encrypted and this is good news for the Internet, but not su good for monitoring and security tools. nDPI is already supporting ESNI, and we’ll do our best to keep it update with the latest TLS 1.3 extensions as they will be standardised. ECHO is not the end of TLS fingerprinting, but it can definitively be yet another good reason to look at TLS behaviour instead of limiting the analysis to fingerprint.  Stay tuned for more news about using nDPI and encrypted traffic analysis.