Aut viam inveniam aut faciam, Hannibal 247-182 B.C.
For years ntopng has been a solution for collecting, analysing and visualising network traffic, but with a major limitation. It is too rich in data display and reporting that users needs to be experts in know what they are looking for. If not, they will be lost with all the data you can find on the web GUI, that is the opposite of what we tried to do.
It is now time to go beyond simple threshold analysis, as currently implemented in ntopng (if metric X is above value Y, then alert, when back to a value below Y we’re back to normality), and move towards a better tool able to interpret data in an automatic and autonomous fashion. Ideally the ntopng user interface should be less rich in reports, and much more powerful in telling the user something like: “everything is working as expected, in case there is a problem I will report it”. In the old ntop (non -ng), thanks to RRD we implemented exponential smoothing (Holt-Winters in our case) to detect anomalies. It is now time to implement time series analysis in ntopng too. This is to complement (not to replace) threshold-based detection. In essence we need to still use thresholds for metrics we know (e.g. the DNS positive/error response ratio should be > 50%, otherwise there is probably something wrong happening), and more comprehensive algorithms for detecting changes in behaviour that might be relevant to report to the users.
Contrary to the current trend in the industry that is deploying machine learning even when it’s not necessary, we’ll do our best to create a traffic analysis solution able to give users the traffic analysis they expect without having to use cluster of machines, GPUs or costly cloud-based traffic analysis. It should work on a small Raspberry PI as well on a powerful server. Our work in 2018 on data indexing (BTW we’re still consolidating this work) and rich metric computation , is paving our way to this implementation. Over 2000 years ago, when his generals told Hannibal that it was impossible to cross the Alps by elephant, he said: “I shall either find a way or make one”. This is the plan for data analysis in ntopng: either use the best of existing techniques for achieving our goal, or create something new for our community.
Stay tuned!