It’s not uncommon to see network administrator struggling tracking down packet drop on network equipments at interface level, while having a low average link utilisation. At the end it often turns out to be due to a phenomenon (well) known as microburst. While forwarding data between network links, network equipments absorb spikes with buffers, when buffers fill much quicker than they empty because of a line-rate burst, they overflow and packet loss occurs (yes you drop even though your like is in average little used).
Now it’s clear that having a tool able to monitor our network for bursts in real time is crucial in identifying potential capacity issues. On the other side, the tools we use everyday for monitoring our network won’t show up much as they are unable to monitor micro-second bursts. They provide data at a resolution of seconds, which is enough for measuring average bandwidth, but it’s definitely not enough for detecting microbursts, as they last for a fraction of a second and even a 1 second average can hide them.
Here at ntop we received many requests, in the last years, regarding this topic. This led to the development of a technology able to continuously compute the traffic rate and detect unexpected data bursts reporting them as soon as they occur. This technology has been integrated in n2disk, our traffic recording application, as well as into standalone tools able to analyse both live traffic or PCAP files. This is to combine microburst with packet-to-disk capabilities into a single tool so you can use one box (one tap, one application) to do both things instead of two specialised apps.
The microburst detection feature in n2disk allows you to specify the traffic rate limit as a percentage of the link speed, when this threshold is exceeded the system generates a log.
MICRO-BURST DETECTION [--uburst-detection] | Enable microburst detection. [--uburst-log] | Microbursts log file. [--uburst-win-size] | Window size for microburst check (usec). [--uburst-link-speed] <mbit/s> | Link speed (Mbit/s). [--uburst-threshold] | Traffic threshold wrt link speed (percenage).
Example:
# n2disk -i zc:eth1 -o /storage/ --uburst-detection --uburst-link-speed 100 --uburst-threshold 90 --uburst-win-size 10000 --uburst-log /var/tmp/n2disk/uburst.log # cat /var/tmp/n2disk/uburst.log Start End Duration Kbit Peak-Mbit/s 1477408107.351533705 1477408107.366453800 0.014920095 1378 95.971 1477408110.330377529 1477408110.341709397 0.011331868 1058 94.741
Please bear in mind that:
- Microbursts are computed on a per link basis.
- Microbursts require precise timestamping, better to use a card with hw timestamp, even though any Intel card using our PF_RING ZC drivers and our software timestamping technology is usually precise enough for microsecond-based measurements.
- Using standard kernel drivers lead to unreliable results, due to buffering happening on the machine itself because of the mechanisms for moving packets from the card to the application.
- Port mirroring usually introduces buffering/timing changes, better to use a network tap!