Observing nDPI from the Inside: Introducing USDT Tracepoints

by Ivan Nardi

One of the recurring challenges when embedding the nDPI library into a production application is answering a deceptively simple question: what is nDPI actually doing right now? You know packets are coming in, flows are being classified, and risks are being flagged — but at what rate? With what latency? Are some protocols taking longer to classify than usual? Is that CPU spike caused by a wave of TLS flows, a flood of DNS queries, or something else entirely?

Your application or monitoring backend probably already exports some statistics (total flows classified, protocol distribution, and so on). But those numbers are often too coarse to diagnose real problems: aggregated over a reporting interval they can smooth out brief spikes; they tell you what was classified but not how hard it was to get there; and they lump together traffic from different sources into a single counter, hiding potentially important details. The dashboard confirms something went wrong — it rarely tells you why.

Historically, the next step was to add more detailed logging, recompile, redeploy, and hope you can reproduce the issue. Not ideal.

A more principled approach is to use generic Linux tracing tools such as perf or the BCC toolkit. These are powerful for low-level concerns (hot functions, call stacks, memory allocation), but they cannot easily answer questions about application logic. They see instructions and functions, not flows, protocols, or dissectors. Dynamic tracing via uprobes is more flexible but does not provide a stable interface.

With the introduction of USDT (User-Space Statically Defined Tracing) support in nDPI, there is now a better and simpler way.

---

What is USDT?

USDT is a well-established technique for embedding lightweight, named tracepoints directly into a library or application at compile time. The key word is lightweight: when no tracing tool is attached, a USDT tracepoint costs nothing at runtime. The moment you attach a tracer such as bpftrace and start collecting data, the overhead is usually still negligible.

This is the same approach used by production systems like PostgreSQL and the Linux kernel itself. It allows you to answer deep internal questions about a running process without recompiling, without restarting, and without modifying your application code. Because tracepoints are placed by the developers themselves at meaningful locations, they expose application-level semantics.

For a library like nDPI — often embedded in high-performance traffic analysis pipelines — this kind of introspection can make the difference between guessing and knowing.

---

USDT Tracepoints in nDPI

To enable USDT support, build nDPI with the --enable-usdt-probes flag:

./autogen.sh && ./configure --enable-usdt-probes --enable-debug-build && make

Once compiled, you can list all available tracepoints with:

bpftrace -l 'usdt:./path/to/libndpi.so:ndpi:*'

The tracepoints are grouped under the ndpi provider and cover key internal events — including flow classification, hostname extraction, and IP fragmentation — giving you a structured view of nDPI’s internal state that any eBPF-based tool can observe at runtime.

---

Example 1: Tracing the Source of an IP Fragmentation Spike

IP fragmentation is a rare but tricky aspect of network operation. When your monitoring backend reports an unusual rise in fragment counters, the number alone does not tell you much: is it a single misconfigured host, a tunnel with a suboptimal MTU, or something more systematic?

The fragment_ipv4 probe fires once for every fragmented IPv4 packet processed by nDPI and provides a pointer to the packet’s IP header. This makes it trivial to tally fragment counts per source address in real time — no recompilation, no packet
capture, and no tcpdump filter required.

# Count fragmented packets by source IP
bpftrace -e '
usdt:./path/to/ndpiReader:ndpi:fragment_ipv4 {
  $iph = (struct ndpi_iphdr *)arg0;
  @frags_by_src[ntop($iph->saddr)] = count();
}'

During a fragmentation event, the output might look like:

@frags_by_src[10.0.0.1]:        3
@frags_by_src[10.0.0.47]:       7
@frags_by_src[10.0.10.82]:   9841

One host is responsible for nearly all fragments. The investigation immediately narrows: check whether that host is behind a tunnel (GRE, IPsec, IPIP) with an incorrect MTU, whether it is running software that deliberately fragments packets, or whether it is generating anomalies worth escalating to the security team. Without the tracepoint, answering this question typically requires a full packet capture and offline analysis; with it, the answer arrives in seconds on a live system.

---

Example 2: Real-Time Protocol Distribution and Unknown Traffic Spikes

Sometimes the most useful thing you can do is simply observe the protocol mix in real time. The following one-liner counts every classified flow by application protocol ID and prints a fresh distribution every ten seconds:

bpftrace -e '
usdt:./path/to/ndpiReader:ndpi:flow_classified {
  @protos[arg1] = count();
}
interval:s:10 {
  print(@protos);
  clear(@protos);
}'

Under normal conditions the output is stable — a predictable mix of TLS, DNS, HTTP, and the application protocols that reflect your traffic baseline:

@protos[91]:   8341    # TLS
@protos[5]:    4812    # DNS
@protos[7]:     923    # HTTP
@protos[133]:   411    # Netflix
@protos[0]:      14    # Unknown

What you are watching for is a sudden rise in protocol ID 0 (Unknown). This bucket contains flows that nDPI could not classify, and a spike there — without a corresponding increase in total flow volume — is often an early indicator that something new or unusual is present on the network.

This matters beyond pure visibility. Unknown flows are processed up to the maximum packet budget before classification fails. Under normal conditions this path is rare. When Unknown traffic surges, however, the additional dissection work can significantly increase CPU usage. In other words, the unexplained CPU spike reported by your monitoring system may be caused not by higher traffic volume but by a qualitative change in traffic composition.

---

Get Involved

USDT support in nDPI is still at an early stage. If you are embedding nDPI in your own tools or workflows, we would love to hear whether you find this feature useful and how it could be improved.

For the full list of available probes, their arguments, and additional examples, see the official documentation.

The source code is available on GitHub. Give it a try and let us know what you discover.