Bug Report

This post is outdated. Check this README for updated information.


Submitting a Bug Report

For open-source applications such as ntopng, if you want to report a bug, please open an issue on our official Github page. For commercial applications such as nProbe you can contact us privately or fill this form.

Submitting a Code Patch

If you want to submit a patch please send us a merge request via Github. We will be happy to review it and possibly include in one official branch.

What To Do In Case Of Crash

In the (unlikely) case of crash, it is fundamental to understand the root cause of the crash. Finding the cause of a crash boils down to debugging the application, but don’t worry, you do not have to be a programmer to do that! In the remainder of this page we will show some simple steps that that will allow you to debug any product and report us the resulting information.

Debugging Applications Built from Source

When debugging applications built from source, you have to make sure the compiler has not performed compile-time code optimizations. This kind of optimizations usually make the code (marginally) faster but prevent the application from being debugged easily.

Typically, when using the GNU toolchain, debugging symbols are configured in the Makefile. Therefore, you have to make sure no option such as -O2 or -Os is used there. Another option that you should enable when using the GNU toolchain is  -b.

Once all the optimizations have been turned off, you are ready to execute the application but, to record a meaningful trace of the crash, you have to enable the core dump generation. Enabling core dump generation is something that varies slightly across operating systems and typically requires you to set ulimit -c unlimited (in bash) or limit coredumpsize unlimited (in tcsh).

A core dump is a file that contains the status of the program at the exact moment of the crash. For this reason this file is fundamental to understand what has caused the crash. To obtain this file you have to wait until the application crashes and this can take some time. Thus you have to be patient, and use the application as usual. When the crash happens a core dump should be generated. Depending on the platform the core is saved into the same directory where you have started the application, or on MacOS X it can be found in the /cores directory. Usually the core file is named core, sometimes it has a PID (process ID) appended to it cores.<PID>.

Once you’ve the core you can analyze it. In order to analyze the core file you need to install the gdb debugger. Done that you can debug the core as follow:

  1. Run the gdb: gdb ./<application executable name> -c <path to the core>. At this point the debugger will open a shell where you can run commands.
  2. Backtrace
    (gdb) bt
    #0 0x00007fff96a6ee42 in __semwait_signal ()
    #1 0x00007fff92d55dea in nanosleep ()
    #2 0x00000001060d7601 in _ntopSleepMSWhileSameState (file=0x105fa8d82 "main.c", line=678, ulDelay=5000) at util.c:4103
    #3 0x00000001060d77e9 in _ntopSleepWhileSameState (file=0x105fa8d82 "main.c", line=678, secs=5) at util.c:4131
    #4 0x0000000105f9eb09 in main (argc=6, argv=0x7fff65b99ae0) at main.c:678
  3. Show threads information
    (gdb) info threads
    6 0x00007fff96a6faf2 in read ()
    5 0x00007fff96a6edf2 in select$DARWIN_EXTSN ()
    4 0x00007fff96a6ebca in __psynch_cvwait ()
    3 0x00007fff96a6ee42 in __semwait_signal ()
    2 0x00007fff96a6ee42 in __semwait_signal ()
    * 1 0x00007fff96a6ee42 in __semwait_signal ()

If you can’t figure out what is causing the application to crash, you can send us the generated core dump along with the ntopng binary for inspection.

Debugging Packaged Applications

ntop also distributes binary applications that can easily be debugged even when they run as services. On operating systems with systemd, generating cores for applications running as services require systemd-coredump executable. On ubuntu the executable and its ancillary services can be installed with a single package installation:

apt-get install systemd-coredump

Other distributions may have a slightly different name for the package. Once the package is installed, core dumps will automatically be generated for all the running services that undergo a crash. It may also be necessary to add LimitCORE=infinity to the service unit file’s [Service] section to prevent core dumps from being limited in size.

By default, systemd writes core dumps to the journal. In order to see generated core dumps, run:

coredumpctl list
TIME PID UID GID SIG PRESENT EXE
Wed 2018-05-30 15:02:14 BST 14595 0 0 11 * /bin/bash
Wed 2018-05-30 15:10:20 BST 22290 0 0 11 * /usr/bin/myapp
Wed 2018-05-30 15:50:38 BST 22739 0 0 11 * /usr/bin/myapp

To get the most recent myapp coredump and export it to a file called myapp.core, run:

coredumpctl -o myapp.core dump /usr/bin/myapp

Similarly, core dumps can also be extracted using the pid. For example, the following command can be run to extract core dump related to process with pid 22739 into file myapp.core:

coredumpctl -o myapp.core dump 22739

Once the core file has been extracted from the journal, send it to us along with the application executable so we can fix the crash and distribute a fix.

In the remainder of this page we will show examples of ntopng core dump generation on ubuntu16, one of the most common distributions used with our products.

Debugging ntopng on Ubuntu16

Install the systemd coredump utility

sudo apt-get install systemd-coredump

Edit the ntopng service file and add LimitCORE=infinity to the [Service] section.

$ cat /etc/systemd/system/ntopng.service
[Unit]
Description=ntopng high-speed web-based traffic monitoring and analysis tool
After=network.target syslog.target redis.service pf_ring.service cluster.service mysql.service
Requires=
Wants=pf_ring.service cluster.service
PartOf=pf_ring.service cluster.service

[Service]
LimitCORE=infinity
Type=simple

<file continues...>

Tell systemd to reload the service files

$ sudo systemctl daemon-reload

At this point you can restart ntopng and wait for the crash.

$ sudo systemctl restart ntopng

To make sure the core dump is generated, you can artifically send a “crash” signal to the ntopng process. To do this, get the PID of the process and use the kill command as follows

$ ps aux | grep ntopng
nobody 26108 6.7 2.6 3034576 436848 ? Ssl 18:21 0:26 /usr/local/bin/ntopng /run/ntopng.conf
simone 32642 0.0 0.0 11280 964 pts/1 S+ 18:28 0:00 grep --color=auto ntopng
$ sudo kill -SEGV 26108

Once the process has crashed, the coredumpctl utility will show it in the list of core dumps

$ sudo coredumpctl list
TIME PID UID GID SIG PRESENT EXE
Mon 2018-09-10 18:19:11 CEST 21398 0 0 11 * /usr/bin/ntopng
Mon 2018-09-10 18:19:26 CEST 22092 0 0 11 * /usr/bin/ntopng
Mon 2018-09-10 18:21:47 CEST 23132 0 0 11 * /usr/bin/ntopng

Core dump extraction and analysis can be performed as described above.