Months ago Draios Inc introduced sysdig a kernel module and user-space library for capturing systems events and thus analyse what is happening on a Linux box. The idea has been immediately appealing for us at ntop, this for many reasons:
- With our tools we can analyse network packets, extract metadata (e.g. URLs, network delays, username who has performed a certain action), but up to the system. In essence even though we install ntopng or nProbe onto a Linux box (either physical or virtual), we currently see packets and we miss the system interactions: process A speaks with process B that sends a HTTP request to xyz.com. We see the HTTP request to xyz.com but we have no clue what is the process doing what.
- We have used SNMP many years ago (to be honest I started with OSI) and we know that these paradigms do not cope with dynamic environments. We needed something better.
- Using sysdig is a lot of fun, but it’s yet another environment, that requires its APIs, its programming language (the high-level library is write in C++11)… in essence you have to accept the rules.
- When we have created our tools to play with sysdig, we have realised that merging packets with system calls was not simple as PF_RING was receiving packets on a certain way, sysdig on a similar yet orthogonal way, with the result that merging information coming from the system and from the network was not as easy as we expected.
On the other hand we know that we have been playing with packets for 15 years now, and we did not like to drop all our beloved PF_RING based tools in favour of sysdig, simply because these two approaches were close but not close enough to make our life easy. This has been the driving force to integrate sysdig-generated events into PF_RING. To enable us to reuse all our tools (even non-ntop tools such as tcpdump or wireshark) and see sysdig as yet another network interface, but nothing different than that.
Inside the PF_RING SVN code (this code will go into the next stable PF_RING release that will be out soon),we have integrated sysdig. Our way. Namely once the sysdig kernel module is loaded in the kernel, the user-space PF_RING code will not need any sysdig library. This way we have removed dependencies on sysdig pre-requisites (e.g. C++11) and we have made sysdig a native PF_RING component. Now you can do things like:
root@ubuntu:/home/deri/PF_RING/userland# ./examples/pfcount -i sysdig: -v 1
Using PF_RING v.6.0.2
Capturing from sysdig
# Device RX channels: 1
# Polling threads: 1
22:30:29.932674439 [cpu_id=0][tid=1292][1|> syscall]
22:30:29.932675107 [cpu_id=0][tid=1292][0|< syscall]
22:30:29.932675377 [cpu_id=0][tid=1292][1|> syscall]
22:30:29.932676611 [cpu_id=0][tid=1292][8|< write]
22:30:29.932689184 [cpu_id=0][tid=1292][9|> write]
22:30:29.932692526 [cpu_id=0][tid=1292][82|< select]
22:30:29.932695822 [cpu_id=0][tid=1292][152|< switch]
or (below you will see tcpdump compiled with PF_RING libraries)
root@ubuntu:/home/deri/PF_RING/userland/tcpdump-4.1.1# ./tcpdump -i sysdig:
tcpdump: WARNING: SIOCGIFADDR: sysdig: No such device
Warning: Kernel filter failed: Socket operation on non-socket
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on sysdig, link-type EN10MB (Ethernet), capture size 8192 bytes
20:31:09.596548771 90:13:ae:3a:00:00 (oui Unknown) > a3:de:c4:b0:77:39 (oui Unknown) Null Information, send seq 16, rcv seq 0, Flags [Command], length 18
20:31:09.596554873 90:13:ae:3a:00:00 (oui Unknown) > 79:f6:c4:b0:77:39 (oui Unknown) Null Information, send seq 16, rcv seq 0, Flags [Command], length 18
20:31:09.596556288 90:13:ae:3a:00:00 (oui Unknown) > 00:fc:c4:b0:77:39 (oui Unknown) Null Information, send seq 16, rcv seq 0, Flags [Command], length 18
20:31:09.596556954 90:13:ae:3a:00:00 (oui Unknown) > 9a:fe:c4:b0:77:39 (oui Unknown) Null Information, send seq 37, rcv seq 0, Flags [Command], length 60
20:31:09.596560287 90:13:ae:3a:00:00 (oui Unknown) > 9f:0b:c5:b0:77:39 (oui Unknown) Null Information, send seq 25, rcv seq 0, Flags [Command], length 36
20:31:09.596578107 90:13:ae:3a:00:00 (oui Unknown) > 3b:51:c5:b0:77:39 (oui Unknown) Null Information, send seq 11, rcv seq 0, Flags [Command], length 8
As you have seen sysdig is now a virtual network interface (-i sysdig:) and all the rest is the same. Nice isn’t it?
Well there’s much more than that. In particular you can have fun when using sysdig from PF_RING ZC (NOTE: you do not need a ZC license for using it in combination with sysdig). Some examples (you can read all details in this file):
- Example 1. Hash incoming sysdig events and read them on 2 threads balancing them per PID
PF_RING/userland/examples_zc# ./zbalance -i sysdig: -c 4 -m 0 -r 1 -g 2:3
- Example 2. Hash incoming packets and read them on 2 processes
PF_RING/userland/examples_zc# ./zbalance_ipc -i sysdig: -c 99 -n 2 -m 0 -g 1 PF_RING/userland/examples_zc# ./zcount_ipc -c 99 -i 0 -g 2 -s PF_RING/userland/examples_zc# ./zcount_ipc -c 99 -i 1 -g 3 -s
- Example 3. Hash incoming packets and read them on 2 non-ZC applications
PF_RING/userland/examples_zc# ./zbalance_ipc -i zc:eth2 -c 99 -n 2 -m 0 -g 1 PF_RING/userland/examples# ./pfcount -i zc:99@0 -v 1 -q PF_RING/userland/examples# ./pfcount -i zc:99@1 -v 1 -q
- Example 4. Enqueue incoming sysdig events to a pipeline with 2 threads
PF_RING/userland/examples_zc# ./zpipeline -i sysdig: -c 99 -g 2:3
- Example 5. Enqueue incoming sysdig events to a queue, on another process forward packets from the queue to another queue, send packets from the second queue to an egress interface (perhaps we should first encapsulate the events into a ethernet frame for best results)
PF_RING/userland/examples_zc# ./zpipeline_ipc -i sysdig:,0 -o zc:eth3,1 -n 2 -c 99 -r 1 -t 2 PF_RING/userland/examples_zc# ./zbounce_ipc -c 99 -i 0 -o 1 -g 3
Note that the zbounce_ipc application can run on a VM, and a pipeline with multiple VMs can be created allocating more queues.
In essence using sysdig over PF_RING:
- You do not have anymore the limitation of running 1 (quantity one) sysdig-based application simultaneously.
- You can run your apps on a physical system or on a VM. For instance from inside a VM you can read the events coming from the physical system that hosts the VM. In zero copy at high speed.
- You can use n2disk to write to disk at high speed (being us able to written 20Gbit to disk with n2disk, is also possible to do the same with sysdig), handling system events similar to packets.
- You can read system events and packets simultaneously (e.g. have a look at apps like pfcount_bundle that can read packets from n-devices simultaneously and merge them) using one single API, from your existing app. The only difference is that you have to be able to interpret system events properly similar to what pfcount is doing on the example above.
- All this at line rate, in zero-copy, from a physical host or a VM. Free of charge (no license needed) using PF_RING ZC.
You can read more on this topic on the PF_RING wiki sysdig page.
We hope we can foster the development of sysdig-based applications thanks to our work. Enjoy!