This is a report from one of our users from the field, who decided to use ntopng to monitor a large network. Many thanks to Bjorn for sharing this information with our community.
Our network
Jessa Ziekenhuis is one of the biggest, non-academic, hospitals in Belgium. Spread over 4 campuses, we manage 3 data centres and about 90 data racks. Combined, this leads us to over 6,000 connected (and active) hosts ranging from laptops, desktops, MRIs, ultrasounds,…
Challenges
With hundreds of different specialised (medical) applications, (medical) devices, it’s hard to debug certain issues that are caused by or (as many network engineers will concur) blamed on the network. When those issues occur before devices like firewalls with extensive logging we often had to resort to running Wireshark on the device, (which in many cases isn’t possible or allowed with medical grade devices) or doing port mirrors with our own laptop. With a small network team, debugging issues puts a great strain on available man hours.
The Set-up
Design-wise, we have the 90 access stacks, spread over 3 campuses, connected to a backbone switch/router per campus. The backbone switches/routers are connected to our central core switch. From there, the connections to internal firewalls, external firewalls, and the outside world are made.
We have dark fiber between the different campuses, so we were able to connect each backbone and the core directly to out ntopng machine. This gives us the possibility to do a full port mirror of all access switch uplinks and every uplink to the core.
For capturing traffic between clients on the same switch, we still have the possibility to do RSPAN to a specific capturing VLAN which we also capture on our ntopng server.
The Result
We now have full visibility of the traffic on our network, even more than with classic Netflow or sFlow collection. Every single packet going over the network is captured and made visible in an incredibly easy and performant way.
Possibilities are limitless:
- “Can you do a port mirror and capture of device x to server y?” OK, give me 3 clicks, and I’ll deliver you a pcap of 1-5-10 minutes without even coming from behind my desk.
- “Why is my server under such a heavy load?” Well, maybe because your low specced server is getting hammered with Oracle queries from 200 clients.
- “Why are we seeing traffic on ntopng that we’re not seeing in the firewall logs?” Because we forgot a logging setting on one of our policies
While writing this little piece, we’re using ntopng to determine what three devices on our network are (unknown). With just a few clicks we were able to see where they were communicating and could pin an application engineer on those devices. I saw some heavy traffic between different hypervisors, which some had some performance issue. We’re digging further into this now with ntopng combined with our firewall logs
Two days ago, we had issues with an application that the vendor blamed on network latency. While doing a simple ping request from the client to the server gives a bit of insight into this, being able to see how long the packets from the application itself took over the trip gives much more insight with the Packet Inter-Arrival Time.
For future projects I also see a big role in ntopng. We plan on micro-segmenting our network in the future, so having this much visibility is ideal to test, debug end follow up on firewall policies between the different segments (is traffic passing by the policies or not?). I also plan on using the DPI of ntopng together with our existing firewalls to find security risks on parts of the network where we don’t have firewalls in place. In the past you needed a firewall or security appliance for this, ntopng can be our security appliance for those parts of the network. With extensive behaviour checks, logging and alerting this can be an excellent asset for this.
So yes, with our experiences thus far (and we keep on exploring new possibilities every day), we could have only dreamt of having so much insight into what’s happening on our network. ntopng makes it possible.
Bjorn Scheepers
Network and Security Engineer
Jessa Ziekenhuis (Hasselt, Belgium)