Cisco network traffic monitoring with NfSen/NfDump and NetFlow

A while ago I wrote a quick article on using NTOP to monitor network utilization by using cisco’s NetFlow functionality. This is an excellent tool, and it’s high level reporting is definitely a strength. However, I was after a more detailed report about what happened when. In a network, it is often necessary to keep historic data in order to answer questions like “what traffic was passing yesterday night?”

While ntop does a pretty good job of breaking the traffic down, I still wanted even more detail about what was passing through the network. If possible I wanted to see the individual NetFlow records taken during a specific time period. That’s when I cam across NfSen. While not as “pretty” in it’s GUI as Ntop is, it has an excellent system of being able to use wireshark-like filters to directly query the NetFlow database, giving unparallel granularity in what information you can extract from the database.

Both NetFlow analysers have their place because Ntop is definitely more suited for a high level, historic view of the netowrk, but for network forensics and accounting I think NfSen is better due to the filtering feature. Some admins even incorporate the two.

In this post i’ll dump my notes on installing the nfsen / nfdump system on a squeaky new centos system. There are some really good guides on the internet, which I refer to, but there were some missing / unclear steps

– The main guide that I followed was:
http://poweredbyapathy.com/nfsen-installation-guide/

– First, install apache:

yum install httpd

– In order to enable php:

yum install php53u-common php53u-gd php53u-mcrypt php53u-pear php53u-pecl-memcache php53u-mhash php53u-mysql php53u-xml

– In order to install the rrdtool, I followed:
http://www.squidworks.net/2009/10/how-to-installing-rrdtool-on-centos-5-using-yum/

– I also needed to install a packaged named byacc in order to successfully compile NfSen:

yum install byacc

– Note, when editing the nfsen.conf, can use IPs rather than rely on ports when defining sources. The default nfsen-dist.conf file shows this:

%sources = (
    ‘upstream1’    => { ‘port’    => ‘9995’, ‘col’ => ‘#0000ff’, ‘type’ => ‘netflow’ },
    ‘peer1’        => { ‘port’    => ‘9996’, ‘col’ => ‘#ff0000’ },
);

Basically each source uses a separate port (9995 and 9996). It may be inconvenient opening a port for every NetFlow node on the network, so you can actually use the same port, but define the peer’s IP address like so:

%sources = (
    ‘firewall1’       => { ‘port’ => ‘2055’, ‘IP’ => ‘10.91.1.4’, ‘col’ => ‘#ff0000’, ‘type’ => ‘netflow’ },

     ‘firewall2’       => { ‘port’ => ‘2055’, ‘IP’ => ‘10.91.1.3’, ‘col’ => ‘#0000ff’, ‘type’ => ‘netflow’ },

);

– Added the following to httpd.conf file to allow access to the NfSen GUI:

<Directory "/var/www/nfsen">

    Options Indexes FollowSymLinks
    AllowOverride None
    Order allow,deny
    Allow from all

</Directory>

– In order to ensure NfSen starts automatically on startup, I added

/data/nfsen/etc/nfsen start

to /etc/rc5.d/S99local

– Once up and running you should be able to see the GUI. A quick example of troubleshooting using the tool:

    * Looking at the “home” tab, the bits/s graph shows a spike in traffic at about midnight:

     * Wanting to investigate this in more detail, we can change the “Single Timeslot” to “Time Window” and highlight that spike using the graph:

     * This is where the NfSen filters come into their own. Now under the heading “Netflow Processing” we modify the options to see the “Stat TopN”, that is, the top 20 flows, ordered by the amount of data transferred to a particular destination IP. This should give us a rough idea of what probably caused the spike. I also added a filter to remove any of my cisco IPs:

     * This shows us a particular IP tops the list:

    * As a filter we now insert : “ip 10.91.2.3” and we change the options from “Stat TopN” to “List Flows” and select the “bi-directional” aggregate:

     * We now see which internal IP connected to which external IP, hence causing the spike: