Using NetFlow with nProbe for ntopng

This blog post is about using NetFlow for sending network traffic statistics to an nProbe collector which forwards the flows to the network analyzer ntopng. It refers to my blog post about installing ntopng on a Linux machine. I am sending the NetFlow packets from a Palo Alto Networks firewall.

My current ntopng installation uses a dedicated monitoring ethernet port (mirror port) in order to “see” everything that happens in that net. This has the major disadvantage that it only gets packets from directly connected layer 2 networks and vlans. NetFlow on the other hand can be used to send traffic statistics from different locations to a NetFlow flow collector, in this case to the tool nProbe. This single flow collector can receive flows from different subnets and routers/firewalls and even VPN tunnel interfaces, etc. However, it turned out that the “real-time” functionalities of NetFlow are limited since it only refreshes flows every few seconds/bytes, but does not give a real-time look at the network. It should be used only for statistics but not for real-time troubleshooting.

Some Pre Notes

I am using a Ubuntu 14.04.5 LTS (GNU/Linux 3.16.0-77-generic x86_64) server. At the time of writing, nProbe had version v.7.4.160802 while ntopng was in version v.2.4.160802. Furthermore note that nProbe requires a license.

For general information about NetFlow use Wikipedia or Cisco or RFC 3954. For the other tools, use the official web sites: nProbe and ntopng. The nProbe site offers a detailed documentation PDF. A similar tutorial for installing nProbe is this one.

Installation of nProbe

(Since I already showed how to install ntopng, I will only show how to use nProbe here.) The stable builds for nProbe and ntopng are listed here. That is, to install nProbe, I used the following commands:

Since I want to receive NetFlow packets and forward them to ntopng, nProbe must run in Collector Mode. That is, I am using the following configuration file:

with these entries:

Note the naming of the config file: “nprobe-none.conf“. This is mandatory due to the documentation of nProbe: “When nProbe is used in probe mode it is not bound to any interface as its job is to collect NetFlow from some other device. In this case the configuration file to be created is: nprobe-none.conf.” (To my mind, this is a spelling mistake because it should read “When nProbe is NOT used in probe mode…”. However, it is working.)

Furthermore, an empty “start” file is needed to tell the init process to use this configuration file:

After a start of the service with sudo service nprobe start , ntopng must be configured to use this nProbe instance. Open the configuration file:

and add the following interface (= localhost):

Finally, restart the ntopng process: sudo service ntopng restart .

A netstat view should indicate the listening 2055 UDP port for nProbe, the 5556 TCP port for the connection between nProbe and ntopng, as well as the common 3000 TCP port from the ntopng WebGUI:

Since all services are now configured within configuration files that are referenced in the init scripts, they are started automatically after a system reboot. Great.

Palo Alto NetFlow

I am using a Palo Alto Networks firewall (version 7.1.3) to send NetFlow statistics to the nProbe collector. (More information about NetFlow on Palo.) This is configured in the following way: Adding of a NetFlow Server Profile and referencing this profile on all needed Network Interfaces, such as:

I am using quite fast values for the Template Refresh Rate as well as the Active Timeout. On interfaces with huge amount of traffic other values are probably better.

A small tcpdump capture shows some samples of the NetFlow packets sent by the Palo Alto. The following Wireshark screenshots show a NetFlow template as well as a sample flow:

ntopng Usage

Now here is the usage within ntopng. Simply choose the tcp://127.0.0.1:5556 interface at the upper right side. All features of ntopng remain the same, such as using the Dashboard, the Flows or the Hosts pages. (Refer to my post to see some features.)

However, here comes the problem with NetFlow: It is NOT a real-time application that lets ntopng show every single flow and its bandwidth correctly. It can be used to see a rough view of all flows during the past few seconds, but not its actual throughput at the moment.

Refer to the following two dashboard screenshots from ntopng. The first shows the Realtime Top Application Traffic from the NetFlow probe, while the second one shows the same from the mirror port eth1. The 54 MBit/s peak in the first screenshot is not true at all. In fact, it was a constant download over a few minutes. Whereas the second screenshot from eth1 shows the correct real-time bandwidth usage.

Conclusion

nProbe for ntopng can be used quite easily. It is possible to receive flows from different locations which can be displayed in a single instance of ntopng. However, if the primary goal is to have a real-time look at the network, e.g., which hosts or flows are consuming bandwidth, this approach does not fit. NetFlow data must be used with statistical applications that can report traffic stats, but not with real-time analyzers such as ntopng.

Featured image: “Flow” by Kalle Gustafsson is licensed under CC BY 2.0.

13 thoughts on “Using NetFlow with nProbe for ntopng

  1. Hi,

    I have used this guide to setup nprobe and ntopng on a virtual machine with Ubuntu server with 2 virtual interfaces. one interface is the promisc interface that listens to netflow traffic coming from routers on the internet and the other interface is the local LAN interface where ntopng is listening on for traffic from nprobe. I see that there are packets coming in from the internet to nprobe and it does send it to ntopng, but ntopng only recognizes the traffic as UDP netflow traffic. So it does not analyze what’s in the netflow packets.. Any idea?

    Cheers,
    Roel

    1. Hi Roel,
      to my mind you have not set up the correct scenario. If you are using ntopng on the same machine on which you have the promisc interface then you do NOT need nProbe at all. Please use the other ntopng guide I have written for that.
      This guide here (with nProbe) is only needed if ntopng is NOT running on the same machine.

      1. Hi Johannes,

        Thanks for your quick reply!
        I tried that as well, but that results in the same behavior. The NTOP only sees netflow packets, but cannot analyze / extract the flows from it. So I send netflow traffic from a router on the internet that handles internet traffic to this NTOP server also on the internet. The NTOP only sees netflow packets, but not the traffic between the router and for example the website IPs the users behind the router are going to. It just shows from: router to NTOP = netflow..

          1. I think so, when I only use the NTOP with the local interface and send netflow traffic there, or when I use it with Nprobe, same behavior..
            I saw that you should use the interface in promisc mode, but how can I send netflow traffic to an interface which does not have an IPv4 address?

  2. Your first guide, was amazing, I think you get a LOT of traffic and thankful people for the write-up, but you are missing something in this guide. To run nprobe via the config file, you need to have another line:

    -g=/var/run/nprobe-none.pid

    (–pid-file doesn’t seem to work with the current init.d script on Ubuntu 14)

    I personally added the following as well:

    –daemon-mode

    I’m just getting into this, but here is my writeup so far, its still a WIP but things are progressing well!

    Thanks for the blog!

    https://www.freesoftwareservers.com/wiki/install-and-setup-ntopng-nprobe-collector-netflow-sql-db-ubuntu-14-04-12517425.html

    PS: I’m confused about you saying not to run nprobe and ntopng on the same machine, that seems logical to me. nprobe gets the NetFlow packets from the router via “2055” and sends them to ntopng via “5556”. I have notpng also checking out the network on eth0, but I don’t get as much info as I do from the NetFlow router since its an edge device.

  3. I have followed both of your blog posts and installed ntopng and nprobe on the same server.
    I am able to see the traffic on the interface eth0 (local interface) but probe interface tcp://127.0.0.1:5556 is not showing any traffic.

    I have tested using tcpdump and i am receiving flow from our firewall.

    Where could it have gone wrong. Please guide

    Praneeth K

    1. Hi Praneeth,

      uh, I am sorry but I cannot help you from afar. It sounds quite good if you followed my guide, did not run into errors, and even receive NetFlow data on the interface… I don’t know. Maybe you can try the installation steps again on another computer?

  4. Hello,
    I followed this guide and have netprobe running with active conf:

    –zmq=”tcp://*:5556″ –collector-port=2055 -n=none -i=none -V=9 -g=/var/run/nprobe-none.pid

    and my ntop conf is:

    –pid-path=/var/tmp/ntopng.pid
    –daemon
    –interface=eth0
    –interface=”tcp://127.0.0.1:5556″
    –http-port=3000
    –local-networks=”10.0.0.0/8,192.168.0.0/16,2001:db8::/48″
    –dns-mode=1
    –data-dir=/var/tmp/ntopng
    –disable-autologout
    –community

    Most things seem to work but I get this nagging warning on the traffic dashboard:
    “Warning: There are no talkers for the current host.”

    And the Flows page says: “No Results Found”, but it clearly is getting some flow data since my sent data is broken up by address.

    Also I’m having a hard time viewing the details of the received Comcast connection, which just shows all the traffic aggregated.

    Any thoughts of what could be wrong?

    1. Edit, I should add that at the bottom right the little computer icon has over 100 (status mode remote) and the flows also shows close to that number. I’m just puzzled why I can’t see anything when I click on it.

Leave a Reply

Your email address will not be published. Required fields are marked *