Using NetFlow with nProbe for ntopng

This blog post is about using NetFlow for sending network traffic statistics to an nProbe collector which forwards the flows to the network analyzer ntopng. It refers to my blog post about installing ntopng on a Linux machine. I am sending the NetFlow packets from a Palo Alto Networks firewall.

My current ntopng installation uses a dedicated monitoring ethernet port (mirror port) in order to “see” everything that happens in that net. This has the major disadvantage that it only gets packets from directly connected layer 2 networks and vlans. NetFlow on the other hand can be used to send traffic statistics from different locations to a NetFlow flow collector, in this case to the tool nProbe. This single flow collector can receive flows from different subnets and routers/firewalls and even VPN tunnel interfaces, etc. However, it turned out that the “real-time” functionalities of NetFlow are limited since it only refreshes flows every few seconds/bytes, but does not give a real-time look at the network. It should be used only for statistics but not for real-time troubleshooting.

Some Pre Notes

I am using a Ubuntu 14.04.5 LTS (GNU/Linux 3.16.0-77-generic x86_64) server. At the time of writing, nProbe had version v.7.4.160802 while ntopng was in version v.2.4.160802. Furthermore note that nProbe requires a license.

For general information about NetFlow use Wikipedia or Cisco or RFC 3954. For the other tools, use the official web sites: nProbe and ntopng. The nProbe site offers a detailed documentation PDF. A similar tutorial for installing nProbe is this one.

Installation of nProbe

(Since I already showed how to install ntopng, I will only show how to use nProbe here.) The stable builds for nProbe and ntopng are listed here. That is, to install nProbe, I used the following commands:

Since I want to receive NetFlow packets and forward them to ntopng, nProbe must run in Collector Mode. That is, I am using the following configuration file:

with these entries:

Note the naming of the config file: “nprobe-none.conf“. This is mandatory due to the documentation of nProbe: “When nProbe is used in probe mode it is not bound to any interface as its job is to collect NetFlow from some other device. In this case the configuration file to be created is: nprobe-none.conf.” (To my mind, this is a spelling mistake because it should read “When nProbe is NOT used in probe mode…”. However, it is working.)

Furthermore, an empty “start” file is needed to tell the init process to use this configuration file:

After a start of the service with sudo service nprobe start , ntopng must be configured to use this nProbe instance. Open the configuration file:

and add the following interface (= localhost):

Finally, restart the ntopng process: sudo service ntopng restart .

A netstat view should indicate the listening 2055 UDP port for nProbe, the 5556 TCP port for the connection between nProbe and ntopng, as well as the common 3000 TCP port from the ntopng WebGUI:

Since all services are now configured within configuration files that are referenced in the init scripts, they are started automatically after a system reboot. Great.

Palo Alto NetFlow

I am using a Palo Alto Networks firewall (version 7.1.3) to send NetFlow statistics to the nProbe collector. (More information about NetFlow on Palo.) This is configured in the following way: Adding of a NetFlow Server Profile and referencing this profile on all needed Network Interfaces, such as:

I am using quite fast values for the Template Refresh Rate as well as the Active Timeout. On interfaces with huge amount of traffic other values are probably better.

A small tcpdump capture shows some samples of the NetFlow packets sent by the Palo Alto. The following Wireshark screenshots show a NetFlow template as well as a sample flow:

ntopng Usage

Now here is the usage within ntopng. Simply choose the tcp://127.0.0.1:5556 interface at the upper right side. All features of ntopng remain the same, such as using the Dashboard, the Flows or the Hosts pages. (Refer to my post to see some features.)

However, here comes the problem with NetFlow: It is NOT a real-time application that lets ntopng show every single flow and its bandwidth correctly. It can be used to see a rough view of all flows during the past few seconds, but not its actual throughput at the moment.

Refer to the following two dashboard screenshots from ntopng. The first shows the Realtime Top Application Traffic from the NetFlow probe, while the second one shows the same from the mirror port eth1. The 54 MBit/s peak in the first screenshot is not true at all. In fact, it was a constant download over a few minutes. Whereas the second screenshot from eth1 shows the correct real-time bandwidth usage.

Conclusion

nProbe for ntopng can be used quite easily. It is possible to receive flows from different locations which can be displayed in a single instance of ntopng. However, if the primary goal is to have a real-time look at the network, e.g., which hosts or flows are consuming bandwidth, this approach does not fit. NetFlow data must be used with statistical applications that can report traffic stats, but not with real-time analyzers such as ntopng.

Featured image: “Flow” by Kalle Gustafsson is licensed under CC BY 2.0.

18 thoughts on “Using NetFlow with nProbe for ntopng

  1. Hi,

    I have used this guide to setup nprobe and ntopng on a virtual machine with Ubuntu server with 2 virtual interfaces. one interface is the promisc interface that listens to netflow traffic coming from routers on the internet and the other interface is the local LAN interface where ntopng is listening on for traffic from nprobe. I see that there are packets coming in from the internet to nprobe and it does send it to ntopng, but ntopng only recognizes the traffic as UDP netflow traffic. So it does not analyze what’s in the netflow packets.. Any idea?

    Cheers,
    Roel

    1. Hi Roel,
      to my mind you have not set up the correct scenario. If you are using ntopng on the same machine on which you have the promisc interface then you do NOT need nProbe at all. Please use the other ntopng guide I have written for that.
      This guide here (with nProbe) is only needed if ntopng is NOT running on the same machine.

      1. Hi Johannes,

        Thanks for your quick reply!
        I tried that as well, but that results in the same behavior. The NTOP only sees netflow packets, but cannot analyze / extract the flows from it. So I send netflow traffic from a router on the internet that handles internet traffic to this NTOP server also on the internet. The NTOP only sees netflow packets, but not the traffic between the router and for example the website IPs the users behind the router are going to. It just shows from: router to NTOP = netflow..

          1. I think so, when I only use the NTOP with the local interface and send netflow traffic there, or when I use it with Nprobe, same behavior..
            I saw that you should use the interface in promisc mode, but how can I send netflow traffic to an interface which does not have an IPv4 address?

  2. Your first guide, was amazing, I think you get a LOT of traffic and thankful people for the write-up, but you are missing something in this guide. To run nprobe via the config file, you need to have another line:

    -g=/var/run/nprobe-none.pid

    (–pid-file doesn’t seem to work with the current init.d script on Ubuntu 14)

    I personally added the following as well:

    –daemon-mode

    I’m just getting into this, but here is my writeup so far, its still a WIP but things are progressing well!

    Thanks for the blog!

    https://www.freesoftwareservers.com/wiki/install-and-setup-ntopng-nprobe-collector-netflow-sql-db-ubuntu-14-04-12517425.html

    PS: I’m confused about you saying not to run nprobe and ntopng on the same machine, that seems logical to me. nprobe gets the NetFlow packets from the router via “2055” and sends them to ntopng via “5556”. I have notpng also checking out the network on eth0, but I don’t get as much info as I do from the NetFlow router since its an edge device.

  3. I have followed both of your blog posts and installed ntopng and nprobe on the same server.
    I am able to see the traffic on the interface eth0 (local interface) but probe interface tcp://127.0.0.1:5556 is not showing any traffic.

    I have tested using tcpdump and i am receiving flow from our firewall.

    Where could it have gone wrong. Please guide

    Praneeth K

    1. Hi Praneeth,

      uh, I am sorry but I cannot help you from afar. It sounds quite good if you followed my guide, did not run into errors, and even receive NetFlow data on the interface… I don’t know. Maybe you can try the installation steps again on another computer?

  4. Hello,
    I followed this guide and have netprobe running with active conf:

    –zmq=”tcp://*:5556″ –collector-port=2055 -n=none -i=none -V=9 -g=/var/run/nprobe-none.pid

    and my ntop conf is:

    –pid-path=/var/tmp/ntopng.pid
    –daemon
    –interface=eth0
    –interface=”tcp://127.0.0.1:5556″
    –http-port=3000
    –local-networks=”10.0.0.0/8,192.168.0.0/16,2001:db8::/48″
    –dns-mode=1
    –data-dir=/var/tmp/ntopng
    –disable-autologout
    –community

    Most things seem to work but I get this nagging warning on the traffic dashboard:
    “Warning: There are no talkers for the current host.”

    And the Flows page says: “No Results Found”, but it clearly is getting some flow data since my sent data is broken up by address.

    Also I’m having a hard time viewing the details of the received Comcast connection, which just shows all the traffic aggregated.

    Any thoughts of what could be wrong?

    1. Edit, I should add that at the bottom right the little computer icon has over 100 (status mode remote) and the flows also shows close to that number. I’m just puzzled why I can’t see anything when I click on it.

  5. I’ve tried the configuration. but on the Flows page, sometimes it appears and sometimes not results found. is there a solution?

  6. Hi There,
    I am running into the issue with nprobe and ntopng. I can’t see port 5556 listening on my server:
    [root@cactitemplate ~]# netstat -tulpen
    Active Internet connections (only servers)
    Proto Recv-Q Send-Q Local Address Foreign Address State User Inode PID/Program name
    tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN 0 7776481 4114/sshd
    tcp 0 0 0.0.0.0:3000 0.0.0.0:* LISTEN 0 9366763 14763/ntopng
    tcp 0 0 127.0.0.1:199 0.0.0.0:* LISTEN 0 9101003 12237/snmpd
    tcp 0 0 127.0.0.1:6379 0.0.0.0:* LISTEN 995 9177923 17069/redis-server
    tcp6 0 0 :::80 :::* LISTEN 0 9140869 6376/httpd
    tcp6 0 0 :::22 :::* LISTEN 0 7776483 4114/sshd
    tcp6 0 0 :::443 :::* LISTEN 0 9140879 6376/httpd
    tcp6 0 0 :::3306 :::* LISTEN 27 18880 1140/mysqld
    udp 0 0 10.10.10.230:49359 0.0.0.0:* 0 9366246 14763/ntopng
    udp 0 0 127.0.0.1:323 0.0.0.0:* 0 7790107 13753/chronyd
    udp 0 0 0.0.0.0:49942 0.0.0.0:* 1000 9394658 18894/spine
    udp 0 0 0.0.0.0:39803 0.0.0.0:* 0 9366154 14763/ntopng
    udp 0 0 0.0.0.0:42686 0.0.0.0:* 1000 9394521 18894/spine
    udp 0 0 0.0.0.0:2055 0.0.0.0:* 99 7771963 3389/nprobe
    udp 0 0 0.0.0.0:55307 0.0.0.0:* 0 9390257 18275/nprobe
    udp 0 0 0.0.0.0:161 0.0.0.0:* 0 9101002 12237/snmpd
    udp6 0 0 ::1:323 :::* 0 7790108 13753/chronyd
    udp6 0 0 :::2055 :::* 99 7771964 3389/nprobe

    but with ntopng runtime it states”Collecting Flow on tcp://10.10.10.230:5556
    [root@cactitemplate ntopng]# ntopng –interface=ens32 –interface=”tcp://10.10.10.230:5556″
    05/Jan/2018 12:21:32 [Ntop.cpp:1486] Setting local networks to 127.0.0.0/8
    05/Jan/2018 12:21:32 [Redis.cpp:115] Successfully connected to redis 127.0.0.1:6379@0
    05/Jan/2018 12:21:32 [Redis.cpp:115] Successfully connected to redis 127.0.0.1:6379@0
    05/Jan/2018 12:21:32 [NtopPro.cpp:221] [LICENSE] Reading license from Redis
    05/Jan/2018 12:21:32 [NtopPro.cpp:300] WARNING: [LICENSE] Invalid or missing license
    05/Jan/2018 12:21:32 [NtopPro.cpp:317] WARNING: [LICENSE] ntopng will now run in enterprise edition for 10 minutes
    05/Jan/2018 12:21:32 [NtopPro.cpp:319] WARNING: [LICENSE] before returning to community mode
    05/Jan/2018 12:21:32 [NtopPro.cpp:321] WARNING: [LICENSE] You can buy a permanent license at http://shop.ntop.org
    05/Jan/2018 12:21:32 [NtopPro.cpp:323] WARNING: [LICENSE] or run ntopng in community mode starting
    05/Jan/2018 12:21:32 [NtopPro.cpp:324] WARNING: [LICENSE] ntopng –community
    05/Jan/2018 12:21:36 [PcapInterface.cpp:88] Reading packets from interface ens32…
    05/Jan/2018 12:21:36 [Ntop.cpp:1613] Registered interface ens32 [id: 0]
    05/Jan/2018 12:21:36 [Ntop.cpp:1613] Registered interface tcp://10.10.10.230:5556 [id: 7]
    05/Jan/2018 12:21:36 [main.cpp:301] PID stored in file /var/run/ntopng.pid
    05/Jan/2018 12:21:37 [HTTPserver.cpp:841] Please read https://github.com/ntop/ntopng/blob/dev/doc/README.SSL if you want to enable SSL.
    05/Jan/2018 12:21:37 [Utils.cpp:469] User changed to nobody
    05/Jan/2018 12:21:37 [HTTPserver.cpp:912] Web server dirs [/usr/share/ntopng/httpdocs][/usr/share/ntopng/scripts]
    05/Jan/2018 12:21:37 [HTTPserver.cpp:915] HTTP server listening on port(s) 3000
    05/Jan/2018 12:21:37 [main.cpp:383] Working directory: /var/tmp/ntopng
    05/Jan/2018 12:21:37 [main.cpp:385] Scripts/HTML pages directory: /usr/share/ntopng
    05/Jan/2018 12:21:37 [Ntop.cpp:385] Welcome to ntopng x86_64 v.3.3.180105 – (C) 1998-17 ntop.org
    05/Jan/2018 12:21:37 [Ntop.cpp:395] Built on CentOS Linux release 7.4.1708 (Core)
    05/Jan/2018 12:21:37 [NtopPro.cpp:464] [LICENSE] System Id: 68A65FBA7A06AB23
    05/Jan/2018 12:21:37 [NtopPro.cpp:465] [LICENSE] Edition: Enterprise
    05/Jan/2018 12:21:37 [NtopPro.cpp:466] [LICENSE] License Type: Demo License
    05/Jan/2018 12:21:37 [NtopPro.cpp:475] [LICENSE] Validity: Until Fri Jan 5 12:31:32 2018
    05/Jan/2018 12:21:37 [Ntop.cpp:678] Adding 10.10.10.230/32 as IPv4 interface address for ens32
    05/Jan/2018 12:21:37 [Ntop.cpp:686] Adding 172.24.128.0/21 as IPv4 local network for ens32
    05/Jan/2018 12:21:37 [Ntop.cpp:705] Adding fe80::a653:1adc:9a0:3f48/128 as IPv6 interface address for ens32
    05/Jan/2018 12:21:37 [Ntop.cpp:714] Adding fe80::a653:1adc:9a0:3f48/64 as IPv6 local network for ens32
    05/Jan/2018 12:21:37 [PeriodicActivities.cpp:59] Started periodic activities loop…
    05/Jan/2018 12:21:37 [PeriodicActivities.cpp:104] Each periodic activity script will use 2 threads
    05/Jan/2018 12:21:37 [NetworkInterface.cpp:2326] Started packet polling on interface ens32 [id: 0]…
    05/Jan/2018 12:21:37 [NetworkInterface.cpp:2326] Started packet polling on interface tcp://10.10.10.230:5556 [id: 7]…
    05/Jan/2018 12:21:38 [CollectorInterface.cpp:122] Collecting flows on tcp://10.10.10.230:5556

    what I am missing

Leave a Reply

Your email address will not be published. Required fields are marked *