Basic syslog-ng Installation

This post shows a guideline for a basic installation of the open source syslog-ng daemon in order to store syslog messages from various devices in a separate file for each device.

I am using such an installation for my firewalls, routers, etc., to have an archive with all of its messages. Later on, I can grep through these logfiles and search for specific events. Of course it does not provide any built-in filter or correlation features – it is obviously not a SIEM. However, as a first step it’s better than nothing. ;)

Prerequisites

This tutorial relies on a blank Linux (server) installation such as shown here. I am using an Ubuntu server. I furthermore assume that the reader is aware of its devices that are capable of sending syslog messages. That is: I am only showing the syslog-ng installation and no further details on how to send syslog messages from various devices to the server.

Installation

The first step is to install the syslog-ng package. I am using an Ubuntu server:

[UPDATE] On a fresh 64-bit Ubuntu 14.04.2 LTS, I got an error while trying to install syslog-ng. The following answer found in the Internet works:

[/UPDATE]

The default configuration file is /etc/syslog-ng/syslog-ng.conf. On Ubuntu, it has already a few lines that generate logfiles at /var/log/, e.g., the basic logfile “syslog”, which can be tailed with tail -f /var/log/syslog to see incoming messages from the system itself.

Configuration

I will now show the basic configuration of syslog-ng in order to:

  1. have an own folder for each device with
  2. a new file every day, nested in folders for year and month.

This requires a “source”, “filter” and “destination” which are then bound together. For more detailed configuration commands, this wiki from archlinux gives many good examples.

1) Source

Since the last line in the “syslog-ng.conf” config file ( /etc/syslog-ng/syslog-ng.conf) is @include "/etc/syslog-ng/conf.d/", all configuration files in the folder “conf.d” will be processed, too. Therefore, I generated a new configuration file called “firewalls.conf” in that subfolder sudo nano /etc/syslog-ng/conf.d/firewalls.conf. It has the following lines in it:

(Note: Replace USERNAME and USERGROUP with the name and group of the account from which the logfiles should be wrote to the disk such as “root” and “root”.)

This “source s_udp” object is quite generic and simply listens on udp port 514 on both Internet Protocols (IPv6 and legacy IP) for incoming syslog messages. It must appear only once in the config file.

2a) Generic Destination

The simpliest way to generate a different folder for every device is to use the following destination (without a specific filter). This adds a folder with the “$HOST” attribute, which is most commonly the IP address of the logging device. With this few lines, no more filters/destinations are required to log from many devices. (Thanks to the comment from “T”, who ticked me to search for this solution):

or 2b) Specific Filter & Destination

Another way is to write a filter for incoming log messages in order to save them in a specific destination folder. Following is the template. The only lines to change are the two UPPER CASE variables without the $ sign (i.e., NAMEOFTHEFIREWALL and IPOFTHEFIREWALL):

That is:

  • the filter “f_NAMEOFTHEFIREWALL” filters upon the source IP address from the sending device,
  • the destination “d_NAMEOFTHEFIREWALL” is set to the hierarchical path,
  • and finally the “log” sequence takes any messages from the source and uses the filter to store into the destination path.

These few lines in the template can appear many times in the config file. (Remember: the source s_udp must appear only once.) So you can copy & paste it for every syslog device.

Final Restart

A restart of the syslog-ng daemon is required to have the just added configuration active:

After that, netstat -tulpen shows a few lines which reveal that the port 514 is listening on IPv6 and legacy IP:

That is, all devices are now logging into the syslog-ng server, in my case the /var/log/firewalls directory:

The paths are quite long but structured, e.g.:

Examples

This is how syslog messages from a Palo Alto firewall look like when changing some policy rules:

 

This are some Juniper ScreenOS logs during active Internet connections:

 

And this are some Cisco router log messages:

That’s it. ;)

Featured image: “Erika 9 typewriter” by dr. shordzi is licensed under CC BY-SA 2.0.

16 thoughts on “Basic syslog-ng Installation

  1. Where do you make the filter/destination file? Is it a couple of lines you added to the first .log file or is it a completely different one?
    An answer is much appreciated :)

    1. It does not matter how many configuration files you have. I am using a single one “firewalls.conf” under the “/etc/syslog-ng/conf.d/” folder. This is the important point! You must store the file under this directory!
      In this single “firewalls.conf” file I am using the filters and destinations etc. for a couple of devices.

      The *.log files are created by syslog-ng. You MUST NOT edit them. These are the read-only log files.

      Hope that helps?

  2. Is this really the most efficient way of doing this? I have 100 Firewalls, apache servers and who knows what else that I am going to point to this thing and I rather not have to make 300 destination/filter/Log rules when I could just do it possibly by a few for each source type (Firewall, apache, etc)

    Is there a better way to do that?

    1. Yeah, great questions. Good idea. I just searched for that and modified the blog post and added the “Generic Destination” section. That should fit for you. ;)

  3. Thanks, this is really useful, and work perfectly.

    somehow l have to create the the folder /var/log/$HOST manually, else l will get the following error,
    Jan 15 17:30:03 Test-virtual-machine syslog-ng[48022]: Error opening file for writing; filename=’/var/log/192.168.28.144/192.168.28.144-2016-01-14.log’, error=’No such file or directory (2)’

    This is the folder privilege for the /var/log folder,
    drwxrwxr-x 14 root syslog 4096 Jan 15 17:30 log

  4. Can you share script to send email upon receiving certain event in syslog message received

  5. Hi Johannes,
    I am using syslog-ng 3.0 to send logs to a SIEM I am using.
    Much as I can see the OS logs in the SIEM, I do not see any application logs.
    I want to discount syslog-ng as the fault, as I think the SIEM is failing to parse these logs.
    Have you configured a DB server to send logs to a SIEM, if you could help me with this.

    Thanks
    IBM

    1. Hi Isaac,
      I am sorry, I am not using syslog-ng in exact that manner as you are doing it. I am only using it as a syslog forwarder, e.g., to a Dell CTA. That is, I have another destination with some options which is used for that:
      destination d_DellCTA {
      tcp(“10.10.10.10” port(1470) flags(“threaded”) template(“<$PRI>$R_DATE $SOURCEIP $MSGHDR$MSG\n”) template_escape(no));

      Are you sure that your SIEM is the problem? Have you tested sending the syslogs directly to it? If your SIEM is the problem, what do you want to solve with syslog-ng?

  6. thanks very much for sharing this , it helped me so much so implent syslog-ng in syslog server .

  7. Hi Johannes,

    I tried this on ubuntu14.04 and it works fine, but with ubuntu16.04 it isn’t is there any changes thats required to be made with 16.04, kindly help.

    1. Hey captain,

      I just verified my guide on a “Ubuntu 16.04.2 LTS (GNU/Linux 4.4.0-79-generic x86_64)” and it works without any errors. To install syslog-ng, simply type “sudo apt-get install syslog-ng” <- works fine in my lab. So I am sorry, but I don't know what exactly your problems are.

  8. Good staff Johannes.
    everything is running as expected and works fine so far
    I’m using Ubuntu 14.04
    I’m looking to have a nice GUI with filtering associated with syslog-ng which obviously makes it fabulous tool to search and create nice graphs.
    have you came across this?

Leave a Reply

Your email address will not be published. Required fields are marked *