Basic syslog-ng Installation

This post shows a guideline for a basic installation of the open source syslog-ng daemon in order to store syslog messages from various devices in a separate file for each device.

I am using such an installation for my routers, firewalls, etc., to have an archive with all of its messages. Later on, I can grep through these logfiles and search for specific events. Of course it does not provide any built-in filter or correlation features – it is obviously not a SIEM. However, as a first step, I think it’s better than nothing. 😉

Prerequisites

This tutorial relies on a blank Linux (server) installation such as shown here. I am using a Ubuntu server. I furthermore assume that the reader is aware of its devices that are capable of sending syslog messages. That is: I am only showing the syslog-ng installation and no further details on how to send syslog messages from various devices to the server.

Installation

The first step is to install the syslog-ng package. I am using an Ubuntu server:

[UPDATE] On a fresh 64-bit Ubuntu 14.04.2 LTS, I got an error while trying to install syslog-ng. The following answer found in the Internet works:

[/UPDATE]

The default configuration file is /etc/syslog-ng/syslog-ng.conf . On Ubuntu, it has already a few lines that generate logfiles under /var/log/ , e.g., the basic logfile “syslog”, which can be tailed with tail -f /var/log/syslog  to see incoming messages from the system itself.

Configuration

I will now show the basic configuration of syslog-ng along with a template for devices in order to:

  1. have an own folder for each device with
  2. a new file every day, nested in folders for year and month.

This requires to have a “source”, “filter” and “destination” which are then bound together. For more detailed configuration commands, this wiki from archlinux gives many good examples.

1) Source

Since the last lines in the “syslog-ng.conf” file ( /etc/syslog-ng/syslog-ng.conf ) end with @include "/etc/syslog-ng/conf.d/" , all configuration files in the folder “conf.d” will be processed, too. Therefore, I generated a new configuration file called “firewalls.conf” in that subfolder sudo nano /etc/syslog-ng/conf.d/firewalls.conf . It has the following lines in it:

(Note: Replace USERNAME and USERGROUP with the name and group of the account from which the logfiles should be wrote to the disk.)

This “source s_udp” object is quite general and simply listens on udp port 514 for incoming syslog messages. It must appear only once in the config file.

2a) Specific Filter & Destination

Now it’s time for the template. The only thing to change is the two UPPER CASE variables in the following lines without the $ sign (i.e., NAMEOFTHEFIREWALL and IPOFTHEFIREWALL):

That is:

  • the filter “f_NAMEOFTHEFIREWALL” filters upon the source IP address from the sending device,
  • the destination “d_NAMEOFTHEFIREWALL” is set to the hierarchical path,
  • and finally, the “log” sequence takes any messages from the source and uses the filter to store into the destination path.

These few lines in the template can appear many times in the config file. (Remember: the source s_udp must appear only once.) So you can copy & paste it for every syslog device.

or 2b) Generic Destination

Another (simplier) way to generate a different folder for every device is to use the following destination (without a specific filter). This adds a folder with the “$HOST” attribute, which is most commonly the IP address of the logging device. With this few lines, no more filters/destinations are required to log from many devices. (Thanks to the comment from “T” beneath, who ticked me to search for this solution):

 

Finally Restart

A restart of the syslog-ng daemon is required to have the just added configuration active:

After that, netstat -l  should show a line similar to the following one which reveals that the port 514 is listening:

Now, after adding all my devices to the config file, they are all logging into the syslog-ng server. The paths are quite long but structured, e.g.:

Examples

This is how syslog messages from my Palo Alto firewall look like when changing some policy rules:

 

And here are some Juniper ScreenOS logs during active Internet connections:

 

That’s it. 😉

10 thoughts on “Basic syslog-ng Installation

  1. Where do you make the filter/destination file? Is it a couple of lines you added to the first .log file or is it a completely different one?
    An answer is much appreciated 🙂

    1. It does not matter how many configuration files you have. I am using a single one “firewalls.conf” under the “/etc/syslog-ng/conf.d/” folder. This is the important point! You must store the file under this directory!
      In this single “firewalls.conf” file I am using the filters and destinations etc. for a couple of devices.

      The *.log files are created by syslog-ng. You MUST NOT edit them. These are the read-only log files.

      Hope that helps?

  2. Is this really the most efficient way of doing this? I have 100 Firewalls, apache servers and who knows what else that I am going to point to this thing and I rather not have to make 300 destination/filter/Log rules when I could just do it possibly by a few for each source type (Firewall, apache, etc)

    Is there a better way to do that?

    1. Yeah, great questions. Good idea. I just searched for that and modified the blog post and added the “Generic Destination” section. That should fit for you. 😉

  3. Thanks, this is really useful, and work perfectly.

    somehow l have to create the the folder /var/log/$HOST manually, else l will get the following error,
    Jan 15 17:30:03 Test-virtual-machine syslog-ng[48022]: Error opening file for writing; filename=’/var/log/192.168.28.144/192.168.28.144-2016-01-14.log’, error=’No such file or directory (2)’

    This is the folder privilege for the /var/log folder,
    drwxrwxr-x 14 root syslog 4096 Jan 15 17:30 log

  4. Can you share script to send email upon receiving certain event in syslog message received

  5. Hi Johannes,
    I am using syslog-ng 3.0 to send logs to a SIEM I am using.
    Much as I can see the OS logs in the SIEM, I do not see any application logs.
    I want to discount syslog-ng as the fault, as I think the SIEM is failing to parse these logs.
    Have you configured a DB server to send logs to a SIEM, if you could help me with this.

    Thanks
    IBM

    1. Hi Isaac,
      I am sorry, I am not using syslog-ng in exact that manner as you are doing it. I am only using it as a syslog forwarder, e.g., to a Dell CTA. That is, I have another destination with some options which is used for that:
      destination d_DellCTA {
      tcp(“10.10.10.10” port(1470) flags(“threaded”) template(“< $PRI>$R_DATE $SOURCEIP $MSGHDR$MSG\n”) template_escape(no));

      Are you sure that your SIEM is the problem? Have you tested sending the syslogs directly to it? If your SIEM is the problem, what do you want to solve with syslog-ng?

Leave a Reply

Your email address will not be published. Required fields are marked *