Upgrading sysklog to rsyslog

As a server setup gets more complex, it is possible that the default syslog (sysklogd) lacks the necessary finely grained filtering that may be desired. Common solutions to this are using syslog-ng or rsyslog. I have opted to go with the latter and found it to, so far, be an excellent replacement.

Installing rsyslog

Update: The latest version of Amazon’s Linux AMI uses rsyslog by default.
On Amazon’s Linux AMI, rsyslog (v4.6.2-1.6) is available in the amzn repository and can be installed via yum. At first glance, however, rsyslog appears to conflict with sysklog (specifically, the file /etc/logrotate.d/syslog). Attempting to uninstall sysklog is not an option, as it appears that pretty much everything depends on it). Renaming the conflicing file, of course, does not resolve the dependency.

The solution is to use yum shell to install rsyslog while simultaneously removing sysklog – the dependencies will be resolved without issue.

At the yum shell prompt, enter (wait for each command to execute before continuing):

1. install rsyslog
2. remove syslog
3. run
4. exit

The overall process should resemble the following:

# yum shell

Loaded plugins: fastestmirror, security
Setting up Yum Shell
> install rsyslog
Loading mirror speeds from cached hostfile
amzn                                                                        | 2.1 kB     00:00
Setting up Install Process
> remove sysklogd
Setting up Remove Process
> run
--> Running transaction check
---> Package rsyslog.i686 0:4.6.2-1.6.amzn1 set to be updated
---> Package sysklogd.i386 0:1.4.1-46.8.amzn1 set to be erased
--> Finished Dependency Resolution
 
===================================================================================================
Package              Arch             Version                         Repository             Size
===================================================================================================
Installing:
rsyslog              i686             4.6.2-1.6.amzn1                 amzn                  427 k
Removing:
sysklogd             i386             1.4.1-46.8.amzn1                installed             118 k
 
Transaction Summary
===================================================================================================
Install       1 Package(s)
Upgrade       0 Package(s)
Remove        1 Package(s)
Reinstall     0 Package(s)
Downgrade     0 Package(s)
 
Total size: 427 k
Is this ok [y/N]: y
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Installing     : rsyslog-4.6.2-1.6.amzn1.i686                                                1/2
Erasing        : sysklogd-1.4.1-46.8.amzn1.i386                                              2/2
warning: /etc/syslog.conf saved as /etc/syslog.conf.rpmsave
 
Removed:
sysklogd.i386 0:1.4.1-46.8.amzn1
 
Installed:
rsyslog.i686 0:4.6.2-1.6.amzn1
 
Finished Transaction
> exit
Leaving Shell

You should note that both the install and remove were processed together, and the old syslog.conf was backed up. (Additionally, the conflicting file /etc/logrotate.d/syslog is also backed up and a new (slightly different) one created).

Next steps

Rsyslogd is a drop in replacement for syslogd – it starts with the same default set of rules (any custom selectors you added to syslog.conf are not copied over). It should work without any setup on a system which doesn’t have custom syslog rules setup. For systems with custom rules, you will have to manually copy the rules to the new conf file (most should not need modification). By default, rsyslogd is not started, although it is setup to start at boot. To start rsyslogd, run:

service rsyslog start

You can view startup settings for rsyslog by running:

chkconfig --list rsyslog

Which should return something similar to the following (default):

rsyslog         0:off   1:off   2:on    3:on    4:on    5:on    6:off

rsyslogd does not use the same configuration file as sysklogd, although the format shares many similarities. The file used by rsyslogd is /etc/rsyslog.conf (while the one used by sysklogd was /etc/syslog.conf)

(It is, perhaps, worth noting, however, that (at least on Amazon’s Linux) rsyslogd’s pid file is /var/run/syslogd.pid not /var/run/rsyslogd.pid).

Suhosin Example

As a quick example of one of the available features of rsyslogd that is not found in sysklogd, consider the following scenario:

I have suhosin installed on my server, and some scripts trigger it – however, it sends messages at a debug level, and so do other scripts. This results in suhosin notifications being logged to /var/log/messages as well as a custom log file I had previously setup.

With rsyslog, it is possible to look for a particular part of the message and filter using that, as well as to prevent subsequent selectors from being triggered.

A sample suhosin notification looks like the following:

Feb 12 15:14:01 servername suhosin[30496]: ALERT - script tried to increase memory_limit to 268435456 bytes which is above the allowed value (attacker 'xx.xxx.xx.xxx', file '/path/to/file/test.php', line 10)

While the message may change depending on the event, the program name (suhosin) remains constant and can be used to filter the message.

To setup such a filter, add the following to /etc/rsyslog.conf, at the top of the rules section (before any other rules).

:programname, contains, "suhosin" /var/log/suhosin.log
& ~

This will check the ‘programname’ to see if it ‘contains’ the text ‘suhosin’, and if it does, will redirect the message to the specified log file. The next line, discards the message, so that no further rules are triggered

Oddity with SpamAssassin

After installing rsyslog, I noticed that SpamAssassin would throw the following warning when started:

warn: logger: failed to add syslog method: logger: syslog initialization failed

As such, sa-update, which restarts the service, would generate the same error (and cron would mail it out). To resolve this, modify /etc/sysconfig/spamassassin (Amazon’s Linux, RHEL/CentOS derived) appending --syslog-socket=native to SPAMDOPTIONS. The final result should resemble the following:

/etc/sysconfig/spamassassin:

SPAMDOPTIONS="-d -c -m5 -H --syslog-socket=native"

Process Dies

One of the problems I have found with this setup is that the process appears to die periodically as per the status below:

#service rsyslog status
rsyslogd dead but pid file exists

Further investigation into the problem reveals that when rsyslog receives the HUP signal, it will attempt a complete restart (however, it appears to not quite succeed in this case). The source of the HUP signal appears to be logrotate.

This result can be verified by running:

#kill -HUP `cat /var/run/syslogd.pid`
#service rsyslog status

The status message is the same as the one above.

From this state, rsyslog can be directly started (without even deleting the PID file) via:

#service rsyslog start

While an upgrade of rsyslog may resolve this issue (I haven’t tried), a simpler solution appears to be a minor modification to the configuration:

Edit rsyslog.conf, add the following at the end of the Global Directives section (just before the Rules section):

$HUPisRestart off

Sending the HUP signal no longer results in the termination of the process making rsyslog considerably more usable.

References

http://www.rsyslog.com/doc/property_replacer.html (list of available properties to filter on)

http://www.rsyslog.com/doc/rsyslog_conf_filter.html (configuration options for filtering)

http://blog.gerhards.net/2008/10/new-rsyslog-hup-processing.html (rsyslog HUP processing)