December 14, 2015

rsyslogd dynamically create log file based on msg content

This is tested on a Ubuntu 10.04 system (should work on newer Ubuntu/Debian system too)

1. Create file /etc/rsyslog.d/30test.conf, with the following content:

$template DynFile,"/tmp/test-%msg:7:18%.log"
:msg,startswith," ABCD-001122334455" ?DynFile
#:syslogtag,startswith,"test" ?DynFile
#:syslogtag,startswith,"test" /tmp/test.log

2. Open file /etc/rsyslog.conf, and make the following modification for the following line:
$FileCreateMode 0644
comment out the following lines:
#$PrivDropToUser syslog
#$PrivDropToGroup syslog

3. sudo service rsyslog restart
4. logger -t test "ABCD-001122334455sdfsfsdfsdfsdfsdfs-----------"
5. Now check file /tmp/test-*.log, and you should see file /tmp/test-001122334455.log

To Log message with a defined template
1. First define the template
      $template shortlog,"%msg:10:1000%"
2. Use the template by appending ";template-name" to the end of the output file


a. Property msg starts with a space " ". 
b. Eventually we should try to figure out why dropPriviledge does causes issues here.


A good debugs tip:
1. add /etc/rsyslog.d/debug.conf
*.* /var/log/all.log;RSYSLOG_DebugFormat
#Note here ";RSYSLOG_DebugFormat" is the output template
2. restart rsyslog
3. watch the file /var/log/all.log to see all log messages with property names

Log with UNIX Timestamp
$template unixTS,"%timegenerated:::date-unixtimestamp%,%msg%\n"
:msg,contains,"[UFW " /var/log/ufw.log;unixTS

#Note here ";unixTS" is the output template
Also refer to the rsyslog PDF version of the manual (Chapter 1) for overview of how rsyslogd processes messages)

Syntax Check
rsyslogd -N1

===============UPDATE: New Syntax for Syslog version 7 and newer =============
rsyslog's new syntax is called "RainerScript". To generate dynamic files using output templates, generate a file in /etc/rsyslogd.conf/10-mytest.conf with the following content, and then restart service rsyslogd. The following example only logs message from the application "abc" and the message itself starts with "ABC" to a dynamic file name.

template(name="myfilename" type="string" string="/tmp/my-%programname%.log")
template(name="shortlog" type="string" string="%msg:10:$%\n")
if  $programname == 'abc' and $msg startswith " ABC"  then {
    action(type="omfile" dynaFile="myfilename" template="shortlog" )


=============UPDATE: More examples==================
1. variables are set using "set" and they start with "$!" in Rainer Scripts
2. variables only work inside the if block
3. Inside template the variables are surrounded by %%. e.g. "%$!devid%"
4. "stop" has replaced the "~" to discard a particular message.

template(name="devid" type="string" string="%$!devid%")
if ($programname == 'charon') then {
    if ($msg contains "deleting IKE_SA")  then {
        set $!devid= re_extract($msg, "([A-Z0-9])+-([A-Z0-9])", 0, 0, "unknown");
        #action(type="omfile" file="/var/log/delete.log" template="devid" )
        action(type="omfwd" Target="" Port="15140" Protocol="tcp" template="devid" )
        #action(type="omprog" binary="/usr/bin/" template="devid")

:programname,isequal,"charon" stop

No comments:

Post a Comment