syslog – UDP local to rsyslog and send remote with TCP and compression

This article is to show how to log Nginx’s access logs locally using UDP to the local rsyslog daemon, which will send the logs to a remote rsyslog server using TCP and compression. In general, logs could generate a lot of traffic and using UDP over distant locations could result in packet loss respectively logs’ lines loss. The idea here is to log messages locally using UDP (non-blocking way) to a local Syslog server, which will send the stream to a remote central Syslog server using TCP connections to be sure no packets are lost. In addition, we are going to enable local caching (if the remote server is temporary unreachable) and compression between the two Syslog servers.
Our goal is to use

  • UDP for our client program (Nginx in the case) for non-blocking log writes.
  • TCP between our local machine and the remote syslog server – to be sure not to lose messages on bad connectivity.
  • local caching for our client machine – not to lose messages if the remote syslog is temporary unreachable.
  • compression between the local machine and the remote syslog server.

The configuration and the commands are tested on CentOS 7, CentOS 8 and Ubuntu 18 LTS. Check out UDP remote logging here – nginx remote logging to UDP rsyslog server (CentOS 7).

STEP 1) Configure client’s local rsyslog to accept UDP log messages only if the messages’ tags are “nginx”

Couple of things should be enabled in the local client-size rsyslog daemon:

  • rsyslog to accept UDP messages. Uncomment or add the following under section “Modules” (probably the first section in the file?) in /etc/rsyslog.conf
    $ModLoad imudp
    $UDPServerRun 514
    

    or

    module(load="imudp")
    input(type="imudp" port="514")
    

    The first is the old syntax, which is still supported and the second is the new syntax. For simplicity, all of the following configuration will be using the new syntax, because the old one is depricated.

  • Add a rule to catch the tag containing “nginx” and execute action to forward the messages to the remote server
    if ($syslogtag == 'nginx:') then {
    action(type="omfwd" target="10.10.10.10" port="10514" protocol="tcp" compression.Mode="single" ZipLevel="9"
    queue.filename="forwarding" queue.spoolDirectory="/var/log" queue.size="1000000" queue.type="LinkedList" queue.maxFileSize="1g" queue.SaveOnShutdown="on"
    action.resumeRetryCount="-1")
    & stop
    }
    
  • The options are almost self-explanatory, the important ones are there is no retry limit count of reconnecting to the server, there is in-disk cache of maximum 1G if the remote server is unavailable and the compression per message is turned on. More on actionshttps://www.rsyslog.com/doc/v8-stable/configuration/actions.html, the forward modulehttps://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and the queuehttps://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html

And restart the rsyslog:

systemctl restart rsyslog

STEP 2) Configure the server to accept the messages and write them in separate files.

We are going to use the IP and the date to create the path and the file name of the log files in the rsyslog server. The files should reside on big storage (multiple servers may produce a lot of logs files), so the big storage is mounted in /mnt/storage and we add “logs” sub-directory. Here is the file template – “/mnt/storage/logs/%FROMHOST-IP%/%$YEAR%.%$MONTH%.%$DAY%.log

  1. Enable the TCP syslog reception. In modules section of /etc/rsyslog.conf add or uncomment the following lines
    module(load="imtcp")
    input(type="imtcp" port="10514")
    

    The module name is “imtcp“.

  2. Add a rule to catch the messages from all remote servers. In the rules section of /etc/rsyslog.conf add the following lines at the beggining:
    template (name="from-ip-time" type="string" string="/mnt/storage/logs/%FROMHOST-IP%/%$YEAR%.%$MONTH%.%$DAY%.log")
    if ($fromhost-ip != "127.0.0.1") then {
    action(type="omfile" dirCreateMode="0700" FileCreateMode="0644" dynaFile="from-ip-time")
    & stop
    }
    

    Couple of rsyslog built-in variables are used fromhost-ip, year, month, day. More on the subject – https://www.rsyslog.com/doc/master/configuration/properties.html.
    Create a template on how should the path and the name of the logs look like and then catch the proper messages (those from the remote servers) and execute an action to save the data. Here it is used the omfile and the dynamic path is configured with dynaFile, which takes the template name defined above.

And restart the rsyslog server:

systemctl restart rsyslog

STEP 3) Permissions and firewall.

If SELinux is used and the path to save the file on the server (it’s the same for the clients, but we use the /var/log for the cache on purpose) the machine should be configured accordingly. In the current case, the SELinux is in enforcing mode so rsyslog to be able to write in “/mnt/storage/logs” execute:

[root@logs ~]# semanage fcontext -a -t var_log_t '/mnt/storage/logs(/.*)?'
[root@logs ~]# restorecon -Rv /mnt/storage/logs/
Relabeled /mnt/storage/logs from unconfined_u:object_r:mnt_t:s0 to unconfined_u:object_r:var_log_t:s0

The restorecon is only needed if the directory is created before adding SELinux files context for it (i.e. command with semanage fcontext).

The firewall configuration is the second option, which should be tuned because it is a security thread to expose the rsyslog server without controlling, which could connect to the server. In simple words, just add a rule to accept the incoming TCP connections on port 10514 for all the remote servers’ IPs.
Here is what to do for firewalld:

[root@logs ~]# firewall-cmd --permanent --zone=public --add-rich-rule="rule family="ipv4" source address="10.10.10.200" port protocol="tcp" port="10514" accept"
success
[root@logs ~]# firewall-cmd --reload
success

Replace the value of “source address” with the IP or sub-network of your remote servers.
Using iptables the rule will be (this rule will be inserted on the top of all rules in INPUT chain):

iptables -I INPUT -s 10.10.10.200 -m tcp -p tcp --dport 10514 -j ACCEPT

STEP 4) Configure the Nginx to send access logs locally.

The Nginx will use UDP to send locally the access log entries (so it will be non-blocking) and then the local rsyslog will use TCP to send the messages to the remote gathering log server.
In Nginx configuration is enough to just include another access_log directive (multiple access_log are allowed in a single block) or replace the current one for the local files:

access_log      syslog:server=127.0.0.1:514,facility=local7,tag=nginx,severity=info main;

and reload the nginx:

systemctl reload nginx

Bonus – check the syntax of /etc/rsyslog.conf

[root@logs ~]# rsyslogd -N1
rsyslogd: version 8.24.0-41.el7_7.2, config validation run (level 1), master config /etc/rsyslog.conf
rsyslogd: End of config validation run. Bye.

Bonus – storage directory

Using the above template will generate files in:

[root@logs ~]# find /mnt/storage/logs/
/mnt/storage/logs/
/mnt/storage/logs/10.10.10.200
/mnt/storage/logs/10.10.10.200/2020.02.20.log
/mnt/storage/logs/10.10.10.201
/mnt/storage/logs/10.10.10.201/2020.02.20.log
/mnt/storage/logs/10.10.10.202
/mnt/storage/logs/10.10.10.202/2020.02.20.log

Leave a Reply

Your email address will not be published. Required fields are marked *