syslog – UDP local to syslog-ng and send remote. Forward syslog to remote server.

After writing an article for the rsyslog daemon about forwarding local UDP logging to a remote server using TCP – UDP local to rsyslog and send remote with TCP and compression this time going to use syslog-ng daemon for those who use it as default in their Linux distribution.
As mentioned in the previous article always use a non-blocking way of writing logs using UDP locally and then transfer (forward) the logs to the centralized log server(s). The example here transfers the web server’s access logs to a remote server. The web server is an Nginx web server.
The goal is to use

• UDP for the client program (Nginx in the case) for non-blocking log writes.
• TCP between our local machine and the remote syslog server – to be sure not to lose messages on bad connectivity.
• local caching for our client machine – not to lose messages if the remote syslog is temporary unreachable.

The configuration and the commands are tested on CentOS 7, CentOS 8, Gentoo and Ubuntu 18 LTS. Check out UDP remote logging here – nginx remote logging to UDP rsyslog server (CentOS 7) to see how to build the server-side part – the syslog server accepting the syslog messages and writing them into files.

STEP 1) Listen for local UDP connections

Configuration file /etc/syslog-ng/syslog-ng.conf

source udp_local {
    network(ip(127.0.0.1) port(514) transport("udp") so_rcvbuf(67108864) log_fetch_limit(1000) max-connections(1000) log-iw-size(1000000));
};

STEP 2) Filter only the access logs to be forwarded.

Catch filter for the Nginx access logs. Configuration file /etc/syslog-ng/syslog-ng.conf:

filter filter_nginx_access_log { program(nginx); };

STEP 3) The destitionation server configuration with local caching.

Configuration file /etc/syslog-ng/syslog-ng.conf

destination d_tcp_syslog {
  syslog("100.100.100.100"
         port(10514)
         transport("tcp")
         disk-buffer(mem-buf-length(10000)
                     mem-buf-size(128M) disk-buf-size(1024M)
                     reliable(yes) dir("/var/lib/syslog-ng")
         )
  );
};

The remote server port is 10514, the transport used here is TCP to be sure no packages (aka access logs) will be lost during transmission to the remote server. The buffer size is 1G.

STEP 4) Forward the syslog packages to the remote server.

Configuration file /etc/syslog-ng/syslog-ng.conf

log { source(udp_local); filter(filter_nginx_access_log); destination(d_tcp_syslog); };

And here is the whole /etc/syslog-ng/syslog-ng.conf file:

@version: 3.17
#
# Syslog-ng default configuration file for Gentoo Linux

# https://bugs.gentoo.org/426814
@include "scl.conf"

options {
        threaded(yes);
        chain_hostnames(no);

        # The default action of syslog-ng is to log a STATS line
        # to the file every 10 minutes.  That's pretty ugly after a while.
        # Change it to every 12 hours so you get a nice daily update of
        # how many messages syslog-ng missed (0).
        stats_freq(43200);
        # The default action of syslog-ng is to log a MARK line
        # to the file every 20 minutes.  That's seems high for most
        # people so turn it down to once an hour.  Set it to zero
        # if you don't want the functionality at all.
        mark_freq(3600);
};

source udp_local {
    network(ip(127.0.0.1) port(514) transport("udp") so_rcvbuf(67108864) log_fetch_limit(1000) max-connections(1000) log-iw-size(1000000));
};

filter filter_nginx_access_log { program(nginx); };

destination d_tcp_syslog {
  syslog("100.100.100.100"
         port(10514)
         transport("tcp")
         disk-buffer(mem-buf-length(10000)
                     mem-buf-size(128M) disk-buf-size(1024M)
                     reliable(yes) dir("/var/lib/syslog-ng")
         )
  );
};
log { source(udp_local); filter(filter_nginx_access_log); destination(d_tcp_syslog); };


source src { system(); internal(); };

destination messages { file("/var/log/messages"); };

# By default messages are logged to tty12...
destination console_all { file("/dev/tty12"); };
# ...if you intend to use /dev/console for programs like xconsole
# you can comment out the destination line above that references /dev/tty12
# and uncomment the line below.
#destination console_all { file("/dev/console"); };

log { source(src); destination(messages); };
log { source(src); destination(console_all); };

STEP 5) Web server configuration to write access logs in the local UDP syslog-ng server.

The example here is with the Nginx web server. Just add or replace the access_log Nginx directive (for exmaple, in /etc/nginx/nginx.conf) with:

access_log      syslog:server=127.0.0.1:514,facility=local7,tag=nginx,severity=info main3;

Nginx will use non-blocking UDP syslog local server to write its access logs and then syslog-ng server will forward them to the remote syslog server, which may be any of the syslog available servers like syslog-ng, rsyslog and etc.