syslog-ng | Any IT here? Help Me!

LXC with SELinux and NFS share result in kernel: SELinux: inode_doinit_use_xattr: getxattr returned 2 for dev=0:43 ino=

After staring a new LXC container, the syslog program (Syslog-ng) began to throw thousands of errors with this kind of message:

Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6977140995289226736
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6551465724643968476
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-5980833553552494142
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-8820947409424952637
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-8270463809263745561
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7923279144252216900
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6181977668994943343
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7585065875445167421
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7923279144252216900
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-5826517164673898101
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7585065875445167421
Dec  1 11:01:01 h3 rsyslogd[1147]: imjournal: 3871493 messages lost due to rate-limiting (20000 allowed within 600 seconds)

These messages were logged in thousands. The same time, the NFS statistics showed a strange peak of using getattr. Something was calling getattr thousands times per second. Despite there were no SELinux blocks in audit.log as the dmesg suggested the SELinux might be blamed.
The LXC container is an application container, which has mound bind directory from the host server. The very same directory is an local NFS share (using NFS-Ganesha) of a GlusterFS volume and the PHP files are situated there.

main menu — kernel SELinux inode_doinit_use_xattr getxattr returned 2 nfsstat getattr graph

So the LXC container reads the PHP files from this NFS share. There were no issues to access the files and the application LXC worked just fine.
The problem disappeared when the NFS share was remounted with SELinux permissions using the context word:

node3:/VOL1 /mnt/nfs/VOL1 nfs defaults,hard,noexec,nosuid,_netdev,fsc,noatime,context="system_u:object_r:httpd_sys_rw_content_t:s0" 0 0

All the files are of SELinux label httpd_sys_rw_content_t and after restarting the LXC container there were no SELinux lines in the dmesg and the syslog logs. The administrator should configure the right SELinux permissions to the LXC bound directories. More on why SELinux sometimes does not report on blocks in the audit.log here – Selinux permission denied and no log in audit.log.

Change time zone for syslog messages in syslog-ng

When sending syslog packets to a remote server the time-zone of the current server may lead to problems because the syslog-ng program sets the time-zone with offset number according to the GMT. The remote server, especially if not a syslog-ng one, may interpret the offset as an offset to the UTC (Coordinated Universal Time), which leads times with an hour into the future.

Note, this whole problem is because of the Daylight saving time and there are almost 6 months when the GMT is not equal to the UTC and it is an hour ahead. Probably it is not a good idea to offset according to the GMT, because of the Daylight saving time during the summer, but this is on the syslog-ng development side.
There is one option time-zone(“[time_zone_string]”), which allows to change the time-zone of the destination packets. To avoid misinterprets of the date and time in the message packets the best way is to use it with UTC, so the local system will do the all necessary to convert the local time to UTC properly.
The configuration below uses time-zone(“[time_zone_string]”) from the current local time to UTC, because the current local time zone is EEST (Eastern European Summer Time), which 3 hours ahead of UTC during the summer period and 2 during winter (Eastern European Time – EET).

Relay the local web server logs from the local UDP port to the remote server using reliable TCP connection and changing the local time to UTC of the packets.

#NGINX - web logs
source udp_local {
    network(ip(127.0.0.1) port(514) transport("udp") so_rcvbuf(67108864) log_fetch_limit(1000) max-connections(1000) log-iw-size(1000000));
};

filter filter_nginx_access_log { program(nginx); };
destination d_tcp_syslog {
  syslog("10.10.10.10"
         port(10514) transport("tcp") disk-buffer(mem-buf-length(10000) mem-buf-size(128M) disk-buf-size(1024M) reliable(yes) dir("/var/lib/syslog-ng"))
         time_zone("UTC")
  );
};
log { source(udp_local); filter(filter_nginx_access_log); destination(d_tcp_syslog); };
#NGINX-end

the time_zone(“UTC”) in destination rule will ensure the packets have proper time related to the local server’s time in UTC and the mistake of misinterpreted date time is unlikely on a remote server.
More articles with syslog-ng – https://ahelpme.com/tag/syslog-ng/.

Rename the hostname in syslog-ng

At present, the syslog-ng (syslog-ng Open Source Edition) does not support to set the hostname of the server in the configuration. syslog-ng uses DNS system to resolve the system name or if it is explicitly switch off (with couple of options) it will use the IP. Sometimes the DNS name of the system may be not valid, for example in containers, or just for better naming purposes it is useful to have just a simple option to set the system’s hostname in the syslog packets.

The easiest and best way to change the system’s hostname in the packets is to use substitution rules with rewrite. In fact, the rewrite rule may operate on soft parts of the macros like MESSAGE, PROGRAM, HOST or user defined macros. The syslog message format and its “fields” could be seen in the RFC5424. It’s worth adding it is possible to replace the whole macro or just part of it.
Here is the configuration to set the system’s hostname:

#substitution rule
rewrite my_host { set("my-server-name", value("HOST")); };

#use the rule before the destination!!!
log { source(src); rewrite(my_host); destination(messages);};

The substitution rule should be used before the destination rule to take effect.

Here is a more complex example to set the system’s hostname only to certain packets:
Keep on reading!

syslog – UDP local to syslog-ng and send remote. Forward syslog to remote server.

After writing an article for the rsyslog daemon about forwarding local UDP logging to a remote server using TCP – UDP local to rsyslog and send remote with TCP and compression this time going to use syslog-ng daemon for those who use it as default in their Linux distribution.
As mentioned in the previous article always use a non-blocking way of writing logs using UDP locally and then transfer (forward) the logs to the centralized log server(s). The example here transfers the web server’s access logs to a remote server. The web server is an Nginx web server.
The goal is to use

UDP for the client program (Nginx in the case) for non-blocking log writes.
TCP between our local machine and the remote syslog server – to be sure not to lose messages on bad connectivity.
local caching for our client machine – not to lose messages if the remote syslog is temporary unreachable.

The configuration and the commands are tested on CentOS 7, CentOS 8, Gentoo and Ubuntu 18 LTS. Check out UDP remote logging here – nginx remote logging to UDP rsyslog server (CentOS 7) to see how to build the server-side part – the syslog server accepting the syslog messages and writing them into files.

STEP 1) Listen for local UDP connections

Configuration file /etc/syslog-ng/syslog-ng.conf

source udp_local {
    network(ip(127.0.0.1) port(514) transport("udp") so_rcvbuf(67108864) log_fetch_limit(1000) max-connections(1000) log-iw-size(1000000));
};

Keep on reading!