LXC with SELinux and NFS share result in kernel: SELinux: inode_doinit_use_xattr: getxattr returned 2 for dev=0:43 ino=

After staring a new LXC container, the syslog program (Syslog-ng) began to throw thousands of errors with this kind of message:

Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6977140995289226736
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6551465724643968476
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-5980833553552494142
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-8820947409424952637
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-8270463809263745561
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7923279144252216900
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6181977668994943343
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7585065875445167421
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7923279144252216900
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-5826517164673898101
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7585065875445167421
Dec  1 11:01:01 h3 rsyslogd[1147]: imjournal: 3871493 messages lost due to rate-limiting (20000 allowed within 600 seconds)

These messages were logged in thousands. The same time, the NFS statistics showed a strange peak of using getattr. Something was calling getattr thousands times per second. Despite there were no SELinux blocks in audit.log as the dmesg suggested the SELinux might be blamed.
The LXC container is an application container, which has mound bind directory from the host server. The very same directory is an local NFS share (using NFS-Ganesha) of a GlusterFS volume and the PHP files are situated there.

main menu
kernel SELinux inode_doinit_use_xattr getxattr returned 2 nfsstat getattr graph

So the LXC container reads the PHP files from this NFS share. There were no issues to access the files and the application LXC worked just fine.
The problem disappeared when the NFS share was remounted with SELinux permissions using the context word:

node3:/VOL1 /mnt/nfs/VOL1 nfs defaults,hard,noexec,nosuid,_netdev,fsc,noatime,context="system_u:object_r:httpd_sys_rw_content_t:s0" 0 0

All the files are of SELinux label httpd_sys_rw_content_t and after restarting the LXC container there were no SELinux lines in the dmesg and the syslog logs. The administrator should configure the right SELinux permissions to the LXC bound directories. More on why SELinux sometimes does not report on blocks in the audit.log here – Selinux permission denied and no log in audit.log.

autorelabel failed to relabel the root filesystem on boot when enabling the selinux on CentOS 8

This article is of a kind – “/.autorelabel” file does not work, because the enable of SELINUX ended with unreachable server.

Enabling the SELINUX should be easy as

  1. just editing a text file – /etc/selinux/config to include
    SELINUX=enforcing
    
  2. relabel all (or at least the root) file systems by creating the “/.autorelabel” file.
  3. restarting the system. The boot process will detect the “/.autorelabel” file and relabel the file systems and then it will restart the system in the normal boot order.

But this time the relabeling did not happen as usual (It happened on CentOS 8, but probably could happen in any Linux distribution?). The server never got reachable again and on the screen, there were multiple errors – all of “Permission denied”!

It is better when enabling SELINUX to set “permissive” mode at first and relabel the root file system with “/.autorelabel” and then to enable “enforcing” mode of SELINUX.

Using “permissive” first for the relabel process guarantees you would have your server back after the process because the SELINUX rules are not enforced.
Here is the better procedure of enabling the SELINUX:

  1. just editing a text file – /etc/selinux/config to include
    SELINUX=permissive
    
  2. relabel all (or at least the root) file systems by creating the “/.autorelabel” file.
  3. restarting the system. The boot process will detect the “/.autorelabel” file and relabel the file systems and then it will restart the system in normal boot order.
  4. edit the /etc/selinux/config to enable “enforcing” mode
    SELINUX=enforcing
    
  5. Restart (it’s better) or just enable SELINUX enforcing live with:
    setenforce 1
    

Our screenshots log of the relabel failure process

SCREENSHOT 1) No autorelabel initiated on boot despite the presence of “/.autorelabel” file.

Multple “Permission Denied” errors and many reports from “audit” – the SELINUX log daemon. The host is unreachable – no network started. No logging is possible!

main menu
boot freezing

SCREENSHOT 2) A page up above the first screen – more “Permission Denied” errors.

main menu
boot freezing 2

SCREENSHOT 3) Second page up above the first screen – the SELINUX rules loaded successfully but no autorelabel process initiated.

main menu
boot freezing 3

A successful relabel process on boot

SCREENSHOT 1) Successful start of the relabel process.

We’ve changed the SELINUX mode to be “permissive” and everything is back to normal, the “/.autorelabel” file initiated the relabel on the next boot.

main menu
relabel process on boot initiated

SCREENSHOT 2) The relabeling of the file system is in progress.

There is a progress counter.

main menu
relabel process in progress

SCREENSHOT 3) The relabel process finished successfully and the reboot is initiated.

The next reboot the “/.autorelabel” file won’t exists and the system will boot normally.

main menu
relabel process success and last screen before reboot

Selinux permission denied and no log in audit.log

So you execute a script and get a “Permission denied” and you know you have enabled SELinux. OK to disable the selinux is not an option (and never will be), so the first thing to check is the audit log to see what is the error and what the selinux tools will offer to solve it.

But there are no entries in the audit log when you execute your script!

So you decide to temporarily disable the selinux to check if this permission denied issues is still caused by it with:

setenforce 0

And the script just executes fine no error! Then again you put back the Enforcing with:

setenforce 1
./myscript
Permission Denied

And NO added lines in audit.log (/var/log/audit/audit.log in our system!). Apparently the logging is just fine, because it got sometime entries, but when executing our script, which is just a simple:

 
find /mnt/storA/servers/webroots/

After some research it appeared that

not all AVC denials may be logged when SELinux denies access.

Too many applications and system libraries check for permissions, which might not use or even need after that and the logging could grow exponentially or be less informative for the real cause of a problem!
Keep on reading!

How to proper enable the selinux in a CentOS7 installed server

These days many dedicated servers are offered with automation installation of operating systems and it have never been so easy and fast to pay a server and to get it up in minutes! Yes, we are talking for dedicated machines not virtual servers or cloud ones, but many cloud ones are in the same situation, when the host uses full virtualization.
It is fast, you can choose from many different linux distros and the installation is unattended and happens immediately, but in most cases the selinux is disabled, because is more easy for the support, for the user, for the admin, and for the offered preinstalled software…If you do not have some strange software in most cases it is advisable to enable the selinux, because it is of great security enhancement for your server and for the software in general. It is very simple to enable selinux, but there is an additional step, which if you omit, the server will probably get unusable (probably you won’t be able even to ssh it or login). These steps are tested under CentOS7, but probably works in all other distros, which support selinux!
So here are the steps:

STEP 1) Enable selinux in configuration

Edit the configuration file in

/etc/selinux/config

SELINUX=enforcing

STEP 2) relabel the file system

When using the selinux, there are labels (extended attributes of the file system), which are additional layer of security. Every system comes with prebuild rules instructing what label is set in which file or directory, so when you enable the selinux you must relabel the entire file system (or at least the root partition, to be able to boot normally). This is done with just a line of code below:

touch /.autorelabel

STEP 3) reboot

To take effect the changes made above the server must be rebooted.

reboot

The init process will find the file from step 2 “/.autorelabel” and will initiate a proper relabel according the current selinux rules file, then the server will be rebooted automatically again, the relabel could take time and it depends on the number of files you have in your server, just keep patient.
After the second reboot (which is automatically after the relabeling)

4) Recommendations

USE SELINUX, do not disable it! In most cases it is really simple to configure it in minutes for the need of your special software and for the generic one bet it there are rules offered in the distro’s packet system.

4) Post install check

You can check if the selinux is enabled with

[root@srv ~]# getenforce 
Enforcing
[root@srv ~]#