LXC with SELinux and NFS share result in kernel: SELinux: inode_doinit_use_xattr: getxattr returned 2 for dev=0:43 ino=

After staring a new LXC container, the syslog program (Syslog-ng) began to throw thousands of errors with this kind of message:

Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6977140995289226736
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6551465724643968476
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-5980833553552494142
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-8820947409424952637
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-8270463809263745561
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7923279144252216900
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6181977668994943343
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7585065875445167421
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7923279144252216900
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-5826517164673898101
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7585065875445167421
Dec  1 11:01:01 h3 rsyslogd[1147]: imjournal: 3871493 messages lost due to rate-limiting (20000 allowed within 600 seconds)

These messages were logged in thousands. The same time, the NFS statistics showed a strange peak of using getattr. Something was calling getattr thousands times per second. Despite there were no SELinux blocks in audit.log as the dmesg suggested the SELinux might be blamed.
The LXC container is an application container, which has mound bind directory from the host server. The very same directory is an local NFS share (using NFS-Ganesha) of a GlusterFS volume and the PHP files are situated there.

main menu
kernel SELinux inode_doinit_use_xattr getxattr returned 2 nfsstat getattr graph

So the LXC container reads the PHP files from this NFS share. There were no issues to access the files and the application LXC worked just fine.
The problem disappeared when the NFS share was remounted with SELinux permissions using the context word:

node3:/VOL1 /mnt/nfs/VOL1 nfs defaults,hard,noexec,nosuid,_netdev,fsc,noatime,context="system_u:object_r:httpd_sys_rw_content_t:s0" 0 0

All the files are of SELinux label httpd_sys_rw_content_t and after restarting the LXC container there were no SELinux lines in the dmesg and the syslog logs. The administrator should configure the right SELinux permissions to the LXC bound directories. More on why SELinux sometimes does not report on blocks in the audit.log here – Selinux permission denied and no log in audit.log.

Moving existing Elasticsearch and Kibana installation from CentOS 7 to CentOS Stream 9

main menu
install Elasticsearch and Kibana

Despite having only two additional installed software under CentOS 7 it is not a good idea to just try upgrading only CentOS 7 to CentOS Stream 9. There is no clear and supported path for upgrading from CentOS 7 to CentOS Stream 9 and even to the older one CentOS 8 (or CentOS Stream 8). The best way is to just make a clean install of CentOS Stream 9 and copy all the Elasticsearch and Kibana files and this article is how to do it without problems.
Here is the plan to move the existing installation of Elasticsearch and Kibana services from CentOS 7 to CentOS Stream 9:

  1. Make a clean install of CentOS Stream 9
  2. Update the current Elasticsearch and Kibana installations to their last versions (from their branch or minor versions).
  3. Add Elasticsearch and Kibana repositories to the new system. Tune the system crypto policies.
  4. Install Elasticsearch and Kibana software packages, but do not start the services.
  5. Copy Elasticsearch and Kibana important files such as the index directory and the configuration directories. Check the user and group IDs of the files.
  6. Start the Elasticsearch and Kibana services.

In this example, the installation of the new server is just starting a new LXC container, which will host the Elasticsearch and Kibana services. There is no difference between using a container or a physical machine. With LXC container it is easier to copy the needed files such as the Elasticsearch index files, which may be tens of terabytes or more, and various configuration files.

STEP 1) Make a clean install of CentOS Stream 9

Check out the following article on the purpose – Network installation of CentOS Stream 9 (20220606.0) – minimal server installation or if LXC container is preferred – Run LXC CentOS Stream 9 container with bridged network under CentOS Stream 9.

Creating a LXC container of CentOS Stream 9 is really simple and fast:

[root@srv ~]# lxc-create --template download -n kibana.u1x2.com -- --dist centos --release 9-Stream --arch amd64
The cached copy has expired, re-downloading...
Downloading the image index
Downloading the rootfs
Downloading the metadata
The image cache is now ready
Unpacking the rootfs

---
You just created a Centos 9-Stream x86_64 (20230511_19:27) container.

Then tune the network following the above article. It is a good idea when configuring the network to preserve the original UUIDs and network addresses (MAC address, too) of the LXC containers network and the inner container’s interface.
So copy the UUID from /var/lib/lxc/loganalyzer-old/rootfs/etc/sysconfig/network-scripts/ifcfg-eth0 to the CentOS Stream 9 network configuration – /var/lib/lxc/loganalyzer/rootfs/etc/NetworkManager/system-connections/ethernet-eth0.nmconnection, which uses NetworkManager. And the LXC container’s MAC address: the variable lxc.net.0.hwaddr from /var/lib/lxc/loganalyzer/config to /var/lib/lxc/loganalyzer/config.
The last step is to run the newly installed system. No errors in the output signals for a successful start-up of the LXC container with the name loganalyzer.

[root@srv ~]# lxc-start -n loganalyzer
[root@srv ~]# 

STEP 2) Upgrade the current Elasticsearch and Kibana installations to their last versions (from their branch or minor versions).

For example, if the current Elasticsearch is version 7. It is good to upgrade it to the latest version from 7.x before proceeding with the next steps.
The current installed versions of Elasticsearch and Kibana software are from the branch 77.17.4-1 and the latest version is 7.17.10-1.
Check in the old system with (CentOS 7):

[root@loganalyzer-old ~]# yum list installed|egrep -e "(elasticsearch|kibana)"
elasticsearch.x86_64               7.17.4-1                               @elasticsearch
kibana.x86_64                      7.17.4-1                               @elasticsearch

Keep on reading!

lxc_attach_run_shell: 1333 Permission denied – failed to exec shell

An annoying error when using the LXC container tools like lxc-attach, which is really simple to fix.

[root@srv ~]# lxc-attach -n db-cluster-3
lxc_container: attach.c: lxc_attach_run_shell: 1333 Permission denied - failed to exec shell
[root@srv ~]#

This error just reports the bash shell in the container cannot be started and the SELinux audit file adds some errors, too:

type=AVC msg=audit(1665745824.682:24229): avc:  denied  { entrypoint } for  pid=20646 comm="lxc-attach" path="/usr/bin/bash" dev="md3" ino=111806476 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=unconfined_u:object_r:var_log_t:s0 tclass=file
type=SYSCALL msg=audit(1665745824.682:24229): arch=c000003e syscall=59 success=no exit=-13 a0=24412c6 a1=7ffe87c07170 a2=2443870 a3=7ffe87c08c60 items=0 ppid=20644 pid=20646 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts12 ses=3304 comm="lxc-attach" exe="/usr/bin/lxc-attach" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=PROCTITLE msg=audit(1665745824.682:24229): proctitle=6C78632D617474616368002D6E0064622D636C75737465722D33
type=AVC msg=audit(1665745824.682:24230): avc:  denied  { entrypoint } for  pid=20646 comm="lxc-attach" path="/usr/bin/bash" dev="md3" ino=111806476 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=unconfined_u:object_r:var_log_t:s0 tclass=file
type=SYSCALL msg=audit(1665745824.682:24230): arch=c000003e syscall=59 success=no exit=-13 a0=7f08b5e579a0 a1=7ffe87c07170 a2=2443870 a3=7ffe87c08c60 items=0 ppid=20644 pid=20646 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts12 ses=3304 comm="lxc-attach" exe="/usr/bin/lxc-attach" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=PROCTITLE msg=audit(1665745824.682:24230): proctitle=6C78632D617474616368002D6E0064622D636C75737465722D33

So clearly, the problem is in SELinux, and turn it off temporarily with

setenforce 0

Turning off the SELinux is not the right thing! There are two aspects to the problem:

  • Missing SELinux rules, which are installed with a special package container-selinux
  • Wrong SELinux permissions for the LXC container’s root directory. In most cases, the user just changes the default /var/lib/lxc/[container] to something new and the LXC works, but it breaks some LXC parts.

Installing container-selinux is easy:

dnf install -y container-selinux

Or the old yum:

yum install -y container-selinux

Then check the SELinux attributes with:

[root@srv ~]# ls -altrZ /mnt/storage/servers/mycontainer/
drwxr-xr-x. root root unconfined_u:object_r:var_log_t:s0 ..
-rw-r--r--. root root unconfined_u:object_r:var_log_t:s0 config
drwxrwx---. root root unconfined_u:object_r:var_log_t:s0 .
drwxr-xr-x. root root unconfined_u:object_r:var_log_t:s0 rootfs

The problem is var_log_t, which is an SELinux file context and it should be container_var_lib_t. Stop the container and fix the permissions. If the default directory (/var/lib/lxc) were used, it would not have this problem. Adding the SELinux file context definition to the new directory is mandatory when changing the directory root of a container:

[root@srv ~]# semanage fcontext -a -t container_var_lib_t '/mnt/storage/servers/mycontainer(/.*)?'
[root@srv ~]# restorecon -Rv /mnt/storage/servers/mycontainer/
restorecon reset /mnt/storage/servers/mycontainer context unconfined_u:object_r:var_log_t:s0->unconfined_u:object_r:container_var_lib_t:s0
.....
.....
restorecon reset /mnt/storage/servers/mycontainer/config context unconfined_u:object_r:var_log_t:s0->unconfined_u:object_r:container_var_lib_t:s0

All files permissions under /mnt/storage/servers/mycontainer/ should be fixed with the restorecon. Start the LXC container and try to attach it with lxc-attach. Now, there should not be any errors:

[root@srv ~]# lxc-attach -n mycontainer
[root@mycontainer ~]#

The files’ context is the right one – container_var_lib_t:

[root@srv ~]# ls -altrZ /mnt/storage/servers/mycontainer/
drwxr-xr-x. root root unconfined_u:object_r:var_log_t:s0 ..
-rw-r--r--. root root unconfined_u:object_r:container_var_lib_t:s0 config
drwxrwx---. root root unconfined_u:object_r:container_var_lib_t:s0 .
drwxr-xr-x. root root unconfined_u:object_r:container_var_lib_t:s0 rootfs

More on LXC containershttps://ahelpme.com/category/software/lxc/.

Debug options for LXC and lxc-start when lxc container could not start

Setup and running LXC container is really easy, but sometimes it is unclear why the LXC container could not start. Most of the time, there is a generic error, which says nothing for the real reason:

root@srv ~ # lxc-start -n test-lxc
lxc-start: test-lxc: lxccontainer.c: wait_on_daemonized_start: 867 Received container state "ABORTING" instead of "RUNNING"
lxc-start: test-lxc: tools/lxc_start.c: main: 306 The container failed to start
lxc-start: test-lxc: tools/lxc_start.c: main: 309 To get more details, run the container in foreground mode
lxc-start: test-lxc: tools/lxc_start.c: main: 311 Additional information can be obtained by setting the --logfile and --logpriority options

No specific reason why the LXC container test-lxc can not be started and the lxc-start command failed. There is just an offer to use the logging options and here is how the administrator of the box may do it by including the following lxc-start options:

-l DEBUG –logfile=test-lxc.log –logpriority=9

Here is a real-world example of an old kernel trying to run LXC 4.0
Keep on reading!

Run LXC CentOS 8 container with bridged network under CentOS 8

The LXC container software comes to CentOS 8 with the EPEL 8 repository. LXC is a multiprocesses container, which offers to boot a Linux distribution under container isolation. It is very similar to systemd-nspawn and a bit different from docker containers. LXC containers are used when multiple processes are needed under one container only. In most cases, the LXC container is a fully-featured Linux distribution (systemd or SysV, i.e. init) booted under a Linux container.
There are several major differences between docker/podman containers and LXC:

  • Multiprocesses.
  • Easy configuration modification. Even hot-plugin supported.
  • Unprivileged Linux containers.
  • Complex network setups. Multiple network interfaces connected to different networks, for example.
  • Live systemd, i.e. systemd or SysV init are booted as usual. Much of the software rellies on systemd/udev features and in many cases, it is really hard to run a software without a systemd or init process

Here are the steps to boot a CentOS 8 container under CentOS 8 host server:

STEP 1) Install EPEL repository.

EPEL CentOS 8 repository now includes LXC 3.0 software.

dnf install -y epel-release

STEP 2) Install LXC software and start LXC service.

At present, the LXC software version is 3.0.4. The package lxc-templates includes template scripts to create a Linux distribution environment like CentOS, Ubuntu, Debian, Gentoo, ArchLinux, Oracle, Alpine, and many others and it also includes the configuration templates to start these Linux distributions.

dnf install -y lxc lxc-templates
dnf install -y wget tar

The wget and tar are required if LXC templates installation is going to be performed.

STEP 3) Create a CentOS 8 container with the help of LXC templates and run it.

Use the lxc-templates to prepare a CentOS 8 container environment. The currently available containers are listed here http://images.linuxcontainers.org/. Check out the URL and choose the right container. Here the CentOS 8 amd64 is used.

lxc-create --template download -n mycontainer -- --dist centos --release 8 --arch amd64 --keyserver hkp://keyserver.ubuntu.com

Keep on reading!

tmpfs mount on /dev/shm in LXC container or chroot environment

When using LXC containers booting the lxc container would not populate it as the normal boot process. Or when you create a chroot jail /dev is not mounted or just some devices are created.
There is an option to populate (when using LXC containers) it with minimal required devices:

lxc.autodev = 1

which will create a tmpfs mount under /dev and create some basic devices, it will ensure /dev/shm to be mounted on with tmpfs!
If you omit this option, the /dev directory won’t be populated and will stay with the devices you made or copied when you made the LXC container (or the chroot jail) and /dev/shm will not be mounted using tmps, which could create numerous bad issues.
If you get errors like

 * configure has detected that the sem_open function is broken.
 * Please ensure that /dev/shm is mounted as a tmpfs with mode 1777.

You could mount the /dev/shm of the LXC container or the chroot jail (usually you can tune the size half of the server’s RAM) with

mkdir -p /dev/shm
mount -t tmpfs -o nodev,nosuid,noexec,mode=1777,size=6144m tmpfs /dev/shm

Or reboot your LXC container with a new configuration (probably in the “/var/lxc/[lxc_name]/config”) adding the following line:

lxc.mount.entry = none dev/shm tmpfs nodev,nosuid,noexec,mode=1777,create=dir 0 0

Thus you ensure the /dev/shm to be mounted on tmpfs and all semaphore functions to work properly.

* Real output of Gentoo failed compilation of python package:
 * configure has detected that the sem_open function is broken.
 * Please ensure that /dev/shm is mounted as a tmpfs with mode 1777.
 * ERROR: dev-lang/python-3.3.4-r1::gentoo failed (configure phase):
 *   Broken sem_open function (bug 496328)
 * 
 * Call stack:
 *     ebuild.sh, line 124:  Called src_configure
 *   environment, line 3542:  Called die
 * The specific snippet of code:
 *           die "Broken sem_open function (bug 496328)";
 * 
 * If you need support, post the output of `emerge --info '=dev-lang/python-3.3.4-r1::gentoo'`,
 * the complete build log and the output of `emerge -pqv '=dev-lang/python-3.3.4-r1::gentoo'`.
 * The complete build log is located at '/var/tmp/portage/dev-lang/python-3.3.4-r1/temp/build.log'.
 * The ebuild environment file is located at '/var/tmp/portage/dev-lang/python-3.3.4-r1/temp/environment'.
 * Working directory: '/var/tmp/portage/dev-lang/python-3.3.4-r1/work/x86_64-pc-linux-gnu'
 * S: '/var/tmp/portage/dev-lang/python-3.3.4-r1/work/Python-3.3.4'

>>> Failed to emerge dev-lang/python-3.3.4-r1, Log file: