LXC with SELinux and NFS share result in kernel: SELinux: inode_doinit_use_xattr: getxattr returned 2 for dev=0:43 ino=

After staring a new LXC container, the syslog program (Syslog-ng) began to throw thousands of errors with this kind of message:

Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6977140995289226736
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6551465724643968476
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-5980833553552494142
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-8820947409424952637
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-8270463809263745561
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7923279144252216900
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6181977668994943343
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7585065875445167421
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7923279144252216900
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-5826517164673898101
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7585065875445167421
Dec  1 11:01:01 h3 rsyslogd[1147]: imjournal: 3871493 messages lost due to rate-limiting (20000 allowed within 600 seconds)

These messages were logged in thousands. The same time, the NFS statistics showed a strange peak of using getattr. Something was calling getattr thousands times per second. Despite there were no SELinux blocks in audit.log as the dmesg suggested the SELinux might be blamed.
The LXC container is an application container, which has mound bind directory from the host server. The very same directory is an local NFS share (using NFS-Ganesha) of a GlusterFS volume and the PHP files are situated there.

main menu
kernel SELinux inode_doinit_use_xattr getxattr returned 2 nfsstat getattr graph

So the LXC container reads the PHP files from this NFS share. There were no issues to access the files and the application LXC worked just fine.
The problem disappeared when the NFS share was remounted with SELinux permissions using the context word:

node3:/VOL1 /mnt/nfs/VOL1 nfs defaults,hard,noexec,nosuid,_netdev,fsc,noatime,context="system_u:object_r:httpd_sys_rw_content_t:s0" 0 0

All the files are of SELinux label httpd_sys_rw_content_t and after restarting the LXC container there were no SELinux lines in the dmesg and the syslog logs. The administrator should configure the right SELinux permissions to the LXC bound directories. More on why SELinux sometimes does not report on blocks in the audit.log here – Selinux permission denied and no log in audit.log.

Degraded ext4 performance with millions files because of high system time in ext4_mb_good_group (100% kworker/u32+flush)

A strange behavior has been monitored in a storage server holding more than 20 millions of files, which is not a really big figure, because we have storage with more than 300 millions of files. The server has a RAID5 of 4 x 10T HGST Ultrastar He10 and storage partition formatted with ext4 filesystem with total of 27T size. Despite the inodes are only 7% and the free space is at 80%, the server began to experience high load averages with almost no IO utilization across the disks and the RAID device.
So the numbers are:

  • 20% free space.
  • 93% free inodes.
  • no IO disk utilization above 10-20% of any of the disks. In fact, the IO above 10% is very rear.
  • really high loads above the number of the server’s cores. The server has 8 virtual processors under Linux and the loads are constantly above 10-14.
  • no Disk Sleep blocked processes or Running to justify the high load.
  • the top command reports one or two kernel processes with 100% running – kworker/u32:1+flush-9:3 (so flushing the data takes so many time?)

If the problem were related to the disks or hardware, it would probably have been seen a high utilization across the disks, because the physical operations would need more time to execute. The high IO utilization across any disk, in deed, may result in a high system time (i.e. kernel time). Apparently, there is no high IO utilization across the disks and even no Disk Sleep blocked processes.
The FlameGraph tool may help to suggest the area where the problem related to.
First, install the git and perf tool (there are two interesting links about it with examples – link1 and link2):

dnf install -y perf git perl-open
git clone https://github.com/brendangregg/FlameGraph

Save a sample 60 second data with:

[root@srv ~]# perf record -F 99 -a -g -- sleep 60
[ perf record: Woken up 24 times to write data ]
[ perf record: Captured and wrote 7.132 MB perf.data (35687 samples) ]

And generate a SVG image of all the function, which are executed during this time frame of 60 seconds. It will give a good picture of what has been executed and how much time it took in the kernel and user space.

[root@srv ~]# cd FlameGraph
[root@srv FlameGraph]# ./stackcollapse-perf.pl ../out.perf > out.folded
[root@srv FlameGraph]# ./flamegraph.pl out.folded > kernel.svg

The SVG are shared bellow. It turned out most of the time during this sample period of 60 seconds, the Linux kernel was in ext4_mb_good_group, which is related to the EXT4 file system.
What fixed the problem (for now) – By removing files it was noticed the system time went down and the load went down, too. Then, after removing approximately a million files, the server performance returned to normal as the average other storage server (this one has unique hardware setup). So by removing files, i.e. INODES from the ext4 file system, the load average and the server ext4 performance were completely different from what it had been before.
The interesting part the inodes never went above 7% (around 24 million of 454 million available) and the free space never went down 20%, but when a good deal of files were removed, the bad ext4 performance disappeared. It still could be related to the physical disks or some strange ext4 bug, but this article is intended to report this behavior and to propose a some kind of a solution (even though a workaround).

SCREENSHOT 1) The kernel process running at 100% – kworker/u32:2-flush-9:3.

main menu
kernel process running at 100%

SCREENSHOT 2) Almost half of the time was spent at ext4 related calls.

main menu
ext4 and ext4_mb_good_group half of the time spent

Keep on reading!

Send coins in Groestlcoin network from command line with Groestlcoin Core 25 in year 2023

This article an updated version of Send coins in Groestlcoin network with Groestlcoin Core (cli), because it appeared many things handling the wallets with the command-line have changed.

main menu
groestlcoin-cli sendtoaddress

The basic notes to send and receive are:

  • a Groestlcoin address is required.
  • a Groestlcoin wallet contains Groestlcoin address
  • the Groestlcoin wallet application (i.e. Groestlcoin node, it is the same application) needs a synchronized Groestlcoin to operate properly, such as send and receive Groestlcoin coins.
  • the Groestlcoin wallet may be created, loaded or imported with the Groestlcoin tools.
  • if there is only one Groestlcoin wallet loaded it would be the default one and it would not require additional options to the Groestlcoin tool to specify, which wallet would be used for the current command.

Here are the steps needed to transfer funds under the Linux distro of Ubuntu securely. To be able to transfer money securely without any 3rd party involved (no online wallet or wallets, which is unclear which node they connect to) just funds from your personal Groestlcoin wallet address on your computer to another Groestlcoin wallet address it is needed a running Groestlcoin node using Groestlcoin Core command line tool:groestlcoin-cli – this is command line tool for managing the Groestlcoin wallets.

STEP 1) Install and run a Groestlcoin node with groestlcoind

The first step is to start a Groestlcoin node by following this updated article – Building from source and run a Groestlcoin node (cli only) under Ubuntu 22.04 LTS. It will help the user to build from the official source code the Groestlcoin Core software, which will include the Groestlcoin node and wallet functionalities. All of the file paths and application names are real and correct if the above article was used.
It’s worth mentioning, that the Groestlcoin node must be running and synced with the Groestlcoin network. Here is how to check it:

myuser@mydesktop:~/groestlcoin-core/bin$ ./groestlcoin-cli getblockchaininfo
{
  "chain": "main",
  "blocks": 4734023,
  "headers": 4734023,
  "bestblockhash": "00000000000020d22b156e27af600fcdc3fd315f8896bbdb5ab748260fbfa3dd",
  "difficulty": 457347.0794165397,
  "time": 1693300370,
  "mediantime": 1693300091,
  "verificationprogress": 0.9999998125955512,
  "initialblockdownload": false,
  "chainwork": "00000000000000000000000000000000000000000000016eeb7afc66a3b219fb",
  "size_on_disk": 3040987475,
  "pruned": false,
  "warnings": ""
}

Keep on reading!

List all rules and IPs when using firewalld under CentOS

CentOS 7, CentOS 8, CentOS Stream 8, CentOS Stream 9 use firewalld service for the firewall of the machine. Firewalld service is easily controlled by the command-line tool firewall-cmd, which has a relatively simple syntax. It imposes an abstraction layer over the much more complex Linux kernels backends like iptables, ip6tables, arptables, ebtables, ipsetand, and nftables. The Firewalld organizes the firewall rules with the help of policies and zones, but there is a catch – it is not possible to list all the firewall rules to check whether an IP, network, or ethernet interface takes part in some rule or policy.

main menu
nft list ruleset

Yes, it is possible to list all rules of a firewalld zone (firewall-cmd –list-all), but what if there are multiple zones? First, get all the zone names and then enumerate all the zones with a list command to check for an IP. It is not practical and fast.
There is the other trick, searching through the Firewalld configuration files in the directory /etc/firewalld/, which saves all the permanent rules. But what if there are temporary rules, which are not saved in the configuration?
From CentOS 8.2 (RHEL 8.2) firewalld backend defaults to nftables instead of “iptables”. By using the nftables tools we can list all the rules applied in text or JSON format. It is like a snapshot of all the running rules applied by the Linux kernel firewall infrastructure no matter in which zone and policy. There is no such command with the firewall-cmd tool.

CentOS 8.2 (RHEL 8.2) and above including CentOS Stream 9

There is an export command using the nfs command line tool.

nft list ruleset

Keep on reading!

How to upgrade to CentOS Stream 9 from CentOS Stream 8

This article will show how to update to CentOS Stream 9 from CentOS Stream 8.

main menu
grub entries

If only official repositories are used it is fairly easy to upgrade to the new rolling based CentOS Stream 9 release, which follows the Red Hat Enterprise Linux 9. Using unofficial or users’ repositories CentOS Stream 8 may lead to unstable system or break user’s system following this tutorial. First, check out for CentOS Stream 9 compatibility issues the currently installed repositories if some specific or unofficial are used.
The following article uses a CentOS Stream 9 with only the official default repositories and in addition, the EPEL repository, which has a CentOS Stream 9 support.
How to install – Network installation of CentOS Stream 9 (20220606.0) – minimal server installation with additional information about important CentOS Stream 9 URLs and links and what kind of software the user can expect to have Software and technical details of CentOS Stream 9 minimal install. Camparing the packages of the two systems – a clean install of CentOS Stream 9 and an upgrade from CentOS Stream 8, the clean install has 377 packages installed and the upgrade has 387 packages installed. The strated processes are the same except the chronyd service is not started on boot. chronyd daemon offers a time synchronization.

STEP 1) All installed packages should be updated to the latest versions.

[root@srv ~]# dnf update -y
Last metadata expiration check: 0:09:08 ago on Tue Oct  4 12:12:07 2022.
Dependencies resolved.
Nothing to do.
Complete!

The DNF tool reports that all packages are up-to-date, because there is nothing to upgrade.

STEP 2) A cleanup of all packages, which are not required anymore.

Check orphan and leaves packages. Some of the packages here may be used by the user explicitly, so it is important to know the system. Remove the packages if not used by the system and they are showed as an output of the following commands and they are not used by the user of the system.

[root@srv ~]# dnf repoquery --unneeded
Last metadata expiration check: 0:12:56 ago on Tue Oct  4 12:12:07 2022.
NetworkManager-initscripts-updown-1:1.40.0-1.el8.noarch
grub2-tools-efi-1:2.02-129.el8.x86_64
libmetalink-0:0.1.3-7.el8.x86_64
pciutils-0:3.7.0-1.el8.x86_64
python3-configobj-0:5.0.6-11.el8.noarch
python3-schedutils-0:0.6-6.el8.x86_64
rdma-core-0:41.0-1.el8.x86_64
[root@srv ~]# dnf repoquery --extras
Last metadata expiration check: 0:00:12 ago on Tue Oct  4 12:34:17 2022.

Remove the unneeded packages:
Keep on reading!

git status and bus error on SSD – fix READ errors by recovering part of the file

SSD and Linux encryption may not be the best idea, especially without the TRIM (allow-discards) option (or never executed fstrim?). Nevertheless, this error may occur not only on an SSD device, but just where there is a corrupted file system or device.
In our case, the SSD has some read errors. Apparently, some files or some parts of files could not be read by the git command:

[myuser@dekstop kernel]# git status -v
Bus error 84/115708)

In the case of SSD bad reads, the only working solution is to find and overwrite the problem file(s) or remove the file(s) and recreate them. A more sophisticated solution is to dump the file with dd and skip errors option enabled to another location and then overwrite the old file with the new one. So only the corrupted area of the file will be lost, which in most cases is just one or two sectors, i.e. one or two 512 bytes of data.

STEP 1) Find the bad files with the find command.

Use find Linux command and read all the files with the cat Linux command, so a bad sector will output an input/output error on READ. On write errors won’t be generated, but the sector will be automatically moved to a healthy one (the bad sector is marked and never used more).

[myuser@dekstop kernel]#  find -type f -exec cat {} > /dev/null \;
cat: ./servers/logo_description.txt: Input/output error

If multiple files are found repeat the procedure with each file.

STEP 2) Copy the healthy portion of the file.

The easiest way to remove the error is just to delete the file (or overwrite it), but if the healthy portion of the file is desirable the dd utility may be used to recover it:
Keep on reading!

MPEG-DASH and ClearKey, CENC drm encryption with Nginx, bento4 and dashjs under CentOS 8

The purpose of this article is to demonstrate a simple and plain example of ClearKey DRM encryption using a DASH stream.
Usually, the ClearKey is used only for testing the encryption key and the DRM setup, because the decrypting key is transferred in a plain text to the browser. In simple DRM words, the key is transferred in plain text, and the handle of the decryption is not in some proprietary module such as CMD – Content Decryption Modules. The CMD is a proprietary module in the browsers or the players, which works like a black box when handling the decryption key. The most popular DRMs are Google’s Widevine, Apple’s Fireplay, and Microsoft PlayReady, which work through a proprietary module – CMD (Content Decryption Modules) in the browser (or the OS and player).
All the three DRMs work basically in a similar way:

  • There is a (encryption) key and a (encryption) keyID, which purpose is to identify the (encryption) key.
  • The video file is encrypted with the key and it includes the keyID.
  • The client needs to have the appropriate CMD (Content Decryption Modules) to decrypt the video.
  • The clients receive a license from a license server, which is encrypted data for the CDM on how to decrypt the video identified by the keyID. In fact, the client sends the keyID and receives the proper license (i.e. license binary data) for this keyID. That’s why keyID is included in the encrypted video. Bare in mind, the CMD is proprietary Content Decryption Module offered by the creator of the DRM – Google, Apple, Microsoft or another and it lives in the browser (OS or player). All popular browsers support at least one of the proprietary DRMs.

ClearKey is like the proprietary DRM schemes, but without the CMD (Content Decryption Modules).

The “org.w3.clearkey” Key System uses plain-text clear (unencrypted) key(s) to decrypt the source. No additional client-side content protection is required.

So, in general, there is no need for a license server when using ClearKey DRM.
Of course, an additional attempt to hide the plain-text key could be made using an extension to the client’s player such as javascript modules and etc. In general, it is perceived this approach to be less secure, because it is much easier to debug the javascript code on the client side. More on ClearKeyhttps://www.w3.org/TR/encrypted-media/#clear-key

Here are all the steps from the server till the client to use ClearKey.

STEP 1) Download and install bento4 software.

bento4 is an open source toolkit for manipulating some of the most common video formats – MP4 and DASH/HLS/CMAF media. The download page is https://www.bento4.com/downloads/ and the Linux binary for latest stable version: https://www.bok.net/Bento4/binaries/Bento4-SDK-1-6-0-639.x86_64-unknown-linux.zip. There is also a source code snapshot link.
Download the famous blender video for the demostration: https://download.blender.org/demo/movies/BBB/bbb_sunflower_1080p_30fps_normal.mp4
Download and unpack the binary Bento4-SDK-1-6-0-639.x86_64-unknown-linux.zip.
Keep on reading!

Create MacVTap device using NetworkManager nmcli under CentOS 8

In continuation of NetworkManager management with nmcli, here is a quick Linux console tip for users like CentOS 8 (or all distributions, which use the NetworkManager for managing the networking). How to create a virtualized bridge device MacVTap device with the NetworkManager nmcli command utility, which will preserve all the configuration over reboots.

nmcli connection add type macvlan dev enp0s3 mode bridge tap yes ifname macvtap0 con-name macvtap0 ip4 0.0.0.0/24

The line above creates a virtualized bridged interface and a connection with the name macvtap0. The MAcVTap device with the name macvtap0 is in bridge mode with the physical network interface enp0s3 with manual IP setting. If the IP is not included a DHCP option will be used as default.

There is one big limitation – there is no link between the enp0s3 and macvtap0. When used macvtap0 could receive packets from the network through the enp0s3, but there is no direct link between the two network devices. In simple words, when used in a virtualized environment in a virtual machine the virtual machine may have access to the network shared with the enp0s3, but the virtual machine cannot communicate with the IPs of the enp0s3!

Typically, this is used to make both the guest and the host show up directly on the switch that the host is connected to.

Linux Virtualization, https://virt.kernelnewbies.org/MacVTap

Initial state, only one connection in NetworkManager.

The main server connection with name enp0s3 using the same name network interface enp0s3:

[root@srv ~]# nmcli con
NAME    UUID                                  TYPE      DEVICE 
enp0s3  09497bbf-da59-42b7-a72c-d69369760b36  ethernet  enp0s3
[root@srv ~]# nmcli 
enp0s3: connected to enp0s3
        "Intel 82540EM"
        ethernet (e1000), 08:00:27:03:C9:2E, hw, mtu 1500
        ip4 default
        inet4 192.168.0.20/24
        route4 192.168.0.0/24 metric 100
        route4 0.0.0.0/0 via 192.168.0.1 metric 100
        inet6 fe80::a00:27ff:fe03:c92e/64
        route6 fe80::/64 metric 100

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

DNS configuration:
        servers: 8.8.8.8 1.1.1.1
        interface: enp0s3

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.

Add the MacVTap device with the name macvlan0

[root@srv ~]# nmcli connection add type macvlan dev enp0s3 mode bridge tap yes ifname macvtap0 con-name macvtap0 ip4 0.0.0.0/24
Connection 'macvtap0' (7a5ef04c-ea98-4642-ac5d-4239f715f631) successfully added.

A MacVTap device, a network connection, and a link are established. The name of the MacVTap device and the network connection is macvtap0.

Keep on reading!

Gentoo ERROR: Python module pytevent of version 0.10.2 not found, and bundling disabled

Emerging the sys-libs/ldb-2.3.0-r1 package may fail with an error for a missing Python mode, despite the sys-libs/tevent with a python USE flag is presented in the system:

Checking for system tevent (>=0.10.2)                                                           : yes 
ERROR: Python module pytevent of version 0.10.2 not found, and bundling disabled
 * ERROR: sys-libs/ldb-2.3.0-r1::gentoo failed (configure phase):
 *   configure failed
 * 
 * Call stack:

Indeed, the tevent (>=0.10.2) is found, but not the Python module! And the checking pase of the setup fails.
First, check whether the USE of sys-libs/tevent has python and the right version PYTHON_SINGLE_TARGET=”python3_8″ is used (the Python version may vary here):

root@srv # emerge -pv =tevent-0.10.2::gentoo

[ebuild   R    ] sys-libs/tevent-0.10.2::gentoo  USE="python" ABI_X86="32 (64) (-x32)" PYTHON_SINGLE_TARGET="python3_8 -python3_9" 0 KiB

Total: 1 package (1 reinstall), Size of downloads: 0 KiB

This system uses Python 3.8 and the library sys-libs/tevent was built with this USE flag.

The problem here is the tevent is installed under a its own directory: /usr/lib64/tevent. Using the ldconig utility the problem quickly has been resolved. Just add a file /etc/ld.so.conf.d/tevent_ldb.conf with the path to the library and then regenerate the ldconfig:

root@srv # cat /etc/ld.so.conf.d/tevent_ldb.conf
/usr/lib64/tevent
root@srv # ldconfig
root@srv # 

Do not forget to run the “ldconfig“, because tevent library won’t be added to the LD cache. Emerging the sys-libs/ldb and then samba was successful after this quick workaround! There is a Gentoo bug reported, but the problem still exists – https://bugs.gentoo.org/590026

Gentoo emerge virtualbox- Mesa / GLU: Mesa not found at, Mesa headers not found

Emerging the package app-emulation/virtualbox the following error occurs:

Checking for Mesa / GLU: 
  Mesa not found at -L/usr/X11R6/lib -L/usr/X11R6/lib64 -L/usr/local/lib -lXext -lX11 -lGL -I/usr/local/include or Mesa headers not found
  Check the file /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/configure.log for detailed error information.
Check /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/configure.log for details
 * ERROR: app-emulation/virtualbox-6.1.18::gentoo failed (configure phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line  125:  Called src_configure
 *   environment, line 5504:  Called doecho './configure' '--with-gcc=x86_64-pc-linux-gnu-gcc' '--with-g++=x86_64-pc-linux-gnu-g++' '--disable-dbus' '--disable-kmods' '--disable-alsa' '--disable-docs' '--disable-devmapper' '--disable-pulse' '--disable-python' '--enable-webservice' '--enable-vnc'
 *   environment, line 1538:  Called die
 * The specific snippet of code:
 *       "$@" || die

The configure script reports the mesa is missing, but the package media-libs/mesa is installed. Reinstalling does not fix the problem.
Farther investigation in the logs by checking the configure.log reveals the real problem:

srv ~ # tail -n 16 /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/configure.log
***** Checking Mesa / GLU *****
compiling the following source file:
#include <cstdio>
#include <X11/Xlib.h>
#include <GL/glx.h>
#include <GL/glu.h>
extern "C" int main(void)
{
  return 0;
}
using the following command line:
x86_64-pc-linux-gnu-g++  -fPIC -g -O -Wall -o /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/.tmp_out /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/.tmp_src.cc "-L/usr/X11R6/lib -L/usr/X11R6/lib64 -L/usr/local/lib -lXext -lX11 -lGL -I/usr/local/include"
/var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/.tmp_src.cc:4:10: fatal error: GL/glu.h: No such file or directory
    4 | #include <GL/glu.h>
      |          ^~~~~~~~~~
compilation terminated.

The glu part of mesa is missing. In Gentoo, the glu (https://gitlab.freedesktop.org/mesa/glu) is not included in the media-libs/mesa and it is a separate package media-libs/glu.

The solution is to emerge media-libs/glu and then the app-emulation/virtualbox.

emerge -v media-libs/glu

Another Linux distribution may include glu in the main mesa package.

Here, the conclusion is to always check the configure.log, because it reports the exact error and not to trust the generic output of the configure script.