LXC with SELinux and NFS share result in kernel: SELinux: inode_doinit_use_xattr: getxattr returned 2 for dev=0:43 ino=

After staring a new LXC container, the syslog program (Syslog-ng) began to throw thousands of errors with this kind of message:

Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6977140995289226736
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6551465724643968476
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-5980833553552494142
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-8820947409424952637
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-8270463809263745561
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7923279144252216900
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6181977668994943343
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7585065875445167421
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7923279144252216900
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-5826517164673898101
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7585065875445167421
Dec  1 11:01:01 h3 rsyslogd[1147]: imjournal: 3871493 messages lost due to rate-limiting (20000 allowed within 600 seconds)

These messages were logged in thousands. The same time, the NFS statistics showed a strange peak of using getattr. Something was calling getattr thousands times per second. Despite there were no SELinux blocks in audit.log as the dmesg suggested the SELinux might be blamed.
The LXC container is an application container, which has mound bind directory from the host server. The very same directory is an local NFS share (using NFS-Ganesha) of a GlusterFS volume and the PHP files are situated there.

main menu
kernel SELinux inode_doinit_use_xattr getxattr returned 2 nfsstat getattr graph

So the LXC container reads the PHP files from this NFS share. There were no issues to access the files and the application LXC worked just fine.
The problem disappeared when the NFS share was remounted with SELinux permissions using the context word:

node3:/VOL1 /mnt/nfs/VOL1 nfs defaults,hard,noexec,nosuid,_netdev,fsc,noatime,context="system_u:object_r:httpd_sys_rw_content_t:s0" 0 0

All the files are of SELinux label httpd_sys_rw_content_t and after restarting the LXC container there were no SELinux lines in the dmesg and the syslog logs. The administrator should configure the right SELinux permissions to the LXC bound directories. More on why SELinux sometimes does not report on blocks in the audit.log here – Selinux permission denied and no log in audit.log.

Degraded ext4 performance with millions files because of high system time in ext4_mb_good_group (100% kworker/u32+flush)

A strange behavior has been monitored in a storage server holding more than 20 millions of files, which is not a really big figure, because we have storage with more than 300 millions of files. The server has a RAID5 of 4 x 10T HGST Ultrastar He10 and storage partition formatted with ext4 filesystem with total of 27T size. Despite the inodes are only 7% and the free space is at 80%, the server began to experience high load averages with almost no IO utilization across the disks and the RAID device.
So the numbers are:

  • 20% free space.
  • 93% free inodes.
  • no IO disk utilization above 10-20% of any of the disks. In fact, the IO above 10% is very rear.
  • really high loads above the number of the server’s cores. The server has 8 virtual processors under Linux and the loads are constantly above 10-14.
  • no Disk Sleep blocked processes or Running to justify the high load.
  • the top command reports one or two kernel processes with 100% running – kworker/u32:1+flush-9:3 (so flushing the data takes so many time?)

If the problem were related to the disks or hardware, it would probably have been seen a high utilization across the disks, because the physical operations would need more time to execute. The high IO utilization across any disk, in deed, may result in a high system time (i.e. kernel time). Apparently, there is no high IO utilization across the disks and even no Disk Sleep blocked processes.
The FlameGraph tool may help to suggest the area where the problem related to.
First, install the git and perf tool (there are two interesting links about it with examples – link1 and link2):

dnf install -y perf git perl-open
git clone https://github.com/brendangregg/FlameGraph

Save a sample 60 second data with:

[root@srv ~]# perf record -F 99 -a -g -- sleep 60
[ perf record: Woken up 24 times to write data ]
[ perf record: Captured and wrote 7.132 MB perf.data (35687 samples) ]

And generate a SVG image of all the function, which are executed during this time frame of 60 seconds. It will give a good picture of what has been executed and how much time it took in the kernel and user space.

[root@srv ~]# cd FlameGraph
[root@srv FlameGraph]# ./stackcollapse-perf.pl ../out.perf > out.folded
[root@srv FlameGraph]# ./flamegraph.pl out.folded > kernel.svg

The SVG are shared bellow. It turned out most of the time during this sample period of 60 seconds, the Linux kernel was in ext4_mb_good_group, which is related to the EXT4 file system.
What fixed the problem (for now) – By removing files it was noticed the system time went down and the load went down, too. Then, after removing approximately a million files, the server performance returned to normal as the average other storage server (this one has unique hardware setup). So by removing files, i.e. INODES from the ext4 file system, the load average and the server ext4 performance were completely different from what it had been before.
The interesting part the inodes never went above 7% (around 24 million of 454 million available) and the free space never went down 20%, but when a good deal of files were removed, the bad ext4 performance disappeared. It still could be related to the physical disks or some strange ext4 bug, but this article is intended to report this behavior and to propose a some kind of a solution (even though a workaround).

SCREENSHOT 1) The kernel process running at 100% – kworker/u32:2-flush-9:3.

main menu
kernel process running at 100%

SCREENSHOT 2) Almost half of the time was spent at ext4 related calls.

main menu
ext4 and ext4_mb_good_group half of the time spent

Keep on reading!

nginx with SELinux and ngx_http_lua_module – PANIC: unprotected error in call to Lua API

main menu
runtime code generation failed restricted kernel

The CentOS 8 (CentOS Stream 9, too) might prevent to successfully execute a Lua code in NGINX web server with the modules like ngx_http_lua_module because of insufficient SELinux rules. By default, the web server is not allowed to execute programs that require memory addresses that are both executable and writable. Using Lua code may even crash with a core dump and restart of the NGIX worker process. The error logs contains information for the incident:

PANIC: unprotected error in call to Lua API (runtime code generation failed, restricted kernel?)

In fact, the error message hints what is going on – that the kernel prevents the execution of a code, which might be an addition limit enforced to this system.
Searching through the SELinux audit log shows the solution by enable the httpd_execmem SELinux boolean.

[root@srv ]# ausearch -c 'nginx' --raw | audit2allow


#============= httpd_t ==============

#!!!! This avc can be allowed using the boolean 'httpd_execmem'
allow httpd_t self:process execmem;

Apparently, the Lua code used needs a privileges to execute a code in the memory addresses (execmem). So when using the OpenResty ngx_http_lua_module, which is not distributed with the NGINX server, the server administrator should enable to true the SELinux boolean – httpd_execmem with:

setsebool -P httpd_execmem=1

The “-P” is going to make the value persistent over reboots.

Note, the official manual for the SELinux boolean. By enabling the httpd_execmem, several SELinux rules are applied, which may make the web server less secure, because the web server process is allowed to execute code in the memory.

– httpd_execmem

When enabled, this Boolean allows httpd to execute programs that require memory addresses that are both executable and writable. Enabling this Boolean is not recommended from a security standpoint as it reduces protection against buffer overflows, however certain modules and applications (such as Java and Mono applications) require this privilege.

The AVC (Access Vector Cache) message looks like:

type=AVC msg=audit(1699774004.724:59165857): avc:  denied  { execmem } for  pid=3872876 comm="nginx" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=process permissive=0
type=SYSCALL msg=audit(1699774004.724:59165857): arch=c000003e syscall=10 success=no exit=-13 a0=7f399cd69000 a1=10000 a2=5 a3=38 items=0 ppid=884311 pid=3872876 auid=4294967295 uid=990 gid=987 euid=990 suid=990 fsuid=990 egid=987 sgid=987 fsgid=987 tty=(none) ses=4294967295 comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0 key=(null)ARCH=x86_64 SYSCALL=mprotect AUID="unset" UID="nginx" GID="nginx" EUID="nginx" SUID="nginx" FSUID="nginx" EGID="nginx" SGID="nginx" FSGID="nginx"
type=PROCTITLE msg=audit(1699774004.724:59165857): proctitle=6E67696E783A20776F726B65722070726F63657373

More on NGINXhttps://ahelpme.com/tag/nginx/ or SELinuxhttps://ahelpme.com/tag/selinux/.

Software and technical details of Fedora Server 39 including cockpit screenshots

main menu
cockpit overview on Fedora Linux 39

This article is for those of you who do not want to install a whole new operating system only to discover some technical details about the default installation like disk layout, packages included, software versions, and so on. Here we are going to review in several sections what is like to have a default installation of Fedora 39 Server using a real not virtual machine!
The kernel is 6.2.15 it detects successfully the Threadripper 1950X AMD and the system is stable (we booted in UEFI mode).
The installation procedure uses default options for all installation setups – Minimal network installation of Fedora 39 Server.

Software

With Fedora Server 39 you can have

  • linux kernel – 6.5.10-300 (6.5.10-300.fc39.x86_64)
  • System
    • linux-firmware – version: 20231030, release: 20231030-1.fc39.
    • libc – 2.38 (2.38-10.fc39)
    • GNU GCC – 13.2.1 (13.2.1-4.fc39)
    • OpenSSL – 3.1.1 (3.1.1-4.fc39) and 1.1.1q (1.1.1q-5.fc39)
    • coreutils – 9.3 (9.3-4.fc39)
    • yum – Depricated and replaced with dnf
    • dnf – 4.18.0 (4.18.0-2.fc39). The dnf5 is available, too (5.1.5-1.fc39)
    • rsyslog – 8.2310.0 (8.2310.0-1.fc39)
    • NetworkManager – 1.44.2 (1.44.2-1.fc39)
  • Servers
    • Apache – 2.4.58 (2.4.58-1.fc39)
    • Nginx – 1.24.0 (1.24.0-4.fc39)
    • MySQL server – 8.0.34 (8.0.34-2.fc39)
    • MariaDB server – 10.5.22 (10.5.22-1.fc39)
    • PostgreSQL – 15.4 (15.4-1.fc39)
  • Programming
    • PHP – 8.2.12 (8.2.12-1.fc39)
    • python – The default is 3.12.0 (3.12.0-1.fc39) and many more available – 3.13.0 (3.13.0~a1-1.fc39), 3.11.6 (3.11.6-1.fc39), 3.10.13 (3.10.13-1.fc39), 3.9.18 (3.9.18-1.fc39), 3.8.18 (3.8.18-1.fc39), 3.7.17 (3.7.17-3.fc39), 3.6.15 (3.6.15-20.fc39) and also includes the older 2.7.18 (2.7.18-35.fc39)
    • perl – 5.38.0 (5.38.0-501.fc39)
    • ruby – 3.2.2 (3.2.2-181.fc39)
    • OpenJDK – the latest 21 – 21.0.0.0.35 (21.0.0.0.35-1.rolling.fc39) and also includes 17.0.8.0.7 (17.0.8.0.7-1.fc39), 11.0.20.0.8 (11.0.20.0.8-1.fc39) and 1.8.0.382.b05 (1.8.0.382.b05-2.fc39)
    • Go – 1.21.3 (1.21.3-1.fc39)
    • Rust – 1.73.0-1.fc39 (1.73.0-1.fc39)
    • llvm – the latest 17.0.3 (17.0.3-1.fc39), 16.0.6 (16.0.6-5.fc39), 15.0.7 (15.0.7-4.fc39), 14.0.5 (14.0.5-6.fc39), 13.0.1 (13.0.1-5.fc39), 12.0.1 (12.0.1-9.fc39), 11.1.0 (11.1.0-11.fc39), 8.0.1 (8.0.1-5.fc39) and 7.0.1 (7.0.1-7.fc39.8)
    • Subversion – 1.14.2 (1.14.2-20.fc39)
    • Git – 2.41.0 (2.41.0-2.fc39)

Note: Not all of the above software comes installed by default. The versions above are valid as of November 2023, these are the minimum versions you get with Fedora Server 39 now, and updating it after the initial date may update some of the above packages with newer versions.
Installed packages are 682 occupying 1.7G space:. Note, this is Fedora Server Install, not minimal install. The server install includes the web console – cockpit version 254.

root@srv:~# dnf list installed|wc -l
682
root@srv:~# df -h /
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root   15G  1.7G   14G  12% /

Keep on reading!

Minimal network installation of Fedora 39 Server

This article will show the simple steps of installing a modern Linux Distribution Fedora 39 Server edition. Fedora line offers many bleeding-edge Linux technologies than the more enterprise CentOS of the same RPM Linux family.

In fact, if the user needs a server with the latest Linux stable software Fedora server is the right and easy choice for a server!

It is interesting to compare to the other big rival Ubuntu Server, which has the latest software and upgrade path to the future release.
Here are some basic data from the default installation setup settings:

  1. Installed packages – ~682 occupying 1.7G of space.
  2. 3 partitions when using automatic partition layout – boot efi, boot and lvm.
  3. xfs used for the root and the boot partitions.

The Fedora 39 Server comes and updates to the latest stable Linux:

  • Linux kernel : 6.5.10.
  • Python : 3.12.0
  • GLibc : 2.38
  • OpenSSL : 3.1.1
  • systemd : 254.5

More detailed software overview here – Software and technical details of Fedora Server 39 including cockpit screenshots.

Of course, one can expect the latest version of GCC (13.2.1), PHP (8.2.12), GO (1.21.3), MySQL Server (8.0.34), PostgreSQL (15.4), NGINX (1.24.0), Apache (2.4.58) and so on. Almost all of them are the latest stable version on their Internet sites.
Just be careful, the Fedora life cycle is 13 months from the release to the EOL (End of Life)! Of course, a dist-upgrade is supported and indeed, it has been flawless for years!

We used the following ISO for the installation process from https://getfedora.org/en/server/download/:

https://download.fedoraproject.org/pub/fedora/linux/releases/39/Server/x86_64/iso/Fedora-Server-netinst-x86_64-39-1.5.iso

It is not a LIVE image so you cannot try it before installing it. The easiest way is to download the image and burn it to a DVD or USB stick disk and then follow the installation below (a USB flash drive could be also created from this ISO). The netinstall installation is as simple as having a good Internet connection to download the packages, the installation wizard automatically detects the closest mirror, from which it will download the packages. Essentially, the network does not differ from the ordinary installation except it expects to download the packages from the Internet. The good thing f network installation is that the bootable ISO is just 686Mbytes and the minimal install of the Fedora 39 Server will consume only around 686 Mbytes.
Here is how to make a bootable USB flash drive using dd:

root@srv ~ # dd if=/mnt/media/OS/Fedora/Fedora-Server-netinst-x86_64-39-1.5.iso of=/dev/sdc bs=8M status=progress oflag=direct
620756992 bytes (621 MB, 592 MiB) copied, 5 s, 123 MB/s719169536 bytes (719 MB, 686 MiB) copied, 5.87066 s, 123 MB/s

85+1 records in
85+1 records out
719169536 bytes (719 MB, 686 MiB) copied, 5.9026 s, 122 MB/s

The /dev/sdd is the removable USB drive. Be careful, it probably will with another name on a different system. Find the name by checking the dmesg.

SCREENSHOT 1) Select the UEFI OS (KINGSTON …), which is the USB flash drive connected to the server.

This USB flash drive is created by the Fedora Official ISO described above.

main menu
USB flash BIOS boot

Keep on reading!

List multiple connections with the same name using nmcli NetworkManager under CentOS

When using the NetworkManager it is possible to create multiple connections with the same name, which may result in confusion how to list them all and how to delete the unneeded ones.

main menu
List network connections

It is simple to create a connection with a certain name, activate it and then deactivate it:

[root@srv ~]# nmcli con add type ethernet con-name eno2 ifname eno2 ipv4.method manual ipv4.addresses 192.168.68.10/24
Warning: There are 3 other connections with the name 'eno2'. Reference the connection by its uuid '47488136-83bf-4394-b2aa-3123886ca9a5'
Connection 'eno2' (47488136-83bf-4394-b2aa-3123886ca9a5) successfully added.
[root@srv ~]# nmcli con down eno2
Connection 'eno2' successfully deactivated (D-Bus active path: /org/freedesktop/NetworkManager/ActiveConnection/6)

So after deactivating the “eno2” connection (the real network interface is with the same name) it is possible to create another connection with the name “eno2” using the same (or even other network interface).
If there are multiple connections with the same name, when creating a new one, there is a warning as it is shown above. There is also a line in the nmcli output, which indicates how many connections there are with this name:

 
[root@srv ~]# nmcli
br0: connected to br0
        "br0"
        bridge, AC:1F:6B:F6:F6:3C, sw, mtu 1500
        ip4 default
        inet4 192.168.0.10/24
        route4 default via 192.168.67.61 metric 425
        route4 192.168.0.0/24 metric 425
        inet6 fe80::c1d5:b200:7259:7e4d/64
        route6 fe80::/64 metric 1024

eno1: connected to bridge-slave-eno1
        "Intel I210"
        ethernet (igb), AC:1F:6B:F6:F6:3C, hw, mtu 1500
        master br0

eno2: disconnected
        "Intel I210"
        3 connections available
        ethernet (igb), AC:1F:6B:F6:F6:3D, hw, mtu 1500

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

DNS configuration:
        servers: 8.8.8.8 1.1.1.1
        interface: br0

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.

Under the disconnected connection name “eno2”, there is a line informing there are 3 connections available under this section.

To list all network connections use the short syntax, which will show all the connections identified by their unique identifier (UUID):

main menu
3 connections with the same name

Keep on reading!

Automatically start zram swap device on boot with systemd zram-generator under CentOS Stream 9

zram-generator package will install necessary tools to automate the creation on boot the compressed RAM devices. This article focuses on compressed swap devices.

main menu
install

As of writing this article, the latest version in the package system under CentOS Stream 9 is 0.32. The the latest version on the original page of the software is much higher number 1.1.2 and many of the following configuration options are marked as OBSOLETE, but they work in the 0.32 version included in CentOS Stream 9 (and the new configuration options does not!). That’s why it is important to check the included sample configuration file.
The package installs no configuration file, just a sample configuration file – /usr/share/doc/zram-generator/zram-generator.conf.example.
zram in the kernel space – https://docs.kernel.org/admin-guide/blockdev/zram.html

STEP 1) Install the zram-generator

It is easy and straightforward, just a single package:

[root@srv) ~]# dnf install -y zram-generator
Last metadata expiration check: 3:42:20 ago on Fri 20 Oct 2023 05:18:32 AM UTC.
Dependencies resolved.
==============================================================================================================
 Package                      Architecture         Version                      Repository               Size
==============================================================================================================
Installing:
 zram-generator               x86_64               0.3.2-7.el9                  appstream               409 k

Transaction Summary
==============================================================================================================
Install  1 Package

Total download size: 409 k
Installed size: 983 k
Downloading Packages:
zram-generator-0.3.2-7.el9.x86_64.rpm                                         2.1 MB/s | 409 kB     00:00    
--------------------------------------------------------------------------------------------------------------
Total                                                                         1.3 MB/s | 409 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                      1/1 
  Installing       : zram-generator-0.3.2-7.el9.x86_64                                                    1/1 
  Running scriptlet: zram-generator-0.3.2-7.el9.x86_64                                                    1/1 
  Verifying        : zram-generator-0.3.2-7.el9.x86_64                                                    1/1 

Installed:
  zram-generator-0.3.2-7.el9.x86_64                                                                           

Complete!

STEP 2) Create a configuration and start the service

By default, there is no configuration file. The best place for it is /etc/systemd/zram-generator.conf.
Here is an example, which will fulfill the following:

  • Set the maximum RAM of the system, because it is a hard-coded 9G, by default. And if not redefined the device will be with 9G max memory no matter how much is tuned by the other options.
  • Set the RAM of the compressed device.
  • Set the fraction of the ram, which may be used by the device. Again as with the above configuration option, it is important, because it limits the maximum available memory to allocate for the compressed device.
  • Set the type of the device – filesystem or swap device.

There is the real world example configuration – /etc/systemd/zram-generator.conf

[root@srv ~]# cat /etc/systemd/zram-generator.conf
[zram0]
host-memory-limit = none
max-zram-size = 32768
zram-fraction = 1.0
compression-algorithm = zstd
swap-priority = 100
fs-type = swap

Keep on reading!

Pass-through the NVIDIA card in a LXC container

Pass-through the NVIDIA card to be used in the LXC container is simple enough and there are three simple rules to watch for:

  • mount bind the NVIDIA devices in /dev to the LXC container’s /dev
  • Allow cgroup access for the bound /dev devices.
  • Install the same version of the NVIDIA driver/software under the host and the LXC container or there will be multiple errors of the sort – version mismatch

main menu
config

When using the LXC container pass-through, i.e. mount bind, the video card may be used simultaneously on the host and on all the LXC containers where it is mount bind. Multiple LXC containers share the video device(s).

This is a working LXC 4.0.12 configuration:

# Distribution configuration
lxc.include = /usr/share/lxc/config/common.conf
lxc.arch = x86_64

# Container specific configuration
lxc.rootfs.path = dir:/mnt/storage1/servers/gpu1u/rootfs
lxc.uts.name = gpu1u

# Network configuration
lxc.net.0.type = macvlan
lxc.net.0.link = enp1s0f1
lxc.net.0.macvlan.mode = bridge
lxc.net.0.flags = up
lxc.net.0.name = eth0
lxc.net.0.hwaddr = fe:77:3f:27:15:60

# Allow cgroup access
lxc.cgroup2.devices.allow = c 195:* rwm
lxc.cgroup2.devices.allow = c 234:* rwm
lxc.cgroup2.devices.allow = c 237:* rwm


# Pass through device files
lxc.mount.entry = /dev/nvidia0 dev/nvidia0 none bind,optional,create=file
lxc.mount.entry = /dev/nvidia1 dev/nvidia1 none bind,optional,create=file
lxc.mount.entry = /dev/nvidia2 dev/nvidia2 none bind,optional,create=file
lxc.mount.entry = /dev/nvidia3 dev/nvidia3 none bind,optional,create=file
lxc.mount.entry = /dev/nvidiactl dev/nvidiactl none bind,optional,create=file
lxc.mount.entry = /dev/nvidia-uvm dev/nvidia-uvm none bind,optional,create=file
lxc.mount.entry = /dev/nvidia-modeset dev/nvidia-modeset none bind,optional,create=file
lxc.mount.entry = /dev/nvidia-uvm-tools dev/nvidia-uvm-tools none bind,optional,create=file
lxc.mount.entry = /dev/nvidia-caps dev/nvidia-caps none bind,optional,create=dir


# Autostart
lxc.group = onboot
lxc.start.auto = 1
lxc.start.delay = 10

Keep on reading!

Recover MySQL InnoDB Cluster and Dba.rebootClusterFromCompleteOutage: Argument #2: Invalid options: primary (ArgumentError)

main menu
MySQL 8.0.28 version

Recent version of MySQL 8 implemented more options to the rebootClusterFromCompleteOutage function! Definitely check the link’s manual above and most of the handy second options are implemented in MySQL 8.0.30, so the user’s MySQL InnoDB Cluster crashed and if rebootClusterFromCompleteOutage should be used, but it outputs an error sort of:

 MySQL  db-cluster-1:33060+ ssl  JS > var cluster = dba.rebootClusterFromCompleteOutage()
Restoring the cluster 'mycluster1' from complete outage...

Dba.rebootClusterFromCompleteOutage: Target member is in state ERROR (RuntimeError)

And when trying to use the node, which was healthy before the crash with this function, there is an error, too:

 MySQL  db-cluster-1:33060+ ssl  JS > var cluster = dba.rebootClusterFromCompleteOutage("mycluster1", {primary: "db-cluster-1:3306"});
Dba.rebootClusterFromCompleteOutage: Argument #2: Invalid options: primary (ArgumentError)

So no cluster is available and the database and its data is inaccessible.
Indeed, the initial state of the cluster was really bad and before the restart, the two of three servers were missing or in bad state.

[root@db-cluster-1 ~]# mysqlsh
MySQL Shell 8.0.28

Copyright (c) 2016, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates.
Other names may be trademarks of their respective owners.

Type '\help' or '\?' for help; '\quit' to exit.
 MySQL  JS > \connect clusteradmin@db-cluster-1
Creating a session to 'clusteradmin@db-cluster-1'
Fetching schema names for autocompletion... Press ^C to stop.
Your MySQL connection id is 241708346 (X protocol)
Server version: 8.0.28 MySQL Community Server - GPL
No default schema selected; type \use <schema> to set one.
 MySQL  db-cluster-1:33060+ ssl  JS > var cluster = dba.getCluster()
 MySQL  db-cluster-1:33060+ ssl  JS > cluster.status()
{
    "clusterName": "mycluster1", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "db-cluster-1:3306", 
        "ssl": "REQUIRED", 
        "status": "OK_NO_TOLERANCE", 
        "statusText": "Cluster is NOT tolerant to any failures. 2 members are not active.", 
        "topology": {
            "db-cluster-1:3306": {
                "address": "db-cluster-1:3306", 
                "memberRole": "PRIMARY", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "replicationLag": null, 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.28"
            }, 
            "db-cluster-2:3306": {
                "address": "db-cluster-2:3306", 
                "instanceErrors": [
                    "NOTE: group_replication is stopped."
                ], 
                "memberRole": "SECONDARY", 
                "memberState": "OFFLINE", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "(MISSING)", 
                "version": "8.0.28"
            }, 
            "db-cluster-3:3306": {
                "address": "db-cluster-3:3306", 
                "instanceErrors": [
                    "ERROR: GR Recovery channel receiver stopped with an error: error connecting to master 'mysql_innodb_cluster_2324239842@db-cluster-1:3306' - retry-time: 60 retries: 1 message: Access denied for user 'mysql_innodb_cluster_2324239842'@'10.10.10.11' (using password: YES) (1045) at 2023-09-19 04:37:00.076960", 
                    "ERROR: group_replication has stopped with an error."
                ], 
                "memberRole": "SECONDARY", 
                "memberState": "ERROR", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "(MISSING)", 
                "version": "8.0.28"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "db-cluster-1:3306"
}

The problem here is the MySQL version is 8.0.28, but after MySQL 8.0.30 there are much more features, which can be used in the second argument of rebootClusterFromCompleteOutage including, which server should be considered primary therefore healthy. In fact, the updated rebootClusterFromCompleteOutage of MySQL 8.0.34 version even auto-detected the correct and healthy node and booted the MySQL InnoDB Cluster.
There were no problems with the update from MySQL 8.0.28 to MySQL 8.0.34 and after the MySQL 8.0.34 started, the rebootClusterFromCompleteOutage reconfigured and started the cluster with the right and healthy server auto-detected. In fact, it is safer to use the second argument and set the option, which is the healthy server “{primary: “db-cluster-1:3306″}”.
Keep on reading!

binutils and the error ld: unrecognized option ‘–no-dynamic-linker’

Yet another bites of an old bintuils installed in the system, which leads to an error and failed a building of glibc this time. The last time it was a kernel building failure, check out here – . Most of the time, these kind of errors occurs when upgrading an old system, so as soon as building the new binutils package with emerge it is mandatory to remove the old one to minimize compiling errors of this sort.

main menu
building failure

This time the error under Gentoo system is (but it could happen in any system with old and new binutils!):

/usr/lib/gcc/x86_64-pc-linux-gnu/11/../../../../x86_64-pc-linux-gnu/bin/ld: unrecognized option '--no-dynamic-linker'
/usr/lib/gcc/x86_64-pc-linux-gnu/11/../../../../x86_64-pc-linux-gnu/bin/ld: use the --help option for usage information
collect2: error: ld returned 1 exit status

To fix the error simply remove the (all) old binutils package(s) with emerge command:

[root@srv ~]# emerge -vaC =binutils-2.25-r1
 * This action can remove important packages! In order to be safer, use
 * `emerge -pv --depclean <atom>` to check for reverse dependencies before
 * removing packages.

>>> These are the packages that would be unmerged:

 sys-devel/binutils
    selected: 2.25-r1 
   protected: none 
     omitted: 2.41-r1 

All selected packages: =sys-devel/binutils-2.25-r1

>>> 'Selected' packages are slated for removal.
>>> 'Protected' and 'omitted' packages will not be removed.

Would you like to unmerge these packages? [Yes/No] yes
>>> Waiting 5 seconds before starting...
>>> (Control-C to abort)...
>>> Unmerging in: 5 4 3 2 1
>>> Unmerging (1 of 1) sys-devel/binutils-2.25-r1...
No package files given... Grabbing a set.
<<<          obj /usr/x86_64-pc-linux-gnu/binutils-bin/2.25/strip
<<<          obj /usr/x86_64-pc-linux-gnu/binutils-bin/2.25/strings
<<<          obj /usr/x86_64-pc-linux-gnu/binutils-bin/2.25/size
<<<          obj /usr/x86_64-pc-linux-gnu/binutils-bin/2.25/readelf
......
......

Keep on reading!