nginx with SELinux and ngx_http_lua_module – PANIC: unprotected error in call to Lua API

main menu
runtime code generation failed restricted kernel

The CentOS 8 (CentOS Stream 9, too) might prevent to successfully execute a Lua code in NGINX web server with the modules like ngx_http_lua_module because of insufficient SELinux rules. By default, the web server is not allowed to execute programs that require memory addresses that are both executable and writable. Using Lua code may even crash with a core dump and restart of the NGIX worker process. The error logs contains information for the incident:

PANIC: unprotected error in call to Lua API (runtime code generation failed, restricted kernel?)

In fact, the error message hints what is going on – that the kernel prevents the execution of a code, which might be an addition limit enforced to this system.
Searching through the SELinux audit log shows the solution by enable the httpd_execmem SELinux boolean.

[root@srv ]# ausearch -c 'nginx' --raw | audit2allow


#============= httpd_t ==============

#!!!! This avc can be allowed using the boolean 'httpd_execmem'
allow httpd_t self:process execmem;

Apparently, the Lua code used needs a privileges to execute a code in the memory addresses (execmem). So when using the OpenResty ngx_http_lua_module, which is not distributed with the NGINX server, the server administrator should enable to true the SELinux boolean – httpd_execmem with:

setsebool -P httpd_execmem=1

The “-P” is going to make the value persistent over reboots.

Note, the official manual for the SELinux boolean. By enabling the httpd_execmem, several SELinux rules are applied, which may make the web server less secure, because the web server process is allowed to execute code in the memory.

– httpd_execmem

When enabled, this Boolean allows httpd to execute programs that require memory addresses that are both executable and writable. Enabling this Boolean is not recommended from a security standpoint as it reduces protection against buffer overflows, however certain modules and applications (such as Java and Mono applications) require this privilege.

The AVC (Access Vector Cache) message looks like:

type=AVC msg=audit(1699774004.724:59165857): avc:  denied  { execmem } for  pid=3872876 comm="nginx" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=process permissive=0
type=SYSCALL msg=audit(1699774004.724:59165857): arch=c000003e syscall=10 success=no exit=-13 a0=7f399cd69000 a1=10000 a2=5 a3=38 items=0 ppid=884311 pid=3872876 auid=4294967295 uid=990 gid=987 euid=990 suid=990 fsuid=990 egid=987 sgid=987 fsgid=987 tty=(none) ses=4294967295 comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0 key=(null)ARCH=x86_64 SYSCALL=mprotect AUID="unset" UID="nginx" GID="nginx" EUID="nginx" SUID="nginx" FSUID="nginx" EGID="nginx" SGID="nginx" FSGID="nginx"
type=PROCTITLE msg=audit(1699774004.724:59165857): proctitle=6E67696E783A20776F726B65722070726F63657373

More on NGINXhttps://ahelpme.com/tag/nginx/ or SELinuxhttps://ahelpme.com/tag/selinux/.

Recover MySQL InnoDB Cluster and Dba.rebootClusterFromCompleteOutage: Argument #2: Invalid options: primary (ArgumentError)

main menu
MySQL 8.0.28 version

Recent version of MySQL 8 implemented more options to the rebootClusterFromCompleteOutage function! Definitely check the link’s manual above and most of the handy second options are implemented in MySQL 8.0.30, so the user’s MySQL InnoDB Cluster crashed and if rebootClusterFromCompleteOutage should be used, but it outputs an error sort of:

 MySQL  db-cluster-1:33060+ ssl  JS > var cluster = dba.rebootClusterFromCompleteOutage()
Restoring the cluster 'mycluster1' from complete outage...

Dba.rebootClusterFromCompleteOutage: Target member is in state ERROR (RuntimeError)

And when trying to use the node, which was healthy before the crash with this function, there is an error, too:

 MySQL  db-cluster-1:33060+ ssl  JS > var cluster = dba.rebootClusterFromCompleteOutage("mycluster1", {primary: "db-cluster-1:3306"});
Dba.rebootClusterFromCompleteOutage: Argument #2: Invalid options: primary (ArgumentError)

So no cluster is available and the database and its data is inaccessible.
Indeed, the initial state of the cluster was really bad and before the restart, the two of three servers were missing or in bad state.

[root@db-cluster-1 ~]# mysqlsh
MySQL Shell 8.0.28

Copyright (c) 2016, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates.
Other names may be trademarks of their respective owners.

Type '\help' or '\?' for help; '\quit' to exit.
 MySQL  JS > \connect clusteradmin@db-cluster-1
Creating a session to 'clusteradmin@db-cluster-1'
Fetching schema names for autocompletion... Press ^C to stop.
Your MySQL connection id is 241708346 (X protocol)
Server version: 8.0.28 MySQL Community Server - GPL
No default schema selected; type \use <schema> to set one.
 MySQL  db-cluster-1:33060+ ssl  JS > var cluster = dba.getCluster()
 MySQL  db-cluster-1:33060+ ssl  JS > cluster.status()
{
    "clusterName": "mycluster1", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "db-cluster-1:3306", 
        "ssl": "REQUIRED", 
        "status": "OK_NO_TOLERANCE", 
        "statusText": "Cluster is NOT tolerant to any failures. 2 members are not active.", 
        "topology": {
            "db-cluster-1:3306": {
                "address": "db-cluster-1:3306", 
                "memberRole": "PRIMARY", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "replicationLag": null, 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.28"
            }, 
            "db-cluster-2:3306": {
                "address": "db-cluster-2:3306", 
                "instanceErrors": [
                    "NOTE: group_replication is stopped."
                ], 
                "memberRole": "SECONDARY", 
                "memberState": "OFFLINE", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "(MISSING)", 
                "version": "8.0.28"
            }, 
            "db-cluster-3:3306": {
                "address": "db-cluster-3:3306", 
                "instanceErrors": [
                    "ERROR: GR Recovery channel receiver stopped with an error: error connecting to master 'mysql_innodb_cluster_2324239842@db-cluster-1:3306' - retry-time: 60 retries: 1 message: Access denied for user 'mysql_innodb_cluster_2324239842'@'10.10.10.11' (using password: YES) (1045) at 2023-09-19 04:37:00.076960", 
                    "ERROR: group_replication has stopped with an error."
                ], 
                "memberRole": "SECONDARY", 
                "memberState": "ERROR", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "(MISSING)", 
                "version": "8.0.28"
            }
        }, 
        "topologyMode": "Single-Primary"
    }, 
    "groupInformationSourceMember": "db-cluster-1:3306"
}

The problem here is the MySQL version is 8.0.28, but after MySQL 8.0.30 there are much more features, which can be used in the second argument of rebootClusterFromCompleteOutage including, which server should be considered primary therefore healthy. In fact, the updated rebootClusterFromCompleteOutage of MySQL 8.0.34 version even auto-detected the correct and healthy node and booted the MySQL InnoDB Cluster.
There were no problems with the update from MySQL 8.0.28 to MySQL 8.0.34 and after the MySQL 8.0.34 started, the rebootClusterFromCompleteOutage reconfigured and started the cluster with the right and healthy server auto-detected. In fact, it is safer to use the second argument and set the option, which is the healthy server “{primary: “db-cluster-1:3306″}”.
Keep on reading!

binutils and the error ld: unrecognized option ‘–no-dynamic-linker’

Yet another bites of an old bintuils installed in the system, which leads to an error and failed a building of glibc this time. The last time it was a kernel building failure, check out here – . Most of the time, these kind of errors occurs when upgrading an old system, so as soon as building the new binutils package with emerge it is mandatory to remove the old one to minimize compiling errors of this sort.

main menu
building failure

This time the error under Gentoo system is (but it could happen in any system with old and new binutils!):

/usr/lib/gcc/x86_64-pc-linux-gnu/11/../../../../x86_64-pc-linux-gnu/bin/ld: unrecognized option '--no-dynamic-linker'
/usr/lib/gcc/x86_64-pc-linux-gnu/11/../../../../x86_64-pc-linux-gnu/bin/ld: use the --help option for usage information
collect2: error: ld returned 1 exit status

To fix the error simply remove the (all) old binutils package(s) with emerge command:

[root@srv ~]# emerge -vaC =binutils-2.25-r1
 * This action can remove important packages! In order to be safer, use
 * `emerge -pv --depclean <atom>` to check for reverse dependencies before
 * removing packages.

>>> These are the packages that would be unmerged:

 sys-devel/binutils
    selected: 2.25-r1 
   protected: none 
     omitted: 2.41-r1 

All selected packages: =sys-devel/binutils-2.25-r1

>>> 'Selected' packages are slated for removal.
>>> 'Protected' and 'omitted' packages will not be removed.

Would you like to unmerge these packages? [Yes/No] yes
>>> Waiting 5 seconds before starting...
>>> (Control-C to abort)...
>>> Unmerging in: 5 4 3 2 1
>>> Unmerging (1 of 1) sys-devel/binutils-2.25-r1...
No package files given... Grabbing a set.
<<<          obj /usr/x86_64-pc-linux-gnu/binutils-bin/2.25/strip
<<<          obj /usr/x86_64-pc-linux-gnu/binutils-bin/2.25/strings
<<<          obj /usr/x86_64-pc-linux-gnu/binutils-bin/2.25/size
<<<          obj /usr/x86_64-pc-linux-gnu/binutils-bin/2.25/readelf
......
......

Keep on reading!

how to skip mysqlbinlog reading error – ERROR: Found out of order GTID

main menu
MariaDB select gtid_strict_mode

When trying to read the MariaDB binlog files, the reading may be interrupted with an error message, and the reading will stop:

ERROR: Found out of order GTID. Got 0-3-855835750755 after 0-1-855835750756

Some positions are shown but after 24 lines of 1236849 file, the mariadb-binlog / mysqlbinlog program interrupts with an error.

It turns out the problem is that the mariadb-binlog / mysqlbinlog reads the binlog files with GTID strict mode, because it is enabled by default! But the servers may have turned it off, so the binlog files are in the wrong format for the strict mode reading. The mode to read the binlog files should be the same as the MySQL / MariaDB server mode that created the binlog files.

root@srv binlog # mariadb-binlog  mysql-bin.52349 
/*!50530 SET @@SESSION.PSEUDO_SLAVE_MODE=1*/;
/*!40019 SET @@session.max_delayed_threads=0*/;
/*!50003 SET @OLD_COMPLETION_TYPE=@@COMPLETION_TYPE,COMPLETION_TYPE=0*/;
DELIMITER /*!*/;
# at 4
#230820 10:53:45 server id 1  end_log_pos 256 CRC32 0xffa88ac1        Start: binlog v 4, server v 11.0.2-MariaDB-1:11.0.2+maria~ubu2204-log created 230820 10:53:45
BINLOG '
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA
AAAAAAAAAAAEEwwwwwwwwwwwwww
'/*!*/;
# at 256
#230820 10:53:45 server id 1  end_log_pos 379 CRC32 0xd434c2c6        Gtid list [0-5-76636859354,
# 0-4-847593757253,
# 0-2-857258567526,
# 0-3-855835750755,
# 0-1-855835750756,
# 1-4-8637504]
# at 379
#230820 10:53:45 server id 1  end_log_pos 423 CRC32 0xc23ef7af        Binlog checkpoint mysql-bin.52348
ERROR: Found out of order GTID. Got 0-3-855835750755 after 0-1-855835750756

Show the slave status to determine, which mode is in use:

root@srv binlog # mysql
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 371408455
Server version: 11.0.2-MariaDB-1:11.0.2+maria~ubu2204-log mariadb.org binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> select @@gtid_strict_mode;
+--------------------+
| @@gtid_strict_mode |
+--------------------+
|                  0 |
+--------------------+
1 row in set (0.000 sec)

MariaDB [(none)]>

When the binlog files are generated with gtid_strict_mode = 0 the “–skip-gtid-strict-mode” option should be used with mariadb-binlog / mysqlbinlog to read the binlog files without errors. More on gtid_strict_mode for MariaDB – https://mariadb.com/docs/skysql-dbaas/ref/mdb/system-variables/gtid_strict_mode/.
Related errors GTID slave position – MariaDB/MySQL replication error – Error during XID COMMIT: failed to update GTID state in mysql.gtid_slave_pos. More on MariaDB on this site with tag mariadb tag.

Missing the CentOS Stream 9 CRB repository – nothing provides python3-pyxattr needed by

CentOS Stream 9 CRB repository is the name of the repository, which replaces the old CentOS Stream 8 PowerTools repository.

main menu
enable CRB
The CRB is an official repository, which stands for CodeReady Linux Builder repository. It includes multiple important packages mainly for developer packages (those with “-devel” in the name). The CRB packages may be found here: https://mirror.stream.centos.org/9-stream/CRB/x86_64/os/Packages/.
When installing packages from community official or other repositories they may depend on packages in CRB repositories, but because it is not enabled by default, there will be a nasty error of broken dependencies like:

Error: 
 Problem: cannot install the best candidate for the job
  - nothing provides python3-pyxattr needed by glusterfs-server-11.0-2.el9s.x86_64 from centos-gluster11-test
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)

The package glusterfs-server-11.0-2.el9s.x86_64 needs the package python3-pyxattr, which cannot be found in all the enabled repositories on the system, so it appears the system is broken.

So the python3-pyxattr is part of the CRB repository so just enabling it will solve the problem:

[root@srv ~]# dnf config-manager --set-enabled crb
[root@srv ~]# dnf install -y glusterfs-server
CentOS Stream 9 - BaseOS                                                                 198 kB/s | 9.5 kB     00:00    
CentOS Stream 9 - AppStream                                                               26 kB/s |  10 kB     00:00    
CentOS Stream 9 - CRB                                                                    8.0 MB/s | 5.4 MB     00:00    
Dependencies resolved.
=========================================================================================================================
 Package                                Architecture    Version                     Repository                      Size
=========================================================================================================================
Installing:
 glusterfs-server                       x86_64          11.0-2.el9s                 centos-gluster11-test          1.2 M
Installing dependencies:
 attr                                   x86_64          2.5.1-3.el9                 baseos                          61 k
 device-mapper-event                    x86_64          9:1.02.195-1.el9            baseos                          33 k
 device-mapper-event-libs               x86_64          9:1.02.195-1.el9            baseos                          32 k
 device-mapper-persistent-data          x86_64          0.9.0-13.el9                baseos                         782 k
 glusterfs-cli                          x86_64          11.0-2.el9s                 centos-gluster11-test          185 k
 glusterfs-client-xlators               x86_64          11.0-2.el9s                 centos-gluster11-test          785 k
 glusterfs-fuse                         x86_64          11.0-2.el9s                 centos-gluster11-test          136 k
 glusterfs-selinux                      noarch          2.0.1-1.el9s                centos-gluster11                29 k
 libaio                                 x86_64          0.3.111-13.el9              baseos                          24 k
 libgfapi0                              x86_64          11.0-2.el9s                 centos-gluster11-test           95 k
 libgfchangelog0                        x86_64          11.0-2.el9s                 centos-gluster11-test           34 k
 lvm2                                   x86_64          9:2.03.21-1.el9             baseos                         1.5 M
 lvm2-libs                              x86_64          9:2.03.21-1.el9             baseos                         1.0 M
 python3-pyxattr                        x86_64          0.7.2-4.el9                 crb                             35 k
 rpcbind                                x86_64          1.2.6-5.el9                 baseos                          58 k

Transaction Summary
=========================================================================================================================
Install  16 Packages

Total download size: 6.0 M
.....
.....
  python3-pyxattr-0.7.2-4.el9.x86_64                         rpcbind-1.2.6-5.el9.x86_64                                 

Complete!

Listing packages of the CRB repository is simple enough.

[root@srv ~]# dnf repository-packages crb list
Last metadata expiration check: 1:26:16 ago on Mon 19 Jun 2023 12:50:59 PM UTC.
Installed Packages
python3-pyxattr.x86_64                       0.7.2-4.el9                        @crb
Available Packages
CUnit-devel.i686                             2.1.3-25.el9                       crb 
CUnit-devel.x86_64                           2.1.3-25.el9                       crb 
Judy-devel.i686                              1.0.5-28.el9                       crb 
Judy-devel.x86_64                            1.0.5-28.el9                       crb 
LibRaw-devel.i686                            0.20.2-6.el9                       crb 
LibRaw-devel.x86_64                          0.20.2-6.el9                       crb 
.....
.....

Apparently, CentOS Stream 9 installation should include EPEL and CRB repositories in addition to the base ones.
Almost half of the files are developments files (i.e. “-devel”) packages and others are additional libraries, mainly Python 3 and Perl modules, OpenJDK 17, 11, 1.8.0 slow debug and fast debug, and more.

yii2 and Bad Request (#400) with Missing required parameters: ID

Using Yii 2 is really easy, but for a newbie sometimes the Gii (i.e. the code generator helper tool) may generate a code, which only the index page works and the other pages like view, update generates a strange error of missing ID (note it is capitalized ID):


Bad Request (#400)
Missing required parameters: ID

The above error occurred while the Web server was processing your request.

Please contact us if you think this is a server error. Thank you.

SCREENSHOT 1) Just viewing an item generates the error of a bad request.

main menu
view with id

Here is the tracing log:

error	yii\web\HttpException:400	yii\web\BadRequestHttpException: Missing required parameters: ID in /home/myuser/projects/test-web/vendor/yiisoft/yii2/web/Controller.php:202
Stack trace:
#0 /home/myuser/projects/test-web/vendor/yiisoft/yii2/base/InlineAction.php(51): yii\web\Controller->bindActionParams()
#1 /home/myuser/projects/test-web/vendor/yiisoft/yii2/base/Controller.php(178): yii\base\InlineAction->runWithParams()
#2 /home/myuser/projects/test-web/vendor/yiisoft/yii2/base/Module.php(552): yii\base\Controller->runAction()
#3 /home/myuser/projects/test-web/vendor/yiisoft/yii2/web/Application.php(103): yii\base\Module->runAction()
#4 /home/myuser/projects/test-web/vendor/yiisoft/yii2/base/Application.php(384): yii\web\Application->handleRequest()
#5 /home/myuser/projects/test-web/web/index.php(12): yii\base\Application->run()
#6 {main}

Probably, the problem is in the Controller class of the route you are trying to view. For example, the above URL is http://localhost:8888/odd-type/view?id=1 the route is /odd-type/view and apparently, the OddTypeController class may not find the ID parameter.
SCREENSHOT 2) The function actionView has the parameter $ID, which is used in the function findModel($ID).

main menu
actionView with capitalized ID

Keep on reading!

Gentoo clang-15: error: does not contain a GCC installation

main menu
emerge firefox

Trying to build a package resulted in a building failure because a Clang could not find GNU GCC installation as the error shows.

Executing just x86_64-pc-linux-gnu-clang-15 the same error.

[root@srv ~]# x86_64-pc-linux-gnu-clang-15 
clang-15: error: '/usr/lib/gcc/x86_64-pc-linux-gnu/12.2.0' does not contain a GCC installation
clang-15: error: no input files

Apparently, because the latest upgrade of GNU GCC went from 12.2.0 to 12.2.1_p20221008 and the directory changed to /usr/lib/gcc/x86_64-pc-linux-gnu/12.
The Clang binaries read several configuration files and one of them was not updated when the GCC had been upgraded. The configuration file /etc/clang/gentoo-gcc-install.cfg has the wrong path, because the gcc-config version was an old one or the configuration file is generated only on GCC major version, not in the minor. But the last upgrade didn’t modified the Clang configuration file /etc/clang/gentoo-gcc-install.cfg

To resolve this issue, the user may edit the file manually or just use gcc-config to revert to the older and then to the new GCC version at once. List the currently installed GCC versions in the system and choose one.

[root@srv ~]# gcc-config -l
 [1] x86_64-pc-linux-gnu-8.2.0
 [2] x86_64-pc-linux-gnu-8.3.0
 [3] x86_64-pc-linux-gnu-9.2.0
 [4] x86_64-pc-linux-gnu-10.3.0
 [5] x86_64-pc-linux-gnu-11.3.0
 [6] x86_64-pc-linux-gnu-12 *
[root@srv ~]# gcc-config x86_64-pc-linux-gnu-11.3.0
 * Switching native-compiler to x86_64-pc-linux-gnu-11.3.0 ...
>>> Regenerating /etc/ld.so.cache...                                                                  [ ok ]
 * If you intend to use the gcc from the new profile in an already
 * running shell, please remember to do:

 *   . /etc/profile

[root@srv ~]#  gcc-config x86_64-pc-linux-gnu-12
 * Switching native-compiler to x86_64-pc-linux-gnu-12 ...
>>> Regenerating /etc/ld.so.cache...                                                                  [ ok ]
 * If you intend to use the gcc from the new profile in an already
 * running shell, please remember to do:

 *   . /etc/profile
[root@srv ~]# . /etc/profile

Check the sys-devel/gcc-config for available upgrades and do them if there are!
More Gentoo tips here.

lxc_attach_run_shell: 1333 Permission denied – failed to exec shell

An annoying error when using the LXC container tools like lxc-attach, which is really simple to fix.

[root@srv ~]# lxc-attach -n db-cluster-3
lxc_container: attach.c: lxc_attach_run_shell: 1333 Permission denied - failed to exec shell
[root@srv ~]#

This error just reports the bash shell in the container cannot be started and the SELinux audit file adds some errors, too:

type=AVC msg=audit(1665745824.682:24229): avc:  denied  { entrypoint } for  pid=20646 comm="lxc-attach" path="/usr/bin/bash" dev="md3" ino=111806476 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=unconfined_u:object_r:var_log_t:s0 tclass=file
type=SYSCALL msg=audit(1665745824.682:24229): arch=c000003e syscall=59 success=no exit=-13 a0=24412c6 a1=7ffe87c07170 a2=2443870 a3=7ffe87c08c60 items=0 ppid=20644 pid=20646 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts12 ses=3304 comm="lxc-attach" exe="/usr/bin/lxc-attach" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=PROCTITLE msg=audit(1665745824.682:24229): proctitle=6C78632D617474616368002D6E0064622D636C75737465722D33
type=AVC msg=audit(1665745824.682:24230): avc:  denied  { entrypoint } for  pid=20646 comm="lxc-attach" path="/usr/bin/bash" dev="md3" ino=111806476 scontext=system_u:system_r:unconfined_service_t:s0 tcontext=unconfined_u:object_r:var_log_t:s0 tclass=file
type=SYSCALL msg=audit(1665745824.682:24230): arch=c000003e syscall=59 success=no exit=-13 a0=7f08b5e579a0 a1=7ffe87c07170 a2=2443870 a3=7ffe87c08c60 items=0 ppid=20644 pid=20646 auid=0 uid=0 gid=0 euid=0 suid=0 fsuid=0 egid=0 sgid=0 fsgid=0 tty=pts12 ses=3304 comm="lxc-attach" exe="/usr/bin/lxc-attach" subj=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023 key=(null)
type=PROCTITLE msg=audit(1665745824.682:24230): proctitle=6C78632D617474616368002D6E0064622D636C75737465722D33

So clearly, the problem is in SELinux, and turn it off temporarily with

setenforce 0

Turning off the SELinux is not the right thing! There are two aspects to the problem:

  • Missing SELinux rules, which are installed with a special package container-selinux
  • Wrong SELinux permissions for the LXC container’s root directory. In most cases, the user just changes the default /var/lib/lxc/[container] to something new and the LXC works, but it breaks some LXC parts.

Installing container-selinux is easy:

dnf install -y container-selinux

Or the old yum:

yum install -y container-selinux

Then check the SELinux attributes with:

[root@srv ~]# ls -altrZ /mnt/storage/servers/mycontainer/
drwxr-xr-x. root root unconfined_u:object_r:var_log_t:s0 ..
-rw-r--r--. root root unconfined_u:object_r:var_log_t:s0 config
drwxrwx---. root root unconfined_u:object_r:var_log_t:s0 .
drwxr-xr-x. root root unconfined_u:object_r:var_log_t:s0 rootfs

The problem is var_log_t, which is an SELinux file context and it should be container_var_lib_t. Stop the container and fix the permissions. If the default directory (/var/lib/lxc) were used, it would not have this problem. Adding the SELinux file context definition to the new directory is mandatory when changing the directory root of a container:

[root@srv ~]# semanage fcontext -a -t container_var_lib_t '/mnt/storage/servers/mycontainer(/.*)?'
[root@srv ~]# restorecon -Rv /mnt/storage/servers/mycontainer/
restorecon reset /mnt/storage/servers/mycontainer context unconfined_u:object_r:var_log_t:s0->unconfined_u:object_r:container_var_lib_t:s0
.....
.....
restorecon reset /mnt/storage/servers/mycontainer/config context unconfined_u:object_r:var_log_t:s0->unconfined_u:object_r:container_var_lib_t:s0

All files permissions under /mnt/storage/servers/mycontainer/ should be fixed with the restorecon. Start the LXC container and try to attach it with lxc-attach. Now, there should not be any errors:

[root@srv ~]# lxc-attach -n mycontainer
[root@mycontainer ~]#

The files’ context is the right one – container_var_lib_t:

[root@srv ~]# ls -altrZ /mnt/storage/servers/mycontainer/
drwxr-xr-x. root root unconfined_u:object_r:var_log_t:s0 ..
-rw-r--r--. root root unconfined_u:object_r:container_var_lib_t:s0 config
drwxrwx---. root root unconfined_u:object_r:container_var_lib_t:s0 .
drwxr-xr-x. root root unconfined_u:object_r:container_var_lib_t:s0 rootfs

More on LXC containershttps://ahelpme.com/category/software/lxc/.

Recover from Unable to fetch live group_replication member data from any server in cluster

main menu
MySQL Router Unable to fetch live group_replication

After multiple networking connectivity issues between MySQL InnoDB Cluster nodes, the cluster may break and the MySQL Router begins to log the following messages:

2022-10-11 15:20:48 metadata_cache ERROR [7f2d619fe640] Unable to fetch live group_replication member data from any server in cluster 'mycluster1'
2022-10-11 15:20:49 metadata_cache WARNING [7f2d619fe640] Member db-cluster-2:3306 (05b6c7c7-f285-11ec-adfc-00163e0b38ff) defined in metadata not found in actual Group Replication
2022-10-11 15:20:49 metadata_cache WARNING [7f2d619fe640] db-cluster-1:3306 is not part of quorum for cluster 'mycluster1'
2022-10-11 15:20:49 metadata_cache WARNING [7f2d619fe640] Member db-cluster-1:3306 (8bf2c25f-90ae-11ec-93d1-00163e20a401) defined in metadata not found in actual Group Replication
2022-10-11 15:20:49 metadata_cache WARNING [7f2d619fe640] db-cluster-3:3306 is not part of quorum for cluster 'mycluster1'
2022-10-11 15:20:49 metadata_cache WARNING [7f2d619fe640] Member db-cluster-3:3306 (99856952-90ae-11ec-9a5f-fafd8f1acc17) defined in metadata not found in actual Group Replication
2022-10-11 15:20:49 metadata_cache WARNING [7f2d619fe640] db-cluster-2:3306 is not part of quorum for cluster 'mycluster1'

And in MySQL nodes there are also the errors of unable to connect to 33061:

2022-10-11T15:16:25.728393Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error on opening a connection to peer node db-cluster-1:33061 when joining a group. My local port is: 33061.'
2022-10-11T15:16:25.728714Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error on opening a connection to peer node db-cluster-3:33061 when joining a group. My local port is: 33061.'
2022-10-11T15:16:25.729195Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error on opening a connection to peer node db-cluster-1:33061 when joining a group. My local port is: 33061.'
2022-10-11T15:16:25.729569Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error on opening a connection to peer node db-cluster-3:33061 when joining a group. My local port is: 33061.'
2022-10-11T15:16:25.730154Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error on opening a connection to peer node db-cluster-1:33061 when joining a group. My local port is: 33061.'
2022-10-11T15:16:25.730474Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error on opening a connection to peer node db-cluster-3:33061 when joining a group. My local port is: 33061.'
2022-10-11T15:16:25.730485Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] Error connecting to all peers. Member join failed. Local port: 33061'
2022-10-11T15:16:25.782015Z 0 [ERROR] [MY-011735] [Repl] Plugin group_replication reported: '[GCS] The member was unable to join the group. Local port: 33061'

When a MySQL Cluster node node cannot connect to the 33061 (or 3306X), it may be a signal for a firewall issue or the group replication has not started on this node, which is the case here!

The group replication is not working and the cluster must be recovered. All nodes wait to join an exiting cluster, which is not available. It has not stared yet and it would not start alone even the administrator may restart all the nodes.
Keep on reading!

Delete Glusterfs volume when a peer is down – failed: Some of the peers are down

Deleting GlusterFS volumes may fail with an error, pointing out some of the peers are down, i.e. they are disconnected. Even all the volume’s peers of the volume the user is trying to delete are available, still the error appears and it is not possible to delete the volume.
That’s because GlusterFS by design stores the volume configuration spread to all peers – no matter they host a brick/arbiter of the volume or not. If a peer is a part of a GlusterFS setup, it is mandatory to be available and online in the peer status, to be able to delete a volume.
If the user still wants to delete the volume:

  1. * Force remove the brink, which was hosted on the detached peer. If any!
  2. Detach the disconnected peer from the peers
  3. Delete the volume

Here are real examples with and without a brick on the unavailable peer.
The initial volumes and peers configuration:

[root@srv1 ~]# gluster volume info
 
Volume Name: VOL1
Type: Replicate
Volume ID: 02ff2995-7307-4f3d-aa24-862edda7ce81
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: ng1:/mnt/storage1/glusterfs/brick1
Brick2: ng3:/mnt/storage1/glusterfs/brick1
Brick3: ng1:/mnt/storage1/glusterfs/arbiter1 (arbiter)
Options Reconfigured:
features.scrub: Active
features.bitrot: on
cluster.self-heal-daemon: enable
storage.linux-io_uring: off
client.event-threads: 4
performance.cache-max-file-size: 50MB
performance.parallel-readdir: on
network.inode-lru-limit: 200000
performance.md-cache-timeout: 600
performance.cache-invalidation: on
performance.stat-prefetch: on
features.cache-invalidation-timeout: 600
features.cache-invalidation: on
performance.cache-size: 2048MB
performance.client-io-threads: on
nfs.disable: on
transport.address-family: inet
 
Volume Name: VOL2
Type: Replicate
Volume ID: fc2e82e4-2576-4bb1-b9bf-c6b2aff10ef0
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x (2 + 1) = 3
Transport-type: tcp
Bricks:
Brick1: ng1:/mnt/storage1/glusterfs/brick2
Brick2: ng2:/mnt/storage1/glusterfs/brick2
Brick3: ng1:/mnt/storage1/glusterfs/arbiter2 (arbiter)
Options Reconfigured:
features.scrub: Active
features.bitrot: on
cluster.self-heal-daemon: enable
storage.linux-io_uring: off
performance.parallel-readdir: on
network.compression: off
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
features.cache-invalidation: on

[root@srv ~]# gluster peer status
Number of Peers: 2

Hostname: ng1
Uuid: 7953514b-b52c-4a5c-be03-763c3e24eb4e
State: Peer in Cluster (Connected)

Hostname: ng3
Uuid: 3d273834-eca6-4997-871f-1a282ca90fb0
State: Peer in Cluster (Disconnected)

Delete a GlusterFS volume – all bricks and bricks’ peers are available, but another peer is not.

First, the error, when the disconnected peer is still in peer status list.

[root@srv ~]# gluster volume stop VOL2
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: VOL2: success
[root@srv ~]# gluster volume delete VOL2
Deleting volume will erase all information about the volume. Do you want to continue? (y/n) y
volume delete: VOL2: failed: Some of the peers are down

Keep on reading!