Tag: selinux

LXC with SELinux and NFS share result in kernel: SELinux: inode_doinit_use_xattr: getxattr returned 2 for dev=0:43 ino=

After staring a new LXC container, the syslog program (Syslog-ng) began to throw thousands of errors with this kind of message:

Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6977140995289226736
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6551465724643968476
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-5980833553552494142
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-8820947409424952637
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-8270463809263745561
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7923279144252216900
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-6181977668994943343
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7585065875445167421
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7923279144252216900
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-5826517164673898101
Dec  1 10:50:36 srv kernel: SELinux: inode_doinit_use_xattr:  getxattr returned 2 for dev=0:43 ino=-7585065875445167421
Dec  1 11:01:01 h3 rsyslogd[1147]: imjournal: 3871493 messages lost due to rate-limiting (20000 allowed within 600 seconds)

These messages were logged in thousands. The same time, the NFS statistics showed a strange peak of using getattr. Something was calling getattr thousands times per second. Despite there were no SELinux blocks in audit.log as the dmesg suggested the SELinux might be blamed.
The LXC container is an application container, which has mound bind directory from the host server. The very same directory is an local NFS share (using NFS-Ganesha) of a GlusterFS volume and the PHP files are situated there.

main menu
kernel SELinux inode_doinit_use_xattr getxattr returned 2 nfsstat getattr graph

So the LXC container reads the PHP files from this NFS share. There were no issues to access the files and the application LXC worked just fine.
The problem disappeared when the NFS share was remounted with SELinux permissions using the context word:

node3:/VOL1 /mnt/nfs/VOL1 nfs defaults,hard,noexec,nosuid,_netdev,fsc,noatime,context="system_u:object_r:httpd_sys_rw_content_t:s0" 0 0

All the files are of SELinux label httpd_sys_rw_content_t and after restarting the LXC container there were no SELinux lines in the dmesg and the syslog logs. The administrator should configure the right SELinux permissions to the LXC bound directories. More on why SELinux sometimes does not report on blocks in the audit.log here – Selinux permission denied and no log in audit.log.

nginx with SELinux and ngx_http_lua_module – PANIC: unprotected error in call to Lua API

main menu
runtime code generation failed restricted kernel

The CentOS 8 (CentOS Stream 9, too) might prevent to successfully execute a Lua code in NGINX web server with the modules like ngx_http_lua_module because of insufficient SELinux rules. By default, the web server is not allowed to execute programs that require memory addresses that are both executable and writable. Using Lua code may even crash with a core dump and restart of the NGIX worker process. The error logs contains information for the incident:

PANIC: unprotected error in call to Lua API (runtime code generation failed, restricted kernel?)

In fact, the error message hints what is going on – that the kernel prevents the execution of a code, which might be an addition limit enforced to this system.
Searching through the SELinux audit log shows the solution by enable the httpd_execmem SELinux boolean.

[root@srv ]# ausearch -c 'nginx' --raw | audit2allow


#============= httpd_t ==============

#!!!! This avc can be allowed using the boolean 'httpd_execmem'
allow httpd_t self:process execmem;

Apparently, the Lua code used needs a privileges to execute a code in the memory addresses (execmem). So when using the OpenResty ngx_http_lua_module, which is not distributed with the NGINX server, the server administrator should enable to true the SELinux boolean – httpd_execmem with:

setsebool -P httpd_execmem=1

The “-P” is going to make the value persistent over reboots.

Note, the official manual for the SELinux boolean. By enabling the httpd_execmem, several SELinux rules are applied, which may make the web server less secure, because the web server process is allowed to execute code in the memory.

– httpd_execmem

When enabled, this Boolean allows httpd to execute programs that require memory addresses that are both executable and writable. Enabling this Boolean is not recommended from a security standpoint as it reduces protection against buffer overflows, however certain modules and applications (such as Java and Mono applications) require this privilege.

The AVC (Access Vector Cache) message looks like:

type=AVC msg=audit(1699774004.724:59165857): avc:  denied  { execmem } for  pid=3872876 comm="nginx" scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:system_r:httpd_t:s0 tclass=process permissive=0
type=SYSCALL msg=audit(1699774004.724:59165857): arch=c000003e syscall=10 success=no exit=-13 a0=7f399cd69000 a1=10000 a2=5 a3=38 items=0 ppid=884311 pid=3872876 auid=4294967295 uid=990 gid=987 euid=990 suid=990 fsuid=990 egid=987 sgid=987 fsgid=987 tty=(none) ses=4294967295 comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0 key=(null)ARCH=x86_64 SYSCALL=mprotect AUID="unset" UID="nginx" GID="nginx" EUID="nginx" SUID="nginx" FSUID="nginx" EGID="nginx" SGID="nginx" FSGID="nginx"
type=PROCTITLE msg=audit(1699774004.724:59165857): proctitle=6E67696E783A20776F726B65722070726F63657373

More on NGINX – https://ahelpme.com/tag/nginx/ or SELinux – https://ahelpme.com/tag/selinux/.

Software and technical details of Fedora Server 39 including cockpit screenshots

main menu
cockpit overview on Fedora Linux 39

This article is for those of you who do not want to install a whole new operating system only to discover some technical details about the default installation like disk layout, packages included, software versions, and so on. Here we are going to review in several sections what is like to have a default installation of Fedora 39 Server using a real not virtual machine!
The kernel is 6.2.15 it detects successfully the Threadripper 1950X AMD and the system is stable (we booted in UEFI mode).
The installation procedure uses default options for all installation setups – Minimal network installation of Fedora 39 Server.

Software

With Fedora Server 39 you can have

  • linux kernel – 6.5.10-300 (6.5.10-300.fc39.x86_64)
  • System
    • linux-firmware – version: 20231030, release: 20231030-1.fc39.
    • libc – 2.38 (2.38-10.fc39)
    • GNU GCC – 13.2.1 (13.2.1-4.fc39)
    • OpenSSL – 3.1.1 (3.1.1-4.fc39) and 1.1.1q (1.1.1q-5.fc39)
    • coreutils – 9.3 (9.3-4.fc39)
    • yum – Depricated and replaced with dnf
    • dnf – 4.18.0 (4.18.0-2.fc39). The dnf5 is available, too (5.1.5-1.fc39)
    • rsyslog – 8.2310.0 (8.2310.0-1.fc39)
    • NetworkManager – 1.44.2 (1.44.2-1.fc39)
  • Servers
    • Apache – 2.4.58 (2.4.58-1.fc39)
    • Nginx – 1.24.0 (1.24.0-4.fc39)
    • MySQL server – 8.0.34 (8.0.34-2.fc39)
    • MariaDB server – 10.5.22 (10.5.22-1.fc39)
    • PostgreSQL – 15.4 (15.4-1.fc39)
  • Programming
    • PHP – 8.2.12 (8.2.12-1.fc39)
    • python – The default is 3.12.0 (3.12.0-1.fc39) and many more available – 3.13.0 (3.13.0~a1-1.fc39), 3.11.6 (3.11.6-1.fc39), 3.10.13 (3.10.13-1.fc39), 3.9.18 (3.9.18-1.fc39), 3.8.18 (3.8.18-1.fc39), 3.7.17 (3.7.17-3.fc39), 3.6.15 (3.6.15-20.fc39) and also includes the older 2.7.18 (2.7.18-35.fc39)
    • perl – 5.38.0 (5.38.0-501.fc39)
    • ruby – 3.2.2 (3.2.2-181.fc39)
    • OpenJDK – the latest 21 – 21.0.0.0.35 (21.0.0.0.35-1.rolling.fc39) and also includes 17.0.8.0.7 (17.0.8.0.7-1.fc39), 11.0.20.0.8 (11.0.20.0.8-1.fc39) and 1.8.0.382.b05 (1.8.0.382.b05-2.fc39)
    • Go – 1.21.3 (1.21.3-1.fc39)
    • Rust – 1.73.0-1.fc39 (1.73.0-1.fc39)
    • llvm – the latest 17.0.3 (17.0.3-1.fc39), 16.0.6 (16.0.6-5.fc39), 15.0.7 (15.0.7-4.fc39), 14.0.5 (14.0.5-6.fc39), 13.0.1 (13.0.1-5.fc39), 12.0.1 (12.0.1-9.fc39), 11.1.0 (11.1.0-11.fc39), 8.0.1 (8.0.1-5.fc39) and 7.0.1 (7.0.1-7.fc39.8)
    • Subversion – 1.14.2 (1.14.2-20.fc39)
    • Git – 2.41.0 (2.41.0-2.fc39)

Note: Not all of the above software comes installed by default. The versions above are valid as of November 2023, these are the minimum versions you get with Fedora Server 39 now, and updating it after the initial date may update some of the above packages with newer versions.
Installed packages are 682 occupying 1.7G space:. Note, this is Fedora Server Install, not minimal install. The server install includes the web console – cockpit version 254.

root@srv:~# dnf list installed|wc -l
682
root@srv:~# df -h /
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root   15G  1.7G   14G  12% /

Keep on reading!

Pages: Page 1, Page 2, Page 3, Page 4

Software and technical details of Fedora Server 38 including cockpit screenshots

main menu
System Overview

This article is for those of you who do not want to install a whole new operating system only to discover some technical details about the default installation like disk layout, packages included, software versions, and so on. Here we are going to review in several sections what is like to have a default installation of Fedora 38 Server using a real not virtual machine!
The kernel is 6.2.15 it detects successfully the Threadripper 1950X AMD and the system is stable (we booted in UEFI mode).
The installation procedure uses default options for all installation setups – Minimal network installation of Fedora 38 Server

Software

With Fedora Server 38 you can have

  • linux kernel – 6.2.15 (6.2.15-300.fc38.x86_64)
  • System
    • linux-firmware – version: 20230515, release: 20230515-150.fc38.
    • libc – 2.37 (2.37-4.fc38)
    • GNU GCC – 13.1.1 (13.1.1-2.fc38)
    • OpenSSL – 3.0.8 (3.0.8-2.fc38) and 1.1.1q (1.1.1q-4.fc38)
    • coreutils – 9.1-12 (9.1-12.fc38)
    • yum – Depricated and replaced with dnf
    • dnf – 4.15.1 (4.15.1-1.fc38)
    • rsyslog – 8.2210.0 (8.2210.0-4.fc38)
    • NetworkManager – 1.42.6 (1.42.6-1.fc38)
  • Servers
    • Apache – 2.4.57 (2.4.57-1.fc38)
    • Nginx – 1.24.0 (1.24.0-1.fc38)
    • MySQL server – 8.0.33 (8.0.33-2.fc38)
    • MariaDB server – 10.5.19 (10.5.19-2.fc38)
    • PostgreSQL – 15.1 (15.1-2.fc38)
  • Programming
    • PHP – 8.2.6 (8.2.6-1.fc38)
    • python – The default is 3.11.3 (3.11.3-2.fc38) and many more available – 3.12.0 (3.12.0~a7-1.fc38), 3.10.11 (3.10.11-1.fc38), 3.9.16 (3.9.16-3.fc38), 3.8.16 (3.8.16-3.fc38), 3.7.16 (3.7.16-3.fc38), 3.6.15 (3.6.15-17.fc38) and also includes the older 2.7.18 (2.7.18-31.fc38)
    • perl – 5.36.1 (5.36.1-497.fc38)
    • ruby – 3.2.2 (3.2.2-180.fc38)
    • OpenJDK – the latest 20 – 20.0.1.0.9 (20.0.1.0.9-8.rolling.fc38) and also includes 17.0.7.0.7 (17.0.7.0.7-5.fc38), 11.0.19.0.7 (11.0.19.0.7-1.fc38) and 1.8.0.362.b09 (1.8.0.362.b09-2.fc38)
    • Go – 1.20.4 (1.20.4-1.fc38)
    • Rust – 1.69.0 (1.69.0-2.fc38)
    • llvm – the latest 16.0.4 (16.0.4-1.fc38), 15.0.7 (15.0.7-4.fc38), 14.0.5 (14.0.5-5.fc38), 13.0.1 (13.0.1-4.fc38), 12.0.1-8.fc38 (12.0.1-8.fc38), 11.1.0 (11.1.0-10.fc38), 8.0.1 (8.0.1-4.fc38) and 7.0.1 (7.0.1-7.fc38)
    • Subversion – 1.14.2 (1.14.2-13.fc38)
    • Git – 2.40.1 (2.40.1-1.fc38)

Note: Not all of the above software comes installed by default. The versions above are valid as of May 2023, these are the minimum versions you get with Fedora Server 38 now, and updating it after the initial date may update some of the above packages with newer versions.

Installed packages are 679 occupying 1.8G space:. Note, this is Fedora Server Install, not minimal install. The server install includes the web console – cockpit version 254.

[root@srv ~]# dnf list installed|wc -l
674
[root@srv ~]# df -h /
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/fedora-root   15G  1.7G   14G  12% /

Keep on reading!

Pages: Page 1, Page 2, Page 3, Page 4

Software and technical details of Fedora Server 37 including cockpit screenshots

main menu
System Overview

This article is for those of you who do not want to install a whole new operating system only to discover some technical details about the default installation like disk layout, packages included, software versions, and so on. Here we are going to review in several sections what is like to have a default installation of Fedora 37 Server using a real not virtual machine!
The kernel is 6.0.11 it detects successfully the Threadripper 1950X AMD and the system is stable (we booted in UEFI mode).
The installation procedure uses default options for all installation setups – Minimal network installation of Fedora 37 Server

Software

With Fedora Server 37 you can have

  • linux kernel – 6.0.11 (6.0.11-300.fc37.x86_64)
  • System
    • linux-firmware – version: 20221109, release: 20221109-144.fc37.
    • libc – 2.36 (2.36-8.fc37)
    • GNU GCC – 12.2.1 (12.2.1-4.fc37)
    • OpenSSL – 3.0.5 (1:3.0.5-3.fc37) and 1.1.1q (1:1.1.1q-2.fc37)
    • coreutils – 9.1 (9.1-6.fc37)
    • yum – Depricated and replaced with dnf
    • dnf – 4.14.0 (4.14.0-1.fc37)
    • rsyslog – 8.2204.0 (8.2204.0-3.fc37)
    • NetworkManager – 1.40.6 (1:1.40.6-1.fc37)
  • Servers
    • Apache – 2.4.54 (2.4.54-5.fc37)
    • Nginx – 1.22.1 (1:1.22.1-1.fc37)
    • MySQL server – 8.0.31 (8.0.31-1.fc37)
    • MariaDB server – 10.5.18 (3:10.5.18-1.fc37)
    • PostgreSQL – 14.3 (14.3-8.fc37)
  • Programming
    • PHP – 8.1.13 (8.1.13-1.fc37)
    • python – The default is 3.11.0 (3.11.0-1.fc37) and many more available – 3.10.8 (3.10.8-3.fc37), 3.12.0 (3.12.0~a2-1.fc37), 3.9.15 (3.9.15-3.fc37), 3.8.15 (3.8.15-2.fc37), 3.7.15 (3.7.15-2.fc37), 3.6.15 (3.6.15-14.fc37) and also includes the older 2.7.18 (2.7.18-25.fc37)
    • perl – 5.36.0 (4:5.36.0-492.fc37)
    • ruby – 3.1.3 (3.1.3-172.fc37)
    • OpenJDK – the latest 19 – 19.0.1.0.10 (1:19.0.1.0.10-2.rolling.fc37) and also includes 1:17.0.5.0.8 (1:17.0.5.0.8-1.fc37), 11.0.17.0.8 (1:11.0.17.0.8-1.fc37) and 1:1.8.0.352 (1:1.8.0.352.b08-2.fc37)
    • Go – 1.19.3 (1.19.3-2.fc37)
    • Rust – 1.65.0 (1.65.0-1.fc37)
    • llvm – the latest 15.0.4 (15.0.4-1.fc37), 14.0.0 (14.0.0-1.fc36) and the old 7.0.1 (7.0.1-7.fc36.4), 8.0.1 (8.0.1-3.fc37), 9.0.1 (9.0.1-15.fc35), 10.0.0 (10.0.0-13.fc35), 11.1.0 (11.1.0-6.fc35), 12.0.1 (12.0.1-2.fc35) and 13.0.1 (13.0.1-2.fc37)
    • Subversion – 1.14.2 (1.14.2-8.fc37)
    • Git – 2.38.1 (2.38.1-1.fc37)

Note: Not all of the above software comes installed by default. The versions above are valid as of December 2022, these are the minimum versions you get with Fedora Server 37 now, and updating it after the initial date may update some of the above packages with newer versions.

Installed packages are 679 occupying 1.8G space:. Note, this is Fedora Server Install, not minimal install. The server install includes the web console – cockpit version 254.

[root@srv ~]# dnf list installed|wc -l
679
[root@srv ~]# df -h /
Filesystem                      Size  Used Avail Use% Mounted on
/dev/mapper/fedora_fedora-root   15G  1.8G   14G  12% /

Keep on reading!

Pages: Page 1, Page 2, Page 3, Page 4

Software and technical details of CentOS Stream 9 minimal install

This article is for those of you who do not want to install a whole new operating system only to discover some technical details about the default installation like disk layout, packages included, software versions, and so on. Here we are going to review in several sections what is like to have a default installation of CentOS Stream 9 using a real not virtual machine!
Here are some useful URLs:

How to install CentOS Stream 9 – Network installation of CentOS Stream 9 (20220606.0) – minimal server installation
The kernel is 5.14.0 it detects successfully the Threadripper 1950X AMD and the system is stable (we booted in UEFI mode).
The installation procedure uses default options for all installation setups – Network installation of CentOS Stream 9 (20220606.0) – minimal server installation

Software

With CentOS Stream 9 (20220606.0) you could have

  • linux kernel – 5.14.0 (5.14.0-109.el9.x86_64)
  • System
    • linux-firmware – version: 20220509, release: 20220509-126.el9.
    • libc – 2.34 (2.34-32.el9)
    • systemd – 250-7 (250-7.el9)
    • GNU GCC – 11.3.1 (gcc-11.3.1-2.el9)
    • OpenSSL – 3.0.1 (3.0.1-33.el9) and 1.1.1k (compat-openssl11-1.1.1k-4.el9)
    • coreutils – 8.32 (8.32-31.el9)
    • yum – deprecated and replaced with dnf
    • dnf – 4.12.0 (4.12.0-2.el9)
    • rsyslog – 8.2102.0 (8.2102.0-105.el9)
    • NetworkManager – 1.39.6 (1:1.39.6-1.el9)
  • Servers
    • Apache – 2.4.53 (2.4.53-2.el9)
    • Nginx – 1.20.1 (1.20.1-10.el9)
    • MySQL server – 8.0.28 (8.0.28-1.el9)
    • MariaDB server – 10.5.13 (10.5.13-2.el9)
    • PostgreSQL – 13.7 (13.7-1.el9)
  • Programming
    • PHP – 8.0.13 (8.0.13-1.el9)
    • python – 3.9.10 (3.9.10-2.el9)
    • perl – 5.32.1 (5.32.1-479.el9)
    • ruby – 3.0.3 (3.0.3-159.el9)
    • OpenJDK – 17.0.3.0.7 (17.0.3.0.7-1.el9), 11.0.15.0.10 (11.0.15.0.10-1.el9) and 1.8.0.332.b09 (1.8.0.332.b09-1.el9)
    • Go – 1.17.5 (1.17.5-1.el9)
    • Rust – 1.61.0 (1.61.0-1.el9)
    • llvm – 14.0.0 (14.0.0-2.el9)
    • Subversion – 1.14.1 (1.14.1-5.el9)
    • Git – 2.31.1 (2.31.1-2.el9.2)

Note: Not all of the above software comes installed by default. The versions above are valid as of June 2022, these are the minimal versions you get with CentOS Stream 9 (20220606.0) now and updating it after the initial date may update some of the above packages with newer versions.

Installed packages are 376 occupying 1.6G space:. Note, this is CentOS Stream 9 Minimal Install, not server or server with GUI.

[root@srv ~]# dnf list installed|wc -l
377
[root@srv ~]# df -h /
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/cs_srv-root   70G  1.6G   69G   3% /

Keep on reading!

Pages: Page 1, Page 2, Page 3, Page 4

Simple export of a ext4 directory with NFS Ganesha 3.5 server in CentOS 8 with SELinux enforcing

In fact, this article is a continuation of the previous NFS Ganesha article – Simple export of an ext4 directory with NFS Ganesha 3.5 server in CentOS 8 without SELinux because it has the same purpose to export a directory residing on an ext4 file system under CentOS 8 Stream, but this time the SELinux is enabled and it is in enforcing mode! There is a need for this additional article because the SELinux is not enabled in many user configurations (despite being wrong!) and the SELinux configuration may add complexity to the first article, which could lead to misleading thoughts. The previous article might be a little bit more detailed, so the reader could check it, too.
It’s worth mentioning the key points of NFS-Ganesha:

  • a user-mode file sharing server
  • supports NFS 3, 4.x and 9P
  • using plugins for different file systems
  • CentOS Storage Special Interest Group offers a file repository with NFS-Ganesha server
  • supports file systems like ext4, xfs, brtfs, zfs and more. There are sample configurations: https://github.com/phdeniel/nfs-ganesha/tree/master/src/config_samples
  • supports cluster and/or distributed file systems like GlusterFS, Ceph, GPFS, HPSS, Lustre
  • Current version 3.5 and it is included in the official SIG CentOS Storage Special Interest Group repository.

This article assumes the reader has a clean CentOS 8 Stream installation with SELinux in enforcing mode.

STEP 1) Install the repository and NFS-Ganesha software

NFS-Ganesha 3 packages are from the CentOS Storage SIG repository, which is a good repository and may be trusted.

dnf install -y centos-release-nfs-ganesha30
dnf install -y nfs-ganesha nfs-ganesha-vfs nfs-ganesha-selinux

STEP 2) Configuration for exporting a directory.

There are two files under /etc/ganesha/:

ganesha.conf
vfs.conf

ganesha.conf includes global configuration and NFS share configuration. Each export path begins with the keyword EXPORT followed by a block ebraced by brackets {}.
vfs.conf includes a simple example for the VFS plugin, but this configuration file is not used by the NFS Ganesha server. It is just a sample file.
Here is a simple configuration, which exports /mnt/storage with Read/Write permissions to a single IP. Just add at the end of the file /etc/ganesha/ganesha.conf contains:

 
EXPORT
{
        Export_Id = 2;
        Path = /mnt/storage1;
        Pseudo = /mnt/storage1;
        Protocols = 3,4;
        Access_Type = RW;
        Squash = None;
        FSAL
        {
                Name = VFS;
        }
        CLIENT
        {
                Clients = 192.168.0.12;
        }
}

STEP 3) Start the server and mount the exported directory. Configure the firewall.

Start the server, enable the service to start on boot and then configure the firewall to pass the NFS requests:

systemctl start nfs-ganesha
systemctl enable nfs-ganesha
firewall-cmd --permanent --zone=public --add-service=nfs
firewall-cmd --reload

Keep on reading!

gitlab in podman cannot create unix sockets in glusterfs because of SELinux

Installing gitlab-ee (and gitlab-ce) under CentOS 7 with enabled SELinux (i.e. enforcing mode) looped endlessly the container in restarting the installation process! There were multiple errors for missing sockets in the podman logs of the gitlab container. Here are some of the errors:
Missing postgresql unix socket in “/var/opt/gitlab/postgresql”:

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
              PG::ConnectionBad: could not connect to server: No such file or directory
                Is the server running locally and accepting
                connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => gitlab:db:configure
              (See full trace by running task with --trace)
    
    
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
.....
.....
Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200915-35-lemic5" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200915-35-lemic5" ----
Ran "bash"  "/tmp/chef-script20200915-35-lemic5" returned 1

Missing redis socket in 

Running handlers:
There was an error running gitlab-ctl reconfigure:

redis_service[redis] (redis::enable line 19) had an error: RuntimeError: ruby_block[warn pending redis restart] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/resources/service.rb line 65) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket INFO` failed with a non-zero exit code (1)
stdout: 
stderr: Could not connect to Redis at /var/opt/gitlab/redis/redis.socket: No such file or directory

It should be noted that the /var/opt/gitlab directory has been mapped in /mnt/storage/podman/gitlab/data. GlusterFS is used for /mnt/storage, so the gitlab files resides on a GlusterFS volume.

ERROR 1) Cannot create unix socket.

Checking the /var/log/audit/audit.log reveiled the problem immediately:
Keep on reading!

collectd nginx plugin: curl_easy_perform failed because of selinux

Enabling the Nginx plugin for collectd under CentOS (or any other system using SELinux) might be confusing for a newbie. Most sources on the Internet would just install collectd-nginx:

yum install -y collectd-nginx

and configure it in the nginx.conf and collectd.conf. Still, the statistics might not work as expected, the collectd may not be able to gather statistics from the Nginx.

SELinux may prevent collectd (plugin) daemon to connect to Nginx and gather statistics from the Nginx stats page.

Checking the collectd log and it reports a problem:
Keep on reading!

syslog – UDP local to rsyslog and send remote with TCP and compression

This article is to show how to log Nginx’s access logs locally using UDP to the local rsyslog daemon, which will send the logs to a remote rsyslog server using TCP and compression. In general, logs could generate a lot of traffic and using UDP over distant locations could result in packet loss respectively logs’ lines loss. The idea here is to log messages locally using UDP (non-blocking way) to a local Syslog server, which will send the stream to a remote central Syslog server using TCP connections to be sure no packets are lost. In addition, we are going to enable local caching (if the remote server is temporary unreachable) and compression between the two Syslog servers.
Our goal is to use

  • UDP for our client program (Nginx in the case) for non-blocking log writes.
  • TCP between our local machine and the remote syslog server – to be sure not to lose messages on bad connectivity.
  • local caching for our client machine – not to lose messages if the remote syslog is temporary unreachable.
  • compression between the local machine and the remote syslog server.

The configuration and the commands are tested on CentOS 7, CentOS 8 and Ubuntu 18 LTS. Check out UDP remote logging here – nginx remote logging to UDP rsyslog server (CentOS 7).

STEP 1) Configure client’s local rsyslog to accept UDP log messages only if the messages’ tags are “nginx”

Couple of things should be enabled in the local client-size rsyslog daemon:

  • rsyslog to accept UDP messages. Uncomment or add the following under section “Modules” (probably the first section in the file?) in /etc/rsyslog.conf 
    $ModLoad imudp
$UDPServerRun 514

    or

    module(load="imudp")
input(type="imudp" port="514")

    The first is the old syntax, which is still supported and the second is the new syntax. For simplicity, all of the following configuration will be using the new syntax, because the old one is depricated.

  • Add a rule to catch the tag containing “nginx” and execute action to forward the messages to the remote server 
    if ($syslogtag == 'nginx:') then {
action(type="omfwd" target="10.10.10.10" port="10514" protocol="tcp" compression.Mode="single" ZipLevel="9"
queue.filename="forwarding" queue.spoolDirectory="/var/log" queue.size="1000000" queue.type="LinkedList" queue.maxFileSize="1g" queue.SaveOnShutdown="on"
action.resumeRetryCount="-1")
& stop
}

    • The options are almost self-explanatory, the important ones are there is no retry limit count of reconnecting to the server, there is in-disk cache of maximum 1G if the remote server is unavailable and the compression per message is turned on. More on actions – https://www.rsyslog.com/doc/v8-stable/configuration/actions.html, the forward module – https://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and the queue – https://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html

And restart the rsyslog:

systemctl restart rsyslog

Keep on reading!