gitlab in podman cannot create unix sockets in glusterfs because of SELinux

Installing gitlab-ee (and gitlab-ce) under CentOS 7 with enabled SELinux (i.e. enforcing mode) looped endlessly the container in restarting the installation process! There were multiple errors for missing sockets in the podman logs of the gitlab container. Here are some of the errors:
Missing postgresql unix socket in “/var/opt/gitlab/postgresql”:

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
              PG::ConnectionBad: could not connect to server: No such file or directory
                Is the server running locally and accepting
                connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => gitlab:db:configure
              (See full trace by running task with --trace)
    
    
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
.....
.....
Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200915-35-lemic5" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200915-35-lemic5" ----
Ran "bash"  "/tmp/chef-script20200915-35-lemic5" returned 1

Missing redis socket in

Running handlers:
There was an error running gitlab-ctl reconfigure:

redis_service[redis] (redis::enable line 19) had an error: RuntimeError: ruby_block[warn pending redis restart] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/resources/service.rb line 65) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket INFO` failed with a non-zero exit code (1)
stdout: 
stderr: Could not connect to Redis at /var/opt/gitlab/redis/redis.socket: No such file or directory

It should be noted that the /var/opt/gitlab directory has been mapped in /mnt/storage/podman/gitlab/data. GlusterFS is used for /mnt/storage, so the gitlab files resides on a GlusterFS volume.

ERROR 1) Cannot create unix socket.

Checking the /var/log/audit/audit.log reveiled the problem immediately:
Keep on reading!

collectd nginx plugin: curl_easy_perform failed because of selinux

Enabling the Nginx plugin for collectd under CentOS (or any other system using SELinux) might be confusing for a newbie. Most sources on the Internet would just install collectd-nginx:

yum install -y collectd-nginx

and configure it in the nginx.conf and collectd.conf. Still, the statistics might not work as expected, the collectd may not be able to gather statistics from the Nginx.

SELinux may prevent collectd (plugin) daemon to connect to Nginx and gather statistics from the Nginx stats page.

Checking the collectd log and it reports a problem:
Keep on reading!

syslog – UDP local to rsyslog and send remote with TCP and compression

This article is to show how to log Nginx’s access logs locally using UDP to the local rsyslog daemon, which will send the logs to a remote rsyslog server using TCP and compression. In general, logs could generate a lot of traffic and using UDP over distant locations could result in packet loss respectively logs’ lines loss. The idea here is to log messages locally using UDP (non-blocking way) to a local Syslog server, which will send the stream to a remote central Syslog server using TCP connections to be sure no packets are lost. In addition, we are going to enable local caching (if the remote server is temporary unreachable) and compression between the two Syslog servers.
Our goal is to use

  • UDP for our client program (Nginx in the case) for non-blocking log writes.
  • TCP between our local machine and the remote syslog server – to be sure not to lose messages on bad connectivity.
  • local caching for our client machine – not to lose messages if the remote syslog is temporary unreachable.
  • compression between the local machine and the remote syslog server.

The configuration and the commands are tested on CentOS 7, CentOS 8 and Ubuntu 18 LTS. Check out UDP remote logging here – nginx remote logging to UDP rsyslog server (CentOS 7).

STEP 1) Configure client’s local rsyslog to accept UDP log messages only if the messages’ tags are “nginx”

Couple of things should be enabled in the local client-size rsyslog daemon:

  • rsyslog to accept UDP messages. Uncomment or add the following under section “Modules” (probably the first section in the file?) in /etc/rsyslog.conf
    $ModLoad imudp
    $UDPServerRun 514
    

    or

    module(load="imudp")
    input(type="imudp" port="514")
    

    The first is the old syntax, which is still supported and the second is the new syntax. For simplicity, all of the following configuration will be using the new syntax, because the old one is depricated.

  • Add a rule to catch the tag containing “nginx” and execute action to forward the messages to the remote server
    if ($syslogtag == 'nginx:') then {
    action(type="omfwd" target="10.10.10.10" port="10514" protocol="tcp" compression.Mode="single" ZipLevel="9"
    queue.filename="forwarding" queue.spoolDirectory="/var/log" queue.size="1000000" queue.type="LinkedList" queue.maxFileSize="1g" queue.SaveOnShutdown="on"
    action.resumeRetryCount="-1")
    & stop
    }
    
  • The options are almost self-explanatory, the important ones are there is no retry limit count of reconnecting to the server, there is in-disk cache of maximum 1G if the remote server is unavailable and the compression per message is turned on. More on actionshttps://www.rsyslog.com/doc/v8-stable/configuration/actions.html, the forward modulehttps://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and the queuehttps://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html

And restart the rsyslog:

systemctl restart rsyslog

Keep on reading!

autorelabel failed to relabel the root filesystem on boot when enabling the selinux on CentOS 8

This article is of a kind – “/.autorelabel” file does not work, because the enable of SELINUX ended with unreachable server.

Enabling the SELINUX should be easy as

  1. just editing a text file – /etc/selinux/config to include
    SELINUX=enforcing
    
  2. relabel all (or at least the root) file systems by creating the “/.autorelabel” file.
  3. restarting the system. The boot process will detect the “/.autorelabel” file and relabel the file systems and then it will restart the system in the normal boot order.

But this time the relabeling did not happen as usual (It happened on CentOS 8, but probably could happen in any Linux distribution?). The server never got reachable again and on the screen, there were multiple errors – all of “Permission denied”!

It is better when enabling SELINUX to set “permissive” mode at first and relabel the root file system with “/.autorelabel” and then to enable “enforcing” mode of SELINUX.

Using “permissive” first for the relabel process guarantees you would have your server back after the process because the SELINUX rules are not enforced.
Here is the better procedure of enabling the SELINUX:

  1. just editing a text file – /etc/selinux/config to include
    SELINUX=permissive
    
  2. relabel all (or at least the root) file systems by creating the “/.autorelabel” file.
  3. restarting the system. The boot process will detect the “/.autorelabel” file and relabel the file systems and then it will restart the system in normal boot order.
  4. edit the /etc/selinux/config to enable “enforcing” mode
    SELINUX=enforcing
    
  5. Restart (it’s better) or just enable SELINUX enforcing live with:
    setenforce 1
    

Our screenshots log of the relabel failure process

SCREENSHOT 1) No autorelabel initiated on boot despite the presence of “/.autorelabel” file.

Multple “Permission Denied” errors and many reports from “audit” – the SELINUX log daemon. The host is unreachable – no network started. No logging is possible!

main menu
boot freezing

SCREENSHOT 2) A page up above the first screen – more “Permission Denied” errors.

main menu
boot freezing 2

SCREENSHOT 3) Second page up above the first screen – the SELINUX rules loaded successfully but no autorelabel process initiated.

main menu
boot freezing 3

A successful relabel process on boot

SCREENSHOT 1) Successful start of the relabel process.

We’ve changed the SELINUX mode to be “permissive” and everything is back to normal, the “/.autorelabel” file initiated the relabel on the next boot.

main menu
relabel process on boot initiated

SCREENSHOT 2) The relabeling of the file system is in progress.

There is a progress counter.

main menu
relabel process in progress

SCREENSHOT 3) The relabel process finished successfully and the reboot is initiated.

The next reboot the “/.autorelabel” file won’t exists and the system will boot normally.

main menu
relabel process success and last screen before reboot

Technical details of Fedora Workstation 31 (Gnome GUI)

This article is for those of you who do not want to install a whole new operating system only to discover some technical details about the default installation like disk layout, packages included, software versions, and so on. Here we are going to review in several sections what is like to have a default installation of Fedora Workstation 31 (Gnome) using a real not virtual machine!
The kernel is 5.3.7 it detects successfully the Threadripper 1950X AMD and the system is stable (we booted in UEFI mode).

Software

With Fedora 31 (Workstation Edition) you can have

  • linux kernel – 5.3.7 (5.3.7-301.fc31.x86_64)
  • Graphical User Interface
    • Xorg X server (Xwayland) – 1.20.5
    • GNOME (the GUI) – 3.34.1
    • K Desktop Environment – ships with https://spins.fedoraproject.org/en/kde/ Use this spin to install KDE.
  • System
    • linux-firmware – version: 20190923, release: 20191022-103.fc31.
    • QT – 5.12.5
    • libc – 2.30
    • GNU GCC – 9.2.1
    • OpenSSL – 1.1.1d
    • coreutils – 8.31
    • yum – Depricated and replaced with dnf
    • dnf – 4.2.9
    • cups – 2.2.12
    • rsyslog – 8.1907.0
    • NetworkManager – 1.20.4
  • Servers

Technical details of a default CentOS 8.0.1950 minimal installation

This article is for those of you who do not want to install a whole new operating system only to discover some technical details about the default installation like disk layout, packages included, software versions, and so on. Here we are going to review in several sections what is like to have a default installation of CentOS 8.0.
Despite the kernel is 4.18 it detects successfully the new RYZEN/Threadripper AMD and the system is stable (we booted in UEFI mode).

Software

The CentOS 8.0 (8.0.1950) you can have

  • linux kernel – 4.18.0 (4.18.0-80.7.1.el8_0.x86_64)
  • Graphical User Interface
    • Xorg X server – 1.20.3
    • GNOME (the GUI) – 3.28.2
    • K Desktop Environment – NO, it’s depricated and not included in the release.
  • System
    • linux-firmware – version: 20190111, release: 92.gitd9fb2ee6.el8.
    • QT – 5.11.1
    • libc – 2.28
    • GNU GCC – 8.2.1
    • OpenSSL – 1.1.1
    • coreutils – 8.30
    • yum – Depricated and replaced with dnf
    • dnf – 4.0.9.2
    • cups – 2.2.6
    • rsyslog – 8.37.0
  • Servers
    • Apache – 2.4.37
    • Nginx – 1.14.1
    • MySQL server – 8.0.13
    • MariaDB server – 10.3.11
    • PostgreSQL – 10.6
  • Programming
    • PHP – 7.2.11
    • python – 2.7.15 and also includes 3.6.8
    • perl – 5.26.3
    • ruby – 2.5.3
    • OpenJDK – 11.0.4.11 and also includes 1.8.0.222.b10
    • Go – 1.11.5
    • Rust – 1.31.0
    • Subversion – 1.10.2
    • Git – 2.18.1

Note: Not all of the above software comes installed by default.
The installation procedure you can find here – How to do a network installation of CentOS 8 (8.0.1950) – minimal server installation Minimal install with “Standard” Addons selected.
Installed packages are 547 occupying 1.8G space:.

[root@srv ~]# dnf list installed|wc -l
547
[root@srv ~]# df -h /
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/cl_srv-root   50G  1.8G   49G   4% /

Note: if you do not select “Standard Addon” the installation occupies less space – 1.6Gbytes and installs only 397 packages.
Keep on reading!

Failed to start Security Audit Service, Authorization Manager and Login Service

A power outrage caused one of our servers to shut down unexpectedly and after it had been powered up the server did not show up. The server was unreachable and apparently, the network did not bring up the interfaces.
Loading the IPMI KVM Console and rebooting the server there were three errors on the screen during the boot up of the CentOS 7:

[FAILED] Failed to start Security Audit Service.
See 'systemctl status auditd.service' for details.
....
....
[FAILED] Failed to start Authorization Manager.
See 'systemctl status polkit.service' for details.
....
....
[FAILED] Failed to start Login Service.
See 'systemctl status systemd-logind.service' for details.

And after the above last line, the system stopped loading.
The disks are clean, but there was no login service, so you cannot log in to the server through the keyboard and the monitor! There was no network as mentioned above, which meant no logging at all in the server. You might not know, but if auditd service is enabled you probably use Selinux!

STEP 1) Failed to start the three important services – Security Audit Service, Authorization Manager and Login Service.

So we ended up with unability to log in our server.

main menu

Not sure what exactly caused this problem (seems strange a perfectly working Selinux enabled CentOS 7 server to have miss-labeled files in the root only because of an unexpected shutdown), but to be able to fix the issue and bring back your server to life

you need a rescue CD/USB/DVD/PXE Server to boot from and mount the disks and relabel your root file system.

STEP 1) Boot from a rescue CD/USB/DVD/PXE Server.

In our case, we used the IPMI KVM Console and mounted a Gentoo ISO disk and then booted from it to have a bash shell in our system. Our root resides on software RAID 1, so cat the /proc/mdstat and mount your root file system somewhere (/mnt/gentoo is there by default…)

STEP 2) Booted in our rescue Gentoo CD and mount your root file system.

main menu
Rescue Gentoo CD

STEP 2) create a file “.autorelabel” in the mounting point of your root file system.

So in our case, we mounted our CentOS 7 root file system in /mnt/gentoo and you must create a file with patch “/mnt/gentoo/.autorelabel”. umount and reboot. And a few minutes later your server will be back from the dead. A quick and handful advice – edit your /etc/fstab to mount only the root file system by commenting out all other big storage mounts – of course, if it is possible. We have big storage with millions of files in /mnt/storage-01 and we put the “#” to comment out the line with it – we do not want to wait for relabeling this file system, because the problem apparently is in our root file system! If it is possible (it is highly recommended) to relabel only the root file system in such situations to be able to regain shell control over your server fast.

Bonus – booted in rescue but no logs

OK, we booted to the rescue and tried to see what was the error (with journalctl in chrooted /mnt/gentoo), which did not allow auditd, polkit and systemd-logind to fail to start, but it appeared by default the systemd logs are not persistent on the disk in CentOS 7, so when you reboot in rescue you do not have systemd logs from the last boot! As a piece of additional advice here you may consider enabling persistent systemd logs!

rsync and selinux – opendir failed: Permission denied

Selinux could sometime mess up with your setup. Let’s say you configured your rsync daemon but still, you get the error related to permissions when executing the rsync to copy files!

rsync: opendir "/." (in backup2) failed: Permission denied (13)

Apparently, the rsync client connects to the server and it finds there is a section name “backup2”, but still no permission despite you explicitly set in the section uid and ig to be root (uid=0 and gid=0 in the section)!

The most common reason is

selinux denies rsync process to open the directory exported by the path in your rsync configuration file.

By default, Selinux will deny access to any of the files and directories in your system! In most cases here what can you help:

setsebool -P rsync_export_all_ro=1

rsync_export_all_ro will export any files and directories read-only and requests like above will not be denied.
The capital letter “-P” is to set it permanently for the system over reboots.
Keep on reading!

Selinux permission denied and no log in audit.log

So you execute a script and get a “Permission denied” and you know you have enabled SELinux. OK to disable the selinux is not an option (and never will be), so the first thing to check is the audit log to see what is the error and what the selinux tools will offer to solve it.

But there are no entries in the audit log when you execute your script!

So you decide to temporarily disable the selinux to check if this permission denied issues is still caused by it with:

setenforce 0

And the script just executes fine no error! Then again you put back the Enforcing with:

setenforce 1
./myscript
Permission Denied

And NO added lines in audit.log (/var/log/audit/audit.log in our system!). Apparently the logging is just fine, because it got sometime entries, but when executing our script, which is just a simple:

 
find /mnt/storA/servers/webroots/

After some research it appeared that

not all AVC denials may be logged when SELinux denies access.

Too many applications and system libraries check for permissions, which might not use or even need after that and the logging could grow exponentially or be less informative for the real cause of a problem!
Keep on reading!

Technical details of a default CentOS 6.10 minimal installation

This article is for those of you who do not want to install a whole new operating system only to discover some technical details about the default installation like, disk layout, packages included, software versions and so on. Here we are going to review in several sections what is like to have a default installation of CentOS 6.10.
Despite the old kernel, as you can see branch 2.6.x (2.6.32) it detects successfully the new RYZEN/Threadripper AMD and the system is stable (though we could not boot in UEFI mode!).

Software

The CentOS 6.10 you can have

  • Xorg X server – 1.17.4
  • GNOME (the GUI) – 2.28.2
  • K Desktop Environment – Workspace 4.3.4
  • linux kernel – 2.6.32 (2.6.32-754.el6)
  • linux-firmware – no version, multiple packages for the drivers
  • QT – 4.6.2
  • libc – 2.12
  • gnu gcc – 4.4.7
  • coreutils – 8.4
  • yum – 3.2.29
  • cups – 1.4.2
  • MySQL server – 5.1.73
  • PHP – 5.3.3
  • python – 2.6.6
  • perl – 5.10.1
  • ruby – 1.8.7.374
  • OpenJDK – 1.8.0.191.b12 and also includes 1.6.0.41 and 1.7.0.191

Note: Not all of the above software comes installed by default.
Installed packages are 268 occupying 717M space:

[root@srv ~]# yum list installed|wc -l
268
[root@srv ~]# df -h /
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/vg_srv-lv_root
                       50G  717M   46G   2% /

Keep on reading!