gitlab in podman cannot create unix sockets in glusterfs because of SELinux

Installing gitlab-ee (and gitlab-ce) under CentOS 7 with enabled SELinux (i.e. enforcing mode) looped endlessly the container in restarting the installation process! There were multiple errors for missing sockets in the podman logs of the gitlab container. Here are some of the errors:
Missing postgresql unix socket in “/var/opt/gitlab/postgresql”:

Recipe: gitlab::database_migrations
  * bash[migrate gitlab-rails database] action run
    [execute] rake aborted!
              PG::ConnectionBad: could not connect to server: No such file or directory
                Is the server running locally and accepting
                connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
              /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
              /opt/gitlab/embedded/bin/bundle:23:in `load'
              /opt/gitlab/embedded/bin/bundle:23:in `<main>'
              Tasks: TOP => gitlab:db:configure
              (See full trace by running task with --trace)
    
    
    Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
.....
.....
Running handlers:
There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20200915-35-lemic5" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
        Is the server running locally and accepting
        connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR: 
---- End output of "bash"  "/tmp/chef-script20200915-35-lemic5" ----
Ran "bash"  "/tmp/chef-script20200915-35-lemic5" returned 1

Missing redis socket in

Running handlers:
There was an error running gitlab-ctl reconfigure:

redis_service[redis] (redis::enable line 19) had an error: RuntimeError: ruby_block[warn pending redis restart] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/resources/service.rb line 65) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket INFO` failed with a non-zero exit code (1)
stdout: 
stderr: Could not connect to Redis at /var/opt/gitlab/redis/redis.socket: No such file or directory

It should be noted that the /var/opt/gitlab directory has been mapped in /mnt/storage/podman/gitlab/data. GlusterFS is used for /mnt/storage, so the gitlab files resides on a GlusterFS volume.

ERROR 1) Cannot create unix socket.

Checking the /var/log/audit/audit.log reveiled the problem immediately:
Keep on reading!

collectd nginx plugin: curl_easy_perform failed because of selinux

Enabling the Nginx plugin for collectd under CentOS (or any other system using SELinux) might be confusing for a newbie. Most sources on the Internet would just install collectd-nginx:

yum install -y collectd-nginx

and configure it in the nginx.conf and collectd.conf. Still, the statistics might not work as expected, the collectd may not be able to gather statistics from the Nginx.

SELinux may prevent collectd (plugin) daemon to connect to Nginx and gather statistics from the Nginx stats page.

Checking the collectd log and it reports a problem:
Keep on reading!

Dracut boot failed with missing device – exit and continue normal booting!

This issue deserves a much more article, in fact, a straightforward tip:

You may be able to continue a normal boot only by typing “exit” and hitting enter in the “Dracut” console.

Most of the time this Dracut console entering is caused because the system administrator of the server/machine added, replaced or deleted a RAID or similar device and forgot to update the configuration (grub2 probably). And in most of these cases, the raid is not critical for machine normal boot from the root partition, but it may be critical for the services lately. Booting in normal mode, even without some devices, is the main goal because under the normal mode it easier to repair the system.
Check out the two articles on the topic (especially the first one):

SCREENSHOT 1) Just type “exit” and hit enter.

It’s worth noting that if you executed some commands in the console and/or mounted devices to test they are with healthy file system or for whatever reason you did it, the boot process may not continue after typeing exit and probablly a reboot is required. The server will go once more in this mode and then just typing will work.

main menu
type exit

Keep on reading!

CentOS 7 – Your kernel headers for kernel cannot be found at – missing kernel-devel

Getting the following error may be deceiving:

Error! echo
Your kernel headers for kernel 3.10.0-1062.1.1.el7.x86_64 cannot be found at
/lib/modules/3.10.0-1062.1.1.el7.x86_64/build or /lib/modules/3.10.0-1062.1.1.el7.x86_64/source.

Because you may have already installed the kernel-headers package for the current kernel and still to get the same error. So what is missing?

In fact, the kernel headers for compiling a kernel module is in kernel-devel package.

[root@localhost ~]# yum install kernel-devel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.wwfx.net
 * extras: mirror.wwfx.net
 * updates: mirror.wwfx.net
Resolving Dependencies
--> Running transaction check
---> Package kernel-devel.x86_64 0:3.10.0-1062.1.1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================
 Package                                   Arch                                Version                                           Repository                            Size
============================================================================================================================================================================
Installing:
 kernel-devel                              x86_64                              3.10.0-1062.1.1.el7                               updates                               18 M

Transaction Summary
============================================================================================================================================================================
Install  1 Package

Total download size: 18 M
Installed size: 38 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
kernel-devel-3.10.0-1062.1.1.el7.x86_64.rpm                                                                                                          |  18 MB  00:00:02     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : kernel-devel-3.10.0-1062.1.1.el7.x86_64                                                                                                                  1/1 
  Verifying  : kernel-devel-3.10.0-1062.1.1.el7.x86_64                                                                                                                  1/1 

Installed:
  kernel-devel.x86_64 0:3.10.0-1062.1.1.el7                                                                                                                                 

Complete!

If you have used other Linux distribution the “kernel headers”/”linux headers” package just means what it is named. In the CentOS 7 world there are two packages:

[root@localhost ~]# yum info kernel-devel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.wwfx.net
 * extras: mirror.wwfx.net
 * updates: mirror.wwfx.net
Installed Packages
Name        : kernel-devel
Arch        : x86_64
Version     : 3.10.0
Release     : 1062.1.1.el7
Size        : 38 M
Repo        : installed
From repo   : updates
Summary     : Development package for building kernel modules to match the kernel
URL         : http://www.kernel.org/
License     : GPLv2
Description : This package provides kernel headers and makefiles sufficient to build modules
            : against the kernel package.

[root@localhost ~]# yum info kernel-headers
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirror.wwfx.net
 * epel: mirrors.neterra.net
 * extras: mirror.wwfx.net
 * updates: mirror.wwfx.net
Installed Packages
Name        : kernel-headers
Arch        : x86_64
Version     : 3.10.0
Release     : 1062.1.1.el7
Size        : 3.7 M
Repo        : installed
From repo   : updates
Summary     : Header files for the Linux kernel for use by glibc
URL         : http://www.kernel.org/
License     : GPLv2
Description : Kernel-headers includes the C header files that specify the interface
            : between the Linux kernel and userspace libraries and programs.  The
            : header files define structures and constants that are needed for
            : building most standard programs and are also needed for rebuilding the
            : glibc package.

CentOS 7 – Dependency Resolution – Error – Requires: dkms – missing epel repository

Quick not note for those not familiar with the CentOS 7 peculiarity and especially the repository peculiarity.
Receiving the follwoing error:

--> Finished Dependency Resolution
Error: Package: 3:kmod-nvidia-latest-dkms-418.87.00-2.el7.x86_64 (cuda)
           Requires: dkms
 You could try using --skip-broken to work around the problem
 You could try running: rpm -Va --nofiles --nodigest

It means you need a package (or meta-package, which might pull multiple packages and dependencies offering a big framework, for example), which could not be found in the existing repositories. In this very case, we need the DKMS (Dynamic Kernel Module Support) – https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support

The DKMS is offered in epel repository and it could not be found in the CentOS 7 official repositories. Just add the epel repository.

[root@localhost ~]# yum install -y epel-release
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: mirrors.daticum.com
 * extras: mirrors.daticum.com
 * updates: mirrors.daticum.com
Resolving Dependencies
--> Running transaction check
---> Package epel-release.noarch 0:7-11 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

============================================================================================================================================================================
 Package                                       Arch                                    Version                                Repository                               Size
============================================================================================================================================================================
Installing:
 epel-release                                  noarch                                  7-11                                   extras                                   15 k

Transaction Summary
============================================================================================================================================================================
Install  1 Package

Total download size: 15 k
Installed size: 24 k
Downloading packages:
epel-release-7-11.noarch.rpm                                                                                                                         |  15 kB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : epel-release-7-11.noarch                                                                                                                                 1/1 
  Verifying  : epel-release-7-11.noarch                                                                                                                                 1/1 

Installed:
  epel-release.noarch 0:7-11                                                                                                                                                

Complete!

And rerun your first install yum line. Now you won’t receive the DKMS error.

Install CUDA and NVIDIA video driver under CentOS 7

Nvidia CUDA Toolkit supports CentOS 7 and it is relatively simple to install it. Nvidia provides three types of installation – a big setup file, a big rpm file and an official Nvidia repository, which we are going to use it in this article. The Nvidia repository contains the Nvidia video driver for the Nvidia video cards like GeForce, GTX, RTX and so on. You may need CUDA Toolkit if you are a game developer or you want to build yourself some of the mining software like XMR-STAK.
In this article, we are going to use the NVIDIA official repository for CUDA and the video driver module. There are other ways to install CUDA, which are not the purpose of this article. Using an official repository is the best practice for installing software on your system.

STEP 1) Update and install the NVIDIA official repository.

yum update -y
yum install -y yum-utils
yum-config-manager --add-repo http://developer.download.nvidia.com/compute/cuda/repos/rhel7/x86_64/cuda-rhel7.repo

Keep on reading!

How to compile xmr-stak (2.10) under CentOS 7 for CPU mining cryptocurrencies in September 2019

A time to refresh our old article on how to compile xmr-stak for CPU mining with the new version and this time a new GNU GCC version (version 8.3, the last article we used 7.x – How to compile xmr-stak (2.4.5) under CentOS 7 for CPU mining cryptocurrencies). Always use the latest available GNU GCC packages because the latest version of GNU GCC could add some optimizations to the binary compiled code and you may have a CPU miner with better performance!
Thanks to xmr-stak we can have one application capable of mining many different cryptocurrencies based on different algorithms. XMR-STAK is GPU and CPU miner and here we present only the CPU ability under CentOS 7 using our AMD Threadripper 1950X.
The software in this article:

  • CentOS 7 – CentOS Linux release 7.6.1810 (Core)
  • GNU GCC – gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC)
  • XMR-STAK – 2.10.7

As said many times working with crypto-currency it is mandatory to do the things yourself – do not trust any binary made by someone on the Internet. It is easy to build your miner yourself with the code from the official repository!

So here are the steps to build the XMR-STAK for CPU mining:

STEP 1) Update your system and install the following dependencies

Always start with update command and then install the dependencies in order first install all the new repositories and then the dependency binaries.

sudo yum update -y
sudo yum install -y centos-release-scl epel-release
sudo yum install -y cmake3 devtoolset-8-gcc* hwloc-devel libmicrohttpd-devel openssl openssl-devel make git screen wget

We are going to use GNU GCC 8 to build the XMR-STAK. More on the subject of how to install GNU GCC 8 and what is “devtoolset” here – How to install GNU GCC 8 on CentOS 7.
Keep on reading!

How to install GNU GCC 8 on CentOS 7

It has been long after releasing the GNU GCC 8.x, but at last, there is a trusted repository, which has offered us packages for GNU GCC 8.x, which won’t break your system! Many of us prefer CentOS 7 because it offers free enterprise-class operating system and as mentioned in our article before – How to install new gcc and development tools under CentOS 7 there are a couple of approved external repositories for CentOS, which you can trust https://wiki.centos.org/AdditionalResources/Repositories. In one of them Software Collection – https://www.softwarecollections.org/en/scls/ several months ago the GNU GCC 8.x packages were added!
At present, the GNU GCC version is gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3).
Here are the steps how you can install GNU GCC 8 and how you can use it:

STEP 1) Update your system and install the repository in your system

The commands:

yum update -y
yum -y install centos-release-scl

Keep on reading!

CentOS 7 dracut-initqueue timeout and could not boot – warning /dev/disk/by-id/md-uuid- does not exist

Let’s say you update your software raid layout – create, delete or modify your software raid and reboot the system and your server does not start normally. After loading your remote video console (KVM) you see the boot process reports for a missing device and you are under console (dracut console). Your system is in “Emergency mode”.

The warning:

dracut-initqueue[504]: Warning: dracut-initqueue timeout - starting timeout scripts
dracut-initqueue[504]: Warning: dracut-initqueue timeout - starting timeout scripts
dracut-initqueue[504]: Warning: dracut-initqueue timeout - starting timeout scripts
....
....
dracut-initqueue[504]: Warning: could not boot.
dracut-initqueue[504]: Warning: /dev/disk/by-id/md-uuid-2fdc509e:8dd05ed3:c2350cb4:ea5a620d does not exist
      Starting Dracut Emergency Shell...
Warning: /dev/disk/by-id/md-uuid-2fdc509e:8dd05ed3:c2350cb4:ea5a620d does not exist

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


dracut:/#

SCREENSHOT 1) The boot process reports mutiple warning messages of dracut-initqueue timeout, because a drive cannot be found.

main menu
Warning: dracut-initqueue timeout – starting timeout scripts

Keep on reading!

Failed to start Security Audit Service, Authorization Manager and Login Service

A power outrage caused one of our servers to shut down unexpectedly and after it had been powered up the server did not show up. The server was unreachable and apparently, the network did not bring up the interfaces.
Loading the IPMI KVM Console and rebooting the server there were three errors on the screen during the boot up of the CentOS 7:

[FAILED] Failed to start Security Audit Service.
See 'systemctl status auditd.service' for details.
....
....
[FAILED] Failed to start Authorization Manager.
See 'systemctl status polkit.service' for details.
....
....
[FAILED] Failed to start Login Service.
See 'systemctl status systemd-logind.service' for details.

And after the above last line, the system stopped loading.
The disks are clean, but there was no login service, so you cannot log in to the server through the keyboard and the monitor! There was no network as mentioned above, which meant no logging at all in the server. You might not know, but if auditd service is enabled you probably use Selinux!

STEP 1) Failed to start the three important services – Security Audit Service, Authorization Manager and Login Service.

So we ended up with unability to log in our server.

main menu

Not sure what exactly caused this problem (seems strange a perfectly working Selinux enabled CentOS 7 server to have miss-labeled files in the root only because of an unexpected shutdown), but to be able to fix the issue and bring back your server to life

you need a rescue CD/USB/DVD/PXE Server to boot from and mount the disks and relabel your root file system.

STEP 1) Boot from a rescue CD/USB/DVD/PXE Server.

In our case, we used the IPMI KVM Console and mounted a Gentoo ISO disk and then booted from it to have a bash shell in our system. Our root resides on software RAID 1, so cat the /proc/mdstat and mount your root file system somewhere (/mnt/gentoo is there by default…)

STEP 2) Booted in our rescue Gentoo CD and mount your root file system.

main menu
Rescue Gentoo CD

STEP 2) create a file “.autorelabel” in the mounting point of your root file system.

So in our case, we mounted our CentOS 7 root file system in /mnt/gentoo and you must create a file with patch “/mnt/gentoo/.autorelabel”. umount and reboot. And a few minutes later your server will be back from the dead. A quick and handful advice – edit your /etc/fstab to mount only the root file system by commenting out all other big storage mounts – of course, if it is possible. We have big storage with millions of files in /mnt/storage-01 and we put the “#” to comment out the line with it – we do not want to wait for relabeling this file system, because the problem apparently is in our root file system! If it is possible (it is highly recommended) to relabel only the root file system in such situations to be able to regain shell control over your server fast.

Bonus – booted in rescue but no logs

OK, we booted to the rescue and tried to see what was the error (with journalctl in chrooted /mnt/gentoo), which did not allow auditd, polkit and systemd-logind to fail to start, but it appeared by default the systemd logs are not persistent on the disk in CentOS 7, so when you reboot in rescue you do not have systemd logs from the last boot! As a piece of additional advice here you may consider enabling persistent systemd logs!