Installing gitlab-ee (and gitlab-ce) under CentOS 7 with enabled SELinux (i.e. enforcing mode) looped endlessly the container in restarting the installation process! There were multiple errors for missing sockets in the podman logs of the gitlab container. Here are some of the errors:
Missing postgresql unix socket in “/var/opt/gitlab/postgresql”:
Recipe: gitlab::database_migrations
* bash[migrate gitlab-rails database] action run
[execute] rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
.....
.....
Running handlers:
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 55) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20200915-35-lemic5" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:53:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20200915-35-lemic5" ----
Ran "bash" "/tmp/chef-script20200915-35-lemic5" returned 1
Missing redis socket in
Running handlers:
There was an error running gitlab-ctl reconfigure:
redis_service[redis] (redis::enable line 19) had an error: RuntimeError: ruby_block[warn pending redis restart] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/redis/resources/service.rb line 65) had an error: RuntimeError: Execution of the command `/opt/gitlab/embedded/bin/redis-cli -s /var/opt/gitlab/redis/redis.socket INFO` failed with a non-zero exit code (1)
stdout:
stderr: Could not connect to Redis at /var/opt/gitlab/redis/redis.socket: No such file or directory
It should be noted that the /var/opt/gitlab directory has been mapped in /mnt/storage/podman/gitlab/data. GlusterFS is used for /mnt/storage, so the gitlab files resides on a GlusterFS volume.
ERROR 1) Cannot create unix socket.
Checking the /var/log/audit/audit.log reveiled the problem immediately: Keep on reading!
Error! echo
Your kernel headers for kernel 3.10.0-1062.1.1.el7.x86_64 cannot be found at
/lib/modules/3.10.0-1062.1.1.el7.x86_64/build or /lib/modules/3.10.0-1062.1.1.el7.x86_64/source.
Because you may have already installed the kernel-headers package for the current kernel and still to get the same error. So what is missing?
In fact, the kernel headers for compiling a kernel module is in kernel-devel package.
[root@localhost ~]# yum install kernel-devel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.wwfx.net
* extras: mirror.wwfx.net
* updates: mirror.wwfx.net
Resolving Dependencies
--> Running transaction check
---> Package kernel-devel.x86_64 0:3.10.0-1062.1.1.el7 will be installed
--> Finished Dependency Resolution
Dependencies Resolved
============================================================================================================================================================================
Package Arch Version Repository Size
============================================================================================================================================================================
Installing:
kernel-devel x86_64 3.10.0-1062.1.1.el7 updates 18 M
Transaction Summary
============================================================================================================================================================================
Install 1 Package
Total download size: 18 M
Installed size: 38 M
Is this ok [y/d/N]: y
Downloading packages:
Delta RPMs disabled because /usr/bin/applydeltarpm not installed.
kernel-devel-3.10.0-1062.1.1.el7.x86_64.rpm | 18 MB 00:00:02
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
Installing : kernel-devel-3.10.0-1062.1.1.el7.x86_64 1/1
Verifying : kernel-devel-3.10.0-1062.1.1.el7.x86_64 1/1
Installed:
kernel-devel.x86_64 0:3.10.0-1062.1.1.el7
Complete!
If you have used other Linux distribution the “kernel headers”/”linux headers” package just means what it is named. In the CentOS 7 world there are two packages:
[root@localhost ~]# yum info kernel-devel
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.wwfx.net
* extras: mirror.wwfx.net
* updates: mirror.wwfx.net
Installed Packages
Name : kernel-devel
Arch : x86_64
Version : 3.10.0
Release : 1062.1.1.el7
Size : 38 M
Repo : installed
From repo : updates
Summary : Development package for building kernel modules to match the kernel
URL : http://www.kernel.org/
License : GPLv2
Description : This package provides kernel headers and makefiles sufficient to build modules
: against the kernel package.
[root@localhost ~]# yum info kernel-headers
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirror.wwfx.net
* epel: mirrors.neterra.net
* extras: mirror.wwfx.net
* updates: mirror.wwfx.net
Installed Packages
Name : kernel-headers
Arch : x86_64
Version : 3.10.0
Release : 1062.1.1.el7
Size : 3.7 M
Repo : installed
From repo : updates
Summary : Header files for the Linux kernel for use by glibc
URL : http://www.kernel.org/
License : GPLv2
Description : Kernel-headers includes the C header files that specify the interface
: between the Linux kernel and userspace libraries and programs. The
: header files define structures and constants that are needed for
: building most standard programs and are also needed for rebuilding the
: glibc package.
Quick note for those not familiar with the CentOS 7 peculiarity and especially the repository peculiarity.
Receiving the follwoing error:
--> Finished Dependency Resolution
Error: Package: 3:kmod-nvidia-latest-dkms-418.87.00-2.el7.x86_64 (cuda)
Requires: dkms
You could try using --skip-broken to work around the problem
You could try running: rpm -Va --nofiles --nodigest
It means you need a package (or meta-package, which might pull multiple packages and dependencies offering a big framework, for example), which could not be found in the existing repositories. In this very case, we need the DKMS (Dynamic Kernel Module Support) – https://en.wikipedia.org/wiki/Dynamic_Kernel_Module_Support
The DKMS is offered in epel repository and it could not be found in the CentOS 7 official repositories. Just add the epel repository.
Nvidia CUDA Toolkit supports CentOS 7 and it is relatively simple to install it. Nvidia provides three types of installation – a big setup file, a big rpm file and an official Nvidia repository, which we are going to use it in this article. The Nvidia repository contains the Nvidia video driver for the Nvidia video cards like GeForce, GTX, RTX and so on. You may need CUDA Toolkit if you are a game developer or you want to build yourself some of the mining software like XMR-STAK.
In this article, we are going to use the NVIDIA official repository for CUDA and the video driver module. There are other ways to install CUDA, which are not the purpose of this article. Using an official repository is the best practice for installing software on your system.
STEP 1) Update and install the NVIDIA official repository.
A time to refresh our old article on how to compile xmr-stak for CPU mining with the new version and this time a new GNU GCC version (version 8.3, the last article we used 7.x – How to compile xmr-stak (2.4.5) under CentOS 7 for CPU mining cryptocurrencies). Always use the latest available GNU GCC packages because the latest version of GNU GCC could add some optimizations to the binary compiled code and you may have a CPU miner with better performance!
Thanks to xmr-stak we can have one application capable of mining many different cryptocurrencies based on different algorithms. XMR-STAK is GPU and CPU miner and here we present only the CPU ability under CentOS 7 using our AMD Threadripper 1950X.
The software in this article:
CentOS 7 – CentOS Linux release 7.6.1810 (Core)
GNU GCC – gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC)
As said many times working with crypto-currency it is mandatory to do the things yourself – do not trust any binary made by someone on the Internet. It is easy to build your miner yourself with the code from the official repository!
So here are the steps to build the XMR-STAK for CPU mining:
STEP 1) Update your system and install the following dependencies
Always start with update command and then install the dependencies in order first install all the new repositories and then the dependency binaries.
It has been long after releasing the GNU GCC 8.x, but at last, there is a trusted repository, which has offered us packages for GNU GCC 8.x, which won’t break your system! Many of us prefer CentOS 7 because it offers free enterprise-class operating system and as mentioned in our article before – How to install new gcc and development tools under CentOS 7 there are a couple of approved external repositories for CentOS, which you can trust https://wiki.centos.org/AdditionalResources/Repositories. In one of them Software Collection – https://www.softwarecollections.org/en/scls/ several months ago the GNU GCC 8.x packages were added!
At present, the GNU GCC version is gcc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3).
Here are the steps how you can install GNU GCC 8 and how you can use it:
STEP 1) Update your system and install the repository in your system
is really old (probably 7-9 years), but still it works, so you can check if you are with the latest and greatest firmware. Hope the latest fixes more things than it beaks. To flash the firmware you need Megaraid cli and the firmware file, the two files you check in the sub-directories of https://www.supermicro.com/wftp/driver/SAS/LSI/2108/Firmware/ They are still there despite this product is discontinued. In this URL these are the latest, tested and verified versions by Supermicro so it is advisable to download them from this link or at least use the same versions if they are not available (in the future, now they are still available).
As you know LSI (they bought 3ware RAID in 2009) was bought by Avago (2013), then Avago bought Broadcom (2016 and renamed itself to Broadcom, 2018), so not so easy to find stuff for such old hardware (which still works). So this old MegaRAID controller is better managed by MegaCli despite you can do it with “storcli”, which is a modification of the tw_cli utility of 3ware RAID.
It could happen to lose your Grafana admin password and luckily there is a pretty easy way for the system administrator of the Grafana server to reset the admin password.
There is really good and simple page of how you can reset the admin password with grafana-cli in the documentation of Grafana website. Probably you can check it, too.
The problem is that most distributions like the one we use (CentOS 7) rearrange the home directory of the Grafana software and if you try to use grafana-cli you’ll end up with the error mentioned in the section below the solution we offer you here! And you really could mess it up and hurry to say this software is buggy!
First, “grafana-cli admin reset-admin-password ” needs to know where is your configuration file and Grafana home directory, because from the configuration it will extract what kind of back-end server you use like sqlite3 or MySQL (or other) and the path or login data if needed to the back-end, then it will reset the password with the given one. At the bottom of this article, there is another method of resetting the Grafana admin password without grafana-cli – manual reset in the database.
So first you should check with what command-line arguments was your instance of Grafana server started: Keep on reading!
Here are the commands needed to load a new kernel without rebooting your server or desktop computer. Why you need this? Sometime rebooting a server could take 5 to 10 minutes and loading a new kernel is just up to a minute. In fact in most cases loading the new kernel and starting the system then is just under 20-30 seconds, so upgrading your server even with new kernel is super easy lately. We tested it on CentOS 7 server and it was successful. The system uses systemd and the process is really easy and safe for the systems.
When the processes is initiated the system shutdowns normally (shutting down all running service with systemd) and then load the system immediately with the new kernel and starts the services as usual!
So no need to worry about unflushed data or not proper shutdown of a service! It’s like a normal reboot but without a hardware reboot and is a lot faster!
Here is what is required to load a kernel without hardware rebooting your computer box:
kexec-tools
Load the new kernel, initram file and the command line arguments with “kexec”
Here is a real world example with all the output:
As you can see it is important to load the initram file and the exact arguments to the kernel. You should take them from the grub 2 configuration – /boot/grub2/grub.cfg
Here is the real output:
[root@srv ~]# yum -y update
Loaded plugins: fastestmirror
Determining fastest mirrors
* base: mirrors.neterra.net
* extras: mirrors.neterra.net
* updates: mirrors.neterra.net
base | 3.6 kB 00:00:00
centos-sclo-rh | 3.0 kB 00:00:00
centos-sclo-sclo | 2.9 kB 00:00:00
extras | 3.4 kB 00:00:00
updates | 3.4 kB 00:00:00
(1/4): extras/7/x86_64/primary_db | 187 kB 00:00:00
(2/4): updates/7/x86_64/primary_db | 5.2 MB 00:00:01
(3/4): centos-sclo-sclo/x86_64/primary_db | 292 kB 00:00:01
(4/4): centos-sclo-rh/x86_64/primary_db | 3.7 MB 00:00:02
Resolving Dependencies
--> Running transaction check
---> Package kernel.x86_64 0:3.10.0-862.11.6.el7 will be installed
---> Package kernel-headers.x86_64 0:3.10.0-862.3.2.el7 will be updated
---> Package kernel-headers.x86_64 0:3.10.0-862.11.6.el7 will be an update
---> Package kernel-tools.x86_64 0:3.10.0-862.3.2.el7 will be updated
---> Package kernel-tools.x86_64 0:3.10.0-862.11.6.el7 will be an update
---> Package kernel-tools-libs.x86_64 0:3.10.0-862.3.2.el7 will be updated
---> Package kernel-tools-libs.x86_64 0:3.10.0-862.11.6.el7 will be an update
....
....
[root@srv ~]# yum install -y kexec-tools
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
* base: mirrors.neterra.net
* extras: mirrors.neterra.net
* updates: mirrors.neterra.net
Package kexec-tools-2.0.15-13.el7.x86_64 already installed and latest version
Nothing to do
So we performed an update and there was a new kernel kernel.x86_64 0:3.10.0-862.11.6.el7, which we would like to load without hardware reboot.
Here is our new kernel in “/boot”
[root@srv ~]# ls -altr /boot/
total 171812
-rw-------. 1 root root 3228420 22 Aug 2017 System.map-3.10.0-693.el7.x86_64
-rw-r--r--. 1 root root 140894 22 Aug 2017 config-3.10.0-693.el7.x86_64
-rw-r--r--. 1 root root 166 22 Aug 2017 .vmlinuz-3.10.0-693.el7.x86_64.hmac
-rwxr-xr-x. 1 root root 5877760 22 Aug 2017 vmlinuz-3.10.0-693.el7.x86_64
-rw-r--r--. 1 root root 293027 22 Aug 2017 symvers-3.10.0-693.el7.x86_64.gz
drwxr-xr-x. 3 root root 17 20 Feb 2018 efi
drwxr-xr-x. 2 root root 27 20 Feb 2018 grub
-rw-r--r--. 1 root root 611343 20 Feb 2018 initrd-plymouth.img
-rw-------. 1 root root 51315067 20 Feb 2018 initramfs-0-rescue-cc7889764e86441b8d1eb54e29e81a91.img
-rwxr-xr-x. 1 root root 5877760 20 Feb 2018 vmlinuz-0-rescue-cc7889764e86441b8d1eb54e29e81a91
-rw-------. 1 root root 3409912 21 May 23,50 System.map-3.10.0-862.3.2.el7.x86_64
-rw-r--r--. 1 root root 147823 21 May 23,50 config-3.10.0-862.3.2.el7.x86_64
-rw-r--r--. 1 root root 170 21 May 23,50 .vmlinuz-3.10.0-862.3.2.el7.x86_64.hmac
-rwxr-xr-x. 1 root root 6228832 21 May 23,50 vmlinuz-3.10.0-862.3.2.el7.x86_64
-rw-r--r--. 1 root root 304943 21 May 23,52 symvers-3.10.0-862.3.2.el7.x86_64.gz
dr-xr-xr-x. 17 root root 224 14 Jun 7,18 ..
-rw-------. 1 root root 12997841 14 Jun 7,19 initramfs-3.10.0-693.el7.x86_64kdump.img
-rw-------. 1 root root 20771492 14 Jun 7,20 initramfs-3.10.0-693.el7.x86_64.img
-rw-------. 1 root root 13007444 14 Jun 7,23 initramfs-3.10.0-862.3.2.el7.x86_64kdump.img
-rw-r--r--. 1 root root 147859 14 Aug 22,02 config-3.10.0-862.11.6.el7.x86_64
-rw-------. 1 root root 3414344 14 Aug 22,02 System.map-3.10.0-862.11.6.el7.x86_64
-rw-r--r--. 1 root root 171 14 Aug 22,02 .vmlinuz-3.10.0-862.11.6.el7.x86_64.hmac
-rwxr-xr-x. 1 root root 6242208 14 Aug 22,02 vmlinuz-3.10.0-862.11.6.el7.x86_64
-rw-r--r--. 1 root root 305158 14 Aug 22,05 symvers-3.10.0-862.11.6.el7.x86_64.gz
-rw-------. 1 root root 20774393 5 Sep 15,20 initramfs-3.10.0-862.11.6.el7.x86_64.img
drwx------. 5 root root 97 5 Sep 15,20 grub2
dr-xr-xr-x. 5 root root 4096 5 Sep 15,21 .
-rw-------. 1 root root 20782404 5 Sep 15,21 initramfs-3.10.0-862.3.2.el7.x86_64.img
Now we know the kernel and initram file names we just check the kernel arguments in the kernel, load them with kexec and start an systemd target to load the new kernel:
As you can see systemd performs a normal shutdown of all services and targets. Normal Shutdown
The ssh connection is immediately closed because the reboot is initiated.
After 10-15 seconds our host is alive and the new kernel is loaded successfully:
root@test ~ $ ssh root@srv
root@srv's password:
Last login: Wed Sep 5 15:16:08 2018 from test
[root@srv ~]# uname -a
Linux srv.local 3.10.0-862.11.6.el7.x86_64 #1 SMP Tue Aug 14 21:49:04 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
[root@srv ~]#
It’s been a while since a new tcp congestion has been developed (by Google), which has the potential to replace the default cubic one! The TCP BBR tries to keep being fast and improves the network connectivity when there is packet loss. Enabling it on the server does not require it to enable it on the client to function BBR algorithm properly. So probably we’ll see the switch to this new tcp congestion control in the next major linux distros releases.
Here we present how to enable TCP BBR on your CentOS 7 box. As you know the CentOS 7 uses kernel 3.10.X and the TCP BBR was introduced in version 4.9+ kernels and in version 4.13 and on it has a tcp-level pacing so you do not need the qdisc “fq” to be enabled, when using TCP BBR. So we need a new kernel version 4.9 and on and the easiest and safest way to do it to use the mainline kernel from https://www.elrepo.org/. We have a detailed howto for this if you need more information on using mainline kernel with your CentOS 7.
Here are the steps, you need to be root to execute the following commands:
We use technologies like cookies to store and/or access device information. We do this to improve browsing experience and to show (non-) personalized ads. Consenting to these technologies will allow us to process data such as browsing behavior or unique IDs on this site. Not consenting or withdrawing consent, may adversely affect certain features and functions.
Functional
Always active
The technical storage or access is strictly necessary for the legitimate purpose of enabling the use of a specific service explicitly requested by the subscriber or user, or for the sole purpose of carrying out the transmission of a communication over an electronic communications network.
Preferences
The technical storage or access is necessary for the legitimate purpose of storing preferences that are not requested by the subscriber or user.
Statistics
The technical storage or access that is used exclusively for statistical purposes.The technical storage or access that is used exclusively for anonymous statistical purposes. Without a subpoena, voluntary compliance on the part of your Internet Service Provider, or additional records from a third party, information stored or retrieved for this purpose alone cannot usually be used to identify you.
Marketing
The technical storage or access is required to create user profiles to send advertising, or to track the user on a website or across several websites for similar marketing purposes.
