Install CentOS 8 over the old OS and preserve the storage partitions

Always put your root partition separate from the storage (aka data) partitions. root partition should be only for system files and nothing more! Keeping this simple rule you may easily change your operating system (or clean install or clean upgrade) without deleting the user’s data thus preserving the old storage partitions.
Our storage has 2 storage partitions, which means they hosts only data and no system files and there are separate partitions for Linux booting (grub2) and system files (root partition). Here is the partitions layout:

[root@srv0 ~]# parted /dev/sda --script print
Model: AVAGO SMC3108 (scsi)
Disk /dev/sda: 48.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  53.7GB  53.7GB                     raid
 2      53.7GB  54.8GB  1075MB                     raid
 3      54.8GB  55.0GB  211MB   fat16              raid
 4      55.0GB  69.4GB  14.4GB                     raid
 5      71.8GB  48.0TB  47.9TB  ext4

[root@srv0 ~]# parted /dev/sdb --script print
Model: AVAGO SMC3108 (scsi)
Disk /dev/sdb: 48.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  53.7GB  53.7GB                     raid
 2      53.7GB  54.8GB  1075MB                     raid
 3      54.8GB  55.0GB  211MB   fat16              raid
 4      55.0GB  69.4GB  14.4GB                     raid
 5      71.8GB  48.0TB  47.9TB  ext4
[root@srv0 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G  250M   16G   2% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/md127       50G  1.7G   45G   4% /
/dev/md125      991M  133M  792M  15% /boot
/dev/md124      201M   12M  190M   6% /boot/efi
/dev/sda5        44T   23T   22T  52% /mnt/storage1
/dev/sdb5        44T   14T   30T  32% /mnt/storage2
tmpfs           3.2G     0  3.2G   0% /run/user/0

Of course, when there are partitions above 2T the GPT is mandatory.
You can skip the software RAID1 setup if you use only one controller or you have system partitions only in one disk (virtual drive and so on). Here we have two hardware controllers, which we want to use both for the system partitions.
4 RAID1 devices:

  1. EFI partition (/boot/efi)
  2. swap partition
  3. boot parition (/boot)
  4. root partition (/)

The best practice is have total between 30G and 50G for the 4 partitions (in fact, boot partition could be skipped). Have in mind most modern Linux distributions cannot be installed on less than 10G~20G and for optimal results just separate between 30G and 50G for 4 partitions above (or 3 if you choose to skip the boot one).

Upgrade to CentOS 8 with clean install over our old CentOS 7 system partitions preserving the big data partitions.

Couple of things before start:

  • UEFI installation will be selected. So boot in UEFI mode.
  • IPMI KVM is used to install the new Linux distribution – CentOS 8
  • The installation disk is mounted in the Virtual CD/DVD IPMI KVM device – with Mount and boot ISO file from windows share in Supermicro IPMI Virtual media (CD-ROM)
  • All system parititions will be removed (grub, boot, root) and a clean minimal installation will be performed.
  • Network installation – using CentOS-8-x86_64-1905-boot.iso

SCREENSHOT 1) The Server is starting. This is the IPMI KVM window. Press F11 to Boot in Boot Menu.

main menu
SUPERMICRO Server starting – KVM

Keep on reading!

Install gitlab-ce (community edition) in docker container with HTTPS and docker registry

This article is a howto install of the official docker gitlab-ce (GitLab Community Edition). GitLab maintains a docker image in the Docker registry and this is the best way to install GitLab.
In this article you are going to learn how:

  • to install the GitLab CE in docker
  • to enable HTTPS (SSL) web support to your GitLab
  • to enable the docker registry functionality of GitLab

To install GitLab docker image in your Linux distribution all you need is a working docker environment and started docker daemon. As you know, installing software with docker will allow you to keep your main system clean and let you use a fined tuned installation from the official developer (creator). As mentioned already, the GitLab maintains an official GitLab image in the Docker Registry so you may expect everything to work smoothly and better than if you make an installation in a clean Linux distribution like CentOS, Ubuntu and so on. In this article, we will include the most important docker commands to control and configure the GitLab docker container and even if you are not familiar with the Docker software they are simple enough to use them and prefer this method over GitLab normal installation.

GitLab has integrated the Docker Container Registry in GitLab Container Registry and now with GitLab you can have a local Docker registry containing all project’s docker images!

Just to note, the Docker Registry is the place for the Docker (aka Linux) images.
Using GitLab Container Registry with CI/CD (continuous integration and continuous delivery) you can create automatically test, staging, development and production docker images.
Keep on reading!

Debug Ubuntu preseed failure – select and install software

Preparing the preseed for unattended installation sometimes could be challenging. This article presents the right way to analyze an installation failure in one of the main steps – “select and install software”.
There is a ubuntu installation preseed file for our Bionic unattended installation, which uses the “pkgsel” to install first packages in the new system:

d-i pkgsel/include string openssh-server wget vim git python gpg ntp
d-i pkgsel/upgrade select full-upgrade
d-i pkgsel/update-policy select unattended-upgrades

When an installation step in the preseed of a unattended installation fails the setup stops with a “Continue” confirmation.

main menu
“select and install software” – step failed

Here is what you can do to check what exactly fails in step “select and install software”:

  1. Start a shell in the current installation boot. Press “Ctrl+Alt+F2” to start the shell. You may use “Ctrl+Alt+F3” and “Ctrl+Alt+F4” for two more consoles and “Ctrl+Alt+F1” to return to the installation wizard.
  2. Check the /var/log/syslog, in which file the debconf writes the logging information.
  3. Find the lines where the step “select and install software” starts and look for errors after that. In this file, you can see all the step titles during, which the setup passes and they are named the same way the windows’ titles during the installation wizard.

Here is the real world output

Presing the “CTRL+ALT+F2” to start the BusyBox built-in shell, which is ash not bash!

Be careful there are some difference between ash and bash.

main menu
Installation wizard – BusyBox built-in shell (ash)

Last 20 lines shows the problem – pkgsel failed to install packages in step “select and install software”.

The installation wizard stops.

main menu
debconf logging using syslog – pkgsel

The problem is in the package “ntp”, the setup cannot install the “ntp” package because of unmet dependencies.

Because it is not so important to install ntp at this stage we added the package to the script executed in “preseed/late_command” and removed the package from the pkgsel line in the preseed file. In general, our problem was because we set local repositories for the bionic packages, but the setup cannot update list of available packages when the you set Bionic mirror to be unofficial local repository.

main menu
Package because of unmet dependencies

bonding – write error – device or resource busy – operation not permitted

Recently, there was a little bit of confusion when following the article about activating network bonding without ifenslave – How to enable Linux bonding without ifenslave. At first, there were couple of errors:

livecd ~ # echo balance-alb > /sys/class/net/bond0/bonding/mode
-bash: echo: write error: Device or resource busy
livecd ~ # echo "+enp129s0f0" > /sys/class/net/bond0/bonding/slaves
-bash: echo: write error: Operation not permitted

Or similar error when changing the bonding mode:

livecd ~ # echo 4 > /sys/class/net/bond0/bonding/mode
-bash: echo: write error: Directory not empty
livecd ~ # echo 802.3ad > /sys/class/net/bond0/bonding/mode
-bash: echo: write error: Directory not empty

The server just booted in rescue live cd and there is no active network configuration:

SCREENSHOT 1) Apparently, the /sys/class/net/bond0/bonding/mode and /sys/class/net/bond0/bonding/slaves are in read only state.

No writes means no new configuration could be installed and the bonding cannot be configured (reconfigured).

main menu
device or resource busy – operation not permitted

Bonding mode could be changed only when the bonding device is in DOWN state.

Network interfaces could be added to the boding device only if they were in DOWN state, too.

In addition, changing bonding mode could only happen if there were no network interfaces added to the bonding interface.

Keep on reading!

Adding bonding interface to CentOS 8 – editing configuration files only

This article shows what files to add if you want to add a bonding interface under CentOS 8 without invoking the Network manager command utility.
Our goal is to use one boding group with the name bond0 in LACP (aka 802.3ad) mode (but it could be any of the other types) with two networks 10Gbps interfaces. The setup resented here uses NetworkManager, which handles the loading of bonding module properly.

In fact, the network-scripts are now deprecated and they are missing from the system (but they still exist in the additional package – “network-scripts”, how knows till when? do not rely on them!).

The configuration files are with the same syntax as under CentOS 7, but this time the network manager parses them. The ifup and ifdown still exist and they just call the Network manager when executed (unless the “network-scripts” package is installed). If you need to enable bonding without any configuration files (for emergency situations) you may still use – How to enable Linux bonding without ifenslave

What do you need:

  • Ensure you have installed: “iputils” and “NetworkManager” packages
    dnf install -y NetworkManager iputils
    
  • Ensure the NetworkManager service is running
    systemctl enable NetworkManager
    systemctl start NetworkManager
    

STEP 1) Configure the bonding device

The boding interface’s name will be bond0 and the configuration will be located in /etc/sysconfig/network-scripts/ifcfg-bond0

BONDING_OPTS="mode=4 miimon=100"
TYPE=Bond
BONDING_MASTER=yes
BOOTPROTO=none
IPADDR0=192.168.0.100
PREFIX0=24
GATEWAY0=192.168.0.1
DNS1=8.8.8.8
DNS2=8.8.4.4
IPV4_FAILURE_FATAL=no
NAME=bond0
UUID=e19e2059-2e31-4143-915a-cdc11d19c9d6
DEVICE=bond0
ONBOOT=yes

Keep on reading!

Install aptly under Ubuntu 18 LTS with nginx serving the packages and the first steps

This article is how to install aptly software, which offers easy Debian repository management.
First, few words for aptly and what tasks are really simple to do:

  • Mirror an existing (remote) repository. Make a local copy of Debian or Ubuntu repostories for all your internal infrastructure.
  • Create your own repositories
  • Create snapshots of repositories and mirrors.
  • Merge repositories
  • Make diff between repositories (in fact snapshots of repositories, but you may make a mirror of an repository and then make a snapshot and then make a diff with some other snapshot to see the changes between the different repositories or the time the snapshots are made).
  • Remove or add individual packages from official mirrored repositories.
  • Use api calls to manage the repositories. HTTP REST API is still in development, but a big part of it works.

For more information you may visit the official documentation page – https://www.aptly.info/doc/overview/

We are going to install the aptly and despite it could be used to serve the repository files we will use the Nginx web server for this work. Nginx is a more fast and reliable web server with easy installation of SSL certificates for our repositories.
The aptly is included in official Ubuntu repositories in the component universe, but at present, it is 2 to 3 versions behind the stable one from the aptly site, so we are going to use their repository to install aptly. Still, if you do not want to use
Keep on reading!

CentOS 8 dracut-initqueue timeout and could not boot – warning /dev/disk/by-id/md-uuid- does not exist – inactive raids

Booting the CentOS 8 failed with

dracut-initqueue timeout and could not boot – warning /dev/disk/by-id/md-uuid- does not exist

we have an article on the subject for CentOS 7 – CentOS 7 dracut-initqueue timeout and could not boot – warning /dev/disk/by-id/md-uuid- does not exist and we continue with another issue with the same error.
Most times when you get this error there is a mistake in some UUID for the file system or the RAID devices. But this time our software raid was autodetected with the right disks but it ended in INACTIVE STATE. Software raid in INACTIVE STATE cannot be used so we are in “Emergency mode”:

SCREENSHOT 1) Without root partition the boot process enters the dracut emergency mode.

main menu
Entering emergency mode

SCREENSHOT 2) Software md raid device information – missing “Personalities” for the raid groups.

Loaded modules in the kernel and missing raid kernel modules.

main menu
Missing raid1 kernel module in initram file

To summarize it up:

  • The disks are detected, so we drivers for SATA/SAS controller is loaded correctly.
  • The software raid autodetected the MD devices, but they are in “INACTIVE STATE”. The RAID “Personalities” is missing.

Keep on reading!

Gentoo – updating perl and problems like perl-core/ is blocking virtual/perl-

When upgrading Perl under Gentoo it’s almost typical to have blocks of the kind:

[blocks B      ] <perl-core/File-Path-2.160.0 ("<perl-core/File-Path-2.160.0" is blocking virtual/perl-File-Path-2.160.0)
[blocks B      ] <perl-core/Module-CoreList-5.201.905.220 ("<perl-core/Module-CoreList-5.201.905.220" is blocking virtual/perl-Module-CoreList-5.201.905.220)
[blocks B      ] <perl-core/Archive-Tar-2.320.0 ("<perl-core/Archive-Tar-2.320.0" is blocking virtual/perl-Archive-Tar-2.320.0)

These blocks appeared when we tried updaring the Perl 5.26 to 5.30 (dev-lang/perl-5.30.0:0/5.30::gentoo from dev-lang/perl-5.26.2:0). This update is a major update.
So what does it mean?
The virtual package and the perl-core package should be the same version, but it appears there are no corresponding same versions to the virtual package versions (). The problem is that the update depends on a newer version of File-Path, Module-CoreList, and Archive-Tar but there are only virtual packages with the required versions. Virtual packages are just meta-packages and they do not install any module. First, what are the virtual/[perl-module] and perl-core/[perl-module]:

  • perl-core/[perl-module] – perl independent modules. In fact, the same module may be independent and it may be part of the build-in packages from the Perl main package – dev-lang/perl. Sometimes you may choose the independent package, because of a newer version than the version included in the dev-lang/perl package.
  • virtual/[perl-module] – perl meta-package, which ensures the installation of the module. The module could be an independent package from above or module, which is included in the main Perl package (in dev-lang/perl)

To solve the blocks of this kind “perl-core/ is blocking virtual/perl-” you must check for a newer version of perl-core/* if there is no newer version, just to remove the package and let the setup (emerge) use the module included in dev-lang/perl.

In our case, there are no newer versions of the three packages and we remove them with:

emerge -Cv perl-core/File-Path perl-core/Module-CoreList perl-core/Archive-Tar

And then emerge with no problems the new version of Perl with:

emerge --verbose --verbose-conflicts --backtrack=200000 $(qlist -IC|grep perl|sort|uniq) media-gfx/inkscape media-gfx/imagemagick dev-php/pecl-imagick

For detail explaination of why we use verbose-conflicts, backtrack, qlist and three excplicitly added packages you may read our article on similar subject – Gentoo updating perl with many masked and blocked packages
You may read the offcial documentation for more information – https://wiki.gentoo.org/wiki/Perl#Introduction

Updating our Perl from dev-lang/perl-5.26.2:0 to dev-lang/perl-5.30.0:0/5.30

Keep on reading!

Install Cobbler 2.6 under Ubuntu 16.04 LTS from Cobbler source – manual intallation

This article will show how to install Cobbler 2.6.11 (the last from the 2.6 branches) under Ubuntu 16.04 LTS. There is Cobbler package version 2.4.x in the Ubuntu 16 LTS, but Cobbler 2.4 is really old and some options and features are not available anymore, so installing from the Ubuntu package system Cobbler 2.4 would probably just waste your time and you will move to 2.6 or even later. In fact, Ubuntu 18 LTS removed the Cobbler packages at all and only manual install (aka installation from sources is only available or from an anonymous repository like PPA).
Here are the steps to install a working Cobbler 2.6 under Ubuntu 16.04 (Xenial Xerus).

Note: All commands are executed as root user, so it is much easier to just execute ones “sudo su” before you begin following the steps below.

STEP 1) Update your system to the latest state and install the Cobbler 2.6.11 dependencies

Update the system:

apt update
apt upgrade -y

Install the Cobbler 2.6 dependencies:

apt install -y make git python-yaml python-cheetah python-netaddr python-simplejson libapache2-mod-wsgi python-django atftpd debmirror apache2 python-urlgrabber fence-agents isc-dhcp-server

STEP 2) Download and install the Cobbler 2.6.11 source

cd /root
wget https://github.com/cobbler/cobbler/archive/v2.6.11.tar.gz
tar xf v2.6.11.tar.gz
cd cobbler-2.6.11
make install

This will install Cobbler and Cobbler web by replacing all configurations of old Cobbler install! The python files will be installed under “/usr/local”, so a big part of Cobbler will be installed in /usr/local/lib/python2.7/dist-packages/cobbler.

STEP 3) Enable Cobbler in apache2

a2enconf cobbler cobbler_web
a2enmod proxy proxy_http rewrite ssl
a2ensite default-ssl
systemctl enable apache2

Enable apache2 configuration file for cobbler web, enable the apache2 modules and the default HTTPS apache2 virtual host, because the web interface won’t work with HTTP. Opening the Cobbler web interface using HTTP will result in forbidden error (HTTP error 403). The Cobbler web interface is limited to use HTTPS.
Keep on reading!

Cobbler web – AppRegistryNotReady – Apps aren’t loaded yet

If you happen to use Cobbler 2.6 (even the last release of 2.6 – 2.6.11) you may encounter the following error when you try loading the Cobbler web interface:

AppRegistryNotReady at /distro/list

Apps aren't loaded yet.

Request Method: 	GET
Request URL: 	https://192.168.0.25/cobbler_web/distro/list
Django Version: 	1.8.7
Exception Type: 	AppRegistryNotReady
Exception Value: 	

Apps aren't loaded yet.

Exception Location: 	/usr/lib/python2.7/dist-packages/django/apps/registry.py in check_apps_ready, line 124
Python Executable: 	/usr/bin/python
Python Version: 	2.7.12
Python Path: 	

['/usr/local/share/cobbler/web/cobbler_web',
 '/usr/local/share/cobbler/web',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages']

Server time: 	Sun, 13 Oct 2019 21:24:30 -0400

The Cobbler web does not work properly. The Django application is not started properly.

main menu
Cobbler web – web error

It is a known error and it is easily fixable by editing the Cobbler web source file “/usr/local/share/cobbler/web/cobbler.wsgi” (or /usr/share/cobbler/web/cobbler.wsgi) and replacing the two lines:

    import django.core.handlers.wsgi
    _application = django.core.handlers.wsgi.WSGIHandler()

With the code:

    from django.core.wsgi import get_wsgi_application
    _application = get_wsgi_application()

Note the “_application” variable starts with underline.

In fact, Cobbler 2.4 has the same issue, but the variable is “application” without underline in the start and there is no return line at the end of the file (in 2.6 there is a return clause and you must leave it there!)

import django.core.handlers.wsgi
application = django.core.handlers.wsgi.WSGIHandler()

replace with

from django.core.wsgi import get_wsgi_application
application = get_wsgi_application()

It is highly unlikely to use Cobbler 2.4, but still, if you need it and you come across this page this is the fix for the Cobbler web and be careful with the variable name _applicationapplication for the various Cobbler versions.