CentOS 8 add a storage driver (megaraid_sas) when booting the installation disk

Installing CentOS 8 in relatively old hardware maybe a real challenge because of an old hardware device like storage, network, or both.
This article shows how to make the CentOS 8 Installation wizard detect the storage – a hardware controller AOC-USAS2LP-H8iR (smc2108 with LSI 2108). Unfortunately, the CentOS 8 (in fact, RHEL 8 removed the support, too) team decided to remove support for the LSI SAS2008/2108/2116 storage controllers by removing the “megaraid_sas” kernel driver. There are still servers in production with similar controllers, which were sold 4-5 years ago from the big vendors such as DELL, HP, and so on.

The method here is to boot the installation CD/USB with modified kernel boot parameters to include an URL link to the installation driver iso (where the megaraid_sas driver is included).

The offered way to load the megaraid_sas (or any other driver) includes:

  1. Use assisted driver update to load an elrepo driver ISO during the first stage of the CentOS 8 Installation Wizard. elrepo is a famous community efford – http://elrepo.org/tiki/tiki-index.php. More on the assited diver update here – https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/8/html/performing_an_advanced_rhel_installation/updating-drivers-during-installation_installing-rhel-as-an-experienced-user#performing-an-assisted-driver-update_updating-drivers-during-installation
  2. Configure the network of the server to be able to download the driver ISO in the early stage of the CentOS 8 Installation Wizard. Add boot parameters to set up a valid network configuration.

The installation CD/USB can download an iso with kernel drivers. And of course, to download a file from the Internet a network should be set in the earliest stage of the CentOS 8 installation wizard.
The added string to the boot CD/USB CentOS 8 Installation disk is:

 inst.dd=https://elrepo.org/linux/dud/el8/x86_64/dd-megaraid_sas-07.710.50.00-1.el8_2.elrepo.iso ip=10.10.10.10::10.10.10.1:255.255.255.0::enp8s0f0:off nameserver=8.8.8.8

SCREENSHOT 1) Select with the arrows “Install CentOS Linux 8” and hit “TAB” button to edit the boot parameters.

As shown in the picture just add ” inst.dd=https://elrepo.org/linux/dud/el8/x86_64/dd-megaraid_sas-07.710.50.00-1.el8_2.elrepo.iso ip=10.10.10.10::10.10.10.1:255.255.255.0::enp8s0f0:off nameserver=8.8.8.8″. The “inst.dd” instructs the installation wizard where are the driver ISO located. The “ip” and “nameserver” command just sets a proper network in the early stage of the CentOS 8 Installation wizard to be able to download the driver ISO. Setting the network by these parameters is really important, because the download of the driver iso happens in this early stage of loading the installation wizard. Replace the IP and the whole network configuration if needed.

main menu
Installation wizard edit boot parameters

Keep on reading!

Set up (802.3ad LACP) bonding when installing CentOS 8

This article is to show how the user could install CentOS 8 (the steps are the same with CentOS 7) with a much complex network setup such as Bonding device in 802.3ad mode (LACP – Link Aggregation Control Protocol).
The whole installation procedure is not included here, but there are couple of other article on the subject “Install CentOS 7 or CentOS 8”:

Similar configuration files will be generated as in Configure Bond (802.3ad LACP) device in CentOS 8 – configuration files

SCREENSHOT 1) Click on “Network and Host Name” to configure the machine networking.

main menu
Installation Summary – Network and Host Name

Keep on reading!

syslog – UDP local to rsyslog and send remote with TCP and compression

This article is to show how to log Nginx’s access logs locally using UDP to the local rsyslog daemon, which will send the logs to a remote rsyslog server using TCP and compression. In general, logs could generate a lot of traffic and using UDP over distant locations could result in packet loss respectively logs’ lines loss. The idea here is to log messages locally using UDP (non-blocking way) to a local Syslog server, which will send the stream to a remote central Syslog server using TCP connections to be sure no packets are lost. In addition, we are going to enable local caching (if the remote server is temporary unreachable) and compression between the two Syslog servers.
Our goal is to use

  • UDP for our client program (Nginx in the case) for non-blocking log writes.
  • TCP between our local machine and the remote syslog server – to be sure not to lose messages on bad connectivity.
  • local caching for our client machine – not to lose messages if the remote syslog is temporary unreachable.
  • compression between the local machine and the remote syslog server.

The configuration and the commands are tested on CentOS 7, CentOS 8 and Ubuntu 18 LTS. Check out UDP remote logging here – nginx remote logging to UDP rsyslog server (CentOS 7).

STEP 1) Configure client’s local rsyslog to accept UDP log messages only if the messages’ tags are “nginx”

Couple of things should be enabled in the local client-size rsyslog daemon:

  • rsyslog to accept UDP messages. Uncomment or add the following under section “Modules” (probably the first section in the file?) in /etc/rsyslog.conf
    $ModLoad imudp
    $UDPServerRun 514
    

    or

    module(load="imudp")
    input(type="imudp" port="514")
    

    The first is the old syntax, which is still supported and the second is the new syntax. For simplicity, all of the following configuration will be using the new syntax, because the old one is depricated.

  • Add a rule to catch the tag containing “nginx” and execute action to forward the messages to the remote server
    if ($syslogtag == 'nginx:') then {
    action(type="omfwd" target="10.10.10.10" port="10514" protocol="tcp" compression.Mode="single" ZipLevel="9"
    queue.filename="forwarding" queue.spoolDirectory="/var/log" queue.size="1000000" queue.type="LinkedList" queue.maxFileSize="1g" queue.SaveOnShutdown="on"
    action.resumeRetryCount="-1")
    & stop
    }
    
  • The options are almost self-explanatory, the important ones are there is no retry limit count of reconnecting to the server, there is in-disk cache of maximum 1G if the remote server is unavailable and the compression per message is turned on. More on actionshttps://www.rsyslog.com/doc/v8-stable/configuration/actions.html, the forward modulehttps://www.rsyslog.com/doc/v8-stable/configuration/modules/omfwd.html and the queuehttps://www.rsyslog.com/doc/v8-stable/rainerscript/queue_parameters.html

And restart the rsyslog:

systemctl restart rsyslog

Keep on reading!

Install Fedora 31 KDE Plasma Desktop (KDE GUI)

This tutorial will show you the simple steps of installing a modern Linux Distribution Fedora 31 KDE Plasma Desktop with KDE for the user graphical interface. First, we present the basic steps for installing the Operating system in addition to your present operating systems (here we have two: Windows 10) and then you can see some screenshots of the installed system and the look and feel of it. We have another tutorial showing more screenshots of the installed and working Fedora 31 (Gnome and KDE plasma) – so you can decide which of them to try first – coming soon.

The Fedora 31 KDE Plasma Desktop comes with

  • Xorg X server – 1.20.5 XWayland is used by default
  • linux kernel – 5.3.7
  • KDE Plasma version: 5.16.5
  • KDE Frameworks version: 5.61.0
  • QT version: 5.12.5

The installation process is very similar to the old Install Fedora 27 KDE Plasma Desktop and Install Fedora 29 KDE Plasma Desktop (KDE GUI). Our system is relatively new – Asus X399 with AMD Ryzen Threadripper 1950X and NVIDIA 1080 TI and the setup loaded successfully and there were no problems till the end.

We used the following ISO for the installation process:

https://download.fedoraproject.org/pub/fedora/linux/releases/31/Spins/x86_64/iso/Fedora-KDE-Live-x86_64-31-1.9.iso

It is a LIVE image so you can try it before installing it. The easiest way is just to download the image and burn it to a DVD disk and then follow the installation below:

SCREENSHOT 1) Here is our “UEFI BIOS->Boot->Boot Override” and in most modern motherboard you can choose to override the default boot devices.

Choose the “UEFI: HL-DT-STDVDRAM…” to boot and install Fedora KDE Plasma Desktop 31 with UEFI support. You should do this, because most of the new hardware like video cards would not work properly without beeing in UEFI mode.

main menu
Boot from DVD/USB Installation

Keep on reading!

Install CentOS 8 over the old OS and preserve the storage partitions

Always put your root partition separate from the storage (aka data) partitions. root partition should be only for system files and nothing more! Keeping this simple rule you may easily change your operating system (or clean install or clean upgrade) without deleting the user’s data thus preserving the old storage partitions.
Our storage has 2 storage partitions, which means they hosts only data and no system files and there are separate partitions for Linux booting (grub2) and system files (root partition). Here is the partitions layout:

[root@srv0 ~]# parted /dev/sda --script print
Model: AVAGO SMC3108 (scsi)
Disk /dev/sda: 48.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  53.7GB  53.7GB                     raid
 2      53.7GB  54.8GB  1075MB                     raid
 3      54.8GB  55.0GB  211MB   fat16              raid
 4      55.0GB  69.4GB  14.4GB                     raid
 5      71.8GB  48.0TB  47.9TB  ext4

[root@srv0 ~]# parted /dev/sdb --script print
Model: AVAGO SMC3108 (scsi)
Disk /dev/sdb: 48.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  53.7GB  53.7GB                     raid
 2      53.7GB  54.8GB  1075MB                     raid
 3      54.8GB  55.0GB  211MB   fat16              raid
 4      55.0GB  69.4GB  14.4GB                     raid
 5      71.8GB  48.0TB  47.9TB  ext4
[root@srv0 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G  250M   16G   2% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/md127       50G  1.7G   45G   4% /
/dev/md125      991M  133M  792M  15% /boot
/dev/md124      201M   12M  190M   6% /boot/efi
/dev/sda5        44T   23T   22T  52% /mnt/storage1
/dev/sdb5        44T   14T   30T  32% /mnt/storage2
tmpfs           3.2G     0  3.2G   0% /run/user/0

Of course, when there are partitions above 2T the GPT is mandatory.
You can skip the software RAID1 setup if you use only one controller or you have system partitions only in one disk (virtual drive and so on). Here we have two hardware controllers, which we want to use both for the system partitions.
4 RAID1 devices:

  1. EFI partition (/boot/efi)
  2. swap partition
  3. boot parition (/boot)
  4. root partition (/)

The best practice is have total between 30G and 50G for the 4 partitions (in fact, boot partition could be skipped). Have in mind most modern Linux distributions cannot be installed on less than 10G~20G and for optimal results just separate between 30G and 50G for 4 partitions above (or 3 if you choose to skip the boot one).

Upgrade to CentOS 8 with clean install over our old CentOS 7 system partitions preserving the big data partitions.

Couple of things before start:

  • UEFI installation will be selected. So boot in UEFI mode.
  • IPMI KVM is used to install the new Linux distribution – CentOS 8
  • The installation disk is mounted in the Virtual CD/DVD IPMI KVM device – with Mount and boot ISO file from windows share in Supermicro IPMI Virtual media (CD-ROM)
  • All system parititions will be removed (grub, boot, root) and a clean minimal installation will be performed.
  • Network installation – using CentOS-8-x86_64-1905-boot.iso

SCREENSHOT 1) The Server is starting. This is the IPMI KVM window. Press F11 to Boot in Boot Menu.

main menu
SUPERMICRO Server starting – KVM

Keep on reading!

Debug Ubuntu preseed failure – select and install software

Preparing the preseed for unattended installation sometimes could be challenging. This article presents the right way to analyze an installation failure in one of the main steps – “select and install software”.
There is a ubuntu installation preseed file for our Bionic unattended installation, which uses the “pkgsel” to install first packages in the new system:

d-i pkgsel/include string openssh-server wget vim git python gpg ntp
d-i pkgsel/upgrade select full-upgrade
d-i pkgsel/update-policy select unattended-upgrades

When an installation step in the preseed of a unattended installation fails the setup stops with a “Continue” confirmation.

main menu
“select and install software” – step failed

Here is what you can do to check what exactly fails in step “select and install software”:

  1. Start a shell in the current installation boot. Press “Ctrl+Alt+F2” to start the shell. You may use “Ctrl+Alt+F3” and “Ctrl+Alt+F4” for two more consoles and “Ctrl+Alt+F1” to return to the installation wizard.
  2. Check the /var/log/syslog, in which file the debconf writes the logging information.
  3. Find the lines where the step “select and install software” starts and look for errors after that. In this file, you can see all the step titles during, which the setup passes and they are named the same way the windows’ titles during the installation wizard.

Here is the real world output

Presing the “CTRL+ALT+F2” to start the BusyBox built-in shell, which is ash not bash!

Be careful there are some difference between ash and bash.

main menu
Installation wizard – BusyBox built-in shell (ash)

Last 20 lines shows the problem – pkgsel failed to install packages in step “select and install software”.

The installation wizard stops.

main menu
debconf logging using syslog – pkgsel

The problem is in the package “ntp”, the setup cannot install the “ntp” package because of unmet dependencies.

Because it is not so important to install ntp at this stage we added the package to the script executed in “preseed/late_command” and removed the package from the pkgsel line in the preseed file. In general, our problem was because we set local repositories for the bionic packages, but the setup cannot update list of available packages when the you set Bionic mirror to be unofficial local repository.

main menu
Package because of unmet dependencies