linux | Any IT here? Help Me!

How to upgrade to CentOS Stream 9 from CentOS Stream 8

This article will show how to update to CentOS Stream 9 from CentOS Stream 8.

If only official repositories are used it is fairly easy to upgrade to the new rolling based CentOS Stream 9 release, which follows the Red Hat Enterprise Linux 9. Using unofficial or users’ repositories CentOS Stream 8 may lead to unstable system or break user’s system following this tutorial. First, check out for CentOS Stream 9 compatibility issues the currently installed repositories if some specific or unofficial are used.
The following article uses a CentOS Stream 9 with only the official default repositories and in addition, the EPEL repository, which has a CentOS Stream 9 support.
How to install – Network installation of CentOS Stream 9 (20220606.0) – minimal server installation with additional information about important CentOS Stream 9 URLs and links and what kind of software the user can expect to have Software and technical details of CentOS Stream 9 minimal install. Camparing the packages of the two systems – a clean install of CentOS Stream 9 and an upgrade from CentOS Stream 8, the clean install has 377 packages installed and the upgrade has 387 packages installed. The strated processes are the same except the chronyd service is not started on boot. chronyd daemon offers a time synchronization.

STEP 1) All installed packages should be updated to the latest versions.

[root@srv ~]# dnf update -y
Last metadata expiration check: 0:09:08 ago on Tue Oct  4 12:12:07 2022.
Dependencies resolved.
Nothing to do.
Complete!

The DNF tool reports that all packages are up-to-date, because there is nothing to upgrade.

STEP 2) A cleanup of all packages, which are not required anymore.

Check orphan and leaves packages. Some of the packages here may be used by the user explicitly, so it is important to know the system. Remove the packages if not used by the system and they are showed as an output of the following commands and they are not used by the user of the system.

[root@srv ~]# dnf repoquery --unneeded
Last metadata expiration check: 0:12:56 ago on Tue Oct  4 12:12:07 2022.
NetworkManager-initscripts-updown-1:1.40.0-1.el8.noarch
grub2-tools-efi-1:2.02-129.el8.x86_64
libmetalink-0:0.1.3-7.el8.x86_64
pciutils-0:3.7.0-1.el8.x86_64
python3-configobj-0:5.0.6-11.el8.noarch
python3-schedutils-0:0.6-6.el8.x86_64
rdma-core-0:41.0-1.el8.x86_64
[root@srv ~]# dnf repoquery --extras
Last metadata expiration check: 0:00:12 ago on Tue Oct  4 12:34:17 2022.

Remove the unneeded packages:
Keep on reading!

git status and bus error on SSD – fix READ errors by recovering part of the file

SSD and Linux encryption may not be the best idea, especially without the TRIM (allow-discards) option (or never executed fstrim?). Nevertheless, this error may occur not only on an SSD device, but just where there is a corrupted file system or device.
In our case, the SSD has some read errors. Apparently, some files or some parts of files could not be read by the git command:

[myuser@dekstop kernel]# git status -v
Bus error 84/115708)

In the case of SSD bad reads, the only working solution is to find and overwrite the problem file(s) or remove the file(s) and recreate them. A more sophisticated solution is to dump the file with dd and skip errors option enabled to another location and then overwrite the old file with the new one. So only the corrupted area of the file will be lost, which in most cases is just one or two sectors, i.e. one or two 512 bytes of data.

STEP 1) Find the bad files with the find command.

Use find Linux command and read all the files with the cat Linux command, so a bad sector will output an input/output error on READ. On write errors won’t be generated, but the sector will be automatically moved to a healthy one (the bad sector is marked and never used more).

[myuser@dekstop kernel]#  find -type f -exec cat {} > /dev/null \;
cat: ./servers/logo_description.txt: Input/output error

If multiple files are found repeat the procedure with each file.

STEP 2) Copy the healthy portion of the file.

The easiest way to remove the error is just to delete the file (or overwrite it), but if the healthy portion of the file is desirable the dd utility may be used to recover it:
Keep on reading!

MPEG-DASH and ClearKey, CENC drm encryption with Nginx, bento4 and dashjs under CentOS 8

The purpose of this article is to demonstrate a simple and plain example of ClearKey DRM encryption using a DASH stream.
Usually, the ClearKey is used only for testing the encryption key and the DRM setup, because the decrypting key is transferred in a plain text to the browser. In simple DRM words, the key is transferred in plain text, and the handle of the decryption is not in some proprietary module such as CMD – Content Decryption Modules. The CMD is a proprietary module in the browsers or the players, which works like a black box when handling the decryption key. The most popular DRMs are Google’s Widevine, Apple’s Fireplay, and Microsoft PlayReady, which work through a proprietary module – CMD (Content Decryption Modules) in the browser (or the OS and player).
All the three DRMs work basically in a similar way:

There is a (encryption) key and a (encryption) keyID, which purpose is to identify the (encryption) key.
The video file is encrypted with the key and it includes the keyID.
The client needs to have the appropriate CMD (Content Decryption Modules) to decrypt the video.
The clients receive a license from a license server, which is encrypted data for the CDM on how to decrypt the video identified by the keyID. In fact, the client sends the keyID and receives the proper license (i.e. license binary data) for this keyID. That’s why keyID is included in the encrypted video. Bare in mind, the CMD is proprietary Content Decryption Module offered by the creator of the DRM – Google, Apple, Microsoft or another and it lives in the browser (OS or player). All popular browsers support at least one of the proprietary DRMs.

ClearKey is like the proprietary DRM schemes, but without the CMD (Content Decryption Modules).

The “org.w3.clearkey” Key System uses plain-text clear (unencrypted) key(s) to decrypt the source. No additional client-side content protection is required.

So, in general, there is no need for a license server when using ClearKey DRM.
Of course, an additional attempt to hide the plain-text key could be made using an extension to the client’s player such as javascript modules and etc. In general, it is perceived this approach to be less secure, because it is much easier to debug the javascript code on the client side. More on ClearKey – https://www.w3.org/TR/encrypted-media/#clear-key

Here are all the steps from the server till the client to use ClearKey.

STEP 1) Download and install bento4 software.

bento4 is an open source toolkit for manipulating some of the most common video formats – MP4 and DASH/HLS/CMAF media. The download page is https://www.bento4.com/downloads/ and the Linux binary for latest stable version: https://www.bok.net/Bento4/binaries/Bento4-SDK-1-6-0-639.x86_64-unknown-linux.zip. There is also a source code snapshot link.
Download the famous blender video for the demostration: https://download.blender.org/demo/movies/BBB/bbb_sunflower_1080p_30fps_normal.mp4
Download and unpack the binary Bento4-SDK-1-6-0-639.x86_64-unknown-linux.zip.
Keep on reading!

Create MacVTap device using NetworkManager nmcli under CentOS 8

In continuation of NetworkManager management with nmcli, here is a quick Linux console tip for users like CentOS 8 (or all distributions, which use the NetworkManager for managing the networking). How to create a virtualized bridge device MacVTap device with the NetworkManager nmcli command utility, which will preserve all the configuration over reboots.

nmcli connection add type macvlan dev enp0s3 mode bridge tap yes ifname macvtap0 con-name macvtap0 ip4 0.0.0.0/24

The line above creates a virtualized bridged interface and a connection with the name macvtap0. The MAcVTap device with the name macvtap0 is in bridge mode with the physical network interface enp0s3 with manual IP setting. If the IP is not included a DHCP option will be used as default.

There is one big limitation – there is no link between the enp0s3 and macvtap0. When used macvtap0 could receive packets from the network through the enp0s3, but there is no direct link between the two network devices. In simple words, when used in a virtualized environment in a virtual machine the virtual machine may have access to the network shared with the enp0s3, but the virtual machine cannot communicate with the IPs of the enp0s3!

Typically, this is used to make both the guest and the host show up directly on the switch that the host is connected to.

Linux Virtualization, https://virt.kernelnewbies.org/MacVTap

Initial state, only one connection in NetworkManager.

The main server connection with name enp0s3 using the same name network interface enp0s3:

[root@srv ~]# nmcli con
NAME    UUID                                  TYPE      DEVICE 
enp0s3  09497bbf-da59-42b7-a72c-d69369760b36  ethernet  enp0s3
[root@srv ~]# nmcli 
enp0s3: connected to enp0s3
        "Intel 82540EM"
        ethernet (e1000), 08:00:27:03:C9:2E, hw, mtu 1500
        ip4 default
        inet4 192.168.0.20/24
        route4 192.168.0.0/24 metric 100
        route4 0.0.0.0/0 via 192.168.0.1 metric 100
        inet6 fe80::a00:27ff:fe03:c92e/64
        route6 fe80::/64 metric 100

lo: unmanaged
        "lo"
        loopback (unknown), 00:00:00:00:00:00, sw, mtu 65536

DNS configuration:
        servers: 8.8.8.8 1.1.1.1
        interface: enp0s3

Use "nmcli device show" to get complete information about known devices and
"nmcli connection show" to get an overview on active connection profiles.

Consult nmcli(1) and nmcli-examples(7) manual pages for complete usage details.

Add the MacVTap device with the name macvlan0

[root@srv ~]# nmcli connection add type macvlan dev enp0s3 mode bridge tap yes ifname macvtap0 con-name macvtap0 ip4 0.0.0.0/24
Connection 'macvtap0' (7a5ef04c-ea98-4642-ac5d-4239f715f631) successfully added.

A MacVTap device, a network connection, and a link are established. The name of the MacVTap device and the network connection is macvtap0.

Keep on reading!

Gentoo ERROR: Python module pytevent of version 0.10.2 not found, and bundling disabled

Emerging the sys-libs/ldb-2.3.0-r1 package may fail with an error for a missing Python mode, despite the sys-libs/tevent with a python USE flag is presented in the system:

Checking for system tevent (>=0.10.2)                                                           : yes 
ERROR: Python module pytevent of version 0.10.2 not found, and bundling disabled
 * ERROR: sys-libs/ldb-2.3.0-r1::gentoo failed (configure phase):
 *   configure failed
 * 
 * Call stack:

Indeed, the tevent (>=0.10.2) is found, but not the Python module! And the checking pase of the setup fails.
First, check whether the USE of sys-libs/tevent has python and the right version PYTHON_SINGLE_TARGET=”python3_8″ is used (the Python version may vary here):

root@srv # emerge -pv =tevent-0.10.2::gentoo

[ebuild   R    ] sys-libs/tevent-0.10.2::gentoo  USE="python" ABI_X86="32 (64) (-x32)" PYTHON_SINGLE_TARGET="python3_8 -python3_9" 0 KiB

Total: 1 package (1 reinstall), Size of downloads: 0 KiB

This system uses Python 3.8 and the library sys-libs/tevent was built with this USE flag.

The problem here is the tevent is installed under a its own directory: /usr/lib64/tevent. Using the ldconig utility the problem quickly has been resolved. Just add a file /etc/ld.so.conf.d/tevent_ldb.conf with the path to the library and then regenerate the ldconfig:

root@srv # cat /etc/ld.so.conf.d/tevent_ldb.conf
/usr/lib64/tevent
root@srv # ldconfig
root@srv #

Do not forget to run the “ldconfig“, because tevent library won’t be added to the LD cache. Emerging the sys-libs/ldb and then samba was successful after this quick workaround! There is a Gentoo bug reported, but the problem still exists – https://bugs.gentoo.org/590026

Gentoo emerge virtualbox- Mesa / GLU: Mesa not found at, Mesa headers not found

Emerging the package app-emulation/virtualbox the following error occurs:

Checking for Mesa / GLU: 
  Mesa not found at -L/usr/X11R6/lib -L/usr/X11R6/lib64 -L/usr/local/lib -lXext -lX11 -lGL -I/usr/local/include or Mesa headers not found
  Check the file /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/configure.log for detailed error information.
Check /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/configure.log for details
 * ERROR: app-emulation/virtualbox-6.1.18::gentoo failed (configure phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line  125:  Called src_configure
 *   environment, line 5504:  Called doecho './configure' '--with-gcc=x86_64-pc-linux-gnu-gcc' '--with-g++=x86_64-pc-linux-gnu-g++' '--disable-dbus' '--disable-kmods' '--disable-alsa' '--disable-docs' '--disable-devmapper' '--disable-pulse' '--disable-python' '--enable-webservice' '--enable-vnc'
 *   environment, line 1538:  Called die
 * The specific snippet of code:
 *       "$@" || die

The configure script reports the mesa is missing, but the package media-libs/mesa is installed. Reinstalling does not fix the problem.
Farther investigation in the logs by checking the configure.log reveals the real problem:

srv ~ # tail -n 16 /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/configure.log
***** Checking Mesa / GLU *****
compiling the following source file:
#include <cstdio>
#include <X11/Xlib.h>
#include <GL/glx.h>
#include <GL/glu.h>
extern "C" int main(void)
{
  return 0;
}
using the following command line:
x86_64-pc-linux-gnu-g++  -fPIC -g -O -Wall -o /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/.tmp_out /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/.tmp_src.cc "-L/usr/X11R6/lib -L/usr/X11R6/lib64 -L/usr/local/lib -lXext -lX11 -lGL -I/usr/local/include"
/var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/.tmp_src.cc:4:10: fatal error: GL/glu.h: No such file or directory
    4 | #include <GL/glu.h>
      |          ^~~~~~~~~~
compilation terminated.

The glu part of mesa is missing. In Gentoo, the glu (https://gitlab.freedesktop.org/mesa/glu) is not included in the media-libs/mesa and it is a separate package media-libs/glu.

The solution is to emerge media-libs/glu and then the app-emulation/virtualbox.

emerge -v media-libs/glu

Another Linux distribution may include glu in the main mesa package.

Here, the conclusion is to always check the configure.log, because it reports the exact error and not to trust the generic output of the configure script.

Simple export of a ext4 directory with NFS Ganesha 3.5 server in CentOS 8 with SELinux enforcing

In fact, this article is a continuation of the previous NFS Ganesha article – Simple export of an ext4 directory with NFS Ganesha 3.5 server in CentOS 8 without SELinux because it has the same purpose to export a directory residing on an ext4 file system under CentOS 8 Stream, but this time the SELinux is enabled and it is in enforcing mode! There is a need for this additional article because the SELinux is not enabled in many user configurations (despite being wrong!) and the SELinux configuration may add complexity to the first article, which could lead to misleading thoughts. The previous article might be a little bit more detailed, so the reader could check it, too.
It’s worth mentioning the key points of NFS-Ganesha:

a user-mode file sharing server
supports NFS 3, 4.x and 9P
using plugins for different file systems
CentOS Storage Special Interest Group offers a file repository with NFS-Ganesha server
supports file systems like ext4, xfs, brtfs, zfs and more. There are sample configurations: https://github.com/phdeniel/nfs-ganesha/tree/master/src/config_samples
supports cluster and/or distributed file systems like GlusterFS, Ceph, GPFS, HPSS, Lustre
Current version 3.5 and it is included in the official SIG CentOS Storage Special Interest Group repository.

This article assumes the reader has a clean CentOS 8 Stream installation with SELinux in enforcing mode.

STEP 1) Install the repository and NFS-Ganesha software

NFS-Ganesha 3 packages are from the CentOS Storage SIG repository, which is a good repository and may be trusted.

dnf install -y centos-release-nfs-ganesha30
dnf install -y nfs-ganesha nfs-ganesha-vfs nfs-ganesha-selinux

STEP 2) Configuration for exporting a directory.

There are two files under /etc/ganesha/:

ganesha.conf
vfs.conf

ganesha.conf includes global configuration and NFS share configuration. Each export path begins with the keyword EXPORT followed by a block ebraced by brackets {}.
vfs.conf includes a simple example for the VFS plugin, but this configuration file is not used by the NFS Ganesha server. It is just a sample file.
Here is a simple configuration, which exports /mnt/storage with Read/Write permissions to a single IP. Just add at the end of the file /etc/ganesha/ganesha.conf contains:

 
EXPORT
{
        Export_Id = 2;
        Path = /mnt/storage1;
        Pseudo = /mnt/storage1;
        Protocols = 3,4;
        Access_Type = RW;
        Squash = None;
        FSAL
        {
                Name = VFS;
        }
        CLIENT
        {
                Clients = 192.168.0.12;
        }
}

STEP 3) Start the server and mount the exported directory. Configure the firewall.

Start the server, enable the service to start on boot and then configure the firewall to pass the NFS requests:

systemctl start nfs-ganesha
systemctl enable nfs-ganesha
firewall-cmd --permanent --zone=public --add-service=nfs
firewall-cmd --reload

Keep on reading!

Simple export of a ext4 directory with NFS Ganesha 3.5 server in CentOS 8 without SELinux

NFS Ganesha is a user-mode file sharing server, which supports NFS 3 and 4.x versions and 9P. NFS Ganesha has several interesting plugins that support exporting files from the cluster and distributed file systems like Ceph and Glusterfs Exporting a file system with NFS Ganesha is simple enough if you do not use SELinux or SELinux is in permissive mode!
This article is to show how to export a server’s directory using NFS protocol Just to note the NFS-Ganesha is tested and supports ext2/ext3/ext4, xfs, brtfs, zfs file systems as of version 3.5 (check the manual for xfs, brtfs and zfs exports – here are sample configurations for them https://github.com/phdeniel/nfs-ganesha/tree/master/src/config_samples). To be able to export a file directory the VFS Ganesha plugin is used. A clean install of minimal CentOS 8 Stream is used so the installation log may differ significantly from the user’s log but the user will see all the dependencies, which are required for this setup.

STEP 1) Install the repository and NFS-Ganesha software

NFS-Ganesha 3 packages are from the CentOS Storage SIG repository, which is a good repository and may be trusted.

dnf install -y centos-release-nfs-ganesha30
dnf install -y nfs-ganesha nfs-ganesha-vfs

STEP 2) Configuration for exporting a directory.

There are two files under /etc/ganesha/:

ganesha.conf
vfs.conf

 
EXPORT
{
        Export_Id = 2;
        Path = /mnt/storage1;
        Pseudo = /mnt/storage1;
        Protocols = 3,4;
        Access_Type = RW;
        Squash = None;
        FSAL
        {
                Name = VFS;
        }
        CLIENT
        {
                Clients = 192.168.0.12;
        }
}

STEP 3) Start the server and mount the exported directory. Configure the firewall.

Start the server, enable the service to start on boot and then configure the firewall to pass the NFS requests:

systemctl start nfs-ganesha
systemctl enable nfs-ganesha
firewall-cmd --permanent --zone=public --add-service=nfs
firewall-cmd --reload

Keep on reading!

Repairing damaged backup GPT with gdisk

Problem with network shared storage could lead to a damaged file system or even GPT tables, so the gdisk may help in this case.
Here it is a nasty the error:

[root@srv1 ~]# kpartx /dev/loop0
Alternate GPT is invalid, using primary GPT.
loop0p1 : 0 83881984 /dev/loop0 2048
[root@srv1 ~]# parted /dev/loop0
GNU Parted 3.1
Using /dev/loop0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used.
OK/Cancel? OK                                                             
Model: Loopback device (loopback)
Disk /dev/loop0: 42.9GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  42.9GB  42.9GB               primary

(parted)

So parted reports the backup GPT is damaged, but how to fix it? The solution is to use gdisk and use write “w” command in it. gdisk also shows the exact error with the GPT table with “v” option:

[root@srv1 ~]# gdisk 
GPT fdisk (gdisk) version 0.8.10

Type device filename, or press <Enter> to exit: /dev/loop0
Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************

Command (? for help): p
Disk /dev/loop0: 83886080 sectors, 40.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 7EDF123B-FBC4-4C09-B636-922BD165F862
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 83886046
Partitions will be aligned on 2048-sector boundaries
Total free space is 4029 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048        83884031   40.0 GiB    0700  primary

Command (? for help): v

Caution: The CRC for the backup partition table is invalid. This table may
be corrupt. This program will automatically create a new backup partition
table when you save your partitions.

Identified 1 problems!

Command (? for help): p
Disk /dev/loop0: 83886080 sectors, 40.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 7EDF123B-FBC4-4C09-B636-922BD165F862
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 83886046
Partitions will be aligned on 2048-sector boundaries
Total free space is 4029 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048        83884031   40.0 GiB    0700  primary

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/loop0.

Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.

And the GPT backup in this loop device is fixed. Executing parted again reports no problems:

[root@srv1 ~]# parted /dev/loop0 print
Model: Loopback device (loopback)
Disk /dev/loop0: 42.9GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  42.9GB  42.9GB               primary

Verify also reports nor error. More options are available:

[root@srv1 ~]# gdisk /dev/loop0
GPT fdisk (gdisk) version 0.8.10

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): h
b  back up GPT data to a file
c  change a partition's name
d  delete a partition
i  show detailed information on a partition
l  list known partition types
n  add a new partition
o  create a new empty GUID partition table (GPT)
p  print the partition table
q  quit without saving changes
r  recovery and transformation options (experts only)
s  sort partitions
t  change a partition's type code
v  verify disk
w  write table to disk and exit
x  extra functionality (experts only)
?  print this menu

Command (? for help): v

No problems found. 4029 free sectors (2.0 MiB) available in 2
segments, the largest of which is 2015 (1007.5 KiB) in size.

Command (? for help): q

CentOS 8.2 (RHEL 8.2) and above including CentOS Stream 9

STEP 1) All installed packages should be updated to the latest versions.

STEP 2) A cleanup of all packages, which are not required anymore.

STEP 1) Find the bad files with the find command.

STEP 2) Copy the healthy portion of the file.

STEP 1) Download and install bento4 software.

Initial state, only one connection in NetworkManager.

Add the MacVTap device with the name macvlan0

STEP 1) Install the repository and NFS-Ganesha software

STEP 2) Configuration for exporting a directory.

STEP 3) Start the server and mount the exported directory. Configure the firewall.

STEP 1) Install the repository and NFS-Ganesha software

STEP 2) Configuration for exporting a directory.

STEP 3) Start the server and mount the exported directory. Configure the firewall.