Run LXC Ubuntu 22.04 LTS container with bridged network under CentOS Stream 9

In continuation of the previous article Run LXC CentOS Stream 9 container with bridged network under CentOS Stream 9, this time the LXC container will be Ubuntu 22.04 LTS Jammy Jellyfish.
To receive a better understanding why to use LXC or a much detailed information of some steps in this article it is better to visit the previously mention article and the original Run LXC CentOS 8 container with bridged network under CentOS 8.

STEP 1) Install the needed software EPEL repository and the LXC and its dependencies

To install LXC software the EPEL CentOS Stream 9 repository must be installed. At present, the LXC included in CentOS Stream 9 EPEL repository is 4.0.

dnf install -y epel-release
dnf install -y lxc lxc-templates container-selinux
dnf install -y wget tar

lxc-templates uses template “download” to download different Linux distribution images from http://images.linuxcontainers.org/, which now redirects to http://uk.lxd.images.canonical.com/ (an Ubuntu lxd images mirror).
The container-selinux should be installed only if the host, i.e. the CentOS Stream 9 install, is with enabled SELinux. The packages offers additional SELinux rules or for the LXC and LXC tools like lxc-attach and more.

STEP 2) Create a Ubuntu 22.04 LTS with the help of LXC templates

[root@srv ~]# lxc-create --template download -n mycontainer -- --dist centos --release 9-Stream --arch amd64

In addition, there is a “–variant” option along with “--dist” and “--release” to specify which variant to install – default, cloud, desktop or other. There is a variant column in the table on the images’ page mentioned above.
Keep on reading!

Run LXC CentOS Stream 9 container with bridged network under CentOS Stream 9

In continue of the previous article with CentOS 8 – Run LXC CentOS 8 container with bridged network under CentOS 8, here is an updated version with CentOS Stream 9 running LXC container. In this case, the LXC container is CentOS Stream 9, too.
Under CentOS 8, the LXC software is from branch 3.x, but in CentOS Stream 9 the LXC is 4.x and there are some differences in the LXC configuration file.
It’s worth mentioning the differences between docker/podman containers and LXC from the previous article:

  • Multiprocesses.
  • Easy configuration modification. Even hot-plugin supported.
  • Unprivileged Linux containers.
  • Complex network setups. Multiple network interfaces connected to different networks, for example.
  • Live systemd, i.e. systemd or SysV init are booted as usual. Much of the software relies on systemd/udev features and in many cases, it is really hard to run a software without a systemd or init process

Here are the steps to boot a CentOS Stream 9 container under CentOS Stream 9 host server:

STEP 1) Install EPEL repository.

EPEL CentOS Stream 9 repository now includes LXC 4.0 software.

dnf install -y epel-release

STEP 2) Install LXC software and start LXC service.

At present, the LXC software version is 4.0.12. The package lxc-templates includes template scripts to create a Linux distribution environment like CentOS, Ubuntu, Debian, Gentoo, ArchLinux, Oracle, Alpine, and many others and it also includes the configuration templates to start these Linux distributions. In fact, lxc-templates now includes a download script to download images from the Internet.

dnf install -y lxc lxc-templates container-selinux
dnf install -y wget tar

The wget and tar are required if LXC templates installation is going to be performed.
There is an additional package for container’s SELinux, which should be installed before starting the LXC service, because some of the SELinux rules may not apply in the system. If the SELinux is disabled the installation of container-selinux package might be skipped.

STEP 3) Create a CentOS Stream 9 container with the help of LXC templates and run it.

Use the lxc-templates to prepare a CentOS Stream 9 container environment. The currently available containers are listed here http://images.linuxcontainers.org/, which now redirects to http://uk.lxd.images.canonical.com/ (an Ubuntu lxd images mirror). Check out the URL and choose the right container. Here the CentOS Stream 9 amd64, i.e. release 9-Stream, is used.

[root@srv ~]# lxc-create --template download -n mycontainer -- --dist centos --release 9-Stream --arch amd64

In addition, there is a “–variant” option along with “--dist” and “--release” to specify which variant to install – default, cloud, desktop or other. There is a variant column in the table on the images’ page mentioned above.
Keep on reading!

Install CentOS Stream 9 Workstation (Gnome GUI)

This is the latest CentOS version with a graphical interface Gnome for a workstation. If you are a developer or just a Linux user, which want to have a pretty stable Operating System, the CentOS Stream 9 may be an option for you. CentOS Stream 9 is based and follows the RedHat Enterprise Linux 9 – paid Linux for the enterprise world, which is available for free thanks to the Open-source Software. CentOS Stream 9 will receive all the updates from the paid Linux system RHEL 9 and the support will be 10 years from now. 5 years of full support and 5 more years of security updates. The use of CentOS Stream 9 assures the user to have a stable and secure Linux operating system, which will not bring fundamental changes and breaks things periodically as of a more enthusiastic Linux Distributions. More information for the system here – Software and technical details of CentOS Stream 9 minimal install and Software comparison Ubuntu server 22.04 LTS vs CentOS Stream 9 head-to-head
The CentOS Stream 9 has a generic installation wizard for multiple type of installations – server, server with gui, user workstation and so on. This article is to show what options to enable to install a user workstation with CentOS Stream 9 with a graphical interface – Gnome. Most of the people think CentOS as a server Linux Distribution, but in fact, it is ideal for a workstation, too, especially with the grade of stability and security these days.
This article uses network installation with the following media: http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/iso/CentOS-Stream-9-latest-x86_64-boot.iso which always points to the latest release of the CentOS Stream 9. The network installation will choose automatically the best mirror to download the packages for the system. There is an option to use an off-line installation with an 8G ISO disk. Check out for more ISOs here – http://mirror.stream.centos.org/9-stream/BaseOS/x86_64/iso/

SCREENSHOT 1) Boot from the UEFI DVD-ROM device.

It is the same with the USB bootable removable drive. Choose the UEFI USB drive and boot the installation live drive.

main menu
UEFI BIOS DVD-ROM boot

Keep on reading!

Software comparison Ubuntu server 22.04 LTS vs CentOS Stream 9 head-to-head

The following article compares two different and very popular for servers Linux distros with Ubuntu Server 22.04 LTS (Jammy) and CentOS Stream 9. These are the latest versions released in the past month. Bear in mind, that they have different release cycle, but the these are the system with log support up to 2032! The two Linux distors Ubuntu Server 22.04 LTS (Jammy) and CentOS Stream 9 support is till 2032! Here is the links to their life cycle and what kind of support could be expected in the time frame of 10 years:

In the Desktop world upgrading to the new and latest version of a Linux distribution is almost mandatory, but in the server world, upgrading is more complicated. The first step in updating a server is to check what software versions come with the new distribution version and then check whether the running custom (application) software supports the software versions. For example, updating to a new distribution version, which comes with PHP 8.1, but the current application supports only 7.4 is not very wise and in addition, the current version may have years of support in the future.

Lately, with the virtualization and container software and it is more common to choose a long-living Linux distribution for the host and more short-living with bleeding edge technologies for the guest environments. So check out the Fedora 36 Server Edition articles in the site – Minimal network installation of Fedora 36 Server, Software and technical details of Fedora Server 36 including cockpit screenshots and the comparison software table between Ubuntu 22.04 LTS and Fedora 36 Server EditionSoftware comparison Ubuntu server 22.04 LTS vs Fedora 36 server edition – head-to-head.

Software comparison table – Ubuntu 22.06 LTS vs CentOS Stream 9 (20220606.0) as of June 2022:

SoftwareUbuntu 22.04CentOS Stream 9
Support10 years (April 2032)10 years (31.03.2032)
Linux kernel5.15.05.14.0
libc2.352.34
OpenSSL.
3.0.2
1.1.1k
3.0.1
GNU GCC9.4.0
10.3.0
11.2.0
12-20220302
.
.
11.3.1
.
PHP8.18.0.13
Python2.7.18
3.10.4
.
3.9.10
Perl5.34.05.32.1
Ruby3.03.0.3
OpenJDK8u312-b07
11.0.15
17.0.3
18~36ea-1
1.8.0.332.b09
11.0.15.0.10
17.0.3.0.7
.
Go lang1.13.8
1.17.3
1.18.1
.
1.17.5
.
Rust1.58.11.61.0
llvm11.1.0
12.0.1
13.0.1
14.0.0
.
.
.
14.0.0
nodejs12.22.9
.
.
16.14.0
Subversion1.14.11.14.1
Git2.34.12.31.1
Apache2.4.522.4.53
Nginx1.18.01.20.1
MySQL server8.0.298.0.28
MariaDB10.6.710.5.13
PostgreSQL14.213.7
SQLite2.8.17
3.37.2
.
3.34.1
Xorg X server1.22.1.11.20.11
Gnome Shell42.040.10
qemu6.27.0.0
docker|podman20.10.124.1.0
lxc5.0.0.

For more detailed overview the two systems check out the following articles – Software and technical overview of Ubuntu server 22.04 LTS and Software and technical details of CentOS Stream 9 minimal install.

Copy of the Life cycle plans of Ubuntu server 22.04 and CentOS Stream 9 (Red Hat Enterprise Linux 9) of the official sources only for time-freeze feature:

Ubuntu 22.04 LTS
CentOS Stream 9 (RHEL 9)

rsync server under CentOS 8 with SELinux enabled

Here is a quick and useful tip on how to run a rsync daemon under CentOS 8 with SELinux in Enforcing mode.
There are three basic steps:

  1. rsync daemon installation and configuration.
  2. firewall configuration.
  3. SELinux configuration.

STEP 1) rsync daemon installation and configuration.

Under CentOS 8 rsync daemon files are in a separate rpm package rsync-daemon (more on the subject rsync daemon in CentOS 8):

[root@srv ~]# dnf install -y rsync-daemon
Last metadata expiration check: 2:45:48 ago on Thu Apr  7 07:40:42 2022.
Dependencies resolved.
==============================================================================================================
 Package                     Architecture          Version                        Repository             Size
==============================================================================================================
Installing:
 rsync-daemon                noarch                3.1.3-14.el8                   baseos                 43 k

Transaction Summary
==============================================================================================================
Install  1 Package

Total download size: 43 k
Installed size: 17 k
Downloading Packages:
rsync-daemon-3.1.3-14.el8.noarch.rpm                                           98 kB/s |  43 kB     00:00    
--------------------------------------------------------------------------------------------------------------
Total                                                                          81 kB/s |  43 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                      1/1 
  Installing       : rsync-daemon-3.1.3-14.el8.noarch                                                     1/1 
  Running scriptlet: rsync-daemon-3.1.3-14.el8.noarch                                                     1/1 
  Verifying        : rsync-daemon-3.1.3-14.el8.noarch                                                     1/1 

Installed:
  rsync-daemon-3.1.3-14.el8.noarch                                                                            

Complete!

Keep on reading!

Replace current interface configuration with a bridge device using nmcli (NetworkManager)

This article shows how the primary network interface could be replaced by a bridge device and the network interface becomes a part of the bridge as a slave device without reboot or restart of the server. Using nmcli under CentOS 8 (and probably any other Linux distribution like Ubuntu, which uses NetworkManager to configure network devices).
The main steps are:

  1. Create a connection profile of a bridge device.
  2. Set the same network configuration as the primary network to the bridge device.
  3. Create a connection profile for the primary interface device as a slave network device to the newly created bridge.
  4. Delete the current primary connection, which is using the primary network device and configuration.
  5. Reload the bridge connection profile to take effect. The bridge device will actually begin to work.

The main goal is not to reboot the server or lose the connection to the server. The primary network interface is the only connection on the server and losing it the server is going to be unreachable. So the last two steps should be performed in the background or a script or a detached terminal (like screen).
Here are all the commands in one place:

nmcli connection add type bridge ifname br0 con-name br0 ipv4.method manual ipv4.addresses "192.168.0.20/24" ipv4.gateway "192.168.0.1" ipv4.dns "8.8.8.8 1.1.1.1"
nmcli con add type bridge-slave ifname enp0s3 master br0
nmcli con del "enp0s3"; nmcli con reload "br0" &

Here is the detailed information for the above commands:
Keep on reading!

libelf was not found in the pkg-config search path

Building from source under CentOS the user may stumble on some compilation errors and most of them are for missing -devel packages. Here is such example with not so easy to find the name of a missing library:

[/tmp/netdata-libbpf-El77Ld/libbpf-0.0.9_netdata-1/src]# env CFLAGS=-fPIC CXXFLAGS= LDFLAGS= BUILD_STATIC_ONLY=y OBJDIR=build DESTDIR=.. make install 
Package libelf was not found in the pkg-config search path.
Perhaps you should add the directory containing `libelf.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libelf' found
mkdir -p build/staticobjs
cc -I. -I../include -I../include/uapi -DCOMPAT_NEED_REALLOCARRAY -fPIC -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c bpf.c -o build/staticobjs/bpf.o
cc -I. -I../include -I../include/uapi -DCOMPAT_NEED_REALLOCARRAY -fPIC -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c btf.c -o build/staticobjs/btf.o
btf.c:17:18: fatal error: gelf.h: No such file or directory
 #include <gelf.h>
                  ^
compilation terminated.
make: *** [build/staticobjs/btf.o] Error 1
 FAILED   

The missing development library file is with the name: elfutils-libelf-devel. Installing the package with yum or dnf will resolve the above error:

yum install -y elfutils-libelf-devel

Or for CentOS 8 and newer Fedora versions:

dnf install -y elfutils-libelf-devel

Configure Bond (802.3ad LACP) device in CentOS 8 – configuration files

Upgrading to a bond device is a common step when the server exhausts its current network port bandwidth.
The hardware setup of the bond example here is:

  • two 10G network cards – ens1f0 and ens1f0
  • bond name – bond0
  • bond mode – 802.3ad – Link Aggregation Control Protocol (LACP)

The systemd reconfiguration procedure consists of:

  • Stop the network target
    systemctl stop network
    
  • Set several configuration files – network device files for the network interfaces, bonding interface – master and slave devices.
  • Start the network target
    systemctl start network
    

*Note: the 802.3ad bonding mode needs aditional configuration in the switch of which the server is connected.

The example here is using CentOS 8 configuration file to make a permanent (i.e. persistent over reboots using the CentOS 8 network configuration files) bonding configuration.
Check out the official bonding documentation for all modes and options – https://www.kernel.org/doc/Documentation/networking/bonding.txt.

CONF 1) Configure the network interfaces.

The interface should be in down state in the configuration file.
Interface 1 – /etc/sysconfig/network-scripts/ifcfg-ens1f0:

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens1f0
UUID=3b399a23-570e-45ed-9369-4ff5b87efb2c
DEVICE=ens1f0
ONBOOT=no

Interface 2 – /etc/sysconfig/network-scripts/ifcfg-ens1f1:

TYPE=Ethernet
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=dhcp
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME=ens1f1
UUID=ecdc5d5b-9739-4424-9d67-362411974281
DEVICE=ens1f1
ONBOOT=no

CONF 2) Configure bonding master device – create a bonding group bond0

This device should be started up at boot.
Bonding device 1 – with name bond0 – /etc/sysconfig/network-scripts/ifcfg-Bond_connection_1:

BONDING_OPTS="downdelay=200 miimon=100 mode=802.3ad updelay=200"
TYPE=Bond
BONDING_MASTER=yes
PROXY_METHOD=none
BROWSER_ONLY=no
BOOTPROTO=none
IPADDR=10.10.10.10
PREFIX=24
GATEWAY=10.10.10.1
DNS1=10.10.10.2
DNS2=10.10.10.3
DEFROUTE=yes
IPV4_FAILURE_FATAL=no
IPV6INIT=yes
IPV6_AUTOCONF=yes
IPV6_DEFROUTE=yes
IPV6_FAILURE_FATAL=no
IPV6_PRIVACY=no
IPV6_ADDR_GEN_MODE=stable-privacy
NAME="Bond connection 1"
UUID=f0a35f9a-20e4-484e-850c-689128642555
DEVICE=bond0
ONBOOT=yes

BONDING_OPTS are specific options for the bonding group with name bond0 and the bonding mode is set here, too.

CONF 3) Configure bonding slave devices – the two network cards

Adding the two network cards to the bonding group bond0. These devices should be started up at boot.
Interface 1 – /etc/sysconfig/network-scripts/ifcfg-bond0_slave_1:

HWADDR=90:E2:BA:8A:13:8C
TYPE=Ethernet
NAME="bond0 slave 1"
UUID=c49e0ced-6411-41fa-9a3b-a01a430664a7
DEVICE=ens1f0
ONBOOT=yes
MASTER=bond0
SLAVE=yes

Interface 2 – /etc/sysconfig/network-scripts/ifcfg-bond0_slave_2:

HWADDR=90:E2:BA:8A:13:8D
TYPE=Ethernet
NAME="bond0 slave 2"
UUID=90de1cad-1d9f-48cb-8e5a-7d8bfdde91d2
DEVICE=ens1f1
ONBOOT=yes
MASTER=bond0
SLAVE=yes

CentOS 8 dracut-initqueue timeout and could not boot – warning /dev/disk/by-id/md-uuid- does not exist – inactive raids

Booting the CentOS 8 failed with

dracut-initqueue timeout and could not boot – warning /dev/disk/by-id/md-uuid- does not exist

we have an article on the subject for CentOS 7 – CentOS 7 dracut-initqueue timeout and could not boot – warning /dev/disk/by-id/md-uuid- does not exist and we continue with another issue with the same error.
Most times when you get this error there is a mistake in some UUID for the file system or the RAID devices. But this time our software raid was autodetected with the right disks but it ended in INACTIVE STATE. Software raid in INACTIVE STATE cannot be used so we are in “Emergency mode”:

SCREENSHOT 1) Without root partition the boot process enters the dracut emergency mode.

main menu
Entering emergency mode

SCREENSHOT 2) Software md raid device information – missing “Personalities” for the raid groups.

Loaded modules in the kernel and missing raid kernel modules.

main menu
Missing raid1 kernel module in initram file

To summarize it up:

  • The disks are detected, so we drivers for SATA/SAS controller is loaded correctly.
  • The software raid autodetected the MD devices, but they are in “INACTIVE STATE”. The RAID “Personalities” is missing.

Keep on reading!

pycurl.h: fatal error: openssl/ssl.h: No such file or directory

If you encounter this error trying to install a pip module or compile a program under the console you surely miss OpenSSL development packages!
pip also may build a packages in your system and it could depend on generic library headers like in this case OpenSSL, which the installer (pip) won’t bring them and it will output an error as you can see

myuser@srv # sudo pip install pycurl pygeoip psutil
Collecting pycurl
  Using cached https://files.pythonhosted.org/packages/e8/e4/0dbb8735407189f00b33d84122b9be52c790c7c3b25286826f4e1bdb7bde/pycurl-7.43.0.2.tar.gz
Requirement already satisfied (use --upgrade to upgrade): pygeoip in /usr/local/lib/python2.7/dist-packages
Requirement already satisfied (use --upgrade to upgrade): psutil in /usr/lib/python2.7/dist-packages
Building wheels for collected packages: pycurl
  Running setup.py bdist_wheel for pycurl ... error
  Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AbCshS/pycurl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" bdist_wheel -d /tmp/tmpqVNq1upip-wheel- --python-tag cp27:
  Using curl-config (libcurl 7.47.0)
  running bdist_wheel
  running build
  running build_py
  creating build
  creating build/lib.linux-x86_64-2.7
  creating build/lib.linux-x86_64-2.7/curl
  copying python/curl/__init__.py -> build/lib.linux-x86_64-2.7/curl
  running build_ext
  building 'pycurl' extension
  creating build/temp.linux-x86_64-2.7
  creating build/temp.linux-x86_64-2.7/src
  x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPYCURL_VERSION="7.43.0.2" -DHAVE_CURL_SSL=1 -DHAVE_CURL_OPENSSL=1 -DHAVE_CURL_SSL=1 -I/usr/include/python2.7 -c src/docstrings.c -o build/temp.linux-x86_64-2.7/src/docstrings.o
  In file included from src/docstrings.c:4:0:
  src/pycurl.h:164:28: fatal error: openssl/ssl.h: No such file or directory
  compilation terminated.
  error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
  
  ----------------------------------------
  Failed building wheel for pycurl
  Running setup.py clean for pycurl
Failed to build pycurl
Installing collected packages: pycurl
  Running setup.py install for pycurl ... error
    Complete output from command /usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AbCshS/pycurl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-oea_jq-record/install-record.txt --single-version-externally-managed --compile:
    Using curl-config (libcurl 7.47.0)
    running install
    running build
    running build_py
    creating build
    creating build/lib.linux-x86_64-2.7
    creating build/lib.linux-x86_64-2.7/curl
    copying python/curl/__init__.py -> build/lib.linux-x86_64-2.7/curl
    running build_ext
    building 'pycurl' extension
    creating build/temp.linux-x86_64-2.7
    creating build/temp.linux-x86_64-2.7/src
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fno-strict-aliasing -Wdate-time -D_FORTIFY_SOURCE=2 -g -fstack-protector-strong -Wformat -Werror=format-security -fPIC -DPYCURL_VERSION="7.43.0.2" -DHAVE_CURL_SSL=1 -DHAVE_CURL_OPENSSL=1 -DHAVE_CURL_SSL=1 -I/usr/include/python2.7 -c src/docstrings.c -o build/temp.linux-x86_64-2.7/src/docstrings.o
    In file included from src/docstrings.c:4:0:
    src/pycurl.h:164:28: fatal error: openssl/ssl.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1
    
    ----------------------------------------
Command "/usr/bin/python -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-AbCshS/pycurl/setup.py';exec(compile(getattr(tokenize, 'open', open)(__file__).read().replace('\r\n', '\n'), __file__, 'exec'))" install --record /tmp/pip-oea_jq-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-AbCshS/pycurl/
You are using pip version 8.1.1, however version 18.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command

Keep on reading!