collectd nginx plugin: curl_easy_perform failed because of selinux

Enabling the Nginx plugin for collectd under CentOS (or any other system using SELinux) might be confusing for a newbie. Most sources on the Internet would just install collectd-nginx:

yum install -y collectd-nginx

and configure it in the nginx.conf and collectd.conf. Still, the statistics might not work as expected, the collectd may not be able to gather statistics from the Nginx.

SELinux may prevent collectd (plugin) daemon to connect to Nginx and gather statistics from the Nginx stats page.

Checking the collectd log and it reports a problem:
Keep on reading!

Dracut boot failed with missing device – exit and continue normal booting!

This issue deserves a much more article, in fact, a straightforward tip:

You may be able to continue a normal boot only by typing “exit” and hitting enter in the “Dracut” console.

Most of the time this Dracut console entering is caused because the system administrator of the server/machine added, replaced or deleted a RAID or similar device and forgot to update the configuration (grub2 probably). And in most of these cases, the raid is not critical for machine normal boot from the root partition, but it may be critical for the services lately. Booting in normal mode, even without some devices, is the main goal because under the normal mode it easier to repair the system.
Check out the two articles on the topic (especially the first one):

SCREENSHOT 1) Just type “exit” and hit enter.

It’s worth noting that if you executed some commands in the console and/or mounted devices to test they are with healthy file system or for whatever reason you did it, the boot process may not continue after typeing exit and probablly a reboot is required. The server will go once more in this mode and then just typing will work.

main menu
type exit

Keep on reading!

podman – Error adding network: failed to allocate for range 0: 10.88.0.46 has been allocated after server reboot

We’ve just stumbled on the following error with one of our podman CentOS 8 servers after restart:

[root@srv ~]# podman start mysql-slave
ERRO[0000] Error adding network: failed to allocate for range 0: 10.88.0.46 has been allocated to c97823be46832ddebbce29f3f51e3091620188710cb7ace246e173a7a981baed, duplicate allocation is not allowed 
ERRO[0000] Error while adding pod to CNI network "podman": failed to allocate for range 0: 10.88.0.46 has been allocated to c97823be46832ddebbce29f3f51e3091620188710cb7ace246e173a7a981baed, duplicate allocation is not allowed 
Error: unable to start container "mysql-slave": error configuring network namespace for container c97823be46832ddebbce29f3f51e3091620188710cb7ace246e173a7a981baed: failed to allocate for range 0: 10.88.0.46 has been allocated to c97823be46832ddebbce29f3f51e3091620188710cb7ace246e173a7a981baed, duplicate allocation is not allowed

Apparently, something got wrong, because the two containers were fine before restarting and they were multiple times stopped, started and restarted.

The solution is to remove IP-named files in /var/lib/cni/networks/podman and start the podman containers again.

It resembles to a bug https://github.com/containers/libpod/issues/3759, which should have already been closed by the new minor CentOS 8 releases.

The interesting part is that the container we are trying to start mysql-slave has c97823be46832ddebbce29f3f51e3091620188710cb7ace246e173a7a981baed, but it reports it cannot allocate it, because it has already been allocated to a container with the same ID. That’s the problem:

The IP-named files in /var/lib/cni/networks/podman were not removed when the podman container had stopped.

Typically, when a podman container executes a stop command, the process should remove the files in /var/lib/cni/networks/podman. Before restarting the CentOS 8 server you may need to stop the podman containers for now.

[root@srv ~]# cd /var/lib/cni/networks/podman
[root@srv podman]# ls -altr
total 24
-rwxr-x---. 1 root root    0  3 Dec  0,43 lock
drwxr-xr-x. 3 root root 4096  3 Dec  0,43 ..
-rw-r--r--. 1 root root   64  9 Dec 18,34 10.88.0.46
-rw-r--r--. 1 root root   64 16 Dec 12,01 10.88.0.47
-rw-r--r--. 1 root root   10  1 Mar  9,28 last_reserved_ip.0
-rw-r--r--. 1 root root   70  1 Mar  9,28 10.88.0.49
drwxr-xr-x. 2 root root 4096  1 Mar  9,28 .
[root@srv podman]# rm 10.88.0.46
rm: remove regular file '10.88.0.46'? y
[root@srv podman]# rm 10.88.0.47
rm: remove regular file '10.88.0.47'? y
[root@srv podman]# podman start mysql-slave
mysql-slave
[root@srv podman]# podman ps
CONTAINER ID  IMAGE                           COMMAND               CREATED       STATUS            PORTS  NAMES
c97823be4683  localhost/centos-mysql-5.6:0.9  /entrypoint.sh my...  2 months ago  Up 2 minutes ago         mysql-slave
e96134b31894  docker.io/example/client:latest   start-boinc.sh        2 months ago  Up 6 minutes ago         example-client
[root@srv podman]# ls -altr
общо 20
-rwxr-x---. 1 root root    0  3 Dec  0,43 lock
drwxr-xr-x. 3 root root 4096  3 Dec  0,43 ..
-rw-r--r--. 1 root root   70  1 Mar  9,28 10.88.0.49
-rw-r--r--. 1 root root   10  1 Mar  9,32 last_reserved_ip.0
-rw-r--r--. 1 root root   70  1 Mar  9,32 10.88.0.50
drwxr-xr-x. 2 root root 4096  1 Mar  9,32 .
[root@srv podman]#

We’ve deleted the old IPs (old by date!) 10.88.0.46 and 10.88.0.47 and the mysql-slave container started successfully.

firewalld and podman (or docker) – no internet in the container and could not resolve host

If you happen to use CentOS 8 you have already discovered that Red Hat (i.e. CentOS) switch to podman, which is a fork of docker. So probably the following fix might help to someone, which does not use CentOS 8 or podman. For now, podman and docker are 99.99% the same.
So creating and starting a container is easy and in most cases one command only, but you may stumble on the error your container could not resolve or could not connect to an IP even there is a ping to the IP!
The service in the container may live a happy life without Internet access but just the mapped ports from the outside world. Still, it may happen to need Internet access, let’s say if an update should be performed.
Here is how to fix podman (docker) missing the Internet access in the container:

  • No ping to the outside world. The chances you are missing
    sysctl -w net.ipv4.ip_forward=1
    

    And do not forget to make it permanent by adding the “net.ipv4.ip_forward=1” to /etc/sysctl.conf (or a file “.conf” in /etc/sysctl.d/).

  • ping to the outside IP of the container is available, but no connection to any service is available! Probably the NAT is not enabled in your podman docker configuration. In the case with firewalld, at least, you must enable the masquerade option of the public zone
    firewall-cmd --zone=public --add-masquerade
    firewall-cmd --permanent --zone=public --add-masquerade
    

    The second command with “–permanent” is to make the option permanent over reboots.

The error – Could not resolve host (Name or service not known) despite having servers in /etc/resolv.conf and ping to them!

One may think having IPs in /etc/resolv.conf and ping to them in the container should give the container access to the Internet. But the following error occurs:

[root@srv /]# yum install telnet
Loaded plugins: fastestmirror, ovl
Determining fastest mirrors
 * base: artfiles.org
 * extras: centos.mirror.net-d-sign.de
 * updates: centos.bio.lmu.de
http://mirror.fra10.de.leaseweb.net/centos/7.7.1908/os/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: mirror.fra10.de.leaseweb.net; Unknown error"
Trying other mirror.
http://artfiles.org/centos.org/7.7.1908/os/x86_64/repodata/repomd.xml: [Errno 14] curl#6 - "Could not resolve host: artfiles.org; Unknown error"
Trying other mirror.
^C

Exiting on user cancel
[root@srv /]# ^C
[root@srv /]# ping 8.8.8.8
PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=56 time=5.05 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=56 time=5.06 ms
^C
--- 8.8.8.8 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1000ms
rtt min/avg/max/mdev = 5.050/5.055/5.061/0.071 ms
[root@srv ~]# cat /etc/resolv.conf 
nameserver 8.8.8.8
nameserver 8.8.4.4
[root@srv /]# ping google.com
ping: google.com: Name or service not known

The error 2 – Can’t connect to despite having ping to the IP!

[root@srv /]# ping 2.2.2.2
PING 2.2.2.2 (2.2.2.2) 56(84) bytes of data.
64 bytes from 2.2.2.2: icmp_seq=1 ttl=56 time=9.15 ms
64 bytes from 2.2.2.2: icmp_seq=2 ttl=56 time=9.16 ms
^C
[root@srv2 /]# mysql -h2.2.2.2 -uroot -p
Enter password: 
ERROR 2003 (HY000): Can't connect to MySQL server on '2.2.2.2' (113)
[root@srv2 /]#

Despite having ping the MySQL server on 2.2.2.2 and despite the firewall on 2.2.2.2 allows outside connections the container could not connect to it. And testing other services like HTTP, HTTPS, FTP and so on resulted in “unable to connect“, too. Simply because the NAT (aka masquerade is not enabled in the firewall).

rsync daemon in CentOS 8

Apparently, the team behind the CentOS 8 decided to split the rsync functionality to two packages – one for the client-side and for the server-side, despite the binary rsync is only one and offers the client-size and server-side.
So there two packages in CentOS 8:

  1. rsync – provides the client-side and server-side as usual
  2. rsync-daemon – provides configuration example file and the systemd to start it as a service.

So if you wonder where is your rsync service after installing the rsync package under CentOS 8 you must install additional package “rsync-daemon”.

Of course, you may just create anywhere “rsyncd.conf” (the best place for the configuration file is in /etc, but could be placed anywhere with “–config=PATH/FILE” option) file and start the daemon as usual with “–daemon” option included to have the rsync server-side up and running.

rsync --daemon --config=/etc/rsyncd.conf

Just create yourself the configuration “/etc/rsyncd.conf” file.

Install the rsync program – the client and the daemon

dnf install rsync

Install the configuration and systemd files

dnf install rsync-daemon

rsync and rsync-daemon files

The files included in the two packages:

[root@srv ~]# dnf repoquery -l rsync-daemon
Last metadata expiration check: 0:33:02 ago on Wed 22 Jan 2020 02:57:06 PM UTC.
/etc/rsyncd.conf
/etc/sysconfig/rsyncd
/usr/lib/systemd/system/rsyncd.service
/usr/lib/systemd/system/rsyncd.socket
/usr/lib/systemd/system/rsyncd@.service
/usr/share/man/man5/rsyncd.conf.5.gz
[root@srv ~]# dnf repoquery -l rsync
Last metadata expiration check: 0:33:06 ago on Wed 22 Jan 2020 02:57:06 PM UTC.
/usr/bin/rsync
/usr/lib/.build-id
/usr/lib/.build-id/6e
/usr/lib/.build-id/6e/bdbf5032a966a6293cea003621ba766ab36ff1
/usr/share/doc/rsync
/usr/share/doc/rsync/NEWS
/usr/share/doc/rsync/OLDNEWS
/usr/share/doc/rsync/README
/usr/share/doc/rsync/support
/usr/share/doc/rsync/support/Makefile
/usr/share/doc/rsync/support/atomic-rsync
/usr/share/doc/rsync/support/cvs2includes
/usr/share/doc/rsync/support/deny-rsync
/usr/share/doc/rsync/support/file-attr-restore
/usr/share/doc/rsync/support/files-to-excludes
/usr/share/doc/rsync/support/git-set-file-times
/usr/share/doc/rsync/support/instant-rsyncd
/usr/share/doc/rsync/support/logfilter
/usr/share/doc/rsync/support/lsh
/usr/share/doc/rsync/support/lsh.sh
/usr/share/doc/rsync/support/mapfrom
/usr/share/doc/rsync/support/mapto
/usr/share/doc/rsync/support/mnt-excl
/usr/share/doc/rsync/support/munge-symlinks
/usr/share/doc/rsync/support/rrsync
/usr/share/doc/rsync/support/rsync-no-vanished
/usr/share/doc/rsync/support/rsync-slash-strip
/usr/share/doc/rsync/support/rsyncstats
/usr/share/doc/rsync/support/savetransfer.c
/usr/share/doc/rsync/tech_report.tex
/usr/share/licenses/rsync
/usr/share/licenses/rsync/COPYING
/usr/share/man/man1/rsync.1.gz

Install CentOS 8 over the old OS and preserve the storage partitions

Always put your root partition separate from the storage (aka data) partitions. root partition should be only for system files and nothing more! Keeping this simple rule you may easily change your operating system (or clean install or clean upgrade) without deleting the user’s data thus preserving the old storage partitions.
Our storage has 2 storage partitions, which means they hosts only data and no system files and there are separate partitions for Linux booting (grub2) and system files (root partition). Here is the partitions layout:

[root@srv0 ~]# parted /dev/sda --script print
Model: AVAGO SMC3108 (scsi)
Disk /dev/sda: 48.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  53.7GB  53.7GB                     raid
 2      53.7GB  54.8GB  1075MB                     raid
 3      54.8GB  55.0GB  211MB   fat16              raid
 4      55.0GB  69.4GB  14.4GB                     raid
 5      71.8GB  48.0TB  47.9TB  ext4

[root@srv0 ~]# parted /dev/sdb --script print
Model: AVAGO SMC3108 (scsi)
Disk /dev/sdb: 48.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name  Flags
 1      1049kB  53.7GB  53.7GB                     raid
 2      53.7GB  54.8GB  1075MB                     raid
 3      54.8GB  55.0GB  211MB   fat16              raid
 4      55.0GB  69.4GB  14.4GB                     raid
 5      71.8GB  48.0TB  47.9TB  ext4
[root@srv0 ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs         16G     0   16G   0% /dev
tmpfs            16G     0   16G   0% /dev/shm
tmpfs            16G  250M   16G   2% /run
tmpfs            16G     0   16G   0% /sys/fs/cgroup
/dev/md127       50G  1.7G   45G   4% /
/dev/md125      991M  133M  792M  15% /boot
/dev/md124      201M   12M  190M   6% /boot/efi
/dev/sda5        44T   23T   22T  52% /mnt/storage1
/dev/sdb5        44T   14T   30T  32% /mnt/storage2
tmpfs           3.2G     0  3.2G   0% /run/user/0

Of course, when there are partitions above 2T the GPT is mandatory.
You can skip the software RAID1 setup if you use only one controller or you have system partitions only in one disk (virtual drive and so on). Here we have two hardware controllers, which we want to use both for the system partitions.
4 RAID1 devices:

  1. EFI partition (/boot/efi)
  2. swap partition
  3. boot parition (/boot)
  4. root partition (/)

The best practice is have total between 30G and 50G for the 4 partitions (in fact, boot partition could be skipped). Have in mind most modern Linux distributions cannot be installed on less than 10G~20G and for optimal results just separate between 30G and 50G for 4 partitions above (or 3 if you choose to skip the boot one).

Upgrade to CentOS 8 with clean install over our old CentOS 7 system partitions preserving the big data partitions.

Couple of things before start:

  • UEFI installation will be selected. So boot in UEFI mode.
  • IPMI KVM is used to install the new Linux distribution – CentOS 8
  • The installation disk is mounted in the Virtual CD/DVD IPMI KVM device – with Mount and boot ISO file from windows share in Supermicro IPMI Virtual media (CD-ROM)
  • All system parititions will be removed (grub, boot, root) and a clean minimal installation will be performed.
  • Network installation – using CentOS-8-x86_64-1905-boot.iso

SCREENSHOT 1) The Server is starting. This is the IPMI KVM window. Press F11 to Boot in Boot Menu.

main menu
SUPERMICRO Server starting – KVM

Keep on reading!

Adding bonding interface to CentOS 8 – editing configuration files only

This article shows what files to add if you want to add a bonding interface under CentOS 8 without invoking the Network manager command utility.
Our goal is to use one boding group with the name bond0 in LACP (aka 802.3ad) mode (but it could be any of the other types) with two networks 10Gbps interfaces. The setup resented here uses NetworkManager, which handles the loading of bonding module properly.

In fact, the network-scripts are now deprecated and they are missing from the system (but they still exist in the additional package – “network-scripts”, who knows till when? do not rely on them!).

The configuration files are with the same syntax as under CentOS 7, but this time the network manager parses them. The ifup and ifdown still exist and they just call the Network manager when executed (unless the “network-scripts” package is installed). If you need to enable bonding without any configuration files (for emergency situations) you may still use – How to enable Linux bonding without ifenslave

What do you need:

  • Ensure you have installed: “iputils” and “NetworkManager” packages
    dnf install -y NetworkManager iputils
    
  • Ensure the NetworkManager service is running
    systemctl enable NetworkManager
    systemctl start NetworkManager
    

STEP 1) Configure the bonding device

The boding interface’s name will be bond0 and the configuration will be located in /etc/sysconfig/network-scripts/ifcfg-bond0

BONDING_OPTS="mode=4 miimon=100"
TYPE=Bond
BONDING_MASTER=yes
BOOTPROTO=none
IPADDR0=192.168.0.100
PREFIX0=24
GATEWAY0=192.168.0.1
DNS1=8.8.8.8
DNS2=8.8.4.4
IPV4_FAILURE_FATAL=no
NAME=bond0
UUID=e19e2059-2e31-4143-915a-cdc11d19c9d6
DEVICE=bond0
ONBOOT=yes

Keep on reading!

CentOS 8 dracut-initqueue timeout and could not boot – warning /dev/disk/by-id/md-uuid- does not exist – inactive raids

Booting the CentOS 8 failed with

dracut-initqueue timeout and could not boot – warning /dev/disk/by-id/md-uuid- does not exist

we have an article on the subject for CentOS 7 – CentOS 7 dracut-initqueue timeout and could not boot – warning /dev/disk/by-id/md-uuid- does not exist and we continue with another issue with the same error.
Most times when you get this error there is a mistake in some UUID for the file system or the RAID devices. But this time our software raid was autodetected with the right disks but it ended in INACTIVE STATE. Software raid in INACTIVE STATE cannot be used so we are in “Emergency mode”:

SCREENSHOT 1) Without root partition the boot process enters the dracut emergency mode.

main menu
Entering emergency mode

SCREENSHOT 2) Software md raid device information – missing “Personalities” for the raid groups.

Loaded modules in the kernel and missing raid kernel modules.

main menu
Missing raid1 kernel module in initram file

To summarize it up:

  • The disks are detected, so we drivers for SATA/SAS controller is loaded correctly.
  • The software raid autodetected the MD devices, but they are in “INACTIVE STATE”. The RAID “Personalities” is missing.

Keep on reading!

Technical details of a default CentOS 8.0.1950 minimal installation

This article is for those of you who do not want to install a whole new operating system only to discover some technical details about the default installation like disk layout, packages included, software versions, and so on. Here we are going to review in several sections what is like to have a default installation of CentOS 8.0.
Despite the kernel is 4.18 it detects successfully the new RYZEN/Threadripper AMD and the system is stable (we booted in UEFI mode).

Software

The CentOS 8.0 (8.0.1950) you can have

  • linux kernel – 4.18.0 (4.18.0-80.7.1.el8_0.x86_64)
  • Graphical User Interface
    • Xorg X server – 1.20.3
    • GNOME (the GUI) – 3.28.2
    • K Desktop Environment – NO, it’s depricated and not included in the release.
  • System
    • linux-firmware – version: 20190111, release: 92.gitd9fb2ee6.el8.
    • QT – 5.11.1
    • libc – 2.28
    • GNU GCC – 8.2.1
    • OpenSSL – 1.1.1
    • coreutils – 8.30
    • yum – Depricated and replaced with dnf
    • dnf – 4.0.9.2
    • cups – 2.2.6
    • rsyslog – 8.37.0
  • Servers
    • Apache – 2.4.37
    • Nginx – 1.14.1
    • MySQL server – 8.0.13
    • MariaDB server – 10.3.11
    • PostgreSQL – 10.6
  • Programming
    • PHP – 7.2.11
    • python – 2.7.15 and also includes 3.6.8
    • perl – 5.26.3
    • ruby – 2.5.3
    • OpenJDK – 11.0.4.11 and also includes 1.8.0.222.b10
    • Go – 1.11.5
    • Rust – 1.31.0
    • Subversion – 1.10.2
    • Git – 2.18.1

Note: Not all of the above software comes installed by default.
The installation procedure you can find here – How to do a network installation of CentOS 8 (8.0.1950) – minimal server installation Minimal install with “Standard” Addons selected.
Installed packages are 547 occupying 1.8G space:.

[root@srv ~]# dnf list installed|wc -l
547
[root@srv ~]# df -h /
Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/cl_srv-root   50G  1.8G   49G   4% /

Note: if you do not select “Standard Addon” the installation occupies less space – 1.6Gbytes and installs only 397 packages.
Keep on reading!

How to do a network installation of CentOS 8 (8.0.1950) – minimal server installation

Minimal net install is useful when a dedicated server is installed from a IPMI KVM or Dell iDRAC, HP iLO, IBM IMM or where the initial client side download of files need to be minimal.
For amd64 CentOS 8 BaseOS the net install bootable media is located here (now the current latest release is 8.0.1950, but you can check the last directory with 8. for the time you follow this howto):

http://mirror.leaseweb.com/centos/8/isos/x86_64/CentOS-8-x86_64-1905-boot.iso

Note there is no minimal CD for offline installation anymore. Boot CD is to just boot and make “netinstall” installation and a big fat DVD of 6.6Gbytes to install offline.
Software details of CentOS 8 minimal install could be found here – Technical details of a default CentOS 8.0.1950 minimal installation

Download it and put it on a cd or usb, the boot from it and follow the steps bellow:

STEP 1) Enter the BIOS of your machine (desktop or server) to change the boot order.

You can try F11 on most servers to have the boot menu. Here we use DEL to enter the UEFI BIOS.

main menu
Enter BIOS – change boot order

Keep on reading!