Category: software

Unable to continue upgrading an old cacti 0.8.8 to the latest version 1.2.18

Upgrading an old instance of cacti monitoring software may become a challenge, because of multiple new recommendations and requirements for the latest version 1.2.x.
There are a couple of b recommendations like memory limit and maximum execution time and multiple plugin requirements, which if not fulfilled the setup cannot continue. Second, there are the MySQL recommendations and there is an option innodb_file_format, which in general, is recommended to be Barracuda, but by default, in older version of MySQL use Antelope!

Upgrading from CACTI version 0.8.8 is successful to CACTI version 1.2.3, but then the upgrade process just began restarting and failed to upgrade to the final target CACTI version 1.2.18 because of the old MySQL InnoDB table format – Antelope.

Despite the Barracuda is just recommended and the upgrade process continues through the steps of the setup wizard, it just suddenly stops and returns to the welcome install screen.
Setting the option innodb_file_format resolves the problem and the upgrade setup finishes successfully the upgrade from CACTI version 0.8.8 (apparently with an intermediate upgrade to CACTI version 1.2.3) to CACTI version 1.2.18.

innodb_file_format=barracuda

Probably, this option will be a mandatory MySQL option for upgrading to a newer CACTI version after 1.2.3.

Several screenshots of recommendations and requirements for upgrading to CACTI 1.2.28

Keep on reading!

Transfer only a list of files with rsync

Transferring a list of files from one server to another maybe not so easy as it looks like, if the list consists of files with symlinks in the paths. Consider the following example:

/mnt/storage1/dir1/subdir1/file1
/mnt/storage1/dir1/subdir2/file2
/mnt/storage1/dir1/subdir3/file3

But what if the subdir3 is a symlink to a sub-directory:

/mnt/storage1/dir1/subdir3 -> /mnt/storage2/dir1/subdir3

STEP 1) Generate a list of only files following the symlinks if they exist.

The best way is to use Linux command find:

find -L /mnt/storage1/ -type f &> find.files.log

The option “-L” instructs the find to follow symbolic links and the test for the file (-type f) will always match against the type of the file that the symbolic link points to and not the symlink itself.

STEP 2) rsync the list of the files

There are several options (–copy-links –copy-dirlinks), which must be used with rsync command to be able to transfer the list of files, which may include symlink directories in the files’ paths:

rsync --copy-links --copy-dirlinks --partial --files-from=/root/find.files.log --verbose --progress --stats --times --perms --owner --group 10.10.10.10::storage /

The command above uses the rsync daemon started on the source server with shared root under the name “root” in the rsync configuration. Here is a sample rsync daemon configuration (/etc/rsyncd.conf):

pid file = /run/rsyncd.pid
use chroot = yes
read only = yes
hosts allow = 10.10.10.10/24
hosts deny = *

[root]
        uid=0
        gid=0
        path = /
        comment = root partition
        exclude = /proc /sys

Of course, a relative path could be used in the rsync daemon configuration if the file generated is also a relative path. For simplicity, here the whole real path is used.

In addition, the rsync could be used without a rsync daemon, but in conjunction with ssh daemon:

rsync --rsh='ssh -p 22 -l root' --copy-links --copy-dirlinks --partial --files-from=/root/find.files.log --verbose --progress --stats --times --perms --owner --group 10.10.10.10:/ /

Again, because the files in the find.files.log are with absolute paths the root / must be used in the rsync command. Relative paths may be used, too.

alter table error 1215 (HY000): Cannot add foreign key constraint

Adding a foreign key may result in the following error:

mysql> ALTER TABLE `table1` ADD CONSTRAINT FK_table2_id FOREIGN KEY (`table2_id`) REFERENCES table2(`ID`);
ERROR 1215 (HY000): Cannot add foreign key constraint

This error can occur because in couple of cases and it is not very informative, but basically, it says there is some kind of compatibility issue between the table1.table_id and table2.ID. And the three most common problems, besides the tables and columns exists, are:

  1. One of the two table are not Innodb. 
    SHOW CREATE TABLE `table1`;
  2. There is a value (or values) in table1.table_id, which does not exists in table2.ID.
    3. SELECT `table1`.`table2_id`, `table2`.`ID` FROM `table1` LEFT JOIN `table2` ON `table1`.`table2_id`=`table2`.`ID`;

    Execute a left join to check whether there are NULL values in the table1.table2_id. If NULL values exist a record with the same missing IDs should be inserted in table2.ID. this kind of check may be stopped temporarily with:

    SET FOREIGN_KEY_CHECKS=0;

    Change it back to 1 after the ALTER clause.

  3. The type and the attributes of the columns are different. The attributes such as UNSINGED may cause a problem, too! 
    ALTER TABLE `table1` CHANGE `table2_id` `table2_id` INT(11) UNSIGNED NOT NULL;

    In this case, the column types are the same, but the attributes are different and the MySQL server throws the error! Change the signed int to unsigned with the above command. And after the table1.table2_id and table2.ID are of the same type and they have the same attributes, the alter command will be executed successfully.

Here are the initial structure of above tables:

CREATE TABLE `table1` (
  `ID` int(11) NOT NULL AUTO_INCREMENT,
  `table2_id` int(11) NOT NULL,
  PRIMARY KEY (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=220 DEFAULT CHARSET=utf8
CREATE TABLE `table2` (
  `ID` int(10) unsigned NOT NULL AUTO_INCREMENT,
  `col1` int(11) NOT NULL DEFAULT '0',
  PRIMARY KEY (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=642 DEFAULT CHARSET=utf8

And after a successful ALTER statement, the table1 will looks like:

ALTER TABLE `table1` ADD CONSTRAINT `FK_table2_id` FOREIGN KEY (`table2_id`) REFERENCES `table1`(`ID`);
Query OK, 216 rows affected (0.92 sec)
Records: 216  Duplicates: 0  Warnings: 0

CREATE TABLE `table1` (
  `ID` int(11) NOT NULL AUTO_INCREMENT,
  `table2_id` int(11) NOT NULL,
  PRIMARY KEY (`ID`),
  KEY `FK_table2_id` (`table1_id`),
  CONSTRAINT `FK_table2_id` FOREIGN KEY (`table2_id`) REFERENCES `table2` (`ID`)
) ENGINE=InnoDB AUTO_INCREMENT=220 DEFAULT CHARSET=utf8

Create and export a GlusterFS volume with NFS-Ganesha in CentOS 8

GlusterFS built-in NFS server supports only NFS version 3. GlusterFS offers NFS exports using NFS-Ganesha, which supports NFS version 3 and 4 protocols.
NFS-Ganesha server is a user-mode file sharing server, which offers a GlusterFS plugin to export GlusterFS volumes. In the following article, the NSF-Ganesha and GlusterFS are installed and a simple GlusterFS volume is created and then exported through NFS 3 and 4 version protocols.
The version of the software in this article:

  • CentOS Stream release 8 (25.04.2021)
  • GlusterFS 8.4
  • NFS-Ganesha 3.5

STEP 1) Install GlusterFS.

dnf install -y centos-release-gluster
dnf install -y glusterfs-server

The first line will installs a new repository under the SIG management – https://wiki.centos.org/SpecialInterestGroup/Storage. The second line installs the GlusterFS server.

STEP 2) Install NFS-Ganesha.

dnf install -y centos-release-nfs-ganesha30
dnf install -y nfs-ganesha nfs-ganesha-gluster

The first line again installs a new repository under the SIG management and the second line installs the NFS-Ganesha server with Gluster plugin.

STEP 3) Create GlusterFS volume

Start the GlusterFS server and create a simple 3 replicas volume with:
Start the GlusterFS on all the three nodes and enable the GlusterFS communication between the three nodes using firewall-cmd utility. So execute the following commands:

systemctl start glusterd
firewall-cmd --permanent --new-zone=glusternodes
firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.200
firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.201
firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.202
firewall-cmd --permanent --zone=glusternodes --add-service=glusterfs
firewall-cmd --reload

On the first node create the GlusterFS volume. First, add the glnode2 and glnode3 to the cluster.

gluster peer probe glnode2
gluster peer probe glnode3
gluster volume create VOL1 replica 3 transport tcp glnode1:/mnt/storage/gluster/brick glnode2:/mnt/storage/gluster/brick glnode3:/mnt/storage/gluster/brick
gluster volume start VOL1

Keep on reading!

glusterfs with localhost (127.0.0.1) nodes on different servers – glusterfs volume with 3 replicas

Binding the GlusterFS nodes on a physical interface may lead to local availability problems even for replication nodes. Bringing down the physical interface will bring down the nodes, even the local replica for the local mounts and applications.

  • 100% local uptime. The local replica will be 100% available. The loopback interface is always in the upstate! Network interfaces are not 100% upstate, because of network reload or cable unplug. Cable unplug (or port down for a switch configuration reload) could lead to a short time unavailable of the local node even for the local system!
  • node resolve rely on /etc/hosts records, not network and remote DNS system.
  • No speed limit. The read from the local system through the loopback interface could easily increase above 1G or even 10G. Probably, building nodes with replicas over a 1G network is much more affected than a network with 10G connectivity. Reading from a node relying on a loopback interface could pass 10G, even though the server is connected to a 1G network!

An addition note – another kind of the proposed solution here is to use a virtual interface to bind the IP of the GlusterFS brick. The most common type of virtual interface is using a bridge interface for the IP.

The example here is to bring a GlusterFS volume in replication mode with 3 servers, i.e. 3 replicas. Each server may mount locally the GlusterFS volume with name VOL1 and it would not get unavailable if the main interface

  • server’s hostname: node1, hostname for the replica brick glnode1. IP: 192.168.0.20, but the node1 locally is resolved as 127.0.0.1 through /etc/hosts.
  • server’s hostname: node2, hostname for the replica brick glnode2. IP: 192.168.0.30, but the node1 locally is resolved as 127.0.0.1 through /etc/hosts.
  • server’s hostname: node3, hostname for the replica brick glnode3 IP: 192.168.0.31, but the node3 locally is resolved as 127.0.0.1 through /etc/hosts.

Of course, the server’s hostname could be used, but it better to have a separate domain for the GlusterFS bricks. Sometimes server hostnames should be a real IP or some software may rely on it, too.

And here are all the commands to bring up the GlusterFS volume on 3 servers:

STEP 1) Install GlusterFS software and initial configuration.

There are GlisterFS packages in the official CentOS 8, but a newer version is supported in the Storage SIG. The GlusterFS version installed in this article is 8.4. Install the software under node1, node2, and node3.

yum install -y centos-release-gluster
yum install -y glusterfs-server

Add the following lines at the end of node1:/etc/hosts file.

127.0.0.1 glnode1
192.168.0.30 glnode2
192.168.0.31 glnode3

Add the following lines at the end of node2:/etc/hosts file.

192.168.0.20 glnode1
127.0.0.1 glnode2
192.168.0.31 glnode3

Add the following lines at the end of node3:/etc/hosts file.

192.168.0.20 glnode1   
192.168.0.30 glnode2
127.0.0.1 glnode3

Start the GlusterFS service on the three nodes

systemctl start glusterd

Mount the storage device if any and make the directory where the GlusterFS brick will reside:

mount /mnt/storage/
mkdir -p /mnt/storage/gluster/brick

STEP 2) Configure the firewall.

CentOS 8 uses firewalld and here a new zone for the GlusterFS is created and the GlusterFS service is added in the whitelist of the new zone. The three IPs of the nodes are also added in the new zone:

firewall-cmd --permanent --new-zone=glusternodes
firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.20
firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.30
firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.31
firewall-cmd --permanent --zone=glusternodes --add-service=glusterfs
firewall-cmd --reload

STEP 3) Add peers to the GlusterFS cluster and create a 3 node replica volume.

gluster peer probe glnode2
gluster peer probe glnode3
gluster volume create VOL1 replica 3 transport tcp glnode1:/mnt/storage/gluster/brick glnode2:/mnt/storage/gluster/brick glnode3:/mnt/storage/gluster/brick
gluster volume start VOL1

The GlusterFS volume can be mounted with:

mount -t glusterfs glnode1:/VOL1 /mnt/VOL1/

And /etc/fstab sample line:

glnode1:/VOL1 /mnt/VOL1 glusterfs defaults,noatime,direct-io-mode=disable 0 0

Always use the local hostname for the current node server. If you would like to mount the volume VOL1 on node1, use glnode1:/VOL1 and so on.
Bringing down the physical interface of the server, which is connected to the Internet (aka the network interface with the real IP) would not make the GlusterFS brick unavailable for the local mounts and applications.

In a cluster with only replicas, the local application will just continue using the mounted GlusterFS volume (or native GlusterFS clients) relying only on the local Gluster brick till the main Internet connection comes back.

Create 3 node replica volume – the whole output

[root@node1 ~]# yum install -y centos-release-gluster
CentOS Stream 8 - AppStream                                                                                 5.8 MB/s | 6.7 MB     00:01    
CentOS Stream 8 - BaseOS                                                                                    2.0 MB/s | 2.3 MB     00:01    
CentOS Stream 8 - Extras                                                                                     22 kB/s | 9.1 kB     00:00    
Dependencies resolved.
============================================================================================================================================
 Package                                          Architecture              Version                         Repository                 Size
============================================================================================================================================
Installing:
 centos-release-gluster8                          noarch                    1.0-1.el8                       extras                    9.3 k
Installing dependencies:
 centos-release-storage-common                    noarch                    2-2.el8                         extras                    9.4 k

Transaction Summary
============================================================================================================================================
Install  2 Packages

Total download size: 19 k
Installed size: 2.4 k
Downloading Packages:
(1/2): centos-release-gluster8-1.0-1.el8.noarch.rpm                                                         136 kB/s | 9.3 kB     00:00    
(2/2): centos-release-storage-common-2-2.el8.noarch.rpm                                                     145 kB/s | 9.4 kB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                        27 kB/s |  19 kB     00:00     
warning: /var/cache/dnf/extras-9705a089504ff150/packages/centos-release-gluster8-1.0-1.el8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
CentOS Stream 8 - Extras                                                                                    725 kB/s | 1.6 kB     00:00    
Importing GPG key 0x8483C65D:
 Userid     : "CentOS (CentOS Official Signing Key) <security@centos.org>"
 Fingerprint: 99DB 70FA E1D7 CE22 7FB6 4882 05B5 55B3 8483 C65D
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                    1/1 
  Installing       : centos-release-storage-common-2-2.el8.noarch                                                                       1/2 
  Installing       : centos-release-gluster8-1.0-1.el8.noarch                                                                           2/2 
  Running scriptlet: centos-release-gluster8-1.0-1.el8.noarch                                                                           2/2 
  Verifying        : centos-release-gluster8-1.0-1.el8.noarch                                                                           1/2 
  Verifying        : centos-release-storage-common-2-2.el8.noarch                                                                       2/2 

Installed:
  centos-release-gluster8-1.0-1.el8.noarch                           centos-release-storage-common-2-2.el8.noarch                          

Complete!
[root@node1 ~]# yum install -y glusterfs-server
Last metadata expiration check: 0:00:14 ago on Wed Apr 14 13:01:50 2021.
Dependencies resolved.
============================================================================================================================================
 Package                                      Architecture          Version                            Repository                      Size
============================================================================================================================================
Installing:
 glusterfs-server                             x86_64                8.4-1.el8                          centos-gluster8                1.4 M
Installing dependencies:
 attr                                         x86_64                2.4.48-3.el8                       baseos                          68 k
 device-mapper-event                          x86_64                8:1.02.175-5.el8                   baseos                         269 k
 device-mapper-event-libs                     x86_64                8:1.02.175-5.el8                   baseos                         269 k
 device-mapper-persistent-data                x86_64                0.8.5-4.el8                        baseos                         468 k
 glusterfs                                    x86_64                8.4-1.el8                          centos-gluster8                689 k
 glusterfs-cli                                x86_64                8.4-1.el8                          centos-gluster8                214 k
 glusterfs-client-xlators                     x86_64                8.4-1.el8                          centos-gluster8                899 k
 glusterfs-fuse                               x86_64                8.4-1.el8                          centos-gluster8                171 k
 libaio                                       x86_64                0.3.112-1.el8                      baseos                          33 k
 libgfapi0                                    x86_64                8.4-1.el8                          centos-gluster8                125 k
 libgfchangelog0                              x86_64                8.4-1.el8                          centos-gluster8                 67 k
 libgfrpc0                                    x86_64                8.4-1.el8                          centos-gluster8                 89 k
 libgfxdr0                                    x86_64                8.4-1.el8                          centos-gluster8                 61 k
 libglusterd0                                 x86_64                8.4-1.el8                          centos-gluster8                 45 k
 libglusterfs0                                x86_64                8.4-1.el8                          centos-gluster8                350 k
 lvm2                                         x86_64                8:2.03.11-5.el8                    baseos                         1.6 M
 lvm2-libs                                    x86_64                8:2.03.11-5.el8                    baseos                         1.1 M
 psmisc                                       x86_64                23.1-5.el8                         baseos                         151 k
 python3-pyxattr                              x86_64                0.5.3-18.el8                       centos-gluster8                 35 k
 rpcbind                                      x86_64                1.2.5-8.el8                        baseos                          70 k
 userspace-rcu                                x86_64                0.10.1-4.el8                       baseos                         101 k

Transaction Summary
============================================================================================================================================
Install  22 Packages

Total download size: 8.2 M
Installed size: 24 M
Downloading Packages:
(1/22): glusterfs-cli-8.4-1.el8.x86_64.rpm                                                                  1.1 MB/s | 214 kB     00:00    
(2/22): glusterfs-8.4-1.el8.x86_64.rpm                                                                      2.6 MB/s | 689 kB     00:00    
(3/22): glusterfs-fuse-8.4-1.el8.x86_64.rpm                                                                 1.1 MB/s | 171 kB     00:00    
(4/22): glusterfs-client-xlators-8.4-1.el8.x86_64.rpm                                                       2.3 MB/s | 899 kB     00:00    
(5/22): libgfapi0-8.4-1.el8.x86_64.rpm                                                                      1.8 MB/s | 125 kB     00:00    
(6/22): libgfchangelog0-8.4-1.el8.x86_64.rpm                                                                755 kB/s |  67 kB     00:00    
(7/22): libgfrpc0-8.4-1.el8.x86_64.rpm                                                                      756 kB/s |  89 kB     00:00    
(8/22): libgfxdr0-8.4-1.el8.x86_64.rpm                                                                      579 kB/s |  61 kB     00:00    
(9/22): libglusterd0-8.4-1.el8.x86_64.rpm                                                                   641 kB/s |  45 kB     00:00    
(10/22): glusterfs-server-8.4-1.el8.x86_64.rpm                                                              3.4 MB/s | 1.4 MB     00:00    
(11/22): libglusterfs0-8.4-1.el8.x86_64.rpm                                                                 1.0 MB/s | 350 kB     00:00    
(12/22): python3-pyxattr-0.5.3-18.el8.x86_64.rpm                                                             97 kB/s |  35 kB     00:00    
(13/22): attr-2.4.48-3.el8.x86_64.rpm                                                                        85 kB/s |  68 kB     00:00    
(14/22): device-mapper-event-1.02.175-5.el8.x86_64.rpm                                                      342 kB/s | 269 kB     00:00    
(15/22): device-mapper-event-libs-1.02.175-5.el8.x86_64.rpm                                                 338 kB/s | 269 kB     00:00    
(16/22): libaio-0.3.112-1.el8.x86_64.rpm                                                                    679 kB/s |  33 kB     00:00    
(17/22): device-mapper-persistent-data-0.8.5-4.el8.x86_64.rpm                                               1.5 MB/s | 468 kB     00:00    
(18/22): psmisc-23.1-5.el8.x86_64.rpm                                                                       1.5 MB/s | 151 kB     00:00    
(19/22): rpcbind-1.2.5-8.el8.x86_64.rpm                                                                     1.2 MB/s |  70 kB     00:00    
(20/22): lvm2-libs-2.03.11-5.el8.x86_64.rpm                                                                 3.1 MB/s | 1.1 MB     00:00    
(21/22): userspace-rcu-0.10.1-4.el8.x86_64.rpm                                                              474 kB/s | 101 kB     00:00    
(22/22): lvm2-2.03.11-5.el8.x86_64.rpm                                                                      3.3 MB/s | 1.6 MB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                       2.8 MB/s | 8.2 MB     00:02     
warning: /var/cache/dnf/centos-gluster8-ae72c2c38de8ee20/packages/glusterfs-8.4-1.el8.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID e451e5b5: NOKEY
CentOS-8 - Gluster 8                                                                                        1.0 MB/s | 1.0 kB     00:00    
Importing GPG key 0xE451E5B5:
 Userid     : "CentOS Storage SIG (http://wiki.centos.org/SpecialInterestGroup/Storage) <security@centos.org>"
 Fingerprint: 7412 9C0B 173B 071A 3775 951A D4A2 E50B E451 E5B5
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                    1/1 
  Installing       : libgfxdr0-8.4-1.el8.x86_64                                                                                        1/22 
  Running scriptlet: libgfxdr0-8.4-1.el8.x86_64                                                                                        1/22 
  Installing       : libglusterfs0-8.4-1.el8.x86_64                                                                                    2/22 
  Running scriptlet: libglusterfs0-8.4-1.el8.x86_64                                                                                    2/22 
  Installing       : libgfrpc0-8.4-1.el8.x86_64                                                                                        3/22 
  Running scriptlet: libgfrpc0-8.4-1.el8.x86_64                                                                                        3/22 
  Installing       : libaio-0.3.112-1.el8.x86_64                                                                                       4/22 
  Installing       : glusterfs-client-xlators-8.4-1.el8.x86_64                                                                         5/22 
  Installing       : device-mapper-event-libs-8:1.02.175-5.el8.x86_64                                                                  6/22 
  Running scriptlet: glusterfs-8.4-1.el8.x86_64                                                                                        7/22 
  Installing       : glusterfs-8.4-1.el8.x86_64                                                                                        7/22 
  Running scriptlet: glusterfs-8.4-1.el8.x86_64                                                                                        7/22 
  Installing       : libglusterd0-8.4-1.el8.x86_64                                                                                     8/22 
  Running scriptlet: libglusterd0-8.4-1.el8.x86_64                                                                                     8/22 
  Installing       : glusterfs-cli-8.4-1.el8.x86_64                                                                                    9/22 
  Installing       : device-mapper-event-8:1.02.175-5.el8.x86_64                                                                      10/22 
  Running scriptlet: device-mapper-event-8:1.02.175-5.el8.x86_64                                                                      10/22 
  Installing       : lvm2-libs-8:2.03.11-5.el8.x86_64                                                                                 11/22 
  Installing       : libgfapi0-8.4-1.el8.x86_64                                                                                       12/22 
  Running scriptlet: libgfapi0-8.4-1.el8.x86_64                                                                                       12/22 
  Installing       : device-mapper-persistent-data-0.8.5-4.el8.x86_64                                                                 13/22 
  Installing       : lvm2-8:2.03.11-5.el8.x86_64                                                                                      14/22 
  Running scriptlet: lvm2-8:2.03.11-5.el8.x86_64                                                                                      14/22 
  Installing       : libgfchangelog0-8.4-1.el8.x86_64                                                                                 15/22 
  Running scriptlet: libgfchangelog0-8.4-1.el8.x86_64                                                                                 15/22 
  Installing       : userspace-rcu-0.10.1-4.el8.x86_64                                                                                16/22 
  Running scriptlet: userspace-rcu-0.10.1-4.el8.x86_64                                                                                16/22 
  Running scriptlet: rpcbind-1.2.5-8.el8.x86_64                                                                                       17/22 
  Installing       : rpcbind-1.2.5-8.el8.x86_64                                                                                       17/22 
  Running scriptlet: rpcbind-1.2.5-8.el8.x86_64                                                                                       17/22 
  Installing       : psmisc-23.1-5.el8.x86_64                                                                                         18/22 
  Installing       : attr-2.4.48-3.el8.x86_64                                                                                         19/22 
  Installing       : glusterfs-fuse-8.4-1.el8.x86_64                                                                                  20/22 
  Installing       : python3-pyxattr-0.5.3-18.el8.x86_64                                                                              21/22 
  Installing       : glusterfs-server-8.4-1.el8.x86_64                                                                                22/22 
  Running scriptlet: glusterfs-server-8.4-1.el8.x86_64                                                                                22/22 
  Verifying        : glusterfs-8.4-1.el8.x86_64                                                                                        1/22 
  Verifying        : glusterfs-cli-8.4-1.el8.x86_64                                                                                    2/22 
  Verifying        : glusterfs-client-xlators-8.4-1.el8.x86_64                                                                         3/22 
  Verifying        : glusterfs-fuse-8.4-1.el8.x86_64                                                                                   4/22 
  Verifying        : glusterfs-server-8.4-1.el8.x86_64                                                                                 5/22 
  Verifying        : libgfapi0-8.4-1.el8.x86_64                                                                                        6/22 
  Verifying        : libgfchangelog0-8.4-1.el8.x86_64                                                                                  7/22 
  Verifying        : libgfrpc0-8.4-1.el8.x86_64                                                                                        8/22 
  Verifying        : libgfxdr0-8.4-1.el8.x86_64                                                                                        9/22 
  Verifying        : libglusterd0-8.4-1.el8.x86_64                                                                                    10/22 
  Verifying        : libglusterfs0-8.4-1.el8.x86_64                                                                                   11/22 
  Verifying        : python3-pyxattr-0.5.3-18.el8.x86_64                                                                              12/22 
  Verifying        : attr-2.4.48-3.el8.x86_64                                                                                         13/22 
  Verifying        : device-mapper-event-8:1.02.175-5.el8.x86_64                                                                      14/22 
  Verifying        : device-mapper-event-libs-8:1.02.175-5.el8.x86_64                                                                 15/22 
  Verifying        : device-mapper-persistent-data-0.8.5-4.el8.x86_64                                                                 16/22 
  Verifying        : libaio-0.3.112-1.el8.x86_64                                                                                      17/22 
  Verifying        : lvm2-8:2.03.11-5.el8.x86_64                                                                                      18/22 
  Verifying        : lvm2-libs-8:2.03.11-5.el8.x86_64                                                                                 19/22 
  Verifying        : psmisc-23.1-5.el8.x86_64                                                                                         20/22 
  Verifying        : rpcbind-1.2.5-8.el8.x86_64                                                                                       21/22 
  Verifying        : userspace-rcu-0.10.1-4.el8.x86_64                                                                                22/22 

Installed:
  attr-2.4.48-3.el8.x86_64                                             device-mapper-event-8:1.02.175-5.el8.x86_64                         
  device-mapper-event-libs-8:1.02.175-5.el8.x86_64                     device-mapper-persistent-data-0.8.5-4.el8.x86_64                    
  glusterfs-8.4-1.el8.x86_64                                           glusterfs-cli-8.4-1.el8.x86_64                                      
  glusterfs-client-xlators-8.4-1.el8.x86_64                            glusterfs-fuse-8.4-1.el8.x86_64                                     
  glusterfs-server-8.4-1.el8.x86_64                                    libaio-0.3.112-1.el8.x86_64                                         
  libgfapi0-8.4-1.el8.x86_64                                           libgfchangelog0-8.4-1.el8.x86_64                                    
  libgfrpc0-8.4-1.el8.x86_64                                           libgfxdr0-8.4-1.el8.x86_64                                          
  libglusterd0-8.4-1.el8.x86_64                                        libglusterfs0-8.4-1.el8.x86_64                                      
  lvm2-8:2.03.11-5.el8.x86_64                                          lvm2-libs-8:2.03.11-5.el8.x86_64                                    
  psmisc-23.1-5.el8.x86_64                                             python3-pyxattr-0.5.3-18.el8.x86_64                                 
  rpcbind-1.2.5-8.el8.x86_64                                           userspace-rcu-0.10.1-4.el8.x86_64                                   

Complete!
[root@node1 ~]# systemctl start glusterd
[root@node1 ~]# systemctl status glusterd
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-04-14 13:06:39 UTC; 3s ago
     Docs: man:glusterd(8)
  Process: 10151 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCC>
 Main PID: 10152 (glusterd)
    Tasks: 9 (limit: 11409)
   Memory: 5.6M
   CGroup: /system.slice/glusterd.service
           └─10152 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO

Apr 14 13:06:39 node1 systemd[1]: Starting GlusterFS, a clustered file-system server...
Apr 14 13:06:39 node1 systemd[1]: Started GlusterFS, a clustered file-system server.
[root@node1 ~]# tail -n 3 /etc/hosts 
127.0.0.1 glnode1
192.168.0.30 glnode2
192.168.0.31 glnode3
[root@node1 ~]# ping glnode1
PING glnode1 (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.058 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.050 ms
^C
--- glnode1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1016ms
rtt min/avg/max/mdev = 0.050/0.054/0.058/0.004 ms
[root@node1 ~]# mkdir -p /mnt/storage/gluster/brick
[root@node1 ~]# firewall-cmd --permanent --new-zone=glusternodes
success
[root@node1 ~]# firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.20
success
[root@node1 ~]# firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.30
success
[root@node1 ~]# firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.31
success
[root@node1 ~]# firewall-cmd --permanent --zone=glusternodes --add-service=glusterfs
success
[root@node1 ~]# firewall-cmd --reload
success
[root@node1 ~]# firewall-cmd --zone=glusternodes --list-all
glusternodes (active)
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 192.168.0.20 192.168.0.30 192.168.0.31
  services: glusterfs
  ports: 
  protocols: 
  forward: no
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:
[root@node1 ~]# gluster peer probe glnode2
peer probe: success
[root@node1 ~]# gluster peer probe glnode3
peer probe: success
[root@node1 ~]# gluster peer status
Number of Peers: 2

Hostname: glnode2
Uuid: ab63f0a0-0a72-4fcd-9f34-b88040d1a8e3
State: Peer in Cluster (Connected)

Hostname: glnode3
Uuid: 439ccd19-a95e-427c-ab72-6b65effcbe06
State: Peer in Cluster (Connected)
[root@node1 ~]# gluster volume create VOL1 replica 3 transport tcp glnode1:/mnt/storage/gluster/brick glnode2:/mnt/storage/gluster/brick glnode3:/mnt/storage/gluster/brick
volume create: VOL1: success: please start the volume to access data
[root@node1 ~]# gluster volume start VOL1
volume start: VOL1: success
[root@node1 ~]# gluster volume info VOL1
 
Volume Name: VOL1
Type: Replicate
Volume ID: 7da7bb05-2c9b-464b-b3f9-8940eeb5b0bb
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: glnode1:/mnt/storage/gluster/brick
Brick2: glnode2:/mnt/storage/gluster/brick
Brick3: glnode3:/mnt/storage/gluster/brick
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root@node1 storage]# mkdir -p /mnt/VOL1
[root@node1 storage]# tail -n 1 /etc/fstab 
glnode1:/VOL1 /mnt/VOL1 glusterfs defaults,noatime,direct-io-mode=disable 0 0
[root@node1 storage]# mount /mnt/VOL1/
[root@node1 storage]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        892M     0  892M   0% /dev
tmpfs           909M     0  909M   0% /dev/shm
tmpfs           909M  8.5M  901M   1% /run
tmpfs           909M     0  909M   0% /sys/fs/cgroup
/dev/sda1        95G  1.5G   89G   2% /
/dev/sda3       976M  148M  762M  17% /boot
tmpfs           182M     0  182M   0% /run/user/0
glnode1:/VOL1    95G  2.4G   89G   3% /mnt/VOL1
[root@node1 storage]# ls -altr /mnt/VOL1/
total 8
drwxr-xr-x. 3 root root 4096 Apr 14 13:31 .
drwxr-xr-x. 4 root root 4096 Apr 14 13:34 ..

Simple export of a ext4 directory with NFS Ganesha 3.5 server in CentOS 8 with SELinux enforcing

In fact, this article is a continuation of the previous NFS Ganesha article – Simple export of an ext4 directory with NFS Ganesha 3.5 server in CentOS 8 without SELinux because it has the same purpose to export a directory residing on an ext4 file system under CentOS 8 Stream, but this time the SELinux is enabled and it is in enforcing mode! There is a need for this additional article because the SELinux is not enabled in many user configurations (despite being wrong!) and the SELinux configuration may add complexity to the first article, which could lead to misleading thoughts. The previous article might be a little bit more detailed, so the reader could check it, too.
It’s worth mentioning the key points of NFS-Ganesha:

  • a user-mode file sharing server
  • supports NFS 3, 4.x and 9P
  • using plugins for different file systems
  • CentOS Storage Special Interest Group offers a file repository with NFS-Ganesha server
  • supports file systems like ext4, xfs, brtfs, zfs and more. There are sample configurations: https://github.com/phdeniel/nfs-ganesha/tree/master/src/config_samples
  • supports cluster and/or distributed file systems like GlusterFS, Ceph, GPFS, HPSS, Lustre
  • Current version 3.5 and it is included in the official SIG CentOS Storage Special Interest Group repository.

This article assumes the reader has a clean CentOS 8 Stream installation with SELinux in enforcing mode.

STEP 1) Install the repository and NFS-Ganesha software

NFS-Ganesha 3 packages are from the CentOS Storage SIG repository, which is a good repository and may be trusted.

dnf install -y centos-release-nfs-ganesha30
dnf install -y nfs-ganesha nfs-ganesha-vfs nfs-ganesha-selinux

STEP 2) Configuration for exporting a directory.

There are two files under /etc/ganesha/:

ganesha.conf
vfs.conf

ganesha.conf includes global configuration and NFS share configuration. Each export path begins with the keyword EXPORT followed by a block ebraced by brackets {}.
vfs.conf includes a simple example for the VFS plugin, but this configuration file is not used by the NFS Ganesha server. It is just a sample file.
Here is a simple configuration, which exports /mnt/storage with Read/Write permissions to a single IP. Just add at the end of the file /etc/ganesha/ganesha.conf contains:

 
EXPORT
{
        Export_Id = 2;
        Path = /mnt/storage1;
        Pseudo = /mnt/storage1;
        Protocols = 3,4;
        Access_Type = RW;
        Squash = None;
        FSAL
        {
                Name = VFS;
        }
        CLIENT
        {
                Clients = 192.168.0.12;
        }
}

STEP 3) Start the server and mount the exported directory. Configure the firewall.

Start the server, enable the service to start on boot and then configure the firewall to pass the NFS requests:

systemctl start nfs-ganesha
systemctl enable nfs-ganesha
firewall-cmd --permanent --zone=public --add-service=nfs
firewall-cmd --reload

Keep on reading!

Simple export of a ext4 directory with NFS Ganesha 3.5 server in CentOS 8 without SELinux

NFS Ganesha is a user-mode file sharing server, which supports NFS 3 and 4.x versions and 9P. NFS Ganesha has several interesting plugins that support exporting files from the cluster and distributed file systems like Ceph and Glusterfs Exporting a file system with NFS Ganesha is simple enough if you do not use SELinux or SELinux is in permissive mode!
This article is to show how to export a server’s directory using NFS protocol Just to note the NFS-Ganesha is tested and supports ext2/ext3/ext4, xfs, brtfs, zfs file systems as of version 3.5 (check the manual for xfs, brtfs and zfs exports – here are sample configurations for them https://github.com/phdeniel/nfs-ganesha/tree/master/src/config_samples). To be able to export a file directory the VFS Ganesha plugin is used. A clean install of minimal CentOS 8 Stream is used so the installation log may differ significantly from the user’s log but the user will see all the dependencies, which are required for this setup.

STEP 1) Install the repository and NFS-Ganesha software

NFS-Ganesha 3 packages are from the CentOS Storage SIG repository, which is a good repository and may be trusted.

dnf install -y centos-release-nfs-ganesha30
dnf install -y nfs-ganesha nfs-ganesha-vfs

STEP 2) Configuration for exporting a directory.

There are two files under /etc/ganesha/:

ganesha.conf
vfs.conf

ganesha.conf includes global configuration and NFS share configuration. Each export path begins with the keyword EXPORT followed by a block ebraced by brackets {}.
vfs.conf includes a simple example for the VFS plugin, but this configuration file is not used by the NFS Ganesha server. It is just a sample file.
Here is a simple configuration, which exports /mnt/storage with Read/Write permissions to a single IP. Just add at the end of the file /etc/ganesha/ganesha.conf contains:

 
EXPORT
{
        Export_Id = 2;
        Path = /mnt/storage1;
        Pseudo = /mnt/storage1;
        Protocols = 3,4;
        Access_Type = RW;
        Squash = None;
        FSAL
        {
                Name = VFS;
        }
        CLIENT
        {
                Clients = 192.168.0.12;
        }
}

STEP 3) Start the server and mount the exported directory. Configure the firewall.

Start the server, enable the service to start on boot and then configure the firewall to pass the NFS requests:

systemctl start nfs-ganesha
systemctl enable nfs-ganesha
firewall-cmd --permanent --zone=public --add-service=nfs
firewall-cmd --reload

Keep on reading!

Stopping the glusterfs volume releases disk sleep process hangs

A quick tip for GlusterFS volume. There are multiple possible reasons for a Linux process to hang in “Disk Sleep” state, which even the KILL -9 cannot interrupt:

  • a bug in GlusterFS
  • just bad options turn on online
  • other device relying on a GlusterFS, which is unavailable.
[17294588.184470] INFO: task gdisk:12505 blocked for more than 120 seconds.
[17294588.184538] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[17294588.184628] gdisk           D ffff8ce01fb9acc0     0 12505  26866 0x00000080
[17294588.184780] Call Trace:
[17294588.184844]  [<ffffffffbaed3d81>] ? __wake_up_common_lock+0x91/0xc0
[17294588.184910]  [<ffffffffbb585da9>] schedule+0x29/0x70
[17294588.184974]  [<ffffffffbb5838b1>] schedule_timeout+0x221/0x2d0
[17294588.185037]  [<ffffffffbaed3dc3>] ? __wake_up+0x13/0x20
[17294588.185102]  [<ffffffffc0a05d2e>] ? loop_make_request+0x12e/0x210 [loop]
[17294588.185169]  [<ffffffffbaf06d32>] ? ktime_get_ts64+0x52/0xf0
[17294588.185232]  [<ffffffffbb58549d>] io_schedule_timeout+0xad/0x130
[17294588.185304]  [<ffffffffbb5863dd>] wait_for_completion_io+0xfd/0x140
[17294588.185369]  [<ffffffffbaedb990>] ? wake_up_state+0x20/0x20
[17294588.185468]  [<ffffffffbb157e64>] blkdev_issue_flush+0xb4/0x110
[17294588.185533]  [<ffffffffbb08d335>] blkdev_fsync+0x35/0x50
[17294588.185598]  [<ffffffffbb082f57>] do_fsync+0x67/0xb0
[17294588.185671]  [<ffffffffbb083240>] SyS_fsync+0x10/0x20
[17294588.185734]  [<ffffffffbb592ed2>] system_call_fastpath+0x25/0x2a
[17294708.187598] INFO: task gdisk:12505 blocked for more than 120 seconds.
[17294708.187664] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[17294708.187753] gdisk           D ffff8ce01fb9acc0     0 12505  26866 0x00000080
[17294708.187905] Call Trace:
[17294708.187968]  [<ffffffffbaed3d81>] ? __wake_up_common_lock+0x91/0xc0
[17294708.188033]  [<ffffffffbb585da9>] schedule+0x29/0x70
[17294708.188096]  [<ffffffffbb5838b1>] schedule_timeout+0x221/0x2d0
[17294708.188159]  [<ffffffffbaed3dc3>] ? __wake_up+0x13/0x20
[17294708.188223]  [<ffffffffc0a05d2e>] ? loop_make_request+0x12e/0x210 [loop]
[17294708.188289]  [<ffffffffbaf06d32>] ? ktime_get_ts64+0x52/0xf0
[17294708.188352]  [<ffffffffbb58549d>] io_schedule_timeout+0xad/0x130
[17294708.188416]  [<ffffffffbb5863dd>] wait_for_completion_io+0xfd/0x140
[17294708.188480]  [<ffffffffbaedb990>] ? wake_up_state+0x20/0x20
[17294708.188545]  [<ffffffffbb157e64>] blkdev_issue_flush+0xb4/0x110
[17294708.188624]  [<ffffffffbb08d335>] blkdev_fsync+0x35/0x50
[17294708.188690]  [<ffffffffbb082f57>] do_fsync+0x67/0xb0
[17294708.188754]  [<ffffffffbb083240>] SyS_fsync+0x10/0x20
[17294708.188828]  [<ffffffffbb592ed2>] system_call_fastpath+0x25/0x2a

The above example of dmesg log shows the gdisk process stuck in “Disk Sleep” state, because of a loop device from a file on an unavailable GlusterFS volume! Kill -9 won’t help, the process will remain in this bad state and even a restart would be difficult to perform!

[root@srv1 ~]# gluster volume stop VOL2 
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: VOL2: success
[root@srv1 ~]# gluster volume start VOL2 
volume start: VOL2: success

The solution is to stop the GlusterFS Volume and all the blocked processes on bad devices such as above would be released. The processes will carry on executing or will end their execution after issuing a stop command to the volume. No problem to start the GlusterFS volume immediately after the stop!
NOTE: executing STOP command would affect all servers using this volume. The volume becomes inaccessible for all!

MySQL slave upgrade: Slave failed to initialize relay log info structure from the repository

MySQL slave after upgrade from 5.6.x to 5.7.x may throw the following error:

mysql> START SLAVE;
ERROR 1872 (HY000): Slave failed to initialize relay log info structure from the repository

The best solution for this error is to:

  • Master server – mysqldump the database with –master-data=1 –single-transaction
  • On the slave server issue command “RESET SLAVE;”
  • On the slave server import the dump sql file and issue “CHANGE MASTER” command with the meta data written in the sql dump
  • On the slave server issue START SLAVE to start the replication.

Here is an a real world example:
First, mysqldump in the master with

root@master ~ # mysqldump --master-data=1 --single-transaction mydb > /root/mydb.sql
root@master ~ # grep "CHANGE MASTER" media.sql 
CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.023283', MASTER_LOG_POS=537774724;

And then copy the dump file to the slave server and import it and issue several specific slave commands:

root@slave ~ # mysql < /root/mydb.sql
root@slave ~ # mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 23
Server version: 5.7.31-log Gentoo Linux mysql-5.7.31

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> reset slave;
Query OK, 0 rows affected (0.01 sec)
mysql> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.023283', MASTER_LOG_POS=537774724;
Query OK, 0 rows affected (0.00 sec)
mysql> START SLAVE;
Query OK, 0 rows affected (0.00 sec)

mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Queueing master event to the relay log
                  Master_Host: 10.10.10.10
                  Master_User: ruser
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.023283
          Read_Master_Log_Pos: 641769286
               Relay_Log_File: slave-relay-bin.000002
                Relay_Log_Pos: 90874706
        Relay_Master_Log_File: mysql-bin.023283
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: mydb.%
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 628649113
              Relay_Log_Space: 103995088
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 2395
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 101
                  Master_UUID: cd1bcebb-cc27-11e8-90c9-801844f2c4d8
             Master_Info_File: /mnt/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Reading event from the relay log
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
1 row in set (0.00 sec)

The replication is advancing. It is 2395 seconds behind the master.
Keep on reading!

VBoxManage: error: Failed to initialize COM! NS_ERROR_FILE_TARGET_DOES_NOT_EXIST (0x80520006)

What an error! And the VirtualBox stopped loading anymore! This error:

myuser@srv ~ $ VBoxManage list vms
VBoxManage: error: Failed to initialize COM! (hrc=NS_ERROR_FILE_TARGET_DOES_NOT_EXIST)

and here with the GIU, which offers a little bit more information about the error ID:
This error occurs despite the successfully loaded VirtualBox kernel modules!

main menu
VBoxManage: error: Failed to initialize COM! NS_ERROR_FILE_TARGET_DOES_NOT_EXIST (0x80520006)

The two errors are the same and hint there is a file missing (or files?)? In our case, after upgrading from virtualbox-bin (a binary package) to a virtualbox (a package, which actually builds the VirtualBox on the system) under Gentoo, but apparently, this error could occur to everyone, who tries to build yourself the VirtualBox bundle.

The solution is simple: just DO NOT CHANGE the installation path of the Virtualbox software, which by default is /opt/VirtualBox.

The path is hardcoded in the sources and cannot be changed! At present version 6.1.16, we could not find a build option to change it! Of course, the linking /opt/VirtualBox would do the trick to move the installation physically away from the /opt, but the path must be valid /opt/VirtualBox.

In Gentoo, the emerge default installation goes to “/usr/lib64/virtualbox“, so the maintainer of the package changed the path and Virtualbox stopped working! Or a user build it and install it in any other location should link the /opt/VirtualBox to the installation directory. For example, in Gentoo, the fix will be:

root@srv ~ $ ln -s /usr/lib64/virtualbox /opt/VirtualBox
root@srv ~ $ exit
myuser@srv ~ $ VBoxManage list vms
"gentoo_raw" {55346caf-04db-4d88-831a-111111111111}
"diskless" {44346caf-c952-5555-b8a3-111111111111}
"diskless-linux" {44346caf-424f-487f-ae8d-111111111111}
"centos7-netinstall" {44346caf-4e86-441b-8d1e-111111111111}

A simple link would bring back Virtualbox to live.

Bonus

Probably, there is a configuration file “xpti.dat” in two or more locations: ~/.config/VirtualBox/xpti.dat and ~/.VirtualBox/xpti.dat, which is generated on every start of a VirtualBox. In the file there are configuration lines like:
Keep on reading!