Any IT here? Help Me!

Caching NFS files with cachefilesd

A great tool for caching a network filesystem like NFS mounts is cachefilesd! It is easy to use it and a good deal of stats can be retrieved from the tool. More on how it works here https://www.kernel.org/doc/Documentation/filesystems/caching/fscache.txt

Here are quick steps to cache an NFS mounts (it works with NFS-Ganesha servers, too):

Install the daemon tool cachefilesd
Check the configuration file /etc/cachefilesd.conf. In most cases, no need to edit the file! Just check the disk limits if they are good.
Start the cachefilesd daemon.
Mount the network directories with “fsc” option. Umount and mount them all if they’ve been already mounted. The fsc is mandatory option to enable file cacheing of a network mount.
Check stats to see if the file cching is working properly.

The example below is under CentOS 8, but it is almost the same in most Linux distributions.

STEP 1) Install the daemon tool cachefilesd

This is straight forward, just install it with the package manager:

[root@srv ~]# dnf install cachefilesd
Last metadata expiration check: 2:33:44 ago on Tue 08 Dec 2020 07:18:01 AM UTC.
Dependencies resolved.
=============================================================================================================================================================================================
 Package                                        Architecture                              Version                                            Repository                                 Size
=============================================================================================================================================================================================
Installing:
 cachefilesd                                    x86_64                                    0.10.10-4.el8                                      BaseOS                                     43 k

Transaction Summary
=============================================================================================================================================================================================
Install  1 Package

Total download size: 43 k
Installed size: 71 k
Is this ok [y/N]: y
Downloading Packages:
cachefilesd-0.10.10-4.el8.x86_64.rpm                                                                                                                         3.1 MB/s |  43 kB     00:00    
---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                        2.8 MB/s |  43 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                     1/1 
  Installing       : cachefilesd-0.10.10-4.el8.x86_64                                                                                                                                    1/1 
  Running scriptlet: cachefilesd-0.10.10-4.el8.x86_64                                                                                                                                    1/1 
  Verifying        : cachefilesd-0.10.10-4.el8.x86_64                                                                                                                                    1/1 

Installed:
  cachefilesd-0.10.10-4.el8.x86_64                                                                                                                                                           

Complete!

STEP 2) Check the configuration file and tune for your system.

In most cases, the defaults in /etc/cachefilesd.conf are good to start with:

dir /var/cache/fscache
tag mycache
brun 10%
bcull 7%
bstop 3%
frun 10%
fcull 7%
fstop 3%

# Assuming you're using SELinux with the default security policy included in
# this package
secctx system_u:system_r:cachefiles_kernel_t:s0

The directory where the cache will reside and the lines with the percentages are for disk space limitation. “brun 10%” means cache can runs freely till the disk space drops below 10%. “bcull 7%” – culling the cache when the free space drops below “7%” and more in the man page (or https://linux.die.net/man/5/cachefilesd.conf).
So if one maintains disk free space below 10% the configuration file should be edited.

STEP 3) Start the cachefilesd daemon.

And enable on boot to start automatically.

[root@srv ~]# systemctl start cachefilesd
[root@srv ~]# systemctl enable cachefilesd
Created symlink /etc/systemd/system/multi-user.target.wants/cachefilesd.service → /usr/lib/systemd/system/cachefilesd.service.
[root@srv ~]# systemctl status cachefilesd
● cachefilesd.service - Local network file caching management daemon
   Loaded: loaded (/usr/lib/systemd/system/cachefilesd.service; enabled; vendor preset: disabled)
   Active: active (running) since Tue 2020-12-08 10:01:24 UTC; 11s ago
 Main PID: 29786 (cachefilesd)
    Tasks: 1 (limit: 408616)
   Memory: 2.5M
   CGroup: /system.slice/cachefilesd.service
           └─29786 /usr/sbin/cachefilesd -n -f /etc/cachefilesd.conf

Dec 08 10:01:24 srv systemd[1]: Starting Local network file caching management daemon...
Dec 08 10:01:24 srv systemd[1]: Started Local network file caching management daemon.
Dec 08 10:01:24 srv cachefilesd[29786]: About to bind cache
Dec 08 10:01:24 srv cachefilesd[29786]: Bound cache
Dec 08 10:01:24 srv cachefilesd[29786]: Daemon Started

The status command shows the daemon cachefilesd is running. But does it cache?

STEP 4) Mount the network filesystems with option fsc

To make cachefilesd cache a network mount the option fsc must be included in the mount options. Remount may not work correctly, so to be sure a full umount/mount should be executed. Here is an example /etc/fstab file:

192.168.0.1:/mnt/storage /mnt/storage  nfs defaults,hard,intr,noexec,nosuid,_netdev,fsc,vers=4 0 0

And then mount with simple command:

mount /mnt/storage

Check whether the mounts if the FS cache is used. FSC must be “yes”.

[root@srv ~]# cat /proc/fs/nfsfs/volumes
NV SERVER   PORT DEV          FSID                              FSC
v4 c0a80001  801 0:41         d4098a2af096148:ec7560388cbe5b83  yes

There is a proc file for cache statistics:

[root@srv ~]# cat /proc/fs/fscache/stats
FS-Cache statistics
Cookies: idx=49 dat=4385599 spc=0
Objects: alc=43666 nal=0 avl=43666 ded=36002
ChkAux : non=0 ok=12289 upd=0 obs=761
Pages  : mrk=24915179 unc=24492585
Acquire: n=4385648 nul=0 noc=0 ok=4385648 nbf=0 oom=0
Lookups: n=43666 neg=31372 pos=12294 crt=31372 tmo=0
Invals : n=1 run=1
Updates: n=0 nul=0 run=1
Relinqs: n=4377930 nul=0 wcr=0 rtr=0
AttrChg: n=0 ok=0 nbf=0 oom=0 run=0
Allocs : n=0 ok=0 wt=0 nbf=0 int=0
Allocs : ops=0 owt=0 abt=0
Retrvls: n=751549 ok=716860 wt=21436 nod=34689 nbf=0 int=0 oom=0
Retrvls: ops=751549 owt=9158 abt=0
Stores : n=550412 ok=550412 agn=0 nbf=0 oom=0
Stores : ops=33238 run=583650 pgs=550412 rxd=550412 olm=0
VmScan : nos=23963352 gon=0 bsy=0 can=0 wt=0
Ops    : pend=9160 run=784788 enq=26874960 can=0 rej=0
Ops    : ini=1301962 dfr=265 rel=1301962 gc=265
CacheOp: alo=0 luo=0 luc=0 gro=0
CacheOp: inv=0 upo=0 dro=0 pto=0 atc=0 syn=0
CacheOp: rap=0 ras=0 alp=0 als=0 wrp=0 ucp=0 dsp=0
CacheEv: nsp=761 stl=0 rtr=0 cul=0

And here is the cache directory filled with files. If there are no files, the FS cache is not used, probably the mount is not mounted with FSC! Umount and mount the mounts again.

[root@srv ~]# find /var/cache/fscache|head -n 20
/var/cache/fscache
/var/cache/fscache/graveyard
/var/cache/fscache/cache
/var/cache/fscache/cache/@4a
/var/cache/fscache/cache/@4a/I03nfs
/var/cache/fscache/cache/@4a/I03nfs/@84
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68/J1100000000000Q0goaWH946iIn7oUMELrd80000000040000g00Kb000wFe000jt000oG3000000040000g000000000
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68/J1100000000000Q0goaWH946iIn7oUMELrd80000000040000g00Kb000wFe000jt000oG3000000040000g000000000/@8b
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68/J1100000000000Q0goaWH946iIn7oUMELrd80000000040000g00Kb000wFe000jt000oG3000000040000g000000000/@8b/EA0g00sg0200n8000000ixBMHyy9gdcUm_O8ewl7XMW1n8cXgyl60
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68/J1100000000000Q0goaWH946iIn7oUMELrd80000000040000g00Kb000wFe000jt000oG3000000040000g000000000/@8b/EA0g00sg0200n8000000ixBMHyy9gdcUm_O8ewl7XZG2n84qfxl60
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68/J1100000000000Q0goaWH946iIn7oUMELrd80000000040000g00Kb000wFe000jt000oG3000000040000g000000000/@8b/EA0g00sg0200n8000000ixBMHyy9gdcUm_O8ewl7XQM2n8IQ4El60
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68/J1100000000000Q0goaWH946iIn7oUMELrd80000000040000g00Kb000wFe000jt000oG3000000040000g000000000/@8b/EA0g00sg0200n8000000ixBMHyy9gdcUm_O8ewl7XZO2n8Ms7kl60
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68/J1100000000000Q0goaWH946iIn7oUMELrd80000000040000g00Kb000wFe000jt000oG3000000040000g000000000/@8b/EA0g00sg0200n8000000ixBMHyy9gdcUm_O8ewl7XvQ2n88dKgl60
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68/J1100000000000Q0goaWH946iIn7oUMELrd80000000040000g00Kb000wFe000jt000oG3000000040000g000000000/@8b/EA0g00sg0200n8000000ixBMHyy9gdcUm_O8ewl7XV40o8cREC6b0
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68/J1100000000000Q0goaWH946iIn7oUMELrd80000000040000g00Kb000wFe000jt000oG3000000040000g000000000/@8b/EA0g00sg0200n8000000ixBMHyy9gdcUm_O8ewl7X3Y1n8MHO_l60
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68/J1100000000000Q0goaWH946iIn7oUMELrd80000000040000g00Kb000wFe000jt000oG3000000040000g000000000/@8b/EA0g00sg0200n8000000ixBMHyy9gdcUm_O8ewl7Xhkwn80f23zb0
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68/J1100000000000Q0goaWH946iIn7oUMELrd80000000040000g00Kb000wFe000jt000oG3000000040000g000000000/@8b/EA0g00sg0200n8000000ixBMHyy9gdcUm_O8ewl7XEQ2n8IuAll60
/var/cache/fscache/cache/@4a/I03nfs/@84/Jc0010800840000cG0400/@68/J1100000000000Q0goaWH946iIn7oUMELrd80000000040000g00Kb000wFe000jt000oG3000000040000g000000000/@8b/EA0g00sg0200n8000000ixBMHyy9gdcUm_O8ewl7XoO2n8A7Bll60
[root@srv ~]# du -d 1 -h /var/cache/fscache
4.0K    /var/cache/fscache/graveyard
3.8G    /var/cache/fscache/cache
3.8G    /var/cache/fscache

There are 3.8G in the cache.

Create and export a GlusterFS volume with NFS-Ganesha in CentOS 8

GlusterFS built-in NFS server supports only NFS version 3. GlusterFS offers NFS exports using NFS-Ganesha, which supports NFS version 3 and 4 protocols.
NFS-Ganesha server is a user-mode file sharing server, which offers a GlusterFS plugin to export GlusterFS volumes. In the following article, the NSF-Ganesha and GlusterFS are installed and a simple GlusterFS volume is created and then exported through NFS 3 and 4 version protocols.
The version of the software in this article:

CentOS Stream release 8 (25.04.2021)
GlusterFS 8.4
NFS-Ganesha 3.5

STEP 1) Install GlusterFS.

dnf install -y centos-release-gluster
dnf install -y glusterfs-server

The first line will installs a new repository under the SIG management – https://wiki.centos.org/SpecialInterestGroup/Storage. The second line installs the GlusterFS server.

STEP 2) Install NFS-Ganesha.

dnf install -y centos-release-nfs-ganesha30
dnf install -y nfs-ganesha nfs-ganesha-gluster

The first line again installs a new repository under the SIG management and the second line installs the NFS-Ganesha server with Gluster plugin.

STEP 3) Create GlusterFS volume

Start the GlusterFS server and create a simple 3 replicas volume with:
Start the GlusterFS on all the three nodes and enable the GlusterFS communication between the three nodes using firewall-cmd utility. So execute the following commands:

systemctl start glusterd
firewall-cmd --permanent --new-zone=glusternodes
firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.200
firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.201
firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.202
firewall-cmd --permanent --zone=glusternodes --add-service=glusterfs
firewall-cmd --reload

On the first node create the GlusterFS volume. First, add the glnode2 and glnode3 to the cluster.

gluster peer probe glnode2
gluster peer probe glnode3
gluster volume create VOL1 replica 3 transport tcp glnode1:/mnt/storage/gluster/brick glnode2:/mnt/storage/gluster/brick glnode3:/mnt/storage/gluster/brick
gluster volume start VOL1

Keep on reading!

Gentoo emerge virtualbox- Mesa / GLU: Mesa not found at, Mesa headers not found

Emerging the package app-emulation/virtualbox the following error occurs:

Checking for Mesa / GLU: 
  Mesa not found at -L/usr/X11R6/lib -L/usr/X11R6/lib64 -L/usr/local/lib -lXext -lX11 -lGL -I/usr/local/include or Mesa headers not found
  Check the file /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/configure.log for detailed error information.
Check /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/configure.log for details
 * ERROR: app-emulation/virtualbox-6.1.18::gentoo failed (configure phase):
 *   (no error message)
 * 
 * Call stack:
 *     ebuild.sh, line  125:  Called src_configure
 *   environment, line 5504:  Called doecho './configure' '--with-gcc=x86_64-pc-linux-gnu-gcc' '--with-g++=x86_64-pc-linux-gnu-g++' '--disable-dbus' '--disable-kmods' '--disable-alsa' '--disable-docs' '--disable-devmapper' '--disable-pulse' '--disable-python' '--enable-webservice' '--enable-vnc'
 *   environment, line 1538:  Called die
 * The specific snippet of code:
 *       "$@" || die

The configure script reports the mesa is missing, but the package media-libs/mesa is installed. Reinstalling does not fix the problem.
Farther investigation in the logs by checking the configure.log reveals the real problem:

srv ~ # tail -n 16 /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/configure.log
***** Checking Mesa / GLU *****
compiling the following source file:
#include <cstdio>
#include <X11/Xlib.h>
#include <GL/glx.h>
#include <GL/glu.h>
extern "C" int main(void)
{
  return 0;
}
using the following command line:
x86_64-pc-linux-gnu-g++  -fPIC -g -O -Wall -o /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/.tmp_out /var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/.tmp_src.cc "-L/usr/X11R6/lib -L/usr/X11R6/lib64 -L/usr/local/lib -lXext -lX11 -lGL -I/usr/local/include"
/var/tmp/portage/app-emulation/virtualbox-6.1.18/work/VirtualBox-6.1.18/.tmp_src.cc:4:10: fatal error: GL/glu.h: No such file or directory
    4 | #include <GL/glu.h>
      |          ^~~~~~~~~~
compilation terminated.

The glu part of mesa is missing. In Gentoo, the glu (https://gitlab.freedesktop.org/mesa/glu) is not included in the media-libs/mesa and it is a separate package media-libs/glu.

The solution is to emerge media-libs/glu and then the app-emulation/virtualbox.

emerge -v media-libs/glu

Another Linux distribution may include glu in the main mesa package.

Here, the conclusion is to always check the configure.log, because it reports the exact error and not to trust the generic output of the configure script.

glusterfs with localhost (127.0.0.1) nodes on different servers – glusterfs volume with 3 replicas

Binding the GlusterFS nodes on a physical interface may lead to local availability problems even for replication nodes. Bringing down the physical interface will bring down the nodes, even the local replica for the local mounts and applications.

100% local uptime. The local replica will be 100% available. The loopback interface is always in the upstate! Network interfaces are not 100% upstate, because of network reload or cable unplug. Cable unplug (or port down for a switch configuration reload) could lead to a short time unavailable of the local node even for the local system!
node resolve rely on /etc/hosts records, not network and remote DNS system.
No speed limit. The read from the local system through the loopback interface could easily increase above 1G or even 10G. Probably, building nodes with replicas over a 1G network is much more affected than a network with 10G connectivity. Reading from a node relying on a loopback interface could pass 10G, even though the server is connected to a 1G network!

An addition note – another kind of the proposed solution here is to use a virtual interface to bind the IP of the GlusterFS brick. The most common type of virtual interface is using a bridge interface for the IP.

The example here is to bring a GlusterFS volume in replication mode with 3 servers, i.e. 3 replicas. Each server may mount locally the GlusterFS volume with name VOL1 and it would not get unavailable if the main interface

server’s hostname: node1, hostname for the replica brick glnode1. IP: 192.168.0.20, but the node1 locally is resolved as 127.0.0.1 through /etc/hosts.
server’s hostname: node2, hostname for the replica brick glnode2. IP: 192.168.0.30, but the node1 locally is resolved as 127.0.0.1 through /etc/hosts.
server’s hostname: node3, hostname for the replica brick glnode3 IP: 192.168.0.31, but the node3 locally is resolved as 127.0.0.1 through /etc/hosts.

Of course, the server’s hostname could be used, but it better to have a separate domain for the GlusterFS bricks. Sometimes server hostnames should be a real IP or some software may rely on it, too.

And here are all the commands to bring up the GlusterFS volume on 3 servers:

STEP 1) Install GlusterFS software and initial configuration.

There are GlisterFS packages in the official CentOS 8, but a newer version is supported in the Storage SIG. The GlusterFS version installed in this article is 8.4. Install the software under node1, node2, and node3.

yum install -y centos-release-gluster
yum install -y glusterfs-server

Add the following lines at the end of node1:/etc/hosts file.

127.0.0.1 glnode1
192.168.0.30 glnode2
192.168.0.31 glnode3

Add the following lines at the end of node2:/etc/hosts file.

192.168.0.20 glnode1
127.0.0.1 glnode2
192.168.0.31 glnode3

Add the following lines at the end of node3:/etc/hosts file.

192.168.0.20 glnode1   
192.168.0.30 glnode2
127.0.0.1 glnode3

Start the GlusterFS service on the three nodes

systemctl start glusterd

Mount the storage device if any and make the directory where the GlusterFS brick will reside:

mount /mnt/storage/
mkdir -p /mnt/storage/gluster/brick

STEP 2) Configure the firewall.

CentOS 8 uses firewalld and here a new zone for the GlusterFS is created and the GlusterFS service is added in the whitelist of the new zone. The three IPs of the nodes are also added in the new zone:

firewall-cmd --permanent --new-zone=glusternodes
firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.20
firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.30
firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.31
firewall-cmd --permanent --zone=glusternodes --add-service=glusterfs
firewall-cmd --reload

STEP 3) Add peers to the GlusterFS cluster and create a 3 node replica volume.

gluster peer probe glnode2
gluster peer probe glnode3
gluster volume create VOL1 replica 3 transport tcp glnode1:/mnt/storage/gluster/brick glnode2:/mnt/storage/gluster/brick glnode3:/mnt/storage/gluster/brick
gluster volume start VOL1

The GlusterFS volume can be mounted with:

mount -t glusterfs glnode1:/VOL1 /mnt/VOL1/

And /etc/fstab sample line:

glnode1:/VOL1 /mnt/VOL1 glusterfs defaults,noatime,direct-io-mode=disable 0 0

Always use the local hostname for the current node server. If you would like to mount the volume VOL1 on node1, use glnode1:/VOL1 and so on.
Bringing down the physical interface of the server, which is connected to the Internet (aka the network interface with the real IP) would not make the GlusterFS brick unavailable for the local mounts and applications.

In a cluster with only replicas, the local application will just continue using the mounted GlusterFS volume (or native GlusterFS clients) relying only on the local Gluster brick till the main Internet connection comes back.

Create 3 node replica volume – the whole output

[root@node1 ~]# yum install -y centos-release-gluster
CentOS Stream 8 - AppStream                                                                                 5.8 MB/s | 6.7 MB     00:01    
CentOS Stream 8 - BaseOS                                                                                    2.0 MB/s | 2.3 MB     00:01    
CentOS Stream 8 - Extras                                                                                     22 kB/s | 9.1 kB     00:00    
Dependencies resolved.
============================================================================================================================================
 Package                                          Architecture              Version                         Repository                 Size
============================================================================================================================================
Installing:
 centos-release-gluster8                          noarch                    1.0-1.el8                       extras                    9.3 k
Installing dependencies:
 centos-release-storage-common                    noarch                    2-2.el8                         extras                    9.4 k

Transaction Summary
============================================================================================================================================
Install  2 Packages

Total download size: 19 k
Installed size: 2.4 k
Downloading Packages:
(1/2): centos-release-gluster8-1.0-1.el8.noarch.rpm                                                         136 kB/s | 9.3 kB     00:00    
(2/2): centos-release-storage-common-2-2.el8.noarch.rpm                                                     145 kB/s | 9.4 kB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                        27 kB/s |  19 kB     00:00     
warning: /var/cache/dnf/extras-9705a089504ff150/packages/centos-release-gluster8-1.0-1.el8.noarch.rpm: Header V3 RSA/SHA256 Signature, key ID 8483c65d: NOKEY
CentOS Stream 8 - Extras                                                                                    725 kB/s | 1.6 kB     00:00    
Importing GPG key 0x8483C65D:
 Userid     : "CentOS (CentOS Official Signing Key) <security@centos.org>"
 Fingerprint: 99DB 70FA E1D7 CE22 7FB6 4882 05B5 55B3 8483 C65D
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-centosofficial
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                    1/1 
  Installing       : centos-release-storage-common-2-2.el8.noarch                                                                       1/2 
  Installing       : centos-release-gluster8-1.0-1.el8.noarch                                                                           2/2 
  Running scriptlet: centos-release-gluster8-1.0-1.el8.noarch                                                                           2/2 
  Verifying        : centos-release-gluster8-1.0-1.el8.noarch                                                                           1/2 
  Verifying        : centos-release-storage-common-2-2.el8.noarch                                                                       2/2 

Installed:
  centos-release-gluster8-1.0-1.el8.noarch                           centos-release-storage-common-2-2.el8.noarch                          

Complete!
[root@node1 ~]# yum install -y glusterfs-server
Last metadata expiration check: 0:00:14 ago on Wed Apr 14 13:01:50 2021.
Dependencies resolved.
============================================================================================================================================
 Package                                      Architecture          Version                            Repository                      Size
============================================================================================================================================
Installing:
 glusterfs-server                             x86_64                8.4-1.el8                          centos-gluster8                1.4 M
Installing dependencies:
 attr                                         x86_64                2.4.48-3.el8                       baseos                          68 k
 device-mapper-event                          x86_64                8:1.02.175-5.el8                   baseos                         269 k
 device-mapper-event-libs                     x86_64                8:1.02.175-5.el8                   baseos                         269 k
 device-mapper-persistent-data                x86_64                0.8.5-4.el8                        baseos                         468 k
 glusterfs                                    x86_64                8.4-1.el8                          centos-gluster8                689 k
 glusterfs-cli                                x86_64                8.4-1.el8                          centos-gluster8                214 k
 glusterfs-client-xlators                     x86_64                8.4-1.el8                          centos-gluster8                899 k
 glusterfs-fuse                               x86_64                8.4-1.el8                          centos-gluster8                171 k
 libaio                                       x86_64                0.3.112-1.el8                      baseos                          33 k
 libgfapi0                                    x86_64                8.4-1.el8                          centos-gluster8                125 k
 libgfchangelog0                              x86_64                8.4-1.el8                          centos-gluster8                 67 k
 libgfrpc0                                    x86_64                8.4-1.el8                          centos-gluster8                 89 k
 libgfxdr0                                    x86_64                8.4-1.el8                          centos-gluster8                 61 k
 libglusterd0                                 x86_64                8.4-1.el8                          centos-gluster8                 45 k
 libglusterfs0                                x86_64                8.4-1.el8                          centos-gluster8                350 k
 lvm2                                         x86_64                8:2.03.11-5.el8                    baseos                         1.6 M
 lvm2-libs                                    x86_64                8:2.03.11-5.el8                    baseos                         1.1 M
 psmisc                                       x86_64                23.1-5.el8                         baseos                         151 k
 python3-pyxattr                              x86_64                0.5.3-18.el8                       centos-gluster8                 35 k
 rpcbind                                      x86_64                1.2.5-8.el8                        baseos                          70 k
 userspace-rcu                                x86_64                0.10.1-4.el8                       baseos                         101 k

Transaction Summary
============================================================================================================================================
Install  22 Packages

Total download size: 8.2 M
Installed size: 24 M
Downloading Packages:
(1/22): glusterfs-cli-8.4-1.el8.x86_64.rpm                                                                  1.1 MB/s | 214 kB     00:00    
(2/22): glusterfs-8.4-1.el8.x86_64.rpm                                                                      2.6 MB/s | 689 kB     00:00    
(3/22): glusterfs-fuse-8.4-1.el8.x86_64.rpm                                                                 1.1 MB/s | 171 kB     00:00    
(4/22): glusterfs-client-xlators-8.4-1.el8.x86_64.rpm                                                       2.3 MB/s | 899 kB     00:00    
(5/22): libgfapi0-8.4-1.el8.x86_64.rpm                                                                      1.8 MB/s | 125 kB     00:00    
(6/22): libgfchangelog0-8.4-1.el8.x86_64.rpm                                                                755 kB/s |  67 kB     00:00    
(7/22): libgfrpc0-8.4-1.el8.x86_64.rpm                                                                      756 kB/s |  89 kB     00:00    
(8/22): libgfxdr0-8.4-1.el8.x86_64.rpm                                                                      579 kB/s |  61 kB     00:00    
(9/22): libglusterd0-8.4-1.el8.x86_64.rpm                                                                   641 kB/s |  45 kB     00:00    
(10/22): glusterfs-server-8.4-1.el8.x86_64.rpm                                                              3.4 MB/s | 1.4 MB     00:00    
(11/22): libglusterfs0-8.4-1.el8.x86_64.rpm                                                                 1.0 MB/s | 350 kB     00:00    
(12/22): python3-pyxattr-0.5.3-18.el8.x86_64.rpm                                                             97 kB/s |  35 kB     00:00    
(13/22): attr-2.4.48-3.el8.x86_64.rpm                                                                        85 kB/s |  68 kB     00:00    
(14/22): device-mapper-event-1.02.175-5.el8.x86_64.rpm                                                      342 kB/s | 269 kB     00:00    
(15/22): device-mapper-event-libs-1.02.175-5.el8.x86_64.rpm                                                 338 kB/s | 269 kB     00:00    
(16/22): libaio-0.3.112-1.el8.x86_64.rpm                                                                    679 kB/s |  33 kB     00:00    
(17/22): device-mapper-persistent-data-0.8.5-4.el8.x86_64.rpm                                               1.5 MB/s | 468 kB     00:00    
(18/22): psmisc-23.1-5.el8.x86_64.rpm                                                                       1.5 MB/s | 151 kB     00:00    
(19/22): rpcbind-1.2.5-8.el8.x86_64.rpm                                                                     1.2 MB/s |  70 kB     00:00    
(20/22): lvm2-libs-2.03.11-5.el8.x86_64.rpm                                                                 3.1 MB/s | 1.1 MB     00:00    
(21/22): userspace-rcu-0.10.1-4.el8.x86_64.rpm                                                              474 kB/s | 101 kB     00:00    
(22/22): lvm2-2.03.11-5.el8.x86_64.rpm                                                                      3.3 MB/s | 1.6 MB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                       2.8 MB/s | 8.2 MB     00:02     
warning: /var/cache/dnf/centos-gluster8-ae72c2c38de8ee20/packages/glusterfs-8.4-1.el8.x86_64.rpm: Header V4 RSA/SHA1 Signature, key ID e451e5b5: NOKEY
CentOS-8 - Gluster 8                                                                                        1.0 MB/s | 1.0 kB     00:00    
Importing GPG key 0xE451E5B5:
 Userid     : "CentOS Storage SIG (http://wiki.centos.org/SpecialInterestGroup/Storage) <security@centos.org>"
 Fingerprint: 7412 9C0B 173B 071A 3775 951A D4A2 E50B E451 E5B5
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Storage
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                    1/1 
  Installing       : libgfxdr0-8.4-1.el8.x86_64                                                                                        1/22 
  Running scriptlet: libgfxdr0-8.4-1.el8.x86_64                                                                                        1/22 
  Installing       : libglusterfs0-8.4-1.el8.x86_64                                                                                    2/22 
  Running scriptlet: libglusterfs0-8.4-1.el8.x86_64                                                                                    2/22 
  Installing       : libgfrpc0-8.4-1.el8.x86_64                                                                                        3/22 
  Running scriptlet: libgfrpc0-8.4-1.el8.x86_64                                                                                        3/22 
  Installing       : libaio-0.3.112-1.el8.x86_64                                                                                       4/22 
  Installing       : glusterfs-client-xlators-8.4-1.el8.x86_64                                                                         5/22 
  Installing       : device-mapper-event-libs-8:1.02.175-5.el8.x86_64                                                                  6/22 
  Running scriptlet: glusterfs-8.4-1.el8.x86_64                                                                                        7/22 
  Installing       : glusterfs-8.4-1.el8.x86_64                                                                                        7/22 
  Running scriptlet: glusterfs-8.4-1.el8.x86_64                                                                                        7/22 
  Installing       : libglusterd0-8.4-1.el8.x86_64                                                                                     8/22 
  Running scriptlet: libglusterd0-8.4-1.el8.x86_64                                                                                     8/22 
  Installing       : glusterfs-cli-8.4-1.el8.x86_64                                                                                    9/22 
  Installing       : device-mapper-event-8:1.02.175-5.el8.x86_64                                                                      10/22 
  Running scriptlet: device-mapper-event-8:1.02.175-5.el8.x86_64                                                                      10/22 
  Installing       : lvm2-libs-8:2.03.11-5.el8.x86_64                                                                                 11/22 
  Installing       : libgfapi0-8.4-1.el8.x86_64                                                                                       12/22 
  Running scriptlet: libgfapi0-8.4-1.el8.x86_64                                                                                       12/22 
  Installing       : device-mapper-persistent-data-0.8.5-4.el8.x86_64                                                                 13/22 
  Installing       : lvm2-8:2.03.11-5.el8.x86_64                                                                                      14/22 
  Running scriptlet: lvm2-8:2.03.11-5.el8.x86_64                                                                                      14/22 
  Installing       : libgfchangelog0-8.4-1.el8.x86_64                                                                                 15/22 
  Running scriptlet: libgfchangelog0-8.4-1.el8.x86_64                                                                                 15/22 
  Installing       : userspace-rcu-0.10.1-4.el8.x86_64                                                                                16/22 
  Running scriptlet: userspace-rcu-0.10.1-4.el8.x86_64                                                                                16/22 
  Running scriptlet: rpcbind-1.2.5-8.el8.x86_64                                                                                       17/22 
  Installing       : rpcbind-1.2.5-8.el8.x86_64                                                                                       17/22 
  Running scriptlet: rpcbind-1.2.5-8.el8.x86_64                                                                                       17/22 
  Installing       : psmisc-23.1-5.el8.x86_64                                                                                         18/22 
  Installing       : attr-2.4.48-3.el8.x86_64                                                                                         19/22 
  Installing       : glusterfs-fuse-8.4-1.el8.x86_64                                                                                  20/22 
  Installing       : python3-pyxattr-0.5.3-18.el8.x86_64                                                                              21/22 
  Installing       : glusterfs-server-8.4-1.el8.x86_64                                                                                22/22 
  Running scriptlet: glusterfs-server-8.4-1.el8.x86_64                                                                                22/22 
  Verifying        : glusterfs-8.4-1.el8.x86_64                                                                                        1/22 
  Verifying        : glusterfs-cli-8.4-1.el8.x86_64                                                                                    2/22 
  Verifying        : glusterfs-client-xlators-8.4-1.el8.x86_64                                                                         3/22 
  Verifying        : glusterfs-fuse-8.4-1.el8.x86_64                                                                                   4/22 
  Verifying        : glusterfs-server-8.4-1.el8.x86_64                                                                                 5/22 
  Verifying        : libgfapi0-8.4-1.el8.x86_64                                                                                        6/22 
  Verifying        : libgfchangelog0-8.4-1.el8.x86_64                                                                                  7/22 
  Verifying        : libgfrpc0-8.4-1.el8.x86_64                                                                                        8/22 
  Verifying        : libgfxdr0-8.4-1.el8.x86_64                                                                                        9/22 
  Verifying        : libglusterd0-8.4-1.el8.x86_64                                                                                    10/22 
  Verifying        : libglusterfs0-8.4-1.el8.x86_64                                                                                   11/22 
  Verifying        : python3-pyxattr-0.5.3-18.el8.x86_64                                                                              12/22 
  Verifying        : attr-2.4.48-3.el8.x86_64                                                                                         13/22 
  Verifying        : device-mapper-event-8:1.02.175-5.el8.x86_64                                                                      14/22 
  Verifying        : device-mapper-event-libs-8:1.02.175-5.el8.x86_64                                                                 15/22 
  Verifying        : device-mapper-persistent-data-0.8.5-4.el8.x86_64                                                                 16/22 
  Verifying        : libaio-0.3.112-1.el8.x86_64                                                                                      17/22 
  Verifying        : lvm2-8:2.03.11-5.el8.x86_64                                                                                      18/22 
  Verifying        : lvm2-libs-8:2.03.11-5.el8.x86_64                                                                                 19/22 
  Verifying        : psmisc-23.1-5.el8.x86_64                                                                                         20/22 
  Verifying        : rpcbind-1.2.5-8.el8.x86_64                                                                                       21/22 
  Verifying        : userspace-rcu-0.10.1-4.el8.x86_64                                                                                22/22 

Installed:
  attr-2.4.48-3.el8.x86_64                                             device-mapper-event-8:1.02.175-5.el8.x86_64                         
  device-mapper-event-libs-8:1.02.175-5.el8.x86_64                     device-mapper-persistent-data-0.8.5-4.el8.x86_64                    
  glusterfs-8.4-1.el8.x86_64                                           glusterfs-cli-8.4-1.el8.x86_64                                      
  glusterfs-client-xlators-8.4-1.el8.x86_64                            glusterfs-fuse-8.4-1.el8.x86_64                                     
  glusterfs-server-8.4-1.el8.x86_64                                    libaio-0.3.112-1.el8.x86_64                                         
  libgfapi0-8.4-1.el8.x86_64                                           libgfchangelog0-8.4-1.el8.x86_64                                    
  libgfrpc0-8.4-1.el8.x86_64                                           libgfxdr0-8.4-1.el8.x86_64                                          
  libglusterd0-8.4-1.el8.x86_64                                        libglusterfs0-8.4-1.el8.x86_64                                      
  lvm2-8:2.03.11-5.el8.x86_64                                          lvm2-libs-8:2.03.11-5.el8.x86_64                                    
  psmisc-23.1-5.el8.x86_64                                             python3-pyxattr-0.5.3-18.el8.x86_64                                 
  rpcbind-1.2.5-8.el8.x86_64                                           userspace-rcu-0.10.1-4.el8.x86_64                                   

Complete!
[root@node1 ~]# systemctl start glusterd
[root@node1 ~]# systemctl status glusterd
● glusterd.service - GlusterFS, a clustered file-system server
   Loaded: loaded (/usr/lib/systemd/system/glusterd.service; enabled; vendor preset: enabled)
   Active: active (running) since Wed 2021-04-14 13:06:39 UTC; 3s ago
     Docs: man:glusterd(8)
  Process: 10151 ExecStart=/usr/sbin/glusterd -p /var/run/glusterd.pid --log-level $LOG_LEVEL $GLUSTERD_OPTIONS (code=exited, status=0/SUCC>
 Main PID: 10152 (glusterd)
    Tasks: 9 (limit: 11409)
   Memory: 5.6M
   CGroup: /system.slice/glusterd.service
           └─10152 /usr/sbin/glusterd -p /var/run/glusterd.pid --log-level INFO

Apr 14 13:06:39 node1 systemd[1]: Starting GlusterFS, a clustered file-system server...
Apr 14 13:06:39 node1 systemd[1]: Started GlusterFS, a clustered file-system server.
[root@node1 ~]# tail -n 3 /etc/hosts 
127.0.0.1 glnode1
192.168.0.30 glnode2
192.168.0.31 glnode3
[root@node1 ~]# ping glnode1
PING glnode1 (127.0.0.1) 56(84) bytes of data.
64 bytes from localhost (127.0.0.1): icmp_seq=1 ttl=64 time=0.058 ms
64 bytes from localhost (127.0.0.1): icmp_seq=2 ttl=64 time=0.050 ms
^C
--- glnode1 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1016ms
rtt min/avg/max/mdev = 0.050/0.054/0.058/0.004 ms
[root@node1 ~]# mkdir -p /mnt/storage/gluster/brick
[root@node1 ~]# firewall-cmd --permanent --new-zone=glusternodes
success
[root@node1 ~]# firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.20
success
[root@node1 ~]# firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.30
success
[root@node1 ~]# firewall-cmd --permanent --zone=glusternodes --add-source=192.168.0.31
success
[root@node1 ~]# firewall-cmd --permanent --zone=glusternodes --add-service=glusterfs
success
[root@node1 ~]# firewall-cmd --reload
success
[root@node1 ~]# firewall-cmd --zone=glusternodes --list-all
glusternodes (active)
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 192.168.0.20 192.168.0.30 192.168.0.31
  services: glusterfs
  ports: 
  protocols: 
  forward: no
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules:
[root@node1 ~]# gluster peer probe glnode2
peer probe: success
[root@node1 ~]# gluster peer probe glnode3
peer probe: success
[root@node1 ~]# gluster peer status
Number of Peers: 2

Hostname: glnode2
Uuid: ab63f0a0-0a72-4fcd-9f34-b88040d1a8e3
State: Peer in Cluster (Connected)

Hostname: glnode3
Uuid: 439ccd19-a95e-427c-ab72-6b65effcbe06
State: Peer in Cluster (Connected)
[root@node1 ~]# gluster volume create VOL1 replica 3 transport tcp glnode1:/mnt/storage/gluster/brick glnode2:/mnt/storage/gluster/brick glnode3:/mnt/storage/gluster/brick
volume create: VOL1: success: please start the volume to access data
[root@node1 ~]# gluster volume start VOL1
volume start: VOL1: success
[root@node1 ~]# gluster volume info VOL1
 
Volume Name: VOL1
Type: Replicate
Volume ID: 7da7bb05-2c9b-464b-b3f9-8940eeb5b0bb
Status: Started
Snapshot Count: 0
Number of Bricks: 1 x 3 = 3
Transport-type: tcp
Bricks:
Brick1: glnode1:/mnt/storage/gluster/brick
Brick2: glnode2:/mnt/storage/gluster/brick
Brick3: glnode3:/mnt/storage/gluster/brick
Options Reconfigured:
storage.fips-mode-rchecksum: on
transport.address-family: inet
nfs.disable: on
performance.client-io-threads: off
[root@node1 storage]# mkdir -p /mnt/VOL1
[root@node1 storage]# tail -n 1 /etc/fstab 
glnode1:/VOL1 /mnt/VOL1 glusterfs defaults,noatime,direct-io-mode=disable 0 0
[root@node1 storage]# mount /mnt/VOL1/
[root@node1 storage]# df -h
Filesystem      Size  Used Avail Use% Mounted on
devtmpfs        892M     0  892M   0% /dev
tmpfs           909M     0  909M   0% /dev/shm
tmpfs           909M  8.5M  901M   1% /run
tmpfs           909M     0  909M   0% /sys/fs/cgroup
/dev/sda1        95G  1.5G   89G   2% /
/dev/sda3       976M  148M  762M  17% /boot
tmpfs           182M     0  182M   0% /run/user/0
glnode1:/VOL1    95G  2.4G   89G   3% /mnt/VOL1
[root@node1 storage]# ls -altr /mnt/VOL1/
total 8
drwxr-xr-x. 3 root root 4096 Apr 14 13:31 .
drwxr-xr-x. 4 root root 4096 Apr 14 13:34 ..

Simple export of a ext4 directory with NFS Ganesha 3.5 server in CentOS 8 with SELinux enforcing

In fact, this article is a continuation of the previous NFS Ganesha article – Simple export of an ext4 directory with NFS Ganesha 3.5 server in CentOS 8 without SELinux because it has the same purpose to export a directory residing on an ext4 file system under CentOS 8 Stream, but this time the SELinux is enabled and it is in enforcing mode! There is a need for this additional article because the SELinux is not enabled in many user configurations (despite being wrong!) and the SELinux configuration may add complexity to the first article, which could lead to misleading thoughts. The previous article might be a little bit more detailed, so the reader could check it, too.
It’s worth mentioning the key points of NFS-Ganesha:

a user-mode file sharing server
supports NFS 3, 4.x and 9P
using plugins for different file systems
CentOS Storage Special Interest Group offers a file repository with NFS-Ganesha server
supports file systems like ext4, xfs, brtfs, zfs and more. There are sample configurations: https://github.com/phdeniel/nfs-ganesha/tree/master/src/config_samples
supports cluster and/or distributed file systems like GlusterFS, Ceph, GPFS, HPSS, Lustre
Current version 3.5 and it is included in the official SIG CentOS Storage Special Interest Group repository.

This article assumes the reader has a clean CentOS 8 Stream installation with SELinux in enforcing mode.

STEP 1) Install the repository and NFS-Ganesha software

NFS-Ganesha 3 packages are from the CentOS Storage SIG repository, which is a good repository and may be trusted.

dnf install -y centos-release-nfs-ganesha30
dnf install -y nfs-ganesha nfs-ganesha-vfs nfs-ganesha-selinux

STEP 2) Configuration for exporting a directory.

There are two files under /etc/ganesha/:

ganesha.conf
vfs.conf

ganesha.conf includes global configuration and NFS share configuration. Each export path begins with the keyword EXPORT followed by a block ebraced by brackets {}.
vfs.conf includes a simple example for the VFS plugin, but this configuration file is not used by the NFS Ganesha server. It is just a sample file.
Here is a simple configuration, which exports /mnt/storage with Read/Write permissions to a single IP. Just add at the end of the file /etc/ganesha/ganesha.conf contains:

 
EXPORT
{
        Export_Id = 2;
        Path = /mnt/storage1;
        Pseudo = /mnt/storage1;
        Protocols = 3,4;
        Access_Type = RW;
        Squash = None;
        FSAL
        {
                Name = VFS;
        }
        CLIENT
        {
                Clients = 192.168.0.12;
        }
}

STEP 3) Start the server and mount the exported directory. Configure the firewall.

Start the server, enable the service to start on boot and then configure the firewall to pass the NFS requests:

systemctl start nfs-ganesha
systemctl enable nfs-ganesha
firewall-cmd --permanent --zone=public --add-service=nfs
firewall-cmd --reload

Keep on reading!

Simple export of a ext4 directory with NFS Ganesha 3.5 server in CentOS 8 without SELinux

NFS Ganesha is a user-mode file sharing server, which supports NFS 3 and 4.x versions and 9P. NFS Ganesha has several interesting plugins that support exporting files from the cluster and distributed file systems like Ceph and Glusterfs Exporting a file system with NFS Ganesha is simple enough if you do not use SELinux or SELinux is in permissive mode!
This article is to show how to export a server’s directory using NFS protocol Just to note the NFS-Ganesha is tested and supports ext2/ext3/ext4, xfs, brtfs, zfs file systems as of version 3.5 (check the manual for xfs, brtfs and zfs exports – here are sample configurations for them https://github.com/phdeniel/nfs-ganesha/tree/master/src/config_samples). To be able to export a file directory the VFS Ganesha plugin is used. A clean install of minimal CentOS 8 Stream is used so the installation log may differ significantly from the user’s log but the user will see all the dependencies, which are required for this setup.

STEP 1) Install the repository and NFS-Ganesha software

NFS-Ganesha 3 packages are from the CentOS Storage SIG repository, which is a good repository and may be trusted.

dnf install -y centos-release-nfs-ganesha30
dnf install -y nfs-ganesha nfs-ganesha-vfs

STEP 2) Configuration for exporting a directory.

There are two files under /etc/ganesha/:

ganesha.conf
vfs.conf

 
EXPORT
{
        Export_Id = 2;
        Path = /mnt/storage1;
        Pseudo = /mnt/storage1;
        Protocols = 3,4;
        Access_Type = RW;
        Squash = None;
        FSAL
        {
                Name = VFS;
        }
        CLIENT
        {
                Clients = 192.168.0.12;
        }
}

STEP 3) Start the server and mount the exported directory. Configure the firewall.

Start the server, enable the service to start on boot and then configure the firewall to pass the NFS requests:

systemctl start nfs-ganesha
systemctl enable nfs-ganesha
firewall-cmd --permanent --zone=public --add-service=nfs
firewall-cmd --reload

Keep on reading!

Stopping the glusterfs volume releases disk sleep process hangs

A quick tip for GlusterFS volume. There are multiple possible reasons for a Linux process to hang in “Disk Sleep” state, which even the KILL -9 cannot interrupt:

a bug in GlusterFS
just bad options turn on online
other device relying on a GlusterFS, which is unavailable.

[17294588.184470] INFO: task gdisk:12505 blocked for more than 120 seconds.
[17294588.184538] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[17294588.184628] gdisk           D ffff8ce01fb9acc0     0 12505  26866 0x00000080
[17294588.184780] Call Trace:
[17294588.184844]  [<ffffffffbaed3d81>] ? __wake_up_common_lock+0x91/0xc0
[17294588.184910]  [<ffffffffbb585da9>] schedule+0x29/0x70
[17294588.184974]  [<ffffffffbb5838b1>] schedule_timeout+0x221/0x2d0
[17294588.185037]  [<ffffffffbaed3dc3>] ? __wake_up+0x13/0x20
[17294588.185102]  [<ffffffffc0a05d2e>] ? loop_make_request+0x12e/0x210 [loop]
[17294588.185169]  [<ffffffffbaf06d32>] ? ktime_get_ts64+0x52/0xf0
[17294588.185232]  [<ffffffffbb58549d>] io_schedule_timeout+0xad/0x130
[17294588.185304]  [<ffffffffbb5863dd>] wait_for_completion_io+0xfd/0x140
[17294588.185369]  [<ffffffffbaedb990>] ? wake_up_state+0x20/0x20
[17294588.185468]  [<ffffffffbb157e64>] blkdev_issue_flush+0xb4/0x110
[17294588.185533]  [<ffffffffbb08d335>] blkdev_fsync+0x35/0x50
[17294588.185598]  [<ffffffffbb082f57>] do_fsync+0x67/0xb0
[17294588.185671]  [<ffffffffbb083240>] SyS_fsync+0x10/0x20
[17294588.185734]  [<ffffffffbb592ed2>] system_call_fastpath+0x25/0x2a
[17294708.187598] INFO: task gdisk:12505 blocked for more than 120 seconds.
[17294708.187664] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[17294708.187753] gdisk           D ffff8ce01fb9acc0     0 12505  26866 0x00000080
[17294708.187905] Call Trace:
[17294708.187968]  [<ffffffffbaed3d81>] ? __wake_up_common_lock+0x91/0xc0
[17294708.188033]  [<ffffffffbb585da9>] schedule+0x29/0x70
[17294708.188096]  [<ffffffffbb5838b1>] schedule_timeout+0x221/0x2d0
[17294708.188159]  [<ffffffffbaed3dc3>] ? __wake_up+0x13/0x20
[17294708.188223]  [<ffffffffc0a05d2e>] ? loop_make_request+0x12e/0x210 [loop]
[17294708.188289]  [<ffffffffbaf06d32>] ? ktime_get_ts64+0x52/0xf0
[17294708.188352]  [<ffffffffbb58549d>] io_schedule_timeout+0xad/0x130
[17294708.188416]  [<ffffffffbb5863dd>] wait_for_completion_io+0xfd/0x140
[17294708.188480]  [<ffffffffbaedb990>] ? wake_up_state+0x20/0x20
[17294708.188545]  [<ffffffffbb157e64>] blkdev_issue_flush+0xb4/0x110
[17294708.188624]  [<ffffffffbb08d335>] blkdev_fsync+0x35/0x50
[17294708.188690]  [<ffffffffbb082f57>] do_fsync+0x67/0xb0
[17294708.188754]  [<ffffffffbb083240>] SyS_fsync+0x10/0x20
[17294708.188828]  [<ffffffffbb592ed2>] system_call_fastpath+0x25/0x2a

The above example of dmesg log shows the gdisk process stuck in “Disk Sleep” state, because of a loop device from a file on an unavailable GlusterFS volume! Kill -9 won’t help, the process will remain in this bad state and even a restart would be difficult to perform!

[root@srv1 ~]# gluster volume stop VOL2 
Stopping volume will make its data inaccessible. Do you want to continue? (y/n) y
volume stop: VOL2: success
[root@srv1 ~]# gluster volume start VOL2 
volume start: VOL2: success

The solution is to stop the GlusterFS Volume and all the blocked processes on bad devices such as above would be released. The processes will carry on executing or will end their execution after issuing a stop command to the volume. No problem to start the GlusterFS volume immediately after the stop!
NOTE: executing STOP command would affect all servers using this volume. The volume becomes inaccessible for all!

Repairing damaged backup GPT with gdisk

Problem with network shared storage could lead to a damaged file system or even GPT tables, so the gdisk may help in this case.
Here it is a nasty the error:

[root@srv1 ~]# kpartx /dev/loop0
Alternate GPT is invalid, using primary GPT.
loop0p1 : 0 83881984 /dev/loop0 2048
[root@srv1 ~]# parted /dev/loop0
GNU Parted 3.1
Using /dev/loop0
Welcome to GNU Parted! Type 'help' to view a list of commands.
(parted) p                                                                
Error: The backup GPT table is corrupt, but the primary appears OK, so that will be used.
OK/Cancel? OK                                                             
Model: Loopback device (loopback)
Disk /dev/loop0: 42.9GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  42.9GB  42.9GB               primary

(parted)

So parted reports the backup GPT is damaged, but how to fix it? The solution is to use gdisk and use write “w” command in it. gdisk also shows the exact error with the GPT table with “v” option:

[root@srv1 ~]# gdisk 
GPT fdisk (gdisk) version 0.8.10

Type device filename, or press <Enter> to exit: /dev/loop0
Warning! Main and backup partition tables differ! Use the 'c' and 'e' options
on the recovery & transformation menu to examine the two tables.

Warning! One or more CRCs don't match. You should repair the disk!

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: damaged

****************************************************************************
Caution: Found protective or hybrid MBR and corrupt GPT. Using GPT, but disk
verification and recovery are STRONGLY recommended.
****************************************************************************

Command (? for help): p
Disk /dev/loop0: 83886080 sectors, 40.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 7EDF123B-FBC4-4C09-B636-922BD165F862
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 83886046
Partitions will be aligned on 2048-sector boundaries
Total free space is 4029 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048        83884031   40.0 GiB    0700  primary

Command (? for help): v

Caution: The CRC for the backup partition table is invalid. This table may
be corrupt. This program will automatically create a new backup partition
table when you save your partitions.

Identified 1 problems!

Command (? for help): p
Disk /dev/loop0: 83886080 sectors, 40.0 GiB
Logical sector size: 512 bytes
Disk identifier (GUID): 7EDF123B-FBC4-4C09-B636-922BD165F862
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 83886046
Partitions will be aligned on 2048-sector boundaries
Total free space is 4029 sectors (2.0 MiB)

Number  Start (sector)    End (sector)  Size       Code  Name
   1            2048        83884031   40.0 GiB    0700  primary

Command (? for help): w

Final checks complete. About to write GPT data. THIS WILL OVERWRITE EXISTING
PARTITIONS!!

Do you want to proceed? (Y/N): Y
OK; writing new GUID partition table (GPT) to /dev/loop0.

Warning: The kernel is still using the old partition table.
The new table will be used at the next reboot.
The operation has completed successfully.

And the GPT backup in this loop device is fixed. Executing parted again reports no problems:

[root@srv1 ~]# parted /dev/loop0 print
Model: Loopback device (loopback)
Disk /dev/loop0: 42.9GB
Sector size (logical/physical): 512B/512B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system  Name     Flags
 1      1049kB  42.9GB  42.9GB               primary

Verify also reports nor error. More options are available:

[root@srv1 ~]# gdisk /dev/loop0
GPT fdisk (gdisk) version 0.8.10

Partition table scan:
  MBR: protective
  BSD: not present
  APM: not present
  GPT: present

Found valid GPT with protective MBR; using GPT.

Command (? for help): h
b  back up GPT data to a file
c  change a partition's name
d  delete a partition
i  show detailed information on a partition
l  list known partition types
n  add a new partition
o  create a new empty GUID partition table (GPT)
p  print the partition table
q  quit without saving changes
r  recovery and transformation options (experts only)
s  sort partitions
t  change a partition's type code
v  verify disk
w  write table to disk and exit
x  extra functionality (experts only)
?  print this menu

Command (? for help): v

No problems found. 4029 free sectors (2.0 MiB) available in 2
segments, the largest of which is 2015 (1007.5 KiB) in size.

Command (? for help): q

Force losetup detach of local file after it became unavailable

Mounting a file as loop devices is simple enough operation! But what if the file just disappears because of the network storage got unreachable?

In our case the shared network storage had got unavailable and the ext4 file system of the loop device got read-only! Unfortunately after the reset of the shared network storage and mounting it in the same place, the loop device (/dev/loop0) still maintained the old file descriptor to the unavailable file. And the losetup could not detach the device, at all.

root@srv ~# losetup -d /dev/loop0
root@srv ~# losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop0         0      0         0  0 /mnt/storage4/servers/test_raw_image.img
[root@lsrv1 ~]# kpartx -df /dev/loop0
read error, sector 0
read error, sector 1
read error, sector 29

The above command could not detach the device. Unfortunately, the losetup does not have the force detach, so the server ended with blocked loop0 device pointing to unavailable file. kpartx does not work, neither.

The solution is to use dmsetup!

root@srv ~# dmsetup remove /dev/mapper/loop0p1
root@srv ~# losetup -l
root@srv ~#

And there is no loop device any more. The wrong pointing loop device has been removed successfully! Now the user can use the loop0 for another device and in many cases, this helps to umount the filesystem!

MySQL slave upgrade: Slave failed to initialize relay log info structure from the repository

MySQL slave after upgrade from 5.6.x to 5.7.x may throw the following error:

mysql> START SLAVE;
ERROR 1872 (HY000): Slave failed to initialize relay log info structure from the repository

The best solution for this error is to:

Master server – mysqldump the database with –master-data=1 –single-transaction
On the slave server issue command “RESET SLAVE;”
On the slave server import the dump sql file and issue “CHANGE MASTER” command with the meta data written in the sql dump
On the slave server issue START SLAVE to start the replication.

Here is an a real world example:
First, mysqldump in the master with

root@master ~ # mysqldump --master-data=1 --single-transaction mydb > /root/mydb.sql
root@master ~ # grep "CHANGE MASTER" media.sql 
CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.023283', MASTER_LOG_POS=537774724;

And then copy the dump file to the slave server and import it and issue several specific slave commands:

root@slave ~ # mysql < /root/mydb.sql
root@slave ~ # mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 23
Server version: 5.7.31-log Gentoo Linux mysql-5.7.31

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> reset slave;
Query OK, 0 rows affected (0.01 sec)
mysql> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.023283', MASTER_LOG_POS=537774724;
Query OK, 0 rows affected (0.00 sec)
mysql> START SLAVE;
Query OK, 0 rows affected (0.00 sec)

mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Queueing master event to the relay log
                  Master_Host: 10.10.10.10
                  Master_User: ruser
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.023283
          Read_Master_Log_Pos: 641769286
               Relay_Log_File: slave-relay-bin.000002
                Relay_Log_Pos: 90874706
        Relay_Master_Log_File: mysql-bin.023283
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: mydb.%
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 628649113
              Relay_Log_Space: 103995088
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 2395
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 101
                  Master_UUID: cd1bcebb-cc27-11e8-90c9-801844f2c4d8
             Master_Info_File: /mnt/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Reading event from the relay log
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
1 row in set (0.00 sec)

The replication is advancing. It is 2395 seconds behind the master.
Keep on reading!