Force losetup detach of local file after it became unavailable

Mounting a file as loop devices is simple enough operation! But what if the file just disappears because of the network storage got unreachable?

In our case the shared network storage had got unavailable and the ext4 file system of the loop device got read-only! Unfortunately after the reset of the shared network storage and mounting it in the same place, the loop device (/dev/loop0) still maintained the old file descriptor to the unavailable file. And the losetup could not detach the device, at all.

root@srv ~# losetup -d /dev/loop0
root@srv ~# losetup -l
NAME       SIZELIMIT OFFSET AUTOCLEAR RO BACK-FILE
/dev/loop0         0      0         0  0 /mnt/storage4/servers/test_raw_image.img
[root@lsrv1 ~]# kpartx -df /dev/loop0
read error, sector 0
read error, sector 1
read error, sector 29

The above command could not detach the device. Unfortunately, the losetup does not have the force detach, so the server ended with blocked loop0 device pointing to unavailable file. kpartx does not work, neither.

The solution is to use dmsetup!

root@srv ~# dmsetup remove /dev/mapper/loop0p1
root@srv ~# losetup -l
root@srv ~#

And there is no loop device any more. The wrong pointing loop device has been removed successfully! Now the user can use the loop0 for another device and in many cases, this helps to umount the filesystem!

MySQL slave upgrade: Slave failed to initialize relay log info structure from the repository

MySQL slave after upgrade from 5.6.x to 5.7.x may throw the following error:

mysql> START SLAVE;
ERROR 1872 (HY000): Slave failed to initialize relay log info structure from the repository

The best solution for this error is to:

  • Master server – mysqldump the database with –master-data=1 –single-transaction
  • On the slave server issue command “RESET SLAVE;”
  • On the slave server import the dump sql file and issue “CHANGE MASTER” command with the meta data written in the sql dump
  • On the slave server issue START SLAVE to start the replication.

Here is an a real world example:
First, mysqldump in the master with

root@master ~ # mysqldump --master-data=1 --single-transaction mydb > /root/mydb.sql
root@master ~ # grep "CHANGE MASTER" media.sql 
CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.023283', MASTER_LOG_POS=537774724;

And then copy the dump file to the slave server and import it and issue several specific slave commands:

root@slave ~ # mysql < /root/mydb.sql
root@slave ~ # mysql
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 23
Server version: 5.7.31-log Gentoo Linux mysql-5.7.31

Copyright (c) 2000, 2020, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> reset slave;
Query OK, 0 rows affected (0.01 sec)
mysql> CHANGE MASTER TO MASTER_LOG_FILE='mysql-bin.023283', MASTER_LOG_POS=537774724;
Query OK, 0 rows affected (0.00 sec)
mysql> START SLAVE;
Query OK, 0 rows affected (0.00 sec)

mysql> show slave status\G;
*************************** 1. row ***************************
               Slave_IO_State: Queueing master event to the relay log
                  Master_Host: 10.10.10.10
                  Master_User: ruser
                  Master_Port: 3306
                Connect_Retry: 60
              Master_Log_File: mysql-bin.023283
          Read_Master_Log_Pos: 641769286
               Relay_Log_File: slave-relay-bin.000002
                Relay_Log_Pos: 90874706
        Relay_Master_Log_File: mysql-bin.023283
             Slave_IO_Running: Yes
            Slave_SQL_Running: Yes
              Replicate_Do_DB: 
          Replicate_Ignore_DB: 
           Replicate_Do_Table: 
       Replicate_Ignore_Table: 
      Replicate_Wild_Do_Table: mydb.%
  Replicate_Wild_Ignore_Table: 
                   Last_Errno: 0
                   Last_Error: 
                 Skip_Counter: 0
          Exec_Master_Log_Pos: 628649113
              Relay_Log_Space: 103995088
              Until_Condition: None
               Until_Log_File: 
                Until_Log_Pos: 0
           Master_SSL_Allowed: No
           Master_SSL_CA_File: 
           Master_SSL_CA_Path: 
              Master_SSL_Cert: 
            Master_SSL_Cipher: 
               Master_SSL_Key: 
        Seconds_Behind_Master: 2395
Master_SSL_Verify_Server_Cert: No
                Last_IO_Errno: 0
                Last_IO_Error: 
               Last_SQL_Errno: 0
               Last_SQL_Error: 
  Replicate_Ignore_Server_Ids: 
             Master_Server_Id: 101
                  Master_UUID: cd1bcebb-cc27-11e8-90c9-801844f2c4d8
             Master_Info_File: /mnt/mysql/master.info
                    SQL_Delay: 0
          SQL_Remaining_Delay: NULL
      Slave_SQL_Running_State: Reading event from the relay log
           Master_Retry_Count: 86400
                  Master_Bind: 
      Last_IO_Error_Timestamp: 
     Last_SQL_Error_Timestamp: 
               Master_SSL_Crl: 
           Master_SSL_Crlpath: 
           Retrieved_Gtid_Set: 
            Executed_Gtid_Set: 
                Auto_Position: 0
         Replicate_Rewrite_DB: 
                 Channel_Name: 
           Master_TLS_Version: 
1 row in set (0.00 sec)

The replication is advancing. It is 2395 seconds behind the master.
Keep on reading!

libelf was not found in the pkg-config search path

Building from source under CentOS the user may stumble on some compilation errors and most of them are for missing -devel packages. Here is such example with not so easy to find the name of a missing library:

[/tmp/netdata-libbpf-El77Ld/libbpf-0.0.9_netdata-1/src]# env CFLAGS=-fPIC CXXFLAGS= LDFLAGS= BUILD_STATIC_ONLY=y OBJDIR=build DESTDIR=.. make install 
Package libelf was not found in the pkg-config search path.
Perhaps you should add the directory containing `libelf.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libelf' found
mkdir -p build/staticobjs
cc -I. -I../include -I../include/uapi -DCOMPAT_NEED_REALLOCARRAY -fPIC -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c bpf.c -o build/staticobjs/bpf.o
cc -I. -I../include -I../include/uapi -DCOMPAT_NEED_REALLOCARRAY -fPIC -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64   -c btf.c -o build/staticobjs/btf.o
btf.c:17:18: fatal error: gelf.h: No such file or directory
 #include <gelf.h>
                  ^
compilation terminated.
make: *** [build/staticobjs/btf.o] Error 1
 FAILED   

The missing development library file is with the name: elfutils-libelf-devel. Installing the package with yum or dnf will resolve the above error:

yum install -y elfutils-libelf-devel

Or for CentOS 8 and newer Fedora versions:

dnf install -y elfutils-libelf-devel

VBoxManage: error: Failed to initialize COM! NS_ERROR_FILE_TARGET_DOES_NOT_EXIST (0x80520006)

What an error! And the VirtualBox stopped loading anymore! This error:

myuser@srv ~ $ VBoxManage list vms
VBoxManage: error: Failed to initialize COM! (hrc=NS_ERROR_FILE_TARGET_DOES_NOT_EXIST)

and here with the GIU, which offers a little bit more information about the error ID:
This error occurs despite the successfully loaded VirtualBox kernel modules!

main menu
VBoxManage: error: Failed to initialize COM! NS_ERROR_FILE_TARGET_DOES_NOT_EXIST (0x80520006)

The two errors are the same and hint there is a file missing (or files?)? In our case, after upgrading from virtualbox-bin (a binary package) to a virtualbox (a package, which actually builds the VirtualBox on the system) under Gentoo, but apparently, this error could occur to everyone, who tries to build yourself the VirtualBox bundle.

The solution is simple: just DO NOT CHANGE the installation path of the Virtualbox software, which by default is /opt/VirtualBox.

The path is hardcoded in the sources and cannot be changed! At present version 6.1.16, we could not find a build option to change it! Of course, the linking /opt/VirtualBox would do the trick to move the installation physically away from the /opt, but the path must be valid /opt/VirtualBox.

In Gentoo, the emerge default installation goes to “/usr/lib64/virtualbox“, so the maintainer of the package changed the path and Virtualbox stopped working! Or a user build it and install it in any other location should link the /opt/VirtualBox to the installation directory. For example, in Gentoo, the fix will be:

root@srv ~ $ ln -s /usr/lib64/virtualbox /opt/VirtualBox
root@srv ~ $ exit
myuser@srv ~ $ VBoxManage list vms
"gentoo_raw" {55346caf-04db-4d88-831a-111111111111}
"diskless" {44346caf-c952-5555-b8a3-111111111111}
"diskless-linux" {44346caf-424f-487f-ae8d-111111111111}
"centos7-netinstall" {44346caf-4e86-441b-8d1e-111111111111}

A simple link would bring back Virtualbox to live.

Bonus

Probably, there is a configuration file “xpti.dat” in two or more locations: ~/.config/VirtualBox/xpti.dat and ~/.VirtualBox/xpti.dat, which is generated on every start of a VirtualBox. In the file there are configuration lines like:
Keep on reading!

removing the default kernel in CentOS 8 – remove elrepo kernel

Removing the default kernel aka the loaded kernel in CentOS 8 maybe challenging because the package is protected and cannot be removed by the yum or dnf.
Here is the case: an elrepo kernel-ml loaded and the dnf prints it cannot remove the package, because it is protected:

[root@srv ~]# dnf remove kernel-ml kernel-ml-core kernel-ml-modules
Error: 
 Problem: The operation would result in removing the following protected packages: kernel-ml-core
(try to add '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
[root@srv ~]# uname -a
Linux srv.localhost 5.10.4-1.el8.elrepo.x86_64 #1 SMP Tue Dec 29 11:04:23 EST 2020 x86_64 x86_64 x86_64 GNU/Linux
[root@srv ~]# grubby --default-kernel
/boot/vmlinuz-5.10.4-1.el8.elrepo.x86_64

The system is booted up with the kernel we are trying to remove, which is impossible.

The solution is to set a new default kernel and load it. Then dnf will be able to remove the first kernel.

For CentOS 7, just use the yum instead of dnf command.
Using grubby is really easy and straightforward:

STEP 1) List all installed and available to boot kernels

[root@srv ~]# grubby --info=ALL |grep ^kernel
kernel="/boot/vmlinuz-5.10.4-1.el8.elrepo.x86_64"
kernel="/boot/vmlinuz-4.18.0-259.el8.x86_64"
kernel="/boot/vmlinuz-4.18.0-257.el8.x86_64"
kernel="/boot/vmlinuz-0-rescue-45e12f0814fd4947b99cbdcb88950361"

STEP 2) Select the kernel to load the next time

[root@srv ~]# grubby --set-default "/boot/vmlinuz-4.18.0-259.el8.x86_64"
The default is /boot/loader/entries/45e12f0814fd4947b99cbdcb88950361-4.18.0-259.el8.x86_64.conf with index 1 and kernel /boot/vmlinuz-4.18.0-259.el8.x86_64

Keep on reading!

Gentoo emerge GO lang failed – atomic_amd64x.go: too many errors

Upgrading GO lang under Gentoo maybe a little bit tricky. The upgrading go lang from 1.13.7 to 1.15.5 failed with strange error:

# runtime/internal/atomic
/usr/lib/go/src/runtime/internal/atomic/atomic_amd64x.go:18:6: Load redeclared in this block
        previous declaration at /usr/lib/go/src/runtime/internal/atomic/atomic_amd64.go:16:24
/usr/lib/go/src/runtime/internal/atomic/atomic_amd64x.go:24:6: Loadp redeclared in this block
        previous declaration at /usr/lib/go/src/runtime/internal/atomic/atomic_amd64.go:22:32
/usr/lib/go/src/runtime/internal/atomic/atomic_amd64x.go:30:6: Load64 redeclared in this block
        previous declaration at /usr/lib/go/src/runtime/internal/atomic/atomic_amd64.go:28:26
/usr/lib/go/src/runtime/internal/atomic/atomic_amd64x.go:36:6: LoadAcq redeclared in this block
        previous declaration at /usr/lib/go/src/runtime/internal/atomic/atomic_amd64.go:34:27
/usr/lib/go/src/runtime/internal/atomic/atomic_amd64x.go:41:6: Xadd redeclared in this block
        previous declaration at /usr/lib/go/src/runtime/internal/atomic/atomic_amd64.go:39:37
/usr/lib/go/src/runtime/internal/atomic/atomic_amd64x.go:44:6: Xadd64 redeclared in this block
        previous declaration at /usr/lib/go/src/runtime/internal/atomic/atomic_amd64.go:42:39
/usr/lib/go/src/runtime/internal/atomic/atomic_amd64x.go:47:6: Xadduintptr redeclared in this block
        previous declaration at /usr/lib/go/src/runtime/internal/atomic/atomic_amd64.go:45:47
/usr/lib/go/src/runtime/internal/atomic/atomic_amd64x.go:50:6: Xchg redeclared in this block
        previous declaration at /usr/lib/go/src/runtime/internal/atomic/atomic_amd64.go:48:36
/usr/lib/go/src/runtime/internal/atomic/atomic_amd64x.go:53:6: Xchg64 redeclared in this block
        previous declaration at /usr/lib/go/src/runtime/internal/atomic/atomic_amd64.go:51:38
/usr/lib/go/src/runtime/internal/atomic/atomic_amd64x.go:56:6: Xchguintptr redeclared in this block
        previous declaration at /usr/lib/go/src/runtime/internal/atomic/atomic_amd64.go:54:45
/usr/lib/go/src/runtime/internal/atomic/atomic_amd64x.go:56:6: too many errors

Googling a little bit and it appeared there might be a conflict with the old version in the same directory. Deleting the temporary build directory didn’t help…

Removing the GO lang package by unmerge command and then emerging the newest GO lang package is successful.

So the solution is to unmerge it and then immediately emerge the newest version with:

emerge -vC dev-lang/go
emerge -v dev-lang/go

The whole Gentoo output of the failed emerge command

Keep on reading!

Booting network installation from ipxe disk using IPMI KVM

There is a project for extended PXE Boot features https://ipxe.org/. This article is not for describing what this project may offer, but to show how to boot any Linux distribution (in fact, Windows 10, too) network installation wizard using the virtual CD/DVD of an IPMI KVM, DELL’s DRAC, HP iLO, IBM RSA/IMM and in general, KVM over IP.
Using the iPXE CD bootable disk mounted in the virtual CD/DVD of the server’s remote console (IPMI KVM and so on) will allow:

  • Booting from a CD/DVD with only a 1M size.
  • Extends the PXE features of the server’s network card.
  • Manual set IP address, i.e. not relying on DHCP server. In addition of the DHCP feature, but DHCP feature requires DHCP server, which is not always the case.
  • Load a Linux kernel and initramfs from a URL.
  • Boot a Linux live or installation CD/DVD from an URL. The server could load the instllation wizard from an official mirror in the Internet.
  • Manual install – boot from 1M CD and continue with multi gigabyte installation from an URL. For comparision the CentOS 8 network installation disk is more than 600M versus 1M iPXE CD disk. Booting directly from a 600M CentOS 8 network installation disk is unstable and really slow when the disk is mounted in user’s KVM. And not alwyas is possible to mount a disk next to the server location (or in the same co-location).
  • Automated install – simple unattended installation with kickstart files without the need of speacial features of the dedicated service provider.
  • No software installation or code writting needed.

This article uses the iPXE CD to boot and manually set an IP and then load the Linux kernel and initramfs of the CentOS 8 installation disk using an official URL mirror on the Internet. All types of server’s KVM, which supports CD/DVD virtual device, can be used.

Just 1 Mbytes of CD/DVD is required to boot an installation of a (connected to the Internet) server/machine.

Here are the steps and correct (all lines are tested) command lines to boot an installation wizard. The server is a SUPERMICRO server with IPMI KVM for remote management.
The iPXE ISO file is located here http://boot.ipxe.org/ipxe.iso

SCREENSHOT 1) Open the IPMI KVM and click on “Virtual Storage” menu to open the image mount dialog.

main menu
Virtual Storage menu

Keep on reading!

gentoo network interface with hyphen in the name

Using the OpenRC (i.e. init system) and network names with special symbols like hyphen in the name may lead to errors of “command not found” and “No such file or directory

The hyphen in the network interface name must be replaced in the configuration file with an underline and the init name file should be with the hyphen.

Proper configuration for network interface name with hyphen mv-eth0

  • In the configuration file /etc/conf.d/net:
    config_mv_eth0="
    192.168.0.202/24
    "
    routes_mv_eth0="
    default via 192.168.0.1
    "
    
  • The network interface init file is with hyphen:
    root@srv /etc/init.d # ln -s net.lo net.mv-eth0
    

And starting the network is successful:

root@srv ~ # /etc/init.d/net.mv-eth0 start
 * Caching service dependencies ...                                                                                                                                                     [ ok ]
 * Bringing up interface mv-eth0
 *   Caching network module dependencies
 *   192.168.0.202/24 ...                                                                                                                                                               [ ok ]
 *   Adding routes
 *     default via 192.168.0.1 ...                                                                                                                                                      [ ok ]
 *   Waiting for tentative IPv6 addresses to complete DAD (5 seconds) ..

Virtualization software may include to the network interface name not so typical alphabets. For example, systemd-nspawn will give name to the guest’s macvlan network with mv-{host_network_name} and iv-{host_network_name} for ipvlan.

Wrong configuration with a hyphen in the network interface name.

The configuration file /etc/conf.d/net:

config_mv-eth0="
192.168.0.202/24
"
routes_mv-eth0="
default via 192.168.0.1
"

Starting the network with such configuration will result in multiple errors:

root@srv ~ # /etc/init.d # /etc/init.d/net.mv-eth0 start
 * Caching service dependencies ...
/etc/init.d/../conf.d/net: line 3: config_mv-eth0=
192.168.0.202/24
: No such file or directory
/etc/init.d/../conf.d/net: line 6: $'routes_mv-eth0=\ndefault via 192.168.0.1\n': command not found
/etc/init.d/../conf.d/net: line 3: config_mv-eth0=
192.168.0.202/24
: No such file or directory
/etc/init.d/../conf.d/net: line 6: $'routes_mv-eth0=\ndefault via 192.168.0.1\n': command not found                                                                                  [ ok ]
/etc/init.d/../conf.d/net: line 3: config_mv-eth0=
192.168.0.202/24
: No such file or directory
/etc/init.d/../conf.d/net: line 6: $'routes_mv-eth0=\ndefault via 192.168.0.1\n': command not found
 * net.mv-eth0: error loading /etc/init.d/../conf.d/net
 * ERROR: net.mv-eth0 failed to start

xdg and autostart in Linux X server regardless the desktop environment

There is a tool xdg, which manages application integration with the different GUI Desktops in the Linux world. One of the features it offers is to autostart an application when the X window system starts and it is perfectly normal to have a bunch of running programs that cannot be found in the Windowing manager settings like KDE System Settings -> Autostart, GNOME Tweak tool and Autostart and so on.

xdg offers autostart of Linux appilcations mainly Desktop when the GUI windowing system starts

There two main paths to look for entries to autostart:

  1. /etc/xdg/autostart – called system-wide and most of the application will place files when they are installed.
  2. [user’s home]/.config/autostart – user’s applications to start when the user logs in .

With xdg autostart feature the user can explain himself why the Windowing systems like KDE or GNOME start tens of applications (not exactly related to the base GUI windowing system).

There is a security problem here, which is sometimes installing a package will place an autostart file there because the maintainer decided it is important but the package might be just a dependency and the next time the user logs in unwanted program might execute and open ports!

For example, Rygel is an open-source UPnP/DLNA MediaServer and it might be installed as a dependency but it places an autostart file, which starts a UPnP/DLNA server and exports the /home/[user’s directory]/Videos, /home/[user’s directory]/Pictures and more to the local network. Another example is with the GNOME index system tracker and the tracker-store, which may easily eat the RAM, disk, CPU, battery on a system without GNOME but with a different GUI!

Here is what a typical Ubuntu 18.04 system might autostart

Keep on reading!

starting Hashicorp vault in server mode under docker container

Running Hashicorp vault in development mode is really easy, but starting the vault in server mode under a docker container may have some changes described in this article.

There are several simple steps, which is hard to get in one place, to run a Hashicorp vault in server mode (under docker):

  1. Prepare the directories to map in the docker. The data in the directories will be safe and won’t be deleted if the container is deleted.
  2. Prepare an initial base configuration to start the server. Without it, the server won’t startup. Even it is really simple.
  3. Start the Hashicorp vault process in a docker container.
  4. Initiliza the vault. During this step, the server will generate the database backend storage (files or in-memory or cloud backends) and 5 unseal keys and an administrative root token will be generated. To manage the vault an administrative user is required.
  5. Unseal the vault. Unencrypt the database backend to use the service with at least three commands and three different unseal keys generated during the initialization step.
  6. Login with the administrative user and enable vault engine to store values (or generate tokens, passwords, and so on). The example here enables the secret engine to store key:value backend. Check out the secrets engines – https://www.vaultproject.io/docs/secrets

STEP 1) Summary of the mapped directories in the docker

Three directories are preserved:

  • /vault/config – contains configuration files in HCL or JSON format.
  • /vault/data – the place, where the encrypted database files will be kept only if a similar storage engine is used like “file” or “raft” storages. More information here – https://www.vaultproject.io/docs/configuration/storage
  • /vault/log – writing persistent audit logs. This feature should be enabled explicitly in the configuration.

The base directory used is /srv/vault/. And the three direcotries are created as follow and will be mapped in the docker container:

mkdir -p /srv/vault/config /srv/vault/data /srv/vault/log

The server’s directory /srv/vault/config will be mapped in docker’s directory /vault/config and the other two, too.

STEP 2) Initial base configuration

The initial configuration file is placed in /vault/config/config.hcl and is using HCL format – https://github.com/hashicorp/hcl. The initial configuration is minimal:

storage "raft" {
  path    = "/vault/data"
  node_id = "node1"
}

listener "tcp" {
  address     = "127.0.0.1:8200"
  tls_disable = 1
}

disable_mlock = true

api_addr = "http://127.0.0.1:8200"
cluster_addr = "https://127.0.0.1:8201"
ui = true

Place the file in /srv/vault/config/config.hcl

STEP 3) Start the Hashicorp vault server in docker

Mapping the three directories.

root@srv ~ # docker run --cap-add=IPC_LOCK -v /srv/vault/config:/vault/config -v /srv/vault/data:/vault/data -v /srv/vault/logs:/vault/logs --name=srv-vault vault server
==> Vault server configuration:

             Api Address: http://127.0.0.1:8200
                     Cgo: disabled
         Cluster Address: https://127.0.0.1:8201
              Go Version: go1.14.7
              Listener 1: tcp (addr: "127.0.0.1:8200", cluster address: "127.0.0.1:8201", max_request_duration: "1m30s", max_request_size: "33554432", tls: "disabled")
               Log Level: info
                   Mlock: supported: true, enabled: false
           Recovery Mode: false
                 Storage: raft (HA available)
                 Version: Vault v1.5.3
             Version Sha: 9fcd81405feb320390b9d71e15a691c3bc1daeef

==> Vault server started! Log data will stream in below:

Keep on reading!