Remove disk (all partitions) from software RAID1 with mdadm and change layout of the disk

The following article is to show how to remove healthy partitions from software RAID1 devices to change the layout of the disk and then add them back to the array.
The mdadm is the tool to manipulate the software RAID devices under Linux and it is part of all Linux distributions (some don’t install it by default so it may need to be installed).

Software RAID layout

[root@srv ~]# cat /proc/mdstat 
Personalities : [raid1] 
md125 : active raid1 sda4[1] sdb3[0]
      1047552 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md126 : active raid1 sdb2[0] sda3[1]
      32867328 blocks super 1.2 [2/2] [UU]
md127 : active raid1 sda2[1] sdb1[0]
      52427776 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>

STEP 1) Make the partitions faulty.

The partitions cannot be removed if they are not faulty.

[root@srv ~]# mdadm --fail /dev/md125 /dev/sdb3
mdadm: set /dev/sdb3 faulty in /dev/md125
[root@srv ~]# mdadm --fail /dev/md126 /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md126
[root@srv ~]# mdadm --fail /dev/md127 /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md127

Keep on reading!

bonding – write error – device or resource busy – operation not permitted

Recently, there was a little bit of confusion when following the article about activating network bonding without ifenslave – How to enable Linux bonding without ifenslave. At first, there were couple of errors:

livecd ~ # echo balance-alb > /sys/class/net/bond0/bonding/mode
-bash: echo: write error: Device or resource busy
livecd ~ # echo "+enp129s0f0" > /sys/class/net/bond0/bonding/slaves
-bash: echo: write error: Operation not permitted

Or similar error when changing the bonding mode:

livecd ~ # echo 4 > /sys/class/net/bond0/bonding/mode
-bash: echo: write error: Directory not empty
livecd ~ # echo 802.3ad > /sys/class/net/bond0/bonding/mode
-bash: echo: write error: Directory not empty

The server just booted in rescue live cd and there is no active network configuration:

SCREENSHOT 1) Apparently, the /sys/class/net/bond0/bonding/mode and /sys/class/net/bond0/bonding/slaves are in read only state.

No writes means no new configuration could be installed and the bonding cannot be configured (reconfigured).

main menu
device or resource busy – operation not permitted

Bonding mode could be changed only when the bonding device is in DOWN state.

Network interfaces could be added to the boding device only if they were in DOWN state, too.

In addition, changing bonding mode could only happen if there were no network interfaces added to the bonding interface.

Keep on reading!

Install aptly under Ubuntu 18 LTS with nginx serving the packages and the first steps

This article is how to install aptly software, which offers easy Debian repository management.
First, few words for aptly and what tasks are really simple to do:

  • Mirror an existing (remote) repository. Make a local copy of Debian or Ubuntu repostories for all your internal infrastructure.
  • Create your own repositories
  • Create snapshots of repositories and mirrors.
  • Merge repositories
  • Make diff between repositories (in fact snapshots of repositories, but you may make a mirror of an repository and then make a snapshot and then make a diff with some other snapshot to see the changes between the different repositories or the time the snapshots are made).
  • Remove or add individual packages from official mirrored repositories.
  • Use api calls to manage the repositories. HTTP REST API is still in development, but a big part of it works.

For more information you may visit the official documentation page –

We are going to install the aptly and despite it could be used to serve the repository files we will use the Nginx web server for this work. Nginx is a more fast and reliable web server with easy installation of SSL certificates for our repositories.
The aptly is included in official Ubuntu repositories in the component universe, but at present, it is 2 to 3 versions behind the stable one from the aptly site, so we are going to use their repository to install aptly. Still, if you do not want to use
Keep on reading!

How to compile xmr-stak (2.10) under CentOS 7 for CPU mining cryptocurrencies in September 2019

A time to refresh our old article on how to compile xmr-stak for CPU mining with the new version and this time a new GNU GCC version (version 8.3, the last article we used 7.x – How to compile xmr-stak (2.4.5) under CentOS 7 for CPU mining cryptocurrencies). Always use the latest available GNU GCC packages because the latest version of GNU GCC could add some optimizations to the binary compiled code and you may have a CPU miner with better performance!
Thanks to xmr-stak we can have one application capable of mining many different cryptocurrencies based on different algorithms. XMR-STAK is GPU and CPU miner and here we present only the CPU ability under CentOS 7 using our AMD Threadripper 1950X.
The software in this article:

  • CentOS 7 – CentOS Linux release 7.6.1810 (Core)
  • GNU GCC – gcc version 8.3.1 20190311 (Red Hat 8.3.1-3) (GCC)
  • XMR-STAK – 2.10.7

As said many times working with crypto-currency it is mandatory to do the things yourself – do not trust any binary made by someone on the Internet. It is easy to build your miner yourself with the code from the official repository!

So here are the steps to build the XMR-STAK for CPU mining:

STEP 1) Update your system and install the following dependencies

Always start with update command and then install the dependencies in order first install all the new repositories and then the dependency binaries.

sudo yum update -y
sudo yum install -y centos-release-scl epel-release
sudo yum install -y cmake3 devtoolset-8-gcc* hwloc-devel libmicrohttpd-devel openssl openssl-devel make git screen wget

We are going to use GNU GCC 8 to build the XMR-STAK. More on the subject of how to install GNU GCC 8 and what is “devtoolset” here – How to install GNU GCC 8 on CentOS 7.
Keep on reading!

root cannot delete, move or change a file – Operation not permitted or Permission denied – immutable attribute

If you are the root user and some file (files or directories) cannot be deleted, removed, renamed or changed you probably deal with the immutable attribute set on (by a colleague of yours – installation setups tend to not set such attributes).

Here is what it looks like to have such a file

root@srv.remote /etc/apache2/vhosts.d # mv /root/old/apache/
mv: cannot move `' to `/root/old/apache/': Operation not permitted
root@srv.remote /etc/apache2/vhosts.d # lsattr
root@srv.remote /etc/apache2/vhosts.d # rm
rm: cannot remove `': Operation not permitted
root@srv.remote /etc/apache2/vhosts.d # echo "teeest" >>
-bash: Permission denied

Here is how you can set the attribute off.

You need first to set off the file’s immutable attribute and then to do whatever you intended to do in the first place – delete, rename, change and so on. Y

chattr -i filename.txt

In continuation of our example above:

root@srv.remote /etc/apache2/vhosts.d # chattr -i
root@srv.remote /etc/apache2/vhosts.d # lsattr
root@srv.remote /etc/apache2/vhosts.d # mv /root/old/apache/
root@srv.remote /etc/apache2/vhosts.d #

As you can see no immutable attribute no problem to move the file!

And just not note you need to install a package with the name e2fsprogs (not always in the default installation) in your Linux distribution to have lsattr, chattr and more!

Really bad performance when going from Write-Back to Write-Through in a LSI controller

Ever wonder what is the impact of write-through of an LSI controller in a real-world streaming server? Have no wonder anymore!

you can get several (multiple?) times slower with the write-through mode than if your controller were using the write-back mode of the cache

And it could happen any moment because when charging the battery of the LSI controller and you have set “No Write Cache if Bad BBU” the write-through would kick in. Of course, you can make a schedule for the battery charging/discharging process, but in general, it will happen and it will hurt your IO performance a lot!

In simple words a write operation is successful only if the controller confirms the write operation on all disks, no matter the data has already been in the cache.

This mode puts pressure on the disks and Write-Through is a known destroyer of hard disks! You can read a lot of administrator’s feedback on the Internet about crashed disks using write-through mode (and sometimes several simultaneously on one machine losing all your data even it would have redundancy with some of the RAID setups like RAID1, RAID5, RAID6, RAID10 and so).

srv ~ # sudo megacli -ldinfo -lall -aall
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :system
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 13.781 TB
Sector Size         : 512
Mirror Data         : 13.781 TB
State               : Optimal
Strip Size          : 128 KB
Number Of Drives per span:2
Span Depth          : 6
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAdaptive, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: Yes
Cache Cade Type : Read Only

Exit Code: 0x00

As you can see our default cache policy is WriteBack and “No Write Cache if Bad BBU”, the BBU is not bad, but charging!
Keep on reading!

PHP 7.2 or PHP 7.3 with mcrypt – manual build

Newer PHP versions do not include PHP mcrypt library. The mcrypt module was part of PHP 5 till 7.1, in which it was deprecated and removed in 7.2. If you open the documentation for mcrypt PHP functions you will see:

This function has been DEPRECATED as of PHP 7.1.0 and REMOVED as of PHP 7.2.0. Relying on this function is highly discouraged.

The mcrypt module is now in PHP PECL (repository for PHP Extensions) in As you can see in the description, this is a legacy module, which

Provides bindings for the unmaintained libmcrypt.

, so it is strongly recommended to replace it with OpenSSL (for example).
Still, if you need this legacy module – mcrypt and :

You may want to manually build the mcrypt module for your current installed PHP. Of course, the generic dependencies are:

  1. libmcrypt and its headers (if the Linux distribution) splits the binary and the headers
  2. GNU GCC
  3. PHP 7.2+
  4. download the latest mcrypt module source from For example, now it is
mkdir /root/mcrypt-php-module-manual
cd /root/mcrypt-php-module-manual
tar xzf mcrypt-1.0.2.tgz
cd mcrypt-1.0.2
libtoolize --force
make install

Do not use “make -j N” (“make -j 8”, for example), because it may fail to compile.
Keep on reading!

using portage eix for the first time – cannot open database file

Installing “app-portage/eix” in Gentoo to manage your portage updates you might encounter this error, when trying to use “eix” for the first time:

Writing database file /var/cache/eix/portage.eix...
cannot open database file /var/cache/eix/portage.eix for writing (mode = 'wb')

The chances are missing directory “/var/cache/eix/” or the user:group of the “/var/cache/eix/” is root:root, which is NOT right.

The user:group must be “portage:portage”.

So the solution is really simple:

mkdir -p /var/cache/eix
chown portage:portage /var/cache/eix

Output – the errors you might get

Using the eix-sync failed with:

root@srv1 ~ # eix-sync 
 * eix-cache does not exist
 * Running eix-update
Reading Portage settings...
Building database (/var/cache/eix/portage.eix)...
[0] "gentoo" /usr/portage/ (cache: metadata-md5-or-flat)
     Reading category 167|167 (100) Finished             
[1] "myportage" /usr/local/myportage (cache: parse|ebuild*#metadata-md5#metadata-flat#assign)
     Reading category 167|167 (100) Finished    
Applying masks...
Calculating hash tables...
Writing database file /var/cache/eix/portage.eix...
cannot open database file /var/cache/eix/portage.eix for writing (mode = 'wb')
 * eix-update failed
 * Time statistics:
     6 seconds for initial eix-update
     6 seconds total

Using the “eix-update” failed, too.

root@srv ~ # eix-update 
Reading Portage settings...
Building database (/var/cache/eix/portage.eix)...
[0] "gentoo" /usr/portage/ (cache: metadata-md5-or-flat)
     Reading category 167|167 (100) Finished             
[1] "myportage" /usr/local/myportage (cache: parse|ebuild*#metadata-md5#metadata-flat#assign)
     Reading category 167|167 (100) Finished    
Applying masks...
Calculating hash tables...
Writing database file /var/cache/eix/portage.eix...
cannot open database file /var/cache/eix/portage.eix for writing (mode = 'wb')

Output 2 – Successful update with eix

root@srv ~ # eix-update 
Reading Portage settings...
Building database (/var/cache/eix/portage.eix)...
[0] "gentoo" /usr/portage/ (cache: metadata-md5-or-flat)
     Reading category 167|167 (100) Finished             
[1] "myportage" /usr/local/myportage (cache: parse|ebuild*#metadata-md5#metadata-flat#assign)
     Reading category 167|167 (100) Finished    
Applying masks...
Calculating hash tables...
Writing database file /var/cache/eix/portage.eix...
Database contains 19544 packages in 167 categories

Kernel loads only single processor on multi-processor system – ACPI: x2apic entry ignored

There multiple reports on this issue with different processors

kernel, which worked perfectly on multiple systems, loads on our new server and only one processor is shown

Just for the record, the SMP is enabled in the kernel (and in the BIOS – Hyperthreading and multicores are enabled, too):

root@srv ~ # zcat /proc/config.gz | grep 'CONFIG_SMP'

The problem is x2APIC Support in the BIOS of your server.

Apparently, our kernel (version 4.18.12) missed the kernel feature:

root@srv ~ # zcat /proc/config.gz | grep -i 'x2apic'

You can see no kernel configuration entry “CONFIG_X86_X2APIC=y” is shown from the above command.

And if your BIOS enables the support of x2APIC you may end up with just one processor under Linux.

This was the case in our server. The x2APIC support is enabled in the BIOS and our kernel (version 4.18.12) does not have CONFIG_X86_X2APIC enabled.
To fix this issue you first might disable the feature in the BIOS and you are going to have all your processors shown and they could be used to compile fast your new kernel (of course, in the case you use custom kernel) after you enable the feature in the kernel CONFIG_X86_X2APIC, which is under
Kernel Configuration —> [*] Support x2apic. The asterisk means it is enabled, so build your kernel. Check and enable this “Device Drivers –> IOMMU Hardware Support –> Support for Interrupt Remapping”, too.
Here you can see how to enable and disable processor x2APIC support in HP ProLiant DL160 Gen9 (2 processors Intel(R) Xeon(R) CPU E5-2630 v4 @ 2.20GHz) – Enable or Disable the processor x2APIC support in HP ProLiant DL160 Gen9.
Keep on reading!

bind – dns server queries statistics with statistics-file

There is an option “statistics-file” in the BIND9 configuration for query statistics. It will give you statistics for

  • Incoming Requests – total number of queries
  • Incoming Queries – queries by record type
  • Outgoing Queries – queries by record type per view
  • Name Server Statistics – extended queries statistics by network connection type (UDP, TCP or IPv4 and IPv6 interface), by type of answer (authoritative, non authoritative and so on) and more
  • Zone Maintenance Statistics – transfer and system queries
  • Resolver Statistics – recursive queries
  • Cache DB RRsets – cached resources records sets
  • Socket I/O Statistics – statistics numbers for UDP, TCP (for both IPv4 and IPV6) sockets and connections opened, closed, failure
  • Per Zone Query Statistics – so you can see how many queries you have for a zone in a view (and the transfers if the server is a slave)

In named.conf:

options {
    statistics-file "/var/log/named.stats";
    zone-statistics yes ;

But if you check in /var/log this file might be missing even your BIND server has been running for months!

This is because the statistics and the file is generated on request and is a snapshot at the moment you do the request

To request from the BIND server to generate the file is pretty easy:

root@srv ~ # rndc stats
root@srv ~ #

No standard output and you should have the stats file generated:

root@srv ~ # ls -altr /var/log/named.stats 
-rw-r--r-- 1 named named 174997 Apr  7 01:43 /var/log/named.stats

The generated requests are appended in the file with a UNIX timestamp.

--- Statistics Dump --- (1550561292)
+++ Statistics Dump +++ (1551233218)

Keep on reading!