Inactive array – mdadm: Cannot get array info for /dev/md126

Replacing a disk maybe sometimes challenging, especially with software RAID. If the software RAID1 went inactive this article might be for you!
Booting from a LIVECD or a rescue PXE system and all RAID devices got inactive despite the loaded personalities. We have similar article on the subject – Recovering MD array and mdadm: Cannot get array info for /dev/md0

livecd ~ # cat /proc/mdstat 
Personalities : [raid6] [raid5] [raid4] [raid0] [raid1] [raid10] [linear] [multipath] 
md125 : inactive sdb3[1](S)
      1047552 blocks super 1.2
       
md126 : inactive sdb1[1](S)
      52427776 blocks super 1.2
       
md127 : inactive sdb2[1](S)
      16515072 blocks super 1.2
       
unused devices: <none>

Despite the personalities are loaded, which means the kernel modules are successfully loaded – “[raid6] [raid5] [raid4] [raid0] [raid1] [raid10] [linear] [multipath] “. Still, something got wrong and the device’s personality is unrecognized and is inactive state.
A device in inactive state cannot be recovered and it cannot be added disks:

livecd ~ # mdadm --add /dev/md125 /dev/sda3
mdadm: Cannot get array info for /dev/md125

In general, to recover a RAID in inactive state:

  1. Check if the kernel modules are loaded. If the RAID setups are using RAID1, the “Personalities” line in /proc/mdstat should include it as “[raid1]”
  2. Try to run the device with “mdadm –run”
  3. Add the missing device to the RAID device with “mdadm –add” if the status of the RAID device goes to “active (auto-read-only)” or just “active”.
  4. Wait for the RAID device to recover.

Keep on reading!

Remove disk (all partitions) from software RAID1 with mdadm and change layout of the disk

The following article is to show how to remove healthy partitions from software RAID1 devices to change the layout of the disk and then add them back to the array.
The mdadm is the tool to manipulate the software RAID devices under Linux and it is part of all Linux distributions (some don’t install it by default so it may need to be installed).

Software RAID layout

[root@srv ~]# cat /proc/mdstat 
Personalities : [raid1] 
md125 : active raid1 sda4[1] sdb3[0]
      1047552 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

md126 : active raid1 sdb2[0] sda3[1]
      32867328 blocks super 1.2 [2/2] [UU]
      
md127 : active raid1 sda2[1] sdb1[0]
      52427776 blocks super 1.2 [2/2] [UU]
      bitmap: 0/1 pages [0KB], 65536KB chunk

unused devices: <none>

STEP 1) Make the partitions faulty.

The partitions cannot be removed if they are not faulty.

[root@srv ~]# mdadm --fail /dev/md125 /dev/sdb3
mdadm: set /dev/sdb3 faulty in /dev/md125
[root@srv ~]# mdadm --fail /dev/md126 /dev/sdb2
mdadm: set /dev/sdb2 faulty in /dev/md126
[root@srv ~]# mdadm --fail /dev/md127 /dev/sdb1
mdadm: set /dev/sdb1 faulty in /dev/md127

Keep on reading!

grub2: grub-install: error: disk mduuid not found even after the partition has bios_grub on

This tutorial is for all of us that has done everything by the book with parted and still they receive an error when installing grub2 to the boot sector!

srv@local ~ # grub2-install /dev/sda
Installing for i386-pc platform.
grub2-install: error: disk `mduuid/613f3f0bb202bf03a5664e17b3d568a0' not found. 

The solution is relatively simple:

Boot from a rescue disk and reinstall grub from there!

The problem is that currently loaded kernel remembers the old device, which was deleted, and probably you won’t be able to fresh the metadata in the memory. In fact, if you receive this error when booted in a rescue disc you probably have done some changes on the layout of the disks or the partitions or the RAID devices and you must reboot the machine again and then JUST reinstall the grub.
Such problems could have happened if you deleted partitions or made some disk layout changes (using parted?) on mounted or partitions in use and the kernel could loaded the partition changes in memory. The parted reports the changes will take effect after system reboot.

STEP 1) Check the devices and mount the root and boot

They might be on the same device here they are on different MD devices (in the case, software array). md1 is the boot and md2 is the root.

root@rescue ~ # cat /proc/mdstat 
Personalities : [raid1] 
md3 : active raid1 sda5[3] sdb5[2]
      422716416 blocks super 1.2 [2/2] [UU]
      bitmap: 0/4 pages [0KB], 65536KB chunk

md2 : active raid1 sda3[3] sdb3[2]
      31440896 blocks super 1.2 [2/2] [UU]
      
md1 : active raid1 sda2[3] sdb2[2]
      523712 blocks super 1.2 [2/2] [UU]
      
md0 : active raid1 sda1[3] sdb1[2]
      33521664 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>
root@rescue ~ # mount /dev/md2 /mnt/
root@rescue ~ # mount /dev/md1 /mnt/boot/
root@rescue ~ # ls -altr /mnt/
total 396K
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 srv
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 opt
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 mnt
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 media
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 home
lrwxrwxrwx.  1 root root    8 May 14  2018 sbin -> usr/sbin
lrwxrwxrwx.  1 root root    9 May 14  2018 lib64 -> usr/lib64
lrwxrwxrwx.  1 root root    7 May 14  2018 lib -> usr/lib
lrwxrwxrwx.  1 root root    7 May 14  2018 bin -> usr/bin
drwxr-xr-x. 13 root root 4.0K May 14  2018 usr
drwx------.  2 root root  16K Feb 12  2019 lost+found
drwxr-xr-x   2 root root 4.0K Feb 12  2019 boot
drwxr-xr-x   2 root root 4.0K Feb 12  2019 storage1
drwxr-xr-x   2 root root 4.0K Feb 12  2019 sys
drwxr-xr-x   2 root root 4.0K Feb 12  2019 dev
drwxr-xr-x   2 root root 4.0K Feb 12  2019 prochttps://www.google.com/search?client=firefox-b-d&q=samsung+tab10+2020
drwxr-xr-x   3 root root 4.0K Feb 12  2019 run
-rw-r-----.  1 root root  575 Feb 12  2019 installimage.conf
-rw-r-----.  1 root root  13K Feb 12  2019 installimage.debug
drwxr-xr-x. 20 root root 4.0K Feb 12  2019 var
drwxr-xr-x. 85 root root 4.0K Nov  2  2019 etc
-rw-r--r--.  1 root root 291K Nov  2  2019 .readahead
drwxr-xr-x. 19 root root 4.0K Nov  2  2019 .
dr-xr-x---.  6 root root 4.0K May  4 19:24 root
drwxrwxrwt.  7 root root 4.0K May  8 14:14 tmp
drwxr-xr-x   1 root root  160 May  8 16:57 ..
root@rescue ~ # ls -altr /mnt/boot/
total 194M
drwxr-xr-x.  3 root root 1.0K Sep 18  2017 efi
-rw-------.  1 root root  49M Sep 18  2017 initramfs-0-rescue-9063ac396d784f4c997ceacdd0590c25.img
-rwxr-xr-x.  1 root root 5.7M Sep 18  2017 vmlinuz-0-rescue-9063ac396d784f4c997ceacdd0590c25
-rw-------.  1 root root 3.4M Feb  1  2019 System.map-3.10.0-957.5.1.el7.x86_64
-rw-r--r--.  1 root root 149K Feb  1  2019 config-https://www.google.com/search?client=firefox-b-d&q=samsung+tab10+20203.10.0-957.5.1.el7.x86_64
-rw-r--r--.  1 root root  170 Feb  1  2019 .vmlinuz-3.10.0-957.5.1.el7.x86_64.hmac
-rwxr-xr-x.  1 root root 6.4M Feb  1  2019 vmlinuz-3.10.0-957.5.1.el7.x86_64
-rw-r--r--.  1 root root 307K Feb  1  2019 symvers-3.10.0-957.5.1.el7.x86_64.gz
drwx------.  2 root root  12K Feb 12  2019 lost+found
drwxr-xr-x.  2 root root 1.0K Feb 12  2019 grub
-rw-------.  1 root root 3.5M Oct 18  2019 System.map-3.10.0-1062.4.1.el7.x86_64
-rw-r--r--.  1 root root 150K Oct 18  2019 config-3.10.0-1062.4.1.el7.x86_64
-rw-r--r--.  1 root root  171 Oct 18  2019 .vmlinuz-3.10.0-1062.4.1.el7.x86_64.hmac
-rwxr-xr-x.  1 root root 6.5M Oct 18  2019 vmlinuz-3.10.0-1062.4.1.el7.x86_64
-rw-r--r--.  1 root root 312K Oct 18  2019 symvers-3.10.0-1062.4.1.el7.x86_64.gz
-rw-------.  1 root root  13M Nov  2  2019 initramfs-3.10.0-957.5.1.el7.x86_64kdump.img
-rw-------.  1 root root  47M Nov  2md  2019 initramfs-3.10.0-1062.4.1.el7.x86_64.img
-rw-------.  1 root root  46M Nov  2  2019 initramfs-3.10.0-957.5.1.el7.x86_64.img
-rw-------.  1 root root  13M Nov  2  2019 initramfs-3.10.0-1062.4.1.el7.x86_64kdump.img
dr-xr-xr-x.  6 root root 1.0K Nov  2  2019 .
drwxr-xr-x. 19 root root 4.0K Nov  2  2019 ..
drwx------.  5 root root 1.0K May  4 16:11 grub2

STEP 2) mount the dev, proc and sys relative to the root mount above and chroot in the root.

After chroot it is recommended to

root@rescue ~ # mount -o bind /dev /mnt/dev
root@rescue ~ # mount -o bind /proc /mnt/proc
root@rescue ~ # mount -o bind /sys /mnt/sys
root@rescue ~ # chroot /mnt/
root@rescue / # cd
root@rescue ~ # . /etc/profile

STEP 3)Install grub2

Unmount all mounted directories above and reboot.

root@rescue ~ # grub2-install /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.
root@rescue ~ # grub2-install /dev/sdb
Installing for i386-pc platform.
Installation finished. No error reported.
root@rescue ~ # exit
root@rescue ~ # umount /mnt/boot 
root@rescue ~ # umount /mnt/dev
root@rescue ~ # umount /mnt/proc 
root@rescue ~ # umount /mnt/sys
root@rescue ~ # umount /mnt/
root@rescue ~ # reboot

* If you are using UEFI enabled boot you probably need more options for the grub installation

Something like that for the grub2 installation (but it is specific for your distro – the path for efi directory, just find it under /boot and put the right path – nothing special!):

grub-install --recheck --target=x86_64-efi --efi-directory=/boot/efi/ /dev/sda

Where does this error come from?

If you search the ID by part of it you will discover it under /dev/disk/by-id/, which is a link to md1, but still there is an error for missing device, because the kernel loaded the device as md-name-rescue:1. And if you reboot the kernel the old and wrong name won’t exists. As mentioned above this could have happened if you deleted partitions or made some disk layout changes (using parted?) on mounted or partitions in use and the kernel could loaded the partition changes.

[root@srv47 ~]# cd /dev/disk/by-id/
[root@srv47 by-id]# ls -altr
total 0
drwxr-xr-x. 5 root root 100  4 May  9,14 ..
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-uuid-e4023626:e09f7c39:20ed5720:2ef1b5af -> ../../md0
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-uuid-ae63857c:b8d537f5:4c09ae48:48148f59 -> ../../md3
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-uuid-4a28f3a7:94dc6dc7:a40a7084:c21463d7 -> ../../md2
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-name-rescue:3 -> ../../md3
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-name-rescue:2 -> ../../md2
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-name-rescue:0 -> ../../md0
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-uuid-613f3f0b:b202bf03:a5664e17:b3d568a0 -> ../../md1
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-name-rescue:1 -> ../../md1
drwxr-xr-x. 2 root root 680  4 May 12,16 .
lrwxrwxrwx. 1 root root   9  8 May 12,01 wwn-0x500a075116bd3e33 -> ../../sda
lrwxrwxrwx. 1 root root   9  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33 -> ../../sda
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part5 -> ../../sda5
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part4 -> ../../sda4
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part3 -> ../../sda3
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part2 -> ../../sda2
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part5 -> ../../sda5
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part4 -> ../../sda4
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part3 -> ../../sda3
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part2 -> ../../sda2
lrwxrwxrwx. 1 root root   9  8 May 12,01 wwn-0x500a07511bb48b38 -> ../../sdb
lrwxrwxrwx. 1 root root   9  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38 -> ../../sdb
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part5 -> ../../sdb5
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part4 -> ../../sdb4
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part3 -> ../../sdb3
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part2 -> ../../sdb2
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part5 -> ../../sdb5
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part4 -> ../../sdb4
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part3 -> ../../sdb3
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part2 -> ../../sdb2
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part1 -> ../../sdb1