Tag: grub2

Install or reinstall CentOS Stream 9 grub 2 under UEFI system

Under CentOS Stream 9 the grub2-install tool cannot install the bootloader manager Grub 2 when using UEFI-based systems. In fact, the grub2-install command installs only the old legacy mode Grub 2.
The grub2-mkconfig generates the Grub configuration file and adds an UEFI entry in the UEFI enabled BIOS of the systems. The grub2-mkconfig adds a boot entry with path to the /boot/efi/EFI/centos/shimx64.efi, which is a small and simple program to start the Grub 2 program grubx64.efi in the same directory. All this method is because of the UEFI Secure Boot support. Using the previous method it was not possible to support UEFI Secure Boot though it was possible to boot in UEFI mode.
There are some considerations to take in mind when using this new version of Grub 2:

  • grub2-mkconfig has two functions – generates the grub.cfg AND adds boot entry in the UEFI BIOS!
  • The efi partition with this file may not be the first partition.
  • It is an ordinary partition, i.e. it is not mandatory to be of ESP type.
  • The system CANNOT boot without a boot entry in the UEFI BIOS.
  • Booting in UEFI mode only by pointing the disk in the BIOS (or in the boot manager) may not occur. No auto detection and there is no boot program installed in the first sectors of the disk or what so ever. Even, the default install of CentOS Stream 9 creates the efi disk as second partition and without ESP flag enabled.
  • Grub 2 uses /boot/efi/EFI/centos/grub.cfg and this file should hold the latest output of grub2-mkconfig for successful booting after installation.
  • Some dedicated servers’ providers forbid any modification or changes to the server’s BIOS (even with UEFI BIOS), so those servers cannot boot in UEFI mode with the newer Grub 2 versions. Install the legacy mode Grub 2 with bios_grub flag partition and grub2-install.
  • Now, Grub 2 with this method supports UEFI Secure Boot.
  • Use grubby command to select the default kernel. More information here – removing the default kernel in CentOS 8 – remove elrepo kernel

More information might be available here – https://docs.fedoraproject.org/en-US/quick-docs/bootloading-with-grub2/ and https://fedoraproject.org/wiki/GRUB_2.

Note: The CentOS Stream 8 also have this new Grub 2 version, which supports only this kind of Grub 2 installation. So the procedure may be the same for CentOS Stream 8.

Grub 2 installation under CentOS Stream 9 and UEFI mode.

[root@srv ~]# grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg
Generating grub configuration file ...
Adding boot menu entry for UEFI Firmware Settings ...
done

The above command will recreate the grub.cfg with the proper kernels and kernel command-line arguments from /etc/default/grub and it will add a boot entry in the UEFI BIOS pointing to the “(partition,file-system-UUID)/EFI/centos/shimx64.efi“.
No grub2-install is needed to install the bootloader program anywhere in the first sectors of disk!

SCREENSHOT 1) The grub2-mkconfig tool will add this boot entry with name “CentOS Linux” with the full path to bootloader program shimx64.efi.

The full path is HD(1,GPT, C31EFB93-690C-4137-B589-0A962BE4B960,0x800,0x11D800)/EFI/centos/shimx64.efi. Most UEFI BIOS setups allow the user to add manually this entry by browsing to the file when adding boot entry.

main menu
BIOS boot entry and shimx64

Keep on reading!

CentOS 7 dracut-initqueue timeout and could not boot – warning /dev/disk/by-id/md-uuid- does not exist

Let’s say you update your software raid layout – create, delete or modify your software raid and reboot the system and your server does not start normally. After loading your remote video console (KVM) you see the boot process reports for a missing device and you are under console (dracut console). Your system is in “Emergency mode”.

The warning:

dracut-initqueue[504]: Warning: dracut-initqueue timeout - starting timeout scripts
dracut-initqueue[504]: Warning: dracut-initqueue timeout - starting timeout scripts
dracut-initqueue[504]: Warning: dracut-initqueue timeout - starting timeout scripts
....
....
dracut-initqueue[504]: Warning: could not boot.
dracut-initqueue[504]: Warning: /dev/disk/by-id/md-uuid-2fdc509e:8dd05ed3:c2350cb4:ea5a620d does not exist
      Starting Dracut Emergency Shell...
Warning: /dev/disk/by-id/md-uuid-2fdc509e:8dd05ed3:c2350cb4:ea5a620d does not exist

Generating "/run/initramfs/rdsosreport.txt"


Entering emergency mode. Exit the shell to continue.
Type "journalctl" to view system logs.
You might want to save "/run/initramfs/rdsosreport.txt" to a USB stick or /boot
after mounting them and attach it to a bug report.


dracut:/#

This article is for problems, which occur after manipulating a storage RAID devices, not the system (root) or boot devices!!! If the missing device the RAID “md-uuid-2fdc509e:8dd05ed3:c2350cb4:ea5a620d does not exist” is either the root or the boot device, the propose solution here would not help with just exiting the Emergency Shell! In those cases, when the missing device is the root or boot before exiting the Emergency Shell the problem must be resolved, so the devices and their file system should be available. There is another article on the subject, which may help the reader in such cases – CentOS 8 dracut-initqueue timeout and could not boot – warning /dev/disk/by-id/md-uuid- does not exist – inactive raids.

SCREENSHOT 1) The boot process reports mutiple warning messages of dracut-initqueue timeout, because a drive cannot be found.

main menu
Warning: dracut-initqueue timeout – starting timeout scripts

Keep on reading!

Centos 7 Server hangs up on boot after deleting a software raid (mdadm device)

We have a CentOS 7 server with a simple two hard drives setup in RAID1 of total 4 devices for boot, root, swap and storage. The storage device (/dev/md5) was removed and recreated with RAID0 for better performance, because the server was promoted as only cache server. Then the server was restarted and it never went up.
On IPMI KVM it just started loading the kernel and hanged up after several seconds without any additional information:

The kernel loads the mdadm devices and do not continue and the device md5 is missing.

main menu
CentOS 7 kernel loading the mdadm RAID devices

To boot successfully you must remove the missing device

On the Grub 2 menu press “e” and you’ll get this screen. Here you can edit all lines if you need. You must remove the last rd.md.uuid in our case or the one you deleted. Remove it and press Ctrl+x to load the kernel.

main menu
Grub 2 edit

There are two options you can do:

  • OPTION 1) Remove rd.md.uuid option of your old mdadm device
  • OPTION 2) Replace the ID in rd.md.uuid= with the new ID of the mdadm device.

Each of these two options could be used to solve the booting problem. Edit /etc/default/grub and replace or remove rd.md.uuid and generate the grub.conf.
You can find old mdadm ID in /etc/mdadm.conf (if you have not replace it there).

[root@srv ~]# cat /etc/mdadm.conf 
ARRAY /dev/md2 level=raid1 num-devices=2 metadata=0.90 UUID=9c08f218:cd5c0f8f:d96bc0d1:57b77e99
ARRAY /dev/md3 level=raid1 num-devices=2 metadata=1.2 name=2035110:swap UUID=1f74a2e0:757bfb9f:9c860e50:325f37cb
ARRAY /dev/md4 level=raid1 num-devices=2 metadata=1.2 name=2035110:root UUID=29bf4aa8:b7dae21a:45f4c188:baea4c13
ARRAY /dev/md5 level=raid1 num-devices=2 metadata=1.2 name=2035110:storage1 UUID=e6eb2590:b767be36:c76bb869:45ff0c3c
[root@srv ~]# mdadm --detail --scan
ARRAY /dev/md2 metadata=0.90 UUID=9c08f218:cd5c0f8f:d96bc0d1:57b77e99
ARRAY /dev/md3 metadata=1.2 name=2035110:swap UUID=1f74a2e0:757bfb9f:9c860e50:325f37cb
ARRAY /dev/md4 metadata=1.2 name=2035110:root UUID=29bf4aa8:b7dae21a:45f4c188:baea4c13
ARRAY /dev/md/5 metadata=1.2 name=s2035110:5 UUID=901074eb:16ba7c5b:0af69934:e9444102
[root@srv ~]# mdadm --detail --scan > /etc/mdadm.conf

Here is our old /etc/default/grub:

[root@srv ~]# cat /etc/default/grub 
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL="serial console"
GRUB_SERIAL_COMMAND="serial --speed=115200"
GRUB_CMDLINE_LINUX="rd.md.uuid=9c08f218:cd5c0f8f:d96bc0d1:57b77e99 rd.md.uuid=1f74a2e0:757bfb9f:9c860e50:325f37cb rd.md.uuid=29bf4aa8:b7dae21a:45f4c188:baea4c13 rd.md.uuid=e6eb2590:b767be36:c76bb869:45ff0c3c console=tty0 crashkernel=auto console=ttyS0,115200 net.ifnames=1"
GRUB_DISABLE_RECOVERY="true"

Here we edit our /boot/grub2/grub.cfg, replace the old uuid and generate grub.cfg (legacy BIOS):

[root@srv ~]# cat /etc/default/grub 
GRUB_TIMEOUT=1
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL="serial console"
GRUB_SERIAL_COMMAND="serial --speed=115200"
GRUB_CMDLINE_LINUX="rd.md.uuid=9c08f218:cd5c0f8f:d96bc0d1:57b77e99 rd.md.uuid=1f74a2e0:757bfb9f:9c860e50:325f37cb rd.md.uuid=29bf4aa8:b7dae21a:45f4c188:baea4c13 rd.md.uuid=901074eb:16ba7c5b:0af69934:e9444102 console=tty0 crashkernel=auto console=ttyS0,115200 net.ifnames=1"
[root@srv ~]# grub2-mkconfig -o /boot/grub2/grub.cfg 
Generating grub configuration file ...
Found linux image: /boot/vmlinuz-3.10.0-957.5.1.el7.x86_64
Found initrd image: /boot/initramfs-3.10.0-957.5.1.el7.x86_64.img
Found linux image: /boot/vmlinuz-0-rescue-05cb8c7b39fe0f70e3ce97e5beab809d
Found initrd image: /boot/initramfs-0-rescue-05cb8c7b39fe0f70e3ce97e5beab809d.img
done
[root@srv ~]# reboot

Use this for UEFI BIOS boot:
First check if /boot and /boot/efi are mounted and if not you must mount them with:

mount /boot
mount /boot/efi

Generate the grub.cfg

grub2-mkconfig -o /boot/efi/EFI/centos/grub.cfg

Bonus

In fact when the original device was removed and added a new one we formatted it as usual. But it was not possible to mount it, you just execute mount 

/dev/md5 /mnt/stor1

no error, but no mount could be found, the device was not mounted and when you execute 

umount /mnt/stor1

The OS told the “/mnt/stor1” was not mounted. Several more tries were made unsuccessfully to mount the “/dev/md5”, then the restart was performed and the server never went up.
Suppose the systemd just did not allow to mount the device because of the boot parameters rd.md.uuid!

grub2: grub-install: error: disk mduuid not found even after the partition has bios_grub on

This tutorial is for all of us that has done everything by the book with parted and still they receive an error when installing grub2 to the boot sector!

srv@local ~ # grub2-install /dev/sda
Installing for i386-pc platform.
grub2-install: error: disk `mduuid/613f3f0bb202bf03a5664e17b3d568a0' not found.

The solution is relatively simple:

Boot from a rescue disk and reinstall grub from there!

The problem is that currently loaded kernel remembers the old device, which was deleted, and probably you won’t be able to fresh the metadata in the memory. In fact, if you receive this error when booted in a rescue disc you probably have done some changes on the layout of the disks or the partitions or the RAID devices and you must reboot the machine again and then JUST reinstall the grub.
Such problems could have happened if you deleted partitions or made some disk layout changes (using parted?) on mounted or partitions in use and the kernel could loaded the partition changes in memory. The parted reports the changes will take effect after system reboot.

STEP 1) Check the devices and mount the root and boot

They might be on the same device here they are on different MD devices (in the case, software array). md1 is the boot and md2 is the root.

root@rescue ~ # cat /proc/mdstat 
Personalities : [raid1] 
md3 : active raid1 sda5[3] sdb5[2]
      422716416 blocks super 1.2 [2/2] [UU]
      bitmap: 0/4 pages [0KB], 65536KB chunk

md2 : active raid1 sda3[3] sdb3[2]
      31440896 blocks super 1.2 [2/2] [UU]
      
md1 : active raid1 sda2[3] sdb2[2]
      523712 blocks super 1.2 [2/2] [UU]
      
md0 : active raid1 sda1[3] sdb1[2]
      33521664 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>
root@rescue ~ # mount /dev/md2 /mnt/
root@rescue ~ # mount /dev/md1 /mnt/boot/
root@rescue ~ # ls -altr /mnt/
total 396K
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 srv
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 opt
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 mnt
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 media
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 home
lrwxrwxrwx.  1 root root    8 May 14  2018 sbin -> usr/sbin
lrwxrwxrwx.  1 root root    9 May 14  2018 lib64 -> usr/lib64
lrwxrwxrwx.  1 root root    7 May 14  2018 lib -> usr/lib
lrwxrwxrwx.  1 root root    7 May 14  2018 bin -> usr/bin
drwxr-xr-x. 13 root root 4.0K May 14  2018 usr
drwx------.  2 root root  16K Feb 12  2019 lost+found
drwxr-xr-x   2 root root 4.0K Feb 12  2019 boot
drwxr-xr-x   2 root root 4.0K Feb 12  2019 storage1
drwxr-xr-x   2 root root 4.0K Feb 12  2019 sys
drwxr-xr-x   2 root root 4.0K Feb 12  2019 dev
drwxr-xr-x   2 root root 4.0K Feb 12  2019 prochttps://www.google.com/search?client=firefox-b-d&q=samsung+tab10+2020
drwxr-xr-x   3 root root 4.0K Feb 12  2019 run
-rw-r-----.  1 root root  575 Feb 12  2019 installimage.conf
-rw-r-----.  1 root root  13K Feb 12  2019 installimage.debug
drwxr-xr-x. 20 root root 4.0K Feb 12  2019 var
drwxr-xr-x. 85 root root 4.0K Nov  2  2019 etc
-rw-r--r--.  1 root root 291K Nov  2  2019 .readahead
drwxr-xr-x. 19 root root 4.0K Nov  2  2019 .
dr-xr-x---.  6 root root 4.0K May  4 19:24 root
drwxrwxrwt.  7 root root 4.0K May  8 14:14 tmp
drwxr-xr-x   1 root root  160 May  8 16:57 ..
root@rescue ~ # ls -altr /mnt/boot/
total 194M
drwxr-xr-x.  3 root root 1.0K Sep 18  2017 efi
-rw-------.  1 root root  49M Sep 18  2017 initramfs-0-rescue-9063ac396d784f4c997ceacdd0590c25.img
-rwxr-xr-x.  1 root root 5.7M Sep 18  2017 vmlinuz-0-rescue-9063ac396d784f4c997ceacdd0590c25
-rw-------.  1 root root 3.4M Feb  1  2019 System.map-3.10.0-957.5.1.el7.x86_64
-rw-r--r--.  1 root root 149K Feb  1  2019 config-https://www.google.com/search?client=firefox-b-d&q=samsung+tab10+20203.10.0-957.5.1.el7.x86_64
-rw-r--r--.  1 root root  170 Feb  1  2019 .vmlinuz-3.10.0-957.5.1.el7.x86_64.hmac
-rwxr-xr-x.  1 root root 6.4M Feb  1  2019 vmlinuz-3.10.0-957.5.1.el7.x86_64
-rw-r--r--.  1 root root 307K Feb  1  2019 symvers-3.10.0-957.5.1.el7.x86_64.gz
drwx------.  2 root root  12K Feb 12  2019 lost+found
drwxr-xr-x.  2 root root 1.0K Feb 12  2019 grub
-rw-------.  1 root root 3.5M Oct 18  2019 System.map-3.10.0-1062.4.1.el7.x86_64
-rw-r--r--.  1 root root 150K Oct 18  2019 config-3.10.0-1062.4.1.el7.x86_64
-rw-r--r--.  1 root root  171 Oct 18  2019 .vmlinuz-3.10.0-1062.4.1.el7.x86_64.hmac
-rwxr-xr-x.  1 root root 6.5M Oct 18  2019 vmlinuz-3.10.0-1062.4.1.el7.x86_64
-rw-r--r--.  1 root root 312K Oct 18  2019 symvers-3.10.0-1062.4.1.el7.x86_64.gz
-rw-------.  1 root root  13M Nov  2  2019 initramfs-3.10.0-957.5.1.el7.x86_64kdump.img
-rw-------.  1 root root  47M Nov  2md  2019 initramfs-3.10.0-1062.4.1.el7.x86_64.img
-rw-------.  1 root root  46M Nov  2  2019 initramfs-3.10.0-957.5.1.el7.x86_64.img
-rw-------.  1 root root  13M Nov  2  2019 initramfs-3.10.0-1062.4.1.el7.x86_64kdump.img
dr-xr-xr-x.  6 root root 1.0K Nov  2  2019 .
drwxr-xr-x. 19 root root 4.0K Nov  2  2019 ..
drwx------.  5 root root 1.0K May  4 16:11 grub2

STEP 2) mount the dev, proc and sys relative to the root mount above and chroot in the root.

After chroot it is recommended to 

root@rescue ~ # mount -o bind /dev /mnt/dev
root@rescue ~ # mount -o bind /proc /mnt/proc
root@rescue ~ # mount -o bind /sys /mnt/sys
root@rescue ~ # chroot /mnt/
root@rescue / # cd
root@rescue ~ # . /etc/profile

STEP 3)Install grub2

Unmount all mounted directories above and reboot. 

root@rescue ~ # grub2-install /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.
root@rescue ~ # grub2-install /dev/sdb
Installing for i386-pc platform.
Installation finished. No error reported.
root@rescue ~ # exit
root@rescue ~ # umount /mnt/boot 
root@rescue ~ # umount /mnt/dev
root@rescue ~ # umount /mnt/proc 
root@rescue ~ # umount /mnt/sys
root@rescue ~ # umount /mnt/
root@rescue ~ # reboot

* If you are using UEFI enabled boot you probably need more options for the grub installation

Something like that for the grub2 installation (but it is specific for your distro – the path for efi directory, just find it under /boot and put the right path – nothing special!):

grub-install --recheck --target=x86_64-efi --efi-directory=/boot/efi/ /dev/sda

Where does this error come from?

If you search the ID by part of it you will discover it under /dev/disk/by-id/, which is a link to md1, but still there is an error for missing device, because the kernel loaded the device as md-name-rescue:1. And if you reboot the kernel the old and wrong name won’t exists. As mentioned above this could have happened if you deleted partitions or made some disk layout changes (using parted?) on mounted or partitions in use and the kernel could loaded the partition changes.

[root@srv47 ~]# cd /dev/disk/by-id/
[root@srv47 by-id]# ls -altr
total 0
drwxr-xr-x. 5 root root 100  4 May  9,14 ..
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-uuid-e4023626:e09f7c39:20ed5720:2ef1b5af -> ../../md0
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-uuid-ae63857c:b8d537f5:4c09ae48:48148f59 -> ../../md3
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-uuid-4a28f3a7:94dc6dc7:a40a7084:c21463d7 -> ../../md2
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-name-rescue:3 -> ../../md3
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-name-rescue:2 -> ../../md2
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-name-rescue:0 -> ../../md0
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-uuid-613f3f0b:b202bf03:a5664e17:b3d568a0 -> ../../md1
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-name-rescue:1 -> ../../md1
drwxr-xr-x. 2 root root 680  4 May 12,16 .
lrwxrwxrwx. 1 root root   9  8 May 12,01 wwn-0x500a075116bd3e33 -> ../../sda
lrwxrwxrwx. 1 root root   9  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33 -> ../../sda
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part5 -> ../../sda5
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part4 -> ../../sda4
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part3 -> ../../sda3
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part2 -> ../../sda2
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part5 -> ../../sda5
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part4 -> ../../sda4
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part3 -> ../../sda3
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part2 -> ../../sda2
lrwxrwxrwx. 1 root root   9  8 May 12,01 wwn-0x500a07511bb48b38 -> ../../sdb
lrwxrwxrwx. 1 root root   9  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38 -> ../../sdb
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part5 -> ../../sdb5
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part4 -> ../../sdb4
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part3 -> ../../sdb3
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part2 -> ../../sdb2
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part5 -> ../../sdb5
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part4 -> ../../sdb4
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part3 -> ../../sdb3
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part2 -> ../../sdb2
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part1 -> ../../sdb1