Following the article SSD cache device to a hard disk drive using LVM the initial setup is:
- Single slow 2T harddisk. So no redundancy. If it fails all data is gone.
- SSD cache device for the slow harddisk above.
And here is how to handle a slow device failure! The data is all gone because there is no redundancy option when using a single device for data. The data is not valuable, because it is a cache server. This article will show what to expect when the slow device fails and how to replace it.
The original slow device is missing and replaced by a new one and the partitions are as follow:
- /dev/sda4 – the slow device. New device.
- /dev/sdb5 – the SSD device. Still in the LVM2 Volume Group and the Logical Volume.
Physical Volume is missing marked as “[unknown]”. The pvdisplay shows metadata information for the missing device. The vgdisplay and lvdisplay show information for the group and the logical volume, but the logical volume is in not available status. So the logical volume cannot be used, which is kind of normal when only the cache device is present.
Slow drive failure and the LVM2 status
[root@srv ~]# ssm list ------------------------------------------------------ Device Free Used Total Pool ------------------------------------------------------ /dev/sda 2.00 TB /dev/sda1 0.00 KB 50.00 GB 50.03 GB md /dev/sda2 0.00 KB 15.75 GB 15.76 GB md /dev/sda3 0.00 KB 1023.00 MB 1.00 GB md /dev/sda4 1.93 TB /dev/sdb 894.25 GB /dev/sdb1 0.00 KB 50.00 GB 50.03 GB md /dev/sdb2 0.00 KB 15.75 GB 15.76 GB md /dev/sdb3 0.00 KB 1023.00 MB 1.00 GB md /dev/sdb4 1.00 KB /dev/sdb5 0.00 KB 675.00 GB 675.00 GB VG_storage1 /dev/sdb6 152.46 GB [unknown] 0.00 KB 1.93 TB 1.93 TB VG_storage1 ------------------------------------------------------ ----------------------------------------------------- Pool Type Devices Free Used Total ----------------------------------------------------- VG_storage1 lvm 2 0.00 KB 2.59 TB 2.59 TB ----------------------------------------------------- ------------------------------------------------------------------------------ Volume Pool Volume size FS FS size Free Type Mount point ------------------------------------------------------------------------------ /dev/md125 md 50.00 GB ext4 50.00 GB 44.41 GB raid1 / /dev/md126 md 1023.00 MB ext4 1023.00 MB 788.84 MB raid1 /boot /dev/md127 md 15.75 GB raid1 /dev/sdb6 152.46 GB ext4 152.46 GB 145.77 GB ------------------------------------------------------------------------------ ---------------------------------------------------------------------------------- Snapshot Origin Pool Volume size Type ---------------------------------------------------------------------------------- /dev/VG_storage1/lv_storage1 [lv_storage1_corig] VG_storage1 1.93 TB cache ---------------------------------------------------------------------------------- [root@srv ~]# pvdisplay WARNING: Device for PV IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd not found or rejected by a filter. Couldn't find device with uuid IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd. --- Physical volume --- PV Name /dev/sdb5 VG Name VG_storage1 PV Size 675.00 GiB / not usable 4.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 172799 Free PE 0 Allocated PE 172799 PV UUID oLn3hh-ROFU-WSW8-0m8P-YLWY-Akoz-nCxh96 --- Physical volume --- PV Name [unknown] VG Name VG_storage1 PV Size 1.93 TiB / not usable 4.00 MiB Allocatable yes (but full) PE Size 4.00 MiB Total PE 507188 Free PE 0 Allocated PE 507188 PV UUID IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd [root@srv ~]# pvscan WARNING: Device for PV IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd not found or rejected by a filter. Couldn't find device with uuid IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd. PV /dev/sdb5 VG VG_storage1 lvm2 [<675.00 GiB / 0 free] PV [unknown] VG VG_storage1 lvm2 [1.93 TiB / 0 free] Total: 2 [2.59 TiB] / in use: 2 [2.59 TiB] / in no VG: 0 [0 ] [root@srv ~]# vgdisplay WARNING: Device for PV IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd not found or rejected by a filter. Couldn't find device with uuid IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd. --- Volume group --- VG Name VG_storage1 System ID Format lvm2 Metadata Areas 1 Metadata Sequence No 10 VG Access read/write VG Status resizable MAX LV 0 Cur LV 1 Open LV 0 Max PV 0 Cur PV 2 Act PV 1 VG Size 2.59 TiB PE Size 4.00 MiB Total PE 679987 Alloc PE / Size 679987 / 2.59 TiB Free PE / Size 0 / 0 VG UUID eZ2ZIb-jcDl-kPLj-oFwJ-LLuN-VxLD-rVJclP [root@srv ~]# vgscan Reading volume groups from cache. WARNING: Device for PV IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd not found or rejected by a filter. Couldn't find device with uuid IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd. Found volume group "VG_storage1" using metadata type lvm2 [root@srv ~]# lvdisplay WARNING: Device for PV IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd not found or rejected by a filter. Couldn't find device with uuid IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd. --- Logical volume --- LV Path /dev/VG_storage1/lv_storage1 LV Name lv_storage1 VG Name VG_storage1 LV UUID NFWlWF-VmSO-HVr4-72RW-YY82-1ax2-92cI6P LV Write Access read/write LV Creation host, time srv.example.com, 2019-10-25 17:18:41 +0000 LV Cache pool name lv_cache LV Cache origin name lv_storage1_corig LV Status NOT available LV Size 1.93 TiB Current LE 507188 Segments 1 Allocation inherit Read ahead sectors auto [root@srv ~]# lvscan WARNING: Device for PV IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd not found or rejected by a filter. Couldn't find device with uuid IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd. inactive '/dev/VG_storage1/lv_storage1' [1.93 TiB] inherit [root@srv ~]# lvs lvs lvscan [root@srv ~]# lvs -a WARNING: Device for PV IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd not found or rejected by a filter. Couldn't find device with uuid IfF78Q-KV3N-GH94-6wvU-23ku-jC20-S8tcBd. LV VG Attr LSize Pool Origin Data% Meta% Move Log Cpy%Sync Convert [lv_cache] VG_storage1 Cwi---C--- 674.90g [lv_cache_cdata] VG_storage1 Cwi------- 674.90g [lv_cache_cmeta] VG_storage1 ewi------- 48.00m lv_storage1 VG_storage1 Cwi---C-p- 1.93t [lv_cache] [lv_storage1_corig] [lv_storage1_corig] VG_storage1 owi---C-p- 1.93t [lvol0_pmspare] VG_storage1 ewi------- 48.00m