mdadm assembles AVAGO/LSI MegaRAID controller RAID 5 array

Author:

It is possible to read data with the software Linux raid using mdadm tool from a RAID 5 array created with the hardware raid controller AVAGO MegaRAID 9361-4i (LSI SAS3108).

main menu
mdadm E sdb

Here, how a RAID 5 array with 3 hard drives and 1 SSD ( with CacheCade in write-through mode) is assembled by the mdadm and Linux software raid:

livecd ~ # cat /proc/mdstat 
Personalities : [raid0] [raid6] [raid5] [raid4] 
md125 : active raid0 sda[0]
      937164800 blocks super external:/md127/1 1024k chunks
      
md126 : active raid5 sdb[2] sdc[1] sdd[0]
      23436722176 blocks super external:/md127/0 level 5, 1024k chunk, algorithm 2 [3/3] [UUU]
      [==============>......]  resync = 72.0% (8438937704/11718361088) finish=336.8min speed=162234K/sec
      
md127 : inactive sdb[3](S) sda[2](S) sdd[1](S) sdc[0](S)
      2100568 blocks super external:ddf
       
unused devices: <none>

Note, it is essential that the CacheCade device is in write-through mode, which means the cache device is used only for reading and the data on the RAID array is consistent and written on it. The RAID 5 array was created here – AVAGO MegaRAID SAS-9361-4i with CacheCade – create a new virtual drive RAID5 with SSD caching. It seams possible for the data to be consistent if the CacheCade is write-back mode if there were few small writes and orderly shutdown prior to the removal of the AVAGO MegaRAID 9361-4i.
So, the above devices use proprietary LSI format, but here Linux software raid supports some of them:

  • md125 – the SSD device, which is a read cache only.
  • md126 – 3 hard drives in RAID 5 array.
  • md127 – logical device, which provides transparent interface to the

The important device is md126 and can be mounted under some live Linux CD/USB. Further, the md125 is a device, which has GPT partition table with 5 partitions:

livecd ~ # parted /dev/md126 --script print
Model: Linux Software RAID Array (md)
Disk /dev/md126: 24.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name                  Flags
 1      1049kB  211MB   210MB   fat16           EFI System Partition  boot, esp
 2      211MB   1285MB  1074MB  ext4                                  msftdata
 3      1285MB  23.9TB  23.9TB  ext4                                  msftdata
 4      23.9TB  24.0TB  53.7GB  ext4                                  msftdata
 5      24.0TB  24.0TB  16.8GB  linux-swap(v1)                        swap


There is CentOS 7 installed on the /dev/md126 and the root device is on /dev/md126p4. To mount it just use the /dev/md126p4:

livecd ~ # mount /dev/md126p4 /mnt/centos/
livecd ~ # ls -altr /mnt/gentoo/
total 364
drwxr-xr-x  2 root root   4096 Apr 11  2018 srv
drwxr-xr-x  2 root root   4096 Apr 11  2018 mnt
drwxr-xr-x  2 root root   4096 Apr 11  2018 media
drwxr-xr-x  2 root root   4096 Apr 11  2018 home
drwx------  2 root root  16384 Dec 19  2018 lost+found
drwxr-xr-x  2 root root   4096 Dec 19  2018 boot
drwxr-xr-x  2 root root   4096 Dec 19  2018 dev
drwxr-xr-x  2 root root   4096 Dec 19  2018 proc
drwxr-xr-x  2 root root   4096 Dec 19  2018 run
drwxr-xr-x  2 root root   4096 Dec 19  2018 storage1
drwxr-xr-x  2 root root   4096 Dec 19  2018 sys
lrwxrwxrwx  1 root root      7 Dec 19  2018 bin -> usr/bin
lrwxrwxrwx  1 root root      9 Dec 19  2018 lib64 -> usr/lib64
lrwxrwxrwx  1 root root      7 Dec 19  2018 lib -> usr/lib
lrwxrwxrwx  1 root root      8 Dec 19  2018 sbin -> usr/sbin
drwxr-xr-x 13 root root   4096 Dec 19  2018 usr
drwxr-xr-x 20 root root   4096 Dec 19  2018 var
drwxr-xr-x  3 root root   4096 Jan  3  2020 opt
-rw-r--r--  1 root root 284329 Jan 20 14:29 .readahead
dr-xr-xr-x 19 root root   4096 Jan 20 14:29 .
drwxr-xr-x 88 root root   4096 Apr 25 10:58 etc
dr-xr-x--- 19 root root   4096 Apr 25 11:44 root
drwxrwxrwt  9 root root   4096 May 18 03:36 tmp

Here are all the LSI metadata headers in the disks, which are used by the LSI controller and now by the Linux software raid to assemble the devices md125, md126 and md127.
The CacheCade SSD device md125 with sda drive:

livecd ~ # mdadm -E /dev/sda
/dev/sda:
          Magic : de11de11
        Version : 01.00.00
Controller GUID : 4C534920:20202020:534B3833:35363737:31310000:20300000
                  (LSI     SK83567711)
 Container GUID : 4C534920:20202020:1000005D:10009363:494B870F:38C4623D
                  (LSI      12/19/18 11:35:43)
            Seq : 0000f399
  Redundant hdr : yes
  Virtual Disks : 2

      VD GUID[0] : 4C534920:20202020:1000005D:10009363:494B870F:1C61722B
                  (LSI      12/19/18 11:35:43)
         unit[0] : 0
        state[0] : Optimal, Consistent
   init state[0] : QuickInit in Progress
       access[0] : Read/Write
         Name[0] : storage1

      VD GUID[1] : 4C534920:20202020:1000005D:10009363:4CA5A634:2A1CFA65
                  (LSI      09/30/20 09:13:24)
         unit[1] : 1
        state[1] : Optimal, Consistent
   init state[1] : Not Initialised
       access[1] : Read/Write
         Name[1] : 
 Raid Devices[1] : 1 (3@0K)
   Chunk Size[1] : 2048 sectors
   Raid Level[1] : RAID0
  Device Size[1] : 937164800
   Array Size[1] : 937164800

 Physical Disks : 255
      Number    RefNo      Size       Device      Type/State
         0    4df1bbc6  11718361088K                 active/Online
         1    d1c4f0e2  11718361088K                 active/Online
         2    7926d1d9  11718361088K                 active/Online
         3    37272945  937164800K /dev/sda        active/Online

The 3 hard drives in RAID 5 with the data. Note the 

livecd ~ # mdadm -E /dev/sdb
/dev/sdb:
          Magic : de11de11
        Version : 01.00.00
Controller GUID : 4C534920:20202020:534B3833:35363737:31310000:20300000
                  (LSI     SK83567711)
 Container GUID : 4C534920:20202020:1000005D:10009363:494B870F:38C4623D
                  (LSI      12/19/18 11:35:43)
            Seq : 0000f399
  Redundant hdr : yes
  Virtual Disks : 2

      VD GUID[0] : 4C534920:20202020:1000005D:10009363:494B870F:1C61722B
                  (LSI      12/19/18 11:35:43)
         unit[0] : 0
        state[0] : Optimal, Consistent
   init state[0] : QuickInit in Progress
       access[0] : Read/Write
         Name[0] : storage1
 Raid Devices[0] : 3 (0@0K 1@0K 2@0K)
   Chunk Size[0] : 2048 sectors
   Raid Level[0] : RAID5
  Device Size[0] : 11718361088
   Array Size[0] : 23436722176

      VD GUID[1] : 4C534920:20202020:1000005D:10009363:4CA5A634:2A1CFA65
                  (LSI      09/30/20 09:13:24)
         unit[1] : 1
        state[1] : Optimal, Consistent
   init state[1] : Not Initialised
       access[1] : Read/Write
         Name[1] : 

 Physical Disks : 255
      Number    RefNo      Size       Device      Type/State
         0    4df1bbc6  11718361088K /dev/sdb        active/Online
         1    d1c4f0e2  11718361088K                 active/Online
         2    7926d1d9  11718361088K                 active/Online
         3    37272945  937164800K                 active/Online
livecd ~ # mdadm -E /dev/sdc
/dev/sdc:
          Magic : de11de11
        Version : 01.00.00
Controller GUID : 4C534920:20202020:534B3833:35363737:31310000:20300000
                  (LSI     SK83567711)
 Container GUID : 4C534920:20202020:1000005D:10009363:494B870F:38C4623D
                  (LSI      12/19/18 11:35:43)
            Seq : 0000f399
  Redundant hdr : yes
  Virtual Disks : 2

      VD GUID[0] : 4C534920:20202020:1000005D:10009363:494B870F:1C61722B
                  (LSI      12/19/18 11:35:43)
         unit[0] : 0
        state[0] : Optimal, Consistent
   init state[0] : QuickInit in Progress
       access[0] : Read/Write
         Name[0] : storage1
 Raid Devices[0] : 3 (0@0K 1@0K 2@0K)
   Chunk Size[0] : 2048 sectors
   Raid Level[0] : RAID5
  Device Size[0] : 11718361088
   Array Size[0] : 23436722176

      VD GUID[1] : 4C534920:20202020:1000005D:10009363:4CA5A634:2A1CFA65
                  (LSI      09/30/20 09:13:24)
         unit[1] : 1
        state[1] : Optimal, Consistent
   init state[1] : Not Initialised
       access[1] : Read/Write
         Name[1] : 

 Physical Disks : 255
      Number    RefNo      Size       Device      Type/State
         0    4df1bbc6  11718361088K                 active/Online
         1    d1c4f0e2  11718361088K /dev/sdc        active/Online
         2    7926d1d9  11718361088K                 active/Online
         3    37272945  937164800K                 active/Online
livecd ~ # mdadm -E /dev/sdd
/dev/sdd:
          Magic : de11de11
        Version : 01.00.00
Controller GUID : 4C534920:20202020:534B3833:35363737:31310000:20300000
                  (LSI     SK83567711)
 Container GUID : 4C534920:20202020:1000005D:10009363:494B870F:38C4623D
                  (LSI      12/19/18 11:35:43)
            Seq : 0000f399
  Redundant hdr : yes
  Virtual Disks : 2

      VD GUID[0] : 4C534920:20202020:1000005D:10009363:494B870F:1C61722B
                  (LSI      12/19/18 11:35:43)
         unit[0] : 0
        state[0] : Optimal, Consistent
   init state[0] : QuickInit in Progress
       access[0] : Read/Write
         Name[0] : storage1
 Raid Devices[0] : 3 (0@0K 1@0K 2@0K)
   Chunk Size[0] : 2048 sectors
   Raid Level[0] : RAID5
  Device Size[0] : 11718361088
   Array Size[0] : 23436722176

      VD GUID[1] : 4C534920:20202020:1000005D:10009363:4CA5A634:2A1CFA65
                  (LSI      09/30/20 09:13:24)
         unit[1] : 1
        state[1] : Optimal, Consistent
   init state[1] : Not Initialised
       access[1] : Read/Write
         Name[1] : 

 Physical Disks : 255
      Number    RefNo      Size       Device      Type/State
         0    4df1bbc6  11718361088K                 active/Online
         1    d1c4f0e2  11718361088K                 active/Online
         2    7926d1d9  11718361088K /dev/sdd        active/Online
         3    37272945  937164800K                 active/Online

Each drive reports it is in “state[1] : Optimal, Consistent” and that there is an ongoing process of “QuickInit in Progress” (Quick initialization, which means, the data are checked for consistency reasons).

The md127 is not important and cannot be accessed without the LSI hardware controller.

To summer it up – the data in RAID 5 array is perfectly accessible after removing the hardware controller AVAGO MegaRAID 9361-4i and connecting the SSD and the hard drives directly to built-in the SATA controller on the server’s mainboard (SATA controller: Intel Corporation Q170/Q150/B150/H170/H110/Z170/CM236 Chipset SATA Controller [AHCI Mode]). The RAID 5 array was created when the disks were connected to the AVAGO MegaRAID 9361-4i controller and using the LSI MEgaRAID Configuration Utility – AVAGO MegaRAID SAS-9361-4i with CacheCade – create a new virtual drive RAID5 with SSD caching

Categories: LSI
Tags: , , , , , , ,

Leave a Reply

Your email address will not be published. Required fields are marked *