mdadm assembles AVAGO/LSI MegaRAID controller RAID 5 array

It is possible to read data with the software Linux raid using mdadm tool from a RAID 5 array created with the hardware raid controller AVAGO MegaRAID 9361-4i (LSI SAS3108).

main menu
mdadm E sdb

Here, how a RAID 5 array with 3 hard drives and 1 SSD ( with CacheCade in write-through mode) is assembled by the mdadm and Linux software raid:

livecd ~ # cat /proc/mdstat 
Personalities : [raid0] [raid6] [raid5] [raid4] 
md125 : active raid0 sda[0]
      937164800 blocks super external:/md127/1 1024k chunks
md126 : active raid5 sdb[2] sdc[1] sdd[0]
      23436722176 blocks super external:/md127/0 level 5, 1024k chunk, algorithm 2 [3/3] [UUU]
      [==============>......]  resync = 72.0% (8438937704/11718361088) finish=336.8min speed=162234K/sec
md127 : inactive sdb[3](S) sda[2](S) sdd[1](S) sdc[0](S)
      2100568 blocks super external:ddf
unused devices: <none>

Note, it is essential that the CacheCade device is in write-through mode, which means the cache device is used only for reading and the data on the RAID array is consistent and written on it. The RAID 5 array was created here – AVAGO MegaRAID SAS-9361-4i with CacheCade – create a new virtual drive RAID5 with SSD caching. It seams possible for the data to be consistent if the CacheCade is write-back mode if there were few small writes and orderly shutdown prior to the removal of the AVAGO MegaRAID 9361-4i.
So, the above devices use proprietary LSI format, but here Linux software raid supports some of them:

  • md125 – the SSD device, which is a read cache only.
  • md126 – 3 hard drives in RAID 5 array.
  • md127 – logical device, which provides transparent interface to the

The important device is md126 and can be mounted under some live Linux CD/USB. Further, the md125 is a device, which has GPT partition table with 5 partitions:

livecd ~ # parted /dev/md126 --script print
Model: Linux Software RAID Array (md)
Disk /dev/md126: 24.0TB
Sector size (logical/physical): 512B/4096B
Partition Table: gpt
Disk Flags: 

Number  Start   End     Size    File system     Name                  Flags
 1      1049kB  211MB   210MB   fat16           EFI System Partition  boot, esp
 2      211MB   1285MB  1074MB  ext4                                  msftdata
 3      1285MB  23.9TB  23.9TB  ext4                                  msftdata
 4      23.9TB  24.0TB  53.7GB  ext4                                  msftdata
 5      24.0TB  24.0TB  16.8GB  linux-swap(v1)                        swap

Keep on reading!

Really bad performance when going from Write-Back to Write-Through in a LSI controller

Ever wonder what is the impact of write-through of an LSI controller in a real-world streaming server? Have no wonder anymore!

you can get several (multiple?) times slower with the write-through mode than if your controller were using the write-back mode of the cache

And it could happen any moment because when charging the battery of the LSI controller and you have set “No Write Cache if Bad BBU” the write-through would kick in. Of course, you can make a schedule for the battery charging/discharging process, but in general, it will happen and it will hurt your IO performance a lot!

In simple words a write operation is successful only if the controller confirms the write operation on all disks, no matter the data has already been in the cache.

This mode puts pressure on the disks and Write-Through is a known destroyer of hard disks! You can read a lot of administrator’s feedback on the Internet about crashed disks using write-through mode (and sometimes several simultaneously on one machine losing all your data even it would have redundancy with some of the RAID setups like RAID1, RAID5, RAID6, RAID10 and so).

srv ~ # sudo megacli -ldinfo -lall -aall
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name                :system
RAID Level          : Primary-1, Secondary-0, RAID Level Qualifier-0
Size                : 13.781 TB
Sector Size         : 512
Mirror Data         : 13.781 TB
State               : Optimal
Strip Size          : 128 KB
Number Of Drives per span:2
Span Depth          : 6
Default Cache Policy: WriteBack, ReadAdaptive, Direct, No Write Cache if Bad BBU
Current Cache Policy: WriteThrough, ReadAdaptive, Direct, No Write Cache if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy   : Disk's Default
Encryption Type     : None
Bad Blocks Exist: No
Is VD Cached: Yes
Cache Cade Type : Read Only

Exit Code: 0x00

As you can see our default cache policy is WriteBack and “No Write Cache if Bad BBU”, the BBU is not bad, but charging!
Keep on reading!