Increase the IO performance of big storage consisting of hard drives with the CacheCade feature with MegaRAID hardware controllers. Here are the commands to ensure adding an SSD to cache storage of hard drives and the whole output:
storcli64 /c0 add VD cachecade r0 drives=252:0 WB storcli64 /c0/v0 set ssdcaching=on
Summary and explanation
- The controller used is AVAGO MegaRAID SAS9361-4i
- Three 12T hard drives and 1 SSD 1T.
- RAID5 of 3 hard drive. The name of the virtual drive is v0.
- The CacheCade is in RAID0. The name of the virtual drive is v1.
- The CacheCade is in WriteBack mode. The WriteBack mode is dangerous when the CacheCade device is RAID0, because there is no redundancy, but it’s OK if the purpose of the machine is just a proxy cache. WriteBack mode leads to much more performance to gain from the cache device over slow device sotrage, but should be carefully planned when using RAID0 cache (or single) device.
- Using storcli – the cli command line tool to manage a (LSI) MegaRAID.
- Add CacheCade RAID0 device by the first command. It will create a virtual drive RAID0 with the name v1.
- Ensure the caching is enabled for the virtual device it should be by explicitly setting it with the second command. Adding only a CacheCade device may not add the cache virtual device to the existing virtual devices!
Output of a real world example
The status of the virtual device and the disks before the operation:
[root@srv ~]# /opt/MegaRAID/storcli/storcli64 /c0 show Generating detailed summary of the adapter, it may take a while to complete. CLI Version = 007.0709.0000.0000 Aug 14, 2018 Operating system = Linux 3.10.0-1127.19.1.el7.x86_64 Controller = 0 Status = Success Description = None Product Name = AVAGO MegaRAID SAS 9361-4i Serial Number = SK11111111 SAS Address = 500605b00ec0fdb0 PCI Address = 00:01:00:00 System Time = 09/30/2020 09:13:41 Mfg. Date = 08/29/18 Controller Time = 09/30/2020 09:13:07 FW Package Build = 24.21.0-0067 BIOS Version = 6.36.00.3_4.19.08.00_0x06180203 FW Version = 4.680.00-8392 Driver Name = megaraid_sas Driver Version = 07.710.50.00-rh1 Current Personality = RAID-Mode Vendor Id = 0x1000 Device Id = 0x5D SubVendor Id = 0x1000 SubDevice Id = 0x9363 Host Interface = PCI-E Device Interface = SAS-12G Bus Number = 1 Device Number = 0 Function Number = 0 Drive Groups = 1 TOPOLOGY : ======== ---------------------------------------------------------------------------- DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace TR ---------------------------------------------------------------------------- 0 - - - - RAID5 Optl N 21.827 TB dflt N N dflt N N 0 0 - - - RAID5 Optl N 21.827 TB dflt N N dflt N N 0 0 0 252:1 5 DRIVE Onln Y 10.913 TB dflt N N dflt - N 0 0 1 252:2 6 DRIVE Onln Y 10.913 TB dflt N N dflt - N 0 0 2 252:3 7 DRIVE Onln Y 10.913 TB dflt N N dflt - N ---------------------------------------------------------------------------- DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Dgrd=Degraded Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present TR=Transport Ready Virtual Drives = 1 VD LIST : ======= ------------------------------------------------------------------ DG/VD TYPE State Access Consist Cache Cac sCC Size Name ------------------------------------------------------------------ 0/0 RAID5 Optl RW Yes RAWBD - ON 21.827 TB storageA ------------------------------------------------------------------ Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|Dgrd=Degraded Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|TRANS=TransportReady|B=Blocked| Consist=Consistent|R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack| AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled Check Consistency Physical Drives = 4 PD LIST : ======= --------------------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type --------------------------------------------------------------------------------------- 252:0 4 UGood - 893.75 GB SATA SSD N N 512B SAMSUNG MZ7KM960HMJP-00005 U - 252:1 5 Onln 0 10.913 TB SATA HDD N N 512B ST12000NM0007-2A1101 U - 252:2 6 Onln 0 10.913 TB SATA HDD N N 512B ST12000NM0007-2A1101 U - 252:3 7 Onln 0 10.913 TB SATA HDD N N 512B ST12000NM001G-2MV103 U - --------------------------------------------------------------------------------------- EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info SeSz-Sector Size|Sp-Spun|U-Up|D-Down/PowerSave|T-Transition|F-Foreign UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded Cachevault_Info : =============== ------------------------------------ Model State Temp Mode MfgDate ------------------------------------ CVPM02 Optimal 26C - 2018/05/19 ------------------------------------ [root@srv ~]# /opt/MegaRAID/storcli/storcli64 /c0/v0 show CLI Version = 007.0709.0000.0000 Aug 14, 2018 Operating system = Linux 3.10.0-1127.19.1.el7.x86_64 Controller = 0 Status = Success Description = None Virtual Drives : ============== ------------------------------------------------------------------ DG/VD TYPE State Access Consist Cache Cac sCC Size Name ------------------------------------------------------------------ 0/0 RAID5 Optl RW Yes RAWBD - ON 21.827 TB storageA ------------------------------------------------------------------ Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|Dgrd=Degraded Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|TRANS=TransportReady|B=Blocked| Consist=Consistent|R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack| AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled Check Consistency
The SSD is in UGood mode, which is ready to be used in a virtual drive.
Add CacheCade device and show the slow device. In the example, the slow device did not enable the SSD caching device after it was added!
[root@srv ~]# /opt/MegaRAID/storcli/storcli64 /c0 add VD cachecade r0 drives=252:0 WB CLI Version = 007.0709.0000.0000 Aug 14, 2018 Operating system = Linux 3.10.0-1127.19.1.el7.x86_64 Controller = 0 Status = Success Description = Add VD Succeeded [root@srv ~]# /opt/MegaRAID/storcli/storcli64 /c0/v0 show CLI Version = 007.0709.0000.0000 Aug 14, 2018 Operating system = Linux 3.10.0-1127.19.1.el7.x86_64 Controller = 0 Status = Success Description = None Virtual Drives : ============== ------------------------------------------------------------------ DG/VD TYPE State Access Consist Cache Cac sCC Size Name ------------------------------------------------------------------ 0/0 RAID5 Optl RW Yes RAWBD - ON 21.827 TB storageA ------------------------------------------------------------------ Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|Dgrd=Degraded Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|TRANS=TransportReady|B=Blocked| Consist=Consistent|R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack| AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled Check Consistency [root@srv ~]# /opt/MegaRAID/storcli/storcli64 /c0/v0 set ssdcaching=on CLI Version = 007.0709.0000.0000 Aug 14, 2018 Operating system = Linux 3.10.0-1127.19.1.el7.x86_64 Controller = 0 Status = Success Description = None Detailed Status : =============== ----------------------------------------- VD Property Value Status ErrCd ErrMsg ----------------------------------------- 0 SSDCaching On Success 0 - -----------------------------------------
After setting the SSDCaching property the slow virtual drive’s “Cac” column will output the mode it uses the Cache drive.
The show command for the slow virtual drive and all virtual drives. The show commands for the all virtual drives (only /c0) will show information for the CacheCade drive, too.
[root@srv ~]# /opt/MegaRAID/storcli/storcli64 /c0/v0 show CLI Version = 007.0709.0000.0000 Aug 14, 2018 Operating system = Linux 3.10.0-1127.19.1.el7.x86_64 Controller = 0 Status = Success Description = None Virtual Drives : ============== ------------------------------------------------------------------ DG/VD TYPE State Access Consist Cache Cac sCC Size Name ------------------------------------------------------------------ 0/0 RAID5 Optl RW Yes RAWBD RW ON 21.827 TB storageA ------------------------------------------------------------------ Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|Dgrd=Degraded Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|TRANS=TransportReady|B=Blocked| Consist=Consistent|R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack| AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled Check Consistency [root@srv ~]# /opt/MegaRAID/storcli/storcli64 /c0 show Generating detailed summary of the adapter, it may take a while to complete. CLI Version = 007.0709.0000.0000 Aug 14, 2018 Operating system = Linux 3.10.0-1127.19.1.el7.x86_64 Controller = 0 Status = Success Description = None Product Name = AVAGO MegaRAID SAS 9361-4i Serial Number = SK11111111 SAS Address = 500605b00ec0fdb0 PCI Address = 00:01:00:00 System Time = 09/30/2020 09:14:29 Mfg. Date = 08/29/18 Controller Time = 09/30/2020 09:13:56 FW Package Build = 24.21.0-0067 BIOS Version = 6.36.00.3_4.19.08.00_0x06180203 FW Version = 4.680.00-8392 Driver Name = megaraid_sas Driver Version = 07.710.50.00-rh1 Current Personality = RAID-Mode Vendor Id = 0x1000 Device Id = 0x5D SubVendor Id = 0x1000 SubDevice Id = 0x9363 Host Interface = PCI-E Device Interface = SAS-12G Bus Number = 1 Device Number = 0 Function Number = 0 Drive Groups = 2 TOPOLOGY : ======== ---------------------------------------------------------------------------- DG Arr Row EID:Slot DID Type State BT Size PDC PI SED DS3 FSpace TR ---------------------------------------------------------------------------- 0 - - - - RAID5 Optl N 21.827 TB dflt N N dflt N N 0 0 - - - RAID5 Optl N 21.827 TB dflt N N dflt N N 0 0 0 252:1 5 DRIVE Onln Y 10.913 TB dflt N N dflt - N 0 0 1 252:2 6 DRIVE Onln Y 10.913 TB dflt N N dflt - N 0 0 2 252:3 7 DRIVE Onln Y 10.913 TB dflt N N dflt - N 1 - - - - Cac0 Optl N 893.75 GB dflt N N none N N 1 0 - - - RAID0 Optl N 893.75 GB dflt N N none N N 1 0 0 252:0 4 DRIVE Onln N 893.75 GB dflt N N none - N ---------------------------------------------------------------------------- DG=Disk Group Index|Arr=Array Index|Row=Row Index|EID=Enclosure Device ID DID=Device ID|Type=Drive Type|Onln=Online|Rbld=Rebuild|Dgrd=Degraded Pdgd=Partially degraded|Offln=Offline|BT=Background Task Active PDC=PD Cache|PI=Protection Info|SED=Self Encrypting Drive|Frgn=Foreign DS3=Dimmer Switch 3|dflt=Default|Msng=Missing|FSpace=Free Space Present TR=Transport Ready Virtual Drives = 2 VD LIST : ======= ------------------------------------------------------------------ DG/VD TYPE State Access Consist Cache Cac sCC Size Name ------------------------------------------------------------------ 0/0 RAID5 Optl RW Yes RAWBD RW ON 21.827 TB storageA 1/1 Cac0 Optl RW Yes RWBD - ON 893.75 GB ------------------------------------------------------------------ Cac=CacheCade|Rec=Recovery|OfLn=OffLine|Pdgd=Partially Degraded|Dgrd=Degraded Optl=Optimal|RO=Read Only|RW=Read Write|HD=Hidden|TRANS=TransportReady|B=Blocked| Consist=Consistent|R=Read Ahead Always|NR=No Read Ahead|WB=WriteBack| AWB=Always WriteBack|WT=WriteThrough|C=Cached IO|D=Direct IO|sCC=Scheduled Check Consistency Physical Drives = 4 PD LIST : ======= --------------------------------------------------------------------------------------- EID:Slt DID State DG Size Intf Med SED PI SeSz Model Sp Type --------------------------------------------------------------------------------------- 252:0 4 Onln 1 893.75 GB SATA SSD N N 512B SAMSUNG MZ7KM960HMJP-00005 U - 252:1 5 Onln 0 10.913 TB SATA HDD N N 512B ST12000NM0007-2A1101 U - 252:2 6 Onln 0 10.913 TB SATA HDD N N 512B ST12000NM0007-2A1101 U - 252:3 7 Onln 0 10.913 TB SATA HDD N N 512B ST12000NM001G-2MV103 U - --------------------------------------------------------------------------------------- EID-Enclosure Device ID|Slt-Slot No.|DID-Device ID|DG-DriveGroup DHS-Dedicated Hot Spare|UGood-Unconfigured Good|GHS-Global Hotspare UBad-Unconfigured Bad|Onln-Online|Offln-Offline|Intf-Interface Med-Media Type|SED-Self Encryptive Drive|PI-Protection Info SeSz-Sector Size|Sp-Spun|U-Up|D-Down/PowerSave|T-Transition|F-Foreign UGUnsp-Unsupported|UGShld-UnConfigured shielded|HSPShld-Hotspare shielded CFShld-Configured shielded|Cpybck-CopyBack|CBShld-Copyback Shielded Cachevault_Info : =============== ------------------------------------ Model State Temp Mode MfgDate ------------------------------------ CVPM02 Optimal 26C - 2018/05/19 ------------------------------------