It happened to one of our old LSI MegaRAID 2108 controllers (AOC-USAS2LP-H8iR (smc2108) with 36 disk, 32x2T and 4x8T) to freeze and most of the processes hang up with Disk sleep. The server was up, the network was working, but no login could be successful. A hard reset was executed with the IPMI KVM. The server started up, the MegaRAID controller booted with a warning that it was shutdown unexpectedly so there could be possible loss of data and to accept it by pressing any key or “C” to boot in the WebBIOS of the controller.
To summarize it up: the LSI controller hangs up when is in the following modes:
- Background Initialization
- Check Consistency
Aborting and disabling the modes above let out controller to work till replacement. If you experience any kind of strange disk hangs or freezes you can try our solution here! Check below to see how to do it yourself.
Here is the output after which the server hangs up and the disk array is unreadable for the operating system:
Nov 09 01:04:33 srv0 kernel: [ 2089.129528] sd 0:2:0:0: [sda] ABORT Nov 09 01:04:34 srv0 kernel: [ 2090.927714] sd 0:2:1:0: [sdb] ABORT Nov 09 01:07:34 srv0 kernel: [ 2269.128495] sd 0:2:0:0: [sda] ABORT Nov 09 01:07:34 srv0 kernel: [ 2270.108198] sd 0:2:1:0: [sdb] ABORT
Several resets and the situation got worse. At first the server had been up an running for at least 30-40 minutes and then it began to hang up the first 2-5 minutes! A fast update of the firmware was performed with the latest firmware, but with no success, the server’s controller still just hung up with no error and apparently not responding to the ABORT commands of the kernel, which would wait for a response infinitely – just issuing new commands after 2-3 minutes!
Several resets took us to understand that one of the virtual drives is in “Background Initialization” mode probably because of the several resets (therefore unexpected shutdowns for the controller).
[root@srv0 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -ldinfo -LALL -aALL Adapter 0 -- Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name : RAID Level : Primary-6, Secondary-0, RAID Level Qualifier-3 Size : 29.088 TB Parity Size : 3.635 TB State : Optimal Strip Size : 1.0 MB Number Of Drives : 18 Span Depth : 1 Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Encryption Type : None Default Power Savings Policy: Controller Defined Current Power Savings Policy: Automatic Can spin up in 1 minute: Yes LD has drives that support T10 power conditions: No LD's IO profile supports MAX power savings with cached writes: No Bad Blocks Exist: No Is VD Cached: Yes Cache Cade Type : Read Only Virtual Drive: 1 (Target Id: 1) Name : RAID Level : Primary-6, Secondary-0, RAID Level Qualifier-3 Size : 29.088 TB Parity Size : 3.635 TB State : Optimal Strip Size : 1.0 MB Number Of Drives : 18 Span Depth : 1 Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Ongoing Progresses: Background Initialization: Completed 5%, Taken 53 min. Encryption Type : None Default Power Savings Policy: Controller Defined Current Power Savings Policy: Automatic Can spin up in 1 minute: Yes LD has drives that support T10 power conditions: No LD's IO profile supports MAX power savings with cached writes: No Bad Blocks Exist: No Is VD Cached: Yes Cache Cade Type : Read Only
So we decided to abort it and not only abort but disable it because we do not want to start again in the future
[root@srv0 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -LDBI -Abort -LALL -aALL
Background Initialization on VD #0 is not in Progress.
Abort Background Initialization on Virtual Drive 1 (target id: 1) Success.
Exit Code: 0x00
[root@srv0 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -LDBI -Dsbl -LALL -aALL
Background Initialization is Disabled on VD 0 (target id: 0) on Adapter 0
Background Initialization is Disabled on VD 1 (target id: 1) on Adapter 0
Exit Code: 0x00
And to be absolutely sure we checked it with those three commands below. No background initialization at the moment and disabled setting in the controller’s configuration
[root@srv0 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -ldinfo -LALL -aALL
Adapter 0 -- Virtual Drive Information:
Virtual Drive: 0 (Target Id: 0)
Name :
RAID Level : Primary-6, Secondary-0, RAID Level Qualifier-3
Size : 29.088 TB
Is VD emulated : Yes
Parity Size : 3.635 TB
State : Optimal
Strip Size : 1.0 MB
Number Of Drives : 18
Span Depth : 1
Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
Virtual Drive: 1 (Target Id: 1)
Name :
RAID Level : Primary-6, Secondary-0, RAID Level Qualifier-3
Size : 29.088 TB
Is VD emulated : Yes
Parity Size : 3.635 TB
State : Optimal
Strip Size : 1.0 MB
Number Of Drives : 18
Span Depth : 1
Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU
Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU
Default Access Policy: Read/Write
Current Access Policy: Read/Write
Disk Cache Policy : Enabled
Encryption Type : None
Bad Blocks Exist: No
Is VD Cached: No
[root@srv0 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -LDBI -ProgDsply -LALL -aALL
Background Initialization on VD #0 is not in Progress.
Background Initialization on VD #0 is not in Progress.
Background Initialization on VD #1 is not in Progress.
Background Initialization on VD #1 is not in Progress.
Exit Code: 0x00
[root@srv0 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -LDBI -getSetting -LALL -aALL
Background Initialization is Disabled on VD 0 (target id: 0) on Adapter 0
Background Initialization is Disabled on VD 1 (target id: 1) on Adapter 0
Exit Code: 0x00
And WOW the server uptime got 3 hours and then it hung up again, the same situation! But this time it was clear it was not from the Background Initialization! The server was reset again (we made several power off/on and even unplugged the cords from the power grid) and issuing information command with the megacli utility showed that the controller this time is in
Check Consistency
[root@srv0 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -ldinfo -LALL -aALL Adapter 0 -- Virtual Drive Information: Virtual Drive: 0 (Target Id: 0) Name : RAID Level : Primary-6, Secondary-0, RAID Level Qualifier-3 Size : 29.088 TB Is VD emulated : Yes Parity Size : 3.635 TB State : Optimal Strip Size : 1.0 MB Number Of Drives : 18 Span Depth : 1 Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Ongoing Progresses: Check Consistency : Completed 0%, Taken 9 min. Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Virtual Drive: 1 (Target Id: 1) Name : RAID Level : Primary-6, Secondary-0, RAID Level Qualifier-3 Size : 29.088 TB Is VD emulated : Yes Parity Size : 3.635 TB State : Optimal Strip Size : 1.0 MB Number Of Drives : 18 Span Depth : 1 Default Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU Current Cache Policy: WriteBack, ReadAhead, Cached, Write Cache OK if Bad BBU Default Access Policy: Read/Write Current Access Policy: Read/Write Disk Cache Policy : Enabled Ongoing Progresses: Check Consistency : Completed 0%, Taken 9 min. Encryption Type : None Bad Blocks Exist: No Is VD Cached: No Exit Code: 0x00
So second guess for the night what if we abort and disable the “Check Consistency” and we did it fast enough before a controller freeze:
[root@srv0 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -LDCC -Abort -lall -aall
Abort Check Consistency on Virtual Drive 0 (target id: 0) Success.
Abort Check Consistency on Virtual Drive 1 (target id: 1) Success.
Exit Code: 0x00
[root@srv0 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -AdpCcSched -Dsbl -aALL
Adapter 0: Scheduled CC mode is set to Disabled.
Exit Code: 0x00
And then we discovered the controller had a “Check Consistency” scheduler with
[root@srv0 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -AdpCcSched -Info -aALL
Adapter #0
Operation Mode: Concurrent
Execution Delay: 168
Next start time: 11/17/2018, 03:00:00
Current State: Stopped
Number of iterations: 1
Number of VD completed: 2
Excluded VDs : None
Exit Code: 0x00
The above output is before the disable command for the “Check Consistency”. After you disable “Check Consistency” the output will be:
[root@srv0 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -AdpCcSched -Info -aALL
Adapter #0
Operation Mode: Disabled
Execution Delay: 168
Next start time: 07/28/2135, 02:00:00
Current State: Stopped
Number of iterations: 1
Number of VD completed: 2
Excluded VDs : None
Exit Code: 0x00
“Back to the future”!
What a coincidence 3 hours after we experienced the problem with the “Background Initialization” we had a weekly scheduled “Check Consistency” task! And the controller got freezed again! So be sure you disable the “Check Consistency”, too!
There is only one more mode: “Initialization”, which is only started after a new logical drive is created:
[root@srv0 ~]# /opt/MegaRAID/MegaCli/MegaCli64 -LDInit -Abort -LALL -aALL
Initialization on VD #0 is not in progress.
Initialization on VD #1 is not in progress.
Exit Code: 0x00
And this mode cannot start without manual administration command.