Update Supermicro BMC/IPMI Firmware – under Linux console

Here you will see our log of upgrading the Supermicro IPMI firmware with the cli tool included in the firmware package for your IPMI unit under Linux console.
If your server has built-in IPMI unit in the motherboard there will be a firmware for it next to the BIOS firmware in the Supermicro site. You go to the page of your Supermicro page and on the left part you have also the BIOS and IPMI firmware links. The IPMI firmware package has a Windows/DOS and Linux executable files to flash the firmware under the console.
So here we flash a new firmware to our motherboard is X10SLM+-F.

Here you can see left “Links & Resources” and click on ” BMC/IPMI Firmware” to download the latest IPMI firmware for your motherboard.

main menu
Motherboard X10SLM+-F page in Supermicro site

Upload the downloaded file in your server.

STEP 1) Unpack the firmware file downloaded from Supermicro site.

Here we include the verbose output of “tar” so you can see what files are included. The files we use here are highlighted.

[root@srv ~]# ls -altr
total 25904
drwxr-xr-x. 94 root root    81920  3 Feb 17,42 ..
drwxr-xr-x.  2 root root     4096  3 Feb 17,43 .
-rw-r--r--.  1 root root 26432121  3 Feb 17,43 REDFISH_X10_372.zip
[root@srv ~]# mkdir REDFISH_X10_372
[root@srv ~]# cd REDFISH_X10_372/
[root@srv ~/REDFISH_X10_372]# unzip ../REDFISH_X10_372.zip 
Archive:  ../REDFISH_X10_372.zip
  inflating: Redfish_Ref_Guide_2.0.pdf  
   creating: 2.07/
   creating: 2.07/dos/
  inflating: 2.07/dos/AdUpdate.exe   
   creating: 2.07/linux/
   creating: 2.07/linux/x32/
  inflating: 2.07/linux/x32/AlUpdate  
   creating: 2.07/linux/x64/
  inflating: 2.07/linux/x64/AlUpdate  
  inflating: 2.07/ReleaseNote.txt    
   creating: 2.07/windows/
   creating: 2.07/windows/x32/
  inflating: 2.07/windows/x32/AwUpdate.exe  
  inflating: 2.07/windows/x32/phymem32.sys  
  inflating: 2.07/windows/x32/pmdll32.dll  
  inflating: 2.07/windows/x32/superbmc32.sys  
  inflating: 2.07/windows/x32/superdll_ssm32.dll  
   creating: 2.07/windows/x64/
  inflating: 2.07/windows/x64/AwUpdate.exe  
  inflating: 2.07/windows/x64/phymem64.sys  
  inflating: 2.07/windows/x64/pmdll64.dll  
  inflating: 2.07/windows/x64/superbmc.sys  
  inflating: 2.07/windows/x64/superdll_ssm64.dll  
  inflating: IPMI Firmware Update_NEW.doc  
  inflating: REDFISH_X10_372.bin

There are 5 version of the flash utility Linux 32bit and 64bit, Windows 32bit and 64bit and a dos version.

STEP 2) Flash the BCM/IPMI firmware.

We choose here not to preserve configuration, because some old features might be incompatible with the new one. It is not mandatory to do it in fact we also tested with “to preserve” the old configuration and we have no problems afterwards.
We do not change almost anything in the IPMI configuration except admin password and the network settings and when flashing under the OS you have the ability to reconfigure it after the flashing process. Your server is up and running and you can use “ipmitool” to configure the IPMI module.
The whole process took about 15 minutes.

[root@srv ~/REDFISH_X10_372]# 2.07/linux/x64/AlUpdate -f REDFISH_X10_372.bin -r n
sh: cls: command not found
*****************************************************************************
* ATEN Technology, Inc.                                                     *
*****************************************************************************
* FUNCTION   :  IPMI FIRMWARE UPDATE UTILITY                                *
* VERSION    :  2.07                                                        *
* BUILD DATE :  Jul 13 2016                                                 *
* USAGE      :                                                              *
*             (1)Update FIRMWARE : AlUpdate -f filename.bin [OPTION]        *
*             (2)Dump FIRMWARE   : AlUpdate -d filename                     *
*             (3)Restore CONFIG  : AlUpdate -c -f filename.bin              *
*             (4)Backup CONFIG   : AlUpdate -c -d filename.bin              *
*****************************************************************************
* OPTION                                                                    *
*   -i the IPMI channel, currently, kcs and lan are supported               *
* LAN channel specific arguments                                            *
*   -h remote BMC address and RMCP+ port, (default port is 623)             *
*   -u IPMI user name                                                       *
*   -p IPMI password correlated to IPMI user name                           *
*   -r Preserve Configuration (default is Preserve)                         *
*      n:No Preserve, reset to factory default settings                     *
*      y:Preserve, keep all of the settings                                 *
*   -c IPMI configuration backup/restore                                    *
*      -f [restore.bin] Restore configurations                              *
*      -d [backup.bin] Backup configurations                                *
*****************************************************************************
* EXAMPLE                                                                   *
*   we like to upgrade firmware through KCS channel                         *
*   AlUpdate -f fwuperade.bin -i kcs -r y                                   *
*   AlUpdate -d fwdump.bin -i kcs -r y                                      *
*                                                                           *
*   we like to restore/backup IPMI config through KCS channel               *
*   AlUpdate -c -f restore.bin -i kcs -r y                                  *
*   AlUpdate -c -d backup.bin -i kcs -r y                                   *
*                                                                           *
*   we like to upgrade firmware through LAN channel with                    *
*   - BMC IP address 10.11.12.13 port 623                                   *
*   - IPMI username is usr                                                  *
*   - Password for alice is pwd                                             *
*   - Preserve Configuration                                                *
*   AlUpdate -f fw.bin -i lan -h 10.11.12.13 623 -u usr -p pwd -r y         *
*   AlUpdate -d fwdump.bin -i lan -h 10.11.12.13 623 -u usr -p pwd -r y     *
*                                                                           *
*   we like to restore/backup IPMI config through LAN channel with          *
*   - BMC IP address 10.11.12.13 port 623                                   *
*   - IPMI username is usr                                                  *
*   - Password for alice is pwd                                             *
*   - Preserve Configuration                                                *
*   AlUpdate -c -f fw.bin -i lan -h 10.11.12.13 623 -u usr -p pwd           *
*   AlUpdate -c -d fwdump.bin -i lan -h 10.11.12.13 623 -u usr -p pwd       *
*****************************************************************************

2.07/linux/x64/AlUpdate -f REDFISH_X10_372.bin -r n 
Try open dev ipmi0....
Check if this file is valid................
If the FW update fails,PLEASE TRY AGAIN
Load part 0   126008 bytes, [Ok]                       
Load part 1 14635008 bytes, [Ok]                       
Load part 2  1537585 bytes, [Ok]                       
Load part 3  8081440 bytes, [Ok]                       
Load part 4   262144 bytes, [Ok]                       



                 If the FW update fails. PLEASE WAIT 5 MINS AND REMOVE THE AC...
new firmware is updating...100%
Update Complete,Please wait for BMC reboot, about 1 min                       
[root@srv ~/REDFISH_X10_372]# 

All the lines starting with “Load part” will shows progress percentages like:

Load part 1 14635008 bytes,  4137K bytes   29%"

And the line starting with “new firmware is updating…” also shows like:

new firmware is updating...28%

In dmesg you can see your IPMI module resets:

[root@conv1 ~]# dmesg
[1954154.242383] usb 3-7: USB disconnect, device number 2
[1954154.242385] usb 3-7.1: USB disconnect, device number 3
[1954185.337154] usb 3-7: new high-speed USB device number 4 using xhci_hcd
[1954185.501356] usb 3-7: New USB device found, idVendor=0557, idProduct=7000
[1954185.501358] usb 3-7: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[1954185.501879] hub 3-7:1.0: USB hub found
[1954185.501923] hub 3-7:1.0: 4 ports detected
[1954185.899168] usb 3-7.1: new low-speed USB device number 5 using xhci_hcd
[1954185.999375] usb 3-7.1: New USB device found, idVendor=0557, idProduct=2419
[1954185.999376] usb 3-7.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[1954186.000708] input: HID 0557:2419 as /devices/pci0000:00/0000:00:14.0/usb3/3-7/3-7.1/3-7.1:1.0/input/input10
[1954186.051346] hid-generic 0003:0557:2419.0003: input,hidraw0: USB HID v1.00 Keyboard [HID 0557:2419] on usb-0000:00:14.0-7.1/input0
[1954186.052050] input: HID 0557:2419 as /devices/pci0000:00/0000:00:14.0/usb3/3-7/3-7.1/3-7.1:1.1/input/input11
[1954186.052423] hid-generic 0003:0557:2419.0004: input,hidraw1: USB HID v1.00 Mouse [HID 0557:2419] on usb-0000:00:14.0-7.1/input1
[1954199.668503] usb 3-7.1: USB disconnect, device number 5
[1954201.450533] usb 3-7.1: new low-speed USB device number 6 using xhci_hcd
[1954201.550755] usb 3-7.1: New USB device found, idVendor=0557, idProduct=2419
[1954201.550756] usb 3-7.1: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[1954201.552044] input: HID 0557:2419 as /devices/pci0000:00/0000:00:14.0/usb3/3-7/3-7.1/3-7.1:1.0/input/input12
[1954201.602658] hid-generic 0003:0557:2419.0005: input,hidraw0: USB HID v1.00 Keyboard [HID 0557:2419] on usb-0000:00:14.0-7.1/input0
[1954201.603372] input: HID 0557:2419 as /devices/pci0000:00/0000:00:14.0/usb3/3-7/3-7.1/3-7.1:1.1/input/input13
[1954201.603729] hid-generic 0003:0557:2419.0006: input,hidraw1: USB HID v1.00 Mouse [HID 0557:2419] on usb-0000:00:14.0-7.1/input1

Mount and boot ISO file from windows share in Supermicro IPMI Virtual media (CD-ROM)

This tutorial shows you how to load any ISO file in the virtual CD/DVD-ROM of a Supermicro server and boot from it if it is bootable. You could install operating systems from a ISO file or just share a data. Here we use an installation DVD disk of CentOS 7 to boot. This tutorial expect there is a windows (samba) share on the local network if you need to do it you can check here – Configure and mount samba share in Supermicro IPMI Virtual media (CD-ROM)
Loading an ISO disk in the (virtual) CD/DVD-ROM device attached to your server could be of a great help to the system administrators, you could use diagnostic disks, update disks (BIOS and firmware of devices), install multiple operating systems including MS Windows, share data and many more! You could use ISO disk right from your computer or from a server next to the server you want to use the ISO to speed up access.

STEP 1) When you are logged in the Supermicro IPMI web interface click on Virtual Media menu and then CD-ROM Image.

main menu
CD-ROM Image

Keep on reading!

Configure and mount samba share in Supermicro IPMI Virtual media (CD-ROM)

Having many Supermicro servers we used multiple times Virtual Media to install, diagnose and rescue Supermicro server. It is really simple to open the Console Redirection – java web start and just mount the ISO file from Virtual Media -> ISO File, but this way if your server is not local to your network and it is located in a colocation as it should be the connection is slow and in many cases bogus! Because it uses UDP it happened many times to remount or just to lose the connection and the media to disappear in the middle of the booting/loading process from the installation media, for example. And probably you have noticed there is additional option in the web interface of mounting ISO file from a windows share. Of course, in linux world it might be samba share as in our case and the share could be easily configured on a server in your colocation.
And we noticed was several times we have no problems using it, but some time the share could not be even saved and respectively could not be mounted. No error reported, just the edit boxes resets to blank and apparently everything was the same as the previous box, which it worked as a charm!
Here are the steps to enable one of your CentOS 7 servers to share a resource and to use it in your Supermicro IPMI KVM. The server is selinux enabled and the policy is Enforcing.

STEP 1) Install samba server in CentOS 7

[root@srv0 ~]# yum -y install samba samba-client samba-common policycoreutils-python
Loaded plugins: fastestmirror
Determining fastest mirrors
epel/x86_64/metalink                                                                                                                           |  30 kB  00:00:00     
 * base: mirrors.neterra.net
 * epel: mirrors.neterra.net
 * extras: mirrors.neterra.net
 * updates: mirrors.neterra.net
base                                                                                                                                           | 3.6 kB  00:00:00     
epel                                                                                                                                           | 3.2 kB  00:00:00     
extras                                                                                                                                         | 3.4 kB  00:00:00     
updates                                                                                                                                        | 3.4 kB  00:00:00     
(1/7): base/7/x86_64/group_gz                                                                                                                  | 166 kB  00:00:00     
(2/7): epel/x86_64/group_gz                                                                                                                    |  88 kB  00:00:00     
(3/7): base/7/x86_64/primary_db                                                                                                                | 5.9 MB  00:00:00     
(4/7): extras/7/x86_64/primary_db                                                                                                              | 147 kB  00:00:00     
(5/7): updates/7/x86_64/primary_db                                                                                                             | 2.0 MB  00:00:00     
(6/7): epel/x86_64/updateinfo                                                                                                                  | 932 kB  00:00:00     
(7/7): epel/x86_64/primary                                                                                                                     | 3.5 MB  00:00:00     
epel                                                                                                                                                      12584/12584
Package samba-client-4.7.1-6.el7.x86_64 already installed and latest version
Package samba-common-4.7.1-6.el7.noarch already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package samba.x86_64 0:4.7.1-6.el7 will be installed
--> Processing Dependency: samba-libs = 4.7.1-6.el7 for package: samba-4.7.1-6.el7.x86_64
--> Processing Dependency: samba-common-tools = 4.7.1-6.el7 for package: samba-4.7.1-6.el7.x86_64
--> Processing Dependency: libxattr-tdb-samba4.so(SAMBA_4.7.1)(64bit) for package: samba-4.7.1-6.el7.x86_64
--> Processing Dependency: libaio.so.1(LIBAIO_0.4)(64bit) for package: samba-4.7.1-6.el7.x86_64
--> Processing Dependency: libaio.so.1(LIBAIO_0.1)(64bit) for package: samba-4.7.1-6.el7.x86_64
--> Processing Dependency: libxattr-tdb-samba4.so()(64bit) for package: samba-4.7.1-6.el7.x86_64
--> Processing Dependency: libaio.so.1()(64bit) for package: samba-4.7.1-6.el7.x86_64
--> Running transaction check
---> Package libaio.x86_64 0:0.3.109-13.el7 will be installed
---> Package samba-common-tools.x86_64 0:4.7.1-6.el7 will be installed
---> Package samba-libs.x86_64 0:4.7.1-6.el7 will be installed
--> Processing Dependency: libpytalloc-util.so.2(PYTALLOC_UTIL_2.1.9)(64bit) for package: samba-libs-4.7.1-6.el7.x86_64
--> Processing Dependency: libpytalloc-util.so.2(PYTALLOC_UTIL_2.1.6)(64bit) for package: samba-libs-4.7.1-6.el7.x86_64
--> Processing Dependency: libpytalloc-util.so.2(PYTALLOC_UTIL_2.0.6)(64bit) for package: samba-libs-4.7.1-6.el7.x86_64
--> Processing Dependency: libpytalloc-util.so.2()(64bit) for package: samba-libs-4.7.1-6.el7.x86_64
--> Running transaction check
---> Package pytalloc.x86_64 0:2.1.10-1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================================
 Package                                        Arch                               Version                                     Repository                        Size
======================================================================================================================================================================
Installing:
 samba                                          x86_64                             4.7.1-6.el7                                 base                             661 k
Installing for dependencies:
 libaio                                         x86_64                             0.3.109-13.el7                              base                              24 k
 pytalloc                                       x86_64                             2.1.10-1.el7                                base                              17 k
 samba-common-tools                             x86_64                             4.7.1-6.el7                                 base                             463 k
 samba-libs                                     x86_64                             4.7.1-6.el7                                 base                             275 k

Transaction Summary
======================================================================================================================================================================
Install  1 Package (+4 Dependent packages)

Total download size: 1.4 M
Installed size: 3.8 M
Downloading packages:
(1/5): libaio-0.3.109-13.el7.x86_64.rpm                                                                                                        |  24 kB  00:00:00     
(2/5): pytalloc-2.1.10-1.el7.x86_64.rpm                                                                                                        |  17 kB  00:00:00     
(3/5): samba-4.7.1-6.el7.x86_64.rpm                                                                                                            | 661 kB  00:00:00     
(4/5): samba-common-tools-4.7.1-6.el7.x86_64.rpm                                                                                               | 463 kB  00:00:00     
(5/5): samba-libs-4.7.1-6.el7.x86_64.rpm                                                                                                       | 275 kB  00:00:00     
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                 4.5 MB/s | 1.4 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : pytalloc-2.1.10-1.el7.x86_64                                                                                                                       1/5 
  Installing : samba-libs-4.7.1-6.el7.x86_64                                                                                                                      2/5 
  Installing : samba-common-tools-4.7.1-6.el7.x86_64                                                                                                              3/5 
  Installing : libaio-0.3.109-13.el7.x86_64                                                                                                                       4/5 
  Installing : samba-4.7.1-6.el7.x86_64                                                                                                                           5/5 
  Verifying  : libaio-0.3.109-13.el7.x86_64                                                                                                                       1/5 
  Verifying  : samba-libs-4.7.1-6.el7.x86_64                                                                                                                      2/5 
  Verifying  : samba-common-tools-4.7.1-6.el7.x86_64                                                                                                              3/5 
  Verifying  : samba-4.7.1-6.el7.x86_64                                                                                                                           4/5 
  Verifying  : pytalloc-2.1.10-1.el7.x86_64                                                                                                                       5/5 

Installed:
  samba.x86_64 0:4.7.1-6.el7                                                                                                                                          

Dependency Installed:
  libaio.x86_64 0:0.3.109-13.el7        pytalloc.x86_64 0:2.1.10-1.el7        samba-common-tools.x86_64 0:4.7.1-6.el7        samba-libs.x86_64 0:4.7.1-6.el7       

Complete!
[root@srv0 ~]#

STEP 2) Configure samba server CentOS 7 for the purpose of using it in IPMI Virtual share.

We are going to use a share without login credentials, because our KVM IP are always local ones and accessed via a vpn network and in addition only the network of the IPMI IPs could access the share (the samba server has a firewall configured).
Set the configuration file of the samba server – one directory storing the files shared with no login credentials (no username/password means anonymous login).
The configuration file is:

/etc/samba/smb.conf

[global]
workgroup = WINSHARE
server string = Samba Server %v
netbios name = centossrv
security = user
map to guest = bad user
dns proxy = no
#============================ Share Definitions ============================== 
[share]
path = /mnt/storage1/samba
browsable =yes
writable = no
guest ok = yes
read only = yes

As you can see we use “/mnt/storage1/samba” for our directory where the ISO files will be located. Change this path if you want to put your ISO files somewhere else.
Set the right permissions for the directory and selinux (if you server is not selinux enabled, you could skip the selinux part) and run the samba daemon:

[root@srv0 ~]# mkdir /mnt/storage1/samba
[root@srv0 ~]# chown -R nobody:nobody /mnt/storage1/samba/
[root@srv0 ~]# semanage fcontext -a -t samba_share_t '/mnt/storage1/samba(/.*)?'
[root@srv0 ~]# restorecon -Rv /mnt/storage1/samba/
restorecon reset /mnt/storage1/samba context unconfined_u:object_r:unlabeled_t:s0->unconfined_u:object_r:samba_share_t:s0
[root@srv0 ~]# cd /mnt/storage1/samba/
[root@srv0 samba]# wget http://mirror.leaseweb.com/centos/7.5.1804/isos/x86_64/CentOS-7-x86_64-Minimal-1804.iso
--2018-06-01 14:15:42--  http://mirror.leaseweb.com/centos/7.5.1804/isos/x86_64/CentOS-7-x86_64-Minimal-1804.iso
Resolving mirror.leaseweb.com (mirror.leaseweb.com)... 37.58.58.140, 2a00:c98:2030:a034::21
Connecting to mirror.leaseweb.com (mirror.leaseweb.com)|37.58.58.140|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 950009856 (906M) [application/octet-stream]
Saving to: ‘CentOS-7-x86_64-Minimal-1804.iso’

100%[============================================================================================================================>] 950,009,856 40.1MB/s   in 33s    

2018-06-01 14:16:15 (27.3 MB/s) - ‘CentOS-7-x86_64-Minimal-1804.iso’ saved [950009856/950009856]

[root@srv0 samba]# chown nobody:nobody CentOS-7-x86_64-Minimal-1804.iso
[root@srv0 samba]# systemctl start smb
[root@srv0 samba]# systemctl status smb
● smb.service - Samba SMB Daemon
   Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-06-01 14:17:57 UTC; 20s ago
 Main PID: 31961 (smbd)
   Status: "smbd: ready to serve connections..."
    Tasks: 4
   Memory: 19.0M
   CGroup: /system.slice/smb.service
           ├─31961 /usr/sbin/smbd --foreground --no-process-group
           ├─31964 /usr/sbin/smbd --foreground --no-process-group
           ├─31965 /usr/sbin/smbd --foreground --no-process-group
           └─31966 /usr/sbin/smbd --foreground --no-process-group

Jun 01 14:17:56 srv0@local systemd[1]: Starting Samba SMB Daemon...
Jun 01 14:17:57 srv0@local smbd[31961]: [2018/06/01 14:17:57.761913,  0] ../lib/util/become_daemon.c:124(daemon_ready)
Jun 01 14:17:57 srv0@local smbd[31961]:   STATUS=daemon 'smbd' finished starting up and ready to serve connections
Jun 01 14:17:57 srv0@local systemd[1]: Started Samba SMB Daemon.

Configure the firewall to allow only connections from the IPMI KVM IP networks (or a single IP if you need to expose it in the Internet). You can work with build in zone “trusted”, but here we prefer more generic approach, which could be used not only for local IP networks but for real IPs:

[root@srv0 samba]# firewall-cmd --new-zone=smbshare --permanent
success
[root@srv0 samba]# firewall-cmd --zone=smbshare --add-source=192.168.7.0/24 --permanent
success
[root@srv0 samba]# firewall-cmd --zone=smbshare --add-service=samba --permanent
success
[root@srv0 samba]# firewall-cmd --zone=smbshare --add-service=samba-client --permanent
success
[root@srv00 samba]# firewall-cmd --reload
success
[root@srv00 samba]# firewall-cmd --zone=smbshare --list-all
smbshare (active)
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 192.168.7.0/24
  services: samba samba-client
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

[root@srv00 samba]#

STEP 3) Mount a disk in IPMI Virutal Media and boot from it

You could check our additional tutorial for it here –

STEP 3.1) Fill “Share host” and “Path image” only, because we configured our samba share without a password

main menu
Image on Windows Share

STEP 3.2) Upon a successful configuration saving you’ll get a confirmation dialog.

If no confirmation is shown you would not be able to mount the share, check out our Troubleshooting below!

main menu
Save configuration of Image on Windows Share

STEP 3.3) After saving the configuration click on “Mount” to load your ISO file in the virtual CD-ROM.

main menu
Mount Image on Windows Share

STEP 3.4) This dialog is always shown no matter there is such share or not: “Please check the device status to confirm whether the image is mounted/unmounted.”

main menu
Check device status

STEP 3.5) If the Supermicro IPMI accessed successfully the share resource it would mount it (load the ISO file as s CD in the virtual CD-ROM).

If not the three Device 1,2,3 will show the same: “No disk emulation set.”, which means the samba share is not accessible, check the permissions for the file and the firewall first (and the logs files, too).

main menu
There is an iso file mounted

* Troubleshooting

– if you click on button “Save” and just nothing happen – no dialog for successful saving and no dialog at all, probably there is a Javascript error, try to open the web interface from a different browser with clean history or click ctrl+F5 when loading the Virtual Media page! More in Cannot save and mount a Supermicro IPMI Virtual media mount – javascript error?

Cannot save and mount a Supermicro IPMI Virtual media mount – javascript error?

If you have multiple Supermicro servers with many different versions of IPMI KVM software installed it could happen your browser to cache some of the JavaScript and other static content to reuse them, but it could very unpleasant when you have different version of IPMI software on different servers and you might notice strange behavior of the web interface! Especially if you use ssh tunneling to access your multiple Supermicro IPMI KVMs from local IP on your computerTunneling the IPMI/KVM ports over ssh (supermicro ipmi ports)

One of the big problems we have when we wanted to mount a Virtual Media

from a windows share (samba share in our case) in IPMI KVM web interface -> Virtual Media -> CD-ROM image -> Save and when the Save is clicked just nothing happen (sometimes it triggers a reload of the iframe) – no error nothing and no confirmation for successful save! At first it seems the web interface accepted the “Share Host” and “Path to Image”:

main menu
Fill “Share Host” and “Path to Image”

but when you click “Mount” it does not mount the media:
main menu
no disk emulation set

and when you reload the CD-ROM image page you get again blank edit boxes or (the old values):
main menu
Old values

Probably a refresh will get the values blank:
main menu
Blank edit boxes

And if you check your browser console you’ll see there is a JavaScript error:

Uncaught ReferenceError: FocusOnErrorSpecificCharSet3 is not defined

The error might be different, this was in our case. the problem was

the browser cached “https://192.168.0.170/js/utils.js”

from one of the previous servers and there the version of the IPMI KVM software was different and apparently the

/js/utils.js

was throwing an error and not working (this function did not exist in some older Supermicro IPMI KVM versions, the file is there but it is slightly different). The solution is so simple!

Just refresh the page with CTRL+F5 or delete the history or use another browser.

Such a simple problem, but could lead to big problems if you try to use the mount virtual media. In fact look for problems in the JavaScript if you cannot save the configuration in the “Share Host” and “Path to Image”, because when saving the IPMI do not check if there is a live “Share Host” with a windows/samba share and an image there, the software just check for special in “Shared” characters like:

var SpeficCharFilter = /[,; &'"<>\\=$|^?*~`()\[\]\{\}#%]/;

And for the password:

var SpeficCharFilter = /[,; &'"<>\\=$|^?~`()\[\]\{\}#%]/;

But in both cases you’ll get an alert with an error.

So to summer it up if you put IP and a path to the windows share of Virtual Media and click “Save” and nothing happen – no confirmation for successful saving you got a JavaScript error and probably your browser cached one of the JavaScript files, the solution is simple just refresh with CTRL+F5 or load from different browser!
We often use ssh tunneling for IPMI KVM accessTunneling the IPMI/KVM ports over ssh (supermicro ipmi ports) and different version of the static files of the supermicro IPMI web interface are cached locally, which as you can see could have really bad consequences!

Working Save button

click Save button

main menu
Fill “Share Host” and “Path to Image”

Confirmation when everything is OK

main menu
The confirmation of image on windows share has been successfully set.

SUPERMICRO IPMI/KVM module tips – reset the unit and the admin password

After the previous howto “SUPERMICRO IPMI to use one of the one interfaces or dedicated LAN port” (in the howto is showed how to install the needed tool for managing the IPMI/KVM unit under console) of setting the network configuration there are a couple of interesting and important tips when working with the IPMI/KVM module. Here are they are:

  1. Reset IPMI/KVM module – sometimes it happen the keyboard or mouse not to work when the Console Redirection is loaded, it is easy to reset the unit from the web interface, but there are case when the web interface is not working – so ssh to your server and try one of the following commands:
    * warm reset – it’s like a reboot, inform the IPMI/KVM to reboot itself.

    ipmitool -I open bmc reset warm
    

    It does not work in all situations! So try a cold reset
    * cold reset – resets the IPMI/KVM, it’s like unplug and plug the power to the unit.

    ipmitool -I open bmc reset cold
    
  2. Reset the configuration of an IPMI/KVM module to factory defaults. It is useful when something goes wrong when upgrading the firmware of the unit and the old configuration is not supported or it says it is, but at the end the unit does not work properly. In rare cases it might help when the KVM (Keyboard, Video, Monitor part aka Console redirection does not work)
    Here is the command for resetting to factory defaults:

    ipmitool -I open raw 0x3c 0x40
    
  3. Reset admin password – reset the password for the administrator login of the IPMI/KVM unit. It’s trivial losing the password so with the help of the local console to the server you can reset the password to a simple one and then change it from the web interface.
    ipmitool -I open user set password 2 ADMIN
    

    The number “2” is the ID of the user, check it with:

    [root@srv0 ~]# ipmitool -I open user list
    ID  Name             Callin  Link Auth  IPMI Msg   Channel Priv Limit
    1                    true    false      false      Unknown (0x00)
    2   ADMIN            true    false      false      Unknown (0x00)
    3                    true    false      false      Unknown (0x00)
    4                    true    false      false      Unknown (0x00)
    5                    true    false      false      Unknown (0x00)
    6                    true    false      false      Unknown (0x00)
    7                    true    false      false      Unknown (0x00)
    8                    true    false      false      Unknown (0x00)
    9                    true    false      false      Unknown (0x00)
    10                   true    false      false      Unknown (0x00)
    

    Sometimes if a hacker got to your IPMI/KVM you could see the user table with the above command. There was a serious bug aka backdoor in some of these units, the ID of the ADMIN user or even the username could be changed, so you should use the list command to list the current user table.
    Use set name to set the username of the user.

    ipmitool -I open user set name 2 ADMIN
    
  4. Set a new network configuration. It’s worth mentioning again the howto for this purpose – “SUPERMICRO IPMI to use one of the one interfaces or dedicated LAN port

All commands using the network option of the ipmitool

ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN bmc reset warm
ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN bmc reset cold
ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN raw 0x3c 0x40
ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN user set password 2 ADMIN
ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN user list

The IP 192.168.7.150 is the IP of your IPMI/KVM module, which you want to change with the above commands.

Tunneling the IPMI/KVM ports over ssh (supermicro ipmi ports)

The best security for the remote management unit in your server such as IPMI/KVM is to have local IP. All IPMI/KVM IP should be switched to a separated switch and a local sub-network used for the LAN Settings. So to be able to connect to the IPMI/KVM module you need a VPN connection to gain access to the local sub-network used for your servers’ management modules. However, sometimes the VPN cannot be used or it just happened the server is down, or you are at a place restricting unknown ports (or ports above 1024), which your VPN uses (that’s why the VPN server should use only one port from the most popular – 80, 443, but that’s a thing for another howto…) and so on. So you end with no ability to connect to the VPN server or you think you do not need at all a VPN server, because you always could use

openssh

to do the trick of tunneling ports from your computer to the IPMI/KVM module of your server through a server, which has an access to the local sub-network of the IPMI/KVM modules.

So here is what you need to get to the remote management of your server just using ssh for tunneling:

STEP 1) A server, which has access to the IP network of the IPMI/KVM modules.

Let’s say you set to all your servers’ IPMI/KVM modules IPs from network 192.168.7.0/24, so your server must have an IP from 192.168.7.0/24, for example 192.168.7.1, add it as an alias or to a dedicated LAN connected to the switch, in which of all your IPMI/KVM modules are plugged in. This server will be used as a transfer point to a selected IPMI/KVM IP.

STEP 2) Tunnel local selected ports using ssh to the server from STEP 1)

Use this command:

ssh -N -L 127.0.0.1:80:[IPMI-IP]:80 -L 127.0.0.1:443:[IPMI-IP]:443 -L 127.0.0.1:5900:[IPMI-IP]:5900 -L 127.0.0.1:623:[IPMI-IP]:623 root@[SERVER-IP]

For example using 192.168.7.150 for an IPMI/KVM IP:

[root@srv0 ~]# ssh -N -L 127.0.0.1:80:192.168.7.150:80 -L 127.0.0.1:443:192.168.7.150:443 -L 127.0.0.1:5900:192.168.7.150:5900 -L 127.0.0.1:623:192.168.7.150:623 root@example-server.com

With the above command you can use the web interface (https://127.0.0.1/, you could replace 127.0.0.1 with a local IP or a local IP alias of your machine), the java web start “Console Redirection” (the KVM – Keyboard, Video and Mouse) and you can mount Virtual Media from your computer to your server’s virtual CD/DVD device. Unfortunately to use properly the Virtual CD/DVD you must tunnel the UDP on port 623 (not only TCP 623), which is a little bit tricky. To tunnel the UDP packets

socat – Multipurpose relay (SOcket CAT)

program must be used.

STEP 3) Tunnel local selected ports using ssh to the server from STEP 1) and UDP port using socat

[root@srv0 ~]# socat -T15 udp4-recvfrom:623,reuseaddr,fork tcp:localhost:8000
[root@srv0 ~]# ssh -L8000:localhost:8000 -L 127.0.0.1:80:192.168.7.150:80 -L 127.0.0.1:443:192.168.7.150:443 -L 127.0.0.1:5900:192.168.7.150:5900 -L 127.0.0.1:623:192.168.7.150:623 root@example-server.com socat tcp4-listen:8000,reuseaddr,fork UDP:192.168.7.150:623

This will start a UDP listening socket on localhost port 8000. Every packet will be relayed using TCP to localhost 8000, which will be tunneled using ssh command to the remote server, where there is a started another socat listening TCP socket on port 8000, which will relay every packet to the UDP port 623 of IP 192.168.7.150. Replace the IP 192.168.7.150 with your IPMI/KVM IP.

* Here are the required ports for SUPERMICRO IPMI functionality in X9 and X10 motherboards

  • X9-motherboards, the ports are

    TCP Ports
    HTTP: 80
    HTTPS: 443
    SSH: 22
    WSMAN: 5985
    Video: 5901
    KVM: 5900
    CD/USB: 5120
    Floppy: 5123
    Virtual Media: 623
    SNMP: 161

    UDP ports:
    IPMI: 623

  • For X10-motherboards, the ports are

    TCP Ports
    HTTP: 80
    HTTPS: 443
    SSH: 22
    WSMAN: 5985
    Video: 5901
    KVM: 5900 , 3520
    CD/USB: 5120
    Floppy: 5123
    Virtual Media: 623
    SNMP: 161

    UDP ports:
    IPMI: 623

You could add the required port to the ssh command above if you need it!

Virtual Device mounted successfully

Successful mount in Console Redirection with Virtual Media:

main menu
Virtual Storage

if you are logged in the server and mount an ISO with the Virtual Device you’ll probably have this in “dmesg”:

[46683751.661063] usb 2-1.3.2: new high-speed USB device number 8 using ehci-pci
[46683751.795048] usb 2-1.3.2: New USB device found, idVendor=0ea0, idProduct=1111
[46683751.795051] usb 2-1.3.2: New USB device strings: Mfr=0, Product=0, SerialNumber=0
[46683751.795365] usb-storage 2-1.3.2:1.0: USB Mass Storage device detected
[46683751.795553] scsi6 : usb-storage 2-1.3.2:1.0
[46683752.795730] scsi 6:0:0:0: CD-ROM            ATEN     Virtual CDROM    YS0J PQ: 0 ANSI: 0 CCS
[46683752.806839] sr0: scsi3-mmc drive: 40x/40x cd/rw xa/form2 cdda tray
[46683752.806842] cdrom: Uniform CD-ROM driver Revision: 3.20
[46683752.806933] sr 6:0:0:0: Attached scsi CD-ROM sr0
[46683752.806971] sr 6:0:0:0: Attached scsi generic sg1 type 5

SUPERMICRO IPMI to use one of the LAN interfaces or dedicated LAN port

If you happen to have a Supermicro server and you want to change the default behavior of the IPMI LAN interface, which is

Failover – on boot check whether the dedicated LAN port is connected if so use the it, otherwise use the shared LAN1

So if change it there are some magic commands to change this default behavior:

  • Always use dedicated LAN:
    within the server under console:

    ipmitool -I open raw 0x30 0x70 0x0c 1 0
    

    from remote using the network:

    ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN raw 0x30 0x70 0x0c 1 0
    

    Sometimes the output of the last command (that using the lanplus) will output:

    Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70)
    

    But it sets the value despite the error output “Unable to send”. You could check it with the read command (the last example).

  • Always use shared LAN1:
    within the server under console:

    ipmitool -I open raw 0x30 0x70 0xc 1 1 
    

    from remote using the network:

    ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN raw 0x30 0x70 0x0c 1 1
    

    Sometimes the output of the last command (that using the lanplus) will output:

    Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70)
    

    But it sets the value despite the error output “Unable to send”. You could check it with the read command (the last example).

  • Always use failover (factory default):
    within the server under console:

    ipmitool -I open raw 0x30 0x70 0xc 1 2
    

    from remote using the network:

    ipmitool -I lanplus -H 192.168.7.150 -U ADMIN -P ADMIN raw 0x30 0x70 0x0c 1 2
    
  • Sometimes the output of the last command (that using the lanplus) will output:

    Unable to send RAW command (channel=0x0 netfn=0x30 lun=0x0 cmd=0x70)
    

    But it sets the value despite the error output “Unable to send”. You could check it with the read command (the last example).

Get the current value with:

[root@srv0 ~]# ipmitool -I open raw 0x30 0x70 0x0c 0
 02
[root@srv0 ~]#

Default (failover): you will see 02
Onboard LAN: you will see 01
Dedicated LAN: you will see 00

The 192.168.7.157 is the IP of the IPMI KVM module and the -U ADMIN and -P ADMIN are username and the password login details to the module (ADMIN/ADMIN are just default settings for the Supermicro IPMI/KVM)

* Here you can set the LAN IP configuration – “Set IP to the IPMI/KVM server module with ipmitool

Set IP to the IPMI/KVM server module with ipmitool

IPMI/KVM module are pretty useful add-on module to every server. In fact every server should have IPMI module installed for a fast management of the server in critical cases!
Here are the commands to set a static IP to the IPMI/KVM module with ipmitool using a console to the server:

ipmitool -I open lan set 1 ipsrc static
ipmitool -I open lan set 1 ipaddr [IPADDR]
ipmitool -I open lan set 1 netmask [NETMASK]
ipmitool -I open lan set 1 defgw ipaddr [GW IPADDR]
ipmitool -I open lan set 1 access on
  • [IPADDR] – the IP address of the IPMI/KVM
  • [NETMASK] – the netmask of the network
  • [GW IPADDR] – the gateway of the network

Here is a real world example of setting properly the LAN settings of the IPMI module.

[root@srv0 ~]# ipmitool -I open lan set 1 ipsrc static
[root@srv0 ~]# ipmitool -I open lan set 1 ipaddr 192.168.6.45
Setting LAN IP Address to 192.168.6.45
[root@srv0 ~]# ipmitool -I open lan set 1 netmask 255.255.255.0
Setting LAN Subnet Mask to 255.255.255.0
[root@srv0 ~]# ipmitool -I open lan set 1 defgw ipaddr 192.168.6.1
Setting LAN Default Gateway IP to 192.168.6.1
[root@srv0 ~]# ipmitool -I open lan set 1 access on
Set Channel Access for channel 1 was successful.
[root@srv0 ~]#

To see the current settings use:

[root@srv0 ~]# ipmitool -I open lan print
Set in Progress         : Set Complete
Auth Type Support       : NONE MD2 MD5 PASSWORD 
Auth Type Enable        : Callback : MD2 MD5 PASSWORD 
                        : User     : MD2 MD5 PASSWORD 
                        : Operator : MD2 MD5 PASSWORD 
                        : Admin    : MD2 MD5 PASSWORD 
                        : OEM      : MD2 MD5 PASSWORD 
IP Address Source       : Static Address
IP Address              : 192.168.6.45
Subnet Mask             : 255.255.255.0
MAC Address             : 00:25:90:18:8b:c9
SNMP Community String   : public
IP Header               : TTL=0x00 Flags=0x00 Precedence=0x00 TOS=0x00
BMC ARP Control         : ARP Responses Enabled, Gratuitous ARP Disabled
Default Gateway IP      : 192.168.6.1
Default Gateway MAC     : 00:00:00:00:00:00
Backup Gateway IP       : 0.0.0.0
Backup Gateway MAC      : 00:00:00:00:00:00
802.1q VLAN ID          : Disabled
802.1q VLAN Priority    : 0
RMCP+ Cipher Suites     : 1,2,3,6,7,8,11,12
Cipher Suite Priv Max   : aaaaXXaaaXXaaXX
                        :     X=Cipher Suite Unused
                        :     c=CALLBACK
                        :     u=USER
                        :     o=OPERATOR
                        :     a=ADMIN
                        :     O=OEM
Bad Password Threshold  : Not Available

*Dependencies

Installation of ipmitool:

  • CentOS 7
    yum -y install ipmitool
    
  • Ubuntu 16+
  • apt-get install ipmitool
    
  • Gentoo
    emerge -vu sys-apps/ipmitool
    

*Troubleshooting

If you receive errors when you execute ipmitool:

[root@srv0 ~]# ipmitool -I open lan set 1 ipaddr 192.168.6.45
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
[root@srv0 ~]# ipmitool -I open lan set 1 netmask 255.255.255.0
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory
[root@srv0 ~]# ipmitool -I open lan set 1 defgw ipaddr 192.168.6.1
Could not open device at /dev/ipmi0 or /dev/ipmi/0 or /dev/ipmidev/0: No such file or directory

The kernel module for the IPMI/KVM is not loaded by the system, so just execute:

[root@srv0 ~]# modprobe ipmi_si
[root@srv0 ~]# modprobe ipmi_devintf

And then you could use ipmitool commands above to set the network configuration of the IPMI/KVM add-on module.