mysql – Error ‘Column count of mysql.user is wrong. Expected 45, found 43. The table is probably corrupted’ on query.

If you

upgraded your MySQL server (from 5.6 to 5.7 or above)

or

imported a MySQL dump SQL file from older version

than your current server you may encounter when granting permissions to a user:

Error 'Column count of mysql.user is wrong. Expected 45, found 43. The table is probably corrupted' on query. Default database: ''. Query: 'GRANT REPLICATION SLAVE ON *.* TO 'replusr'@'144.76.156.182''

Do not panic probably it is not corrupted just continue reading.

There is the simple fix, just

execute mysql_upgrade

It will automatically detect what to upgrade and it will upgrade it:

[myuser@mysql1 ~]# screen -R upgrade
[myuser@mysql1 ~]# mysql_upgrade 
Checking if update is needed.
Checking server version.
Running queries to upgrade MySQL server.
Checking system database.
mysql.columns_priv                                 OK
mysql.db                                           OK
mysql.engine_cost                                  OK
mysql.event                                        OK
mysql.func                                         OK
mysql.general_log                                  OK
mysql.gtid_executed                                OK
mysql.help_category                                OK
mysql.help_keyword                                 OK
mysql.help_relation                                OK
mysql.help_topic                                   OK
mysql.host                                         OK
mysql.innodb_index_stats                           OK
mysql.innodb_table_stats                           OK
mysql.ndb_binlog_index                             OK
mysql.plugin                                       OK
mysql.proc                                         OK
mysql.procs_priv                                   OK
mysql.proxies_priv                                 OK
mysql.server_cost                                  OK
mysql.servers                                      OK
mysql.slave_master_info                            OK
mysql.slave_relay_log_info                         OK
mysql.slave_worker_info                            OK
mysql.slow_log                                     OK
mysql.tables_priv                                  OK
mysql.time_zone                                    OK
mysql.time_zone_leap_second                        OK
mysql.time_zone_name                               OK
mysql.time_zone_transition                         OK
mysql.time_zone_transition_type                    OK
mysql.user                                         OK
The sys schema is already up to date (version 1.5.1).
Found 0 sys functions, but expected 22. Re-installing the sys schema.
Upgrading the sys schema.
Checking databases.
phpmyadmin.pma__bookmark                           OK
phpmyadmin.pma__central_columns                    OK
phpmyadmin.pma__column_info                        OK
phpmyadmin.pma__designer_settings                  OK
phpmyadmin.pma__export_templates                   OK
phpmyadmin.pma__favorite                           OK
phpmyadmin.pma__history                            OK
phpmyadmin.pma__navigationhiding                   OK
phpmyadmin.pma__pdf_pages                          OK
phpmyadmin.pma__recent                             OK
phpmyadmin.pma__relation                           OK
phpmyadmin.pma__savedsearches                      OK
phpmyadmin.pma__table_coords                       OK
phpmyadmin.pma__table_info                         OK
phpmyadmin.pma__table_uiprefs                      OK
phpmyadmin.pma__tracking                           OK
phpmyadmin.pma__userconfig                         OK
phpmyadmin.pma__usergroups                         OK
phpmyadmin.pma__users                              OK
sys.sys_config                                     OK
db1.access                                         OK
db1.users                                          OK
db1.objects                                        OK
db1.isp                                            OK
db1.desc                                           OK
Upgrade process completed successfully.
Checking if update is needed.

It works when the server is up and running and it is a good idea to execute the command in a screen.
It does not need to be logged as root, but mysql_upgrade does need to have the root MySQL password. In the example above it did not asked for password, because we have it in ~/.my.cnf file.

Just to note you might upgraded a long before this error to appear!

If you do not use a certain functionality you could live up happily with the old mysql.user scheme (and all old mysql.* tables). In our case we upgraded one of our slaves and several days after when a grant command on the master was issued the replication just stopped with this error! Of course, if someone were used the command in our slave the error would have appeared there sooner.
We also had case where old MySQL SQL dump file (5.6) was imported in a newer MySQL server 5.7 and there had been no issues for weeks till the GRANT command.

perror

Th error code is 1805.

[myuser@mysql1 ~]# perror 1805
MySQL error code 1805 (ER_COL_COUNT_DOESNT_MATCH_CORRUPTED_V2): Column count of %s.%s is wrong. Expected %d, found %d. The table is probably corrupted

Supermicro server cannot enter BIOS with F2, DEL or other when UEFI mode OS is installed

If you happen to have a supermicro server (X10SLH-F) and install Linux in UEFI mode in our case CentOS 7 and you want to enter the BIOS you’ll be surprised that you cannot with the keys provided in the very same BIOS boot screen – F2, DEL. The F11 and F12 also does not work for menu selection and network boot!

Even if you manage to press the DEL key and you see on the screen “Entering BIOS setup…” – the server WON’T enter BIOS, but will continue with the UEFI BIOS boot drive!

So what to do? Ammm break temporary your system by removing (renaming or moving) the EFI directory in your efi boot partition, resetting your server and holding pressed DEL key (again) on all start up screens of the server. When the UEFI BIOS boot entry is not valid any more and there are no other boot devices (and probably because we pressed DEL key) we were able to enter in the BIOS without remote hands on the collocation side or any other intervention on the server.

[root@srv ~]# mv /boot/efi/EFI/ /boot/efi/EFI_org
[root@srv ~]# reboot

This is the path in CentOS 7 and our standard partition layout:

[root@srv ~]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3         26G  4.5G    20G  19% /
devtmpfs         7.8G     0   7.8G   0% /dev
tmpfs            7.8G     0   7.8G   0% /dev/shm
tmpfs            7.8G  8.5M   7.8G   1% /run
tmpfs            7.8G     0   7.8G   0% /sys/fs/cgroup
/dev/sda2        976M   98M   812M  11% /boot
/dev/sda1        200M  9.8M   191M   5% /boot/efi
tmpfs            1.6G     0   1.6G   0% /run/user/0

DO NOT forget to remove all other (virtual) CD/DVD ROM Devices and temporary disable your network PXE Server (if you have any in the network)

Because it when the UEFI BIOS cannot find the EFI file saved in the UEFI BIOS BOOT drive it might follow the boot order before entering the BIOS!

Enter the bios by remote console on our X9 boards with UEFI bios

Apparently there is an issue with X8 and X9 supermicro boards in UEFI mode BIOS: https://www.supermicro.com/support/faqs/faq.cfm?faq=14029
So for someone it could be useful pressing and holding “ESC” + “-” or F4 to enter the UEFI BIOS, but we could not make it because of the IPMI KVM we used to manage the server.

Update supermicro X10SLM-F firmware BIOS under Linux with the SUM cli

Here is how we updated our Supermicro server X10SLM-F with the latest firmware at the moment.

Our current BIOS firmware version is 2.0

[root@srv ~]# lshw|grep -A 14 "core$"
  *-core
       description: Motherboard
       product: X10SLM-F
       vendor: Supermicro
       physical id: 0
       version: 1.02
       serial: 11111111111
       slot: To be filled by O.E.M.
     *-firmware
          description: BIOS
          vendor: American Megatrends Inc.
          physical id: 0
          version: 2.0
          date: 04/24/2014
          size: 64KiB

Keep on reading!

sed with delimiter – any other single character in replacing words or characters

It is not so known fact that this powerful unix world command

sed

could be used with other delimiter than “/” (when replacing words or characters), which is used in 100% of the time in the Internet examples.
You probably know the syntax from the manual like:

s/regexp/replacement/
              Attempt to match regexp against the pattern space.  If successful, replace that portion matched with replacement.  The replacement may  con‐
              tain  the special character & to refer to that portion of the pattern space which matched, and the special escapes \1 through \9 to refer to
              the corresponding matching sub-expressions in the regexp.

The Internet examples always use “/” as shown in the man, BUT you CAN use “#” instead of “/” for example:

s#regexp#replacement#g

or

s:regexp:replacement:g

or you can even use letter character (not a special character as “!@#$%#^”):

sRregexpRreplacementRg

and with “r” lower case (you see not so readable, but possible!)

srregexprreplacementrg

Here are the examples:

sed 's@else@elllllse@g' test.php
sed 's#else#elllllse#g' test.php
sed 's:else:elllllse:g' test.php
sed 's~else~elllllse~g' test.php
sed 's!else!elllllse!g' test.php
sed 'spelsepelllllsepg' test.php
sed 'sRelseRelllllseRg' test.php
#could be paired with "-i", too
sed 'spelsepelllllsepg' -i test.php
sed 'sRelseRelllllseRg' -i test.php

What does it mean to you? The simple implication is you are not forced to escape characters in your regexp and replacement part. Consider you want to replace part of a unix/linux path string: /home/myuser/Desktop/mydirectory1/myfile.log if you use the default “/” you MUST escape all the “/” in your string:

sed 's/\/home\/myuser\/Desktop\/mydirectory1\/myfile.log/\/home\/user\/Desktop\/mydirectory2\/myfile.log/g' test.log

versus the simpler and more readable version with “#”

sed 's#/home/myuser/Desktop/mydirectory1/myfile.log#/home/user/Desktop/mydirectory2/myfile.log#g' test.log

And consider you use an variable:

cat file.test | sed "s/\[version\]/${PKGVERSION}/g"

* additional explanations:

  • /g (or what ever character is used for delimiter like “@”, “:”, “~” and so on) is for “Apply the replacement to all matches to the regexp, not just the first.”
  • -i (used in some of the examples above) is for inline replacement – the file you add in the sed command will be modified and the replacement will be saved in the file (the default behavior is to show the modified output in the standard output – console output)

Clear or delete systemd logs

Systemd linux distros use Journald service to collect and store logs in the system. Here are a couple of tips if you have problems with the space they occupy. It is good that all systemd linux distros support it – CentOS 7, Ubuntu 16+, Fedora, OpenSuse and so on.

TIP 1) Remove the old archive logs older than 10 days with

Time based removal of old logs. It will remove the old files. This command won’t change the configuration, so it just has a temporary effect.

journalctl --vacuum-time=10d

TIP 2) Remove the old archive logs greater than 1G

Size based removal of old logs. It will reduce the size of the logs to this specified size. This command won’t change the configuration, so it just has a temporary effect.

journalctl --vacuum-size=1024M

TIP 3) Show disk usage

[root@srv ~]# journalctl --disk-usage
Archived and active journals take up 785.5M on disk.

TIP 4) Show all logs and information for them

Where are the log files, the size they occupy, the time period of the entries in them and more:

[root@srv0 ~]# journalctl --header
File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system.journal
File ID: 7deaff4610a94c82aab85386597e825b
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ONLINE
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 1239405
Tail Sequential Number: 1341511
Head Realtime Timestamp: mon 2018-06-25 10:09:09 UTC
Tail Realtime Timestamp: tue 2018-06-28 00:07:10 UTC
Tail Monotonic Timestamp: 1month 2d 21h 6min 47.287s
Objects: 258136
Entry Objects: 102107
Data Objects: 129585
Data Hash Table Fill: 72.5%
Field Objects: 36
Field Hash Table Fill: 10.8%
Tag Objects: 0
Entry Array Objects: 26406
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-000000000011512d-00056f3f3ff56172.journal
File ID: a8e1f042bc144df78f37005b0b555a82
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 1134893
Tail Sequential Number: 1239404
Head Realtime Timestamp: fri 2018-06-22 18:32:10 UTC
Tail Realtime Timestamp: mon 2018-06-25 10:09:09 UTC
Tail Monotonic Timestamp: 1month 7h 8min 45.991s
Objects: 263445
Entry Objects: 104512
Data Objects: 131953
Data Hash Table Fill: 73.8%
Field Objects: 34
Field Hash Table Fill: 10.2%
Tag Objects: 0
Entry Array Objects: 26944
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-00000000000fc083-00056f0d6ef70320.journal
File ID: 007272b1f181487d83116bee96c40c30
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 1032323
Tail Sequential Number: 1134892
Head Realtime Timestamp: wed 2018-06-20 07:06:10 UTC
Tail Realtime Timestamp: fri 2018-06-22 18:32:10 UTC
Tail Monotonic Timestamp: 4w 2h 1min 46.980s
Objects: 263998
Entry Objects: 102570
Data Objects: 133456
Data Hash Table Fill: 74.7%
Field Objects: 37
Field Hash Table Fill: 11.1%
Tag Objects: 0
Entry Array Objects: 27933
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-00000000000e3f67-00056ee09376d287.journal
File ID: 4c268268a6e34d6297c7ae9ca01fc31b
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: yes
Head Sequential Number: 933735
Tail Sequential Number: 1032322
Head Realtime Timestamp: mon 2018-06-18 01:35:09 UTC
Tail Realtime Timestamp: wed 2018-06-20 07:06:10 UTC
Tail Monotonic Timestamp: 3w 4d 14h 35min 47.248s
Objects: 261498
Entry Objects: 98588
Data Objects: 134059
Data Hash Table Fill: 75.0%
Field Objects: 40
Field Hash Table Fill: 12.0%
Tag Objects: 0
Entry Array Objects: 28809
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-00000000000cbf3b-00056eb3ea255b2a.journal
File ID: 7019a721bebe4e00b76410a6e7052c6d
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: yes
Head Sequential Number: 835387
Tail Sequential Number: 933734
Head Realtime Timestamp: fri 2018-06-15 20:18:10 UTC
Tail Realtime Timestamp: mon 2018-06-18 01:35:09 UTC
Tail Monotonic Timestamp: 3w 2d 9h 4min 45.972s
Objects: 260794
Entry Objects: 98348
Data Objects: 134060
Data Hash Table Fill: 75.0%
Field Objects: 33
Field Hash Table Fill: 9.9%
Tag Objects: 0
Entry Array Objects: 28351
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-00000000000b2fac-00056e827d3375dd.journal
File ID: 61c16e8acc364d97b8c0c7cc2afcb6ec
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 733100
Tail Sequential Number: 835386
Head Realtime Timestamp: wed 2018-06-13 09:20:08 UTC
Tail Realtime Timestamp: fri 2018-06-15 20:18:10 UTC
Tail Monotonic Timestamp: 3w 3h 47min 46.827s
Objects: 264298
Entry Objects: 102287
Data Objects: 134043
Data Hash Table Fill: 75.0%
Field Objects: 33
Field Hash Table Fill: 9.9%
Tag Objects: 0
Entry Array Objects: 27933
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-000000000009966a-00056e4cbbb6ebfc.journal
File ID: c90df7bdc4654ab7a4ac955c74a779a3
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 628330
Tail Sequential Number: 733099
Head Realtime Timestamp: sun 2018-06-10 17:12:09 UTC
Tail Realtime Timestamp: wed 2018-06-13 09:19:10 UTC
Tail Monotonic Timestamp: 2w 4d 16h 48min 46.993s
Objects: 263295
Entry Objects: 104770
Data Objects: 131816
Data Hash Table Fill: 73.7%
Field Objects: 37
Field Hash Table Fill: 11.1%
Tag Objects: 0
Entry Array Objects: 26670
Disk usage: 98.1M

File Path: /run/log/journal/bb5aa69bfa2b483386443ede7a92a45a/system@541a59d9a68342d3980d86e565a48c7d-000000000007fd20-00056e16f3135f30.journal
File ID: 32915d651dd84728b9e4a33a97b480cf
Machine ID: bb5aa69bfa2b483386443ede7a92a45a
Boot ID: e89507fa3f814b8a96f9d70914c120b0
Sequential Number ID: 541a59d9a68342d3980d86e565a48c7d
State: ARCHIVED
Compatible Flags:
Incompatible Flags: COMPRESSED-XZ
Header size: 240
Arena size: 102956816
Data Hash Table Size: 178744
Field Hash Table Size: 333
Rotate Suggested: no
Head Sequential Number: 523552
Tail Sequential Number: 628329
Head Realtime Timestamp: fri 2018-06-08 01:02:10 UTC
Tail Realtime Timestamp: sun 2018-06-10 17:12:09 UTC
Tail Monotonic Timestamp: 2w 2d 41min 46.200s
Objects: 263287
Entry Objects: 104778
Data Objects: 131696
Data Hash Table Fill: 73.7%
Field Objects: 36
Field Hash Table Fill: 10.8%
Tag Objects: 0
Entry Array Objects: 26775
Disk usage: 98.1M
[root@srv0 ~]#

* Deleting logs

[root@srv0 ~]# journalctl --vacuum-size=128M
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a33241-000570199db5be3c.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a34463-0005701a0b24d0fc.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3568c-0005701a79e388c7.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a36899-0005701ae529f439.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a37ac8-0005701b50781f62.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a38cde-0005701bbd3b60c3.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a39ef6-0005701c28dcf40e.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3b0f1-0005701c9416ee94.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3c309-0005701cffe19015.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3d521-0005701d6c79b9cb.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3e748-0005701dd7c60e79.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a3f92d-0005701e48413eb8.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a40b0a-0005701ec83a0c74.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a41ce2-0005701f3fdcadc6.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a42f04-0005701fabfefbca.journal (8.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a4410e-000570201757a256.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a60a8f-0005702ac025bb0c.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a7d254-000570359f2bd7eb.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000a99a34-000570405b5e812e.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000ab639b-0005704b05553c7d.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000ad2c3e-00057055cb403fae.journal (120.0M).
Deleted archived journal /var/log/journal/9be717e698354ec481abb641cf4085c3/system@79d0440cc1174c2db132b707a0567bcb-0000000000aef5c8-000570607686fd78.journal (120.0M).
Vacuuming done, freed 960.0M of archived journals from /var/log/journal/9be717e698354ec481abb641cf4085c3.

Replace an old IP with new one in all files of all sub-directories recursively

Here is a quick Linux tip for those who want to replace their old IP with new one for all files in a given directory and all its sub-directories recursively:

find [path-to-directory] -type f -print0 | xargs -0 sed -i 's[old-IP-escape-dot]/[new-IP]/g'

Quick example:

find /etc/nginx/ -type f -print0 | xargs -0 sed -i 's/192\.168\.10\.124/10.10.10.224/g'

As you can see the directory is “/etc/nginx” and replace it with the directory where are your (configuration) files are. We are replacing old IP=192.168.10.124 with the new one 10.10.10.224, so after the execution of the above line you’ll get modified files with IP 10.10.10.224.
You must escape the dot “.” in the IP!

Configure and mount samba share in Supermicro IPMI Virtual media (CD-ROM)

Having many Supermicro servers we used multiple times Virtual Media to install, diagnose and rescue Supermicro server. It is really simple to open the Console Redirection – java web start and just mount the ISO file from Virtual Media -> ISO File, but this way if your server is not local to your network and it is located in a colocation as it should be the connection is slow and in many cases bogus! Because it uses UDP it happened many times to remount or just to lose the connection and the media to disappear in the middle of the booting/loading process from the installation media, for example. And probably you have noticed there is additional option in the web interface of mounting ISO file from a windows share. Of course, in linux world it might be samba share as in our case and the share could be easily configured on a server in your colocation.
And we noticed was several times we have no problems using it, but some time the share could not be even saved and respectively could not be mounted. No error reported, just the edit boxes resets to blank and apparently everything was the same as the previous box, which it worked as a charm!
Here are the steps to enable one of your CentOS 7 servers to share a resource and to use it in your Supermicro IPMI KVM. The server is selinux enabled and the policy is Enforcing.

STEP 1) Install samba server in CentOS 7

[root@srv0 ~]# yum -y install samba samba-client samba-common policycoreutils-python
Loaded plugins: fastestmirror
Determining fastest mirrors
epel/x86_64/metalink                                                                                                                           |  30 kB  00:00:00     
 * base: mirrors.neterra.net
 * epel: mirrors.neterra.net
 * extras: mirrors.neterra.net
 * updates: mirrors.neterra.net
base                                                                                                                                           | 3.6 kB  00:00:00     
epel                                                                                                                                           | 3.2 kB  00:00:00     
extras                                                                                                                                         | 3.4 kB  00:00:00     
updates                                                                                                                                        | 3.4 kB  00:00:00     
(1/7): base/7/x86_64/group_gz                                                                                                                  | 166 kB  00:00:00     
(2/7): epel/x86_64/group_gz                                                                                                                    |  88 kB  00:00:00     
(3/7): base/7/x86_64/primary_db                                                                                                                | 5.9 MB  00:00:00     
(4/7): extras/7/x86_64/primary_db                                                                                                              | 147 kB  00:00:00     
(5/7): updates/7/x86_64/primary_db                                                                                                             | 2.0 MB  00:00:00     
(6/7): epel/x86_64/updateinfo                                                                                                                  | 932 kB  00:00:00     
(7/7): epel/x86_64/primary                                                                                                                     | 3.5 MB  00:00:00     
epel                                                                                                                                                      12584/12584
Package samba-client-4.7.1-6.el7.x86_64 already installed and latest version
Package samba-common-4.7.1-6.el7.noarch already installed and latest version
Resolving Dependencies
--> Running transaction check
---> Package samba.x86_64 0:4.7.1-6.el7 will be installed
--> Processing Dependency: samba-libs = 4.7.1-6.el7 for package: samba-4.7.1-6.el7.x86_64
--> Processing Dependency: samba-common-tools = 4.7.1-6.el7 for package: samba-4.7.1-6.el7.x86_64
--> Processing Dependency: libxattr-tdb-samba4.so(SAMBA_4.7.1)(64bit) for package: samba-4.7.1-6.el7.x86_64
--> Processing Dependency: libaio.so.1(LIBAIO_0.4)(64bit) for package: samba-4.7.1-6.el7.x86_64
--> Processing Dependency: libaio.so.1(LIBAIO_0.1)(64bit) for package: samba-4.7.1-6.el7.x86_64
--> Processing Dependency: libxattr-tdb-samba4.so()(64bit) for package: samba-4.7.1-6.el7.x86_64
--> Processing Dependency: libaio.so.1()(64bit) for package: samba-4.7.1-6.el7.x86_64
--> Running transaction check
---> Package libaio.x86_64 0:0.3.109-13.el7 will be installed
---> Package samba-common-tools.x86_64 0:4.7.1-6.el7 will be installed
---> Package samba-libs.x86_64 0:4.7.1-6.el7 will be installed
--> Processing Dependency: libpytalloc-util.so.2(PYTALLOC_UTIL_2.1.9)(64bit) for package: samba-libs-4.7.1-6.el7.x86_64
--> Processing Dependency: libpytalloc-util.so.2(PYTALLOC_UTIL_2.1.6)(64bit) for package: samba-libs-4.7.1-6.el7.x86_64
--> Processing Dependency: libpytalloc-util.so.2(PYTALLOC_UTIL_2.0.6)(64bit) for package: samba-libs-4.7.1-6.el7.x86_64
--> Processing Dependency: libpytalloc-util.so.2()(64bit) for package: samba-libs-4.7.1-6.el7.x86_64
--> Running transaction check
---> Package pytalloc.x86_64 0:2.1.10-1.el7 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

======================================================================================================================================================================
 Package                                        Arch                               Version                                     Repository                        Size
======================================================================================================================================================================
Installing:
 samba                                          x86_64                             4.7.1-6.el7                                 base                             661 k
Installing for dependencies:
 libaio                                         x86_64                             0.3.109-13.el7                              base                              24 k
 pytalloc                                       x86_64                             2.1.10-1.el7                                base                              17 k
 samba-common-tools                             x86_64                             4.7.1-6.el7                                 base                             463 k
 samba-libs                                     x86_64                             4.7.1-6.el7                                 base                             275 k

Transaction Summary
======================================================================================================================================================================
Install  1 Package (+4 Dependent packages)

Total download size: 1.4 M
Installed size: 3.8 M
Downloading packages:
(1/5): libaio-0.3.109-13.el7.x86_64.rpm                                                                                                        |  24 kB  00:00:00     
(2/5): pytalloc-2.1.10-1.el7.x86_64.rpm                                                                                                        |  17 kB  00:00:00     
(3/5): samba-4.7.1-6.el7.x86_64.rpm                                                                                                            | 661 kB  00:00:00     
(4/5): samba-common-tools-4.7.1-6.el7.x86_64.rpm                                                                                               | 463 kB  00:00:00     
(5/5): samba-libs-4.7.1-6.el7.x86_64.rpm                                                                                                       | 275 kB  00:00:00     
----------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                 4.5 MB/s | 1.4 MB  00:00:00     
Running transaction check
Running transaction test
Transaction test succeeded
Running transaction
  Installing : pytalloc-2.1.10-1.el7.x86_64                                                                                                                       1/5 
  Installing : samba-libs-4.7.1-6.el7.x86_64                                                                                                                      2/5 
  Installing : samba-common-tools-4.7.1-6.el7.x86_64                                                                                                              3/5 
  Installing : libaio-0.3.109-13.el7.x86_64                                                                                                                       4/5 
  Installing : samba-4.7.1-6.el7.x86_64                                                                                                                           5/5 
  Verifying  : libaio-0.3.109-13.el7.x86_64                                                                                                                       1/5 
  Verifying  : samba-libs-4.7.1-6.el7.x86_64                                                                                                                      2/5 
  Verifying  : samba-common-tools-4.7.1-6.el7.x86_64                                                                                                              3/5 
  Verifying  : samba-4.7.1-6.el7.x86_64                                                                                                                           4/5 
  Verifying  : pytalloc-2.1.10-1.el7.x86_64                                                                                                                       5/5 

Installed:
  samba.x86_64 0:4.7.1-6.el7                                                                                                                                          

Dependency Installed:
  libaio.x86_64 0:0.3.109-13.el7        pytalloc.x86_64 0:2.1.10-1.el7        samba-common-tools.x86_64 0:4.7.1-6.el7        samba-libs.x86_64 0:4.7.1-6.el7       

Complete!
[root@srv0 ~]#

STEP 2) Configure samba server CentOS 7 for the purpose of using it in IPMI Virtual share.

We are going to use a share without login credentials, because our KVM IP are always local ones and accessed via a vpn network and in addition only the network of the IPMI IPs could access the share (the samba server has a firewall configured).
Set the configuration file of the samba server – one directory storing the files shared with no login credentials (no username/password means anonymous login).
The configuration file is:

/etc/samba/smb.conf

[global]
workgroup = WINSHARE
server string = Samba Server %v
netbios name = centossrv
security = user
map to guest = bad user
dns proxy = no
#============================ Share Definitions ============================== 
[share]
path = /mnt/storage1/samba
browsable =yes
writable = no
guest ok = yes
read only = yes

As you can see we use “/mnt/storage1/samba” for our directory where the ISO files will be located. Change this path if you want to put your ISO files somewhere else.
Set the right permissions for the directory and selinux (if you server is not selinux enabled, you could skip the selinux part) and run the samba daemon:

[root@srv0 ~]# mkdir /mnt/storage1/samba
[root@srv0 ~]# chown -R nobody:nobody /mnt/storage1/samba/
[root@srv0 ~]# semanage fcontext -a -t samba_share_t '/mnt/storage1/samba(/.*)?'
[root@srv0 ~]# restorecon -Rv /mnt/storage1/samba/
restorecon reset /mnt/storage1/samba context unconfined_u:object_r:unlabeled_t:s0->unconfined_u:object_r:samba_share_t:s0
[root@srv0 ~]# cd /mnt/storage1/samba/
[root@srv0 samba]# wget http://mirror.leaseweb.com/centos/7.5.1804/isos/x86_64/CentOS-7-x86_64-Minimal-1804.iso
--2018-06-01 14:15:42--  http://mirror.leaseweb.com/centos/7.5.1804/isos/x86_64/CentOS-7-x86_64-Minimal-1804.iso
Resolving mirror.leaseweb.com (mirror.leaseweb.com)... 37.58.58.140, 2a00:c98:2030:a034::21
Connecting to mirror.leaseweb.com (mirror.leaseweb.com)|37.58.58.140|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 950009856 (906M) [application/octet-stream]
Saving to: ‘CentOS-7-x86_64-Minimal-1804.iso’

100%[============================================================================================================================>] 950,009,856 40.1MB/s   in 33s    

2018-06-01 14:16:15 (27.3 MB/s) - ‘CentOS-7-x86_64-Minimal-1804.iso’ saved [950009856/950009856]

[root@srv0 samba]# chown nobody:nobody CentOS-7-x86_64-Minimal-1804.iso
[root@srv0 samba]# systemctl start smb
[root@srv0 samba]# systemctl status smb
● smb.service - Samba SMB Daemon
   Loaded: loaded (/usr/lib/systemd/system/smb.service; disabled; vendor preset: disabled)
   Active: active (running) since Fri 2018-06-01 14:17:57 UTC; 20s ago
 Main PID: 31961 (smbd)
   Status: "smbd: ready to serve connections..."
    Tasks: 4
   Memory: 19.0M
   CGroup: /system.slice/smb.service
           ├─31961 /usr/sbin/smbd --foreground --no-process-group
           ├─31964 /usr/sbin/smbd --foreground --no-process-group
           ├─31965 /usr/sbin/smbd --foreground --no-process-group
           └─31966 /usr/sbin/smbd --foreground --no-process-group

Jun 01 14:17:56 srv0@local systemd[1]: Starting Samba SMB Daemon...
Jun 01 14:17:57 srv0@local smbd[31961]: [2018/06/01 14:17:57.761913,  0] ../lib/util/become_daemon.c:124(daemon_ready)
Jun 01 14:17:57 srv0@local smbd[31961]:   STATUS=daemon 'smbd' finished starting up and ready to serve connections
Jun 01 14:17:57 srv0@local systemd[1]: Started Samba SMB Daemon.

Configure the firewall to allow only connections from the IPMI KVM IP networks (or a single IP if you need to expose it in the Internet). You can work with build in zone “trusted”, but here we prefer more generic approach, which could be used not only for local IP networks but for real IPs:

[root@srv0 samba]# firewall-cmd --new-zone=smbshare --permanent
success
[root@srv0 samba]# firewall-cmd --zone=smbshare --add-source=192.168.7.0/24 --permanent
success
[root@srv0 samba]# firewall-cmd --zone=smbshare --add-service=samba --permanent
success
[root@srv0 samba]# firewall-cmd --zone=smbshare --add-service=samba-client --permanent
success
[root@srv00 samba]# firewall-cmd --reload
success
[root@srv00 samba]# firewall-cmd --zone=smbshare --list-all
smbshare (active)
  target: default
  icmp-block-inversion: no
  interfaces: 
  sources: 192.168.7.0/24
  services: samba samba-client
  ports: 
  protocols: 
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

[root@srv00 samba]#

STEP 3) Mount a disk in IPMI Virutal Media and boot from it

You could check our additional tutorial for it here –

STEP 3.1) Fill “Share host” and “Path image” only, because we configured our samba share without a password

main menu
Image on Windows Share

STEP 3.2) Upon a successful configuration saving you’ll get a confirmation dialog.

If no confirmation is shown you would not be able to mount the share, check out our Troubleshooting below!

main menu
Save configuration of Image on Windows Share

STEP 3.3) After saving the configuration click on “Mount” to load your ISO file in the virtual CD-ROM.

main menu
Mount Image on Windows Share

STEP 3.4) This dialog is always shown no matter there is such share or not: “Please check the device status to confirm whether the image is mounted/unmounted.”

main menu
Check device status

STEP 3.5) If the Supermicro IPMI accessed successfully the share resource it would mount it (load the ISO file as s CD in the virtual CD-ROM).

If not the three Device 1,2,3 will show the same: “No disk emulation set.”, which means the samba share is not accessible, check the permissions for the file and the firewall first (and the logs files, too).

main menu
There is an iso file mounted

* Troubleshooting

– if you click on button “Save” and just nothing happen – no dialog for successful saving and no dialog at all, probably there is a Javascript error, try to open the web interface from a different browser with clean history or click ctrl+F5 when loading the Virtual Media page! More in Cannot save and mount a Supermicro IPMI Virtual media mount – javascript error?

Cannot save and mount a Supermicro IPMI Virtual media mount – javascript error?

If you have multiple Supermicro servers with many different versions of IPMI KVM software installed it could happen your browser to cache some of the JavaScript and other static content to reuse them, but it could very unpleasant when you have different version of IPMI software on different servers and you might notice strange behavior of the web interface! Especially if you use ssh tunneling to access your multiple Supermicro IPMI KVMs from local IP on your computerTunneling the IPMI/KVM ports over ssh (supermicro ipmi ports)

One of the big problems we have when we wanted to mount a Virtual Media

from a windows share (samba share in our case) in IPMI KVM web interface -> Virtual Media -> CD-ROM image -> Save and when the Save is clicked just nothing happen (sometimes it triggers a reload of the iframe) – no error nothing and no confirmation for successful save! At first it seems the web interface accepted the “Share Host” and “Path to Image”:

main menu
Fill “Share Host” and “Path to Image”

but when you click “Mount” it does not mount the media:
main menu
no disk emulation set

and when you reload the CD-ROM image page you get again blank edit boxes or (the old values):
main menu
Old values

Probably a refresh will get the values blank:
main menu
Blank edit boxes

And if you check your browser console you’ll see there is a JavaScript error:

Uncaught ReferenceError: FocusOnErrorSpecificCharSet3 is not defined

The error might be different, this was in our case. the problem was

the browser cached “https://192.168.0.170/js/utils.js”

from one of the previous servers and there the version of the IPMI KVM software was different and apparently the

/js/utils.js

was throwing an error and not working (this function did not exist in some older Supermicro IPMI KVM versions, the file is there but it is slightly different). The solution is so simple!

Just refresh the page with CTRL+F5 or delete the history or use another browser.

Such a simple problem, but could lead to big problems if you try to use the mount virtual media. In fact look for problems in the JavaScript if you cannot save the configuration in the “Share Host” and “Path to Image”, because when saving the IPMI do not check if there is a live “Share Host” with a windows/samba share and an image there, the software just check for special in “Shared” characters like:

var SpeficCharFilter = /[,; &'"<>\\=$|^?*~`()\[\]\{\}#%]/;

And for the password:

var SpeficCharFilter = /[,; &'"<>\\=$|^?~`()\[\]\{\}#%]/;

But in both cases you’ll get an alert with an error.

So to summer it up if you put IP and a path to the windows share of Virtual Media and click “Save” and nothing happen – no confirmation for successful saving you got a JavaScript error and probably your browser cached one of the JavaScript files, the solution is simple just refresh with CTRL+F5 or load from different browser!
We often use ssh tunneling for IPMI KVM accessTunneling the IPMI/KVM ports over ssh (supermicro ipmi ports) and different version of the static files of the supermicro IPMI web interface are cached locally, which as you can see could have really bad consequences!

Working Save button

click Save button

main menu
Fill “Share Host” and “Path to Image”

Confirmation when everything is OK

main menu
The confirmation of image on windows share has been successfully set.

Update supermicro server’s firmware BIOS under linux with the SUM cli

The newer Supermicro motherboards have the ability to update the system BIOS from IPMI Web management or with a Supermicro tool called

“Supermicro Update Manager (SUM)”

This tool consists of a linux cli and it works on most of the X9/X10/X11 Supermicro systems. If you have one of these motherboards you probably could update your BIOS under linux console and throwing away your freedos cd prehistoric update process (probably coming soon). If you’ve used Supermicro with for ages you probably are aware of the problem with the updating the BIOS, their tool is an old DOS flash utility, which could be used under MS Windows and with linux servers we got a problem! But not any more! Recent motherboards with built-in IPMI modules have “BIOS Update” option in the web interface (under Maintenance menu), but you need a license (which by the way is not expensive).

main menu
BIOS Update

So if you happen to have such server you could give a try to SUM linux cli (yes they made a linux cli and even a freebsd one!). With the SUM cli you could do more than just update the BIOS (which is pretty important) here are the functions you can do Key Management, System Checks, BIOS Management, BMC Management, System Event Log, CMM Management, Storage Management, Applications (through the network) and for some of them you still need a license, but for BIOS update you do not need if you do it as Supermicro says

In-Band

which basically means from the server you want to update (or in general use it). KEEP IN MIND this method will reset your BIOS options to DEFAULTS!!! In fact you need a license to use preserve settings you can see the output of the SUM help for the command UpdateBios. So save your changes to recover them after the update!
And here we present a BIOS update with SUM on one of our Supermicro servers with motherboard X11SSV-M4F
So the BIOS version was 1.0 reported by lshw:

[srv@local ~]# lshw|head -n 25
srv@local
    description: System
    product: Super Server (To be filled by O.E.M.)
    vendor: Supermicro
    version: 0123456789
    serial: 0123456789
    width: 64 bits
    capabilities: smbios-3.0 dmi-3.0 smp vsyscall32
    configuration: boot=normal chassis=server family=To be filled by O.E.M. sku=To be filled by O.E.M. uuid=00000000-0000-0000-0000-111111111111
  *-core
       description: Motherboard
       product: X11SSV-M4F
       vendor: Supermicro
       physical id: 0
       version: 1.02
       serial: 112233000044
       slot: To be filled by O.E.M.
     *-firmware
          description: BIOS
          vendor: American Megatrends Inc.
          physical id: 0
          version: 1.0
          date: 10/18/2016
          size: 64KiB
          capacity: 15MiB

And here are the steps to do:

STEP 1) Download the SUM cli – Supermicro Update Manager and unpack it in your server

You could download freely the SUM version 2 from here: https://www.supermicro.com/solutions/SMS_SUM.cfm (this link explains what the tool is used for) at the bottom of the page there is a link to download the SUM – https://www.supermicro.com/SwDownload/UserInfo.aspx?sw=0&cat=SUM Here you should fill the form with your name, company and email address and then you will be redirected to the download page, which at present is like the screenshot below:

main menu
Supermicro Update Manager (SUM)

When downloaded the linux version you would have the following files in the archive file:

[srv@local ~]# cd
[srv@local ~]# tar xzvf sum_2.0.0_Linux_x86_64_20171108.tar.gz
sum_2.0.0_Linux_x86_64/
sum_2.0.0_Linux_x86_64/ReleaseNote.txt
sum_2.0.0_Linux_x86_64/sum
sum_2.0.0_Linux_x86_64/ExternalData/
sum_2.0.0_Linux_x86_64/ExternalData/VENID.txt
sum_2.0.0_Linux_x86_64/ExternalData/SMCIPID.txt
sum_2.0.0_Linux_x86_64/driver/
sum_2.0.0_Linux_x86_64/driver/RHL4_x86_64/
sum_2.0.0_Linux_x86_64/driver/RHL4_x86_64/sum_bios.ko
sum_2.0.0_Linux_x86_64/driver/RHL6_x86_64/
sum_2.0.0_Linux_x86_64/driver/RHL6_x86_64/sum_bios.ko
sum_2.0.0_Linux_x86_64/driver/RHL5_x86_64/
sum_2.0.0_Linux_x86_64/driver/RHL5_x86_64/sum_bios.ko
sum_2.0.0_Linux_x86_64/driver/RHL7_x86_64/
sum_2.0.0_Linux_x86_64/driver/RHL7_x86_64/sum_bios.ko
sum_2.0.0_Linux_x86_64/SUM_UserGuide.pdf

The “sum” file binary is the tool you need.

STEP 2) Download the new motherboard BIOS firmware

, which for our board (X11SSV-M4F – https://www.supermicro.com/products/motherboard/Xeon/C236_C232/X11SSV-M4F.cfm) it was version 1.1 with filename “X11SVMF8_308.zip”, unpack the file under X11SVMF8_308

[srv@local ~]# cd
[srv@local ~]# unzip X11SVMF8_308.zip
Archive:  X11SVMF8_308.zip
   creating: X11SVMF8.308/
  inflating: X11SVMF8.308/AFUDOSU.SMC  
  inflating: X11SVMF8.308/CHOICE.SMC  
  inflating: X11SVMF8.308/FDT.smc    
  inflating: X11SVMF8.308/FLASH.BAT  
  inflating: X11SVMF8.308/Readme for UP X11 AMI  BIOS.txt  
  inflating: X11SVMF8.308/X11SVMF8.308

The file “X11SVMF8.308” is the BIOS firmware of the motherboard. As you can see the prehistoric DOS executable flash utility is still distributed and is supported method of updating.

STEP 3) Update the BIOS

[srv@local ~]# cd /root/sum_2.0.0_Linux_x86_64
[srv@local ~]# ./sum -c UpdateBios --file ../X11SVMF8.308/X11SVMF8.308 
Supermicro Update Manager (for UEFI BIOS) 2.0.0 (2017/11/08) (x86_64)
Copyright©2017 Super Micro Computer, Inc. All rights reserved
Reading BIOS flash ..................... (100%)
Checking BIOS ID ...
Checking ME Firmware ...
Comparing FDT for ROM file and flash.... (100%)

***************************<<<<<CRITICAL WARNING>>>>>***************************

ExitCode                = 254
Description             = Manual steps are required
Program Error Code      = 119.18
Error message:
    FDT is different. Please use system reboot(do not shutdown 
    or poweroff system) command to force ME enter manufacturing mode 
    and run UpdateBIOS command again to update BIOS and ME region.

********************************************************************************
[srv@local ~]# reboot

As you can see we got critical warning! You may not have the critical, but if you do you need to run the command second time to really flash the new BIOS firmware. So here it is, AFTER rebooting your server login again and execute the command again:

[srv@local ~]# cd /root/sum_2.0.0_Linux_x86_64
[srv@local ~]# ./sum -c UpdateBios --file ../X11SVMF8.308/X11SVMF8.308 
Supermicro Update Manager (for UEFI BIOS) 2.0.0 (2017/11/08) (x86_64)
Copyright©2017 Super Micro Computer, Inc. All rights reserved
Reading BIOS flash ..................... (100%)
Checking BIOS ID ...
Programming BIOS and ME (including FDT)
Writing BIOS flash ..................... (100%)
Verifying BIOS flash ................... (100%)
Checking ME Firmware ...
Putting ME data to BIOS ................ (100%)
Writing ME region in BIOS flash ...
 - Update success for FDR
 - Updated Recovery Loader to OPRx
 - Updated FPT, MFSB, FTPR and MFS
 - ME Entire Image done
WARNING:Must power cycle or restart the system for the changes to take effect!
[srv@local ~]# reboot

And there you have it you updated the BIOS of your server successfully. KEEP IN MIND the BIOS settings are reset to Defaults! When you restart the server the second time (if you get the critical warning or the first time if you do not) you should load the optimized defaults and change your settings according your needs. Go to BIOS and load the optimized defaults and change the settings according your needs (or your backup).

[srv@local ~]# lshw|head -n 25
srv@local
    description: System
    product: Super Server (To be filled by O.E.M.)
    vendor: Supermicro
    version: 0123456789
    serial: 0123456789
    width: 64 bits
    capabilities: smbios-3.0 dmi-3.0 smp vsyscall32
    configuration: boot=normal chassis=server family=To be filled by O.E.M. sku=To be filled by O.E.M. uuid=00000000-0000-0000-0000-111111111111
  *-core
       description: Motherboard
       product: X11SSV-M4F
       vendor: Supermicro
       physical id: 0
       version: 1.02
       serial: 112233000044
       slot: To be filled by O.E.M.
     *-firmware
          description: BIOS
          vendor: American Megatrends Inc.
          physical id: 0
          version: 1.1
          date: 03/08/2018
          size: 64KiB
          capacity: 15MiB

* SUM cli – UpdateBios help output

[srv@local ~]# ./sum -h -c UpdateBios
Supermicro Update Manager (for UEFI BIOS) 2.0.0 (2017/11/08) (x86_64)
Copyright©2017 Super Micro Computer, Inc. All rights reserved
Description
    Updates BIOS with the given image file.
Required Arguments
    --file    <file name>
Optional Arguments
    --reboot
          Forces the managed system to reboot or power up after operation.
    --flash_smbios
          Overwrites the SMBIOS data
    --preserve_nv
          Preserves the NVRAM region
    --preserve_mer
          Preserves the ME firmware region
    --preserve_setting
          Preserves setting configurations
Usage Modes
    [OOB] [In-Band] [Multiple systems OOB] 
Node Product Key Required
     No for [In-Band]
    Yes for [OOB]
Examples
OOB
  # ./sum -i 192.168.34.56 -u ADMIN -p ADMIN -c UpdateBios --file BIOS.rom 
    --reboot
In-Band
  # ./sum -c UpdateBios --file BIOS.rom --reboot
Multiple systems OOB
  # ./sum -l IP_ADDR_RANGE.txt -u ADMIN -p ADMIN -c UpdateBios --file BIOS.rom 
    --reboot
Notice
    1. For [OOB] and [Multiple systems OOB] usage modes, before executing this 
    command, it is recommended to shutdown the managed system first.
    2. --preserve_setting option is only supported in Purley and the platforms 
    of later versions. Note that --preserve_setting option for in-band usage 
    requires "SFT-OOB-LIC" product key. The preserved setting configurations 
    will be listed in preserved_settings.log.

* SUM cli generic help output


[srv@local ~]# ./sum 
Supermicro Update Manager (for UEFI BIOS) 2.0.0 (2017/11/08) (x86_64)
Copyright©2017 Super Micro Computer, Inc. All rights reserved

NAME
  sum (Supermicro Update Manager)

SYNOPSIS
  sum [OPTIONs] [COMMAND] [COMMAND ARGUMENTS]

OPTIONS
  -h  Shows help information.
  -v  Displays the verbose output.
  -i  <BMC/CMM IP address or BMC/CMM host name>
  -l  <BMC/CMM system list file name. Refer to the user's guide for formatting>
  -u  <BMC/CMM user ID>
  -p  <BMC/CMM user password>
  -c  <command name> (case insensitive)

USAGE MODES
○  Single System Out-Of-Band (OOB) Management [operates on single BMC/CMM]: 
    Must use -i,-u, -p options
○  Single System In-Band Management [operates on local OS]: Do not use -i, -u 
    and -p options
○  Concurrent Systems OOB Management [operates on multiple system BMCs/CMMs]: 
    Replace -i option with -l option

COMMANDS
Function Group             Command Names

Key Management             ActivateProductKey, QueryProductKey, ClearProductKey
System Checks              CheckOOBSupport, CheckAssetInfo, 
                           CheckSystemUtilization, CheckSensorData
BIOS Management            GetBIOSInfo, UpdateBios, GetDefaultBiosCfg, 
                           GetCurrentBiosCfg, ChangeBiosCfg, 
                           LoadDefaultBiosCfg, GetDmiInfo, ChangeDmiInfo, 
                           EditDmiInfo, SetBiosAction
BMC Management             GetBmcInfo, UpdateBmc, GetBmcCfg, ChangeBmcCfg
System Event Log           GetEventLog, ClearEventLog
CMM Management             GetCmmInfo, UpdateCmm, GetCmmCfg, ChangeCmmCfg
Storage Management         GetRaidControllerInfo, UpdateRaidController, 
                           GetRaidCfg, ChangeRaidCfg, GetSataInfo, GetNvmeInfo
Applications               TpmProvision, MountIsoImage, UnmountIsoImage

COMMAND USAGE
  See help message for each command
  Syntax:"  # ./sum -h -c <command name>"
  Notes: 1)Command support is platform dependent. Please refer to Appendix C in 
         the user's guide for platform dependency hints.
         2)If BMC/CMM user ID or password includes special characters, it has 
         to be quoted.

EXAMPLES
OOB
  # ./sum -i 192.168.34.56 -u ADMIN -p ADMIN -c ChangeBmcCfg --file BmcCfg.txt
  # ./sum -i 192.168.34.56 -u ADMIN -p "&123456" -c ChangeBmcCfg --file 
  BmcCfg.txt
Multiple systems OOB
  # ./sum -l IP_ADDR_RANGE.txt -u ADMIN -p ADMIN -c GetBIOSInfo --file BIOS.rom
  # ./sum -l IP_ADDR_RANGE.txt -u ADMIN -p "&123456" -c GetBIOSInfo --file 
  BIOS.rom
In-Band
  # ./sum -c UpdateBios --file BIOS.rom
Help Message
  # ./sum -h -c UpdateBios

grub2: grub-install: error: disk mduuid not found even after the partition has bios_grub on

This tutorial is for all of us that has done everything by the book with parted and still they receive an error when installing grub2 to the boot sector!

srv@local ~ # grub2-install /dev/sda
Installing for i386-pc platform.
grub2-install: error: disk `mduuid/613f3f0bb202bf03a5664e17b3d568a0' not found. 

The solution is relatively simple:

Boot from a rescue disk and reinstall grub from there!

The problem is that currently loaded kernel remembers the old device, which was deleted, and probably you won’t be able to fresh the metadata in the memory. In fact, if you receive this error when booted in a rescue disc you probably have done some changes on the layout of the disks or the partitions or the RAID devices and you must reboot the machine again and then JUST reinstall the grub.
Such problems could have happened if you deleted partitions or made some disk layout changes (using parted?) on mounted or partitions in use and the kernel could loaded the partition changes in memory. The parted reports the changes will take effect after system reboot.

STEP 1) Check the devices and mount the root and boot

They might be on the same device here they are on different MD devices (in the case, software array). md1 is the boot and md2 is the root.

root@rescue ~ # cat /proc/mdstat 
Personalities : [raid1] 
md3 : active raid1 sda5[3] sdb5[2]
      422716416 blocks super 1.2 [2/2] [UU]
      bitmap: 0/4 pages [0KB], 65536KB chunk

md2 : active raid1 sda3[3] sdb3[2]
      31440896 blocks super 1.2 [2/2] [UU]
      
md1 : active raid1 sda2[3] sdb2[2]
      523712 blocks super 1.2 [2/2] [UU]
      
md0 : active raid1 sda1[3] sdb1[2]
      33521664 blocks super 1.2 [2/2] [UU]
      
unused devices: <none>
root@rescue ~ # mount /dev/md2 /mnt/
root@rescue ~ # mount /dev/md1 /mnt/boot/
root@rescue ~ # ls -altr /mnt/
total 396K
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 srv
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 opt
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 mnt
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 media
drwxr-xr-x.  2 root root 4.0K Apr 11  2018 home
lrwxrwxrwx.  1 root root    8 May 14  2018 sbin -> usr/sbin
lrwxrwxrwx.  1 root root    9 May 14  2018 lib64 -> usr/lib64
lrwxrwxrwx.  1 root root    7 May 14  2018 lib -> usr/lib
lrwxrwxrwx.  1 root root    7 May 14  2018 bin -> usr/bin
drwxr-xr-x. 13 root root 4.0K May 14  2018 usr
drwx------.  2 root root  16K Feb 12  2019 lost+found
drwxr-xr-x   2 root root 4.0K Feb 12  2019 boot
drwxr-xr-x   2 root root 4.0K Feb 12  2019 storage1
drwxr-xr-x   2 root root 4.0K Feb 12  2019 sys
drwxr-xr-x   2 root root 4.0K Feb 12  2019 dev
drwxr-xr-x   2 root root 4.0K Feb 12  2019 prochttps://www.google.com/search?client=firefox-b-d&q=samsung+tab10+2020
drwxr-xr-x   3 root root 4.0K Feb 12  2019 run
-rw-r-----.  1 root root  575 Feb 12  2019 installimage.conf
-rw-r-----.  1 root root  13K Feb 12  2019 installimage.debug
drwxr-xr-x. 20 root root 4.0K Feb 12  2019 var
drwxr-xr-x. 85 root root 4.0K Nov  2  2019 etc
-rw-r--r--.  1 root root 291K Nov  2  2019 .readahead
drwxr-xr-x. 19 root root 4.0K Nov  2  2019 .
dr-xr-x---.  6 root root 4.0K May  4 19:24 root
drwxrwxrwt.  7 root root 4.0K May  8 14:14 tmp
drwxr-xr-x   1 root root  160 May  8 16:57 ..
root@rescue ~ # ls -altr /mnt/boot/
total 194M
drwxr-xr-x.  3 root root 1.0K Sep 18  2017 efi
-rw-------.  1 root root  49M Sep 18  2017 initramfs-0-rescue-9063ac396d784f4c997ceacdd0590c25.img
-rwxr-xr-x.  1 root root 5.7M Sep 18  2017 vmlinuz-0-rescue-9063ac396d784f4c997ceacdd0590c25
-rw-------.  1 root root 3.4M Feb  1  2019 System.map-3.10.0-957.5.1.el7.x86_64
-rw-r--r--.  1 root root 149K Feb  1  2019 config-https://www.google.com/search?client=firefox-b-d&q=samsung+tab10+20203.10.0-957.5.1.el7.x86_64
-rw-r--r--.  1 root root  170 Feb  1  2019 .vmlinuz-3.10.0-957.5.1.el7.x86_64.hmac
-rwxr-xr-x.  1 root root 6.4M Feb  1  2019 vmlinuz-3.10.0-957.5.1.el7.x86_64
-rw-r--r--.  1 root root 307K Feb  1  2019 symvers-3.10.0-957.5.1.el7.x86_64.gz
drwx------.  2 root root  12K Feb 12  2019 lost+found
drwxr-xr-x.  2 root root 1.0K Feb 12  2019 grub
-rw-------.  1 root root 3.5M Oct 18  2019 System.map-3.10.0-1062.4.1.el7.x86_64
-rw-r--r--.  1 root root 150K Oct 18  2019 config-3.10.0-1062.4.1.el7.x86_64
-rw-r--r--.  1 root root  171 Oct 18  2019 .vmlinuz-3.10.0-1062.4.1.el7.x86_64.hmac
-rwxr-xr-x.  1 root root 6.5M Oct 18  2019 vmlinuz-3.10.0-1062.4.1.el7.x86_64
-rw-r--r--.  1 root root 312K Oct 18  2019 symvers-3.10.0-1062.4.1.el7.x86_64.gz
-rw-------.  1 root root  13M Nov  2  2019 initramfs-3.10.0-957.5.1.el7.x86_64kdump.img
-rw-------.  1 root root  47M Nov  2md  2019 initramfs-3.10.0-1062.4.1.el7.x86_64.img
-rw-------.  1 root root  46M Nov  2  2019 initramfs-3.10.0-957.5.1.el7.x86_64.img
-rw-------.  1 root root  13M Nov  2  2019 initramfs-3.10.0-1062.4.1.el7.x86_64kdump.img
dr-xr-xr-x.  6 root root 1.0K Nov  2  2019 .
drwxr-xr-x. 19 root root 4.0K Nov  2  2019 ..
drwx------.  5 root root 1.0K May  4 16:11 grub2

STEP 2) mount the dev, proc and sys relative to the root mount above and chroot in the root.

After chroot it is recommended to

root@rescue ~ # mount -o bind /dev /mnt/dev
root@rescue ~ # mount -o bind /proc /mnt/proc
root@rescue ~ # mount -o bind /sys /mnt/sys
root@rescue ~ # chroot /mnt/
root@rescue / # cd
root@rescue ~ # . /etc/profile

STEP 3)Install grub2

Unmount all mounted directories above and reboot.

root@rescue ~ # grub2-install /dev/sda
Installing for i386-pc platform.
Installation finished. No error reported.
root@rescue ~ # grub2-install /dev/sdb
Installing for i386-pc platform.
Installation finished. No error reported.
root@rescue ~ # exit
root@rescue ~ # umount /mnt/boot 
root@rescue ~ # umount /mnt/dev
root@rescue ~ # umount /mnt/proc 
root@rescue ~ # umount /mnt/sys
root@rescue ~ # umount /mnt/
root@rescue ~ # reboot

* If you are using UEFI enabled boot you probably need more options for the grub installation

Something like that for the grub2 installation (but it is specific for your distro – the path for efi directory, just find it under /boot and put the right path – nothing special!):

grub-install --recheck --target=x86_64-efi --efi-directory=/boot/efi/ /dev/sda

Where does this error come from?

If you search the ID by part of it you will discover it under /dev/disk/by-id/, which is a link to md1, but still there is an error for missing device, because the kernel loaded the device as md-name-rescue:1. And if you reboot the kernel the old and wrong name won’t exists. As mentioned above this could have happened if you deleted partitions or made some disk layout changes (using parted?) on mounted or partitions in use and the kernel could loaded the partition changes.

[root@srv47 ~]# cd /dev/disk/by-id/
[root@srv47 by-id]# ls -altr
total 0
drwxr-xr-x. 5 root root 100  4 May  9,14 ..
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-uuid-e4023626:e09f7c39:20ed5720:2ef1b5af -> ../../md0
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-uuid-ae63857c:b8d537f5:4c09ae48:48148f59 -> ../../md3
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-uuid-4a28f3a7:94dc6dc7:a40a7084:c21463d7 -> ../../md2
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-name-rescue:3 -> ../../md3
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-name-rescue:2 -> ../../md2
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-name-rescue:0 -> ../../md0
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-uuid-613f3f0b:b202bf03:a5664e17:b3d568a0 -> ../../md1
lrwxrwxrwx. 1 root root   9  4 May 10,14 md-name-rescue:1 -> ../../md1
drwxr-xr-x. 2 root root 680  4 May 12,16 .
lrwxrwxrwx. 1 root root   9  8 May 12,01 wwn-0x500a075116bd3e33 -> ../../sda
lrwxrwxrwx. 1 root root   9  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33 -> ../../sda
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part1 -> ../../sda1
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part5 -> ../../sda5
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part4 -> ../../sda4
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part3 -> ../../sda3
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a075116bd3e33-part2 -> ../../sda2
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part5 -> ../../sda5
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part4 -> ../../sda4
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part3 -> ../../sda3
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_171416BD3E33-part2 -> ../../sda2
lrwxrwxrwx. 1 root root   9  8 May 12,01 wwn-0x500a07511bb48b38 -> ../../sdb
lrwxrwxrwx. 1 root root   9  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38 -> ../../sdb
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part5 -> ../../sdb5
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part4 -> ../../sdb4
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part3 -> ../../sdb3
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part2 -> ../../sdb2
lrwxrwxrwx. 1 root root  10  8 May 12,01 wwn-0x500a07511bb48b38-part1 -> ../../sdb1
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part5 -> ../../sdb5
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part4 -> ../../sdb4
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part3 -> ../../sdb3
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part2 -> ../../sdb2
lrwxrwxrwx. 1 root root  10  8 May 12,01 ata-Micron_1100_MTFDDAK512TBN_18081BB48B38-part1 -> ../../sdb1