Recover MySQL InnoDB Cluster and Dba.rebootClusterFromCompleteOutage: Argument #2: Invalid options: primary (ArgumentError)

main menu
MySQL 8.0.28 version

Recent version of MySQL 8 implemented more options to the rebootClusterFromCompleteOutage function! Definitely check the link’s manual above and most of the handy second options are implemented in MySQL 8.0.30, so the user’s MySQL InnoDB Cluster crashed and if rebootClusterFromCompleteOutage should be used, but it outputs an error sort of:

 MySQL  db-cluster-1:33060+ ssl  JS > var cluster = dba.rebootClusterFromCompleteOutage()
Restoring the cluster 'mycluster1' from complete outage...

Dba.rebootClusterFromCompleteOutage: Target member is in state ERROR (RuntimeError)

And when trying to use the node, which was healthy before the crash with this function, there is an error, too:

 MySQL  db-cluster-1:33060+ ssl  JS > var cluster = dba.rebootClusterFromCompleteOutage("mycluster1", {primary: "db-cluster-1:3306"});
Dba.rebootClusterFromCompleteOutage: Argument #2: Invalid options: primary (ArgumentError)

So no cluster is available and the database and its data is inaccessible.
Indeed, the initial state of the cluster was really bad and before the restart, the two of three servers were missing or in bad state.

[root@db-cluster-1 ~]# mysqlsh
MySQL Shell 8.0.28

Copyright (c) 2016, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates.
Other names may be trademarks of their respective owners.

Type '\help' or '\?' for help; '\quit' to exit.
 MySQL  JS > \connect clusteradmin@db-cluster-1
Creating a session to 'clusteradmin@db-cluster-1'
Fetching schema names for autocompletion... Press ^C to stop.
Your MySQL connection id is 241708346 (X protocol)
Server version: 8.0.28 MySQL Community Server - GPL
No default schema selected; type \use <schema> to set one.
 MySQL  db-cluster-1:33060+ ssl  JS > var cluster = dba.getCluster()
 MySQL  db-cluster-1:33060+ ssl  JS > cluster.status()
    "clusterName": "mycluster1", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "db-cluster-1:3306", 
        "ssl": "REQUIRED", 
        "status": "OK_NO_TOLERANCE", 
        "statusText": "Cluster is NOT tolerant to any failures. 2 members are not active.", 
        "topology": {
            "db-cluster-1:3306": {
                "address": "db-cluster-1:3306", 
                "memberRole": "PRIMARY", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "replicationLag": null, 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.28"
            "db-cluster-2:3306": {
                "address": "db-cluster-2:3306", 
                "instanceErrors": [
                    "NOTE: group_replication is stopped."
                "memberRole": "SECONDARY", 
                "memberState": "OFFLINE", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "(MISSING)", 
                "version": "8.0.28"
            "db-cluster-3:3306": {
                "address": "db-cluster-3:3306", 
                "instanceErrors": [
                    "ERROR: GR Recovery channel receiver stopped with an error: error connecting to master 'mysql_innodb_cluster_2324239842@db-cluster-1:3306' - retry-time: 60 retries: 1 message: Access denied for user 'mysql_innodb_cluster_2324239842'@'' (using password: YES) (1045) at 2023-09-19 04:37:00.076960", 
                    "ERROR: group_replication has stopped with an error."
                "memberRole": "SECONDARY", 
                "memberState": "ERROR", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "(MISSING)", 
                "version": "8.0.28"
        "topologyMode": "Single-Primary"
    "groupInformationSourceMember": "db-cluster-1:3306"

The problem here is the MySQL version is 8.0.28, but after MySQL 8.0.30 there are much more features, which can be used in the second argument of rebootClusterFromCompleteOutage including, which server should be considered primary therefore healthy. In fact, the updated rebootClusterFromCompleteOutage of MySQL 8.0.34 version even auto-detected the correct and healthy node and booted the MySQL InnoDB Cluster.
There were no problems with the update from MySQL 8.0.28 to MySQL 8.0.34 and after the MySQL 8.0.34 started, the rebootClusterFromCompleteOutage reconfigured and started the cluster with the right and healthy server auto-detected. In fact, it is safer to use the second argument and set the option, which is the healthy server “{primary: “db-cluster-1:3306″}”.
Keep on reading!

binutils and the error ld: unrecognized option ‘–no-dynamic-linker’

Yet another bites of an old bintuils installed in the system, which leads to an error and failed a building of glibc this time. The last time it was a kernel building failure, check out here – . Most of the time, these kind of errors occurs when upgrading an old system, so as soon as building the new binutils package with emerge it is mandatory to remove the old one to minimize compiling errors of this sort.

main menu
building failure

This time the error under Gentoo system is (but it could happen in any system with old and new binutils!):

/usr/lib/gcc/x86_64-pc-linux-gnu/11/../../../../x86_64-pc-linux-gnu/bin/ld: unrecognized option '--no-dynamic-linker'
/usr/lib/gcc/x86_64-pc-linux-gnu/11/../../../../x86_64-pc-linux-gnu/bin/ld: use the --help option for usage information
collect2: error: ld returned 1 exit status

To fix the error simply remove the (all) old binutils package(s) with emerge command:

[root@srv ~]# emerge -vaC =binutils-2.25-r1
 * This action can remove important packages! In order to be safer, use
 * `emerge -pv --depclean <atom>` to check for reverse dependencies before
 * removing packages.

>>> These are the packages that would be unmerged:

    selected: 2.25-r1 
   protected: none 
     omitted: 2.41-r1 

All selected packages: =sys-devel/binutils-2.25-r1

>>> 'Selected' packages are slated for removal.
>>> 'Protected' and 'omitted' packages will not be removed.

Would you like to unmerge these packages? [Yes/No] yes
>>> Waiting 5 seconds before starting...
>>> (Control-C to abort)...
>>> Unmerging in: 5 4 3 2 1
>>> Unmerging (1 of 1) sys-devel/binutils-2.25-r1...
No package files given... Grabbing a set.
<<<          obj /usr/x86_64-pc-linux-gnu/binutils-bin/2.25/strip
<<<          obj /usr/x86_64-pc-linux-gnu/binutils-bin/2.25/strings
<<<          obj /usr/x86_64-pc-linux-gnu/binutils-bin/2.25/size
<<<          obj /usr/x86_64-pc-linux-gnu/binutils-bin/2.25/readelf

Keep on reading!

GPD Win Max2 (2023) with AMD 7840U performance with Cinebench R23

GPD Win Max2 (2023) is a cool small laptop with joysticks for gaming. It has really cool specs, which many of the laptops lack!

main menu
original photo

The model used to test here is equipped with

  • 10.1 inches display – 2560×1600 with 400 nit brightness (Supports 10-point Touch, Active Pens With Up To 4096-level pressure-sensitivity)
  • AMD Ryzen 7 7840U with integrated GPU (RDNA 3) – AMD Radeon 780M Graphics (12CU, 768 shaders)
  • 64GB RAM (LPDDR5x-7500 MT/s)
  • 2T NVME (two slots available!)
  • Oculink (SFF-8612) Port
  • 2 USB 4 ports (2 more USB 3.2 Gen1 ports are available and SD card slot reader)
  • Wi-Fi 6
  • 67Wh battery
  • and many more features.

It’s like a little beast with this CPU and the 64G RAM and adding the integrated RDNA AMD GPU, this little machine offers a gaming performance even with some of the best and resource demanding games, at present. It can run best games with 40-60 FPS on Cyberpunk 2077, Hades, Control, Rise of the Tomb Raider, Horizon Zero Dawn, The Witcher III, the latest Zelda series, Metroid Prime Remastered, in addition to many old games with simulation of PS1/PS2/PS3, Nintendo Switch/Wii and many more. There many games tested here YouTube.

Here are Cinebench R23 test for all predefined TDP Watts with temperatures, FAN and CPU Speeds. The best TDP is probably 8 or 12 Watts max, where the CPU and the machine is not hot, so no FAN noise, at all. The single core performance is almost maxed out and the multi cores is around the half of the possible max performance of the device. The device is not reinstalled and no drivers were updated, the tests are performed on the stock device with the original software and drivers. All tests are performed on battery at least half full. Ambient temperature is 26’C and the idle temperature of the CPU is between 34-36’C. The Motion Assistant is used to live limit the TDP.
The TDP in AMD world means the required cooling with simple words, because the product (i.e. CPU/GPU) can produce such amount of heat.

Cinebench R23 multi cores tests

It always uses all the cores for the tests. The Cinebench R23 points are in yellow, the temperatures are red and the TDP in Watts are in blue. Probably the best TDP related to the CPU performance is between 8 and 18 Watts. The minimal Cinebench R23 points for the 5 TDP are 1968 and the max are around 14000 with 35 Watts TDP. In fact, the CPU offers great performance even with TDP 12 Watts.

main menu
multicore watts temperatures points

Cinebench R23 single core

It is difficult to say how many cores are used and how many are parked, but it is between 6-12 are parked and only two are half used at most. The single core performance is much easier to represent, because it max out on the 8 Watts TDP and it does not change when the TDP are higher. That’s why even 8 Watts TDP is really good for productivity work with max single core, 1/3 performance of the multi core and temperatures and watts usage for maximum battery.

main menu
single core watts temperatures points

Keep on reading!

Edit with systemctl edit to add restart on fail to a service – nfs-ganesha

A quick tip how to edit a service unit file under a c system like CentOS Stream 9 or Ubuntu. The best way is to edit it with the the tool “systemctl edit [service_name]”, which will trigger the default editor to open a temporary copy of the systemd unit file with the service name used with it. The default editor in the console is controlled by “EDITOR” variable and may be changed prior using the systemctl edit. After a successful manipulation of the system unit file the new one will be installed and a reload of the systemd unit files will be triggered with “systemctl daemon-reload” automatically. Indeed, it is just a text edit of a text file, which will do several actions when using “systemctl edit” command.

main menu
systemctl cat service

systemd options ro restart a service on fail are:


Here, the example is to add a restart-on-fail functionality to the nfs-ganesha service (NFS service). The systemctl edit may be used for many other changes to the systemd unit file under the console and it is the easiest and proper way.

SCREENSHOT 1) Use “systemctl edit” to edit a copy of the systemd override unit file.

do not insert anything at the end of the comments or below the second red line comments – “### Lines below this comment will be discarded”. This temporary override file includes a systemd unit file of the service, which is opened for editing. The result override.conf file will only include the added lines, no other comments shown below the second red line.

main menu
systemctl edit opened

Keep on reading!

Change time zone for syslog messages in syslog-ng

When sending syslog packets to a remote server the time-zone of the current server may lead to problems because the syslog-ng program sets the time-zone with offset number according to the GMT. The remote server, especially if not a syslog-ng one, may interpret the offset as an offset to the UTC (Coordinated Universal Time), which leads times with an hour into the future.

main menu
configuration in syslog-ng

Note, this whole problem is because of the Daylight saving time and there are almost 6 months when the GMT is not equal to the UTC and it is an hour ahead. Probably it is not a good idea to offset according to the GMT, because of the Daylight saving time during the summer, but this is on the syslog-ng development side.
There is one option time-zone(“[time_zone_string]”), which allows to change the time-zone of the destination packets. To avoid misinterprets of the date and time in the message packets the best way is to use it with UTC, so the local system will do the all necessary to convert the local time to UTC properly.
The configuration below uses time-zone(“[time_zone_string]”) from the current local time to UTC, because the current local time zone is EEST (Eastern European Summer Time), which 3 hours ahead of UTC during the summer period and 2 during winter (Eastern European Time – EET).

Relay the local web server logs from the local UDP port to the remote server using reliable TCP connection and changing the local time to UTC of the packets.

#NGINX - web logs
source udp_local {
    network(ip( port(514) transport("udp") so_rcvbuf(67108864) log_fetch_limit(1000) max-connections(1000) log-iw-size(1000000));

filter filter_nginx_access_log { program(nginx); };
destination d_tcp_syslog {
         port(10514) transport("tcp") disk-buffer(mem-buf-length(10000) mem-buf-size(128M) disk-buf-size(1024M) reliable(yes) dir("/var/lib/syslog-ng"))
log { source(udp_local); filter(filter_nginx_access_log); destination(d_tcp_syslog); };

the time_zone(“UTC”) in destination rule will ensure the packets have proper time related to the local server’s time in UTC and the mistake of misinterpreted date time is unlikely on a remote server.
More articles with syslog-ng

Rename the hostname in syslog-ng

At present, the syslog-ng (syslog-ng Open Source Edition) does not support to set the hostname of the server in the configuration. syslog-ng uses DNS system to resolve the system name or if it is explicitly switch off (with couple of options) it will use the IP. Sometimes the DNS name of the system may be not valid, for example in containers, or just for better naming purposes it is useful to have just a simple option to set the system’s hostname in the syslog packets.

main menu
rewrite rule

The easiest and best way to change the system’s hostname in the packets is to use substitution rules with rewrite. In fact, the rewrite rule may operate on soft parts of the macros like MESSAGE, PROGRAM, HOST or user defined macros. The syslog message format and its “fields” could be seen in the RFC5424. It’s worth adding it is possible to replace the whole macro or just part of it.
Here is the configuration to set the system’s hostname:

#substitution rule
rewrite my_host { set("my-server-name", value("HOST")); };

#use the rule before the destination!!!
log { source(src); rewrite(my_host); destination(messages);};

The substitution rule should be used before the destination rule to take effect.

Here is a more complex example to set the system’s hostname only to certain packets:
Keep on reading!

Send and receive Groestlcoins with Groestlcoin Core 25.0 graphical user interface

As a continuation of managing Groestlcoin wallet articles Send coins in Groestlcoin network from command line with Groestlcoin Core 25 in year 2023, Build and run the Groestlcoin Core 25.0 node with a graphical wallet and Generate a new Groestlcoin address (wallet) and list wallets and addresses using command line cli verson 25.0.0. This article will show how to transfer Groestlcoins with the official Groestlcoin Core desktop application and the look and feel of the wallet functionality of the Groestlcoin Core.

main menu
Overview balances

The basic topics to manage a Groestlcoin wallet with the official software (ALWAYS use only official software) are:

  • Groestlcoin Core provides two key functionalities – become a Groestlcoin node, i.e. a part of the Groestlcoin network and the Groestlcoin wallet with command line interface and graphical user interface.
  • Groestlcoin Core can generate and Groestlcoin address.
  • a Groestlcoin wallet contains Groestlcoin addresses.
  • the Groestlcoin Core has a wallet functionality application, but before using it the wallet needs to synchronize with the Groestlcoin network. So the Groestlcoin GUI application will start a Groeslcoin node and it will try synchronizing with the network.
  • Groestlcoin Core includes groestlcoin-qt application, which offers the graphical user interface
  • Using groestlcoin-qt is really simple to manage wallets, generate Groestlcoin addresses, and get information for the local Groestlcoin node, which is used by the wallet application.

Here are the steps needed to transfer funds under the Linux distro of Ubuntu securely, but this time from the Gnome. To be able to transfer money securely without any 3rd party involved (no online wallet or wallets, which is unclear which node they connect to) just funds from your personal Groestlcoin wallet address on your computer to another Groestlcoin wallet address it is needed a running Groestlcoin node using Groestlcoin Core graphical user interface:groestlcoin-qt – this application provides basic graphical user interface for managing the Groestlcoin wallets. After the Groestlcoin Core is build and installed (check this article – Build and run the Groestlcoin Core 25.0 node with a graphical wallet), start the /[user_home_directory]/groestlcoin-core/bin/groestlcoin-qt. Apparently, the Groestlcoin site offers a compiled version of Groestlcoin Core (with an installer for Windows, too), but be careful what you download and from where! The best way is to just follow our article to build it yourself under Ubuntu.
More on the Groestlcoin topic –

SCREENSHOT 1) The loading screen at the start of groestlcoin-qt application.

It may take time to start because it is loading the local cached block index and if the node was fully synced lately, it may take a minute or two to load.

main menu
loading screen

Keep on reading!

Send coins in Groestlcoin network from command line with Groestlcoin Core 25 in year 2023

This article an updated version of Send coins in Groestlcoin network with Groestlcoin Core (cli), because it appeared many things handling the wallets with the command-line have changed.

main menu
groestlcoin-cli sendtoaddress

The basic notes to send and receive are:

  • a Groestlcoin address is required.
  • a Groestlcoin wallet contains Groestlcoin address
  • the Groestlcoin wallet application (i.e. Groestlcoin node, it is the same application) needs a synchronized Groestlcoin to operate properly, such as send and receive Groestlcoin coins.
  • the Groestlcoin wallet may be created, loaded or imported with the Groestlcoin tools.
  • if there is only one Groestlcoin wallet loaded it would be the default one and it would not require additional options to the Groestlcoin tool to specify, which wallet would be used for the current command.

Here are the steps needed to transfer funds under the Linux distro of Ubuntu securely. To be able to transfer money securely without any 3rd party involved (no online wallet or wallets, which is unclear which node they connect to) just funds from your personal Groestlcoin wallet address on your computer to another Groestlcoin wallet address it is needed a running Groestlcoin node using Groestlcoin Core command line tool:groestlcoin-cli – this is command line tool for managing the Groestlcoin wallets.

STEP 1) Install and run a Groestlcoin node with groestlcoind

The first step is to start a Groestlcoin node by following this updated article – Building from source and run a Groestlcoin node (cli only) under Ubuntu 22.04 LTS. It will help the user to build from the official source code the Groestlcoin Core software, which will include the Groestlcoin node and wallet functionalities. All of the file paths and application names are real and correct if the above article was used.
It’s worth mentioning, that the Groestlcoin node must be running and synced with the Groestlcoin network. Here is how to check it:

myuser@mydesktop:~/groestlcoin-core/bin$ ./groestlcoin-cli getblockchaininfo
  "chain": "main",
  "blocks": 4734023,
  "headers": 4734023,
  "bestblockhash": "00000000000020d22b156e27af600fcdc3fd315f8896bbdb5ab748260fbfa3dd",
  "difficulty": 457347.0794165397,
  "time": 1693300370,
  "mediantime": 1693300091,
  "verificationprogress": 0.9999998125955512,
  "initialblockdownload": false,
  "chainwork": "00000000000000000000000000000000000000000000016eeb7afc66a3b219fb",
  "size_on_disk": 3040987475,
  "pruned": false,
  "warnings": ""

Keep on reading!

Build and run the Groestlcoin Core 25.0 node with a graphical wallet

In continuation to the article Building from source and run a Groestlcoin node (cli only) under Ubuntu 22.04 LTS, which offers only a command line management to the Groestlcoin wallets, this article will show a more convenient way with a graphical user interface. Groistlcoin Core software includes a graphical user interface for the Groestlcoin wallet.

main menu
node window synced with network

The first thing is to build the Groestlcoin Core software as shown in the link above with some additional dependencies and options.
It’s worth noting there is a package for the graphical Groestlcoin wallet on the official site of Groestlcoin, but the idea behind this series of articles is to build everything from the official source code and to run an own Groestlcoin node, which will be used for the user’s Groestlcoin wallet. Such no trusting anyone or any site, but only the official source code of the project. The security is really important in the crypto world.

Here are the steps to build and run the Groestlcoin node with wallet functionality – command line and graphical user interface. The graphical user interface uses QT5 to build the wallet interface and all libraries are included in the official Ubuntu repositories.

STEP 1) Build and run the Groestlcoin node

For more detailed information check out Building from source and run a Groestlcoin node (cli only) under Ubuntu 22.04 LTS.

First, install the dependencies but with additional libraries including the QT5 and qrencode. Update the system and install the needed additional binaries and libraries to build and run the Groestlcoin core package. These dependencies include the ones needed to build the Groestcoin wallet functionality with GUI! By default, many Linux systems won’t even include dependencies needed to build the wallet.

sudo apt -y update
sudo apt -y upgrade
sudo apt install build-essential libssl-dev libboost-all-dev libtool automake libevent-dev bsdmainutils git make g++ gcc autoconf cpp ngrep iftop sysstat autotools-dev pkg-config libminiupnpc-dev libzmq3-dev libdb-dev libdb5.3++-dev sqlite3 libsqlite3-dev

An additional 1042 Mbytes will be occupied.
Keep on reading!

how to skip mysqlbinlog reading error – ERROR: Found out of order GTID

main menu
MariaDB select gtid_strict_mode

When trying to read the MariaDB binlog files, the reading may be interrupted with an error message, and the reading will stop:

ERROR: Found out of order GTID. Got 0-3-855835750755 after 0-1-855835750756

Some positions are shown but after 24 lines of 1236849 file, the mariadb-binlog / mysqlbinlog program interrupts with an error.

It turns out the problem is that the mariadb-binlog / mysqlbinlog reads the binlog files with GTID strict mode, because it is enabled by default! But the servers may have turned it off, so the binlog files are in the wrong format for the strict mode reading. The mode to read the binlog files should be the same as the MySQL / MariaDB server mode that created the binlog files.

root@srv binlog # mariadb-binlog  mysql-bin.52349 
/*!40019 SET @@session.max_delayed_threads=0*/;
# at 4
#230820 10:53:45 server id 1  end_log_pos 256 CRC32 0xffa88ac1        Start: binlog v 4, server v 11.0.2-MariaDB-1:11.0.2+maria~ubu2204-log created 230820 10:53:45
# at 256
#230820 10:53:45 server id 1  end_log_pos 379 CRC32 0xd434c2c6        Gtid list [0-5-76636859354,
# 0-4-847593757253,
# 0-2-857258567526,
# 0-3-855835750755,
# 0-1-855835750756,
# 1-4-8637504]
# at 379
#230820 10:53:45 server id 1  end_log_pos 423 CRC32 0xc23ef7af        Binlog checkpoint mysql-bin.52348
ERROR: Found out of order GTID. Got 0-3-855835750755 after 0-1-855835750756

Show the slave status to determine, which mode is in use:

root@srv binlog # mysql
mysql: Deprecated program name. It will be removed in a future release, use '/usr/bin/mariadb' instead
Welcome to the MariaDB monitor.  Commands end with ; or \g.
Your MariaDB connection id is 371408455
Server version: 11.0.2-MariaDB-1:11.0.2+maria~ubu2204-log binary distribution

Copyright (c) 2000, 2018, Oracle, MariaDB Corporation Ab and others.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

MariaDB [(none)]> select @@gtid_strict_mode;
| @@gtid_strict_mode |
|                  0 |
1 row in set (0.000 sec)

MariaDB [(none)]>

When the binlog files are generated with gtid_strict_mode = 0 the “–skip-gtid-strict-mode” option should be used with mariadb-binlog / mysqlbinlog to read the binlog files without errors. More on gtid_strict_mode for MariaDB –
Related errors GTID slave position – MariaDB/MySQL replication error – Error during XID COMMIT: failed to update GTID state in mysql.gtid_slave_pos. More on MariaDB on this site with tag mariadb tag.