Recover MySQL InnoDB Cluster and Dba.rebootClusterFromCompleteOutage: Argument #2: Invalid options: primary (ArgumentError)

main menu
MySQL 8.0.28 version

Recent version of MySQL 8 implemented more options to the rebootClusterFromCompleteOutage function! Definitely check the link’s manual above and most of the handy second options are implemented in MySQL 8.0.30, so the user’s MySQL InnoDB Cluster crashed and if rebootClusterFromCompleteOutage should be used, but it outputs an error sort of:

 MySQL  db-cluster-1:33060+ ssl  JS > var cluster = dba.rebootClusterFromCompleteOutage()
Restoring the cluster 'mycluster1' from complete outage...

Dba.rebootClusterFromCompleteOutage: Target member is in state ERROR (RuntimeError)

And when trying to use the node, which was healthy before the crash with this function, there is an error, too:

 MySQL  db-cluster-1:33060+ ssl  JS > var cluster = dba.rebootClusterFromCompleteOutage("mycluster1", {primary: "db-cluster-1:3306"});
Dba.rebootClusterFromCompleteOutage: Argument #2: Invalid options: primary (ArgumentError)

So no cluster is available and the database and its data is inaccessible.
Indeed, the initial state of the cluster was really bad and before the restart, the two of three servers were missing or in bad state.

[root@db-cluster-1 ~]# mysqlsh
MySQL Shell 8.0.28

Copyright (c) 2016, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates.
Other names may be trademarks of their respective owners.

Type '\help' or '\?' for help; '\quit' to exit.
 MySQL  JS > \connect clusteradmin@db-cluster-1
Creating a session to 'clusteradmin@db-cluster-1'
Fetching schema names for autocompletion... Press ^C to stop.
Your MySQL connection id is 241708346 (X protocol)
Server version: 8.0.28 MySQL Community Server - GPL
No default schema selected; type \use <schema> to set one.
 MySQL  db-cluster-1:33060+ ssl  JS > var cluster = dba.getCluster()
 MySQL  db-cluster-1:33060+ ssl  JS > cluster.status()
    "clusterName": "mycluster1", 
    "defaultReplicaSet": {
        "name": "default", 
        "primary": "db-cluster-1:3306", 
        "ssl": "REQUIRED", 
        "status": "OK_NO_TOLERANCE", 
        "statusText": "Cluster is NOT tolerant to any failures. 2 members are not active.", 
        "topology": {
            "db-cluster-1:3306": {
                "address": "db-cluster-1:3306", 
                "memberRole": "PRIMARY", 
                "mode": "R/W", 
                "readReplicas": {}, 
                "replicationLag": null, 
                "role": "HA", 
                "status": "ONLINE", 
                "version": "8.0.28"
            "db-cluster-2:3306": {
                "address": "db-cluster-2:3306", 
                "instanceErrors": [
                    "NOTE: group_replication is stopped."
                "memberRole": "SECONDARY", 
                "memberState": "OFFLINE", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "(MISSING)", 
                "version": "8.0.28"
            "db-cluster-3:3306": {
                "address": "db-cluster-3:3306", 
                "instanceErrors": [
                    "ERROR: GR Recovery channel receiver stopped with an error: error connecting to master 'mysql_innodb_cluster_2324239842@db-cluster-1:3306' - retry-time: 60 retries: 1 message: Access denied for user 'mysql_innodb_cluster_2324239842'@'' (using password: YES) (1045) at 2023-09-19 04:37:00.076960", 
                    "ERROR: group_replication has stopped with an error."
                "memberRole": "SECONDARY", 
                "memberState": "ERROR", 
                "mode": "R/O", 
                "readReplicas": {}, 
                "role": "HA", 
                "status": "(MISSING)", 
                "version": "8.0.28"
        "topologyMode": "Single-Primary"
    "groupInformationSourceMember": "db-cluster-1:3306"

The problem here is the MySQL version is 8.0.28, but after MySQL 8.0.30 there are much more features, which can be used in the second argument of rebootClusterFromCompleteOutage including, which server should be considered primary therefore healthy. In fact, the updated rebootClusterFromCompleteOutage of MySQL 8.0.34 version even auto-detected the correct and healthy node and booted the MySQL InnoDB Cluster.
There were no problems with the update from MySQL 8.0.28 to MySQL 8.0.34 and after the MySQL 8.0.34 started, the rebootClusterFromCompleteOutage reconfigured and started the cluster with the right and healthy server auto-detected. In fact, it is safer to use the second argument and set the option, which is the healthy server “{primary: “db-cluster-1:3306″}”.
Keep on reading!

Install and deploy MySQL 8 InnoDB Cluster with 3 nodes under CentOS 8 and MySQL Router for HA

This article is going to show how to install a MySQL server and deploy a MySQL 8 InnoDB Cluster with three nodes behind a MySQL router to archive a high availability with MySQL database back-end.

In really simple words, MySQL 8.0 InnoDB Cluster is just MySQL replication on steroids – i.e. a little more additional work between the servers in the group before committing the transactions. It uses MySQL Group Replication plugin, which allows the group to operate in two different modes:

  1. a single-primary mode with automatic primary election. Only one server gets the updates.
  2. a multi-master mode – all servers accept the updates. For advanced setups.

Group Replication is bi-directional, the servers communicate with each other and use row replication to replicate the data. The main limitation is that only the MySQL InnoDB engine is supported, because of the transactions support. So the performance (and most features and caveats) of MySQL InnoDB is not impacted by cluster setup and overhead compared to the MySQL in replication mode (or a single server setups) from the previous MySQL versions. Still, all read-write transactions commit only after they have been approved by the group – a verification process providing consensus between the servers. In fact, most of the features like GUIDs, row-based replication (i.e. different replication modes) are developed and available to older versions. The new part is handled by Group Communication System (GCS) protocols, which provide a failure detection mechanism, a group membership service, and a safe and completely ordered message delivery (more on the subject here
In addition to the group replication, MySQL Router 8.0 provides the HAhigh availability. The program, which redirects, fails over, balances to the right server in the group is the MySQL Router. Clients may connect directly to the servers in the group, but only if the clients connect using MySQL router will have HA because Group Replication does not have a built-in method for it. It is worth noting, there could be many MySQL Routers in different servers, they do not need to communicate or synchronize anything with each other. So the router could be installed in the same place, where the application is installed or on a separate dedicated server, or on every MySQL server in the group.

Key points in this article of MySQL InnoDB Cluster deployment:

  • CentOS 8 Stream is used for the operating system
  • SELinux tuning to allow MySQL process to connect the network.
  • CentOS 8 firewall tuning to unblock the nodes traffic between them.
  • Disable mysql package system module to use the official MySQL repository.
  • Three MySQL 8.0.28 server nodes will be installed
  • To create and manage the cluster MySQL Shell 8.0 and dba object in it are used.
  • Three MySQL routers on each MySQL node will be installed.
  • Each server will have the domains of the all three servers in /etc/hosts file – db-cluster-1, db-cluster-2, db-cluster-3.
  • The cluster is in group replication with one primary (i.e. master) and two secondary nodes (i.e. slaves)

STEP 1) Install CentOS 8 Stream.

There is an article with the CentOS 8 – How to do a network installation of CentOS 8 (8.0.1950) – minimal server installation, which installation is essentially the same as CentOS 8 Stream.

STEP 2) Prepare the CentOS 8 Stream to install MySQL 8 server.

At present, the latest MySQL Community edition is 8.0.28. The preferred way to install the MySQL server is to download the RPM repository file from MySQL web site –
Keep on reading!