【Mysql MHA】CentOS7.6+Mysql8.0.16 入坑

1.防火墙

firewall-cmd --add-port=/tcp --permanent
firewall-cmd --reload

2.SELINUX

sed -i 's/SELINUX=enforcing/SELINUX=disabled/g' /etc/selinux/config
setenforce 

3.软件包需求

yum install -y perl-Class-Load.noarch perl-DBD-MySQL

#以下需下载，地址： https://pkgs.org/
yum install -y perl-Log-Dispatch-2.41-.of.el7.noarch.rpm
yum install -y perl-Config-Tiny-2.14-.el7.noarch.rpm
yum install -y perl-Parallel-ForkManager-1.06-.of.el7.noarch.rpm

#MHA包需下载，地址 https://github.com/yoshinorim
mha4mysql-manager-0.58-.el7.centos.noarch.rpm #建议安装在独立服务器，也可选择任一从库安装
mha4mysql-node-0.58-.el7.centos.noarch.rpm #安装在所有节点

4.密码认证插件需要用 mysql_native_password

[mysqld]
default_authentication_plugin=mysql_native_password

alter user rep identified with 'mysql_native_password' by 'rep';
alter user root identified with 'mysql_native_password' by 'root ';

5. 检查配置是否成功

建议所有从库都置为只读模式 read_only=1,但这实际并不影响MHA搭建。

[root@mysql3 soft]# masterha_check_repl --conf=/etc/masterha_default.cnf
Mon Jun  ::  - [info] Reading default configuration from /etc/masterha_default.cnf..
Mon Jun  ::  - [info] Reading application default configuration from /etc/masterha_default.cnf..
Mon Jun  ::  - [info] Reading server configuration from /etc/masterha_default.cnf..
Mon Jun  ::  - [info] MHA::MasterMonitor version 0.58.
Mon Jun  ::  - [info] GTID failover mode =
Mon Jun  ::  - [info] Dead Servers:
Mon Jun  ::  - [info] Alive Servers:
Mon Jun  ::  - [info]   mysql1(192.168.20.81:)
Mon Jun  ::  - [info]   mysql2(192.168.20.82:)
Mon Jun  ::  - [info]   mysql3(192.168.20.83:)
Mon Jun  ::  - [info]   mysql4(192.168.20.84:)
Mon Jun  ::  - [info] Alive Slaves:
Mon Jun  ::  - [info]   mysql2(192.168.20.82:)  Version=8.0. (oldest major version between slaves) log-bin:enabled
Mon Jun  ::  - [info]     GTID ON
Mon Jun  ::  - [info]     Replicating from 192.168.20.81(192.168.20.81:)
Mon Jun  ::  - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Jun  ::  - [info]   mysql3(192.168.20.83:)  Version=8.0. (oldest major version between slaves) log-bin:enabled
Mon Jun  ::  - [info]     GTID ON
Mon Jun  ::  - [info]     Replicating from 192.168.20.81(192.168.20.81:)
Mon Jun  ::  - [info]     Not candidate for the new Master (no_master is set)
Mon Jun  ::  - [info]   mysql4(192.168.20.84:)  Version=8.0. (oldest major version between slaves) log-bin:enabled
Mon Jun  ::  - [info]     GTID ON
Mon Jun  ::  - [info]     Replicating from 192.168.20.81(192.168.20.81:)
Mon Jun  ::  - [info]     Not candidate for the new Master (no_master is set)
Mon Jun  ::  - [info] Current Alive Master: mysql1(192.168.20.81:)
Mon Jun  ::  - [info] Checking slave configurations..
Mon Jun  ::  - [info]  read_only= is not set on slave mysql2(192.168.20.82:).
Mon Jun  ::  - [info] Checking replication filtering settings..
Mon Jun  ::  - [info]  binlog_do_db= , binlog_ignore_db=
Mon Jun  ::  - [info]  Replication filtering check ok.
Mon Jun  ::  - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Mon Jun  ::  - [info] Checking SSH publickey authentication settings on the current master..
Mon Jun  ::  - [info] HealthCheck: SSH to mysql1 is reachable.
Mon Jun  ::  - [info]
mysql1(192.168.20.81:) (current master)
 +--mysql2(192.168.20.82:)
 +--mysql3(192.168.20.83:)
 +--mysql4(192.168.20.84:)

Mon Jun  ::  - [info] Checking replication health on mysql2..
Mon Jun  ::  - [info]  ok.
Mon Jun  ::  - [info] Checking replication health on mysql3..
Mon Jun  ::  - [info]  ok.
Mon Jun  ::  - [info] Checking replication health on mysql4..
Mon Jun  ::  - [info]  ok.
Mon Jun  ::  - [warning] master_ip_failover_script is not defined.
Mon Jun  ::  - [warning] shutdown_script is not defined.
Mon Jun  ::  - [info] Got exit code  (Not master dead).

MySQL Replication Health is OK.

6.以上我用的GTID 复制模式

[mysqld]
enforce_gtid_consistency=ON
gtid_mode=on

--从库执行如下命令，启动复制
change master to
        master_host='192.168.20.81',
        master_port=3306,
        master_auto_position=1;
start slave user='rep' password='rep';

7.MHA 相关命令

masterha_check_ssh --conf=/etc/masterha_default.cnf
masterha_check_repl --conf=/etc/masterha_default.cnf

masterha_master_switch　　　　　　　　#可用来执行 fail over/online master switch
# For master failover
masterha_master_switch --master_state=dead
    --global_conf=/etc/masterha_default.cnf
    --conf=/usr/local/masterha/conf/app1.cnf --dead_master_host=host1
# For online master switch
    masterha_master_switch --master_state=alive
    --global_conf=/etc/masterha_default.cnf
    --conf=/usr/local/masterha/conf/app1.cnf

masterha_master_monitor --global_conf=/etc/masterha_default.cnf --conf=/usr/local/masterha/app1.cnf

masterha_secondary_check -s secondary_host1 -s secondary_host2 .. --user=ssh_username --master_host=host --master_ip=ip --master_port=port

masterha_manager --global_conf=/etc/masterha_default.cnf --conf=/usr/local/masterha/conf/app1.cnf

# 以上的--global_conf=/etc/masterha_default.cnf 参数均可省略

8.MHA 在线切换测试

[root@mysql3 soft]# masterha_master_switch --master_state=alive --conf=/etc/masterha_default.cnf
Mon Jun  ::  - [info] MHA::MasterRotate version 0.58.
Mon Jun  ::  - [info] Starting online master switch..
Mon Jun  ::  - [info]
Mon Jun  ::  - [info] * Phase : Configuration Check Phase..
Mon Jun  ::  - [info]
Mon Jun  ::  - [info] Reading default configuration from /etc/masterha_default.cnf..
Mon Jun  ::  - [info] Reading application default configuration from /etc/masterha_default.cnf..
Mon Jun  ::  - [info] Reading server configuration from /etc/masterha_default.cnf..
Mon Jun  ::  - [info] GTID failover mode =
Mon Jun  ::  - [info] Current Alive Master: mysql1(192.168.20.81:)
Mon Jun  ::  - [info] Alive Slaves:
Mon Jun  ::  - [info]   mysql2(192.168.20.82:)  Version=8.0. (oldest major version between slaves) log-bin:enabled
Mon Jun  ::  - [info]     GTID ON
Mon Jun  ::  - [info]     Replicating from 192.168.20.81(192.168.20.81:)
Mon Jun  ::  - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Jun  ::  - [info]   mysql3(192.168.20.83:)  Version=8.0. (oldest major version between slaves) log-bin:enabled
Mon Jun  ::  - [info]     GTID ON
Mon Jun  ::  - [info]     Replicating from 192.168.20.81(192.168.20.81:)
Mon Jun  ::  - [info]     Not candidate for the new Master (no_master is set)
Mon Jun  ::  - [info]   mysql4(192.168.20.84:)  Version=8.0. (oldest major version between slaves) log-bin:enabled
Mon Jun  ::  - [info]     GTID ON
Mon Jun  ::  - [info]     Replicating from 192.168.20.81(192.168.20.81:)
Mon Jun  ::  - [info]     Not candidate for the new Master (no_master is set)

It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on mysql1(192.168.20.81:)? (YES/no): YES
Mon Jun  ::  - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Mon Jun  ::  - [info]  ok.
Mon Jun  ::  - [info] Checking MHA is not monitoring or doing failover..
Mon Jun  ::  - [info] Checking replication health on mysql2..
Mon Jun  ::  - [info]  ok.
Mon Jun  ::  - [info] Checking replication health on mysql3..
Mon Jun  ::  - [info]  ok.
Mon Jun  ::  - [info] Checking replication health on mysql4..
Mon Jun  ::  - [info]  ok.
Mon Jun  ::  - [info] Searching new master from slaves..
Mon Jun  ::  - [info]  Candidate masters from the configuration file:
Mon Jun  ::  - [info]   mysql1(192.168.20.81:)  Version=8.0. log-bin:enabled
Mon Jun  ::  - [info]     GTID ON
Mon Jun  ::  - [info]   mysql2(192.168.20.82:)  Version=8.0. (oldest major version between slaves) log-bin:enabled
Mon Jun  ::  - [info]     GTID ON
Mon Jun  ::  - [info]     Replicating from 192.168.20.81(192.168.20.81:)
Mon Jun  ::  - [info]     Primary candidate for the new Master (candidate_master is set)
Mon Jun  ::  - [info]  Non-candidate masters:
Mon Jun  ::  - [info]   mysql3(192.168.20.83:)  Version=8.0. (oldest major version between slaves) log-bin:enabled
Mon Jun  ::  - [info]     GTID ON
Mon Jun  ::  - [info]     Replicating from 192.168.20.81(192.168.20.81:)
Mon Jun  ::  - [info]     Not candidate for the new Master (no_master is set)
Mon Jun  ::  - [info]   mysql4(192.168.20.84:)  Version=8.0. (oldest major version between slaves) log-bin:enabled
Mon Jun  ::  - [info]     GTID ON
Mon Jun  ::  - [info]     Replicating from 192.168.20.81(192.168.20.81:)
Mon Jun  ::  - [info]     Not candidate for the new Master (no_master is set)
Mon Jun  ::  - [info]  Searching from candidate_master slaves which have received the latest relay log events..
Mon Jun  ::  - [info]
From:
mysql1(192.168.20.81:) (current master)
 +--mysql2(192.168.20.82:)
 +--mysql3(192.168.20.83:)
 +--mysql4(192.168.20.84:)

To:
mysql2(192.168.20.82:) (new master)
 +--mysql3(192.168.20.83:)
 +--mysql4(192.168.20.84:)

Starting master switch from mysql1(192.168.20.81:) to mysql2(192.168.20.82:)? (yes/NO): yes
Mon Jun  ::  - [info] Checking whether mysql2(192.168.20.82:) is ok for the new master..
Mon Jun  ::  - [info]  ok.
Mon Jun  ::  - [info] ** Phase : Configuration Check Phase completed.
Mon Jun  ::  - [info]
Mon Jun  ::  - [info] * Phase : Rejecting updates Phase..
Mon Jun  ::  - [info]
FLUSH NO_WRITE_TO_BINLOG TABLES is not defined. If you do not disable writes on the current master manually, applications keep writing on the current master. Is it ok to proceed? (yes/NO): yes
Mon Jun  ::  - [info] Locking all tables on the orig master to reject updates from everybody (including root):
Mon Jun  ::  - [info] Executing FLUSH TABLES WITH READ LOCK..
Mon Jun  ::  - [info]  ok.
Mon Jun  ::  - [info] Orig master binlog:pos is binlog.:.
Mon Jun  ::  - [info]  Waiting to execute all relay logs on mysql2(192.168.20.82:)..
Mon Jun  ::  - [info]  master_pos_wait(binlog.:) completed on mysql2(192.168.20.82:). Executed  events.
Mon Jun  ::  - [info]   done.
Mon Jun  ::  - [info] Getting new master's binlog name and position..
Mon Jun  ::  - [info]  binlog.:
Mon Jun  ::  - [info]  All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='mysql2 or 192.168.20.82', MASTER_PORT=, MASTER_AUTO_POSITION=, MASTER_USER='rep', MASTER_PASSWORD='xxx';
Mon Jun  ::  - [info] Setting read_only= on mysql2(192.168.20.82:)..
Mon Jun  ::  - [info]  ok.
Mon Jun  ::  - [info]
Mon Jun  ::  - [info] * Switching slaves in parallel..
Mon Jun  ::  - [info]
Mon Jun  ::  - [info] -- Slave switch on host mysql3(192.168.20.83:) started, pid:
Mon Jun  ::  - [info]
Mon Jun  ::  - [info] -- Slave switch on host mysql4(192.168.20.84:) started, pid:
Mon Jun  ::  - [info]
Mon Jun  ::  - [info] Log messages from mysql3 ...
Mon Jun  ::  - [info]
Mon Jun  ::  - [info]  Waiting to execute all relay logs on mysql3(192.168.20.83:)..
Mon Jun  ::  - [info]  master_pos_wait(binlog.:) completed on mysql3(192.168.20.83:). Executed  events.
Mon Jun  ::  - [info]   done.
Mon Jun  ::  - [info]  Resetting slave mysql3(192.168.20.83:) and starting replication from the new master mysql2(192.168.20.82:)..
Mon Jun  ::  - [info]  Executed CHANGE MASTER.
Mon Jun  ::  - [info]  Slave started.
Mon Jun  ::  - [info] End of log messages from mysql3 ...
Mon Jun  ::  - [info]
Mon Jun  ::  - [info] -- Slave switch on host mysql3(192.168.20.83:) succeeded.
Mon Jun  ::  - [info] Log messages from mysql4 ...
Mon Jun  ::  - [info]
Mon Jun  ::  - [info]  Waiting to execute all relay logs on mysql4(192.168.20.84:)..
Mon Jun  ::  - [info]  master_pos_wait(binlog.:) completed on mysql4(192.168.20.84:). Executed  events.
Mon Jun  ::  - [info]   done.
Mon Jun  ::  - [info]  Resetting slave mysql4(192.168.20.84:) and starting replication from the new master mysql2(192.168.20.82:)..
Mon Jun  ::  - [info]  Executed CHANGE MASTER.
Mon Jun  ::  - [info]  Slave started.
Mon Jun  ::  - [info] End of log messages from mysql4 ...
Mon Jun  ::  - [info]
Mon Jun  ::  - [info] -- Slave switch on host mysql4(192.168.20.84:) succeeded.
Mon Jun  ::  - [info] Unlocking all tables on the orig master:
Mon Jun  ::  - [info] Executing UNLOCK TABLES..
Mon Jun  ::  - [info]  ok.
Mon Jun  ::  - [info] All new slave servers switched successfully.
Mon Jun  ::  - [info]
Mon Jun  ::  - [info] * Phase : New master cleanup phase..
Mon Jun  ::  - [info]
Mon Jun  ::  - [info]  mysql2: Resetting slave info succeeded.
Mon Jun  ::  - [info] Switching master to mysql2(192.168.20.82:) completed successfully.

执行过程中提供了几次YES输入，还有几个重要的语句：

检查阶段：
master 执行 FLUSH NO_WRITE_TO_BINLOG TABLES
搜寻candidate_master= 的slave 节点，选择其为新master 

执行阶段：
FLUSH NO_WRITE_TO_BINLOG TABLES is not defined. If you do not disable writes on the current master manually, applications keep writing on the current master. Is it ok to proceed? (yes/NO): yes
Locking all tables on the orig master to reject updates from everybody (including root):
Executing FLUSH TABLES WITH READ LOCK..
All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='mysql2 or 192.168.20.82', MASTER_PORT=, MASTER_AUTO_POSITION=, MASTER_USER='rep', MASTER_PASSWORD='xxx';
Setting read_only= on mysql2(192.168.20.82:)..
Unlocking all tables on the orig master:
Executing UNLOCK TABLES..

9.在线切换注意点

1. 会从 candidate_master=1的slave节点中选择新master

2.被选择的新master 修改为read_only=0

3.原来的master 修改为read_only=1

4. 切换后，原master 不会转为新master 的 slave。

5.配置文件不会发生改变，所以重启需注意。

 

10.将原master 改为新master 的slave

--将原master 修改为新master 的从库
change master to  master_host='192.168.20.82',master_port=3306,master_auto_position=1;
start slave user='rep' password='rep';
show slave status \G

11. 切换回原来的master

# masterha_master_switch --master_state=alive --conf=/etc/masterha_default.cnf 

同样的需要把node2 改为slave
change master to  master_host='192.168.20.81',master_port=3306,master_auto_position=1;
start slave user='rep' password='rep';
show slave status \G