在CentOS6上配置MHA过程全记录

在CentOS6上配置MHA过程全记录

MHA（Master High Availability）是一款开源的MariaDB or MySQL高可用程序，为MariaDB or MySQL主从复制架构提供了automating master failover功能。
MHA有两种角色：MHA Manager（管理节点）和MHA Node（数据节点），支持自定义扩展组件。
MHA Manager：通常单独部署在一台服务器上以管理多个master/slave集群，每个集群称作一个application。
masterha_check_ssh：MHA依赖的SSH环境检测组件；
masterha_check_repl：MariaDB or MySQL复制环境检测组件；
masterha_manager：MHA管理主程序组件；
masterha_check_status：MHA状态检测组件；
masterha_master_monitor：MariaDB or MySQL master节点可用性监测组件；
masterha_master_switch：master节点切换组件：
masterha_conf_host：添加或删除配置节点的组件；
masterha_stop：关闭MHA服务的组件；
MHA Node：运行在master/slave/manager各节点上。
save_binary_logs：保存和复制master的二进制日志；
apply_diff_relay_logs：识别差异的中继日志事件并应用于其他slave；
purge_relay_logs：清理中继日志（不阻塞SQL线程）；

一、环境；
1-1.OS：
# cat /etc/redhat-release
CentOS release 6.9 (Final)

1-2.Software：
# mysql --version
mysql Ver 15.1 Distrib 5.5.57-MariaDB, for Linux (x86_64) using readline 5.1
MHA：
mha4mysql-manager-0.56-0.el6.noarch.rpm
mha4mysql-node-0.56-0.el6.noarch.rpm

1-3.各节点IP：
MHA Manager：192.168.1.61
Master Node：192.168.1.62
Slave Node 1：192.168.1.63
Slave Node 2：192.168.1.64

1-4.各节点yum源配置：
# wget -O /etc/yum.repos.d/CentOS6-Base-163.repo http://mirrors.163.com/.help/CentOS6-Base-163.repo
# sed -i 's/$releasever/6/g' /etc/yum.repos.d/CentOS6-Base-163.repo
# wget -O /etc/yum.repos.d/CentOS-Base-Ali.repo http://mirrors.aliyun.com/repo/Centos-6.repo
# sed -i 's/$releasever/6/g' /etc/yum.repos.d/CentOS-Base-Ali.repo
# vim /etc/yum.repos.d/epel.repo
[epel]
name=epel
baseurl=https://mirrors.ustc.edu.cn/epel/6Server/x86_64/
gpgcheck=0
enabled=1
# vim /etc/yum.repos.d/mariadb.repo
[mariadb]
name=mariadb
baseurl=http://mirrors.tuna.tsinghua.edu.cn/mariadb//mariadb-5.5.57/yum/centos6-amd64/
gpgcheck=0
enabled=1
//MHA Manager：192.168.1.61，可不配置mariadb.repo。
# yum groupinstall 'Development tools' 'Server Platform Development' -y

二、配置MariaDB or MySQL MS复制架构；
//可配置为MS、MSS等架构，这里配置MSS的半同步复制架构，减少主节点I/O压力，降低二进制日志丢失的风险；
2-1.Master Node：192.168.1.62
# yum install MariaDB-server -y
# vim /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
innodb_file_per_table=ON
skip_name_resolve=ON
log_bin=log-bin
relay_log=relay-log
server_id=1
# /etc/init.d/mysql start
# mysql
MariaDB [(none)]> show master status;
+----------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+----------------+----------+--------------+------------------+
| log-bin.000001 | 245 | | |
+----------------+----------+--------------+------------------+
MariaDB [(none)]> grant replication slave,replication client on *.* to 'repluser'@'192.168.1.%' identified by 'replpass';
MariaDB [(none)]> flush privileges;
MariaDB [(none)]> install plugin rpl_semi_sync_master soname 'semisync_master.so';
MariaDB [(none)]> set global rpl_semi_sync_master_enabled=1;
MariaDB [(none)]> show global variables like '%semi%';
+------------------------------------+-------+
| Variable_name | Value |
+------------------------------------+-------+
| rpl_semi_sync_master_enabled | ON |
| rpl_semi_sync_master_timeout | 10000 |
| rpl_semi_sync_master_trace_level | 32 |
| rpl_semi_sync_master_wait_no_slave | ON |
+------------------------------------+-------+

2-2.Slave Node 1：192.168.1.63
# yum install MariaDB-server -y
# vim /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
innodb_file_per_table=ON
skip_name_resolve=ON
log_bin=log-bin
relay_log=relay-log
server-id=2
read_only=1
relay_log_purge=0
# /etc/init.d/mysql start
# mysql
MariaDB [(none)]> change master to master_host='192.168.1.62',master_user='repluser',master_password='replpass',master_log_file='log-bin.000001',master_log_pos=245;
MariaDB [(none)]> start slave;
MariaDB [(none)]> show slave status \G

2-3.Slave Node 2：192.168.1.64
# yum install MariaDB-server -y
# vim /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
innodb_file_per_table=ON
skip_name_resolve=ON
log_bin=log-bin
relay_log=relay-log
server-id=3
read_only=1
relay_log_purge=0
# /etc/init.d/mysql start
# mysql
MariaDB [(none)]> change master to master_host='192.168.1.62',master_user='repluser',master_password='replpass',master_log_file='log-bin.000001',master_log_pos=245;
MariaDB [(none)]> start slave;
MariaDB [(none)]> show slave status \G

2-4.Master Node：192.168.1.62
MariaDB [(none)]> grant all on *.* to 'mhauser'@'192.168.1.%' identified by 'mhapass';
MariaDB [(none)]> flush privileges;

三、配置各节点免密码通信；
3-1：各server，192.168.1.61-64
# mkdir -pv /root/.ssh
# mkdir -pv /root/rpms

3-2：MHA Manager：192.168.1.61
# ssh-keygen -t rsa -P ''
# cat .ssh/id_rsa.pub > .ssh/authorized_keys
# chmod 600 .ssh/authorized_keys
# scp -p .ssh/id_rsa .ssh/authorized_keys 192.168.1.62:/root/.ssh
# scp -p .ssh/id_rsa .ssh/authorized_keys 192.168.1.63:/root/.ssh
# scp -p .ssh/id_rsa .ssh/authorized_keys 192.168.1.64:/root/.ssh
# ssh 192.168.1.61 'ip addr list'
# ssh 192.168.1.62 'ip addr list'
# ssh 192.168.1.63 'ip addr list'
# ssh 192.168.1.64 'ip addr list'
//可在任意server上通过ssh连接四个server（包含自身）；

四、安装及配置MHA；
4-1：MHA Manager：192.168.1.61
wget https://72003f4c60f5cc941cd1c7d448fc3c99e0aebaa8.googledrive.com/host/0B1lu97m8-haWeHdGWXp0YVVUSlk/mha4mysql-manager-0.57-0.el7.noarch.rpm
wget https://72003f4c60f5cc941cd1c7d448fc3c99e0aebaa8.googledrive.com/host/0B1lu97m8-haWeHdGWXp0YVVUSlk/mha4mysql-node-0.57-0.el7.noarch.rpm
//RHEL7 or CentOS7兼容el6版本，但最好使用el7版本，以避免perl modules类报错。本文档基于CentOS6系统，使用el6版本，请按需下载。因本人无法访问谷歌，此为保存下载链接。

# yum install mha4mysql-manager-0.56-0.el6.noarch.rpm mha4mysql-node-0.56-0.el6.noarch.rpm -y
......
Installed:
mha4mysql-manager.noarch 0:0.56-0.el6 mha4mysql-node.noarch 0:0.56-0.el6
Dependency Installed:
perl-Config-Tiny.noarch 0:2.12-7.1.el6 perl-DBD-MySQL.x86_64 0:4.013-3.el6 perl-DBI.x86_64 0:1.609-4.el6
perl-Email-Date-Format.noarch 0:1.002-5.el6 perl-Log-Dispatch.noarch 0:2.27-1.el6 perl-MIME-Lite.noarch 0:3.027-2.el6
perl-MIME-Types.noarch 0:1.28-2.el6 perl-Mail-Sender.noarch 0:0.8.16-3.el6 perl-Mail-Sendmail.noarch 0:0.79-12.el6
perl-MailTools.noarch 0:2.04-4.el6 perl-Parallel-ForkManager.noarch 0:0.7.9-1.el6 perl-Params-Validate.x86_64 0:0.92-3.el6
perl-Time-HiRes.x86_64 4:1.9721-144.el6 perl-TimeDate.noarch 1:1.16-13.el6
//上传两个软件包至MHA Manager server，通过yum安装解决依赖。

# scp /root/rpms/mha4mysql-node-0.56-0.el6.noarch.rpm 192.168.1.62:/root/rpms/
# scp /root/rpms/mha4mysql-node-0.56-0.el6.noarch.rpm 192.168.1.63:/root/rpms/
# scp /root/rpms/mha4mysql-node-0.56-0.el6.noarch.rpm 192.168.1.64:/root/rpms/

4-2：MHA Manager：192.168.1.61
# mkdir -pv /dbdata/masterha/app1
# mkdir -pv /etc/masterha
# vim /etc/masterha/app1.cnf
[server default]
user=mhauser
password=mhapass
manager_workdir=/dbdata/masterha/app1
manager_log=/dbdata/masterha/app1/manager.log
remote_workdir=/dbdata/masterha/app1
ssh_user=root
repl_user=repluser
repl_password=replpass
ping_interval=1
[server1]
hostname=192.168.1.62
candidate_master=1
#ssh_port=22
[server2]
hostname=192.168.1.63
candidate_master=1
[server3]
hostname=192.168.1.64
#no_master=1

4-3：192.168.1.62-64
# yum install mha4mysql-node-0.56-0.el6.noarch.rpm -y
......
Installed:
mha4mysql-node.noarch 0:0.56-0.el6
Dependency Installed:
perl-DBD-MySQL.x86_64 0:4.013-3.el6

4-4：MHA Manager：192.168.1.61
# masterha_check_ssh --conf=/etc/masterha/app1.cnf
Sat Oct 7 00:57:39 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sat Oct 7 00:57:39 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Sat Oct 7 00:57:39 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Sat Oct 7 00:57:39 2017 - [info] Starting SSH connection tests..
Sat Oct 7 00:57:40 2017 - [debug]
Sat Oct 7 00:57:39 2017 - [debug] Connecting via SSH from root@192.168.1.62(192.168.1.62:22) to root@192.168.1.63(192.168.1.63:22)..
Address 192.168.1.62 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Address 192.168.1.63 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Sat Oct 7 00:57:40 2017 - [debug] ok.
Sat Oct 7 00:57:40 2017 - [debug] Connecting via SSH from root@192.168.1.62(192.168.1.62:22) to root@192.168.1.64(192.168.1.64:22)..
Address 192.168.1.62 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Address 192.168.1.64 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Sat Oct 7 00:57:40 2017 - [debug] ok.
Sat Oct 7 00:57:41 2017 - [debug]
Sat Oct 7 00:57:40 2017 - [debug] Connecting via SSH from root@192.168.1.63(192.168.1.63:22) to root@192.168.1.62(192.168.1.62:22)..
Address 192.168.1.63 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Address 192.168.1.62 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Sat Oct 7 00:57:40 2017 - [debug] ok.
Sat Oct 7 00:57:40 2017 - [debug] Connecting via SSH from root@192.168.1.63(192.168.1.63:22) to root@192.168.1.64(192.168.1.64:22)..
Address 192.168.1.63 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Address 192.168.1.64 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Sat Oct 7 00:57:40 2017 - [debug] ok.
Sat Oct 7 00:57:41 2017 - [debug]
Sat Oct 7 00:57:40 2017 - [debug] Connecting via SSH from root@192.168.1.64(192.168.1.64:22) to root@192.168.1.62(192.168.1.62:22)..
Address 192.168.1.64 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Address 192.168.1.62 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Sat Oct 7 00:57:41 2017 - [debug] ok.
Sat Oct 7 00:57:41 2017 - [debug] Connecting via SSH from root@192.168.1.64(192.168.1.64:22) to root@192.168.1.63(192.168.1.63:22)..
Address 192.168.1.64 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Address 192.168.1.63 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Sat Oct 7 00:57:41 2017 - [debug] ok.
Sat Oct 7 00:57:41 2017 - [info] All SSH connection tests passed successfully.

# masterha_check_repl --conf=/etc/masterha/app1.cnf
Sat Oct 7 02:48:16 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sat Oct 7 02:48:16 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Sat Oct 7 02:48:16 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Sat Oct 7 02:48:16 2017 - [info] MHA::MasterMonitor version 0.56.
Sat Oct 7 02:48:16 2017 - [info] GTID failover mode = 0
Sat Oct 7 02:48:16 2017 - [info] Dead Servers:
Sat Oct 7 02:48:16 2017 - [info] Alive Servers:
Sat Oct 7 02:48:16 2017 - [info] 192.168.1.62(192.168.1.62:3306)
Sat Oct 7 02:48:16 2017 - [info] 192.168.1.63(192.168.1.63:3306)
Sat Oct 7 02:48:16 2017 - [info] 192.168.1.64(192.168.1.64:3306)
Sat Oct 7 02:48:16 2017 - [info] Alive Slaves:
Sat Oct 7 02:48:16 2017 - [info] 192.168.1.63(192.168.1.63:3306) Version=5.5.57-MariaDB (oldest major version between slaves) log-bin:enabled
Sat Oct 7 02:48:16 2017 - [info] Replicating from 192.168.1.62(192.168.1.62:3306)
Sat Oct 7 02:48:16 2017 - [info] Primary candidate for the new Master (candidate_master is set)
Sat Oct 7 02:48:16 2017 - [info] 192.168.1.64(192.168.1.64:3306) Version=5.5.57-MariaDB (oldest major version between slaves) log-bin:enabled
Sat Oct 7 02:48:16 2017 - [info] Replicating from 192.168.1.62(192.168.1.62:3306)
Sat Oct 7 02:48:16 2017 - [info] Current Alive Master: 192.168.1.62(192.168.1.62:3306)
Sat Oct 7 02:48:16 2017 - [info] Checking slave configurations..
Sat Oct 7 02:48:16 2017 - [warning] relay_log_purge=0 is not set on slave 192.168.1.63(192.168.1.63:3306).
Sat Oct 7 02:48:16 2017 - [warning] relay_log_purge=0 is not set on slave 192.168.1.64(192.168.1.64:3306).
Sat Oct 7 02:48:16 2017 - [info] Checking replication filtering settings..
Sat Oct 7 02:48:16 2017 - [info] binlog_do_db= , binlog_ignore_db=
Sat Oct 7 02:48:16 2017 - [info] Replication filtering check ok.
Sat Oct 7 02:48:16 2017 - [info] GTID (with auto-pos) is not supported
Sat Oct 7 02:48:16 2017 - [info] Starting SSH connection tests..
Sat Oct 7 02:48:17 2017 - [info] All SSH connection tests passed successfully.
Sat Oct 7 02:48:17 2017 - [info] Checking MHA Node version..
Sat Oct 7 02:48:18 2017 - [info] Version check ok.
Sat Oct 7 02:48:18 2017 - [info] Checking SSH publickey authentication settings on the current master..
Address 192.168.1.62 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Sat Oct 7 02:48:18 2017 - [info] HealthCheck: SSH to 192.168.1.62 is reachable.
Sat Oct 7 02:48:18 2017 - [info] Master MHA Node version is 0.56.
Sat Oct 7 02:48:18 2017 - [info] Checking recovery script configurations on 192.168.1.62(192.168.1.62:3306)..
Sat Oct 7 02:48:18 2017 - [info] Executing command: save_binary_logs --command=test --start_pos=4 --binlog_dir=/var/lib/mysql,/var/log/mysql --output_file=/dbdata/masterha/app1/save_binary_logs_test --manager_version=0.56 --start_file=log-bin.000001
Sat Oct 7 02:48:18 2017 - [info] Connecting to root@192.168.1.62(192.168.1.62:22)..
Address 192.168.1.62 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Creating /dbdata/masterha/app1 if not exists.. Creating directory /dbdata/masterha/app1.. done.
ok.
Checking output directory is accessible or not..
ok.
Binlog found at /var/lib/mysql, up to log-bin.000001
Sat Oct 7 02:48:18 2017 - [info] Binlog setting check done.
Sat Oct 7 02:48:18 2017 - [info] Checking SSH publickey authentication and checking recovery script configurations on all alive slave servers..
Sat Oct 7 02:48:18 2017 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.1.63 --slave_ip=192.168.1.63 --slave_port=3306 --workdir=/dbdata/masterha/app1 --target_version=5.5.57-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Sat Oct 7 02:48:18 2017 - [info] Connecting to root@192.168.1.63(192.168.1.63:22)..
Address 192.168.1.63 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Creating directory /dbdata/masterha/app1.. done.
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to relay-log.000002
Temporary relay log file is /var/lib/mysql/relay-log.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Sat Oct 7 02:48:19 2017 - [info] Executing command : apply_diff_relay_logs --command=test --slave_user='mhauser' --slave_host=192.168.1.64 --slave_ip=192.168.1.64 --slave_port=3306 --workdir=/dbdata/masterha/app1 --target_version=5.5.57-MariaDB --manager_version=0.56 --relay_log_info=/var/lib/mysql/relay-log.info --relay_dir=/var/lib/mysql/ --slave_pass=xxx
Sat Oct 7 02:48:19 2017 - [info] Connecting to root@192.168.1.64(192.168.1.64:22)..
Address 192.168.1.64 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Creating directory /dbdata/masterha/app1.. done.
Checking slave recovery environment settings..
Opening /var/lib/mysql/relay-log.info ... ok.
Relay log found at /var/lib/mysql, up to relay-log.000002
Temporary relay log file is /var/lib/mysql/relay-log.000002
Testing mysql connection and privileges.. done.
Testing mysqlbinlog output.. done.
Cleaning up test file(s).. done.
Sat Oct 7 02:48:19 2017 - [info] Slaves settings check done.
Sat Oct 7 02:48:19 2017 - [info]
192.168.1.62(192.168.1.62:3306) (current master)
+--192.168.1.63(192.168.1.63:3306)
+--192.168.1.64(192.168.1.64:3306)

Sat Oct 7 02:48:19 2017 - [info] Checking replication health on 192.168.1.63..
Sat Oct 7 02:48:19 2017 - [info] ok.
Sat Oct 7 02:48:19 2017 - [info] Checking replication health on 192.168.1.64..
Sat Oct 7 02:48:19 2017 - [info] ok.
Sat Oct 7 02:48:19 2017 - [warning] master_ip_failover_script is not defined.
Sat Oct 7 02:48:19 2017 - [warning] shutdown_script is not defined.
Sat Oct 7 02:48:19 2017 - [info] Got exit code 0 (Not master dead).
MySQL Replication Health is OK.

# masterha_manager --conf=/etc/masterha/app1.cnf
Sat Oct 7 03:10:48 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sat Oct 7 03:10:48 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Sat Oct 7 03:10:48 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Address 192.168.1.62 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
//masterha_manager程序运行在前台

五、测试故障转移；
5-1：Master Node：192.168.1.62
# killall mysqld mysql_safe

MHA Manager：192.168.1.61
# masterha_manager --conf=/etc/masterha/app1.cnf
Sat Oct 7 03:10:48 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sat Oct 7 03:10:48 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Sat Oct 7 03:10:48 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Address 192.168.1.62 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Creating /dbdata/masterha/app1 if not exists.. ok.
Checking output directory is accessible or not..
ok.
Binlog found at /var/lib/mysql, up to log-bin.000001
Sat Oct 7 03:14:42 2017 - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Sat Oct 7 03:14:42 2017 - [info] Reading application default configuration from /etc/masterha/app1.cnf..
Sat Oct 7 03:14:42 2017 - [info] Reading server configuration from /etc/masterha/app1.cnf..
Address 192.168.1.62 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Address 192.168.1.63 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Address 192.168.1.64 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Address 192.168.1.63 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
Address 192.168.1.64 maps to bogon, but this does not map back to the address - POSSIBLE BREAK-IN ATTEMPT!
#
//masterha_manager完成自动切换后退出

5-2：Slave Node 1：192.168.1.63
MariaDB [(none)]> show slave status \G
Empty set (0.00 sec)
MariaDB [(none)]> show master status;
+----------------+----------+--------------+------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB |
+----------------+----------+--------------+------------------+
| log-bin.000001 | 245 | | |
+----------------+----------+--------------+------------------+
1 row in set (0.00 sec)
MariaDB [(none)]> show global variables like 'read_only';
+---------------+-------+
| Variable_name | Value |
+---------------+-------+
| read_only | OFF |
+---------------+-------+
1 row in set (0.00 sec)

MariaDB [(none)]> grant replication slave,replication client on *.* to 'repluser'@'192.168.1.%' identified by 'replpass';
MariaDB [(none)]> flush privileges;
MariaDB [(none)]> install plugin rpl_semi_sync_master soname 'semisync_master.so';
MariaDB [(none)]> set global rpl_semi_sync_master_enabled=1;
MariaDB [(none)]> show global variables like '%semi%';
+------------------------------------+-------+
| Variable_name | Value |
+------------------------------------+-------+
| rpl_semi_sync_master_enabled | ON |
| rpl_semi_sync_master_timeout | 10000 |
| rpl_semi_sync_master_trace_level | 32 |
| rpl_semi_sync_master_wait_no_slave | ON |
+------------------------------------+-------+
MariaDB [(none)]> grant all on *.* to 'mhauser'@'192.168.1.%' identified by 'mhapass';
MariaDB [(none)]> flush privileges;

5-3：Slave Node 2：192.168.1.64
MariaDB [(none)]> show slave status \G
*************************** 1. row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.1.63
Master_User: repluser
Master_Port: 3306
Connect_Retry: 60
Master_Log_File: log-bin.000001
Read_Master_Log_Pos: 245
Relay_Log_File: relay-log.000002
Relay_Log_Pos: 527
Relay_Master_Log_File: log-bin.000001
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
......

六、修复节点重上线或新节点上线；
6-1：修复节点重上线；
//从当前主节点导入最新备份；（因本文档系模拟实验，无数据，过程略。）
Repaired Node：192.168.1.62
# vim /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
innodb_file_per_table=ON
skip_name_resolve=ON
log_bin=log-bin
relay_log=relay-log
server-id=1
read_only=1
relay_log_purge=0
# /etc/init.d/mysql start
# mysql
MariaDB [(none)]> change master to master_host='192.168.1.63',master_user='repluser',master_password='replpass',master_log_file='log-bin.000001',master_log_pos=245;
//注意：此时的“lig-bin.000001”及“pos”以当前主节点192.168.1.63的查询为依据；
MariaDB [(none)]> start slave;
MariaDB [(none)]> show slave status \G

6-2：新节点上线；
New Node：192.168.1.65
# mkdir -pv /root/.ssh
# mkdir -pv /root/rpms

MHA Manager：192.168.1.61
# scp -p .ssh/id_rsa .ssh/authorized_keys 192.168.1.65:/root/.ssh
# scp /root/rpms/mha4mysql-node-0.56-0.el6.noarch.rpm 192.168.1.62:/root/rpms/

New Node：192.168.1.65
# yum install MariaDB-server -y
//DB安装完成后，要从当前主节点导入最新备份；（因本文档系模拟实验，无数据，过程略。）
# vim /etc/my.cnf
[mysqld]
datadir=/var/lib/mysql
socket=/var/lib/mysql/mysql.sock
innodb_file_per_table=ON
skip_name_resolve=ON
log_bin=log-bin
relay_log=relay-log
server-id=4
read_only=1
relay_log_purge=0
# /etc/init.d/mysql start
# mysql
MariaDB [(none)]> change master to master_host='192.168.1.63',master_user='repluser',master_password='replpass',master_log_file='log-bin.000001',master_log_pos=245;
//注意：此时的“lig-bin.000001”及“pos”以当前主节点192.168.1.63的查询为依据；
MariaDB [(none)]> start slave;
MariaDB [(none)]> show slave status \G

6-3：重新检测SSH、repl、启动masterha_manager；
# nohup masterha_manager --conf=/etc/masterha/app1.cnf [--remove_dead_master_conf --ignore_last_failover < /dev/null] > /dbdata/masterha/app1/manager.log 2>&1 &
//重新运行MHA程序，nohup可剥离当前终端，其他可选择参数及说明：
--remove_dead_master_conf
该参数代表当发生主从切换后，原master的ip将会从配置文件中移除。
--ignore_last_failover
在缺省情况下，如果MHA检测到连续发生宕机，且两次宕机间隔不足8小时的话，则不会进行Failover，之所以这样限制是为了避免ping-pong效应。该参数代表忽略上次MHA触发切换产生的文件，默认情况下，MHA发生切换后会在日志目录，也就是上面我设置的/data产生app1.failover.complete文件，下次再次切换的时候如果发现该目录下存在该文件将不允许触发切换，除非在第一次切换后收到删除该文件，为了方便，这里设置为--ignore_last_failover。