MySQL高可用之MHA切换测试（switchover & failover）
Preface
I've installed MasterHA yesterday,Now let's test the master-slave switch and failover feature.
Framework
| Hostname | IP | Port | Identity | OS Version | MySQL Version |
| zlm2 | 192.168.1.101 | 3306 | master | CentOS 7.0 | 5.7.21 |
| zlm3 | 192.168.1.102 | 3306 | slave/mha-manager | CentOS 7.0 | 5.7.21 |
| null | 192.168.1.200 | null | vip | null | null |
Procedure
Test 1：Manual master switchover
Check state of MHA-manager on zlm3.
[root@zlm3 :: ~]
#masterha_check_ssh --conf=/etc/masterha/app1.conf
Fri Aug :: - [warning] Global configuration file /etc/masterha_default.cnf not found. Skipping.
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Starting SSH connection tests..
Fri Aug :: - [debug]
Fri Aug :: - [debug] Connecting via SSH from root@192.168.1.101(192.168.1.101:) to root@192.168.1.102(192.168.1.102:)..
Fri Aug :: - [debug] ok.
Fri Aug :: - [debug]
Fri Aug :: - [debug] Connecting via SSH from root@192.168.1.102(192.168.1.102:) to root@192.168.1.101(192.168.1.101:)..
Fri Aug :: - [debug] ok.
Fri Aug :: - [info] All SSH connection tests passed successfully. [root@zlm3 :: ~]
#masterha_check_repl --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] MHA::MasterMonitor version 0.56.
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [info] Alive Servers:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] Current Alive Master: 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Checking slave configurations..
Fri Aug :: - [info] read_only= is not set on slave 192.168.1.102(192.168.1.102:).
Fri Aug :: - [info] Checking replication filtering settings..
Fri Aug :: - [info] binlog_do_db= , binlog_ignore_db=
Fri Aug :: - [info] Replication filtering check ok.
Fri Aug :: - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Fri Aug :: - [info] Checking SSH publickey authentication settings on the current master..
ssh_exchange_identification: Connection closed by remote host
Fri Aug :: - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
Fri Aug :: - [info]
192.168.1.101(192.168.1.101:) (current master)
+--192.168.1.102(192.168.1.102:) Fri Aug :: - [info] Checking replication health on 192.168.1.102..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Checking master_ip_failover_script status:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --orig_master_ssh_port=
Fri Aug :: - [info] OK.
Fri Aug :: - [warning] shutdown_script is not defined.
Fri Aug :: - [info] Got exit code (Not master dead). MySQL Replication Health is OK. [root@zlm3 :: ~]
#Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
ssh_exchange_identification: Connection closed by remote host
^C [root@zlm3 :: ~]
#masterha_check_status --conf=/etc/masterha/app1.conf
app1 (pid:) is running(:PING_OK), master:192.168.1.101
Switch master to slave and make it become a new slave of new master.
[root@zlm3 :: ~]
#masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --master_state=alive --new_master_host=192.168.1.102 --orig_master_is_new_slave --running_updates_limit=
Fri Aug :: - [info] MHA::MasterRotate version 0.56.
Fri Aug :: - [info] Starting online master switch..
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Configuration Check Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Current Alive Master: 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set) It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.1.101(192.168.1.101:)? (YES/no): yes
Fri Aug :: - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Checking MHA is not monitoring or doing failover..
Fri Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/MasterRotate.pm, ln142] Getting advisory lock failed on the current master. MHA Monitor runs on the current master. Stop MHA Manager/Monitor and try again.
Fri Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm, ln177] Got ERROR: at /usr/bin/masterha_master_switch line . //It means that we should stop MHA-manager when donging switchover master. [root@zlm3 :: ~]
#masterha_stop --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf
Stopped app1 successfully.
[]+ Exit masterha_manager --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf [root@zlm3 :: ~]
#masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --master_state=alive --new_master_host=192.168.1.102 --orig_master_is_new_slave --running_updates_limit=
Fri Aug :: - [info] MHA::MasterRotate version 0.56.
Fri Aug :: - [info] Starting online master switch..
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Configuration Check Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Current Alive Master: 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set) It is better to execute FLUSH NO_WRITE_TO_BINLOG TABLES on the master before switching. Is it ok to execute on 192.168.1.101(192.168.1.101:)? (YES/no): yes
Fri Aug :: - [info] Executing FLUSH NO_WRITE_TO_BINLOG TABLES. This may take long time..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Checking MHA is not monitoring or doing failover..
Fri Aug :: - [info] Checking replication health on 192.168.1.102..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] 192.168.1.102 can be new master.
Fri Aug :: - [info]
From:
192.168.1.101(192.168.1.101:) (current master)
+--192.168.1.102(192.168.1.102:) To:
192.168.1.102(192.168.1.102:) (new master)
+--192.168.1.101(192.168.1.101:) Starting master switch from 192.168.1.101(192.168.1.101:) to 192.168.1.102(192.168.1.102:)? (yes/NO): yes
Fri Aug :: - [info] Checking whether 192.168.1.102(192.168.1.102:) is ok for the new master..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:): SHOW SLAVE STATUS returned empty result. To check replication filtering rules, temporarily executing CHANGE MASTER to a dummy host.
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:): Resetting slave pointing to the dummy host.
Fri Aug :: - [info] ** Phase : Configuration Check Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Rejecting updates Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Executing master ip online change script to disable write on the current master:
Fri Aug :: - [info] /etc/masterha/master_ip_online_change --command=stop --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --orig_master_user='zlm' --orig_master_password='zlmzlm' --new_master_host=192.168.1.102 --new_master_ip=192.168.1.102 --new_master_port= --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_ssh_port= --new_master_ssh_port= --orig_master_is_new_slave
Unknown option: new_master_ssh_port
Fri Aug :: Set read_only on the new master.. ok.
Fri Aug :: drop vip 10.33.101.239..
ssh_exchange_identification: Connection closed by remote host
Fri Aug :: Waiting all running threads are disconnected.. (max milliseconds)
{'Time' => '','db' => undef,'Id' => '','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'}
Fri Aug :: Waiting all running threads are disconnected.. (max milliseconds)
{'Time' => '','db' => undef,'Id' => '','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'}
Fri Aug :: Waiting all running threads are disconnected.. (max milliseconds)
{'Time' => '','db' => undef,'Id' => '','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'}
Fri Aug :: Set read_only= on the orig master.. ok.
Fri Aug :: Waiting all running queries are disconnected.. (max milliseconds)
{'Time' => '','db' => undef,'Id' => '','User' => 'repl','State' => 'Master has sent all binlog to slave; waiting for more updates','Command' => 'Binlog Dump GTID','Info' => undef,'Host' => 'zlm3:40535'}
Fri Aug :: Killing all application threads..
Fri Aug :: done.
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Locking all tables on the orig master to reject updates from everybody (including root):
Fri Aug :: - [info] Executing FLUSH TABLES WITH READ LOCK..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Orig master binlog:pos is mysql-bin.:.
Fri Aug :: - [info] Waiting to execute all relay logs on 192.168.1.102(192.168.1.102:)..
Fri Aug :: - [info] master_pos_wait(mysql-bin.:) completed on 192.168.1.102(192.168.1.102:). Executed events.
Fri Aug :: - [info] done.
Fri Aug :: - [info] Getting new master's binlog name and position..
Fri Aug :: - [info] mysql-bin.:
Fri Aug :: - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.102', MASTER_PORT=, MASTER_AUTO_POSITION=, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Fri Aug :: - [info] Executing master ip online change script to allow write on the new master:
Fri Aug :: - [info] /etc/masterha/master_ip_online_change --command=start --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --orig_master_user='zlm' --orig_master_password='zlmzlm' --new_master_host=192.168.1.102 --new_master_ip=192.168.1.102 --new_master_port= --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_user=root --new_master_ssh_user=root --orig_master_ssh_port= --new_master_ssh_port= --orig_master_is_new_slave
Unknown option: new_master_ssh_port
Fri Aug :: Set read_only= on the new master.
Fri Aug :: 328259Add vip 10.33.101.239 on p3p1..
ssh_exchange_identification: Connection closed by remote host
Fri Aug :: - [info] ok.
Fri Aug :: - [info]
Fri Aug :: - [info] * Switching slaves in parallel..
Fri Aug :: - [info]
Fri Aug :: - [info] Unlocking all tables on the orig master:
Fri Aug :: - [info] Executing UNLOCK TABLES..
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Starting orig master as a new slave..
Fri Aug :: - [info] Resetting slave 192.168.1.101(192.168.1.101:) and starting replication from the new master 192.168.1.102(192.168.1.102:)..
Fri Aug :: - [info] Executed CHANGE MASTER.
Fri Aug :: - [info] Slave started.
Fri Aug :: - [info] All new slave servers switched successfully.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : New master cleanup phase..
Fri Aug :: - [info]
Fri Aug :: - [info] 192.168.1.102: Resetting slave info succeeded.
Fri Aug :: - [info] Switching master to 192.168.1.102(192.168.1.102:) completed successfully. [root@zlm3 :: ~]
#
Check the master-slave replication status.
//New master(original slave)
(zlm@192.168.1.102 )[(none)]>show master status;
+------------------+----------+--------------+------------------+------------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+------------------------------------------------+
| mysql-bin. | | | | 1b7181ee-6eaf-11e8-998e-080027de0e0e:- |
+------------------+----------+--------------+------------------+------------------------------------------------+
row in set (0.00 sec) (zlm@192.168.1.102 )[(none)]>show slave status\G
Empty set (0.00 sec) //New slave(original master)
(zlm@192.168.1.101 )[(none)]>show master status;
+------------------+----------+--------------+------------------+------------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+------------------------------------------------+
| mysql-bin. | | | | 1b7181ee-6eaf-11e8-998e-080027de0e0e:- |
+------------------+----------+--------------+------------------+------------------------------------------------+
row in set (0.01 sec) (zlm@192.168.1.101 )[(none)]>show slave status\G
*************************** . row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.1.102
Master_User: repl
Master_Port:
Connect_Retry:
Master_Log_File: mysql-bin.
Read_Master_Log_Pos:
Relay_Log_File: relay-bin.
Relay_Log_Pos:
Relay_Master_Log_File: mysql-bin.
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno:
Last_Error:
Skip_Counter:
Exec_Master_Log_Pos:
Relay_Log_Space:
Until_Condition: None
Until_Log_File:
Until_Log_Pos:
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master:
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno:
Last_IO_Error:
Last_SQL_Errno:
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id:
Master_UUID: 842ea497--11e8-83ca-080027de0e0e
Master_Info_File: mysql.slave_master_info
SQL_Delay:
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count:
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 1b7181ee-6eaf-11e8-998e-080027de0e0e:-
Auto_Position:
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
row in set (0.00 sec)
Check the log of MasterHA on zlm3.
[root@zlm3 :: ~]
#cd /var/log/masterha/app1 [root@zlm3 :: /var/log/masterha/app1]
#cat app1.log
Fri Aug :: - [info] MHA::MasterMonitor version 0.56.
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [info] Alive Servers:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] Current Alive Master: 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Checking slave configurations..
Fri Aug :: - [info] read_only= is not set on slave 192.168.1.102(192.168.1.102:).
Fri Aug :: - [info] Checking replication filtering settings..
Fri Aug :: - [info] binlog_do_db= , binlog_ignore_db=
Fri Aug :: - [info] Replication filtering check ok.
Fri Aug :: - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Fri Aug :: - [info] Checking SSH publickey authentication settings on the current master..
Fri Aug :: - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
Fri Aug :: - [info]
192.168.1.101(192.168.1.101:) (current master)
+--192.168.1.102(192.168.1.102:) Fri Aug :: - [info] Checking master_ip_failover_script status:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --orig_master_ssh_port=
Fri Aug :: - [info] OK.
Fri Aug :: - [warning] shutdown_script is not defined.
Fri Aug :: - [info] Set master ping interval seconds.
Fri Aug :: - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Fri Aug :: - [info] Starting ping health check on 192.168.1.101(192.168.1.101:)..
Fri Aug :: - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Fri Aug :: - [info] Got terminate signal. Exit.
Fri Aug :: - [info] MHA::MasterMonitor version 0.56.
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [info] Alive Servers:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] Current Alive Master: 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Checking slave configurations..
Fri Aug :: - [info] read_only= is not set on slave 192.168.1.102(192.168.1.102:).
Fri Aug :: - [info] Checking replication filtering settings..
Fri Aug :: - [info] binlog_do_db= , binlog_ignore_db=
Fri Aug :: - [info] Replication filtering check ok.
Fri Aug :: - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Fri Aug :: - [info] Checking SSH publickey authentication settings on the current master..
Fri Aug :: - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
Fri Aug :: - [info]
192.168.1.101(192.168.1.101:) (current master)
+--192.168.1.102(192.168.1.102:) Fri Aug :: - [info] Checking master_ip_failover_script status:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --orig_master_ssh_port=
Fri Aug :: - [info] OK.
Fri Aug :: - [warning] shutdown_script is not defined.
Fri Aug :: - [info] Set master ping interval seconds.
Fri Aug :: - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Fri Aug :: - [info] Starting ping health check on 192.168.1.101(192.168.1.101:)..
Fri Aug :: - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Fri Aug :: - [info] Got terminate signal. Exit.
Test 2：Manual master failover
Execute "masterha_master_switch" script again to generate a failover on zlm2.
[root@zlm2 :: ~]
#masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --dead_master_host=192.168.1.102 --master_state=dead --new_master_host=192.168.1.101 --ignore_last_failover
--dead_master_ip=<dead_master_ip> is not set. Using 192.168.1.102.
--dead_master_port=<dead_master_port> is not set. Using .
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] MHA::MasterFailover version 0.56.
Fri Aug :: - [info] Starting master failover.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Configuration Check Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/MasterFailover.pm, ln187] None of server is dead. Stop failover.
Fri Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/ManagerUtil.pm, ln177] Got ERROR: at /usr/bin/masterha_master_switch line . //Stop mysqld of master on zlm3.
[root@zlm3 :: ~]
#mysqladmin shutdown [root@zlm3 :: ~]
#ps aux|grep mysqld
mysql 0.0 19.6 pts/ Sl : : mysqld --defaults-file=/data/mysql/mysql3306/my.cnf
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld [root@zlm3 :: ~]
#ps aux|grep mysqld
mysql 0.0 19.6 pts/ Sl : : mysqld --defaults-file=/data/mysql/mysql3306/my.cnf
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld [root@zlm3 :: ~]
#ps aux|grep mysqld
mysql 0.0 19.1 pts/ Sl : : mysqld --defaults-file=/data/mysql/mysql3306/my.cnf
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld [root@zlm3 :: ~]
#ps aux|grep mysqld
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld //Execute the above command again on zlm2.
[root@zlm2 :: ~]
#masterha_master_switch --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf --dead_master_host=192.168.1.102 --master_state=dead --new_master_host=192.168.1.101 --ignore_last_failover
--dead_master_ip=<dead_master_ip> is not set. Using 192.168.1.102.
--dead_master_port=<dead_master_port> is not set. Using .
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] MHA::MasterFailover version 0.56.
Fri Aug :: - [info] Starting master failover.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Configuration Check Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Checking master reachability via MySQL(double check)...
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Alive Servers:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Master 192.168.1.102(192.168.1.102:) is dead. Proceed? (yes/NO): yes
Fri Aug :: - [info] Starting GTID based failover.
Fri Aug :: - [info]
Fri Aug :: - [info] ** Phase : Configuration Check Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Dead Master Shutdown Phase..
Fri Aug :: - [info]
ssh: connect to host 192.168.1.102 port : Connection refused
Fri Aug :: - [warning] HealthCheck: SSH to 192.168.1.102 is NOT reachable.
Fri Aug :: - [info] Forcing shutdown so that applications never connect to the current master..
Fri Aug :: - [info] Executing master IP deactivation script:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --orig_master_host=192.168.1.102 --orig_master_ip=192.168.1.102 --orig_master_port= --command=stop --orig_master_ssh_port=
ssh: connect to host 192.168.1.102 port : Connection refused
Fri Aug :: - [info] done.
Fri Aug :: - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Fri Aug :: - [info] * Phase : Dead Master Shutdown Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Master Recovery Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 3.1: Getting Latest Slaves Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] The latest binary log file/position on all slaves is mysql-bin.:
Fri Aug :: - [info] Latest slaves (Slaves that received relay log files to the latest):
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] The oldest binary log file/position on all slaves is mysql-bin.:
Fri Aug :: - [info] Oldest slaves:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 3.3: Determining New Master Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] 192.168.1.101 can be new master.
Fri Aug :: - [info] New master is 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Starting master failover..
Fri Aug :: - [info]
From:
192.168.1.102(192.168.1.102:) (current master)
+--192.168.1.101(192.168.1.101:) To:
192.168.1.101(192.168.1.101:) (new master) Starting master switch from 192.168.1.102(192.168.1.102:) to 192.168.1.101(192.168.1.101:)? (yes/NO): yes
Fri Aug :: - [info] New master decided manually is 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 3.3: New Master Recovery Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Waiting all logs to be applied..
Fri Aug :: - [info] done.
Fri Aug :: - [info] Getting new master's binlog name and position..
Fri Aug :: - [info] mysql-bin.:
Fri Aug :: - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.101', MASTER_PORT=, MASTER_AUTO_POSITION=, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Fri Aug :: - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin., , 1b7181ee-6eaf-11e8-998e-080027de0e0e:-
Fri Aug :: - [info] Executing master IP activate script:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.1.102 --orig_master_ip=192.168.1.102 --orig_master_port= --new_master_host=192.168.1.101 --new_master_ip=192.168.1.101 --new_master_port= --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_port= --new_master_ssh_port=
Unknown option: new_master_ssh_port
Set read_only= on the new master.
ssh_exchange_identification: Connection closed by remote host
Fri Aug :: - [info] OK.
Fri Aug :: - [info] ** Finished master recovery successfully.
Fri Aug :: - [info] * Phase : Master Recovery Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Slaves Recovery Phase..
Fri Aug :: - [info]
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 4.1: Starting Slaves in parallel..
Fri Aug :: - [info]
Fri Aug :: - [info] All new slave servers recovered successfully.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : New master cleanup phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Resetting slave info on the new master..
Fri Aug :: - [info] 192.168.1.101: Resetting slave info succeeded.
Fri Aug :: - [info] Master failover to 192.168.1.101(192.168.1.101:) completed successfully.
Fri Aug :: - [info] ----- Failover Report ----- app1: MySQL Master failover 192.168.1.102(192.168.1.102:) to 192.168.1.101(192.168.1.101:) succeeded Master 192.168.1.102(192.168.1.102:) is down! Check MHA Manager logs at zlm2 for details. Started manual(interactive) failover.
Invalidated master IP address on 192.168.1.102(192.168.1.102:)
Selected 192.168.1.101(192.168.1.101:) as a new master.
192.168.1.101(192.168.1.101:): OK: Applying all logs succeeded.
192.168.1.101(192.168.1.101:): OK: Activated master IP address.
192.168.1.101(192.168.1.101:): Resetting slave info succeeded.
Master failover to 192.168.1.101(192.168.1.101:) completed successfully.
Check the status of new master on zlm2.
(zlm@192.168.1.101 )[(none)]>show master status;
+------------------+----------+--------------+------------------+------------------------------------------------+
| File | Position | Binlog_Do_DB | Binlog_Ignore_DB | Executed_Gtid_Set |
+------------------+----------+--------------+------------------+------------------------------------------------+
| mysql-bin. | | | | 1b7181ee-6eaf-11e8-998e-080027de0e0e:- |
+------------------+----------+--------------+------------------+------------------------------------------------+
row in set (0.00 sec) (zlm@192.168.1.101 )[(none)]>show slave status\G
Empty set (0.00 sec)
Check the file and log on MasterHA.
[root@zlm2 :: ~]
#cd /var/log/masterha/app1 [root@zlm2 :: /var/log/masterha/app1]
#ls -l
total
-rw-r--r-- root root Aug : app1.failover.complete
-rw-r--r-- root root Aug : app1.log //The option of "--ignore_last_failover" can neglect the influence of existence of "app1.failover.complete".Otherwise,the failover operation will be terminated by error.
//This file will be created after a failover operation and it will be created only on the original slave who wants to become a new master. [root@zlm2 :: /var/log/masterha/app1]
#cat app1.log
Thu Aug :: - [info] MHA::MasterMonitor version 0.56.
Thu Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln301] Got MySQL error when connecting 192.168.1.101(192.168.1.101:) ::Access denied for user 'root'@'zlm2' (using password: NO), but this is not a MySQL crash. Check MySQL server settings.
at /usr/share/perl5/vendor_perl/MHA/ServerManager.pm line .
Thu Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln301] Got MySQL error when connecting 192.168.1.102(192.168.1.102:) ::Access denied for user 'root'@'zlm2' (using password: NO), but this is not a MySQL crash. Check MySQL server settings.
at /usr/share/perl5/vendor_perl/MHA/ServerManager.pm line .
Thu Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/ServerManager.pm, ln309] Got fatal error, stopping operations
Thu Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln424] Error happened on checking configurations. at /usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm line .
Thu Aug :: - [error][/usr/share/perl5/vendor_perl/MHA/MasterMonitor.pm, ln523] Error happened on monitoring servers.
Thu Aug :: - [info] Got exit code (Not master dead).
Thu Aug :: - [info] MHA::MasterMonitor version 0.56.
Thu Aug :: - [info] GTID failover mode =
Thu Aug :: - [info] Dead Servers:
Thu Aug :: - [info] Alive Servers:
Thu Aug :: - [info] 192.168.1.101(192.168.1.101:)
Thu Aug :: - [info] 192.168.1.102(192.168.1.102:)
Thu Aug :: - [info] Alive Slaves:
Thu Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Thu Aug :: - [info] GTID ON
Thu Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Thu Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Thu Aug :: - [info] Current Alive Master: 192.168.1.101(192.168.1.101:)
Thu Aug :: - [info] Checking slave configurations..
Thu Aug :: - [info] read_only= is not set on slave 192.168.1.102(192.168.1.102:).
Thu Aug :: - [info] Checking replication filtering settings..
Thu Aug :: - [info] binlog_do_db= , binlog_ignore_db=
Thu Aug :: - [info] Replication filtering check ok.
Thu Aug :: - [info] GTID (with auto-pos) is supported. Skipping all SSH and Node package checking.
Thu Aug :: - [info] Checking SSH publickey authentication settings on the current master..
Thu Aug :: - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
Thu Aug :: - [info]
192.168.1.101(192.168.1.101:) (current master)
+--192.168.1.102(192.168.1.102:) Thu Aug :: - [info] Checking master_ip_failover_script status:
Thu Aug :: - [info] /etc/masterha/master_ip_failover --command=status --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --orig_master_ssh_port=
Thu Aug :: - [info] OK.
Thu Aug :: - [warning] shutdown_script is not defined.
Thu Aug :: - [info] Set master ping interval seconds.
Thu Aug :: - [warning] secondary_check_script is not defined. It is highly recommended setting it to check master reachability from two or more routes.
Thu Aug :: - [info] Starting ping health check on 192.168.1.101(192.168.1.101:)..
Thu Aug :: - [info] Ping(SELECT) succeeded, waiting until MySQL doesn't respond..
Thu Aug :: - [info] Got terminate signal. Exit.
Test3: Automation of master switchover.
Repair the salve replication on zlm3.
[root@zlm3 :: ~]
#mysql
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is
Server version: 5.7.-log MySQL Community Server (GPL) Copyright (c) , , Oracle and/or its affiliates. All rights reserved. Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners. Type 'help;' or '\h' for help. Type '\c' to clear the current input statement. (zlm@192.168.1.102 )[(none)]>change master to \
-> master_host='192.168.1.101',\
-> master_port=,\
-> master_user='repl',\
-> master_password='repl4slave',\
-> master_auto_position=;
Query OK, rows affected, warnings (0.02 sec) (zlm@192.168.1.102 )[(none)]>show slave status\G
*************************** . row ***************************
Slave_IO_State: Waiting for master to send event
Master_Host: 192.168.1.101
Master_User: repl
Master_Port:
Connect_Retry:
Master_Log_File: mysql-bin.
Read_Master_Log_Pos:
Relay_Log_File: relay-bin.
Relay_Log_Pos:
Relay_Master_Log_File: mysql-bin.
Slave_IO_Running: Yes
Slave_SQL_Running: Yes
Replicate_Do_DB:
Replicate_Ignore_DB:
Replicate_Do_Table:
Replicate_Ignore_Table:
Replicate_Wild_Do_Table:
Replicate_Wild_Ignore_Table:
Last_Errno:
Last_Error:
Skip_Counter:
Exec_Master_Log_Pos:
Relay_Log_Space:
Until_Condition: None
Until_Log_File:
Until_Log_Pos:
Master_SSL_Allowed: No
Master_SSL_CA_File:
Master_SSL_CA_Path:
Master_SSL_Cert:
Master_SSL_Cipher:
Master_SSL_Key:
Seconds_Behind_Master:
Master_SSL_Verify_Server_Cert: No
Last_IO_Errno:
Last_IO_Error:
Last_SQL_Errno:
Last_SQL_Error:
Replicate_Ignore_Server_Ids:
Master_Server_Id:
Master_UUID: 1b7181ee-6eaf-11e8-998e-080027de0e0e
Master_Info_File: mysql.slave_master_info
SQL_Delay:
SQL_Remaining_Delay: NULL
Slave_SQL_Running_State: Slave has read all relay log; waiting for more updates
Master_Retry_Count:
Master_Bind:
Last_IO_Error_Timestamp:
Last_SQL_Error_Timestamp:
Master_SSL_Crl:
Master_SSL_Crlpath:
Retrieved_Gtid_Set:
Executed_Gtid_Set: 1b7181ee-6eaf-11e8-998e-080027de0e0e:-
Auto_Position:
Replicate_Rewrite_DB:
Channel_Name:
Master_TLS_Version:
row in set (0.00 sec)
Start MasterHA-manager.
[root@zlm3 :: /var/log/masterha/app1]
#nohup masterha_manager --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf &
[]
nohup: ignoring input and appending output to ‘nohup.out’ [root@zlm3 :: /var/log/masterha/app1]
#ls -l
total
-rw-r--r-- root root Aug : app1.log
-rw-r--r-- root root Aug : app1.master_status.health //This file is created only when MasterHA-manager is running.It will continuously record the health status between slave and master.
-rw------- root root Aug : nohup.out [root@zlm3 :: /var/log/masterha/app1]
#cat nohup.out
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
ssh_exchange_identification: Connection closed by remote host [root@zlm3 :: /var/log/masterha/app1]
#cat app1.master_status.health
:PING_OK master:192.168.1.101
[root@zlm3 :: /var/log/masterha/app1]
#ps aux|grep manager
root 0.6 2.1 pts/ S : : perl /usr/bin/masterha_manager --conf=/etc/masterha/app1.conf --global_conf=/etc/masterha/masterha_default.conf
root 0.0 0.0 pts/ R+ : : grep --color=auto manager
Kill mysqld on zlm2 to pretend the master is dead.
[root@zlm2 :: /var/log/masterha/app1]
#pkill mysqld [root@zlm2 :: /var/log/masterha/app1]
#ps aux|grep mysqld
root 0.0 0.0 pts/ R+ : : grep --color=auto mysqld
Observe the app1.log on zlm3.
[root@zlm3 :: /var/log/masterha/app1]
#echo ''> app1.log [root@zlm3 :: /var/log/masterha/app1]
#tail -f app1.log Fri Aug :: - [warning] Got error on MySQL select ping: (MySQL server has gone away)
Fri Aug :: - [info] Executing SSH check script: exit
Fri Aug :: - [warning] HealthCheck: SSH to 192.168.1.101 is NOT reachable.
Fri Aug :: - [warning] Got error on MySQL connect: (Can't connect to MySQL server on '192.168.1.101' (111))
Fri Aug :: - [warning] Connection failed time(s)..
Fri Aug :: - [warning] Got error on MySQL connect: (Can't connect to MySQL server on '192.168.1.101' (111))
Fri Aug :: - [warning] Connection failed time(s)..
Fri Aug :: - [warning] Got error on MySQL connect: (Can't connect to MySQL server on '192.168.1.101' (111))
Fri Aug :: - [warning] Connection failed time(s)..
Fri Aug :: - [warning] Master is not reachable from health checker!
Fri Aug :: - [warning] Master 192.168.1.101(192.168.1.101:) is not reachable!
Fri Aug :: - [warning] SSH is NOT reachable.
Fri Aug :: - [info] Connecting to a master server failed. Reading configuration file /etc/masterha/masterha_default.conf and /etc/masterha/app1.conf again, and trying to connect to all servers to check server status..
Fri Aug :: - [info] Reading default configuration from /etc/masterha/masterha_default.conf..
Fri Aug :: - [info] Reading application default configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] Reading server configuration from /etc/masterha/app1.conf..
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Alive Servers:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] Checking slave configurations..
Fri Aug :: - [info] read_only= is not set on slave 192.168.1.102(192.168.1.102:).
Fri Aug :: - [info] Checking replication filtering settings..
Fri Aug :: - [info] Replication filtering check ok.
Fri Aug :: - [info] Master is down!
Fri Aug :: - [info] Terminating monitoring script.
Fri Aug :: - [info] Got exit code (Master dead).
Fri Aug :: - [info] MHA::MasterFailover version 0.56.
Fri Aug :: - [info] Starting master failover.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Configuration Check Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] GTID failover mode =
Fri Aug :: - [info] Dead Servers:
Fri Aug :: - [info] 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Checking master reachability via MySQL(double check)...
Fri Aug :: - [info] ok.
Fri Aug :: - [info] Alive Servers:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Alive Slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] Starting GTID based failover.
Fri Aug :: - [info]
Fri Aug :: - [info] ** Phase : Configuration Check Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Dead Master Shutdown Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Forcing shutdown so that applications never connect to the current master..
Fri Aug :: - [info] Executing master IP deactivation script:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --command=stop --orig_master_ssh_port=
ssh: connect to host 192.168.1.101 port : Connection refused
Fri Aug :: - [info] done.
Fri Aug :: - [warning] shutdown_script is not set. Skipping explicit shutting down of the dead master.
Fri Aug :: - [info] * Phase : Dead Master Shutdown Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Master Recovery Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 3.1: Getting Latest Slaves Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] The latest binary log file/position on all slaves is mysql-bin.:
Fri Aug :: - [info] Latest slaves (Slaves that received relay log files to the latest):
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] The oldest binary log file/position on all slaves is mysql-bin.:
Fri Aug :: - [info] Oldest slaves:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 3.3: Determining New Master Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Searching new master from slaves..
Fri Aug :: - [info] Candidate masters from the configuration file:
Fri Aug :: - [info] 192.168.1.102(192.168.1.102:) Version=5.7.-log (oldest major version between slaves) log-bin:enabled
Fri Aug :: - [info] GTID ON
Fri Aug :: - [info] Replicating from 192.168.1.101(192.168.1.101:)
Fri Aug :: - [info] Primary candidate for the new Master (candidate_master is set)
Fri Aug :: - [info] Non-candidate masters:
Fri Aug :: - [info] Searching from candidate_master slaves which have received the latest relay log events..
Fri Aug :: - [info] New master is 192.168.1.102(192.168.1.102:)
Fri Aug :: - [info] Starting master failover..
Fri Aug :: - [info]
From:
192.168.1.101(192.168.1.101:) (current master)
+--192.168.1.102(192.168.1.102:) To:
192.168.1.102(192.168.1.102:) (new master)
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 3.3: New Master Recovery Phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Waiting all logs to be applied..
Fri Aug :: - [info] done.
Fri Aug :: - [info] Getting new master's binlog name and position..
Fri Aug :: - [info] mysql-bin.:
Fri Aug :: - [info] All other slaves should start replication from here. Statement should be: CHANGE MASTER TO MASTER_HOST='192.168.1.102', MASTER_PORT=, MASTER_AUTO_POSITION=, MASTER_USER='repl', MASTER_PASSWORD='xxx';
Fri Aug :: - [info] Master Recovery succeeded. File:Pos:Exec_Gtid_Set: mysql-bin., , 1b7181ee-6eaf-11e8-998e-080027de0e0e:-
Fri Aug :: - [info] Executing master IP activate script:
Fri Aug :: - [info] /etc/masterha/master_ip_failover --command=start --ssh_user=root --orig_master_host=192.168.1.101 --orig_master_ip=192.168.1.101 --orig_master_port= --new_master_host=192.168.1.102 --new_master_ip=192.168.1.102 --new_master_port= --new_master_user='zlm' --new_master_password='zlmzlm' --orig_master_ssh_port= --new_master_ssh_port=
Unknown option: new_master_ssh_port
Set read_only= on the new master.
ssh_exchange_identification: Connection closed by remote host
Fri Aug :: - [info] OK.
Fri Aug :: - [info] ** Finished master recovery successfully.
Fri Aug :: - [info] * Phase : Master Recovery Phase completed.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : Slaves Recovery Phase..
Fri Aug :: - [info]
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase 4.1: Starting Slaves in parallel..
Fri Aug :: - [info]
Fri Aug :: - [info] All new slave servers recovered successfully.
Fri Aug :: - [info]
Fri Aug :: - [info] * Phase : New master cleanup phase..
Fri Aug :: - [info]
Fri Aug :: - [info] Resetting slave info on the new master..
Fri Aug :: - [info] 192.168.1.102: Resetting slave info succeeded.
Fri Aug :: - [info] Master failover to 192.168.1.102(192.168.1.102:) completed successfully.
Fri Aug :: - [info] ----- Failover Report ----- app1: MySQL Master failover 192.168.1.101(192.168.1.101:) to 192.168.1.102(192.168.1.102:) succeeded Master 192.168.1.101(192.168.1.101:) is down! Check MHA Manager logs at zlm3:/var/log/masterha/app1/app1.log for details. Started automated(non-interactive) failover.
Invalidated master IP address on 192.168.1.101(192.168.1.101:)
Selected 192.168.1.102(192.168.1.102:) as a new master.
192.168.1.102(192.168.1.102:): OK: Applying all logs succeeded.
192.168.1.102(192.168.1.102:): OK: Activated master IP address.
192.168.1.102(192.168.1.102:): Resetting slave info succeeded.
Master failover to 192.168.1.102(192.168.1.102:) completed successfully. //Above failover report shows all the evidence and results of automation master switchover.All of the steps are executed successfully.
MySQL高可用之MHA切换测试（switchover & failover）的更多相关文章
- MySQL高可用方案MHA在线切换的步骤及原理
在日常工作中,会碰到如下的场景,如mysql数据库升级,主服务器硬件升级等,这个时候就需要将写操作切换到另外一台服务器上,那么如何进行在线切换呢?同时,要求切换过程短,对业务的影响比较小. MHA就提 ...
- MySQL高可用方案MHA自动Failover与手动Failover的实践及原理
集群信息 角色 IP地址 ServerID 类型 Master ...
- MySQL高可用方案--MHA部署及故障转移
架构设计及必要配置 主机环境 IP 主机名 担任角色 192.168.192.128 node_master MySQL-Master| ...
- 【DB宝42】MySQL高可用架构MHA+ProxySQL实现读写分离和负载均衡
目录 一.MHA+ProxySQL架构 二.快速搭建MHA环境 2.1 下载MHA镜像 2.2 编辑yml文件,创建MHA相关容器 2.3 安装docker-compose软件(若已安装,可忽略) 2 ...
- mysql高可用方案MHA介绍
mysql高可用方案MHA介绍 概述 MHA是一位日本MySQL大牛用Perl写的一套MySQL故障切换方案,来保证数据库系统的高可用.在宕机的时间内(通常10-30秒内),完成故障切换,部署MHA, ...
- MySQL高可用之MHA (转)
MySQL高可用之MHA MHA简介 MHA是由日本人yoshinorim(原就职于DeNA现就职于FaceBook)开发的比较成熟的MySQL高可用方案.MHA能够在30秒内实现故障切换,并能在故障 ...
- 【DB宝19】在Docker中使用MySQL高可用之MHA
目录 一.MHA简介和架构 1.1 MHA简介 1.2 MHA工具包的组成 1.3 MHA架构 二.准备MHA环境 2.1 下载MHA镜像 2.2 编辑yml文件,创建MHA相关容器 2.3 安装do ...
- MySQL高可用之MHA的搭建 转
http://www.cnblogs.com/muhu/p/4045780.html http://www.cnblogs.com/gomysql/p/3675429.html http://www ...
- MySQL高可用之MHA的搭建
MySQL MHA架构介绍: MHA(Master High Availability)目前在MySQL高可用方面是一个相对成熟的解决方案,它由日本DeNA公司youshimaton(现就职于Face ...
随机推荐
- 【转】应用程序的入口是ActivityThread
ActivityThread运行框架 在分析中,我们可以看到真正对应应用进程的不是Application而是ActivityThread.我们从实际的应用堆栈可以看到: NavitiveStart.m ...
- FreeImage.lib库的配置和简单使用 转
转载地址 http://www.codeweblog.com/win8-1%E4%B8%8Bfreeimage-lib%E5%BA%93%E7%9A%84%E9%85%8D%E7%BD%AE%E5%9 ...
- 消息中间件JMS(二)
之前介绍了ActiveMQ下载与安装,并且启动了.下面进行ActiveMQ的Demo 1. JMS入门Demo 1.1 点对点模式 点对点模式主要建立在一个队列上面,当连接一个队列的时候,发送端不需要 ...
- 02_Linux 终端命令格式
01. 终端命令格式 command [-options] [parameter] 说明: command:命令名,相应功能的英文单词或单词的缩写 [-options]:选项,可用来对命令进行控制,也 ...
- Autofac QuickStart
1 构建应用程序 示例: 我们期望有一个输出工具类,当前希望通过控制台(console)输出,但是又希望仅能在控制台模式下输出.所以我们把输出抽象为一个接口 using System; namespa ...
- asp.net mvc Post上传文件大小限制 （转载）
最近发现在项目中使用jQuery.form插件上传比较大的文件时,上传不了,于是改了下web.config的上传文件最大限制. <configuration> <system.web ...
- JWT如何在Spring Cloud微服务系统中在服务相互调时传递
转载请标明出处: http://blog.csdn.net/forezp/article/details/78676036 本文出自方志朋的博客 在微服务系统中,为了保证微服务系统的安全,常常使用jw ...
- iOS Alamofire插件使用方法
let parameters = [ " ] Alamofire.request("http://110.185.104.100:8888/skproject/HvLogistic ...
- A^B Mod C （51Nod - 1046 ）（快速幂）
给出3个正整数A B C,求A^B Mod C. 例如,3 5 8,3^5 Mod 8 = 3. Input 3个正整数A B C,中间用空格分隔.(1 <= A,B,C <= 10^ ...
- SpringBoot配置全局自定义异常
不同于传统集中时Springmvc 全局异常,具体查看前面的章节https://www.cnblogs.com/zwdx/p/8963311.html 对于springboot框架来讲,这里我就介绍一 ...