mha && 脚本拉起

ip分布：

 vip 192.168.238.222

mha_manager	192.168.238.131	监控服务器
master	192.168.238.128	主库GTID复制模式
slave—1	192.168.238.129	从库，备用主库
slave-2	192.168.238.130	从库
mha版本	0.56

 

 二进制mysql，mysql和mysqlbinlog命令软连接到/usr/lcoal/bin

一主两从模式，采用GTID模式复制（不详述）

master主要配置如下：

master_info_repository = TABLE

relay_log_info_repository = TABLE

gtid_mode = on

enforce_gtid_consistency = 1

log_slave_updates

binlog_format = row

server-id = 3306

log-bin = /home/mysql/log/binlog/mybinlog

sync-master-info=1

slave-parallel-workers=2

binlog-checksum=CRC32

master-verify-checksum=1

slave-sql-verify-checksum=1

binlog-rows-query-log_events=1

slave 主要配置如下 ，已slave-1为例，slave2注意修改server-id

master_info_repository = TABLE

relay_log_info_repository = TABLE

gtid_mode = on

enforce_gtid_consistency = 1

log_slave_updates

binlog_format = row

server-id = 3307

log-bin = /home/mysql/log/binlog/mybinlog

sync-master-info=1

slave-parallel-workers=2

binlog-checksum=CRC32

master-verify-checksum=1

slave-sql-verify-checksum=1

binlog-rows-query-log_events=1

#relay log

relay-log = /home/mysql/log/relaylog/relay-bin

relay-log-index = /home/mysql/log/relaylog/relay-bin.index

relay_log_recovery = 1

relay-log-purge = 1

          

然后再slave上进行change master ：

CHANGE MASTER TO MASTER_HOST='192.168.238.128', MASTER_PORT=3306, MASTER_AUTO_POSITION=1, MASTER_USER='repl', MASTER_PASSWORD='repl123';

     

动态设置read_only=1，原因：切换的时候提升为主的时候可以动态关闭不需要去重启mysql。

 

主从搭建完成之后，在master授权root可以允许manager连入。或者创建一个专门的mha账号也可以

====================

 

安装前，四个节点全部安装上依赖关系

rpm -ivh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

yum install perl-* -y

 

四个节点全部编译安装node

tar xf mha4mysql-node-0.56.tar.gz
cd mha4mysql-node-0.56
perl Makefile.PL

make && make install

 

单单在manager节点上安装manager

tar xf mha4mysql-manager-0.56.tar.gz 
cd mha4mysql-manager-0.56
perl Makefile.PL
make && make install

 

mha_node安装完成之后会有如下工具包

# ll /usr/loca/bin

-r-xr-xr-x 1 root root 15498 Apr 20 10:05 apply_diff_relay_logs  -- 识别差异的中继日志事件并将其差异的事件应用于其他的slave

-r-xr-xr-x 1 root root  4807 Apr 20 10:05 filter_mysqlbinlog -- 过滤mysqlbinlog中的rollback事件（已经不使用）

-r-xr-xr-x 1 root root  7401 Apr 20 10:05 purge_relay_logs -- 清除中继日志（不会阻塞SQL线程）

-r-xr-xr-x 1 root root  7263 Apr 20 10:05 save_binary_logs -- 保存和读取master的binlog

 

 

mha_manager安装完成之后会有如下工具包

# ll /usr/loca/bin

-r-xr-xr-x 1 root root 15498 Apr 20 10:58 apply_diff_relay_logs

-r-xr-xr-x 1 root root  4807 Apr 20 10:58 filter_mysqlbinlog

-r-xr-xr-x 1 root root  1995 Apr 20 11:33 masterha_check_repl  -- 检查复制情况

-r-xr-xr-x 1 root root  1779 Apr 20 11:33 masterha_check_ssh  -- 检查ssh联通性
-r-xr-xr-x 1 root root  1865 Apr 20 11:33 masterha_check_status -- 检查manager状态
-r-xr-xr-x 1 root root  3201 Apr 20 11:33 masterha_conf_host -- 添加或者删除配置的server信息
-r-xr-xr-x 1 root root  2517 Apr 20 11:33 masterha_manager -- 启动mha
-r-xr-xr-x 1 root root  2165 Apr 20 11:33 masterha_master_monitor -- 检测master是否宕机
-r-xr-xr-x 1 root root  2373 Apr 20 11:33 masterha_master_switch -- 控制故障转移切换
-r-xr-xr-x 1 root root  3749 Apr 20 11:33 masterha_secondary_check
-r-xr-xr-x 1 root root  1739 Apr 20 11:33 masterha_stop
-r-xr-xr-x 1 root root  7401 Apr 20 10:58 purge_relay_logs
-r-xr-xr-x 1 root root  7263 Apr 20 10:58 save_binary_logs

 

 

在manager端配置mha的配置文件cat /mha/app1.cnf

#########################

[server default]

manager_workdir=/mha

manager_log=/mha/manager.log

master_binlog_dir=/home/mysql/log/binlog

password=1234567

user=root

ping_interval=1

remote_workdir=/mha

repl_password=repl123

repl_user=repl

ssh_user=root

client_bindir=/home/mysql/app/bin

master_ip_failover_script=/mha/master_ip_failover

[server1]

hostname=192.168.238.128

port=3306

[server2]

hostname=192.168.238.129

port=3306

master_binlog_dir=/home/mysql/log/binlog

candidate_master=1

[server3]

hostname=192.168.238.130

port=3306

master_binlog_dir=/home/mysql/log/binlog

 

在相关路径添加脚本master_ip_failover见附件failover.pl

该脚本功能，进行故障转移，并且绑定vip

 

在manager检测mha_manager配置情况以及集群的存活情况

masterha_check_ssh --conf=/mha/app1.cnf

masterha_check_repl --conf=/mha/app1.cnf

masterha_check_status --conf=/mha/app1.cnf

 

启动mha如下

nohup masterha_manager --conf=/mha/app1.cnf --ignore_last_failover < /dev/null >/mha/manager.log 2>&1 &

 

故障切换流程

停掉主库的mysql

 

查看manager.log

当出现master failover  to ip_addre completed successfully，则说明故障转移成功

 

 

说明，mha_manager完成故障转移之后会退出，属于正常情况，如果需要持续不断的运行，就是用daemontool

 

后续思考： 

故障切换完成之后可以发送邮件到邮件，根据manager是否退出来进行判断

 

故障切换后，拉起原主库作为备库&再启动mha_manager脚本见附件do_mha.sh

 

脚本运行目录/mha

[root@mon mha] # sh do_mha.sh app1

 

该脚本运行原理是，判断manager进程是否存在。如果存在就不执行脚本，因为没有发生过故障切换

如果manager不存在就运行脚本，说明发生过故障切换

（排除人工杀掉manager进程的情况，只基于正常的故障转移）

 

 

 附件

failover.pl

#!/usr/bin/env perl
use strict;
use warnings FATAL => 'all';

use Getopt::Long;

use Net::Ping;
use Switch;

my ($command, $ssh_user, $orig_master_host, $orig_master_ip, $orig_master_port, $new_master_host, $new_master_ip, $new_master_port, $new_master_user, $new_master_password);
GetOptions(
'command=s' => \$command,
'ssh_user=s' => \$ssh_user,
'orig_master_host=s' => \$orig_master_host,
'orig_master_ip=s' => \$orig_master_ip,
'orig_master_port=i' => \$orig_master_port,
'new_master_host=s' => \$new_master_host,
'new_master_ip=s' => \$new_master_ip,
'new_master_port=i' => \$new_master_port,
'new_master_user=s' => \$new_master_user,
'new_master_password=s' => \$new_master_password,
);
my $vip = '192.168.238.222'; # Virtual IP
my $master_srv = '192.168.238.129';
my $timeout = 5;
my $key = "1";
my $gateway = '192.168.238.2';
my $interface = 'eth0';
my $ssh_start_vip = "/sbin/ifconfig $interface:$key $vip;/sbin/arping -I $interface -c 3 -s $vip $gateway >/dev/null 2>&1";
my $ssh_stop_vip = "/sbin/ifconfig $interface:$key down";

exit &main();

sub main {

#print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
print "\n\nIN SCRIPT TEST====$ssh_stop_vip==$ssh_start_vip===\n\n";
if ( $command eq "stop" || $command eq "stopssh" ) {

# $orig_master_host, $orig_master_ip, $orig_master_port are passed.
# If you manage master ip address at global catalog database,
# invalidate orig_master_ip here.
my $exit_code = 1;
eval {
print "Disabling the VIP on old master if the server is still UP: $orig_master_host \n";
my $p=Net::Ping->new('icmp');
&stop_vip() if $p->ping($master_srv, $timeout);
$p->close();
$exit_code = 0;
};
if ($@) {
warn "Got Error: $@\n";
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "start" ) {

# all arguments are passed.
# If you manage master ip address at global catalog database,
# activate new_master_ip here.
# You can also grant write access (create user, set read_only=0, etc) here.
my $exit_code = 10;
eval {
print "Enabling the VIP - $vip on the new master - $new_master_host \n";
&start_vip();
$exit_code = 0;
};
if ($@) {
warn $@;
exit $exit_code;
}
exit $exit_code;
}
elsif ( $command eq "status" ) {
print "Checking the Status of the script.. OK \n";
#`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
`ssh $ssh_user\@$orig_master_host \" $ssh_start_vip \"`;
exit 0;
}
else {
&usage();
exit 1;
}
}

# A simple system call that enable the VIP on the new master
sub start_vip() {
`ssh $ssh_user\@$new_master_host \" $ssh_start_vip \"`;
}
# A simple system call that disable the VIP on the old_master
sub stop_vip() {
`ssh $ssh_user\@$orig_master_host \" $ssh_stop_vip \"`;
}

sub usage {
print
"Usage: master_ip_failover --command=start|stop|stopssh|status --orig_master_host=host --orig_master_ip=ip --orig_master_port=port --new_master_host=host --new_master_ip=ip --new_master_port=port\n";
}

mha.sh

#!/bin/bash
# Script
# Name: do_mha.sh
# APP vendor: MHA
# Linux vendor: RHEL / CentOS

# --------------------------
# variable and file path

# Part:: this script
str_mha_application=$1

# Part:: Linux
# Account info
str_linux_username="root"

# Part:: MySQL
# Account info
str_mysql_username="root"
str_mysql_password="123456"

str_repl_username="repl"
str_repl_password="repl123"

# Part:: MHA
# pid
str_pid_masterha_manager=`ps -ef | grep masterha_manager | grep "$str_mha_application" | grep perl | awk '{print $2}'`

# file
file_conf_mha_application="/mha/$str_mha_application.cnf"

# file: relation / by computed
file_log_mha_manager=`cat $file_conf_mha_application | grep --color manager_log | cut -d'=' -f2`

# variable: ip info
# 如果MHA中MySQL主库的候选服务器数量超过了两台，也许下面这个list参数，就会排上用场
list_ip_candicate=`cat $file_conf_mha_application | grep -B 3 "^candidate" | grep "hostname" | cut -d'=' -f2`

str_ip_orig_master=`cat $file_log_mha_manager | grep --color "MySQL Master failover" | cut -d'(' -f2 | cut -d':' -f1 | tail -n 1`
str_ip_new_master=`cat $file_log_mha_manager | grep --color "MySQL Master failover" | cut -d'(' -f3 | cut -d':' -f1 | tail -n 1`

str_ip_mha_manager="192.168.238.131"

# 为[change master]准备的参数
str_log_file_new_master=""
str_log_pos_new_master=""

# Part:: String SQL
str_sql_mysql_change_master=""

# --------------------------
# function

function do_sql() {
# variable
func_str_ip="$1"
func_str_sql="$2"

# action
# 本场景中不涉及到对MySQL某个库的操作，所以没有选择[db]
mysql -u $str_mysql_username -h $func_str_ip -p"$str_mysql_password" -N -e "$func_str_sql" -P3306
}

# 获取主库状态信息 非GTID复制
#function get_info_mysql_master_new_master() {
# version ONE
#str_log_file_new_master=`do_sql "$str_ip_new_master" "show master status" | awk '{print $1}'`
#str_log_pos_new_master=`do_sql "$str_ip_new_master" "show master status" | awk '{print $2}'`
#}

# 生成orig_master作为slave加入new_master的[change master]SQL命令
function gen_sql_mysql_change_master() {

func_temp_master_host_sed=`cat $file_log_mha_manager | grep --color "All other slaves should start" | tail -n 1 | cut -d',' -f1 | cut -d'=' -f2 | cut -d\' -f2`
func_temp_repl_password_sed=`cat $file_log_mha_manager | grep --color "All other slaves should start" | tail -n 1 | rev | cut -d\' -f2`

echo "======================"
echo "@@ func variable: func_temp_repl_password_sed = $func_temp_repl_password_sed"
echo "======================"
str_sql_mysql_change_master=`cat $file_log_mha_manager | grep --color "All other slaves should start" | tail -n 1 | sed "s/'$func_temp_repl_password_sed'/'$str_repl_password'/g" | cut -d':' -f4`
str_sql_mysql_change_master=`echo $str_sql_mysql_change_master | sed "s/'$func_temp_master_host_sed'/'$str_ip_new_master'/g"`
}

# 对指定主机执行Linux命令
# 前提：
# 1. IP可达
# 2. SSH等价关系
function do_linux_by_ssh() {
# variable
func_str_ip="$1"
func_str_user="$2"
func_str_command="$3"

# action
ssh -T -t $func_str_user@$func_str_ip "$func_str_command"
}

function do_part_orig_master_is_new_slave() {
do_linux_by_ssh "$str_ip_orig_master" "root" "service mysql start"
do_sql "$str_ip_orig_master" "set global read_only=1;"
do_sql "$str_ip_orig_master" "$str_sql_mysql_change_master"
do_sql "$str_ip_orig_master" "start slave;"
}

function do_part_mha_master_manager_start() {
do_linux_by_ssh "$str_ip_mha_manager" "root" "nohup masterha_manager --conf=$file_conf_mha_application --ignore_last_failover &"
}

# 如果PID不存在，则执行该脚本，否则，退出
function runable_by_mha_manager_pid() {
echo "-----------------"
echo "Script for MySQL Master HA"
echo "-----------------"
echo "Begin:: "`date "+|%Y-%m-%d|%H:%M:%S|"`

if [[ "$str_pid_masterha_manager" == "" ]]
then
echo "## masterha_manager is [NOT ALIVED]."
else
echo "## masterha_manager is [ALIVED]."
echo "[masterha_manager] PID is:: $str_pid_masterha_manager"

# do something.
echo "## Exit Script"
exit 0
fi
}

# --------------------------
# action

# 如果PID不存在，则执行该脚本，否则，退出
echo "------------------"
echo "app: runable_by_mha_manager_pid"
runable_by_mha_manager_pid
echo ""

echo "------------------"
echo "app: gen_sql_mysql_change_master"
gen_sql_mysql_change_master
echo ""

echo "------------------"
echo "app: do_part_orig_master_is_new_slave"
do_part_orig_master_is_new_slave
echo ""

echo "------------------"
echo "app: do_part_mha_master_manager_start"
#do_part_mha_master_manager_start 如果脚本在费manager端，则使用该函数，否则就是用如下命令
nohup masterha_manager --conf=$file_conf_mha_application --ignore_last_failover &
echo ""

# --------------------------
# Show time

# ---------
# version one
# ---------
#echo "new master is:: $str_ip_new_master"
#echo "Master log file is:: $str_log_file_new_master"
#echo "Master log POS is:: $str_log_pos_new_master"
#echo "orig master --> new master ## SQL: CHANGE MASTER ## is:: $str_sql_mysql_change_master"

# ---------
# version two
# ---------
echo "================="
echo "MySQL info:"

echo "## Account and Password"
echo "username @ $str_mysql_username"
echo "password @ $str_mysql_password"
echo "--- for REPLICATION ---"
echo "repl @ username ## $str_repl_username"
echo "repl @ password ## $str_repl_password"
echo "## Master Server info"
echo "log file @ Master ## $str_log_file_new_master"
echo "log pos @ Master ## $str_log_pos_new_master"
echo ""

echo "================="
echo "SQL statement:"
echo "[CHANGE MASTER] -->"
echo "$str_sql_mysql_change_master"
echo ""

echo "================="
echo "MasterHA info:"

echo "## File and Path"
echo "MHA Global config file @ $file_conf_mha_global"
echo "MHA Application config file @ $file_conf_mha_application"
echo "MHA Log file:: masterha_manager @ $file_log_mha_manager"

echo "## Architecture"
echo "Candicate Server list::"
echo "$list_ip_candicate"

echo "## IP"
echo "MHA Manager Server:: $str_ip_mha_manager"
echo "Last:: new master:: $str_ip_new_master"
echo "Last:: orig master:: $str_ip_orig_master"
echo ""

# --------------------------
echo "-----------------"
echo "Finished:: "`date "+|%Y-%m-%d|%H:%M:%S|"`
# Done