Lvs+Keepalived实现MySQL高可用

LVS+Keepalived+MySQL高可用配置

本文所有配置前提是已实现MySQL双主备份(MySQL双主)

安装前的准备:

VIP:192.168.0.201

Keepalived:

　　Keepalived-Master:192.168.0.241

　　Keepalived-Backup:192.168.0.277

RealServer:

　　MySQL1:192.168.0.225

　　MySQL2:192.168.0.226

kernel-devel各版本下载地址:http://rpmfind.net/linux/rpm2html/search.php?query=kernel-devel

keepalived下载地址:http://www.keepalived.org/download.html

kernel-devel.2.6.32-431.el6.x86_64.rpm (根据系统实际内核版本)

keepalived-1.2.7.tar.gz

ipvsadm-1.26.tar.gz

依赖包 :

yum install openssl-devle popt-devel popt-static libnl*

一、软件安装

1、安装kernel-devel:

rpm -ivh kernel-devel-2.6.32-431.29.2.el6.x86_64.rpm

2、安装lvs管理工具ipsadm:

tar zxvf ipvsadm-1.26.tar.gz
cd ipvsadm
make
make install

3、安装keepalived:

tar zxvf keepalived-1.2.7.tar.gz
cd keepalived
./configure --prefix=/usr/local/keepalived
make
make install

4、将keepalived添加到系统服务:

cp /usr/local/keepalived/sbin/keepalived /usr/sbin/
cp /usr/local/keepalived/etc/sysconfig/keepalived /etc/sysconfig/
cp /usr/local/keepalived/etc/rc.d/init.d/keepalived /etc/init.d/
mkdir /etc/keepalived
cp /usr/local/keepalived/etc/keepalived/keepalived.conf /etc/keepalived/

二、配置

1、MySQL服务器中的配置 :

将realserver脚本放在/etc/init.d/目录下，执行即可:

#!/bin/sh
VIP=192.168.0.201

sh /etc/rc.d/init.d/functions

case "$1" in
# 禁用本地的ARP请求、绑定本地回环地址
start)
    /sbin/ifconfig lo down
    /sbin/ifconfig lo up
    echo "" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "" >/proc/sys/net/ipv4/conf/all/arp_announce
    /sbin/sysctl -p >/dev/null 2>&1
    /sbin/ifconfig lo:0 $VIP netmask 255.255.255.255 up #在回环地址上绑定VIP，设定掩码，与Direct Server（自身）上的IP保持通信
    /sbin/route add -host $VIP dev lo:0
    echo "LVS-DR real server starts successfully.\n"
    ;;
stop)
    /sbin/ifconfig lo:0 down
    /sbin/route del $VIP >/dev/null 2>&1
    echo "" >/proc/sys/net/ipv4/conf/lo/arp_ignore
    echo "" >/proc/sys/net/ipv4/conf/lo/arp_announce
    echo "" >/proc/sys/net/ipv4/conf/all/arp_ignore
    echo "" >/proc/sys/net/ipv4/conf/all/arp_announce
echo "LVS-DR real server stopped.\n"
    ;;
status)
    isLoOn=`/sbin/ifconfig lo:0 | grep "$VIP"`
    isRoOn=`/bin/netstat -rn | grep "$VIP"`
    if [ "$isLoON" == "" -a "$isRoOn" == "" ]; then
        echo "LVS-DR real server has run yet."
    else
        echo "LVS-DR real server is running."
    fi
    exit 3
    ;;
*)
    echo "Usage: $0 {start|stop|status}"
    exit 1
esac
exit 0

然后使用 ip addr 命令查看VIP是否绑定:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet 192.168.0.201/32 brd 192.168.0.201 scope global lo:0
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:59:f1:3f brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.225/24 brd 192.168.0.255 scope global eth0
    inet6 fe80::5054:ff:fe59:f13f/64 scope link
       valid_lft forever preferred_lft forever

可以看到vip已经绑定在了lo上。两台msyql服务器执行同样的操作。

2、Keepalived的配置:

2.1、首先关闭系统防火墙iptables和selinux

2.2、开启路由转发及非本机ip绑定支持功能:

echo 1 >/proc/sys/net/ipv4/ip_forward

echo 1 >/proc/sys/net/ipv4/ip_nonlocal_bind

2.3、修改Keepalived的配置文件/etc/keepalived/keepalived.conf:

! Configuration File for keepalived

global_defs {
   notification_email {
     123@sina.com
   }
   notification_email_from 321@sina.com
   smtp_server 192.168.0.241
   smtp_connect_timeout 30
   router_id LVS1
}

vrrp_instance VI_1 {
    state BACKUP              ＃角色,若都为BACKUP，则表示当其中一台挂掉时，以另一台为MASTER，(即不会主动抢VIP)
    interface eth0            ＃主备之间健康检查的网卡
    virtual_router_id 51      ＃主备id相同
    priority 100
    advert_int 1
    authentication {
        auth_type PASS        ＃主备之间通信认证
        auth_pass 0711
    }
    virtual_ipaddress {
        192.168.0.201
    }
}

virtual_server 192.168.0.201 3306 {
    delay_loop 6
    lb_algo rr
    lb_kind DR
    nat_mask 255.255.255.0
    persistence_timeout 50
    protocol TCP

    real_server 192.168.0.225 3306 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        connect_prot 3306
        }
    }
    real_server 192.168.0.226 3306 {
        weight 1
        TCP_CHECK {
            connect_timeout 3
            nb_get_retry 3
            delay_before_retry 3
        connect_prot 3306
        }
    }
}

keepalived的Master和Backup配置文件不同的是，

Master: state为MASTER，priority的值比BACKUP大。

BACKUP:state为BACKUP，priority的值比MASTER小。

2.4、启动Keepalived:

service keepalived start

然后使用ipvsadm工具查看集群列表:

主、备结果一样。

[root@localhost ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.201:3306 rr persistent 50
  -> 192.168.0.225:3306           Route   1      0          0
  -> 192.168.0.226:3306           Route   1      0          0         

查看ip地址情况:

MASTER:

[root@localhost ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: p4p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 64:00:6a:34:c3:e2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.241/24 brd 192.168.0.255 scope global p4p1
    inet 192.168.0.201/32 scope global p4p1
    inet6 fe80::6600:6aff:fe34:c3e2/64 scope link
       valid_lft forever preferred_lft forever

BACKUP：

[root@localhost ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:b3:34:e6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.227/24 brd 192.168.0.255 scope global eth0
    inet6 fe80::5054:ff:feb3:34e6/64 scope link
       valid_lft forever preferred_lft forever

可以看到，MASTER上绑定了VIP，而BACKUP没有。

3、测试:

3.1keepalived主从测试:

把MASTER的keepalived服务停掉，service keepalived stop ，然后再查看ip:

[root@localhost ~]# service keepalived stop
Stopping keepalived:                                       [  OK  ]

[root@localhost ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: p4p1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 64:00:6a:34:c3:e2 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.241/24 brd 192.168.0.255 scope global p4p1
    inet6 fe80::6600:6aff:fe34:c3e2/64 scope link
       valid_lft forever preferred_lft forever

发现VIP没了。此时再查看BACKUP的ip:

[root@localhost ~]# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 1000
    link/ether 52:54:00:b3:34:e6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.0.227/24 brd 192.168.0.255 scope global eth0
    inet 192.168.0.201/32 scope global eth0
    inet6 fe80::5054:ff:feb3:34e6/64 scope link
       valid_lft forever preferred_lft forever

VIP成功漂移到了BACKUP上。

同时，通过keepalived的日志也可以看出，当master停掉后，backup主动成为master:

cat /var/log/messages

Jan 25 16:18:28 localhost Keepalived_healthcheckers[1199]: SMTP connection ERROR to [192.168.0.227]:25.
Jan 25 16:42:41 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) Transition to MASTER STATE
Jan 25 16:42:42 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) Entering MASTER STATE
Jan 25 16:42:42 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) setting protocol VIPs.
Jan 25 16:42:42 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.201
Jan 25 16:42:42 localhost Keepalived_healthcheckers[1199]: Netlink reflector reports IP 192.168.0.201 added
Jan 25 16:42:47 localhost Keepalived_vrrp[1200]: VRRP_Instance(VI_1) Sending gratuitous ARPs on eth0 for 192.168.0.201

3.2MySQL高可用测试:

通过VIP连接mysql:

localhost:~ ahaii$ mysql -h192.168.0.201 -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 643
Server version: 5.5.54-log Source distribution

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

连接成功。

同时在keepalived-master上查看lvs分配情况:

[root@localhost ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.201:3306 rr persistent 50
  -> 192.168.0.225:3306           Route   1      0          0
  -> 192.168.0.226:3306           Route   1      1          0
[root@localhost ~]# 

显示现在连接的是226的那台mysql服务器。

此时，将226服务器上的mysql服务停掉，然后再通过vip连接mysql，并查看lvs分配情况:

依然能通过vip连接mysql:

localhost:~ ahaii$ mysql -h192.168.0.201 -uroot -p
Enter password:
Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 37014
Server version: 5.5.54-log Source distribution

Copyright (c) 2000, 2015, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

此时，226已被keepalived集群剔除掉，并且客户端连接的是225服务器 ：

[root@localhost ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.201:3306 rr persistent 50
  -> 192.168.0.225:3306           Route   1      1          0         

当把226服务器的mysql服务重新启动时，keepalived又将其加入到集群中:

[root@localhost ~]# ipvsadm -L -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  192.168.0.201:3306 rr persistent 50
  -> 192.168.0.225:3306           Route   1      1          0
  -> 192.168.0.226:3306           Route   1      0          0         

keepalived剔除、添加226服务器的日志记录如下:

Jan 25 17:00:27 localhost Keepalived_healthcheckers[1429]: TCP connection to [192.168.0.226]:3306 failed !!!
Jan 25 17:00:27 localhost Keepalived_healthcheckers[1429]: Removing service [192.168.0.226]:3306 from VS [192.168.0.201]:3306
Jan 25 17:03:27 localhost Keepalived_healthcheckers[1429]: TCP connection to [192.168.0.226]:3306 success.
Jan 25 17:03:27 localhost Keepalived_healthcheckers[1429]: Adding service [192.168.0.226]:3306 to VS [192.168.0.201]:3306

疑惑:

1、客户点请求VIP时，由于调度器和realserver都有VIP，如何保证是调度器接收请求，而不是realserver？

为了保证是调度器接收客户端的请求，在realserver上都做了如下配置:

echo "" >/proc/sys/net/ipv4/conf/lo/arp_ignore
echo "" >/proc/sys/net/ipv4/conf/lo/arp_announce
echo "" >/proc/sys/net/ipv4/conf/all/arp_ignore
echo "" >/proc/sys/net/ipv4/conf/all/arp_announce

用arp_ignore参数设定当请求的目标地址是本机IP时，对其ARP的查询作出不同级别的响应。

1:表示只回答目标IP地址是本地网卡的arp请求

2:表示只回答目标IP地址是本地网卡的arp请求，并且来访IP与本机IP在同一网段内。

用arp_announce参数设定本机向外宣告自身IP-MAC时的级别。

1:表示尽量避免响应ARP请求中MAC不是本网卡的。如一个主机有多块网卡，其中一块网卡接收到了ARP请求，发现所请求的MAC是本机另一块网卡的，这个时候接收到ARP请求的这块网卡就尽量避免响应。

2:表示总是使用最合适的网卡来响应。一个主机有多块网卡，其中一块网卡接收到了ARP请求，发现所请求的MAC是本机另一块网卡的，这个时候接收到ARP请求的这块网卡就一定不响应，只有发现请求的MAC是自己的才给与响应。

以上设置，就是为了接收客户点请求的是调度器的VIP，而不是realserver的VIP。

2、为什么负载均衡器和realserver上都有VIP呢？

当均衡器收到客户端请求后，根据配置的算法选取一台realserver，将客户端请求报文的目标MAC地址修改为选取的realserver的MAC地址后重新封装该数据帧并发送给选取的realserver。realserver收到该数据帧后检查IP报文发现该报文的目标IP(VIP)是自己，然后对该请求报文做出响应，将响应结果直接发送给客户端(整个过程中源地址不变)。

以上是整个安装配置和测试的过程。