cobbler高可用方案

一、环境准备

主网IP	私网IP	主机名	角色	VIP
10.203.178.125	192.168.10.2	node1	主	10.203.178.137,192.168.10.1
10.203.178.127	192.168.10.3	node2	备	-

二、安装配置dhcp

1、10.203.178.125 配置dhcp服务

a、dhcp服务软件安装

yum install -y dhcpd

b、修改配置文件

vim /etc/dhcp/dhcpd.conf

# cobbler slave dhcpd
ddns-update-style interim;
allow booting;
allow bootp;
ignore client-updates;
set vendorclass = option vendor-class-identifier;
option pxe-system-type code 93 = unsigned integer 16;
subnet 192.168.10.0 netmask 255.255.254.0 {
 option routers             192.168.10.1;
 option domain-name-servers 10.202.32.100;
 option subnet-mask         255.255.254.0;
 range dynamic-bootp        192.168.10.10 192.168.10.199;
 default-lease-time         7200;
 max-lease-time             14400;
 next-server                10.203.178.137;
 class "pxeclients" {
    match if substring (option vendor-class-identifier, 0, 9) = "PXEClient";
    if option pxe-system-type = 00:02 {
            filename "ia64/elilo.efi";
    } else if option pxe-system-type = 00:06 {
            filename "grub/grub-x86.efi";
    } else if option pxe-system-type = 00:07 {
            filename "grub/grub-x86_64.efi";
    } else {
            filename "pxelinux.0";
    }
  }
}

c、启动dhcpd服务：

service dhcpd restart
service dhcpd status

2、10.203.178.127 配置dhcp服务

a、安装dhcp服务

同上

b、配置dhcp服务

同上，但修改下列行到另一个c类地址段

range dynamic-bootp 192.168.11.10 192.168.11.199;

c、启动dhcpd服务：

同上

三、安装配置cobbler

yum install -y cobbler

四、安装配置keepalived

1、10.203.178.125安装keepalived服务

a、软件安装

yum install -y keepalived

b、配置keepalived

vim /etc/keepalived/scripts/keepalived.conf

global_defs {
    router_id pribms.dev
}
vrrp_script chk_http1 {
    script "/etc/keepalived/scripts/check_healthy.sh"
    interval 2
}
vrrp_instance VI_1 {
    state BACKUP
    interface eth0
    virtual_router_id 137
    priority 100
    nopreempt
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 1qaz@WSX
    }
    virtual_ipaddress {
        10.203.178.137
    }
    track_script {
        chk_http1
    }
    master_notify /etc/keepalived/scripts/restart_cobblerd.sh
}
vrrp_instance VI_2 {
    state BACKUP
    interface eth3
    virtual_router_id 101
    priority 100
    nopreempt
    advert_int 1
    authentication {
        auth_type PASS
        auth_pass 2wsx#EDC
    }
    virtual_ipaddress {
        192.168.10.1
    }
}

c、创建 keepalived 健康检查脚本

vim /etc/keepalived/scripts/check_healthy.sh

#!/bin/bash
stop_keepalived() {
    service keepalived stop
    exit 1
}
nc -w 10 -z 127.0.0.1 80 || service httpd restart
nc -w 10 -z 127.0.0.1 8080 || /bin/bash /app/cloud-pri-bms/code/run_app.sh restart
sleep 2
nc -w 10 -z 127.0.0.1 80 || stop_keepalived
nc -w 10 -z 127.0.0.1 8080 || stop_keepalived

d、创建 keepalived 角色切换为主之后的触发脚本

vim /etc/keepalived/scripts/restart_cobblerd.sh

#!/bin/bash
#Auther: Harry
#Date: 2019-09-12
/etc/init.d/cobblerd restart
/usr/bin/cobbler sync

e、启动keepalived并设置开机启动

# 启动服务
service keepalived start
ip a
# 打开开机自启动并验证：
chkconfig keepalived on
chkconfig keepalived --list

2、10.203.178.127安装keepalived服务

a、软件安装

同上

b、配置keepalived

同上

# 修改
priority 99
# 删除
nopreempt

c、创建keepalived监控检查脚本

同上

d、创建 keepalived 角色切换为主之后的触发脚本

同上

e、启动keepalived并设置开机启动

同上

五、安装配置rsync服务：

功能：rysnc客户端可以将指定目录中的文件同步到rsync服务器端

1、10.203.178.125配置rsync服务

a、安装rsync、xinetd、inotify-tools软件

yum install -y rsync xinetd inotify-tools

b、编辑rsyncd.conf配置文件

vim /etc/cobbler/rsync.template

在前面插入下面这段

log file = /var/log/rsyncd.log
pidfile = /var/run/rsyncd.pid
lock file = /var/run/rsync.lock
secrets file = /etc/rsync.pass
motd file = /etc/rsyncd.Motd
[systems]
path = /app/cobbler/config/systems.d/
comment = systemd
uid = root
gid = root
use chroot = no
read only = no
list = no
max connections = 200
timeout = 600
auth users = appfiles_user
hosts allow = 10.203.178.127
[tftpboot]
path = /app/tftpboot/
comment = tftpboot
uid = root
gid = root
use chroot = no
read only = no
list = no
max connections = 200
timeout = 600
auth users = appfiles_user
hosts allow = 10.203.178.127

同步到/etc/rsyncd.conf配置文件中

cobbler sync

c、设置rsync同步的账号和密码

echo "appfiles_user:123456" > /etc/rsync.pass
chmod 600 /etc/rsyncd.conf
chmod 600 /etc/rsync.pass

d、配置rsync同步密码

echo 123456> /etc/keepalived/passwd.txt
chmod 600 /etc/keepalived/passwd.txt

e、设置rsync服务开机自启动

chkconfig rsync on
chkconfig rsync --list

f、配置rsync_inotify.sh脚本文件

同步路径：

src1=/app/cobbler/config/systems.d/
src2=/app/tftpboot/

vim /opt/rsync_scripts/rsync_inotify.sh

#!/bin/bash
#20150608
export PATH=$PATH:/usr/kerberos/sbin:/usr/kerberos/bin:/usr/local/sbin:/usr/local/binsbin:/bin:/usr/sbin:/usr/bin:/root/bin
src1=/app/cobbler/config/systems.d/
src2=/app/tftpboot/
des1=systems
des2=tftpboot
passwdfile=/etc/keepalived/passwd.txt
ip=10.203.178.127
logfile=/var/log/rsync_app.log
if ! grep -q 8192000 /proc/sys/fs/inotify/max_user_watches ; then
echo 8192000 > /proc/sys/fs/inotify/max_user_watches
fi
/usr/bin/inotifywait -mrq --timefmt '%Y-%m-%d %H:%M' --format '%T %w%f %e' -modify,delete,create,attrib $src1 | \
while read file ; do
rsync -vzrtopg --delete --progress $src1 appfiles_user@$ip::$des1 --passworfile=$passwdfile
echo "$file was synced" >> ${logfile} 2>&1
done &
/usr/bin/inotifywait -mrq --timefmt '%Y-%m-%d %H:%M' --format '%T %w%f %e' -modify,delete,create,attrib $src2 | \
while read file ; do
rsync -vzrtopg --delete --progress $src2 appfiles_user@$ip::$des2 --passworfile=$passwdfile
echo "$file was synced" >> ${logfile} 2>&1
done &
sleep 2
date '+%Y-%m-%d %H:%M:%S' >> /var/log/rync_date.log
wait

2、10.203.178.127配置rsync服务

a、安装rsync、xinetd、inotify-tools软件

同上

b、编辑rsyncd.conf配置文件

vim /etc/cobbler/rsync.template

同上

# 修改
hosts allow = 10.203.178.125

同步到/etc/rsyncd.conf配置文件中

同上

c、设置rsync同步的账号和密码

同上

d、配置rsync同步密码

同上

e、重启xinetd服务并设置rsync服务开机自启动

chkconfig rsync on
chkconfig rsync --list
service xinetd restart

f、配置rsync_inotify.sh脚本文件

同上

#修改
ip=10.203.178.125

六、故障恢复：

1、场景简介

正常情况下，10.203.178.125作为主节点提供服务

10.203.178.127作为备节点

主节点上的数据实时同步到备节点

2、主节点10.203.178.125宕机的情况

a、恢复10.203.178.125服务

检查httpd服务
service httpd status
service httpd start
检查cobbler服务
service cobblerd status
service cobblerd start
检查api服务
telnet 127.0.0.1 8080
/bin/bash /app/cloud-pri-bms/code/run_app.sh start

b、同步10.203.178.127数据到10.203.178.125

10.203.178.125检查并启动xinetd服务

/etc/init.d/xinetd status
/etc/init.d/xinetd start        #未启动则手动启动服务

10.203.178.127执行同步脚本并创建文件

nohup sh /opt/rsync_scripts/rsync_inotify.sh >>/app/clog/rsync.log 2>&1 &
touch /app/cobbler/config/systems.d/1111.txt
touch /app/tftpboot/2222.txt

10.203.178.125上验证是否同步：

ll /app/cobbler/config/systems.d/
ll /app/tftpboot/

10.203.178.127上删掉刚才创建的文件

rm /app/cobbler/config/systems.d/1111.txt
rm /app/tftpboot/2222.txt

c、启动10.203.178.125高可用服务

检查keepalived服务并启动
service keepalived status
service keepalived start

3、备节点10.203.178.127宕机的情况

a、恢复10.203.178.127服务

检查httpd服务
service httpd status
service httpd start
检查cobbler服务
service cobblerd status
service cobblerd start
检查api服务
telnet 127.0.0.1 8080
/bin/bash /app/cloud-pri-bms/code/run_app.sh start

b、同步10.203.178.125数据到10.203.178.127

10.203.178.127检查并启动xinetd服务

service xinetd status
service xinetd start

10.203.178.125执行同步脚本并创建文件

touch /app/cobbler/config/systems.d/1111.txt
touch /app/tftpboot/2222.txt

10.203.178.127上验证是否同步

ll /app/cobbler/config/systems.d/
ll /app/tftpboot/

10.203.178.125上删掉刚才创建的文件

rm /app/cobbler/config/systems.d/1111.txt
rm /app/tftpboot/2222.txt

10.203.178.127重启cobbler服务并同步

service cobblerd restart
cobbler sync

c、启动10.203.178.127高可用服务

检查keepalived服务并启动
service keepalived status
service keepalived start