一、需求

在主从结构的集群中,我们假设硬件机器是很脆弱的,随时可能会宕机,当master挂掉之后需要从slave中选出一个节点作为新的master,使用zookeeper可以很简单的实现集群选主功能。

二、分析

下面为了方便叙述,将使用更通用的技术术语,即使用leader表示master,使用follower表示slave。

集群选主涉及到两个问题:

1. 谁来做leader

2. leader挂掉了怎么被follower感知到

首先是第一个问题,谁来做leader,其实可以将这个问题看做是多线程中的互斥锁抢占,锁只有一把,并且只能被一个人抢到,这里就把一个zookeeper上的一个节点/leader-info看做是锁,集群中的每台机器都尝试去创建这个节点(抢占锁),因为zookeeper创建节点是原子性操作,所以只有一台机器能够创建成功其它都会失败,创建成功的那台机器就作为leader,其它机器做follower,一般leader抢占成功了之后会在/leader-info节点上存储一些与自己相关的信息,比如hostname、id之类的,以让follower知道谁抢占成功成为了leader,然后去连接leader进行一些数据交换或指令控制之类的,那就是选主之后的事了不在此篇文章的讨论范围之内。

第二个问题是leader挂掉了怎么通知其它的follower,zookeeper中的节点按照有效时间分为持久节点和临时节点,临时节点跟session绑定,而一个session表示一个客户端,当客户端下线的时候session失效,当session失效的时候跟它绑定的临时节点就会被删除,利用这个特性可以检测节点是否还在存活状态,实现follower对leader下线的感知,只是需要注意在创建/leader-info节点的时候要将其创建为临时节点,然后众多follower都在这个节点上添加一个watcher监听其删除事件,这样当leader挂掉的时候session失效,然后与此session绑定的临时节点会被删除,即/leader-info节点将被删除,同时给所有的follower发送事件通知,follower一看leader挂了就燥起来了,将自己的状态置为looking,开始新一轮的选举。

总结一下选主的流程:

1. 集群中的所有机器将自己置为looking状态,准备开始选举。

2. 所有looking状态的机器尝试去创建/leader-info节点。

3. 创建成功的将自己的状态修改为leader,同时将自己的一些信息写入到/leader-info这个节点上。

4. 创建失败的将自己的状态置为follower,同时尝试从/leader-info获取leader信息进行一些leader改变的逻辑(在这里这个不是重点,打印一下即可),follower在获取/leader-info节点数据的同时要设置一个watcher,监听此节点的删除事件,当节点被删除事件触发时启动新一轮的选举,因为获取数据设置watcher这个操作是原子性的,所以要么这个节点存在获取数据成功,并且设置watcher也成功,要么节点不存在抛出KeeperException.NoNodeException异常。

5. 为什么在follower设置watcher的时候还有可能会抛异常呢,leader不是已经创建了这个节点了吗?因为follower从尝试创建/leader-info节点失败到去获取此节点的数据同时设置watcher这一段操作不是原子性的,在这中间可能会发生一些变故,leader可能刚成为leader就挂掉了(或者因为一些网络抖动原因,总之是session失效了),leader挂掉之后它创建的临时节点就被zookeeper删除了,所以当follower在设置watcher的时候如果检测到KeeperException.NoNodeException,说明之前的leader挂掉了,此时集群中已经没有了leader,follower又燥起来了,它将自己的状态置为looking开始新一轮的选举。

三、实现

Node.java:

package cc11001100.zookeeper.leaderElection;

import cc11001100.zookeeper.utils.ZooKeeperUtil;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooKeeper; import java.io.IOException;
import java.io.UnsupportedEncodingException; /**
* 表示集群中的一个节点
*
* @author CC11001100
*/
public class Node { private Status status;
private String nodeForLeaderInfo;
private ZooKeeper zooKeeper; public Node(String listenerNodeForLeader) throws IOException {
this.nodeForLeaderInfo = listenerNodeForLeader;
this.zooKeeper = ZooKeeperUtil.getZooKeeper();
lookingForLeader();
} public void lookingForLeader() {
status = Status.LOOKING;
try {
String leaderInfo = Thread.currentThread().getName();
// 需要注意这里创建的是临时节点
zooKeeper.create(nodeForLeaderInfo, leaderInfo.getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
// 如果上一步没有抛异常,说明自己已经是leader了
status = Status.LEADER;
String logMsg = Thread.currentThread().getName() + " is leader";
System.out.println(logMsg);
} catch (KeeperException.NodeExistsException e) {
// 节点已经存在,说明leader已经被别人注册成功了,自己是follower
status = Status.FOLLOWER;
try {
byte[] leaderInfoBytes = zooKeeper.getData(nodeForLeaderInfo, event -> {
if (event.getType() == Watcher.Event.EventType.NodeDeleted) {
lookingForLeader();
}
}, null);
String logMsg = Thread.currentThread().getName() + " is follower, master is " + new String(leaderInfoBytes, "UTF-8");
System.out.println(logMsg);
} catch (KeeperException.NoNodeException e1) {
// 如果在获取leader信息的时候报了节点不存在,说明这个leader比较短命,刚抢到leader就又挂掉了
lookingForLeader();
} catch (KeeperException | InterruptedException | UnsupportedEncodingException e1) {
e1.printStackTrace();
}
} catch (KeeperException | InterruptedException e) {
e.printStackTrace();
}
} public void shutdown() {
try {
if (zooKeeper != null) {
zooKeeper.close();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
} public Status getStatus() {
return status;
} // 当前节点的状态,节点的状态必须在这三个中的一个
public enum Status {
LOOKING, // 选举中
LEADER, // 选举完毕,当前节点为leader
FOLLOWER; // 选举完毕,当前节点为follower
} }

LeaderElectionTest.java:

package cc11001100.zookeeper.leaderElection;

import java.io.IOException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong; /**
* @author CC11001100
*/
public class LeaderElectionTest { private static void sleep(long mils) {
try {
TimeUnit.MILLISECONDS.sleep(mils);
} catch (InterruptedException e) {
e.printStackTrace();
}
} public static void main(String[] args) throws IOException { final String LEADER_INFO_NODE = "/leader-info";
int nodeNum = 10;
AtomicLong idGenerator = new AtomicLong();
AtomicInteger activeNodeCount = new AtomicInteger();
while (true) {
if (activeNodeCount.get() >= nodeNum) {
sleep(10);
continue;
} // 线程启动需要一定时间,将线程启动看做开机过程,在开机之前就算一台新的机器加入了
activeNodeCount.incrementAndGet();
new Thread(() -> {
try {
Node node = new Node(LEADER_INFO_NODE);
while (true) {
sleep(1000);
// 这里为了试验就让leader有轻微自杀倾向...
if (node.getStatus() == Node.Status.LEADER && Math.random() < 0.3) {
String logMsg = "----------------------------- " + Thread.currentThread().getName() + " shutdown -----------------------------";
System.out.println(logMsg);
node.shutdown();
break;
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
activeNodeCount.decrementAndGet();
}
}, "node-" + idGenerator.getAndIncrement()).start();
}
} }

控制台输出:

...
node-4 is leader
node-3 is follower, master is node-4
node-0 is follower, master is node-4
node-9 is follower, master is node-4
node-7 is follower, master is node-4
node-5 is follower, master is node-4
node-1 is follower, master is node-4
node-6 is follower, master is node-4
node-8 is follower, master is node-4
node-2 is follower, master is node-4
----------------------------- node-4 shutdown -----------------------------
node-0-EventThread is leader
node-6-EventThread is follower, master is node-0-EventThread
node-3-EventThread is follower, master is node-0-EventThread
node-7-EventThread is follower, master is node-0-EventThread
node-1-EventThread is follower, master is node-0-EventThread
node-5-EventThread is follower, master is node-0-EventThread
node-9-EventThread is follower, master is node-0-EventThread
node-2-EventThread is follower, master is node-0-EventThread
node-8-EventThread is follower, master is node-0-EventThread
node-10 is follower, master is node-0-EventThread
----------------------------- node-0 shutdown -----------------------------
node-6-EventThread is leader
node-7-EventThread is follower, master is node-6-EventThread
node-1-EventThread is follower, master is node-6-EventThread
node-3-EventThread is follower, master is node-6-EventThread
node-10-EventThread is follower, master is node-6-EventThread
node-9-EventThread is follower, master is node-6-EventThread
node-5-EventThread is follower, master is node-6-EventThread
node-2-EventThread is follower, master is node-6-EventThread
node-8-EventThread is follower, master is node-6-EventThread
node-11 is follower, master is node-6-EventThread
----------------------------- node-6 shutdown -----------------------------
node-1-EventThread is leader
node-10-EventThread is follower, master is node-1-EventThread
node-7-EventThread is follower, master is node-1-EventThread
node-11-EventThread is follower, master is node-1-EventThread
node-8-EventThread is follower, master is node-1-EventThread
node-5-EventThread is follower, master is node-1-EventThread
node-9-EventThread is follower, master is node-1-EventThread
node-3-EventThread is follower, master is node-1-EventThread
node-2-EventThread is follower, master is node-1-EventThread
node-12 is follower, master is node-1-EventThread
----------------------------- node-1 shutdown -----------------------------
node-3-EventThread is leader
node-12-EventThread is follower, master is node-3-EventThread
node-11-EventThread is follower, master is node-3-EventThread
node-5-EventThread is follower, master is node-3-EventThread
node-7-EventThread is follower, master is node-3-EventThread
node-9-EventThread is follower, master is node-3-EventThread
node-2-EventThread is follower, master is node-3-EventThread
node-10-EventThread is follower, master is node-3-EventThread
node-8-EventThread is follower, master is node-3-EventThread
node-13 is follower, master is node-3-EventThread
----------------------------- node-3 shutdown -----------------------------
node-5-EventThread is leader
node-13-EventThread is follower, master is node-5-EventThread
node-12-EventThread is follower, master is node-5-EventThread
node-7-EventThread is follower, master is node-5-EventThread
node-11-EventThread is follower, master is node-5-EventThread
node-10-EventThread is follower, master is node-5-EventThread
node-9-EventThread is follower, master is node-5-EventThread
node-2-EventThread is follower, master is node-5-EventThread
node-8-EventThread is follower, master is node-5-EventThread
node-14 is follower, master is node-5-EventThread
----------------------------- node-5 shutdown -----------------------------
node-7-EventThread is leader
node-13-EventThread is follower, master is node-7-EventThread
node-12-EventThread is follower, master is node-7-EventThread
node-9-EventThread is follower, master is node-7-EventThread
node-11-EventThread is follower, master is node-7-EventThread
node-14-EventThread is follower, master is node-7-EventThread
node-10-EventThread is follower, master is node-7-EventThread
node-8-EventThread is follower, master is node-7-EventThread
node-2-EventThread is follower, master is node-7-EventThread
node-15 is follower, master is node-7-EventThread
----------------------------- node-7 shutdown -----------------------------
node-14-EventThread is leader
node-13-EventThread is follower, master is node-14-EventThread
node-11-EventThread is follower, master is node-14-EventThread
node-2-EventThread is follower, master is node-14-EventThread
node-12-EventThread is follower, master is node-14-EventThread
node-15-EventThread is follower, master is node-14-EventThread
node-10-EventThread is follower, master is node-14-EventThread
node-9-EventThread is follower, master is node-14-EventThread
node-8-EventThread is follower, master is node-14-EventThread
node-16 is follower, master is node-14-EventThread
----------------------------- node-14 shutdown -----------------------------
node-13-EventThread is leader
node-12-EventThread is follower, master is node-13-EventThread
node-15-EventThread is follower, master is node-13-EventThread
node-9-EventThread is follower, master is node-13-EventThread
node-10-EventThread is follower, master is node-13-EventThread
node-2-EventThread is follower, master is node-13-EventThread
node-8-EventThread is follower, master is node-13-EventThread
node-11-EventThread is follower, master is node-13-EventThread
node-16-EventThread is follower, master is node-13-EventThread
node-17 is follower, master is node-13-EventThread
...

最后有个需要注意的地方就是是否需要将leader节点设置为顺序临时节点呢?类似于分布式锁那样,这样的话每次唤醒一个节点就可以了,这看上去像是一个可以优化的点。

其实并不是,当leader挂掉的时候必须所有follower都能够感知到leader的变更,即使他们不参与抢主也必须醒来执行leader变更的逻辑。

.

Zookeeper笔记之使用zk实现集群选主的更多相关文章

  1. ZooKeeper实践:(2)集群管理

    前言: 随着业务的扩大,用户的增多,访问量的增加,单机模式已经不能支撑,从而出现了从单机模式->垂直应用模式->集群模式,集群模式诞生了,伴随着一堆问题也油然而生,Master怎么选举,机 ...

  2. zookeeper的单实例和伪集群部署

    原文链接: http://gudaoyufu.com/?p=1395 zookeeper工作方式 ZooKeeper 是一个开源的分布式协调服务,由雅虎创建,是 Google Chubby 的开源实现 ...

  3. ZooKeeper实践:(1)集群管理

    前言: 随着业务的扩大,用户的增多,访问量的增加,单机模式已经不能支撑,从而出现了从单机模式->垂直应用模式->集群模式,集群模式诞生了,伴随着一堆问题也油然而生,Master怎么选举,机 ...

  4. zookeeper在windows下的伪集群模式

    参考:zookeeper在windows下的伪集群模式 踩到的坑: 注意windows下路径需要使用\ dataDir=D:\Program Files\Java\zookeeper-3.4.10-c ...

  5. zk伪集群部署

    jdk 配置 # tar xf jdk-8u161-linux-x64.tar.gz -C /usr/local/ # vim /etc/profile.d/jdk.sh export JAVA_HO ...

  6. Zookeeper节点增删改查与集群搭建(笔记)

    1.上传文件目录说明 上传的文件一般放在 /home/下 安装文件一般在 /usr/local/下 2. 安装zookeeper 2.1将zookeeper-3.4.11.tar.gz拷贝到/home ...

  7. Zookeeper的基本原理(zk架构、zk存储结构、watch机制、独立安装zk、集群间同步复制)

    1.Hbase集群的高可用性与伸缩性 HBase可以实现对Regionserver的监控,当个别Regionserver不可访问时,将其负责的分区分给其他Regionsever,其转移过程较快,因为只 ...

  8. 基于zookeeper的高可用Hadoop HA集群安装

    (1)hadoop2.7.1源码编译 http://aperise.iteye.com/blog/2246856 (2)hadoop2.7.1安装准备 http://aperise.iteye.com ...

  9. zookeeper笔记之基于zk实现分布式锁

    一.分布式锁概述 Java中基于AQS框架提供了一系列的锁,但是当需要在集群中的多台机器上互斥执行一段代码或使用资源时Java提供的这种单机锁就没了用武之地,此时需要使用分布式锁协调它们.分布式锁有很 ...

随机推荐

  1. 1. Python3 环境搭建

    Python3 环境搭建 开门见山,其他关于Python发展史.语言类型.优缺点等等 可以自己去百度百度,这里就不多说了.其实基本想要学这门语言的时候,你已经了解差不多了!!! Python的运行环境 ...

  2. Android 使用 OnTouchListener 接口监听双击或多击事件

    这里是使用 OnTouchListener 实现的监听双击 or 多击的监听器.通过 View.setOnTouchListener ,可以实现在任意 View 上监听双击事件. 网上有许多文章简单的 ...

  3. Final发布 视频展示

    1.视频链接 视频地址:http://v.youku.com/v_show/id_XMzk1OTYyNjE0NA==.html?spm=a2hzp.8244740.0.0 杨老师粉丝群——弹球学成语项 ...

  4. 【SE】Week3 : 四则运算式生成评分工具Extension&Release Version(结对项目)

    Foreword 此次的结对项目终于告一段落,除了本身对软件开发的整体流程有了更深刻的了解外,更深刻的认识应该是结对编程对这一过程的促进作用. 在此想形式性但真心地啰嗦几句,十分感谢能端同学能够不厌其 ...

  5. 从编译DotNetOpenAuth中学到的程序集强签名知识

    1. 背景 最近在研究DotNetOpenAuth——OAuth的一个.NET开源实现,官方网站:http://dotnetopenauth.net/ . 从GitHub签出DotNetOpenAut ...

  6. 为elasticSearch开发c++接口

    一.    ElasticSearch是什么 ElasticSearch是目前开源全文搜索引擎的首选,可以快速存储,搜索和分析海量数据.Stack Overflow,Github等都在使用. Elas ...

  7. Sum 南京网络赛J题

    题意: 统计每个数的因子的对数,如果因子能被某个平方数整除,则不统计在内,每对因子有序 解析: 我们对某个数n进行质因子分解,如果某个质因子的指数大于2则 f(n) = 0, 例 N = X3 * M ...

  8. DynamicSegmentTree

    最近尝试了一下动态开点线段树,英文直译就是Dynamic Open Point Segment Tree,太SB了. 就跟之前的主席树写法差不多. if(!x || x == y) { x = ++t ...

  9. (转)win下修改jdk环境变量后,java版本不变 java -version

    背景:在windows下安装了多个版本的jdk,发现修改环境变量无法切换. win 7环境下修改JAVA_HOME后,在命令行执行:java -version 发现版本没有变化,以为需要重启才行,就把 ...

  10. HDU 6181 第k短路

    Two Paths Time Limit: 4000/2000 MS (Java/Others)    Memory Limit: 153428/153428 K (Java/Others)Total ...