Zookeeper笔记之使用zk实现集群选主
一、需求
在主从结构的集群中,我们假设硬件机器是很脆弱的,随时可能会宕机,当master挂掉之后需要从slave中选出一个节点作为新的master,使用zookeeper可以很简单的实现集群选主功能。
二、分析
下面为了方便叙述,将使用更通用的技术术语,即使用leader表示master,使用follower表示slave。
集群选主涉及到两个问题:
1. 谁来做leader
2. leader挂掉了怎么被follower感知到
首先是第一个问题,谁来做leader,其实可以将这个问题看做是多线程中的互斥锁抢占,锁只有一把,并且只能被一个人抢到,这里就把一个zookeeper上的一个节点/leader-info看做是锁,集群中的每台机器都尝试去创建这个节点(抢占锁),因为zookeeper创建节点是原子性操作,所以只有一台机器能够创建成功其它都会失败,创建成功的那台机器就作为leader,其它机器做follower,一般leader抢占成功了之后会在/leader-info节点上存储一些与自己相关的信息,比如hostname、id之类的,以让follower知道谁抢占成功成为了leader,然后去连接leader进行一些数据交换或指令控制之类的,那就是选主之后的事了不在此篇文章的讨论范围之内。
第二个问题是leader挂掉了怎么通知其它的follower,zookeeper中的节点按照有效时间分为持久节点和临时节点,临时节点跟session绑定,而一个session表示一个客户端,当客户端下线的时候session失效,当session失效的时候跟它绑定的临时节点就会被删除,利用这个特性可以检测节点是否还在存活状态,实现follower对leader下线的感知,只是需要注意在创建/leader-info节点的时候要将其创建为临时节点,然后众多follower都在这个节点上添加一个watcher监听其删除事件,这样当leader挂掉的时候session失效,然后与此session绑定的临时节点会被删除,即/leader-info节点将被删除,同时给所有的follower发送事件通知,follower一看leader挂了就燥起来了,将自己的状态置为looking,开始新一轮的选举。
总结一下选主的流程:
1. 集群中的所有机器将自己置为looking状态,准备开始选举。
2. 所有looking状态的机器尝试去创建/leader-info节点。
3. 创建成功的将自己的状态修改为leader,同时将自己的一些信息写入到/leader-info这个节点上。
4. 创建失败的将自己的状态置为follower,同时尝试从/leader-info获取leader信息进行一些leader改变的逻辑(在这里这个不是重点,打印一下即可),follower在获取/leader-info节点数据的同时要设置一个watcher,监听此节点的删除事件,当节点被删除事件触发时启动新一轮的选举,因为获取数据设置watcher这个操作是原子性的,所以要么这个节点存在获取数据成功,并且设置watcher也成功,要么节点不存在抛出KeeperException.NoNodeException异常。
5. 为什么在follower设置watcher的时候还有可能会抛异常呢,leader不是已经创建了这个节点了吗?因为follower从尝试创建/leader-info节点失败到去获取此节点的数据同时设置watcher这一段操作不是原子性的,在这中间可能会发生一些变故,leader可能刚成为leader就挂掉了(或者因为一些网络抖动原因,总之是session失效了),leader挂掉之后它创建的临时节点就被zookeeper删除了,所以当follower在设置watcher的时候如果检测到KeeperException.NoNodeException,说明之前的leader挂掉了,此时集群中已经没有了leader,follower又燥起来了,它将自己的状态置为looking开始新一轮的选举。
三、实现
Node.java:
package cc11001100.zookeeper.leaderElection; import cc11001100.zookeeper.utils.ZooKeeperUtil;
import org.apache.zookeeper.CreateMode;
import org.apache.zookeeper.KeeperException;
import org.apache.zookeeper.Watcher;
import org.apache.zookeeper.ZooDefs;
import org.apache.zookeeper.ZooKeeper; import java.io.IOException;
import java.io.UnsupportedEncodingException; /**
* 表示集群中的一个节点
*
* @author CC11001100
*/
public class Node { private Status status;
private String nodeForLeaderInfo;
private ZooKeeper zooKeeper; public Node(String listenerNodeForLeader) throws IOException {
this.nodeForLeaderInfo = listenerNodeForLeader;
this.zooKeeper = ZooKeeperUtil.getZooKeeper();
lookingForLeader();
} public void lookingForLeader() {
status = Status.LOOKING;
try {
String leaderInfo = Thread.currentThread().getName();
// 需要注意这里创建的是临时节点
zooKeeper.create(nodeForLeaderInfo, leaderInfo.getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);
// 如果上一步没有抛异常,说明自己已经是leader了
status = Status.LEADER;
String logMsg = Thread.currentThread().getName() + " is leader";
System.out.println(logMsg);
} catch (KeeperException.NodeExistsException e) {
// 节点已经存在,说明leader已经被别人注册成功了,自己是follower
status = Status.FOLLOWER;
try {
byte[] leaderInfoBytes = zooKeeper.getData(nodeForLeaderInfo, event -> {
if (event.getType() == Watcher.Event.EventType.NodeDeleted) {
lookingForLeader();
}
}, null);
String logMsg = Thread.currentThread().getName() + " is follower, master is " + new String(leaderInfoBytes, "UTF-8");
System.out.println(logMsg);
} catch (KeeperException.NoNodeException e1) {
// 如果在获取leader信息的时候报了节点不存在,说明这个leader比较短命,刚抢到leader就又挂掉了
lookingForLeader();
} catch (KeeperException | InterruptedException | UnsupportedEncodingException e1) {
e1.printStackTrace();
}
} catch (KeeperException | InterruptedException e) {
e.printStackTrace();
}
} public void shutdown() {
try {
if (zooKeeper != null) {
zooKeeper.close();
}
} catch (InterruptedException e) {
e.printStackTrace();
}
} public Status getStatus() {
return status;
} // 当前节点的状态,节点的状态必须在这三个中的一个
public enum Status {
LOOKING, // 选举中
LEADER, // 选举完毕,当前节点为leader
FOLLOWER; // 选举完毕,当前节点为follower
} }
LeaderElectionTest.java:
package cc11001100.zookeeper.leaderElection; import java.io.IOException;
import java.util.concurrent.TimeUnit;
import java.util.concurrent.atomic.AtomicInteger;
import java.util.concurrent.atomic.AtomicLong; /**
* @author CC11001100
*/
public class LeaderElectionTest { private static void sleep(long mils) {
try {
TimeUnit.MILLISECONDS.sleep(mils);
} catch (InterruptedException e) {
e.printStackTrace();
}
} public static void main(String[] args) throws IOException { final String LEADER_INFO_NODE = "/leader-info";
int nodeNum = 10;
AtomicLong idGenerator = new AtomicLong();
AtomicInteger activeNodeCount = new AtomicInteger();
while (true) {
if (activeNodeCount.get() >= nodeNum) {
sleep(10);
continue;
} // 线程启动需要一定时间,将线程启动看做开机过程,在开机之前就算一台新的机器加入了
activeNodeCount.incrementAndGet();
new Thread(() -> {
try {
Node node = new Node(LEADER_INFO_NODE);
while (true) {
sleep(1000);
// 这里为了试验就让leader有轻微自杀倾向...
if (node.getStatus() == Node.Status.LEADER && Math.random() < 0.3) {
String logMsg = "----------------------------- " + Thread.currentThread().getName() + " shutdown -----------------------------";
System.out.println(logMsg);
node.shutdown();
break;
}
}
} catch (IOException e) {
e.printStackTrace();
} finally {
activeNodeCount.decrementAndGet();
}
}, "node-" + idGenerator.getAndIncrement()).start();
}
} }
控制台输出:
...
node-4 is leader
node-3 is follower, master is node-4
node-0 is follower, master is node-4
node-9 is follower, master is node-4
node-7 is follower, master is node-4
node-5 is follower, master is node-4
node-1 is follower, master is node-4
node-6 is follower, master is node-4
node-8 is follower, master is node-4
node-2 is follower, master is node-4
----------------------------- node-4 shutdown -----------------------------
node-0-EventThread is leader
node-6-EventThread is follower, master is node-0-EventThread
node-3-EventThread is follower, master is node-0-EventThread
node-7-EventThread is follower, master is node-0-EventThread
node-1-EventThread is follower, master is node-0-EventThread
node-5-EventThread is follower, master is node-0-EventThread
node-9-EventThread is follower, master is node-0-EventThread
node-2-EventThread is follower, master is node-0-EventThread
node-8-EventThread is follower, master is node-0-EventThread
node-10 is follower, master is node-0-EventThread
----------------------------- node-0 shutdown -----------------------------
node-6-EventThread is leader
node-7-EventThread is follower, master is node-6-EventThread
node-1-EventThread is follower, master is node-6-EventThread
node-3-EventThread is follower, master is node-6-EventThread
node-10-EventThread is follower, master is node-6-EventThread
node-9-EventThread is follower, master is node-6-EventThread
node-5-EventThread is follower, master is node-6-EventThread
node-2-EventThread is follower, master is node-6-EventThread
node-8-EventThread is follower, master is node-6-EventThread
node-11 is follower, master is node-6-EventThread
----------------------------- node-6 shutdown -----------------------------
node-1-EventThread is leader
node-10-EventThread is follower, master is node-1-EventThread
node-7-EventThread is follower, master is node-1-EventThread
node-11-EventThread is follower, master is node-1-EventThread
node-8-EventThread is follower, master is node-1-EventThread
node-5-EventThread is follower, master is node-1-EventThread
node-9-EventThread is follower, master is node-1-EventThread
node-3-EventThread is follower, master is node-1-EventThread
node-2-EventThread is follower, master is node-1-EventThread
node-12 is follower, master is node-1-EventThread
----------------------------- node-1 shutdown -----------------------------
node-3-EventThread is leader
node-12-EventThread is follower, master is node-3-EventThread
node-11-EventThread is follower, master is node-3-EventThread
node-5-EventThread is follower, master is node-3-EventThread
node-7-EventThread is follower, master is node-3-EventThread
node-9-EventThread is follower, master is node-3-EventThread
node-2-EventThread is follower, master is node-3-EventThread
node-10-EventThread is follower, master is node-3-EventThread
node-8-EventThread is follower, master is node-3-EventThread
node-13 is follower, master is node-3-EventThread
----------------------------- node-3 shutdown -----------------------------
node-5-EventThread is leader
node-13-EventThread is follower, master is node-5-EventThread
node-12-EventThread is follower, master is node-5-EventThread
node-7-EventThread is follower, master is node-5-EventThread
node-11-EventThread is follower, master is node-5-EventThread
node-10-EventThread is follower, master is node-5-EventThread
node-9-EventThread is follower, master is node-5-EventThread
node-2-EventThread is follower, master is node-5-EventThread
node-8-EventThread is follower, master is node-5-EventThread
node-14 is follower, master is node-5-EventThread
----------------------------- node-5 shutdown -----------------------------
node-7-EventThread is leader
node-13-EventThread is follower, master is node-7-EventThread
node-12-EventThread is follower, master is node-7-EventThread
node-9-EventThread is follower, master is node-7-EventThread
node-11-EventThread is follower, master is node-7-EventThread
node-14-EventThread is follower, master is node-7-EventThread
node-10-EventThread is follower, master is node-7-EventThread
node-8-EventThread is follower, master is node-7-EventThread
node-2-EventThread is follower, master is node-7-EventThread
node-15 is follower, master is node-7-EventThread
----------------------------- node-7 shutdown -----------------------------
node-14-EventThread is leader
node-13-EventThread is follower, master is node-14-EventThread
node-11-EventThread is follower, master is node-14-EventThread
node-2-EventThread is follower, master is node-14-EventThread
node-12-EventThread is follower, master is node-14-EventThread
node-15-EventThread is follower, master is node-14-EventThread
node-10-EventThread is follower, master is node-14-EventThread
node-9-EventThread is follower, master is node-14-EventThread
node-8-EventThread is follower, master is node-14-EventThread
node-16 is follower, master is node-14-EventThread
----------------------------- node-14 shutdown -----------------------------
node-13-EventThread is leader
node-12-EventThread is follower, master is node-13-EventThread
node-15-EventThread is follower, master is node-13-EventThread
node-9-EventThread is follower, master is node-13-EventThread
node-10-EventThread is follower, master is node-13-EventThread
node-2-EventThread is follower, master is node-13-EventThread
node-8-EventThread is follower, master is node-13-EventThread
node-11-EventThread is follower, master is node-13-EventThread
node-16-EventThread is follower, master is node-13-EventThread
node-17 is follower, master is node-13-EventThread
...
最后有个需要注意的地方就是是否需要将leader节点设置为顺序临时节点呢?类似于分布式锁那样,这样的话每次唤醒一个节点就可以了,这看上去像是一个可以优化的点。
其实并不是,当leader挂掉的时候必须所有follower都能够感知到leader的变更,即使他们不参与抢主也必须醒来执行leader变更的逻辑。
.
Zookeeper笔记之使用zk实现集群选主的更多相关文章
- ZooKeeper实践:(2)集群管理
前言: 随着业务的扩大,用户的增多,访问量的增加,单机模式已经不能支撑,从而出现了从单机模式->垂直应用模式->集群模式,集群模式诞生了,伴随着一堆问题也油然而生,Master怎么选举,机 ...
- zookeeper的单实例和伪集群部署
原文链接: http://gudaoyufu.com/?p=1395 zookeeper工作方式 ZooKeeper 是一个开源的分布式协调服务,由雅虎创建,是 Google Chubby 的开源实现 ...
- ZooKeeper实践:(1)集群管理
前言: 随着业务的扩大,用户的增多,访问量的增加,单机模式已经不能支撑,从而出现了从单机模式->垂直应用模式->集群模式,集群模式诞生了,伴随着一堆问题也油然而生,Master怎么选举,机 ...
- zookeeper在windows下的伪集群模式
参考:zookeeper在windows下的伪集群模式 踩到的坑: 注意windows下路径需要使用\ dataDir=D:\Program Files\Java\zookeeper-3.4.10-c ...
- zk伪集群部署
jdk 配置 # tar xf jdk-8u161-linux-x64.tar.gz -C /usr/local/ # vim /etc/profile.d/jdk.sh export JAVA_HO ...
- Zookeeper节点增删改查与集群搭建(笔记)
1.上传文件目录说明 上传的文件一般放在 /home/下 安装文件一般在 /usr/local/下 2. 安装zookeeper 2.1将zookeeper-3.4.11.tar.gz拷贝到/home ...
- Zookeeper的基本原理(zk架构、zk存储结构、watch机制、独立安装zk、集群间同步复制)
1.Hbase集群的高可用性与伸缩性 HBase可以实现对Regionserver的监控,当个别Regionserver不可访问时,将其负责的分区分给其他Regionsever,其转移过程较快,因为只 ...
- 基于zookeeper的高可用Hadoop HA集群安装
(1)hadoop2.7.1源码编译 http://aperise.iteye.com/blog/2246856 (2)hadoop2.7.1安装准备 http://aperise.iteye.com ...
- zookeeper笔记之基于zk实现分布式锁
一.分布式锁概述 Java中基于AQS框架提供了一系列的锁,但是当需要在集群中的多台机器上互斥执行一段代码或使用资源时Java提供的这种单机锁就没了用武之地,此时需要使用分布式锁协调它们.分布式锁有很 ...
随机推荐
- 第二十次ScrumMeeting博客
第二十次ScrumMeeting博客 本次会议于12月11日(一)22时整在3公寓725房间召开,持续20分钟. 与会人员:刘畅.辛德泰.张安澜.赵奕.方科栋. 1. 每个人的工作(有Issue的内容 ...
- PAT甲题题解-1106. Lowest Price in Supply Chain (25)-(dfs计算树的最小层数)
统计树的最小层数以及位于该层数上的叶子节点个数即可. 代码里建树我用了邻接链表的存储方式——链式前向星,不了解的可以参考,非常好用: http://www.cnblogs.com/chenxiwenr ...
- 第二个Sprint ------第四、五、六、七天
27号.28号.29号有事回家,没能及时更新博客. 罗伟业:加减算术----乘除算式 返回一个完整正确(加减.乘除)的算式<考虑到低年级还没有完全掌握四则混合运算> 康取:四则混合运算 ...
- Ns3 构建哑铃型拓扑,并实现两个点的TCP连接(详细请戳全文)
aaarticlea/png;base64,iVBORw0KGgoAAAANSUhEUgAAAaUAAADlCAIAAAB8n38hAAAV8UlEQVR4nO3dMWjjyBoH8CkNj1uc5v
- Python的三种格式化输出
今天刚学了python的三种格式化输出,以前没接触过这么有趣的输出方式,现在来分享一下. #!/user/bin/env python#coding:utf-8#三种格式化输出 #第一种格式化输出na ...
- ElasticSearch 2 (29) - 信息聚合系列之测试驱动
ElasticSearch 2 (29) - 信息聚合系列之测试驱动 摘要 我们可以用以下几页定义不同的聚合和它们的语法,但学习聚合的最佳途径就是用实例来说明.一旦我们获得了聚合的思想,以及如何合理地 ...
- 使用redis防止抢购商品超卖
前言: redis不仅仅是单纯的缓存,它还有一些特殊的功能,在一些特殊场景上很好用. 本篇博文用来测试下使用redis来防止抢购商品超卖问题. 内容: 使用redis的list进行测试 思路是设置一个 ...
- Study From DevOps 学习交流会议
1.今天下午参加了 软件集团的 爱城市的devops的沟通会议,了解到他们是通过bash的方式来执行jenkins的build以及创建 jenkins的 project 等内容.晚上回来简单实验了下. ...
- javaee, javaweb和javase的区别以及各自的知识体系
javaee, javaweb和javase的区别以及各自的知识体系 来源 https://blog.csdn.net/weixin_39297312/article/details/79454642 ...
- CF888G Xor-MST 解题报告
CF888G Xor-MST 题意翻译 给定一个\(n\)个节点的完全图,每个节点有个编号\(a_i\),节点\(i\)和节点\(j\)之间边的权值为\(a_i\ xor\ a_j\),求该图的最小生 ...