一、需求

在主从结构的集群中，我们假设硬件机器是很脆弱的，随时可能会宕机，当master挂掉之后需要从slave中选出一个节点作为新的master，使用zookeeper可以很简单的实现集群选主功能。

二、分析

下面为了方便叙述，将使用更通用的技术术语，即使用leader表示master，使用follower表示slave。

集群选主涉及到两个问题：

1. 谁来做leader

2. leader挂掉了怎么被follower感知到

首先是第一个问题，谁来做leader，其实可以将这个问题看做是多线程中的互斥锁抢占，锁只有一把，并且只能被一个人抢到，这里就把一个zookeeper上的一个节点/leader-info看做是锁，集群中的每台机器都尝试去创建这个节点（抢占锁），因为zookeeper创建节点是原子性操作，所以只有一台机器能够创建成功其它都会失败，创建成功的那台机器就作为leader，其它机器做follower，一般leader抢占成功了之后会在/leader-info节点上存储一些与自己相关的信息，比如hostname、id之类的，以让follower知道谁抢占成功成为了leader，然后去连接leader进行一些数据交换或指令控制之类的，那就是选主之后的事了不在此篇文章的讨论范围之内。

第二个问题是leader挂掉了怎么通知其它的follower，zookeeper中的节点按照有效时间分为持久节点和临时节点，临时节点跟session绑定，而一个session表示一个客户端，当客户端下线的时候session失效，当session失效的时候跟它绑定的临时节点就会被删除，利用这个特性可以检测节点是否还在存活状态，实现follower对leader下线的感知，只是需要注意在创建/leader-info节点的时候要将其创建为临时节点，然后众多follower都在这个节点上添加一个watcher监听其删除事件，这样当leader挂掉的时候session失效，然后与此session绑定的临时节点会被删除，即/leader-info节点将被删除，同时给所有的follower发送事件通知，follower一看leader挂了就燥起来了，将自己的状态置为looking，开始新一轮的选举。

总结一下选主的流程：

1. 集群中的所有机器将自己置为looking状态，准备开始选举。

2. 所有looking状态的机器尝试去创建/leader-info节点。

3. 创建成功的将自己的状态修改为leader，同时将自己的一些信息写入到/leader-info这个节点上。

4. 创建失败的将自己的状态置为follower，同时尝试从/leader-info获取leader信息进行一些leader改变的逻辑（在这里这个不是重点，打印一下即可），follower在获取/leader-info节点数据的同时要设置一个watcher，监听此节点的删除事件，当节点被删除事件触发时启动新一轮的选举，因为获取数据设置watcher这个操作是原子性的，所以要么这个节点存在获取数据成功，并且设置watcher也成功，要么节点不存在抛出KeeperException.NoNodeException异常。

5. 为什么在follower设置watcher的时候还有可能会抛异常呢，leader不是已经创建了这个节点了吗？因为follower从尝试创建/leader-info节点失败到去获取此节点的数据同时设置watcher这一段操作不是原子性的，在这中间可能会发生一些变故，leader可能刚成为leader就挂掉了（或者因为一些网络抖动原因，总之是session失效了），leader挂掉之后它创建的临时节点就被zookeeper删除了，所以当follower在设置watcher的时候如果检测到KeeperException.NoNodeException，说明之前的leader挂掉了，此时集群中已经没有了leader，follower又燥起来了，它将自己的状态置为looking开始新一轮的选举。

三、实现

Node.java：

package cc11001100.zookeeper.leaderElection;

import cc11001100.zookeeper.utils.ZooKeeperUtil;

import org.apache.zookeeper.CreateMode;

import org.apache.zookeeper.KeeperException;

import org.apache.zookeeper.Watcher;

import org.apache.zookeeper.ZooDefs;

import org.apache.zookeeper.ZooKeeper;

import java.io.IOException;

import java.io.UnsupportedEncodingException;

/**

 * 表示集群中的一个节点

 *

 * @author CC11001100

 */

public class Node {

	private Status status;

	private String nodeForLeaderInfo;

	private ZooKeeper zooKeeper;

	public Node(String listenerNodeForLeader) throws IOException {

		this.nodeForLeaderInfo = listenerNodeForLeader;

		this.zooKeeper = ZooKeeperUtil.getZooKeeper();

		lookingForLeader();

	}

	public void lookingForLeader() {

		status = Status.LOOKING;

		try {

			String leaderInfo = Thread.currentThread().getName();

			// 需要注意这里创建的是临时节点

			zooKeeper.create(nodeForLeaderInfo, leaderInfo.getBytes(), ZooDefs.Ids.OPEN_ACL_UNSAFE, CreateMode.EPHEMERAL);

			// 如果上一步没有抛异常，说明自己已经是leader了

			status = Status.LEADER;

			String logMsg = Thread.currentThread().getName() + " is leader";

			System.out.println(logMsg);

		} catch (KeeperException.NodeExistsException e) {

			// 节点已经存在，说明leader已经被别人注册成功了，自己是follower

			status = Status.FOLLOWER;

			try {

				byte[] leaderInfoBytes = zooKeeper.getData(nodeForLeaderInfo, event -> {

					if (event.getType() == Watcher.Event.EventType.NodeDeleted) {

						lookingForLeader();

					}

				}, null);

				String logMsg = Thread.currentThread().getName() + " is follower, master is " + new String(leaderInfoBytes, "UTF-8");

				System.out.println(logMsg);

			} catch (KeeperException.NoNodeException e1) {

				// 如果在获取leader信息的时候报了节点不存在，说明这个leader比较短命，刚抢到leader就又挂掉了

				lookingForLeader();

			} catch (KeeperException | InterruptedException | UnsupportedEncodingException e1) {

				e1.printStackTrace();

			}

		} catch (KeeperException | InterruptedException e) {

			e.printStackTrace();

		}

	}

	public void shutdown() {

		try {

			if (zooKeeper != null) {

				zooKeeper.close();

			}

		} catch (InterruptedException e) {

			e.printStackTrace();

		}

	}

	public Status getStatus() {

		return status;

	}

	// 当前节点的状态，节点的状态必须在这三个中的一个

	public enum Status {

		LOOKING, // 选举中

		LEADER, // 选举完毕，当前节点为leader

		FOLLOWER; // 选举完毕，当前节点为follower

	}

}

LeaderElectionTest.java：

package cc11001100.zookeeper.leaderElection;

import java.io.IOException;

import java.util.concurrent.TimeUnit;

import java.util.concurrent.atomic.AtomicInteger;

import java.util.concurrent.atomic.AtomicLong;

/**

 * @author CC11001100

 */

public class LeaderElectionTest {

	private static void sleep(long mils) {

		try {

			TimeUnit.MILLISECONDS.sleep(mils);

		} catch (InterruptedException e) {

			e.printStackTrace();

		}

	}

	public static void main(String[] args) throws IOException {

		final String LEADER_INFO_NODE = "/leader-info";

		int nodeNum = 10;

		AtomicLong idGenerator = new AtomicLong();

		AtomicInteger activeNodeCount = new AtomicInteger();

		while (true) {

			if (activeNodeCount.get() >= nodeNum) {

				sleep(10);

				continue;

			}

			// 线程启动需要一定时间，将线程启动看做开机过程，在开机之前就算一台新的机器加入了

			activeNodeCount.incrementAndGet();

			new Thread(() -> {

				try {

					Node node = new Node(LEADER_INFO_NODE);

					while (true) {

						sleep(1000);

						// 这里为了试验就让leader有轻微自杀倾向...

						if (node.getStatus() == Node.Status.LEADER && Math.random() < 0.3) {

							String logMsg = "----------------------------- " + Thread.currentThread().getName() + " shutdown -----------------------------";

							System.out.println(logMsg);

							node.shutdown();

							break;

						}

					}

				} catch (IOException e) {

					e.printStackTrace();

				} finally {

					activeNodeCount.decrementAndGet();

				}

			}, "node-" + idGenerator.getAndIncrement()).start();

		}

	}

}

控制台输出：

...

node-4 is leader

node-3 is follower, master is node-4

node-0 is follower, master is node-4

node-9 is follower, master is node-4

node-7 is follower, master is node-4

node-5 is follower, master is node-4

node-1 is follower, master is node-4

node-6 is follower, master is node-4

node-8 is follower, master is node-4

node-2 is follower, master is node-4

----------------------------- node-4 shutdown -----------------------------

node-0-EventThread is leader

node-6-EventThread is follower, master is node-0-EventThread

node-3-EventThread is follower, master is node-0-EventThread

node-7-EventThread is follower, master is node-0-EventThread

node-1-EventThread is follower, master is node-0-EventThread

node-5-EventThread is follower, master is node-0-EventThread

node-9-EventThread is follower, master is node-0-EventThread

node-2-EventThread is follower, master is node-0-EventThread

node-8-EventThread is follower, master is node-0-EventThread

node-10 is follower, master is node-0-EventThread

----------------------------- node-0 shutdown -----------------------------

node-6-EventThread is leader

node-7-EventThread is follower, master is node-6-EventThread

node-1-EventThread is follower, master is node-6-EventThread

node-3-EventThread is follower, master is node-6-EventThread

node-10-EventThread is follower, master is node-6-EventThread

node-9-EventThread is follower, master is node-6-EventThread

node-5-EventThread is follower, master is node-6-EventThread

node-2-EventThread is follower, master is node-6-EventThread

node-8-EventThread is follower, master is node-6-EventThread

node-11 is follower, master is node-6-EventThread

----------------------------- node-6 shutdown -----------------------------

node-1-EventThread is leader

node-10-EventThread is follower, master is node-1-EventThread

node-7-EventThread is follower, master is node-1-EventThread

node-11-EventThread is follower, master is node-1-EventThread

node-8-EventThread is follower, master is node-1-EventThread

node-5-EventThread is follower, master is node-1-EventThread

node-9-EventThread is follower, master is node-1-EventThread

node-3-EventThread is follower, master is node-1-EventThread

node-2-EventThread is follower, master is node-1-EventThread

node-12 is follower, master is node-1-EventThread

----------------------------- node-1 shutdown -----------------------------

node-3-EventThread is leader

node-12-EventThread is follower, master is node-3-EventThread

node-11-EventThread is follower, master is node-3-EventThread

node-5-EventThread is follower, master is node-3-EventThread

node-7-EventThread is follower, master is node-3-EventThread

node-9-EventThread is follower, master is node-3-EventThread

node-2-EventThread is follower, master is node-3-EventThread

node-10-EventThread is follower, master is node-3-EventThread

node-8-EventThread is follower, master is node-3-EventThread

node-13 is follower, master is node-3-EventThread

----------------------------- node-3 shutdown -----------------------------

node-5-EventThread is leader

node-13-EventThread is follower, master is node-5-EventThread

node-12-EventThread is follower, master is node-5-EventThread

node-7-EventThread is follower, master is node-5-EventThread

node-11-EventThread is follower, master is node-5-EventThread

node-10-EventThread is follower, master is node-5-EventThread

node-9-EventThread is follower, master is node-5-EventThread

node-2-EventThread is follower, master is node-5-EventThread

node-8-EventThread is follower, master is node-5-EventThread

node-14 is follower, master is node-5-EventThread

----------------------------- node-5 shutdown -----------------------------

node-7-EventThread is leader

node-13-EventThread is follower, master is node-7-EventThread

node-12-EventThread is follower, master is node-7-EventThread

node-9-EventThread is follower, master is node-7-EventThread

node-11-EventThread is follower, master is node-7-EventThread

node-14-EventThread is follower, master is node-7-EventThread

node-10-EventThread is follower, master is node-7-EventThread

node-8-EventThread is follower, master is node-7-EventThread

node-2-EventThread is follower, master is node-7-EventThread

node-15 is follower, master is node-7-EventThread

----------------------------- node-7 shutdown -----------------------------

node-14-EventThread is leader

node-13-EventThread is follower, master is node-14-EventThread

node-11-EventThread is follower, master is node-14-EventThread

node-2-EventThread is follower, master is node-14-EventThread

node-12-EventThread is follower, master is node-14-EventThread

node-15-EventThread is follower, master is node-14-EventThread

node-10-EventThread is follower, master is node-14-EventThread

node-9-EventThread is follower, master is node-14-EventThread

node-8-EventThread is follower, master is node-14-EventThread

node-16 is follower, master is node-14-EventThread

----------------------------- node-14 shutdown -----------------------------

node-13-EventThread is leader

node-12-EventThread is follower, master is node-13-EventThread

node-15-EventThread is follower, master is node-13-EventThread

node-9-EventThread is follower, master is node-13-EventThread

node-10-EventThread is follower, master is node-13-EventThread

node-2-EventThread is follower, master is node-13-EventThread

node-8-EventThread is follower, master is node-13-EventThread

node-11-EventThread is follower, master is node-13-EventThread

node-16-EventThread is follower, master is node-13-EventThread

node-17 is follower, master is node-13-EventThread

...

最后有个需要注意的地方就是是否需要将leader节点设置为顺序临时节点呢？类似于分布式锁那样，这样的话每次唤醒一个节点就可以了，这看上去像是一个可以优化的点。

其实并不是，当leader挂掉的时候必须所有follower都能够感知到leader的变更，即使他们不参与抢主也必须醒来执行leader变更的逻辑。

.

Zookeeper笔记之使用zk实现集群选主的更多相关文章

  1. ZooKeeper实践：（2）集群管理

    前言: 随着业务的扩大,用户的增多,访问量的增加,单机模式已经不能支撑,从而出现了从单机模式->垂直应用模式->集群模式,集群模式诞生了,伴随着一堆问题也油然而生,Master怎么选举,机 ...

  2. zookeeper的单实例和伪集群部署

    原文链接: http://gudaoyufu.com/?p=1395 zookeeper工作方式 ZooKeeper 是一个开源的分布式协调服务,由雅虎创建,是 Google Chubby 的开源实现 ...

  3. ZooKeeper实践：（1）集群管理

    前言: 随着业务的扩大,用户的增多,访问量的增加,单机模式已经不能支撑,从而出现了从单机模式->垂直应用模式->集群模式,集群模式诞生了,伴随着一堆问题也油然而生,Master怎么选举,机 ...

  4. zookeeper在windows下的伪集群模式

    参考:zookeeper在windows下的伪集群模式 踩到的坑: 注意windows下路径需要使用\ dataDir=D:\Program Files\Java\zookeeper-3.4.10-c ...

  5. zk伪集群部署

    jdk 配置 # tar xf jdk-8u161-linux-x64.tar.gz -C /usr/local/ # vim /etc/profile.d/jdk.sh export JAVA_HO ...

  6. Zookeeper节点增删改查与集群搭建（笔记）

    1.上传文件目录说明 上传的文件一般放在 /home/下 安装文件一般在 /usr/local/下 2. 安装zookeeper 2.1将zookeeper-3.4.11.tar.gz拷贝到/home ...

  7. Zookeeper的基本原理(zk架构、zk存储结构、watch机制、独立安装zk、集群间同步复制)

    1.Hbase集群的高可用性与伸缩性 HBase可以实现对Regionserver的监控,当个别Regionserver不可访问时,将其负责的分区分给其他Regionsever,其转移过程较快,因为只 ...

  8. 基于zookeeper的高可用Hadoop HA集群安装

    (1)hadoop2.7.1源码编译 http://aperise.iteye.com/blog/2246856 (2)hadoop2.7.1安装准备 http://aperise.iteye.com ...

  9. zookeeper笔记之基于zk实现分布式锁

    一.分布式锁概述 Java中基于AQS框架提供了一系列的锁,但是当需要在集群中的多台机器上互斥执行一段代码或使用资源时Java提供的这种单机锁就没了用武之地,此时需要使用分布式锁协调它们.分布式锁有很 ...

随机推荐

  1. Hyperledger Fabric网络节点架构

    Fabric区块链网络的组成  区块链网络结构图 区块链网络组成 组成区块链网络相关的节点 节点是区块链的通信主体,和区块链网络相关的节点有多种类型:客户端(应用).Peer节点.排序服务(Orde ...

  2. Final阶段基于spec评论作品

    组名:杨老师粉丝群 组长:乔静玉 组员:吴奕瑶  刘佳瑞  公冶令鑫  杨磊  刘欣  张宇  卢帝同 一.测试目标:拉格朗日2018——飞词 1.开始界面,只有一个开始游戏按钮,简约且醒目     ...

  3. 【Beta阶段】第五次Scrum Meeting！

    每日任务内容: 本次会议为第五次Scrum Meeting会议~ 由于本次会议项目经理召开时间依旧较晚,在公寓7层召开,女生参与了线上会议. 队员 昨日完成任务 明日要完成任务 刘乾 #167(未完成 ...

  4. 作业一_随笔1_初来乍到：学号&博客地址

    031302540——http://www.cnblogs.com/yyj031302540/ 计算机实验班叶艺洁

  5. DirectoryEntry_Properties属性的遍历(win2008)

    DirectoryEntry root = new DirectoryEntry(@"IIS://localhost/W3SVC"); string PInfo = "& ...

  6. # 蜗牛慢慢爬 LeetCode 21. Merge Two Sorted Lists [Difficulty: Easy]

    题目 Merge two sorted linked lists and return it as a new list. The new list should be made by splicin ...

  7. ThreadLocal 与 Synchronized区别

    相同:ThreadLocal和线程同步机制都是为了解决多线程中相同变量的访问冲突问题.不同:Synchronized同步机制采用了“以时间换空间”的方式,仅提供一份变量,让不同的线程排队访问:而Thr ...

  8. GS 服务器超时时间设置

    工作中 遇到一个超时的问题 与徐庆同学沟通后 了解了下超时时间设置的地方 1.web.congfig问题: 常规路径 C:\Program Files\GenerSoft\bscw_local\web ...

  9. Java之流的分类

    Java I/O流分类:

  10. nowcoder106I Neat Tree (单调栈)

    Richard神犇出给nowcoder的题 用单调栈找到每个点它向右和向左的第一个大于或小于它的位置,然后它作为最大值/最小值的区间就要在这个范围里,那么它的贡献就是这个区间长度乘一乘再减一减 注意一 ...