原创技术文章，转载请注明：转自http://newliferen.github.io/

如何连接ZooKeeper集群

  要想了解ZooKeeper客户端实现原理，首先需要关注一下客户端的使用方式，会使用之后，方便更进一步的了解zk的源码实现。

客户端程序连接ZooKeeper集群代码

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
 
public void () {
    try {
        final CountDownLatch semaphore = new CountDownLatch(1);
        ZooKeeper zkConn = new ZooKeeper(host, timeout, new Watcher() {

            @Override
            public void process(WatchedEvent watchedEvent) {
                if (Event.KeeperState.SyncConnected == watchedEvent.getState()) {
                    if (Event.EventType.None == watchedEvent.getType() && watchedEvent.getPath() == null) {
                        System.out.println("ZK connect success!");
                        semaphore.countDown();
                    }
                }
            }
        });
        semaphore.await();
    } catch (IOException e) {
        e.printStackTrace();
    } catch (InterruptedException e) {
        e.printStackTrace();
    }
}

客户端线程模型

ZooKeeper实现原理

  接下来我们来分析zk客户端的源码实现。ZooKeeper为线程安全类，用户可以放心使用。客户端类库中最主要的三个类：ClientCnxn、SendThread、EventThread，后两者做为前者的内部类存在，当客户端创建ZooKeeper对象时，在ZooKeeper类的构造函数内解析客户端传入的server地址、实例化ClientCnxn对象，并通过ClientCnxn对象的start()方法启动SendThread和EventThread两个线程。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
 
public ZooKeeper(String connectString, int sessionTimeout, Watcher watcher,
           boolean canBeReadOnly)
       throws IOException
   {
       LOG.info("Initiating client connection, connectString=" + connectString
               + " sessionTimeout=" + sessionTimeout + " watcher=" + watcher);

       watchManager.defaultWatcher = watcher;
	// connectString形如：192.168.155.144:2181,192.168.155.155:2181
       ConnectStringParser connectStringParser = new ConnectStringParser(
               connectString);
       // 持有所有server连接对象
       HostProvider hostProvider = new StaticHostProvider(
               connectStringParser.getServerAddresses());
	// 实例化ClientCnxn对象
       cnxn = new ClientCnxn(connectStringParser.getChrootPath(),
               hostProvider, sessionTimeout, this, watchManager,
               getClientCnxnSocket(), canBeReadOnly);
	// 启动客户端线程
       cnxn.start();
   }
 
1
2
3
4
 
public void start() {
       sendThread.start();
       eventThread.start();
}

  其中，SendThread线程负责从与server端建立连接、定时发送心跳消息、从outgoingQueue队列中取得任务，并通过tcp连接将请求发送到server端。SendThread类收到server端响应事件后，向waitingEvents队列中addwatcher对象，由EventThread从waitingEvents队列中获取事件，判断事件类型，回调客户端的Watcher实现类。一个完成的客户端流程就完成了。创建连接时的Watcher实现类将作为zk的默认watcher类。

StaticHostProvider连接持有者

StaticHostProvider类的功能非常简单

  1. 该类持有解析之后的zk server所有连接对象。
  2. 采用轮询算法作为负载均衡策略，从StaticHostProvider的所有连接对象中获取一个连接，这个连提供给ClientCnxn对象使用。
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
 
 public StaticHostProvider(Collection<InetSocketAddress> serverAddresses)
        throws UnknownHostException {
    for (InetSocketAddress address : serverAddresses) {
        InetAddress ia = address.getAddress();
        InetAddress resolvedAddresses[] = InetAddress.getAllByName((ia!=null) ? ia.getHostAddress():
            address.getHostName());
        for (InetAddress resolvedAddress : resolvedAddresses) {
            if (resolvedAddress.toString().startsWith("/") 
                    && resolvedAddress.getAddress() != null) {
                this.serverAddresses.add(
                        new InetSocketAddress(InetAddress.getByAddress(
                                address.getHostName(),
                                resolvedAddress.getAddress()), 
                                address.getPort()));
            } else {
                this.serverAddresses.add(new InetSocketAddress(resolvedAddress.getHostAddress(), address.getPort()));
            }  
        }
    }

    if (this.serverAddresses.isEmpty()) {
        throw new IllegalArgumentException(
                "A HostProvider may not be empty!");
    }
    Collections.shuffle(this.serverAddresses);
}
 
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
 
public InetSocketAddress next(long spinDelay) {
        ++currentIndex;
        if (currentIndex == serverAddresses.size()) {
            currentIndex = 0;
        }
        if (currentIndex == lastIndex && spinDelay > 0) {
            try {
                Thread.sleep(spinDelay);
            } catch (InterruptedException e) {
                LOG.warn("Unexpected exception", e);
            }
        } else if (lastIndex == -1) {
            // We don't want to sleep on the first ever connect attempt.
            lastIndex = 0;
        }

        return serverAddresses.get(currentIndex);
    }

SendThread

SendThread类主要实现的功能有三个：

  • 建立到zk server端的tcp连接
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
 
if (!clientCnxnSocket.isConnected()) {
    if(!isFirstConnect){
        try {
            Thread.sleep(r.nextInt(1000));
        } catch (InterruptedException e) {
            LOG.warn("Unexpected exception", e);
        }
    }
    // don't re-establish connection if we are closing
    if (closing || !state.isAlive()) {
        break;
    }
    startConnect();
    clientCnxnSocket.updateLastSendAndHeard();
}
  • 定时向zk server端发送心跳信息
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
 
if (state.isConnected()) {
	//1000(1 second) is to prevent race condition missing to send the second ping
	//also make sure not to send too many pings when readTimeout is small 
    int timeToNextPing = readTimeout / 2 - clientCnxnSocket.getIdleSend() - 
    		((clientCnxnSocket.getIdleSend() > 1000) ? 1000 : 0);
    //send a ping request either time is due or no packet sent out within MAX_SEND_PING_INTERVAL
    if (timeToNextPing <= 0 || clientCnxnSocket.getIdleSend() > MAX_SEND_PING_INTERVAL) {
        sendPing();
        clientCnxnSocket.updateLastSend();
    } else {
        if (timeToNextPing < to) {
            to = timeToNextPing;
        }
    }
}

// If we are in read-only mode, seek for read/write server
if (state == States.CONNECTEDREADONLY) {
    long now = System.currentTimeMillis();
    int idlePingRwServer = (int) (now - lastPingRwServer);
    if (idlePingRwServer >= pingRwTimeout) {
        lastPingRwServer = now;
        idlePingRwServer = 0;
        pingRwTimeout =
            Math.min(2*pingRwTimeout, maxPingRwTimeout);
        pingRwServer();
    }
    to = Math.min(to, pingRwTimeout - idlePingRwServer);
}
  • 将任务队列中的任务发送到zk server端
1
 
clientCnxnSocket.doTransport(to, pendingQueue, outgoingQueue, ClientCnxn.this);
  • 接收从zk server端返回的响应，并将响应事件加入到EventThread线程的任务队列中，并记录事务id
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    98
    99
    100
    101
    102
    103
    104
    105
    106
    107
    108
    109
    110
    111
    112
    113
    114
    115
    116
    117
    118
    		 
    void readResponse(ByteBuffer incomingBuffer) throws IOException {
        ByteBufferInputStream bbis = new ByteBufferInputStream(
                incomingBuffer);
        BinaryInputArchive bbia = BinaryInputArchive.getArchive(bbis);
        ReplyHeader replyHdr = n 大专栏  ZooKeeper源码阅读——client（二）ew ReplyHeader();

    replyHdr.deserialize(bbia, "header");
        if (replyHdr.getXid() == -2) {
            // -2 is the xid for pings
            if (LOG.isDebugEnabled()) {
                LOG.debug("Got ping response for sessionid: 0x"
                        + Long.toHexString(sessionId)
                        + " after "
                        + ((System.nanoTime() - lastPingSentNs) / 1000000)
                        + "ms");
            }
            return;
        }
        if (replyHdr.getXid() == -4) {
            // -4 is the xid for AuthPacket               
            if(replyHdr.getErr() == KeeperException.Code.AUTHFAILED.intValue()) {
                state = States.AUTH_FAILED;                    
                eventThread.queueEvent( new WatchedEvent(Watcher.Event.EventType.None, 
                        Watcher.Event.KeeperState.AuthFailed, null) );            		            		
            }
            if (LOG.isDebugEnabled()) {
                LOG.debug("Got auth sessionid:0x"
                        + Long.toHexString(sessionId));
            }
            return;
        }
        if (replyHdr.getXid() == -1) {
            // -1 means notification
            if (LOG.isDebugEnabled()) {
                LOG.debug("Got notification sessionid:0x"
                    + Long.toHexString(sessionId));
            }
            WatcherEvent event = new WatcherEvent();
            event.deserialize(bbia, "response");

        // convert from a server path to a client path
            if (chrootPath != null) {
                String serverPath = event.getPath();
                if(serverPath.compareTo(chrootPath)==0)
                    event.setPath("/");
                else if (serverPath.length() > chrootPath.length())
                    event.setPath(serverPath.substring(chrootPath.length()));
                else {
                	LOG.warn("Got server path " + event.getPath()
                			+ " which is too short for chroot path "
                			+ chrootPath);
                }
            }

        WatchedEvent we = new WatchedEvent(event);
            if (LOG.isDebugEnabled()) {
                LOG.debug("Got " + we + " for sessionid 0x"
                        + Long.toHexString(sessionId));
            }

        eventThread.queueEvent( we );
            return;
        }

    // If SASL authentication is currently in progress, construct and
        // send a response packet immediately, rather than queuing a
        // response as with other packets.
        if (clientTunneledAuthenticationInProgress()) {
            GetSASLRequest request = new GetSASLRequest();
            request.deserialize(bbia,"token");
            zooKeeperSaslClient.respondToServer(request.getToken(),
              ClientCnxn.this);
            return;
        }

    Packet packet;
        synchronized (pendingQueue) {
            if (pendingQueue.size() == 0) {
                throw new IOException("Nothing in the queue, but got "
                        + replyHdr.getXid());
            }
            packet = pendingQueue.remove();
        }
        /*
         * Since requests are processed in order, we better get a response
         * to the first request!
         */
        try {
            if (packet.requestHeader.getXid() != replyHdr.getXid()) {
                packet.replyHeader.setErr(
                        KeeperException.Code.CONNECTIONLOSS.intValue());
                throw new IOException("Xid out of order. Got Xid "
                        + replyHdr.getXid() + " with err " +
                        + replyHdr.getErr() +
                        " expected Xid "
                        + packet.requestHeader.getXid()
                        + " for a packet with details: "
                        + packet );
            }

        packet.replyHeader.setXid(replyHdr.getXid());
            packet.replyHeader.setErr(replyHdr.getErr());
            packet.replyHeader.setZxid(replyHdr.getZxid());
            if (replyHdr.getZxid() > 0) {
                lastZxid = replyHdr.getZxid();
            }
            if (packet.response != null && replyHdr.getErr() == 0) {
                packet.response.deserialize(bbia, "response");
            }

        if (LOG.isDebugEnabled()) {
                LOG.debug("Reading reply sessionid:0x"
                        + Long.toHexString(sessionId) + ", packet:: " + packet);
            }
        } finally {
            finishPacket(packet);
        }
    }

EventThread响应事件处理类

EventThread类负责实例化Watcher回调。每一种操作都有对应的事件回调接口，zk客户端代码会分别进行处理。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
 
private void processEvent(Object event) {
	try {
	  if (event instanceof WatcherSetEventPair) {
	      // each watcher will process the event
	      WatcherSetEventPair pair = (WatcherSetEventPair) event;
	      for (Watcher watcher : pair.watchers) {
	          try {
	              watcher.process(pair.event);
	          } catch (Throwable t) {
	              LOG.error("Error while calling watcher ", t);
	          }
	      }
	  } else {
	      Packet p = (Packet) event;
	      int rc = 0;
	      String clientPath = p.clientPath;
	      if (p.replyHeader.getErr() != 0) {
	          rc = p.replyHeader.getErr();
	      }
	      if (p.cb == null) {
	          LOG.warn("Somehow a null cb got to EventThread!");
	      } else if (p.response instanceof ExistsResponse
	              || p.response instanceof SetDataResponse
	              || p.response instanceof SetACLResponse) {
	          StatCallback cb = (StatCallback) p.cb;
	          if (rc == 0) {
	              if (p.response instanceof ExistsResponse) {
	            	  // 调用client传入的callback实现类实现的processResult方法
	                  cb.processResult(rc, clientPath, p.ctx,
	                          ((ExistsResponse) p.response)
	                                  .getStat());
	              } else if (p.response instanceof SetDataResponse) {
	                  cb.processResult(rc, clientPath, p.ctx,
	                          ((SetDataResponse) p.response)
	                                  .getStat());
	              } else if (p.response instanceof SetACLResponse) {
	                  cb.processResult(rc, clientPath, p.ctx,
	                          ((SetACLResponse) p.response)
	                                  .getStat());
	              }
	          } else {
	              cb.processResult(rc, clientPath, p.ctx, null);
	          }
	      } else if (p.response instanceof GetDataResponse) {
	          DataCallback cb = (DataCallback) p.cb;
	          GetDataResponse rsp = (GetDataResponse) p.response;
	          if (rc == 0) {
	              cb.processResult(rc, clientPath, p.ctx, rsp
	                      .getData(), rsp.getStat());
	          } else {
	              cb.processResult(rc, clientPath, p.ctx, null,
	                      null);
	          }
	      } else if (p.response instanceof GetACLResponse) {
	          ACLCallback cb = (ACLCallback) p.cb;
	          GetACLResponse rsp = (GetACLResponse) p.response;
	          if (rc == 0) {
	              cb.processResult(rc, clientPath, p.ctx, rsp
	                      .getAcl(), rsp.getStat());
	          } else {
	              cb.processResult(rc, clientPath, p.ctx, null,
	                      null);
	          }
	      } else if (p.response instanceof GetChildrenResponse) {
	          ChildrenCallback cb = (ChildrenCallback) p.cb;
	          GetChildrenResponse rsp = (GetChildrenResponse) p.response;
	          if (rc == 0) {
	              cb.processResult(rc, clientPath, p.ctx, rsp
	                      .getChildren());
	          } else {
	              cb.processResult(rc, clientPath, p.ctx, null);
	          }
	      } else if (p.response instanceof GetChildren2Response) {
	          Children2Callback cb = (Children2Callback) p.cb;
	          GetChildren2Response rsp = (GetChildren2Response) p.response;
	          if (rc == 0) {
	              cb.processResult(rc, clientPath, p.ctx, rsp
	                      .getChildren(), rsp.getStat());
	          } else {
	              cb.processResult(rc, clientPath, p.ctx, null, null);
	          }
	      } else if (p.response instanceof CreateResponse) {
	          StringCallback cb = (StringCallback) p.cb;
	          CreateResponse rsp = (CreateResponse) p.response;
	          if (rc == 0) {
	              cb.processResult(rc, clientPath, p.ctx,
	                      (chrootPath == null
	                              ? rsp.getPath()
	                              : rsp.getPath()
	                        .substring(chrootPath.length())));
	          } else {
	              cb.processResult(rc, clientPath, p.ctx, null);
	          }
	      } else if (p.cb instanceof VoidCallback) {
	          VoidCallback cb = (VoidCallback) p.cb;
	          cb.processResult(rc, clientPath, p.ctx);
	      }
	  }
	} catch (Throwable t) {
	  LOG.error("Caught unexpected throwable", t);
	}
  }
}

ZooKeeper源码阅读——client（二）的更多相关文章

  1. ZooKeeper源码阅读（二）：客户端

    源代码: http://svn.apache.org/repos/asf/zookeeper/trunk/ 导入eclipse: 在包含build.xml目录下执行ant eclipse将产生.cla ...

  2. 【原】FMDB源码阅读（二）

    [原]FMDB源码阅读(二) 本文转载请注明出处 -- polobymulberry-博客园 1. 前言 上一篇只是简单地过了一下FMDB一个简单例子的基本流程,并没有涉及到FMDB的所有方方面面,比 ...

  3. 【原】AFNetworking源码阅读（二）

    [原]AFNetworking源码阅读(二) 本文转载请注明出处 —— polobymulberry-博客园 1. 前言 上一篇中我们在iOS Example代码中提到了AFHTTPSessionMa ...

  4. 【原】SDWebImage源码阅读（二）

    [原]SDWebImage源码阅读(二) 本文转载请注明出处 —— polobymulberry-博客园 1. 解决上一篇遗留的坑 上一篇中对sd_setImageWithURL函数简单分析了一下,还 ...

  5. 【详解】ThreadPoolExecutor源码阅读（二）

    系列目录 [详解]ThreadPoolExecutor源码阅读(一) [详解]ThreadPoolExecutor源码阅读(二) [详解]ThreadPoolExecutor源码阅读(三) AQS在W ...

  6. Redis源码阅读（二）高可用设计——复制

    Redis源码阅读(二)高可用设计-复制 复制的概念:Redis的复制简单理解就是一个Redis服务器从另一台Redis服务器复制所有的Redis数据库数据,能保持两台Redis服务器的数据库数据一致 ...

  7. Caddy源码阅读（二）启动流程与 Event 事件通知

    Caddy源码阅读(二)启动流程与 Event 事件通知 Preface Caddy 是 Go 语言构建的轻量配置化服务器.https://github.com/caddyserver/caddy C ...

  8. Rpc框架dubbo-client（v2.6.3） 源码阅读（二）

    接上一篇 dubbo-server 之后,再来看一下 dubbo-client 是如何工作的. dubbo提供者服务示例, 其结构是这样的!dubbo://192.168.11.6:20880/com ...

  9. 【 js 基础 】【 源码学习 】backbone 源码阅读（二）

    最近看完了 backbone.js 的源码,这里对于源码的细节就不再赘述了,大家可以 star 我的源码阅读项目(source-code-study)进行参考交流,有详细的源码注释,以及知识总结,同时 ...

随机推荐

  1. TechEmpower 框架性能测试数据 - 新解读

    1. TechEmpower Framework Benchmark 介绍 TechEmpower 框架性能大比拼平台从 2013 年 3 月开始以来已经历经了 18 轮测试,参与这个平台的框架平台产 ...

  2. Cover letter|review|Discussion

    选择期刊考虑影响因子和载文量(流量) 分类:多学科eg:CNS 专业综合:eg:nature子刊:lancet:cell,jacs 细分:eg:CA-A 投完Cover letter后,根据审稿结果修 ...

  3. python类的继承、多继承及其常用魔术方法

    继承 一个类可以派生出一个子类,这个子类可以使用父类的属性及方法,也可以在父类的基础上添加自己的独特属性或方法.属性和方法的继承的顺序是先从自己开始,找不到再去找父类,父类没有再找父类的父类,其尽头就 ...

  4. Python 爬取腾讯招聘职位详情 2019/12/4有效

    我爬取的是Python相关职位,先po上代码,(PS:本人小白,这是跟着B站教学视频学习后,老师留的作业,因为腾讯招聘的网站变动比较大,老师的代码已经无法运行,所以po上),一些想法和过程在后面. f ...

  5. Linux-使用syslog来记录调试信息

    1.有三个函数:openlog.syslog.closelog 2.一般的log信息都在操作系统的/var/log/messages这个文件中存储着,但是ubuntu中是在/var/log/syslo ...

  6. 一个帖子csrf的例子

    服务端 <?php $conn=mysqli_connect('localhost','root','root','csrf'); $data=$_POST; $user=$_POST['use ...

  7. [HNOI2019]校园旅行（建图优化+bfs）

    30分的O(m^2)做法应该比较容易想到:令f[i][j]表示i->j是否有解,然后把每个路径点数不超过2的有解状态(u,v)加入队列,然后弹出队列时,两点分别向两边搜索边,发现颜色一样时,再修 ...

  8. 字符串常用方法总结与StringBuffer基础

    字符串 基本特性 final:字符串被修饰为final,是不能被继承的. immutable:字符串是不可改变的,例如创建了一个字符串对象,就不可改变它,即不能进行增删查改其中的字符.一旦创建好这个字 ...

  9. iTOP-iMX6UL开发板-MiniLinux-CAN测试使用文档

    本文档介绍的是迅为iMX6UL开发板在 MiniLinux 系统环境下 iTOP-iMX6UL CAN 实验调试步骤.给用户提供了“can_libs.rar”.“can_tools.zip”和“can ...

  10. FPGA 状态机-序列检测器verilog

    实现功能:检测出串行输入数据4位Data二进制序列0101,当检测到该序列的时候,out=1,否则out=0 (1)给出状态编码,画出状态图 (2)门电路实现 (3)verilog实现 首先规定Q3Q ...