在分布式系统中，往往需要一些分布式同步原语来做一些协同工作，上一篇文章介绍了Zookeeper的基本原理，本文介绍下基于Zookeeper的Lock和Queue的实现，主要代码都来自Zookeeper的官方recipe。

锁（Lock）

完全分布式锁是全局同步的，这意味着在任何时刻没有两个客户端会同时认为它们都拥有相同的锁，使用 Zookeeper 可以实现分布式锁，需要首先定义一个锁节点（lock root node）。

需要获得锁的客户端按照以下步骤来获取锁：

  1. 保证锁节点（lock root node）这个父根节点的存在，这个节点是每个要获取lock客户端共用的，这个节点是PERSISTENT的。

  2. 第一次需要创建本客户端要获取lock的节点，调用 create( )，并设置 节点为EPHEMERAL_SEQUENTIAL类型，表示该节点为临时的和顺序的。如果获取锁的节点挂掉，则该节点自动失效，可以让其他节点获取锁。

  3. 在父锁节点（lock root node）上调用 getChildren( ) ，不需要设置监视标志。 (为了避免“羊群效应”).

  4. 按照Fair竞争的原则，将步骤3中的子节点（要获取锁的节点）按照节点顺序的大小做排序，取出编号最小的一个节点做为lock的owner，判断自己的节点id是否就为owner id，如果是则返回，lock成功。如果不是则调用 exists( )监听比自己小的前一位的id，关注它锁释放的操作（也就是exist watch）。

  5. 如果第4步监听exist的watch被触发，则继续按4中的原则判断自己是否能获取到lock。

释放锁：需要释放锁的客户端只需要删除在第2步中创建的节点即可。

注意事项：

一个节点的删除只会导致一个客户端被唤醒，因为每个节点只被一个客户端watch，这避免了“羊群效应”。

一个分布式lock的实现：

  1. package org.apache.zookeeper.recipes.lock;
  2.  
  3. import org.slf4j.Logger;
  4. import org.slf4j.LoggerFactory;
  5. import org.apache.zookeeper.KeeperException;
  6. import org.apache.zookeeper.WatchedEvent;
  7. import org.apache.zookeeper.Watcher;
  8. import static org.apache.zookeeper.CreateMode.EPHEMERAL_SEQUENTIAL;
  9. import org.apache.zookeeper.ZooKeeper;
  10. import org.apache.zookeeper.data.ACL;
  11. import org.apache.zookeeper.data.Stat;
  12.  
  13. import java.util.List;
  14. import java.util.SortedSet;
  15. import java.util.TreeSet;
  16.  
  17. /**
  18.  * A <a href="package.html">protocol to implement an exclusive
  19.  *  write lock or to elect a leader</a>. <p/> You invoke {@link #lock()} to
  20.  *  start the process of grabbing the lock; you may get the lock then or it may be
  21.  *  some time later. <p/> You can register a listener so that you are invoked
  22.  *  when you get the lock; otherwise you can ask if you have the lock
  23.  *  by calling {@link #isOwner()}
  24.  *
  25.  */
  26. public class WriteLock extends ProtocolSupport {
  27.     private static final Logger LOG = LoggerFactory.getLogger(WriteLock.class);
  28.  
  29.     private final String dir;
  30.     private String id;
  31.     private ZNodeName idName;
  32.     private String ownerId;
  33.     private String lastChildId;
  34.     private byte[] data = {0x12, 0x34};
  35.     private LockListener callback;
  36.     private LockZooKeeperOperation zop;
  37.  
  38.     /**
  39.      * zookeeper contructor for writelock
  40.      * @param zookeeper zookeeper client instance
  41.      * @param dir the parent path you want to use for locking
  42.      * @param acls the acls that you want to use for all the paths,
  43.      * if null world read/write is used.
  44.      */
  45.     public WriteLock(ZooKeeper zookeeper, String dir, List<ACL> acl) {
  46.         super(zookeeper);
  47.         this.dir = dir;
  48.         if (acl != null) {
  49.             setAcl(acl);
  50.         }
  51.         this.zop = new LockZooKeeperOperation();
  52.     }
  53.  
  54.     /**
  55.      * zookeeper contructor for writelock with callback
  56.      * @param zookeeper the zookeeper client instance
  57.      * @param dir the parent path you want to use for locking
  58.      * @param acl the acls that you want to use for all the paths
  59.      * @param callback the call back instance
  60.      */
  61.     public WriteLock(ZooKeeper zookeeper, String dir, List<ACL> acl,
  62.             LockListener callback) {
  63.         this(zookeeper, dir, acl);
  64.         this.callback = callback;
  65.     }
  66.  
  67.     /**
  68.      * return the current locklistener
  69.      * @return the locklistener
  70.      */
  71.     public LockListener getLockListener() {
  72.         return this.callback;
  73.     }
  74.  
  75.     /**
  76.      * register a different call back listener
  77.      * @param callback the call back instance
  78.      */
  79.     public void setLockListener(LockListener callback) {
  80.         this.callback = callback;
  81.     }
  82.  
  83.     /**
  84.      * Removes the lock or associated znode if
  85.      * you no longer require the lock. this also
  86.      * removes your request in the queue for locking
  87.      * in case you do not already hold the lock.
  88.      * @throws RuntimeException throws a runtime exception
  89.      * if it cannot connect to zookeeper.
  90.      */
  91.     public synchronized void unlock() throws RuntimeException {
  92.  
  93.         if (!isClosed() && id != null) {
  94.             // we don't need to retry this operation in the case of failure
  95.             // as ZK will remove ephemeral files and we don't wanna hang
  96.             // this process when closing if we cannot reconnect to ZK
  97.             try {
  98.                 ZooKeeperOperation zopdel = new ZooKeeperOperation() {
  99.                     public boolean execute() throws KeeperException,
  100.                         InterruptedException {
  101.                         zookeeper.delete(id, -1);
  102.                         return Boolean.TRUE;
  103.                     }
  104.                 };
  105.                 zopdel.execute();
  106.             } catch (InterruptedException e) {
  107.                 LOG.warn("Caught: " + e, e);
  108.                 //set that we have been interrupted.
  109.                Thread.currentThread().interrupt();
  110.             } catch (KeeperException.NoNodeException e) {
  111.                 // do nothing
  112.             } catch (KeeperException e) {
  113.                 LOG.warn("Caught: " + e, e);
  114.                 throw (RuntimeException) new RuntimeException(e.getMessage()).
  115.                     initCause(e);
  116.             }
  117.             finally {
  118.                 if (callback != null) {
  119.                     callback.lockReleased();
  120.                 }
  121.                 id = null;
  122.             }
  123.         }
  124.     }
  125.  
  126.     /**
  127.      * the watcher called on
  128.      * getting watch while watching
  129.      * my predecessor
  130.      */
  131.     private class LockWatcher implements Watcher {
  132.         public void process(WatchedEvent event) {
  133.             // lets either become the leader or watch the new/updated node
  134.             LOG.debug("Watcher fired on path: " + event.getPath() + " state: " +
  135.                     event.getState() + " type " + event.getType());
  136.             try {
  137.                 lock();
  138.             } catch (Exception e) {
  139.                 LOG.warn("Failed to acquire lock: " + e, e);
  140.             }
  141.         }
  142.     }
  143.  
  144.     /**
  145.      * a zoookeeper operation that is mainly responsible
  146.      * for all the magic required for locking.
  147.      */
  148.     private  class LockZooKeeperOperation implements ZooKeeperOperation {
  149.  
  150.         /** find if we have been created earler if not create our node
  151.          *
  152.          * @param prefix the prefix node
  153.          * @param zookeeper teh zookeeper client
  154.          * @param dir the dir paretn
  155.          * @throws KeeperException
  156.          * @throws InterruptedException
  157.          */
  158.         private void findPrefixInChildren(String prefix, ZooKeeper zookeeper, String dir)
  159.             throws KeeperException, InterruptedException {
  160.             List<String> names = zookeeper.getChildren(dir, false);
  161.             for (String name : names) {
  162.                 if (name.startsWith(prefix)) {
  163.                     id = dir + "/" + name;
  164.                     if (LOG.isDebugEnabled()) {
  165.                         LOG.debug("Found id created last time: " + id);
  166.                     }
  167.                     break;
  168.                 }
  169.             }
  170.             if (id == null) {
  171.                 id = zookeeper.create(dir + "/" + prefix, data,
  172.                         getAcl(), EPHEMERAL_SEQUENTIAL);
  173.  
  174.                 if (LOG.isDebugEnabled()) {
  175.                     LOG.debug("Created id: " + id);
  176.                 }
  177.             }
  178.  
  179.         }
  180.  
  181.         /**
  182.          * the command that is run and retried for actually
  183.          * obtaining the lock
  184.          * @return if the command was successful or not
  185.          */
  186.         public boolean execute() throws KeeperException, InterruptedException {
  187.             do {
  188.                 if (id == null) {
  189.                     long sessionId = zookeeper.getSessionId();
  190.                     String prefix = "x-" + sessionId + "-";
  191.                     // lets try look up the current ID if we failed
  192.                     // in the middle of creating the znode
  193.                     findPrefixInChildren(prefix, zookeeper, dir);
  194.                     idName = new ZNodeName(id);
  195.                 }
  196.                 if (id != null) {
  197.                     List<String> names = zookeeper.getChildren(dir, false);
  198.                     if (names.isEmpty()) {
  199.                         LOG.warn("No children in: " + dir + " when we've just " +
  200.                         "created one! Lets recreate it...");
  201.                         // lets force the recreation of the id
  202.                         id = null;
  203.                     } else {
  204.                         // lets sort them explicitly (though they do seem to come back in order ususally :)
  205.                         SortedSet<ZNodeName> sortedNames = new TreeSet<ZNodeName>();
  206.                         for (String name : names) {
  207.                             sortedNames.add(new ZNodeName(dir + "/" + name));
  208.                         }
  209.                         ownerId = sortedNames.first().getName();
  210.                         SortedSet<ZNodeName> lessThanMe = sortedNames.headSet(idName);
  211.                         if (!lessThanMe.isEmpty()) {
  212.                             ZNodeName lastChildName = lessThanMe.last();
  213.                             lastChildId = lastChildName.getName();
  214.                             if (LOG.isDebugEnabled()) {
  215.                                 LOG.debug("watching less than me node: " + lastChildId);
  216.                             }
  217.                             Stat stat = zookeeper.exists(lastChildId, new LockWatcher());
  218.                             if (stat != null) {
  219.                                 return Boolean.FALSE;
  220.                             } else {
  221.                                 LOG.warn("Could not find the" +
  222.                                         " stats for less than me: " + lastChildName.getName());
  223.                             }
  224.                         } else {
  225.                             if (isOwner()) {
  226.                                 if (callback != null) {
  227.                                     callback.lockAcquired();
  228.                                 }
  229.                                 return Boolean.TRUE;
  230.                             }
  231.                         }
  232.                     }
  233.                 }
  234.             }
  235.             while (id == null);
  236.             return Boolean.FALSE;
  237.         }
  238.     };
  239.  
  240.     /**
  241.      * Attempts to acquire the exclusive write lock returning whether or not it was
  242.      * acquired. Note that the exclusive lock may be acquired some time later after
  243.      * this method has been invoked due to the current lock owner going away.
  244.      */
  245.     public synchronized boolean lock() throws KeeperException, InterruptedException {
  246.         if (isClosed()) {
  247.             return false;
  248.         }
  249.         ensurePathExists(dir);
  250.  
  251.         return (Boolean) retryOperation(zop);
  252.     }
  253.  
  254.     /**
  255.      * return the parent dir for lock
  256.      * @return the parent dir used for locks.
  257.      */
  258.     public String getDir() {
  259.         return dir;
  260.     }
  261.  
  262.     /**
  263.      * Returns true if this node is the owner of the
  264.      *  lock (or the leader)
  265.      */
  266.     public boolean isOwner() {
  267.         return id != null && ownerId != null && id.equals(ownerId);
  268.     }
  269.  
  270.     /**
  271.      * return the id for this lock
  272.      * @return the id for this lock
  273.      */
  274.     public String getId() {
  275.        return this.id;
  276.     }
  277. }

注意这里的lock，可能会失败，会尝试多次，每次失败后会Sleep一段时间。

PS：官方的代码有个小bug，id和ownerId应该都是全路径，即id = dir + "/" + name;原代码在findPrefixInChildren里有问题。

用于辅助节点大小顺序排序的类：

  1. package org.apache.zookeeper.recipes.lock;
  2.  
  3. import org.slf4j.Logger;
  4. import org.slf4j.LoggerFactory;
  5.  
  6. /**
  7.  * Represents an ephemeral znode name which has an ordered sequence number and
  8.  * can be sorted in order
  9.  *
  10.  */
  11. class ZNodeName implements Comparable<ZNodeName> {
  12.     private final String name;
  13.     private String prefix;
  14.     private int sequence = -1;
  15.     private static final Logger LOG = LoggerFactory.getLogger(ZNodeName.class);
  16.  
  17.     public ZNodeName(String name) {
  18.         if (name == null) {
  19.             throw new NullPointerException("id cannot be null");
  20.         }
  21.         this.name = name;
  22.         this.prefix = name;
  23.         int idx = name.lastIndexOf('-');
  24.         if (idx >= 0) {
  25.             this.prefix = name.substring(0, idx);
  26.             try {
  27.                 this.sequence = Integer.parseInt(name.substring(idx + 1));
  28.                 // If an exception occurred we misdetected a sequence suffix,
  29.                 // so return -1.
  30.             } catch (NumberFormatException e) {
  31.                 LOG.info("Number format exception for " + idx, e);
  32.             } catch (ArrayIndexOutOfBoundsException e) {
  33.                 LOG.info("Array out of bounds for " + idx, e);
  34.             }
  35.         }
  36.     }
  37.  
  38.     @Override
  39.     public String toString() {
  40.         return name.toString();
  41.     }
  42.  
  43.     @Override
  44.     public boolean equals(Object o) {
  45.         if (this == o)
  46.             return true;
  47.         if (o == null || getClass() != o.getClass())
  48.             return false;
  49.  
  50.         ZNodeName sequence = (ZNodeName) o;
  51.  
  52.         if (!name.equals(sequence.name))
  53.             return false;
  54.  
  55.         return true;
  56.     }
  57.  
  58.     @Override
  59.     public int hashCode() {
  60.         return name.hashCode() + 37;
  61.     }
  62.  
  63.     public int compareTo(ZNodeName that) {
  64.         int s1 = this.sequence;
  65.         int s2 = that.sequence;
  66.         if (s1 == -1 && s2 == -1) {
  67.             return this.name.compareTo(that.name);
  68.         }
  69.         if (s1 == -1) {
  70.             return -1;
  71.         } else if (s2 == -1) {
  72.             return 1;
  73.         } else {
  74.             return s1 - s2;
  75.         }
  76.     }
  77.  
  78.     /**
  79.      * Returns the name of the znode
  80.      */
  81.     public String getName() {
  82.         return name;
  83.     }
  84.  
  85.     /**
  86.      * Returns the sequence number
  87.      */
  88.     public int getZNodeName() {
  89.         return sequence;
  90.     }
  91.  
  92.     /**
  93.      * Returns the text prefix before the sequence number
  94.      */
  95.     public String getPrefix() {
  96.         return prefix;
  97.     }
  98. }

PS：这个ZNodeName类是被我修改过的，官方的代码比较有问题，官方的先用了节点路径的前缀prefix比较，再去比较sequence序号是不对的，这样会导致sessionid小的总是能拿到锁。应该直接比较全局有序的sequence序号，小的先拿到锁，先到先得。

Zookeeper统一操作ZooKeeperOperation接口：

  1. public interface ZooKeeperOperation {
  2.  
  3.     /**
  4.      * Performs the operation - which may be involved multiple times if the connection
  5.      * to ZooKeeper closes during this operation
  6.      *
  7.      * @return the result of the operation or null
  8.      * @throws KeeperException
  9.      * @throws InterruptedException
  10.      */
  11.     public boolean execute() throws KeeperException, InterruptedException;
  12. }

因为Zookeeper的操作会失败，这个类封装了多次尝试：

  1. /**
  2.  *
  3.  * Licensed to the Apache Software Foundation (ASF) under one or more
  4.  * contributor license agreements.  See the NOTICE file distributed with
  5.  * this work for additional information regarding copyright ownership.
  6.  * The ASF licenses this file to You under the Apache License, Version 2.0
  7.  * (the "License"); you may not use this file except in compliance with
  8.  * the License.  You may obtain a copy of the License at
  9.  *
  10.  * http://www.apache.org/licenses/LICENSE-2.0
  11.  *
  12.  * Unless required by applicable law or agreed to in writing, software
  13.  * distributed under the License is distributed on an "AS IS" BASIS,
  14.  * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  15.  * See the License for the specific language governing permissions and
  16.  * limitations under the License.
  17.  */
  18. package org.apache.zookeeper.recipes.lock;
  19.  
  20. import org.slf4j.Logger;
  21. import org.slf4j.LoggerFactory;
  22. import org.apache.zookeeper.CreateMode;
  23. import org.apache.zookeeper.KeeperException;
  24. import org.apache.zookeeper.ZooDefs;
  25. import org.apache.zookeeper.ZooKeeper;
  26. import org.apache.zookeeper.data.ACL;
  27. import org.apache.zookeeper.data.Stat;
  28. import org.apache.zookeeper.recipes.lock.ZooKeeperOperation;
  29.  
  30. import java.util.List;
  31. import java.util.concurrent.atomic.AtomicBoolean;
  32.  
  33. /**
  34.  * A base class for protocol implementations which provides a number of higher
  35.  * level helper methods for working with ZooKeeper along with retrying synchronous
  36.  *  operations if the connection to ZooKeeper closes such as
  37.  *  {@link #retryOperation(ZooKeeperOperation)}
  38.  *
  39.  */
  40. class ProtocolSupport {
  41.     private static final Logger LOG = LoggerFactory.getLogger(ProtocolSupport.class);
  42.  
  43.     protected final ZooKeeper zookeeper;
  44.     private AtomicBoolean closed = new AtomicBoolean(false);
  45.     private long retryDelay = 500L;
  46.     private int retryCount = 10;
  47.     private List<ACL> acl = ZooDefs.Ids.OPEN_ACL_UNSAFE;
  48.  
  49.     public ProtocolSupport(ZooKeeper zookeeper) {
  50.         this.zookeeper = zookeeper;
  51.     }
  52.  
  53.     /**
  54.      * Closes this strategy and releases any ZooKeeper resources; but keeps the
  55.      *  ZooKeeper instance open
  56.      */
  57.     public void close() {
  58.         if (closed.compareAndSet(false, true)) {
  59.             doClose();
  60.         }
  61.     }
  62.  
  63.     /**
  64.      * return zookeeper client instance
  65.      * @return zookeeper client instance
  66.      */
  67.     public ZooKeeper getZookeeper() {
  68.         return zookeeper;
  69.     }
  70.  
  71.     /**
  72.      * return the acl its using
  73.      * @return the acl.
  74.      */
  75.     public List<ACL> getAcl() {
  76.         return acl;
  77.     }
  78.  
  79.     /**
  80.      * set the acl
  81.      * @param acl the acl to set to
  82.      */
  83.     public void setAcl(List<ACL> acl) {
  84.         this.acl = acl;
  85.     }
  86.  
  87.     /**
  88.      * get the retry delay in milliseconds
  89.      * @return the retry delay
  90.      */
  91.     public long getRetryDelay() {
  92.         return retryDelay;
  93.     }
  94.  
  95.     /**
  96.      * Sets the time waited between retry delays
  97.      * @param retryDelay the retry delay
  98.      */
  99.     public void setRetryDelay(long retryDelay) {
  100.         this.retryDelay = retryDelay;
  101.     }
  102.  
  103.     /**
  104.      * Allow derived classes to perform
  105.      * some custom closing operations to release resources
  106.      */
  107.     protected void doClose() {
  108.     }
  109.  
  110.     /**
  111.      * Perform the given operation, retrying if the connection fails
  112.      * @return object. it needs to be cast to the callee's expected
  113.      * return type.
  114.      */
  115.     protected Object retryOperation(ZooKeeperOperation operation)
  116.         throws KeeperException, InterruptedException {
  117.         KeeperException exception = null;
  118.         for (int i = 0; i < retryCount; i++) {
  119.             try {
  120.                 return operation.execute();
  121.             } catch (KeeperException.SessionExpiredException e) {
  122.                 LOG.warn("Session expired for: " + zookeeper + " so reconnecting due to: " + e, e);
  123.                 throw e;
  124.             } catch (KeeperException.ConnectionLossException e) {
  125.                 if (exception == null) {
  126.                     exception = e;
  127.                 }
  128.                 LOG.debug("Attempt " + i + " failed with connection loss so " +
  129.                         "attempting to reconnect: " + e, e);
  130.                 retryDelay(i);
  131.             }
  132.         }
  133.         throw exception;
  134.     }
  135.  
  136.     /**
  137.      * Ensures that the given path exists with no data, the current
  138.      * ACL and no flags
  139.      * @param path
  140.      */
  141.     protected void ensurePathExists(String path) {
  142.         ensureExists(path, null, acl, CreateMode.PERSISTENT);
  143.     }
  144.  
  145.     /**
  146.      * Ensures that the given path exists with the given data, ACL and flags
  147.      * @param path
  148.      * @param acl
  149.      * @param flags
  150.      */
  151.     protected void ensureExists(final String path, final byte[] data,
  152.             final List<ACL> acl, final CreateMode flags) {
  153.         try {
  154.             retryOperation(new ZooKeeperOperation() {
  155.                 public boolean execute() throws KeeperException, InterruptedException {
  156.                     Stat stat = zookeeper.exists(path, false);
  157.                     if (stat != null) {
  158.                         return true;
  159.                     }
  160.                     zookeeper.create(path, data, acl, flags);
  161.                     return true;
  162.                 }
  163.             });
  164.         } catch (KeeperException e) {
  165.             LOG.warn("Caught: " + e, e);
  166.         } catch (InterruptedException e) {
  167.             LOG.warn("Caught: " + e, e);
  168.         }
  169.     }
  170.  
  171.     /**
  172.      * Returns true if this protocol has been closed
  173.      * @return true if this protocol is closed
  174.      */
  175.     protected boolean isClosed() {
  176.         return closed.get();
  177.     }
  178.  
  179.     /**
  180.      * Performs a retry delay if this is not the first attempt
  181.      * @param attemptCount the number of the attempts performed so far
  182.      */
  183.     protected void retryDelay(int attemptCount) {
  184.         if (attemptCount > 0) {
  185.             try {
  186.                 Thread.sleep(attemptCount * retryDelay);
  187.             } catch (InterruptedException e) {
  188.                 LOG.debug("Failed to sleep: " + e, e);
  189.             }
  190.         }
  191.     }
  192. }

这个类是本客户端获取到lock和释放lock的时候触发操作的接口：

  1. public interface LockListener {
  2.     /**
  3.      * call back called when the lock
  4.      * is acquired
  5.      */
  6.     public void lockAcquired();
  7.  
  8.     /**
  9.      * call back called when the lock is
  10.      * released.
  11.      */
  12.     public void lockReleased();
  13. }

队列（Queue）

分布式队列是通用的数据结构，为了在 Zookeeper 中实现分布式队列，首先需要指定一个 Znode 节点作为队列节点（queue node）， 各个分布式客户端通过调用 create() 函数向队列中放入数据，调用create()时节点路径名带"qn-"结尾，并设置顺序（sequence）节点标志。 由于设置了节点的顺序标志，新的路径名具有以下字符串模式："_path-to-queue-node_/qn-X"，X 是唯一自增号。需要从队列中获取数据/移除数据的客户端首先调用 getChildren() 函数，有数据则获取（获取数据后可以删除也可以不删），没有则在队列节点（queue node）上将 watch 设置为 true，等待触发并处理最小序号的节点（即从序号最小的节点中取数据）。

实现步骤基本如下：

前提：需要一个队列root节点dir

入队：使用create()创建节点，将共享数据data放在该节点上，节点类型为PERSISTENT_SEQUENTIAL，永久顺序性的（也可以设置为临时的，看需求）。

出队：因为队列可能为空，2种方式处理：一种如果为空则wait等待，一种返回异常。

等待方式：这里使用了CountDownLatch的等待和Watcher的通知机制，使用了TreeMap的排序获取节点顺序最小的数据（FIFO）。

抛出异常：getChildren()获取队列数据时，如果size==0则抛出异常。

一个分布式Queue的实现，详细代码：

  1. package org.apache.zookeeper.recipes.queue;
  2.  
  3. import java.util.List;
  4. import java.util.NoSuchElementException;
  5. import java.util.TreeMap;
  6. import java.util.concurrent.CountDownLatch;
  7.  
  8. import org.slf4j.Logger;
  9. import org.slf4j.LoggerFactory;
  10. import org.apache.zookeeper.CreateMode;
  11. import org.apache.zookeeper.KeeperException;
  12. import org.apache.zookeeper.WatchedEvent;
  13. import org.apache.zookeeper.Watcher;
  14. import org.apache.zookeeper.ZooDefs;
  15. import org.apache.zookeeper.ZooKeeper;
  16. import org.apache.zookeeper.data.ACL;
  17. import org.apache.zookeeper.data.Stat;
  18.  
  19. /**
  20.  *
  21.  * A <a href="package.html">protocol to implement a distributed queue</a>.
  22.  *
  23.  */
  24. public class DistributedQueue {
  25.     private static final Logger LOG = LoggerFactory.getLogger(DistributedQueue.class);
  26.  
  27.     private final String dir;
  28.  
  29.     private ZooKeeper zookeeper;
  30.     private List<ACL> acl = ZooDefs.Ids.OPEN_ACL_UNSAFE;
  31.  
  32.     private final String prefix = "qn-";
  33.  
  34.     public DistributedQueue(ZooKeeper zookeeper, String dir, List<ACL> acl){
  35.         this.dir = dir;
  36.  
  37.         if(acl != null){
  38.             this.acl = acl;
  39.         }
  40.         this.zookeeper = zookeeper;
  41.  
  42.         //Add root dir first if not exists
  43.         if (zookeeper != null) {
  44.             try {
  45.                 Stat s = zookeeper.exists(dir, false);
  46.                 if (s == null) {
  47.                     zookeeper.create(dir, new byte[0], acl, CreateMode.PERSISTENT);
  48.                 }
  49.             } catch (KeeperException e) {
  50.                 LOG.error(e.toString());
  51.             } catch (InterruptedException e) {
  52.                 LOG.error(e.toString());
  53.             }
  54.         }
  55.     }
  56.  
  57.     /**
  58.      * Returns a Map of the children, ordered by id.
  59.      * @param watcher optional watcher on getChildren() operation.
  60.      * @return map from id to child name for all children
  61.      */
  62.     private TreeMap<Long,String> orderedChildren(Watcher watcher) throws KeeperException, InterruptedException {
  63.         TreeMap<Long,String> orderedChildren = new TreeMap<Long,String>();
  64.  
  65.         List<String> childNames = null;
  66.         try{
  67.             childNames = zookeeper.getChildren(dir, watcher);
  68.         }catch (KeeperException.NoNodeException e){
  69.             throw e;
  70.         }
  71.  
  72.         for(String childName : childNames){
  73.             try{
  74.                 //Check format
  75.                 if(!childName.regionMatches(0, prefix, 0, prefix.length())){
  76.                     LOG.warn("Found child node with improper name: " + childName);
  77.                     continue;
  78.                 }
  79.                 String suffix = childName.substring(prefix.length());
  80.                 Long childId = new Long(suffix);
  81.                 orderedChildren.put(childId,childName);
  82.             }catch(NumberFormatException e){
  83.                 LOG.warn("Found child node with improper format : " + childName + " " + e,e);
  84.             }
  85.         }
  86.  
  87.         return orderedChildren;
  88.     }
  89.  
  90.     /**
  91.      * Find the smallest child node.
  92.      * @return The name of the smallest child node.
  93.      */
  94.     private String smallestChildName() throws KeeperException, InterruptedException {
  95.         long minId = Long.MAX_VALUE;
  96.         String minName = "";
  97.  
  98.         List<String> childNames = null;
  99.  
  100.         try{
  101.             childNames = zookeeper.getChildren(dir, false);
  102.         }catch(KeeperException.NoNodeException e){
  103.             LOG.warn("Caught: " +e,e);
  104.             return null;
  105.         }
  106.  
  107.         for(String childName : childNames){
  108.             try{
  109.                 //Check format
  110.                 if(!childName.regionMatches(0, prefix, 0, prefix.length())){
  111.                     LOG.warn("Found child node with improper name: " + childName);
  112.                     continue;
  113.                 }
  114.                 String suffix = childName.substring(prefix.length());
  115.                 long childId = Long.parseLong(suffix);
  116.                 if(childId < minId){
  117.                     minId = childId;
  118.                     minName = childName;
  119.                 }
  120.             }catch(NumberFormatException e){
  121.                 LOG.warn("Found child node with improper format : " + childName + " " + e,e);
  122.             }
  123.         }
  124.  
  125.         if(minId < Long.MAX_VALUE){
  126.             return minName;
  127.         }else{
  128.             return null;
  129.         }
  130.     }
  131.  
  132.     /**
  133.      * Return the head of the queue without modifying the queue.
  134.      * @return the data at the head of the queue.
  135.      * @throws NoSuchElementException
  136.      * @throws KeeperException
  137.      * @throws InterruptedException
  138.      */
  139.     public byte[] element() throws NoSuchElementException, KeeperException, InterruptedException {
  140.         TreeMap<Long,String> orderedChildren;
  141.  
  142.         // element, take, and remove follow the same pattern.
  143.         // We want to return the child node with the smallest sequence number.
  144.         // Since other clients are remove()ing and take()ing nodes concurrently,
  145.         // the child with the smallest sequence number in orderedChildren might be gone by the time we check.
  146.         // We don't call getChildren again until we have tried the rest of the nodes in sequence order.
  147.         while(true){
  148.             try{
  149.                 orderedChildren = orderedChildren(null);
  150.             }catch(KeeperException.NoNodeException e){
  151.                 throw new NoSuchElementException();
  152.             }
  153.             if(orderedChildren.size() == 0 ) throw new NoSuchElementException();
  154.  
  155.             for(String headNode : orderedChildren.values()){
  156.                 if(headNode != null){
  157.                     try{
  158.                         return zookeeper.getData(dir+"/"+headNode, false, null);
  159.                     }catch(KeeperException.NoNodeException e){
  160.                         //Another client removed the node first, try next
  161.                     }
  162.                 }
  163.             }
  164.  
  165.         }
  166.     }
  167.  
  168.     /**
  169.      * Attempts to remove the head of the queue and return it.
  170.      * @return The former head of the queue
  171.      * @throws NoSuchElementException
  172.      * @throws KeeperException
  173.      * @throws InterruptedException
  174.      */
  175.     public byte[] remove() throws NoSuchElementException, KeeperException, InterruptedException {
  176.         TreeMap<Long,String> orderedChildren;
  177.         // Same as for element.  Should refactor this.
  178.         while(true){
  179.             try{
  180.                 orderedChildren = orderedChildren(null);
  181.             }catch(KeeperException.NoNodeException e){
  182.                 throw new NoSuchElementException();
  183.             }
  184.             if(orderedChildren.size() == 0) throw new NoSuchElementException();
  185.  
  186.             for(String headNode : orderedChildren.values()){
  187.                 String path = dir +"/"+headNode;
  188.                 try{
  189.                     byte[] data = zookeeper.getData(path, false, null);
  190.                     zookeeper.delete(path, -1);
  191.                     return data;
  192.                 }catch(KeeperException.NoNodeException e){
  193.                     // Another client deleted the node first.
  194.                 }
  195.             }
  196.         }
  197.     }
  198.  
  199.     private class LatchChildWatcher implements Watcher {
  200.  
  201.         CountDownLatch latch;
  202.  
  203.         public LatchChildWatcher(){
  204.             latch = new CountDownLatch(1);
  205.         }
  206.  
  207.         public void process(WatchedEvent event){
  208.             LOG.debug("Watcher fired on path: " + event.getPath() + " state: " +
  209.                     event.getState() + " type " + event.getType());
  210.             latch.countDown();
  211.         }
  212.         public void await() throws InterruptedException {
  213.             latch.await();
  214.         }
  215.     }
  216.  
  217.     /**
  218.      * Removes the head of the queue and returns it, blocks until it succeeds.
  219.      * @return The former head of the queue
  220.      * @throws NoSuchElementException
  221.      * @throws KeeperException
  222.      * @throws InterruptedException
  223.      */
  224.     public byte[] take() throws KeeperException, InterruptedException {
  225.         TreeMap<Long,String> orderedChildren;
  226.         // Same as for element.  Should refactor this.
  227.         while(true){
  228.             LatchChildWatcher childWatcher = new LatchChildWatcher();
  229.             try{
  230.                 orderedChildren = orderedChildren(childWatcher);
  231.             }catch(KeeperException.NoNodeException e){
  232.                 zookeeper.create(dir, new byte[0], acl, CreateMode.PERSISTENT);
  233.                 continue;
  234.             }
  235.             if(orderedChildren.size() == 0){
  236.                 childWatcher.await();
  237.                 continue;
  238.             }
  239.  
  240.             for(String headNode : orderedChildren.values()){
  241.                 String path = dir +"/"+headNode;
  242.                 try{
  243.                     byte[] data = zookeeper.getData(path, false, null);
  244.                     zookeeper.delete(path, -1);
  245.                     return data;
  246.                 }catch(KeeperException.NoNodeException e){
  247.                     // Another client deleted the node first.
  248.                 }
  249.             }
  250.         }
  251.     }
  252.  
  253.     /**
  254.      * Inserts data into queue.
  255.      * @param data
  256.      * @return true if data was successfully added
  257.      */
  258.     public boolean offer(byte[] data) throws KeeperException, InterruptedException{
  259.         for(;;){
  260.             try{
  261.                 zookeeper.create(dir+"/"+prefix, data, acl, CreateMode.PERSISTENT_SEQUENTIAL);
  262.                 return true;
  263.             }catch(KeeperException.NoNodeException e){
  264.                 zookeeper.create(dir, new byte[0], acl, CreateMode.PERSISTENT);
  265.             }
  266.         }
  267.     }
  268.  
  269.     /**
  270.      * Returns the data at the first element of the queue, or null if the queue is empty.
  271.      * @return data at the first element of the queue, or null.
  272.      * @throws KeeperException
  273.      * @throws InterruptedException
  274.      */
  275.     public byte[] peek() throws KeeperException, InterruptedException{
  276.         try{
  277.             return element();
  278.         }catch(NoSuchElementException e){
  279.             return null;
  280.         }
  281.     }
  282.  
  283.     /**
  284.      * Attempts to remove the head of the queue and return it. Returns null if the queue is empty.
  285.      * @return Head of the queue or null.
  286.      * @throws KeeperException
  287.      * @throws InterruptedException
  288.      */
  289.     public byte[] poll() throws KeeperException, InterruptedException {
  290.         try{
  291.             return remove();
  292.         }catch(NoSuchElementException e){
  293.             return null;
  294.         }
  295.     }
  296. }

Apache Curator

Curator是一个封装Zookeeper操作的库，使用这个库的好处是Curator帮你管理和Zookeeper的连接，当连接有问题时会自动重试（retry）。

  1. RetryPolicy retryPolicy = new ExponentialBackoffRetry(1000, 3)
  2. CuratorFramework client = CuratorFrameworkFactory.newClient(zookeeperConnectionString, retryPolicy);
  3. client.start();

Curator已经封装了一些常用的Recipes

Distributed Lock

  1. InterProcessMutex lock = new InterProcessMutex(client, lockPath);
  2. if ( lock.acquire(maxWait, waitUnit) )
  3. {
  4.     try
  5.     {
  6.         // do some work inside of the critical section here
  7.     }
  8.     finally
  9.     {
  10.         lock.release();
  11.     }
  12. }

Leader Election

  1. LeaderSelectorListener listener = new LeaderSelectorListenerAdapter()
  2. {
  3.     public void takeLeadership(CuratorFramework client) throws Exception
  4.     {
  5.         // this callback will get called when you are the leader
  6.         // do whatever leader work you need to and only exit
  7.         // this method when you want to relinquish leadership
  8.     }
  9. }
  10.  
  11. LeaderSelector selector = new LeaderSelector(client, path, listener);
  12. selector.autoRequeue();  // not required, but this is behavior that you will probably expect
  13. selector.start(); 

参考：

http://zookeeper.apache.org/doc/trunk/recipes.html

http://curator.apache.org/curator-recipes/index.html

基于ZooKeeper的分布式锁和队列的更多相关文章

  1. 分布式锁（3) ----- 基于zookeeper的分布式锁

    分布式锁系列文章 分布式锁(1) ----- 介绍和基于数据库的分布式锁 分布式锁(2) ----- 基于redis的分布式锁 分布式锁(3) ----- 基于zookeeper的分布式锁 代码:ht ...

  2. 基于 Zookeeper 的分布式锁实现

    1. 背景 最近在学习 Zookeeper,在刚开始接触 Zookeeper 的时候,完全不知道 Zookeeper 有什么用.且很多资料都是将 Zookeeper 描述成一个“类 Unix/Linu ...

  3. 【连载】redis库存操作，分布式锁的四种实现方式[一]--基于zookeeper实现分布式锁

    一.背景 在电商系统中,库存的概念一定是有的,例如配一些商品的库存,做商品秒杀活动等,而由于库存操作频繁且要求原子性操作,所以绝大多数电商系统都用Redis来实现库存的加减,最近公司项目做架构升级,以 ...

  4. 基于Zookeeper的分布式锁(干干干货)

    原文地址: https://juejin.im/post/5df883d96fb9a0163514d97f 介绍 为什么使用锁 锁的出现是为了解决资源争用问题,在单进程环境下的资源争夺可以使用 JDK ...

  5. 基于Zookeeper的分布式锁

    实现分布式锁目前有三种流行方案,分别为基于数据库.Redis.Zookeeper的方案,其中前两种方案网络上有很多资料可以参考,本文不做展开.我们来看下使用Zookeeper如何实现分布式锁. 什么是 ...

  6. 10分钟看懂！基于Zookeeper的分布式锁

    实现分布式锁目前有三种流行方案,分别为基于数据库.Redis.Zookeeper的方案,其中前两种方案网络上有很多资料可以参考,本文不做展开.我们来看下使用Zookeeper如何实现分布式锁. 什么是 ...

  7. 基于ZooKeeper实现——分布式锁与实现

    引言 ZooKeeper是一个分布式的,开放源码的分布式应用程序协调服务,是Google的Chubby一个开源的实现,是Hadoop和Hbase的重要组件.它是一个为分布式应用提供一致性服务的软件,提 ...

  8. 基于zookeeper实现分布式锁和基于redis实现分布所的区别

    1,实现方式不同 zookeeper实现分布式锁:通过创建一个临时节点,创建的成功节点的服务则抢占到分布式锁,可做业务逻辑.当业务逻辑完成,连接中断,节点消失,继续下一轮的锁的抢占. redis实现分 ...

  9. 基于zookeeper实现分布式锁

    Zookeeper是一个分布式的,开放源码的分布式应用程序协调服务,是Hadoop和Hbase的重要组件. 特性: 1.节点数据结构,znode是一个跟Unix文件系统路径相似的节点,可以往这个节点存 ...

随机推荐

  1. 【转】Python实现不同格式打印九九乘法表

    前言:最近在学习Python,学习资源有慕课网上的视频教程.菜鸟教程以及Python官方文档tutorial.虽然了解了Python的基本语法,但是还没有真正意义上输出自己写的代码.代码小白,之前仅学 ...

  2. PHP中的全局变量global和$GLOBALS的区别

    1.global Global的作用是定义全局变量,但是这个全局变量不是应用于整个网站,而是应用于当前页面,包括include或require的所有文件. 但是在函数体内定义的global变量,函数体 ...

  3. [Android]优化相关

    尽量减少布局的层次,最多10层,可以通过LinearLayout向RelativeLayout的转变来减少层的数量 使用ListView的时候,getView方法中的对象尽量重用

  4. SCI英文论文写作- Latex 进阶

    SCI英文论文写作- Latex 进阶   1.设置行间距的方法: %\setlength{\baselineskip}{15pt} \renewcommand{\baselinestretch}{1 ...

  5. HTML5 中的 canvas 画布（二）

    绘制图片 一.绘制图片 context.drawImage()(即把图片放到canvas里) var image = new Image();  // 先创建图片对象 image.src = '图片的 ...

  6. （转）C#根据当前时间获取周，月，季度，年度等时间段的起止时间

    DateTime dt = DateTime.Now; //当前时间 DateTime startWeek = dt.AddDays( - Convert.ToInt32(dt.DayOfWeek.T ...

  7. 【JavaScript】JS 中 原始字符串 和 HTML 字符转换

    参考资料:http://www.sjyhome.com/javascript/js-html-escape.html JS转换HTML转义符 SJY • 发表于:2013年10月05日 17:04 • ...

  8. WebService -- Java 实现之 CXF ( 使用Spring添加拦截器）

    最重要的就是在ApplicationContext.xml下面添加配置 <!-- service provider --> <jaxws:endpoint implementor=& ...

  9. OC编程之道-创建对象之抽象工厂方法

    定义:提供一个创建一系列相关或相互依赖对象的接口,而无需指定他们具体的类.       <AbstractProductA> <AbstractProductB> <Ab ...

  10. ABAP 将单元格设置编辑状态 FORM

    FORM set_style  USING   fieldname                         style TYPE string                 CHANGING ...