Akka源码分析-Cluster-Singleton

　　akka Cluster基本实现原理已经分析过，其实它就是在remote基础上添加了gossip协议，同步各个节点信息，使集群内各节点能够识别。在Cluster中可能会有一个特殊的节点，叫做单例节点。也就是具有某个角色的节点在集群中只能有一个，如果这个节点宕机了，需要把这个角色的工作转移到其他节点。

　　使用单例模式有几个场景。

• 负责集群的一致性决策，跨集群协调工作。比如集群事务。
• 对外部系统的单实例接入。
• 一个master，多个worker
• 集中化的命名服务或路由策略。

　　官网说单例对象在设计时，永远不要成为第一个选择，它可能存在单点性能瓶颈、单点故障等多个问题。这点我是赞同的，这年头都是分布式为王，当你考虑单点的时候，有两种情况：设计错误、业务特殊。单例虽然有那么多缺点，但还是有点用处的，源码分析还是要做的，我们要知其然还要知其所以然。

　　akka的单例模式由akka.cluster.singleton.ClusterSingletonManager.实现，它在集群或角色相同的一组节点管理一个单例actor实例。ClusterSingletonManager 是集群或角色相同节点中最先被启动的actor 。ClusterSingletonManager 会在所有节点中运行时间最长的节点上启动，如果这个节点宕机，则重新选择运行时间最长的节点。如果你要问运行时间最长的信息从哪里获取？我不知道，哈哈，这肯定是在通过gossip协议广播的Member消息中呗。

　　那么问题来了，既然ClusterSingletonManager 的位置会发生漂移，那我怎么跟它通信呢？akka还是很贴心的，它帮你实现了一个akka.cluster.singleton.ClusterSingletonProxy，这个actor会把消息路由给当前的ClusterSingletonManager 单例对象。简单来说就是ClusterSingletonProxy会给运行时间最长的节点发送一个akka.actor.Identify消息，查询单例actor的actorRef。当然了，在查找的期间消息是被缓冲的，超过缓冲大小直接丢弃。读者需要注意的是，akka只是尽量保证同一时刻最多有一个ClusterSingletonManager 单例，但并不保证同一时刻有且仅有一个。

    system.actorOf(
      ClusterSingletonManager.props(
        singletonProps = Props(classOf[Consumer], queue, testActor),
        terminationMessage = End,
        settings = ClusterSingletonManagerSettings(system).withRole("worker")),
      name = "consumer")

　　同样的，我们还是从官网demo入手。上面是ClusterSingletonManager的创建过程，很明显这是创建了一个actor，这个actor内部还有一个Props，也就是会创建对应的子actor：Consumer。ClusterSingletonManager会保证在集群中的Consumer实例只有一个。

    val proxy = system.actorOf(
      ClusterSingletonProxy.props(
        singletonManagerPath = "/user/consumer",
        settings = ClusterSingletonProxySettings(system).withRole("worker")),
      name = "consumerProxy")

　　上面创建了ClusterSingletonProxy，这个proxy路由消息给集群中path为/user/consumer的单例对象，当然了ClusterSingletonProxy是可以有多个的。

　　下面切入正题，来看看ClusterSingletonManager和ClusterSingletonProxy的源码。首先这两个包是在cluster-tools这个包里面。

/**
 * Manages singleton actor instance among all cluster nodes or a group
 * of nodes tagged with a specific role. At most one singleton instance
 * is running at any point in time.
 *
 * The ClusterSingletonManager is supposed to be started on all nodes,
 * or all nodes with specified role, in the cluster with `actorOf`.
 * The actual singleton is started on the oldest node by creating a child
 * actor from the supplied `singletonProps`.
 *
 * The singleton actor is always running on the oldest member with specified role.
 * The oldest member is determined by [[akka.cluster.Member#isOlderThan]].
 * This can change when removing members. A graceful hand over can normally
 * be performed when current oldest node is leaving the cluster. Be aware that
 * there is a short time period when there is no active singleton during the
 * hand-over process.
 */
@DoNotInherit
class ClusterSingletonManager(
  singletonProps:     Props,
  terminationMessage: Any,
  settings:           ClusterSingletonManagerSettings)
  extends Actor with FSM[ClusterSingletonManager.State, ClusterSingletonManager.Data]

　　上面是ClusterSingletonManager的定义，官方注释很多，我只保留了最重要的三段。第一段介绍了什么是单例对象；第二段说ClusterSingletonManager需要在所有的节点启动，它管理的单例对象在集群中只会有一个；第三段说这个单例actor只会在运行时间最长的节点上，重启过程中单例不可用。其实第二点值得我们深究，既然ClusterSingletonManager需要在所有的节点启动，那怎么保持单例呢。其实吧，正如它的名字那样，它只是个管理者，真正的单例actor是通过singletonProps创建的。也就是说，ClusterSingletonManager在节点中互相协调，具体由谁创建整个单例actor。看到这里我都想自己实现一个单例actor了。

　　还要注意的是这个manager继承了FSM接口，应该是为了协调单例的状态什么的。既然这是个actor，那就需要看看这个actor启动时，做了哪些初始化。

 override def preStart(): Unit = {
    super.preStart()
    require(!cluster.isTerminated, "Cluster node must not be terminated")

    // subscribe to cluster changes, re-subscribe when restart
    cluster.subscribe(self, ClusterEvent.InitialStateAsEvents, classOf[MemberRemoved])

    setTimer(CleanupTimer, Cleanup, 1.minute, repeat = true)

    // defer subscription to avoid some jitter when
    // starting/joining several nodes at the same time
    cluster.registerOnMemberUp(self ! StartOldestChangedBuffer)
  }

　　简单来说就是订阅了ClusterEvent.InitialStateAsEvents、MemberRemoved、MemberUp消息，启动了一个timer。这个timer每个1分钟发送一个Cleanup消息，timer的名字是CleanupTimer。收到MemberUp消息会给自己发送StartOldestChangedBuffer。初始化是不是很简单？想想都简单，这就是一个普通actor，只不过需要跟其他actor协调一下，那个是运行时间最久的actor，然后再加上单例actor漂移的功能就行了。

  startWith(Start, Uninitialized)

　　主构造函数中还有上面一段代码需要注意，这是FSM的函数，之前分析过，这是用来设置初始状态的，也就是定义当前ctor的receive。

 when(Start) {
    case Event(StartOldestChangedBuffer, _) ⇒
      oldestChangedBuffer = context.actorOf(Props(classOf[OldestChangedBuffer], role).
        withDispatcher(context.props.dispatcher))
      getNextOldestChanged()
      stay

    case Event(InitialOldestState(oldestOption, safeToBeOldest), _) ⇒
      oldestChangedReceived = true
      if (oldestOption == selfUniqueAddressOption && safeToBeOldest)
        // oldest immediately
        gotoOldest()
      else if (oldestOption == selfUniqueAddressOption)
        goto(BecomingOldest) using BecomingOldestData(None)
      else
        goto(Younger) using YoungerData(oldestOption)
  }

　　很显然Start状态会收到StartOldestChangedBuffer消息。

// started when when self member is Up
  var oldestChangedBuffer: ActorRef = _

　　首先创建了一个OldestChangedBuffer，这个actor会在当前节点变成up状态时启动，作用未知，然后就调用getNextOldestChanged。

/**
     * Notifications of member events that track oldest member is tunneled
     * via this actor (child of ClusterSingletonManager) to be able to deliver
     * one change at a time. Avoiding simultaneous changes simplifies
     * the process in ClusterSingletonManager. ClusterSingletonManager requests
     * next event with `GetNext` when it is ready for it. Only one outstanding
     * `GetNext` request is allowed. Incoming events are buffered and delivered
     * upon `GetNext` request.
     */
    class OldestChangedBuffer(role: Option[String]) extends Actor

　　OldestChangedBuffer官方注释也很清晰，这个actor就是用来监听集群事件，用来判断运行时间最久的节点的。

 def getNextOldestChanged(): Unit =
    if (oldestChangedReceived) {
      oldestChangedReceived = false
      oldestChangedBuffer ! GetNext
    }

　　getNextOldestChanged也比较简单就是给oldestChangedBuffer发送了GetNext消息。

def receive = {
        case state: CurrentClusterState ⇒ handleInitial(state)
        case MemberUp(m)                ⇒ add(m)
        case MemberRemoved(m, _)        ⇒ remove(m)
        case MemberExited(m) if m.uniqueAddress != cluster.selfUniqueAddress ⇒
          remove(m)
        case SelfExiting ⇒
          remove(cluster.readView.self)
          sender() ! Done // reply to ask
        case GetNext if changes.isEmpty ⇒
          context.become(deliverNext, discardOld = false)
        case GetNext ⇒
          sendFirstChange()
      }

　　oldestChangedBuffer收到GetNext消息，会判断changes是不是为空，如果为空，则简单的进行beconme，否则调用sendFirstChange。那changes什么时候赋值呢？

def trackChange(block: () ⇒ Unit): Unit = {
        val before = membersByAge.headOption
        block()
        val after = membersByAge.headOption
        if (before != after)
          changes :+= OldestChanged(after.map(_.uniqueAddress))
      }

      def handleInitial(state: CurrentClusterState): Unit = {
        membersByAge = immutable.SortedSet.empty(ageOrdering) union state.members.filter(m ⇒
          (m.status == MemberStatus.Up || m.status == MemberStatus.Leaving) && matchingRole(m))
        val safeToBeOldest = !state.members.exists { m ⇒ (m.status == MemberStatus.Down || m.status == MemberStatus.Exiting) }
        val initial = InitialOldestState(membersByAge.headOption.map(_.uniqueAddress), safeToBeOldest)
        changes :+= initial
      }

　　简要分析之后，可以发现OldestChangedBuffer在收到集群的相关消息时会调用上面两个函数更改changes的值，其实就是保存运行时间最长节点的信息。但不论哪种情况，其实最终都是会调用sendFirstChange方法。

def sendFirstChange(): Unit = {
        // don't send cluster change events if this node is shutting its self down, just wait for SelfExiting
        if (!cluster.isTerminated) {
          val event = changes.head
          changes = changes.tail
          context.parent ! event
        }
      }

　　这个函数的功能不作过多介绍，其实就是获取集群节点信息，获取运行时间最长的节点信息，发送给context.parent也就是ClusterSingletonManager，只不过会区分当前节点是不是第一个启动的节点，但无论是不是第一个启动，区别都不大。

　　此时ClusterSingletonManager还在Start状态，会收到InitialOldestState消息。假设当前节点就是最老的，那么会调用gotoOldest

def gotoOldest(): State = {
    val singleton = context watch context.actorOf(singletonProps, singletonName)
    logInfo("Singleton manager starting singleton actor [{}]", singleton.path)
    goto(Oldest) using OldestData(singleton)
  }

　　可以看到，如果当前节点是最老的，那么会创建singletonProps，然后转入Oldest状态，且FSM数据就是这个singletonProps的ActorRef。那么后面启动的节点，同样会执行上面的流程，只不过它判断当前节点是最老节点时会失败，不会执行gotoOldest方法，也就不会创建singletonProps，而是转到Younger状态。

when(Younger) {
    case Event(OldestChanged(oldestOption), YoungerData(previousOldestOption)) ⇒
      oldestChangedReceived = true
      if (oldestOption == selfUniqueAddressOption) {
        logInfo("Younger observed OldestChanged: [{} -> myself]", previousOldestOption.map(_.address))
        previousOldestOption match {
          case None                                 ⇒ gotoOldest()
          case Some(prev) if removed.contains(prev) ⇒ gotoOldest()
          case Some(prev) ⇒
            peer(prev.address) ! HandOverToMe
            goto(BecomingOldest) using BecomingOldestData(previousOldestOption)
        }
      } else {
        logInfo("Younger observed OldestChanged: [{} -> {}]", previousOldestOption.map(_.address), oldestOption.map(_.address))
        getNextOldestChanged()
        stay using YoungerData(oldestOption)
      }

    case Event(MemberRemoved(m, _), _) if m.uniqueAddress == cluster.selfUniqueAddress ⇒
      logInfo("Self removed, stopping ClusterSingletonManager")
      stop()

    case Event(MemberRemoved(m, _), _) ⇒
      scheduleDelayedMemberRemoved(m)
      stay

    case Event(DelayedMemberRemoved(m), YoungerData(Some(previousOldest))) if m.uniqueAddress == previousOldest ⇒
      logInfo("Previous oldest removed [{}]", m.address)
      addRemoved(m.uniqueAddress)
      // transition when OldestChanged
      stay using YoungerData(None)

    case Event(HandOverToMe, _) ⇒
      // this node was probably quickly restarted with same hostname:port,
      // confirm that the old singleton instance has been stopped
      sender() ! HandOverDone
      stay
  }

　　如果最老节点发生变化，且当前节点不是最老节点，而当前节点变成了最老节点，会发生什么呢？会命中Younger状态的第一个case店里面的第一个if语句。也就是会给原来的最老节点发送HandOverToMe消息 ，自己变成BecomingOldest状态。

def gotoHandingOver(singleton: ActorRef, singletonTerminated: Boolean, handOverTo: Option[ActorRef]): State = {
    if (singletonTerminated) {
      handOverDone(handOverTo)
    } else {
      handOverTo foreach { _ ! HandOverInProgress }
      singleton ! terminationMessage
      goto(HandingOver) using HandingOverData(singleton, handOverTo)
    }
  }

　　此时原来的最老节点，会调用上面的函数，简单来说既是给原来的singleton这个单例actor发送terminationMessage消息，当然，单例actor收到这个消息后可自行处理，但最终都需要stop。这样，才能给最新的最老节点发送HandOverDone消息。而此时最新的最老节点处于BecomingOldest状态，收到消息后命中以下case。

 case Event(HandOverDone, BecomingOldestData(Some(previousOldest))) ⇒
      if (sender().path.address == previousOldest.address)
        gotoOldest()
      else {
        logInfo(
          "Ignoring HandOverDone in BecomingOldest from [{}]. Expected previous oldest [{}]",
          sender().path.address, previousOldest.address)
        stay
      }

　　也比较简单，就是调用gotoOldest，这个方法的功能之前说过了。

　　但我发现了一个比较重要的问题，那就是如果当前节点处于Oldest状态，而创建的单例对象actor以外终止了，会发生什么呢？从Oldest的处理来看，它就是修改了当前状态数据的singletonTerminated为true，没有其他操作。难道不应该是重启么？很奇怪啊，不管了，先继续分析吧。

/**
 * The `ClusterSingletonProxy` works together with the [[akka.cluster.singleton.ClusterSingletonManager]] to provide a
 * distributed proxy to the singleton actor.
 *
 * The proxy can be started on every node where the singleton needs to be reached and used as if it were the singleton
 * itself. It will then act as a router to the currently running singleton instance. If the singleton is not currently
 * available, e.g., during hand off or startup, the proxy will buffer the messages sent to the singleton and then deliver
 * them when the singleton is finally available. The size of the buffer is configurable and it can be disabled by using
 * a buffer size of 0. When the buffer is full old messages will be dropped when new messages are sent via the proxy.
 *
 * The proxy works by keeping track of the oldest cluster member. When a new oldest member is identified, e.g. because
 * the older one left the cluster, or at startup, the proxy will try to identify the singleton on the oldest member by
 * periodically sending an [[akka.actor.Identify]] message until the singleton responds with its
 * [[akka.actor.ActorIdentity]].
 *
 * Note that this is a best effort implementation: messages can always be lost due to the distributed nature of the
 * actors involved.
 */
final class ClusterSingletonProxy(singletonManagerPath: String, settings: ClusterSingletonProxySettings) extends Actor with ActorLogging

　　官方注释也很详细，ClusterSingletonProxy也会跟踪当前集群的最老节点，然后通过singletonManagerPath进行actorSelection获取单例actor的ActorRef。ClusterSingletonProxy跟OldestChangedBuffer关于集群消息的部分，功能框架有点类似，都是在集群节点变化时调用handleInitial、add、remove三个函数，不同的是这三个函数在ClusterSingletonProxy中还调用了trackChange。

def trackChange(block: () ⇒ Unit): Unit = {
    val before = membersByAge.headOption
    block()
    val after = membersByAge.headOption
    // if the head has changed, I need to find the new singleton
    if (before != after) identifySingleton()
  }

　　trackChange发现节点有变化时，会调用identifySingleton

def identifySingleton() {
    import context.dispatcher
    log.debug("Creating singleton identification timer...")
    identifyCounter += 1
    identifyId = createIdentifyId(identifyCounter)
    singleton = None
    cancelTimer()
    identifyTimer = Some(context.system.scheduler.schedule(0 milliseconds, singletonIdentificationInterval, self, ClusterSingletonProxy.TryToIdentifySingleton))
  }

　　其实就是启动了一个定时器，每隔singletonIdentificationInterval秒给自己发送ClusterSingletonProxy.TryToIdentifySingleton消息，为啥需要间隔时间重复发送了，这主要是为了防止单例还在漂移，无法回应Identify消息。

case ClusterSingletonProxy.TryToIdentifySingleton ⇒
      identifyTimer match {
        case Some(_) ⇒
          membersByAge.headOption foreach { oldest ⇒
            val singletonAddress = RootActorPath(oldest.address) / singletonPath
            log.debug("Trying to identify singleton at [{}]", singletonAddress)
            context.actorSelection(singletonAddress) ! Identify(identifyId)
          }
        case _ ⇒
        // ignore, if the timer is not present it means we have successfully identified
      }

　　很显然收到这个消息，会通过actorSelection发送Identify消息。

case ActorIdentity(identifyId, Some(s)) ⇒
      // if the new singleton is defined, deliver all buffered messages
      log.info("Singleton identified at [{}]", s.path)
      singleton = Some(s)
      context.watch(s)
      cancelTimer()
      sendBuffered()

　　收到ActorIdentity消息后，会取消timer，然后把缓存的消息发送出去。

case msg: Any ⇒
      singleton match {
        case Some(s) ⇒
          if (log.isDebugEnabled)
            log.debug(
              "Forwarding message of type [{}] to current singleton instance at [{}]: {}",
              Logging.simpleName(msg.getClass), s.path)
          s forward msg
        case None ⇒
          buffer(msg)
      }

　　如果singleton已经有值，收到消息后会命中上面的case分支，其实就是简单的forward消息。

　　好了，到此为止，单例模式基本就解释清楚了，其实源码也不是非常复杂，如果不考虑到通用、健壮性，自己实现一个获取更高效。而且在ClusterSingletonManager中，发现单例actor已经stop了，居然没有任何补偿措施，这真是恶心啊，说好的高可用呢，说好的故障恢复呢。另外ClusterSingletonManager需要自己单独的用actorOf进行创建，为啥不用一个扩展来实现呢，简单、方便、易管理。

Cluster Singleton