Apache Kafka源码分析 - autoLeaderRebalanceEnable
在broker的配置中,auto.leader.rebalance.enable (false)
那么这个leader是如何进行rebalance的?
首先在controller启动的时候会打开一个scheduler,
if (config.autoLeaderRebalanceEnable) { //如果打开outoLeaderRebalance,需要把partiton leader由于dead而发生迁徙的,重新迁徙回去
info("starting the partition rebalance scheduler")
autoRebalanceScheduler.startup()
autoRebalanceScheduler.schedule("partition-rebalance-thread", checkAndTriggerPartitionRebalance,
5, config.leaderImbalanceCheckIntervalSeconds, TimeUnit.SECONDS)
}
定期去做,
checkAndTriggerPartitionRebalance
这个函数逻辑,就是找出所有发生过迁移的replica,即
topicsNotInPreferredReplica
并且判断如果满足imbalance比率,即自动触发leader rebalance,将leader迁回perfer replica
关键要理解什么是preferred replicas?
preferredReplicasForTopicsByBrokers =
controllerContext.partitionReplicaAssignment.filterNot(p => deleteTopicManager.isTopicQueuedUpForDeletion(p._1.topic)).groupBy {
case(topicAndPartition, assignedReplicas) => assignedReplicas.head
}
partitionReplicaAssignment: mutable.Map[TopicAndPartition, Seq[Int]]
TopicAndPartition可以通过topic name和partition id来唯一标识一个partition,Seq[int],表示brokerids,表明这个partition的replicas在哪些brokers上面
从partition的ReplicaAssignment里面过滤掉delete的topic,然后按照assignedReplicas.head进行groupby,就是按照Seq中的第一个brokerid
意思就是说,默认每个partition的preferred replica就是第一个被assign的replica
groupby的结果就是,每个broker,和应该以该broker作为leader的所有partition,即
case(leaderBroker, topicAndPartitionsForBroker)
那么找出里面当前leader不是preferred的,即发生过迁移的,
很简单,直接和leaderAndIsr里面的leader进行比较,如果不相等就说明发生过迁徙
topicsNotInPreferredReplica =
topicAndPartitionsForBroker.filter {
case(topicPartition, replicas) => {
controllerContext.partitionLeadershipInfo.contains(topicPartition) &&
controllerContext.partitionLeadershipInfo(topicPartition).leaderAndIsr.leader != leaderBroker
}
}
并且只有当某个broker上的imbalanceRatio大于10%的时候,才会触发rebalance
imbalanceRatio = totalTopicPartitionsNotLedByBroker.toDouble / totalTopicPartitionsForBroker
对每个partition的迁移过程,
首先preferred的broker要是活着的,并且当前是没有partition正在进行reassign或replica election的,说明这个过程是不能并行的,同时做reassign很容易冲突
// do this check only if the broker is live and there are no partitions being reassigned currently
// and preferred replica election is not in progress
if (controllerContext.liveBrokerIds.contains(leaderBroker) &&
controllerContext.partitionsBeingReassigned.size == 0 &&
controllerContext.partitionsUndergoingPreferredReplicaElection.size == 0 &&
!deleteTopicManager.isTopicQueuedUpForDeletion(topicPartition.topic) &&
controllerContext.allTopics.contains(topicPartition.topic)) {
onPreferredReplicaElection(Set(topicPartition), true)
onPreferredReplicaElection
还是通过partitionStateMachine,来改变partition的状态
partitionStateMachine.handleStateChanges(partitions, OnlinePartition, preferredReplicaPartitionLeaderSelector)
partitionStateMachine会另外分析,这里只需要知道,当前partition的状态是,OnlinePartition –> OnlinePartition
并且是以preferredReplicaPartitionLeaderSelector,作为leaderSelector的策略
PreferredReplicaPartitionLeaderSelector
策略很简单,就是把leader换成preferred replica
def selectLeader(topicAndPartition: TopicAndPartition, currentLeaderAndIsr: LeaderAndIsr): (LeaderAndIsr, Seq[Int]) = {
val assignedReplicas = controllerContext.partitionReplicaAssignment(topicAndPartition)
val preferredReplica = assignedReplicas.head //取AR第一个replica作为preferred
// check if preferred replica is the current leader
val currentLeader = controllerContext.partitionLeadershipInfo(topicAndPartition).leaderAndIsr.leader
if (currentLeader == preferredReplica) { //如果当前leader就是preferred就不需要做了
throw new LeaderElectionNotNeededException("Preferred replica %d is already the current leader for partition %s" .format(preferredReplica, topicAndPartition))
} else {
info("Current leader %d for partition %s is not the preferred replica.".format(currentLeader, topicAndPartition) + " Trigerring preferred replica leader election")
// check if preferred replica is not the current leader and is alive and in the isr
if (controllerContext.liveBrokerIds.contains(preferredReplica) && currentLeaderAndIsr.isr.contains(preferredReplica)) { //判断当前preferred replica所在broker是否活,是否在isr中
(new LeaderAndIsr(preferredReplica, currentLeaderAndIsr.leaderEpoch + 1, currentLeaderAndIsr.isr, currentLeaderAndIsr.zkVersion + 1), assignedReplicas) //产生新的leaderAndIsr
} else {
throw new StateChangeFailedException("Preferred replica %d for partition ".format(preferredReplica) +
"%s is either not alive or not in the isr. Current leader and ISR: [%s]".format(topicAndPartition, currentLeaderAndIsr))
}
}
}
}
Apache Kafka源码分析 - autoLeaderRebalanceEnable的更多相关文章
- Apache Kafka源码分析 – Broker Server
1. Kafka.scala 在Kafka的main入口中startup KafkaServerStartable, 而KafkaServerStartable这是对KafkaServer的封装 1: ...
- apache kafka源码分析-Producer分析---转载
原文地址:http://www.aboutyun.com/thread-9938-1-1.html 问题导读1.Kafka提供了Producer类作为java producer的api,此类有几种发送 ...
- Apache Kafka源码分析 - kafka controller
前面已经分析过kafka server的启动过程,以及server所能处理的所有的request,即KafkaApis 剩下的,其实关键就是controller,以及partition和replica ...
- Apache Kafka源码分析 – Controller
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Controller+Internalshttps://cwiki.apache.org ...
- Apache Kafka源码分析 – Log Management
LogManager LogManager会管理broker上所有的logs(在一个log目录下),一个topic的一个partition对应于一个log(一个log子目录)首先loadLogs会加载 ...
- Apache Kafka源码分析 - KafkaApis
kafka apis反映出kafka broker server可以提供哪些服务,broker server主要和producer,consumer,controller有交互,搞清这些api就清楚了 ...
- Apache Kafka源码分析 – Replica and Partition
Replica 对于local replica, 需要记录highWatermarkValue,表示当前已经committed的数据对于remote replica,需要记录logEndOffsetV ...
- Apache Kafka源码分析 - ReplicaStateMachine
startup 在onControllerFailover中被调用, /** * Invoked on successful controller election. First registers ...
- Apache Kafka源码分析 - PartitionStateMachine
startup 在onControllerFailover中被调用, initializePartitionState private def initializePartitionState() { ...
随机推荐
- ThinkPHP的增、删、改、查
一.创建操作 在ThinkPHP使用add方法新增数据到数据库. 使用方法如下: $User = M()->; $data[););')->delete();
- PL/SQL Developer 连接新数据库
1 2 3
- uva 562
Description It's commonly known that the Dutch have invented copper-wire. Two Dutch men were fight ...
- aac格式解析
AAC格式有以下两种: ADIF:Audio Data Interchange Format 音频数据交换格式.这种格式的特征是可以确定的找到这个音频数据的开始,不需进行在音频数据流中间开始的解码,即 ...
- 安装Kali Linux操作系统Kali Linux无线网络渗透
安装Kali Linux操作系统Kali Linux无线网络渗透 Kali Linux是一个基于Debian的Linux发行版,它的前身是BackTrack Linux发行版.在该操作系统中,自带了大 ...
- 阿里云ECS(云服务器)之产品简介
参考阿里产品文档:https://docs.aliyun.com/?spm=5176.100054.3.1.ywnrMX#/pub/ecs/product-introduction/concept
- LCIS HDOJ 4512 吉哥系列故事——完美队形I
题目传送门 题意:中文题面 分析:LCIS应用:设置b[]为a[]的反转,然后LCIS,若相等的是自己本身,则+1, 否则+2 代码: #include <cstdio> #include ...
- LightOJ1031 Easy Game(区间DP)
我可能真想不到这题是区间DP,不过知道是区间DP想了下就AC了. dp[i][j]表示局面为ai...aj先手能获得与后手得分的最大差值 那么转移到当前状态就是枚举中间的位置,分成两边,其中一边先手全 ...
- 操作properties文件,注意抹掉最前面的"file:"
package com.xiewanzhi.property; import java.io.BufferedInputStream; import java.io.File; import java ...
- BZOJ2757 : [SCOI2012]Blinker的仰慕者
BZOJ AC900题纪念~~ 若K>0,则 设f[i][j]表示i位数字,积为j的数字的个数 g[i][j]表示i位数字,积为j的数字的和 DP+Hash预处理 查询时枚举LCP然后统计贡献 ...