(六)storm-kafka源代码走读之PartitionManager
PartitionManager算是storm-kafka的核心类了,如今開始简单分析一下。还是先声明一下,metric部分这里不做分析。
PartitionManager主要负责的是消息的发送、容错处理,所以PartitionManager会有三个集合
- _pending:尚未发送的message的offset集合, 是个TreeSet<Long>()
- failed : 发送失败的offset 集合,是个TreeSet<Long>()
- _waitingToEmit: 存放待发射的message,是个LinkedList。从kafka读到的message所有先放在这里
- Long _emittedToOffset; 从kafka读到的offset。从kafka读到的messages会放入_waitingToEmit,放入这个list。我们就觉得一定会被emit,所以emittedToOffset能够觉得是从kafka读到的offset
- Long _committedTo; 已经写入zk的offset
- lastCompletedOffset() 已经被成功处理的offset,因为message是要在storm里面处理的,当中是可能fail的,所以正在处理的offset是缓存在_pending中的
假设_pending为空,那么lastCompletedOffset=_emittedToOffset
假设_pending不为空,那么lastCompletedOffset为pending list里面第一个offset,由于后面都还在等待ack
public long lastCompletedOffset() {
if (_pending.isEmpty()) {
return _emittedToOffset;
} else {
return _pending.first();
}
}
来看初始化的过程:
public PartitionManager(DynamicPartitionConnections connections, String topologyInstanceId, ZkState state, Map stormConf, SpoutConfig spoutConfig, Partition id) {
_partition = id;
_connections = connections;
_spoutConfig = spoutConfig;
_topologyInstanceId = topologyInstanceId;
_consumer = connections.register(id.host, id.partition);
_state = state;
_stormConf = stormConf;
numberAcked = numberFailed = 0; String jsonTopologyId = null;
Long jsonOffset = null;
String path = committedPath();
try {
Map<Object, Object> json = _state.readJSON(path); // 从zk读取offset
LOG.info("Read partition information from: " + path + " --> " + json );
if (json != null) {
jsonTopologyId = (String) ((Map<Object, Object>) json.get("topology")).get("id");
jsonOffset = (Long) json.get("offset");
}
} catch (Throwable e) {
LOG.warn("Error reading and/or parsing at ZkNode: " + path, e);
} /**
* 依据用户设置的startOffsetTime,值来读取offset(-2 从kafka头開始 -1 是从最新的開始 0 =无 从ZK開始)
**/
Long currentOffset = KafkaUtils.getOffset(_consumer, spoutConfig.topic, id.partition, spoutConfig); if (jsonTopologyId == null || jsonOffset == null) { // failed to parse JSON?
_committedTo = currentOffset;
LOG.info("No partition information found, using configuration to determine offset");
} else if (!topologyInstanceId.equals(jsonTopologyId) && spoutConfig.forceFromStart) {
_committedTo = KafkaUtils.getOffset(_consumer, spoutConfig.topic, id.partition, spoutConfig.startOffsetTime);
LOG.info("Topology change detected and reset from start forced, using configuration to determine offset");
} else {
_committedTo = jsonOffset;
LOG.info("Read last commit offset from zookeeper: " + _committedTo + "; old topology_id: " + jsonTopologyId + " - new topology_id: " + topologyInstanceId );
} /**
* 以下这个if推断是假设当前读取的offset值与提交到zk的值不一致,且相差Long.MAX_VALUE,就觉得中间非常大部分msg发射了没有提交,就把这部分所有放弃,避免重发
* 令_committedTo = currentOffset, 这个是新修复的bug。之前maxOffsetBehind=100000(好像是这个值,这个太小),bug isuue 是 STORM-399 **/
if (currentOffset - _committedTo > spoutConfig.maxOffsetBehind || _committedTo <= 0) {
LOG.info("Last commit offset from zookeeper: " + _committedTo);
_committedTo = currentOffset;
LOG.info("Commit offset " + _committedTo + " is more than " +
spoutConfig.maxOffsetBehind + " behind, resetting to startOffsetTime=" + spoutConfig.startOffsetTime);
} LOG.info("Starting Kafka " + _consumer.host() + ":" + id.partition + " from offset " + _committedTo);
_emittedToOffset = _committedTo; _fetchAPILatencyMax = new CombinedMetric(new MaxMetric());
_fetchAPILatencyMean = new ReducedMetric(new MeanReducer());
_fetchAPICallCount = new CountMetric();
_fetchAPIMessageCount = new CountMetric();
}
刚開始的时候须要读取message,放到_waitingToEmit中。这是fill的过程,看代码
private void fill() {
long start = System.nanoTime();
long offset; // 首先要推断是否有fail的offset, 假设有的话,在须要从这个offset開始往下去读取message,所以这里有重发的可能
final boolean had_failed = !failed.isEmpty(); // Are there failed tuples? If so, fetch those first.
if (had_failed) {
offset = failed.first(); // 取失败的最小的offset值。
} else {
offset = _emittedToOffset;
} ByteBufferMessageSet msgs = KafkaUtils.fetchMessages(_spoutConfig, _consumer, _partition, offset);
long end = System.nanoTime();
long millis = (end - start) / 1000000;
_fetchAPILatencyMax.update(millis);
_fetchAPILatencyMean.update(millis);
_fetchAPICallCount.incr();
if (msgs != null) {
int numMessages = 0; for (MessageAndOffset msg : msgs) {
final Long cur_offset = msg.offset();
if (cur_offset < offset) {
// Skip any old offsets.
continue;
} /**
* 仅仅要是没有失败的或者失败的set中含有该offset(由于失败msg有非常多,我们仅仅是从最小的offset開始读取msg的)
* ,就把这个message放到待发射的list中
**/
if (!had_failed || failed.contains(cur_offset)) {
numMessages += 1;
_pending.add(cur_offset);
_waitingToEmit.add(new MessageAndRealOffset(msg.message(), cur_offset));
_emittedToOffset = Math.max(msg.nextOffset(), _emittedToOffset);
if (had_failed) { // 假设失败列表中含有该offset,就移除,由于要又一次发射了。
failed.remove(cur_offset);
}
}
}
_fetchAPIMessageCount.incrBy(numMessages);
}
}
ByteBufferMessageSet msgs = KafkaUtils.fetchMessages(_spoutConfig, _consumer, _partition, offset);
改动成:
ByteBufferMessageSet msgs = null;
try {
msgs = KafkaUtils.fetchMessages(_spoutConfig, _consumer, _partition, offset);
} catch (UpdateOffsetException e) {
_emittedToOffset = KafkaUtils.getOffset(_consumer, _spoutConfig.topic, _partition.partition, _spoutConfig);
LOG.warn("Using new offset: {}", _emittedToOffset);
// fetch failed, so don't update the metrics
return;
}
public static ByteBufferMessageSet fetchMessages(KafkaConfig config, SimpleConsumer consumer, Partition partition, long offset) {
ByteBufferMessageSet msgs = null;
String topic = config.topic;
int partitionId = partition.partition;
for (int errors = 0; errors < 2 && msgs == null; errors++) { //容忍两次错误
FetchRequestBuilder builder = new FetchRequestBuilder();
FetchRequest fetchRequest = builder.addFetch(topic, partitionId, offset, config.fetchSizeBytes).
clientId(config.clientId).maxWait(config.fetchMaxWait).build();
FetchResponse fetchResponse;
try {
fetchResponse = consumer.fetch(fetchRequest);
} catch (Exception e) {
if (e instanceof ConnectException ||
e instanceof SocketTimeoutException ||
e instanceof IOException ||
e instanceof UnresolvedAddressException
) {
LOG.warn("Network error when fetching messages:", e);
throw new FailedFetchException(e);
} else {
throw new RuntimeException(e);
}
}
if (fetchResponse.hasError()) { // 主要处理offset outofrange的case。通过getOffset从earliest或latest读
KafkaError error = KafkaError.getError(fetchResponse.errorCode(topic, partitionId));
if (error.equals(KafkaError.OFFSET_OUT_OF_RANGE) && config.useStartOffsetTimeIfOffsetOutOfRange && errors == 0) {
long startOffset = getOffset(consumer, topic, partitionId, config.startOffsetTime);
LOG.warn("Got fetch request with offset out of range: [" + offset + "]; " +
"retrying with default start offset time from configuration. " +
"configured start offset time: [" + config.startOffsetTime + "] offset: [" + startOffset + "]");
offset = startOffset;
} else {
String message = "Error fetching data from [" + partition + "] for topic [" + topic + "]: [" + error + "]";
LOG.error(message);
throw new FailedFetchException(message);
}
} else {
msgs = fetchResponse.messageSet(topic, partitionId);
}
}
return msgs;
}
====================== github 上关于fetchMessage已有相关修改。修改代码例如以下====================================
public static ByteBufferMessageSet fetchMessages(KafkaConfig config, SimpleConsumer consumer, Partition partition, long offset) throws UpdateOffsetException {
ByteBufferMessageSet msgs = null;
String topic = config.topic;
int partitionId = partition.partition;
FetchRequestBuilder builder = new FetchRequestBuilder();
FetchRequest fetchRequest = builder.addFetch(topic, partitionId, offset, config.fetchSizeBytes).
clientId(config.clientId).maxWait(config.fetchMaxWait).build();
FetchResponse fetchResponse;
try {
fetchResponse = consumer.fetch(fetchRequest);
} catch (Exception e) {
if (e instanceof ConnectException ||
e instanceof SocketTimeoutException ||
e instanceof IOException ||
e instanceof UnresolvedAddressException
) {
LOG.warn("Network error when fetching messages:", e);
throw new FailedFetchException(e);
} else {
throw new RuntimeException(e);
}
}
if (fetchResponse.hasError()) {
KafkaError error = KafkaError.getError(fetchResponse.errorCode(topic, partitionId));
if (error.equals(KafkaError.OFFSET_OUT_OF_RANGE) && config.useStartOffsetTimeIfOffsetOutOfRange) {
LOG.warn("Got fetch request with offset out of range: [" + offset + "]; " +
"retrying with default start offset time from configuration. " +
"configured start offset time: [" + config.startOffsetTime + "]");
throw new UpdateOffsetException();
} else {
String message = "Error fetching data from [" + partition + "] for topic [" + topic + "]: [" + error + "]";
LOG.error(message);
throw new FailedFetchException(message);
}
} else {
msgs = fetchResponse.messageSet(topic, partitionId);
}
return msgs;
}
去掉了三次容错处理。之所以这么改动的原因大家去看STORM-511 isuue。描写叙述的原因例如以下:
tries to fix offset in local scope and retry fetch request. But if there are no more messages appeared under that specified partition it will never update the PartitionManager, but keep sending tons of requests with invalid offset to Kafka broker. On both
sides Storm and Kafka logs grow extremely quick during that time.
再来看next方法。这种方法就是KafkaSpout 的nextTuple真正调用的方法
//returns false if it's reached the end of current batch
public EmitState next(SpoutOutputCollector collector) {
if (_waitingToEmit.isEmpty()) {
fill(); // 開始时获取message
}
while (true) {
MessageAndRealOffset toEmit = _waitingToEmit.pollFirst(); //每次读取一条
if (toEmit == null) {
return EmitState.NO_EMITTED;
}
// 假设忘记了,能够再返回看下自己定义scheme这篇 : http://blog.csdn.net/wzhg0508/article/details/40874155
Iterable<List<Object>> tups = KafkaUtils.generateTuples(_spoutConfig, toEmit.msg);
if (tups != null) {
for (List<Object> tup : tups) { //这个地方在讲述自己定义Scheme时。提到了
collector.emit(tup, new KafkaMessageId(_partition, toEmit.offset));
}
break; // 这里就是每成功发射一天msg,就break掉,返回emitstate给kafkaSpout的nextTuple中做推断和定时commit成功处理的offset到zk
} else {
ack(toEmit.offset); // ack 做清除工作
}
}
if (!_waitingToEmit.isEmpty()) {
return EmitState.EMITTED_MORE_LEFT;
} else {
return EmitState.EMITTED_END;
}
}
来看ack的操作:
public void ack(Long offset) {
if (!_pending.isEmpty() && _pending.first() < offset - _spoutConfig.maxOffsetBehind) {
// Too many things pending!
_pending.headSet(offset - _spoutConfig.maxOffsetBehind).clear();
}
_pending.remove(offset);
numberAcked++;
}
接下来看下。commit的操作,这个操作也是在kafkaSpout中调用的
public void commit() {
long lastCompletedOffset = lastCompletedOffset();
if (_committedTo != lastCompletedOffset) {
LOG.debug("Writing last completed offset (" + lastCompletedOffset + ") to ZK for " + _partition + " for topology: " + _topologyInstanceId);
Map<Object, Object> data = (Map<Object, Object>) ImmutableMap.builder()
.put("topology", ImmutableMap.of("id", _topologyInstanceId,
"name", _stormConf.get(Config.TOPOLOGY_NAME)))
.put("offset", lastCompletedOffset)
.put("partition", _partition.partition)
.put("broker", ImmutableMap.of("host", _partition.host.host,
"port", _partition.host.port))
.put("topic", _spoutConfig.topic).build();
_state.writeJSON(committedPath(), data); _committedTo = lastCompletedOffset;
LOG.debug("Wrote last completed offset (" + lastCompletedOffset + ") to ZK for " + _partition + " for topology: " + _topologyInstanceId);
} else {
LOG.debug("No new offset for " + _partition + " for topology: " + _topologyInstanceId);
}
}
这个commit非常easy。不用我多做解释了
public void fail(Long offset) {
if (offset < _emittedToOffset - _spoutConfig.maxOffsetBehind) {
LOG.info(
"Skipping failed tuple at offset=" + offset +
" because it's more than maxOffsetBehind=" + _spoutConfig.maxOffsetBehind +
" behind _emittedToOffset=" + _emittedToOffset
);
} else {
LOG.debug("failing at offset=" + offset + " with _pending.size()=" + _pending.size() + " pending and _emittedToOffset=" + _emittedToOffset);
failed.add(offset);
numberFailed++;
if (numberAcked == 0 && numberFailed > _spoutConfig.maxOffsetBehind) {
throw new RuntimeException("Too many tuple failures");
}
}
}
之前storm-kafka-0.8plus的版本号是这种(摘自storm-kafka-0.8-plus
源代码解析)
首先作者没有cache message。而仅仅是cache offset
所以fail的时候,他是无法直接replay的,在他的凝视里面写了,不这样做的原因是怕内存爆掉
所以他的做法是,当一个offset fail的时候, 直接将_emittedToOffset回滚到当前fail的这个offset
下次从Kafka fetch的时候会从_emittedToOffset開始读,这样做的优点就是依赖kafka做replay,问题就是会有反复问题
所以使用时,一定要考虑,能否够接受反复问题
public void fail(Long offset) {
//TODO: should it use in-memory ack set to skip anything that's been acked but not committed? ? ? // things might get crazy with lots of timeouts
if (_emittedToOffset > offset) {
_emittedToOffset = offset;
_pending.tailSet(offset).clear();
}
}
大家对照一下吧
reference
(六)storm-kafka源代码走读之PartitionManager的更多相关文章
- 【原创】Windows平台搭建Kafka源代码开发环境(Eclipse版本)
最近在研究Kafka源代码,需要自己搭建一个开发环境.官网上给出的提示略显简单,照着做了一遍也碰到了一些问题.特此记录下来. 开发环境: Oracle Java 1.7_u71 + Eclipse 4 ...
- Storm+kafka的HelloWorld初体验
从16年4月5号开始学习kafka,后来由于项目需要又涉及到了storm. 经过几天的扫盲,到今天16年4月13日,磕磕碰碰的总算是写了一个kafka+storm的HelloWorld的例子. 为了达 ...
- Kafka 源代码分析.
这里记录kafka源代码笔记.(代码版本是0.8.2.1) kafka的源代码如何下载.这里简单说一下. git clone https://git-wip-us.apache.org/repos/a ...
- kafka源代码环境配置
kafka版本10.0.0.没有采用最新版本是因为项目中目前使用了这个版本. 1.安装gradle 首先进入https://gradle.org/install 查看Install manually- ...
- Kafka设计解析(六)Kafka高性能架构之道
转载自 技术世界,原文链接 Kafka设计解析(六)- Kafka高性能架构之道 本文从宏观架构层面和微观实现层面分析了Kafka如何实现高性能.包含Kafka如何利用Partition实现并行处理和 ...
- storm笔记:Storm+Kafka简单应用
storm笔记:Storm+Kafka简单应用 这几天工作须要使用storm+kafka,基本场景是应用出现错误,发送日志到kafka的某个topic.storm订阅该topic.然后进行兴许处理.场 ...
- 大数据学习——Storm+Kafka+Redis整合
1 pom.xml <?xml version="1.0" encoding="UTF-8"?> <project xmlns="h ...
- hadoop+yarn+hbase+storm+kafka+spark+zookeeper)高可用集群详细配置
配置 hadoop+yarn+hbase+storm+kafka+spark+zookeeper 高可用集群,同时安装相关组建:JDK,MySQL,Hive,Flume 文章目录 环境介绍 节点介绍 ...
- word2vec 中的数学原理具体解释(六)若干源代码细节
word2vec 是 Google 于 2013 年开源推出的一个用于获取 word vector 的工具包,它简单.高效,因此引起了非常多人的关注.因为 word2vec 的作者 Tomas M ...
随机推荐
- linux上重启jboss服务器
ps -ef|grep jboss :查看当前jboss进程 kill -9 进程id :杀掉进程,kill -9发送的信号是SIGKILL,即exit.exit信号不会被系统阻塞 ...
- Qt之qInstallMessageHandler(输出详细日志)
简述 安装之前已定义的消息处理程序,返回一个指向前一个消息处理程序. 消息处理程序是一个函数,用于打印调试信息.警告信息.严重错误和致命的错误的消息.Qt库(debug模式)包含成百上千的警告信息打印 ...
- gdb学习-checkpoint,watch
checkpoint的内容参考: http://blog.chinaunix.net/uid-23629988-id-2943273.html 这一篇主要是checkpoint,在next之前加che ...
- POJ 3904
第一道莫比乌斯反演的题. 建议参看http://www.isnowfy.com/mobius-inversion/ 摘其中部分 证明的话感觉写起来会比较诡异,大家意会吧说一下这个经典题目:令R(M,N ...
- MySQL高可用系列之MHA(二)
一.參数说明 MHA提供了一系列配置參数.深入理解每一个參数的详细含义,对优化配置.合理使用MHA非常重要.非常多高可用性也都是通过合理配置一些參数而实现的. MHA包含例如以下配置參数,分别说明例如 ...
- hiho模拟面试题2 补提交卡 (贪心,枚举)
题目: 时间限制:2000ms 单点时限:1000ms 内存限制:256MB 描写叙述 小Ho给自己定了一个雄伟的目标:连续100天每天坚持在hihoCoder上提交一个程序.100天过去了.小Ho查 ...
- 剑指offer面试题26-复杂链表的复制
题目: 请实现函数ComplexListNode* Clone(ComplexListNode* pHead).复制一个复杂链表. 在复杂链表中.每个节点除了一个m_pNext指针指向下一个节点外,另 ...
- Android动态载入JAR包的实现方法
有时候我们须要使用动态更新技术,简单来说就是:当我们把开发jar包发送给用户后.假设后期升级了部分代码.这时让用户的jar包自己主动更新,而不是用户主动手动地去更新的技术就是动态更新.这个须要使用的技 ...
- linux使用windows中编辑的文件,格式问题
参考:https://blog.csdn.net/yongan1006/article/details/8142527 运行脚本时出现了这样一个错误,打开之后并没有找到所谓的^M,查了之后才知道原来是 ...
- Iterator(迭代器) 和generator
数组是可迭代的 var a = []; console.dir(a); 发现这里有一个Symbol.iterator ,说明它是可迭代的. object 是不可以迭代的 var a = {} cons ...