Kafka Tools
参考,
https://cwiki.apache.org/confluence/display/KAFKA/System+Tools
https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools
http://kafka.apache.org/documentation.html#quickstart
http://kafka.apache.org/documentation.html#operations
为了便于使用,kafka提供了比较强大的Tools,把经常需要使用的整理一下
开关kafka Server
bin/kafka-server-start.sh config/server.properties
bin/kafka-server-stop.sh
JMX_PORT=9999 nohup bin/kafka-server-start.sh config/server.properties &
topic相关
bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
bin/kafka-topics.sh --list --zookeeper localhost:2181
describe topic的详细情况
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test
修改topic的partition,只能增加
bin/kafka-topics.sh --alter --zookeeper localhost:2181 --partitions 3 --topic test
到0.8.2才正式支持删除topic,当前是beta版
/usr/local/rds/kafka/bin/kafka-topics.sh --delete --topic topic_name --zookeeper localhost:2181
注意在配置里面,delete.topic.enable=true
查看有问题的partition
bin/kafka-topics.sh --describe --zookeeper localhost:2181 --unavailable-partitions --topic test
per-topic 修改参数
> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic my-topic --partitions 1
--replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1
> bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic
--config max.message.bytes=128000
> bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic
--deleteConfig max.message.bytes
集群扩展
集群扩展,对于broker还是比较简单的,但是现有的topic上的partition是不会做自动迁移的
需要手工做迁移,但kafka提供了比较方便的工具,
--generate,生成参考的迁移计划
given a list of topics and a list of brokers,工具会给出迁徙方案
把topic完全迁移到新的brokers
> cat topics-to-move.json
{"topics": [{"topic": "foo1"},
{"topic": "foo2"}],
"version":
}
> bin/kafka-reassign-partitions.sh --zookeeper localhost: --topics-to-move-json-file topics-to-move.json --broker-list "5,6" --generate
Current partition replica assignment {"version":,
"partitions":[{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]}]
} Proposed partition reassignment configuration {"version":,
"partitions":[{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]}]
}
给出当前的assignment情况和,迁移方案
我们可以同时保存当前的assignment情况和迁移方案,当前的assignment情况可以用于rollback
--execute,开始执行迁移
> bin/kafka-reassign-partitions.sh --zookeeper localhost: --reassignment-json-file expand-cluster-reassignment.json --execute
Current partition replica assignment {"version":,
"partitions":[{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]}]
} Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":,
"partitions":[{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]}]
}
--verify,check当前的迁移状态
> bin/kafka-reassign-partitions.sh --zookeeper localhost: --reassignment-json-file expand-cluster-reassignment.json --verify
Status of partition reassignment:
Reassignment of partition [foo1,] completed successfully
Reassignment of partition [foo1,] is in progress
Reassignment of partition [foo1,] is in progress
Reassignment of partition [foo2,] completed successfully
Reassignment of partition [foo2,] completed successfully
Reassignment of partition [foo2,] completed successfully
选择topic的某个partition的某些replica进行迁徙
moves partition 0 of topic foo1 to brokers 5,6 and partition 1 of topic foo2 to brokers 2,3
> cat custom-reassignment.json
{"version":,"partitions":[{"topic":"foo1","partition":,"replicas":[,]},{"topic":"foo2","partition":,"replicas":[,]}]}
> bin/kafka-reassign-partitions.sh --zookeeper localhost: --reassignment-json-file custom-reassignment.json --execute
Current partition replica assignment {"version":,
"partitions":[{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]}]
} Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":,
"partitions":[{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]}]
}
brokers下线
当前版本不支持下线的规划,需要到0.8.2才支持,这需要把一个broker上的replica清空
增加replication factor
partition 0的replica数从1增长到3,当前replica存在broker5,在broker6,7上增加replica
> cat increase-replication-factor.json
{"version":,
"partitions":[{"topic":"foo","partition":,"replicas":[,,]}]}
> bin/kafka-reassign-partitions.sh --zookeeper localhost: --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment {"version":,
"partitions":[{"topic":"foo","partition":,"replicas":[]}]} Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":,
"partitions":[{"topic":"foo","partition":,"replicas":[,,]}]}
Producer console
> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message
后面可以任意的输入message,都会发到broker的topic中
Comsumer console
bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning
从头读这个topic,可以重复读到所有数据
我在想为啥,每次都能replay,原来每次都是随机产生一个groupid
consumerProps.put("group.id","console-consumer-" + new Random().nextInt(100000))
Consumer Offset Checker
这个会显示出consumer group的offset情况, 必须参数为--group, 不指定--topic,默认为所有topic
Displays the: Consumer Group, Topic, Partitions, Offset, logSize, Lag, Owner for the specified set of Topics and Consumer Group
bin/kafka-run-
class
.sh kafka.tools.ConsumerOffsetChecker
required argument: [group]
Option Description
------ -----------
--broker-info Print broker info
--group Consumer group.
--help Print this message.
--topic Comma-separated list of consumer
topics (all topics if absent).
--zkconnect ZooKeeper connect string. (default: localhost:2181)
Example,
bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group pv
Group Topic Pid Offset logSize Lag Owner
pv page_visits 0 21 21 0 none
pv page_visits 1 19 19 0 none
pv page_visits 2 20 20 0 none
Export Zookeeper Offsets
将Zk中的offset信息以下面的形式打到file里面去
A utility that retrieves the offsets of broker partitions in ZK and prints to an output file in the following format:
/consumers/group1/offsets/topic1/1-0:286894308
/consumers/group1/offsets/topic1/2-0:284803985
bin/kafka-run-
class
.sh kafka.tools.ExportZkOffsets
required argument: [zkconnect]
Option Description
------ -----------
--group Consumer group.
--help Print this message.
--output-file Output file
--zkconnect ZooKeeper connect string. (default: localhost:2181)
Update Offsets In Zookeeper
这个挺有用,用于replay, kafka的文档有点坑爹,看了不知道咋用,还是看源码才看明白
A utility that updates the offset of every broker partition to the offset of earliest or latest log segment file, in ZK.
bin/kafka-run-
class
.sh kafka.tools.UpdateOffsetsInZK
USAGE: kafka.tools.UpdateOffsetsInZK$ [earliest | latest] consumer.properties topic
Example,
bin/kafka-run-class.sh kafka.tools.UpdateOffsetsInZK earliest config/consumer.properties page_visits
Group Topic Pid Offset logSize Lag Owner
pv page_visits 0 0 21 21 none
pv page_visits 1 0 19 19 none
pv page_visits 2 0 20 20 none
可以看到offset已经被清0,Lag=logSize
更加直接的方式是,直接去Zookeeper里面看
通过zkCli.sh连上后,通过ls查看
Broker Node Registry
/brokers/ids/[0...N] --> host:port (ephemeral node)
Broker Topic Registry
/brokers/topics/[topic]/[0...N] --> nPartions (ephemeral node)
Consumer Id Registry
/consumers/[group_id]/ids/[consumer_id] --> {"topic1": #streams, ..., "topicN": #streams} (ephemeral node)
Consumer Offset Tracking
/consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] --> offset_counter_value ((persistent node)
Partition Owner registry
/consumers/[group_id]/owners/[topic]/[broker_id-partition_id] --> consumer_node_id (ephemeral node)
Kafka Tools的更多相关文章
- 5.1SpringBoot整合Kafka(工具安装Kafka+Tools)
1.工具安装Kafka 上一期我分享了安装zk,下一次我们把Kafka和可视化工具一起搞起来. 注意:这个时候ZK一定要启动成功. zk安装地址:https://www.cnblogs.com/dao ...
- kafka可视化工具kafka tools
一.下载 下载地址 选择windows 傻瓜式安装,选择安装路径,直接下一步就可以了 二. 使用 点击,运行 linux开启9092(broker)端口和2181(zookeeper)然后填写后,确定 ...
- kafka性能参数和压力测试揭秘
转自:http://blog.csdn.net/stark_summer/article/details/50203133 上一篇文章介绍了Kafka在设计上是如何来保证高时效.大吞吐量的,主要的内容 ...
- Kafka 如何读取offset topic内容 (__consumer_offsets)
众所周知,由于Zookeeper并不适合大批量的频繁写入操作,新版Kafka已推荐将consumer的位移信息保存在Kafka内部的topic中,即__consumer_offsets topic,并 ...
- Kafka设计解析(三)- Kafka High Availability (下)
本文转发自Jason’s Blog,原文链接 http://www.jasongj.com/2015/06/08/KafkaColumn3 摘要 本文在上篇文章基础上,更加深入讲解了Kafka的HA机 ...
- Kafka设计解析(一)- Kafka背景及架构介绍
本文转发自Jason’s Blog,原文链接 http://www.jasongj.com/2015/01/02/Kafka深度解析 背景介绍 Kafka简介 Kafka是一种分布式的,基于发布/订阅 ...
- 【原创】Kafka console consumer源代码分析(一)
上一篇中分析了Scala版的console producer代码,这篇文章为读者带来一篇console consumer工作原理分析的随笔.其实不论是哪个consumer,大部分的工作原理都是类似的. ...
- Kafka简介
Kafka简介 转载请注明出处:http://www.cnblogs.com/BYRans/ Apache Kafka发源于LinkedIn,于2011年成为Apache的孵化项目,随后于2012年成 ...
- KAFKA一异常处理记录
kafka-console-consumer.sh --topic TOPIC_KEYWORD --from-beginning --zookeeper localhost报异常,Exception ...
随机推荐
- linux上创建PV/VG/LV
LVM的整体思路是: 首先创建PV-->然后创建VG并将多个PV加到VG里-->然后创建LV-->格式化分区-->mount分区 1.创建PV pvcreate /dev/sd ...
- PLSQL_统计信息系列05_统计信息的比较
20150506 Created By BaoXinjian
- Linux vm运行参数 - overcommit相关的参数
一.前言 终于可以进入Linux kernel内存管理的世界了,但是从哪里入手是一个问题,当面对一个复杂系统的时候,有时候不知道怎么开始.遵守“一切以人为本”的原则,我最终选择先从从userspace ...
- Django ORM 数据库操作
比较有用 转自 http://blog.csdn.net/fgf00/article/details/53678205 一.DjangoORM 创建基本类型及生成数据库表结构 1.简介 2.创建数据库 ...
- Server Process
1.client进行update操作后.其它是怎么协作的? Client进行update操作之后,是由Server Process真正完毕的,分以下几步: 1).须要更新的数据在Data buffer ...
- js获取日期实例之昨天今天和明天、后天
本文介绍了js获取日期的方法,可以获取前天.昨天.今天.明天.后天. 代码: <html> <head> <meta http-equiv="Content-T ...
- SQL游标操作每隔5分钟时间段数据统计信息
Tb_People 表信息: id uname era amount plushtime 1000031 张亮 中年 100000 ...
- python 连接sql server数据库的示例代码
首先,到http://pymssql.sourceforge.net/下载pymssql模块,必须安装这个模块才可以用python连接mysql 以下是sql server的操作代码,需要注意字符集 ...
- Oracle PLSQL Demo - 29.01.Function结构模板 [无入参] [有返回]
CREATE OR REPLACE FUNCTION function_name RETURN DATE AS v_date DATE; BEGIN ; dbms_output.put_line(v_ ...
- mac下borderless的window(无标题栏)如何实现
子类化NSWindow: - (void)awakeFromNib { [selfsetStyleMask:NSBorderlessWindowMask]; [selfsetAcceptsMouseM ...