参考,

https://cwiki.apache.org/confluence/display/KAFKA/System+Tools

https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools

http://kafka.apache.org/documentation.html#quickstart

http://kafka.apache.org/documentation.html#operations

为了便于使用,kafka提供了比较强大的Tools,把经常需要使用的整理一下

开关kafka Server

bin/kafka-server-start.sh config/server.properties
bin/kafka-server-stop.sh
JMX_PORT=9999 nohup bin/kafka-server-start.sh config/server.properties &

topic相关

bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
bin/kafka-topics.sh --list --zookeeper localhost:2181

describe topic的详细情况

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --topic test

修改topic的partition,只能增加

bin/kafka-topics.sh --alter --zookeeper localhost:2181 --partitions 3 --topic test

到0.8.2才正式支持删除topic,当前是beta版

/usr/local/rds/kafka/bin/kafka-topics.sh --delete --topic topic_name --zookeeper localhost:2181

注意在配置里面,delete.topic.enable=true

查看有问题的partition

bin/kafka-topics.sh --describe --zookeeper localhost:2181 --unavailable-partitions --topic test

per-topic 修改参数
> bin/kafka-topics.sh --zookeeper localhost:2181 --create --topic my-topic --partitions 1
--replication-factor 1 --config max.message.bytes=64000 --config flush.messages=1
> bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic
--config max.message.bytes=128000
> bin/kafka-topics.sh --zookeeper localhost:2181 --alter --topic my-topic
--deleteConfig max.message.bytes

集群扩展
集群扩展,对于broker还是比较简单的,但是现有的topic上的partition是不会做自动迁移的

需要手工做迁移,但kafka提供了比较方便的工具,

--generate,生成参考的迁移计划

given a list of topics and a list of brokers,工具会给出迁徙方案

把topic完全迁移到新的brokers

> cat topics-to-move.json
{"topics": [{"topic": "foo1"},
{"topic": "foo2"}],
"version":
}
> bin/kafka-reassign-partitions.sh --zookeeper localhost: --topics-to-move-json-file topics-to-move.json --broker-list "5,6" --generate
Current partition replica assignment {"version":,
"partitions":[{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]}]
} Proposed partition reassignment configuration {"version":,
"partitions":[{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]}]
}

给出当前的assignment情况和,迁移方案

我们可以同时保存当前的assignment情况和迁移方案,当前的assignment情况可以用于rollback

--execute,开始执行迁移

> bin/kafka-reassign-partitions.sh --zookeeper localhost: --reassignment-json-file expand-cluster-reassignment.json --execute
Current partition replica assignment {"version":,
"partitions":[{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]}]
} Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":,
"partitions":[{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]},
{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]}]
}

--verify,check当前的迁移状态

> bin/kafka-reassign-partitions.sh --zookeeper localhost: --reassignment-json-file expand-cluster-reassignment.json --verify
Status of partition reassignment:
Reassignment of partition [foo1,] completed successfully
Reassignment of partition [foo1,] is in progress
Reassignment of partition [foo1,] is in progress
Reassignment of partition [foo2,] completed successfully
Reassignment of partition [foo2,] completed successfully
Reassignment of partition [foo2,] completed successfully

选择topic的某个partition的某些replica进行迁徙

moves partition 0 of topic foo1 to brokers 5,6 and partition 1 of topic foo2 to brokers 2,3

> cat custom-reassignment.json
{"version":,"partitions":[{"topic":"foo1","partition":,"replicas":[,]},{"topic":"foo2","partition":,"replicas":[,]}]}
> bin/kafka-reassign-partitions.sh --zookeeper localhost: --reassignment-json-file custom-reassignment.json --execute
Current partition replica assignment {"version":,
"partitions":[{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]}]
} Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":,
"partitions":[{"topic":"foo1","partition":,"replicas":[,]},
{"topic":"foo2","partition":,"replicas":[,]}]
}

brokers下线

当前版本不支持下线的规划,需要到0.8.2才支持,这需要把一个broker上的replica清空

增加replication factor

partition 0的replica数从1增长到3,当前replica存在broker5,在broker6,7上增加replica

> cat increase-replication-factor.json
{"version":,
"partitions":[{"topic":"foo","partition":,"replicas":[,,]}]}
> bin/kafka-reassign-partitions.sh --zookeeper localhost: --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment {"version":,
"partitions":[{"topic":"foo","partition":,"replicas":[]}]} Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions
{"version":,
"partitions":[{"topic":"foo","partition":,"replicas":[,,]}]}

Producer console

> bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a message
This is another message

后面可以任意的输入message,都会发到broker的topic中

Comsumer console

bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic test --from-beginning

从头读这个topic,可以重复读到所有数据
我在想为啥,每次都能replay,原来每次都是随机产生一个groupid

consumerProps.put("group.id","console-consumer-" + new Random().nextInt(100000))

Consumer Offset Checker

这个会显示出consumer group的offset情况, 必须参数为--group, 不指定--topic,默认为所有topic

Displays the:  Consumer Group, Topic, Partitions, Offset, logSize, Lag, Owner for the specified set of Topics and Consumer Group

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker

required argument: [group]

Option Description

------ -----------

--broker-info Print broker info

--group Consumer group.

--help Print this message.

--topic Comma-separated list of consumer

topics (all topics if absent).

--zkconnect ZooKeeper connect string. (default: localhost:2181)

Example,

bin/kafka-run-class.sh kafka.tools.ConsumerOffsetChecker --group pv

Group           Topic                          Pid Offset          logSize         Lag             Owner

pv              page_visits                    0   21              21              0               none

pv              page_visits                    1   19              19              0               none

pv              page_visits                    2   20              20              0               none

Export Zookeeper Offsets

将Zk中的offset信息以下面的形式打到file里面去

A utility that retrieves the offsets of broker partitions in ZK and prints to an output file in the following format:

/consumers/group1/offsets/topic1/1-0:286894308

/consumers/group1/offsets/topic1/2-0:284803985

bin/kafka-run-class.sh kafka.tools.ExportZkOffsets

required argument: [zkconnect]

Option Description

------ -----------

--group Consumer group.

--help Print this message.

--output-file Output file

--zkconnect ZooKeeper connect string. (default: localhost:2181)

Update Offsets In Zookeeper

这个挺有用,用于replay, kafka的文档有点坑爹,看了不知道咋用,还是看源码才看明白

A utility that updates the offset of every broker partition to the offset of earliest or latest log segment file, in ZK.

bin/kafka-run-class.sh kafka.tools.UpdateOffsetsInZK

USAGE: kafka.tools.UpdateOffsetsInZK$ [earliest | latest] consumer.properties topic

Example,

bin/kafka-run-class.sh kafka.tools.UpdateOffsetsInZK earliest config/consumer.properties  page_visits

Group           Topic                          Pid Offset          logSize         Lag             Owner

pv              page_visits                    0   0               21              21              none

pv              page_visits                    1   0               19              19              none

pv              page_visits                    2   0               20              20              none

可以看到offset已经被清0,Lag=logSize

更加直接的方式是,直接去Zookeeper里面看

通过zkCli.sh连上后,通过ls查看

Broker Node Registry

/brokers/ids/[0...N] --> host:port (ephemeral node)

Broker Topic Registry

/brokers/topics/[topic]/[0...N] --> nPartions (ephemeral node)

Consumer Id Registry

/consumers/[group_id]/ids/[consumer_id] --> {"topic1": #streams, ..., "topicN": #streams} (ephemeral node)

Consumer Offset Tracking

/consumers/[group_id]/offsets/[topic]/[broker_id-partition_id] --> offset_counter_value ((persistent node)

Partition Owner registry

/consumers/[group_id]/owners/[topic]/[broker_id-partition_id] --> consumer_node_id (ephemeral node)

Kafka Tools的更多相关文章

  1. 5.1SpringBoot整合Kafka(工具安装Kafka+Tools)

    1.工具安装Kafka 上一期我分享了安装zk,下一次我们把Kafka和可视化工具一起搞起来. 注意:这个时候ZK一定要启动成功. zk安装地址:https://www.cnblogs.com/dao ...

  2. kafka可视化工具kafka tools

    一.下载 下载地址 选择windows 傻瓜式安装,选择安装路径,直接下一步就可以了 二. 使用 点击,运行 linux开启9092(broker)端口和2181(zookeeper)然后填写后,确定 ...

  3. kafka性能参数和压力测试揭秘

    转自:http://blog.csdn.net/stark_summer/article/details/50203133 上一篇文章介绍了Kafka在设计上是如何来保证高时效.大吞吐量的,主要的内容 ...

  4. Kafka 如何读取offset topic内容 (__consumer_offsets)

    众所周知,由于Zookeeper并不适合大批量的频繁写入操作,新版Kafka已推荐将consumer的位移信息保存在Kafka内部的topic中,即__consumer_offsets topic,并 ...

  5. Kafka设计解析(三)- Kafka High Availability (下)

    本文转发自Jason’s Blog,原文链接 http://www.jasongj.com/2015/06/08/KafkaColumn3 摘要 本文在上篇文章基础上,更加深入讲解了Kafka的HA机 ...

  6. Kafka设计解析(一)- Kafka背景及架构介绍

    本文转发自Jason’s Blog,原文链接 http://www.jasongj.com/2015/01/02/Kafka深度解析 背景介绍 Kafka简介 Kafka是一种分布式的,基于发布/订阅 ...

  7. 【原创】Kafka console consumer源代码分析(一)

    上一篇中分析了Scala版的console producer代码,这篇文章为读者带来一篇console consumer工作原理分析的随笔.其实不论是哪个consumer,大部分的工作原理都是类似的. ...

  8. Kafka简介

    Kafka简介 转载请注明出处:http://www.cnblogs.com/BYRans/ Apache Kafka发源于LinkedIn,于2011年成为Apache的孵化项目,随后于2012年成 ...

  9. KAFKA一异常处理记录

    kafka-console-consumer.sh --topic TOPIC_KEYWORD --from-beginning --zookeeper localhost报异常,Exception ...

随机推荐

  1. linux上创建PV/VG/LV

    LVM的整体思路是: 首先创建PV-->然后创建VG并将多个PV加到VG里-->然后创建LV-->格式化分区-->mount分区 1.创建PV pvcreate /dev/sd ...

  2. PLSQL_统计信息系列05_统计信息的比较

    20150506 Created By BaoXinjian

  3. Linux vm运行参数 - overcommit相关的参数

    一.前言 终于可以进入Linux kernel内存管理的世界了,但是从哪里入手是一个问题,当面对一个复杂系统的时候,有时候不知道怎么开始.遵守“一切以人为本”的原则,我最终选择先从从userspace ...

  4. Django ORM 数据库操作

    比较有用 转自 http://blog.csdn.net/fgf00/article/details/53678205 一.DjangoORM 创建基本类型及生成数据库表结构 1.简介 2.创建数据库 ...

  5. Server Process

    1.client进行update操作后.其它是怎么协作的? Client进行update操作之后,是由Server Process真正完毕的,分以下几步: 1).须要更新的数据在Data buffer ...

  6. js获取日期实例之昨天今天和明天、后天

    本文介绍了js获取日期的方法,可以获取前天.昨天.今天.明天.后天. 代码: <html> <head> <meta http-equiv="Content-T ...

  7. SQL游标操作每隔5分钟时间段数据统计信息

    Tb_People 表信息: id     uname    era     amount    plushtime           1000031    张亮    中年    100000   ...

  8. python 连接sql server数据库的示例代码

    首先,到http://pymssql.sourceforge.net/下载pymssql模块,必须安装这个模块才可以用python连接mysql 以下是sql server的操作代码,需要注意字符集 ...

  9. Oracle PLSQL Demo - 29.01.Function结构模板 [无入参] [有返回]

    CREATE OR REPLACE FUNCTION function_name RETURN DATE AS v_date DATE; BEGIN ; dbms_output.put_line(v_ ...

  10. mac下borderless的window(无标题栏)如何实现

    子类化NSWindow: - (void)awakeFromNib { [selfsetStyleMask:NSBorderlessWindowMask]; [selfsetAcceptsMouseM ...