[bigdata] kafka基本命令 -- 迁移topic partition到指定的broker
版本 0.9.2
创建topic
bin/kafka-topics.sh --create --topic topic_name --partition 6 --replication-factor 1 -zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181
开启console consumer查看消息
bin/kafka-console-consumer.sh --topic rt_live_pcweb -zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181 --from-begin
改变topic的partition数, kafka topic的partition数只能增加,不能减少。
[root@ip-10-27-100-100 kafka]# bin/kafka-topics.sh --zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181 --alter --topic rt_live_pcweb --partitions 8
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Error while executing topic command The number of partitions for a topic can only be increased
kafka.admin.AdminOperationException: The number of partitions for a topic can only be increased
at kafka.admin.AdminUtils$.addPartitions(AdminUtils.scala:98)
at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:109)
at kafka.admin.TopicCommand$$anonfun$alterTopic$1.apply(TopicCommand.scala:93)
at scala.collection.mutable.ResizableArray$class.foreach(ResizableArray.scala:60)
at scala.collection.mutable.ArrayBuffer.foreach(ArrayBuffer.scala:47)
at kafka.admin.TopicCommand$.alterTopic(TopicCommand.scala:93)
at kafka.admin.TopicCommand$.main(TopicCommand.scala:52)
at kafka.admin.TopicCommand.main(TopicCommand.scala)
迁移topic的partition到指定的broker上
1. 将需要迁移的topic按如下格式写成json文件
[root@ip-10-27-111-107 kafka]# cat topics-to-move.json
{"topics":
[{"topic": "rt_live_mobile_new"}],
"version":1
}
2. 执行如下命令,其中 0,1,2,3 为要迁移到的broker id
[root@ip-10-27-111-107 kafka]# bin/kafka-reassign-partitions.sh --zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181 --topics-to-move-json-file topics-to-move.json --broker-list "0,1,2,3" --generate
Current partition replica assignment
{"version":1,"partitions":[{"topic":"rt_live_mobile_new","partition":2,"replicas":[4]},{"topic":"rt_live_mobile_new","partition":1,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":6,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":4,"replicas":[6]},{"topic":"rt_live_mobile_new","partition":8,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":7,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":11,"replicas":[5]},{"topic":"rt_live_mobile_new","partition":3,"replicas":[5]},{"topic":"rt_live_mobile_new","partition":0,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":10,"replicas":[4]},{"topic":"rt_live_mobile_new","partition":5,"replicas":[7]},{"topic":"rt_live_mobile_new","partition":9,"replicas":[3]}]}
Proposed partition reassignment configuration
{"version":1,"partitions":[{"topic":"rt_live_mobile_new","partition":2,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":6,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":1,"replicas":[3]},{"topic":"rt_live_mobile_new","partition":8,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":4,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":7,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":11,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":10,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":0,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":3,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":9,"replicas":[3]},{"topic":"rt_live_mobile_new","partition":5,"replicas":[3]}]}
3. 将上面命令的输出(红色部门)存入json文件reassignment.json中
4. 执行如下命令,即将partition都迁移至broker 0,1,2,3 服务器上。
[root@ip-10-27-111-107 kafka]# bin/kafka-reassign-partitions.sh --zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181 --reassignment-json-file reassignment.json --execute
Current partition replica assignment
{"version":1,"partitions":[{"topic":"rt_live_mobile_new","partition":2,"replicas":[4]},{"topic":"rt_live_mobile_new","partition":1,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":6,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":4,"replicas":[6]},{"topic":"rt_live_mobile_new","partition":8,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":7,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":11,"replicas":[5]},{"topic":"rt_live_mobile_new","partition":3,"replicas":[5]},{"topic":"rt_live_mobile_new","partition":0,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":10,"replicas":[4]},{"topic":"rt_live_mobile_new","partition":5,"replicas":[7]},{"topic":"rt_live_mobile_new","partition":9,"replicas":[3]}]}
Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions {"version":1,"partitions":[{"topic":"rt_live_mobile_new","partition":2,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":4,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":8,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":1,"replicas":[3]},{"topic":"rt_live_mobile_new","partition":6,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":7,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":11,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":3,"replicas":[1]},{"topic":"rt_live_mobile_new","partition":0,"replicas":[2]},{"topic":"rt_live_mobile_new","partition":10,"replicas":[0]},{"topic":"rt_live_mobile_new","partition":5,"replicas":[3]},{"topic":"rt_live_mobile_new","partition":9,"replicas":[3]}]}
5.隔一会之后(partition迁移需要一段时间),执行describe查看,partition已经迁移成功
[root@ip-10-27-111-107 kafka]# bin/kafka-topics.sh --topic rt_live_mobile_new --describe -zookeeper 10.27.100.207:2181,10.27.100.144:2181,10.27.100.145:2181
Topic:rt_live_mobile_new PartitionCount:12 ReplicationFactor:1 Configs:
Topic: rt_live_mobile_new Partition: 0 Leader: 2 Replicas: 2 Isr: 2
Topic: rt_live_mobile_new Partition: 1 Leader: 3 Replicas: 3 Isr: 3
Topic: rt_live_mobile_new Partition: 2 Leader: 0 Replicas: 0 Isr: 0
Topic: rt_live_mobile_new Partition: 3 Leader: 1 Replicas: 1 Isr: 1
Topic: rt_live_mobile_new Partition: 4 Leader: 2 Replicas: 2 Isr: 2
Topic: rt_live_mobile_new Partition: 5 Leader: 3 Replicas: 3 Isr: 3
Topic: rt_live_mobile_new Partition: 6 Leader: 0 Replicas: 0 Isr: 0
Topic: rt_live_mobile_new Partition: 7 Leader: 1 Replicas: 1 Isr: 1
Topic: rt_live_mobile_new Partition: 8 Leader: 2 Replicas: 2 Isr: 2
Topic: rt_live_mobile_new Partition: 9 Leader: 3 Replicas: 3 Isr: 3
Topic: rt_live_mobile_new Partition: 10 Leader: 0 Replicas: 0 Isr: 0
Topic: rt_live_mobile_new Partition: 11 Leader: 1 Replicas: 1 Isr: 1
[bigdata] kafka基本命令 -- 迁移topic partition到指定的broker的更多相关文章
- kafka学习(四)-Topic & Partition
topic中partition存储分布 Topic在逻辑上可以被认为是一个queue.每条消费都必须指定它的topic,可以简单理解为必须指明把这条消息放进哪个queue里.为了使得 Kafka的吞吐 ...
- Kafka Topic Partition Replica Assignment实现原理及资源隔离方案
本文共分为三个部分: Kafka Topic创建方式 Kafka Topic Partitions Assignment实现原理 Kafka资源隔离方案 1. Kafka Topic创建方式 ...
- Kafka Topic Partition Offset 这一长串都是啥?
摘要:Offset 偏移量,是针对于单个partition存在的概念. 本文分享自华为云社区<Kafka Topic Partition Offset 这一长串都是啥?>,作者: gent ...
- 删除kafka的topic及kafka基本命令
kafka的topic默认是不允许被删除的,删除后在topic后会出现”marked for deletion”字样,实际并未删除,现在创建同样的topic会提示topic已经存在. 解决办法: se ...
- kafka数据迁移实践
欢迎大家前往云+社区,获取更多腾讯海量技术实践干货哦~ 作者:mikealzhou 本文重点介绍kafka的两类常见数据迁移方式:1.broker内部不同数据盘之间的分区数据迁移:2.不同broker ...
- Kafka数据迁移
1.概述 Kafka的使用场景非常广泛,一些实时流数据业务场景,均依赖Kafka来做数据分流.而在分布式应用场景中,数据迁移是一个比较常见的问题.关于Kafka集群数据如何迁移,今天笔者将为大家详细介 ...
- kafka的迁移干货
随着业务的发展, 服务器所在网段/机群不允许kafka继续保留在那, 需要移动到先机器上. 哎呀上面是废话,总的说就是: 2台老kafka不要了,数据要迁移到新的2台kafka上面.要求数据不丢失 通 ...
- Kafka动态增加Topic的副本
一.kafka的副本机制 由于Producer和Consumer都只会与Leader角色的分区副本相连,所以kafka需要以集群的组织形式提供主题下的消息高可用.kafka支持主备复制,所以消息具备高 ...
- Kafka数据迁移MaxCompute最佳实践
摘要: 本文向您详细介绍如何使用DataWorks数据同步功能,将Kafka集群上的数据迁移到阿里云MaxCompute大数据计算服务. 前提条件 搭建Kafka集群 进行数据迁移前,您需要保证自己的 ...
随机推荐
- 用drawRect的方式实现一个尺子
用drawRect的方式实现了一个尺子选择器,demo在这里:https://github.com/Phelthas/LXMRulerView 效果如图: 如果不考虑复用的问题,我感觉最简单的实现 ...
- 整理分享原生态mac AndroidStudio的快捷键
使用AndroidStudio开发半年了,一路爬坑至今,刚由Windows转mac一个星期.通过查些资料和自己摸索,记录一些常用的快捷键,犹豫个人不喜欢改快捷键,所以都是原生的.特此分享给大家!欢迎补 ...
- linux Mysql 安装及配置
1.准备 cmake-3.6.0.tar.gz bison-3.0.4.tar.gz mysql-5.7.13.tar.gz (http://dev.mysql.com/get/Downloads/M ...
- 如何定义好一个符合规范的url
描述 进公司没有多久遇到一个问题,定义的url会被大神吐槽说是很渣.之前从来没有注意这块,今天把我们团队的url规范分享给大家. 为什么需要URL规范化 1.网站URL和结构已经成为网站搜索引擎友好的 ...
- Java导出excel
一.介绍 常常有客户这样子要求:你要把我们的报表直接用Excel打开(电信系统.银行系统).或者是:我们已经习惯用Excel打印.这样在我们实际的开发中,很多时候需要实现导入.导出Excel的应用. ...
- Spark——共享变量
Spark执行不少操作时都依赖于闭包函数的调用,此时如果闭包函数使用到了外部变量驱动程序在使用行动操作时传递到集群中各worker节点任务时就会进行一系列操作: 1.驱动程序使将闭包中使用变量封装成对 ...
- Holographic Remoting Player
https://developer.microsoft.com/en-us/windows/holographic/holographic_remoting_player http://www.cnb ...
- ubuntu 14.04 desktop装vnc4server
ubuntu 14.04 desktop上安装vnc4server要装上gnome的一些软件包并修改启动文件~/.vnc/xstartup 问题来源How to make VNC Server wor ...
- [LeetCode] Reconstruct Original Digits from English 从英文中重建数字
Given a non-empty string containing an out-of-order English representation of digits 0-9, output the ...
- 【经验之谈】Git使用之TortoiseGit配置VS详解
前言 上一篇<[经验之谈]Git使用之Windows环境下配置>: 安装 配置和使用 后记 关于vs中使用git网上的教程大家也可以找到,我当时配置的时候也是按照网上找的教程一步一步来的, ...