以下操作可以在mini01、mini02、mini03任意一台操作即可。

1. kafka通过网页管理

2. 创建topic

 # 参数说明 --replication-factor  表示有2个副本

 # --partitions  表示有4个分区

 [yun@mini01 ~]$ kafka-topics.sh --create --zookeeper mini01: --replication-factor  --partitions  --topic test

 Created topic "test".

 [yun@mini01 ~]$ kafka-topics.sh --create --zookeeper mini01: --replication-factor  --partitions  --topic zhang

 Created topic "zhang".

 [yun@mini01 ~]$ kafka-topics.sh --list --zookeeper mini01:  # 再次查看

 zhang

 test

2.1. 各主机信息查看

mini01

 [yun@mini01 logs]$ pwd

 /app/kafka/logs

 [yun@mini01 logs]$ ll

 total

 ………………

 drwxrwxr-x  yun yun    Sep  : test-

 drwxrwxr-x  yun yun    Sep  : test-

 drwxrwxr-x  yun yun    Sep  : test-

mini02

 [yun@mini02 logs]$ pwd

 /app/kafka/logs

 [yun@mini02 logs]$ ll

 total

 ………………

 drwxrwxr-x  yun yun     Sep  : test-

 drwxrwxr-x  yun yun     Sep  : test-

mini03

 [yun@mini03 logs]$ pwd

 /app/kafka/logs

 [yun@mini03 logs]$ ll

 total

 ………………

 drwxrwxr-x  yun yun    Sep  : test-

 drwxrwxr-x  yun yun    Sep  : test-

 drwxrwxr-x  yun yun    Sep  : test-

3. 修改topic

3.1. 增加分区数

注意：分区数不能减少

　　Kafka目前不支持减少主题的分区数量。

 [yun@mini01 ~]$ kafka-topics.sh --list --zookeeper mini01:

 __consumer_offsets

 test

 test01

 test02

 test03

 test04

 zhang

 [yun@mini01 ~]$ kafka-topics.sh --describe --zookeeper mini01: --topic test01

 Topic:test01    PartitionCount:    ReplicationFactor:    Configs:

     Topic: test01    Partition:     Leader:     Replicas:     Isr:

     Topic: test01    Partition:     Leader:     Replicas:     Isr:

     Topic: test01    Partition:     Leader:     Replicas:     Isr:

     Topic: test01    Partition:     Leader:     Replicas:     Isr:

     Topic: test01    Partition:     Leader:     Replicas:     Isr:

 [yun@mini01 ~]$ kafka-topics.sh --alter --zookeeper mini01: --partitions  --topic test01 # 失败，分区数不能减少

 WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected

 Error while executing topic command : The number of partitions for a topic can only be increased. Topic test01 currently has  partitions,  would not be an increase.

 [-- ::,] ERROR org.apache.kafka.common.errors.InvalidPartitionsException: The number of partitions for a topic can only be increased. Topic test01 currently has  partitions,  would not be an increase.

  (kafka.admin.TopicCommand$)

 [yun@mini01 ~]$ kafka-topics.sh --alter --zookeeper mini01: --partitions  --topic test01  # 增加分区数

 WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected

 Adding partitions succeeded!

 [yun@mini01 ~]$ kafka-topics.sh --describe --zookeeper mini01: --topic test01

 Topic:test01    PartitionCount:    ReplicationFactor:    Configs:

     Topic: test01    Partition:     Leader:     Replicas:     Isr:

     Topic: test01    Partition:     Leader:     Replicas:     Isr:

     Topic: test01    Partition:     Leader:     Replicas:     Isr:

     Topic: test01    Partition:     Leader:     Replicas:     Isr:

     Topic: test01    Partition:     Leader:     Replicas:     Isr:

     Topic: test01    Partition:     Leader:     Replicas:     Isr:

     Topic: test01    Partition:     Leader:     Replicas:     Isr:

4. 删除topic

 # server.properties中设置delete.topic.enable=true 【当前版本默认就是true】否则只是标记删除或者直接重启

 [yun@mini01 ~]$ kafka-topics.sh --delete --zookeeper mini01: --topic test

 Topic test is marked for deletion.

 Note: This will have no impact if delete.topic.enable is not set to true.

 [yun@mini01 ~]$ kafka-topics.sh --list --zookeeper mini01:  # 再次查看   只有 zhang，则表示真的删除了

 zhang

5. 查看所有topic

 [yun@mini01 ~]$ kafka-topics.sh --list --zookeeper mini01:

 __consumer_offsets

 test

 zhang

6. 查看某个Topic的详情

 [yun@mini01 ~]$ kafka-topics.sh --describe --zookeeper mini01: --topic zhang

 Topic:zhang    PartitionCount:    ReplicationFactor:    Configs:

     Topic: zhang    Partition:     Leader:     Replicas: ,,    Isr: ,,

     Topic: zhang    Partition:     Leader:     Replicas: ,,    Isr: ,,

     Topic: zhang    Partition:     Leader:     Replicas: ,,    Isr: ,,

     Topic: zhang    Partition:     Leader:     Replicas: ,,    Isr: ,,

7. 通过shell命令生产消息

7.1. 输入单条数据

 [yun@mini01 ~]$ kafka-console-producer.sh --broker-list mini01: --topic zhang

 >

 >

 >

 >

 >

 >

 >

 >

 >

7.2. 批量导入数据

 [yun@mini01 zhangliang]$ kafka-console-producer.sh --broker-list mini01: --topic liang < .info

8. 通过shell命令消费消息

 # --from-beginning 从最开始读取

 # kafka-console-consumer.sh --zookeeper mini01: --from-beginning --topic zhang  # 老版本

 [yun@mini01 ~]$ kafka-console-consumer.sh --bootstrap-server mini01: --from-beginning --topic zhang

9. 消费组消费

9.1. 创建topic

 [yun@mini01 ~]$ kafka-topics.sh --create --zookeeper mini01: --replication-factor  --partitions  --topic order

 Created topic "order".

 [yun@mini01 ~]$ kafka-topics.sh --list --zookeeper mini01:  # 查看所有topic列表

 __consumer_offsets

 order

 test

 zhang

 [yun@mini01 ~]$ kafka-topics.sh --describe --zookeeper mini01: --topic order  # 查看topic详情

 Topic:order    PartitionCount:    ReplicationFactor:    Configs:

     Topic: order    Partition:     Leader:     Replicas:     Isr:

     Topic: order    Partition:     Leader:     Replicas:     Isr:

     Topic: order    Partition:     Leader:     Replicas:     Isr:

     Topic: order    Partition:     Leader:     Replicas:     Isr:

9.2. 生产消息

 [yun@mini01 ~]$ kafka-console-producer.sh --broker-list mini01: --topic order

 >

 >

 >

 >

 >

9.3. 消费组消费消息

mini02机器上运行

 # --group  指定组

 [yun@mini02 ~]$ kafka-console-consumer.sh --bootstrap-server mini01: --topic order --group order-group

mini03机器上运行

 # --group  指定组

 [yun@mini03 ~]$ kafka-console-consumer.sh --bootstrap-server mini01: --topic order --group order-group

 # 新开一个窗口执行

 [yun@mini03 ~]$ kafka-console-consumer.sh --bootstrap-server mini01: --topic order --group order-group

　　表示order-group消费组有3个消费者，消费topic order的信息。

9.4. 消费组消费位置信息查看

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group order-group  # 查看消费情况

 TOPIC    PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG   CONSUMER-ID                                     HOST            CLIENT-ID

 order                                                 consumer--2e9805db-e021--8c62-92f8691fbf20 /172.16.1.13    consumer-

 order                                                 consumer--2e9805db-e021--8c62-92f8691fbf20 /172.16.1.13    consumer-

 order                                                 consumer--9e65dcfb-246f--aaf7-3ee83532237f /172.16.1.13    consumer-

 order                                                 consumer--ee17939d-1ffe-42c7--b19be8acea43 /172.16.1.12    consumer-

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group order-group --members --verbose

 CONSUMER-ID                                     HOST            CLIENT-ID       #PARTITIONS     ASSIGNMENT

 consumer--9e65dcfb-246f--aaf7-3ee83532237f /172.16.1.13    consumer-                     order()

 consumer--2e9805db-e021--8c62-92f8691fbf20 /172.16.1.13    consumer-                     order(,)

 consumer--ee17939d-1ffe-42c7--b19be8acea43 /172.16.1.12    consumer-                     order()

10. 管理消费组

10.1. 查看所有消费组

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --list

 console-consumer-

 console-consumer-

 console-consumer-

 console-consumer-

 console-consumer-

 console-consumer-

 console-consumer-

10.2. 查看消费组消费情况【消费位置】

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group console-consumer-

 Consumer group 'console-consumer-26727' has no active members.

 TOPIC           PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID     HOST            CLIENT-ID

 zhang                                                                -               -               -

 zhang                                                                  -               -               -

 zhang                                                                  -               -               -

 zhang                                                                -               -               -

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group console-consumer-  

 TOPIC  PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG  CONSUMER-ID                                     HOST            CLIENT-ID

 zhang                                            consumer--17c812f0-116b-42a9-88d8-90d1a85949e1 /172.16.1.13    consumer-

 zhang                                            consumer--17c812f0-116b-42a9-88d8-90d1a85949e1 /172.16.1.13    consumer-

 zhang                                            consumer--17c812f0-116b-42a9-88d8-90d1a85949e1 /172.16.1.13    consumer-

 zhang                                            consumer--17c812f0-116b-42a9-88d8-90d1a85949e1 /172.16.1.13    consumer-

--members

 # --members 此选项提供使用者组中所有活动成员的列表。

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group console-consumer- --members  

 CONSUMER-ID                                     HOST            CLIENT-ID       #PARTITIONS

 consumer--17c812f0-116b-42a9-88d8-90d1a85949e1 /172.16.1.13    consumer-

--verbose

 # --verbose 这个选项还提供了分配给每个成员的分区。

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group console-consumer- --members --verbose

 CONSUMER-ID                                     HOST            CLIENT-ID       #PARTITIONS     ASSIGNMENT

 consumer--17c812f0-116b-42a9-88d8-90d1a85949e1 /172.16.1.13    consumer-                     zhang(,,,)

--state

 # --state  这个选项提供了有用的组级信息。

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group console-consumer- --state

 COORDINATOR (ID)          ASSIGNMENT-STRATEGY       STATE                #MEMBERS

 mini01: ()           range                     Stable

10.3. 删除一个或多个用户组

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --list

 console-consumer-

 console-consumer-

 console-consumer-

 console-consumer-

 console-consumer-

 console-consumer-

 console-consumer-

 # 删除一个或多个组

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --delete --group console-consumer- --group console-consumer-

 Deletion of requested consumer groups ('console-consumer-31713', 'console-consumer-11661') was successful.

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --list

 console-consumer-

 console-consumer-

 console-consumer-

 console-consumer-

 console-consumer-

11. 增加副本因子

 [yun@mini01 kafka_20180916]$ kafka-topics.sh --create --zookeeper mini01: --replication-factor  --partitions  --topic order

 Created topic "order".

 [yun@mini01 kafka_20180916]$ kafka-topics.sh --describe --zookeeper mini01: --topic order

 Topic:order    PartitionCount:    ReplicationFactor:    Configs:

     Topic: order    Partition:     Leader:     Replicas:     Isr:

     Topic: order    Partition:     Leader:     Replicas:     Isr:

     Topic: order    Partition:     Leader:     Replicas:     Isr:

     Topic: order    Partition:     Leader:     Replicas:     Isr:

要求：topic order 的副本数由1变为2，之前每个分区在哪台机器上在上面已给出。

说明：part 0分布在群集0,1； part 1分布在集群1,2；part 2 分布在集群2,0；part 3分布在集群0,1。

11.1. 创建一个重新调整计划文件

 [yun@mini01 kafka_20180916]$ cat increase-replication-factor.json

 {

   "version":,

   "partitions":[

     {"topic": "order","partition": ,"replicas": [,]},

     {"topic": "order","partition": ,"replicas": [,]},

     {"topic": "order","partition": ,"replicas": [,]},

     {"topic": "order","partition": ,"replicas": [,]}

   ]

 }

11.2. 语句执行

 [yun@mini01 kafka_20180916]$ kafka-reassign-partitions.sh --zookeeper mini01: --reassignment-json-file increase-replication-factor.json --execute

 Current partition replica assignment

 {"version":,"partitions":[{"topic":"order","partition":,"replicas":[],"log_dirs":["any"]},{"topic":"order","partition":,"replicas":[],"log_dirs":["any"]},{"topic":"order","partition":,"replicas":[],"log_dirs":["any"]},{"topic":"order","partition":,"replicas":[],"log_dirs":["any"]}]}

 Save this to use as the --reassignment-json-file option during rollback

 Successfully started reassignment of partitions.

11.3. 查看是否执行成功

 [yun@mini01 kafka_20180916]$ kafka-reassign-partitions.sh --zookeeper mini01: --reassignment-json-file increase-replication-factor.json --verify

 Status of partition reassignment:

 Reassignment of partition order- completed successfully

 Reassignment of partition order- completed successfully

 Reassignment of partition order- completed successfully

 Reassignment of partition order- completed successfully

11.4. 再次查看该topic详情

 [yun@mini01 kafka_20180916]$ kafka-topics.sh --describe --zookeeper mini01: --topic order  # 由下可见分配成功

 Topic:order    PartitionCount:    ReplicationFactor:    Configs:

     Topic: order    Partition:     Leader:     Replicas: ,    Isr: ,

     Topic: order    Partition:     Leader:     Replicas: ,    Isr: ,

     Topic: order    Partition:     Leader:     Replicas: ,    Isr: ,

     Topic: order    Partition:     Leader:     Replicas: ,    Isr: ,

12. 创建partitions时在broker的分配机制

 kafka-topics.sh --create --zookeeper mini01: --replication-factor  --partitions  --topic test01

 kafka-topics.sh --create --zookeeper mini01: --replication-factor  --partitions  --topic test02

注意在各机器上partitions的分布

 mini01

     test01-

     test01-

     test02-

     test02-

     test02-

 mini02

     test01-

     test01-

     test02-

     test02-

     test02-

     test02-

 mini03

     test01-

     test02-

     test02-

     test02-

     test02-

kafka_2.11-2.0.0_常用操作的更多相关文章

Hbase-2.0.0_02_常用操作
主要是常用的hbase shell命令,包括表的创建与删除,表数据的增删查[hbase没有修改]:以及hbase的导出与导入. 参考教程:HBase教程参考博客:hbase shell基础和常用命令 ...
『心善渊』Selenium3.0基础 — 11、Selenium对元素常用操作
目录 1.Selenium对元素常用操作 2.Selenium对元素的其他操作 1.Selenium对元素常用操作操作说明 click() 单击元素 send_keys() 模拟输入 clear( ...
Oracle 11gR2 11.2.0.1 ( 11.2.0.1的BUG?)：ohasd不能正常启动：ioctl操作：npohasd的问题：【chmod a+wr /var/tmp/.oracle/npohasd】
问题1:执行安装,编译成功后,执行asmca时,失败,无法成功创建后台相关服务问题2:os系统重启后,ohasd无法正常启动,css服务失败原因:11.2.0.1的BUG:/var/tmp/.or ...
C# 字符串常用操作分类： C# 2014-08-22 15:07 238人阅读评论(0) 收藏
string str1 = "C#操作字符串<几种常见方式>如下"; string str2 = "C#操作字符串"; //比较字符串 Co ...
Python 基礎 - 字符串常用操作
字符串常用操作今天就介紹一下常用的字符串操作,都是以 Python3撰寫的首字母變大寫 #!/usr/bin/env python3 # -*- coding:utf-8 -*- name = & ...
select元素javascript常用操作转
/*------------------------------------------------------ *作者:xieyu @ 2007-08-14 *语言:JavaScript *说明:s ...
guava常用操作
Jack47 我思故我在 Google Java编程库Guava介绍本系列想介绍下Java下开源的优秀编程库--Guava[ˈgwɑːvə].它包含了Google在Java项目中使用一些核心库,包含 ...
[转]Mysql命令行常用操作
Mysql命令行常用操作一.从命令行登录MySQL数据库服务器 1.登录使用默认3306端口的MySQL /usr/local/mysql/bin/mysql -u root -p 2.通过TCP连 ...
NSString&NSMutableString常用操作梳理（转）
作者:弦苦授权本站转载. 上一篇梳理了NSArray&NSMutableArray常用操作,这次来梳理一下Objective-C中每天都要用到的字符串处理类——NSString. Objec ...

随机推荐

分布式系统监视zabbix讲解一之zabbix安装--技术流ken
zabbix概述 Zabbix是什么 Zabbix 是由Alexei Vladishev创建,目前由Zabbix SIA在持续开发和支持. Zabbix 是一个企业级的分布式开源监控方案. Zabbi ...
golang的字符串拼接
常用拼接方法字符串拼接在日常开发中是很常见的需求,目前有两种普遍做法: 一种是直接用 += 来拼接 s1 := "Hello" s2 := "World" s ...
DropDownList按照Gridview获取数据获取到的是定义格式
首先需要把DropDownList改成允许服务器返回. 然后绑定的时候需要以下两项. DropDownList1.DataTextField = "name";DropDownLi ...
PHP之单例模式
之前记得有写过PHP的几种这模式.这几天看群里在问单列模式,觉得还是有必要再深入写清楚下..其实单例模式很好理解滴哦单例模式顾名思义,就是只有一个实例,作为对象的创建模式,单例模式确保某一个类只有一 ...
swoole扩展实现真正的数据库连接池
php的数据库连接池一直以来都是一个难题,很多从php语言转向java的项目,大多数原因可能都是因为java有更好的连接池实现.php的mysql扩展提供了长连接的API,但在php机器数量较多,规模 ...
Maven（八）Eclipse创建Web项目（复杂方式）
1. 生成标准的Web工程结构 2. 勾选结尾为webapp的包 3. 生成的文件结构如下 3.1 生成的目录结构若存在错误,缺少servlet.api 3.1.1 添加步骤如下 4.生成后存在的缺点 ...
Android BrocastReceiver解析
简介 BroadcastReceiver(广播接收器)是Android四大组件之一,是一个用来响应系统范围内的广播组件,可以从Android系统和其它app发送或接收广播消息,类似于发布 - 订阅设计 ...
Springboot使用Filter以及踩过的坑
Springboot使用Filter以及踩过的坑在Springboot中使用Filter有两种方式,注解方式,注册bean方式一.注解@WebFilter 1.实现Filter接口(javax.s ...
Github被微软收购，这里整理了16个替代品
微软斥资75亿美元收购以后,鉴于微软和开源竞争的历史,很多开发者都感到惊恐.毕竟,互联网上最大的一块可以自由的净土被微软染指,宝宝不开森.如果你真的担心微软会对Github有所动作,那么这里我列举了1 ...
Socket编程（网络编程）
网络通信的第一要素:IP地址通过IP地址唯一的定位到互联网的主机通过 IP+port(端口号) 来确定互联网的主机上的某个软件 InetAddress:位于java.net包下 getHostNa ...

kafka_2.11-2.0.0_常用操作