参考博文:Kafka消费组(consumer group)

参考博文:kafka 1.0 中文文档(九):操作

参考博文:kafka集群管理工具kafka-manager部署安装

以下操作可以在mini01、mini02、mini03任意一台操作即可

1. kafka通过网页管理

参考博文:kafka集群管理工具kafka-manager部署安装

2. 创建topic

 # 参数说明 --replication-factor  表示有2个副本
# --partitions 表示有4个分区
[yun@mini01 ~]$ kafka-topics.sh --create --zookeeper mini01: --replication-factor --partitions --topic test
Created topic "test".
[yun@mini01 ~]$ kafka-topics.sh --create --zookeeper mini01: --replication-factor --partitions --topic zhang
Created topic "zhang".
[yun@mini01 ~]$ kafka-topics.sh --list --zookeeper mini01: # 再次查看
zhang
test

2.1. 各主机信息查看

mini01

 [yun@mini01 logs]$ pwd
/app/kafka/logs
[yun@mini01 logs]$ ll
total
………………
drwxrwxr-x yun yun Sep : test-
drwxrwxr-x yun yun Sep : test-
drwxrwxr-x yun yun Sep : test-

mini02

 [yun@mini02 logs]$ pwd
/app/kafka/logs
[yun@mini02 logs]$ ll
total
………………
drwxrwxr-x yun yun Sep : test-
drwxrwxr-x yun yun Sep : test-

mini03

 [yun@mini03 logs]$ pwd
/app/kafka/logs
[yun@mini03 logs]$ ll
total
………………
drwxrwxr-x yun yun Sep : test-
drwxrwxr-x yun yun Sep : test-
drwxrwxr-x yun yun Sep : test-

3. 修改topic

3.1. 增加分区数

注意:分区数不能减少

  Kafka目前不支持减少主题的分区数量。

 [yun@mini01 ~]$ kafka-topics.sh --list --zookeeper mini01:
__consumer_offsets
test
test01
test02
test03
test04
zhang
[yun@mini01 ~]$ kafka-topics.sh --describe --zookeeper mini01: --topic test01
Topic:test01 PartitionCount: ReplicationFactor: Configs:
Topic: test01 Partition: Leader: Replicas: Isr:
Topic: test01 Partition: Leader: Replicas: Isr:
Topic: test01 Partition: Leader: Replicas: Isr:
Topic: test01 Partition: Leader: Replicas: Isr:
Topic: test01 Partition: Leader: Replicas: Isr:
[yun@mini01 ~]$ kafka-topics.sh --alter --zookeeper mini01: --partitions --topic test01 # 失败,分区数不能减少
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Error while executing topic command : The number of partitions for a topic can only be increased. Topic test01 currently has partitions, would not be an increase.
[-- ::,] ERROR org.apache.kafka.common.errors.InvalidPartitionsException: The number of partitions for a topic can only be increased. Topic test01 currently has partitions, would not be an increase.
(kafka.admin.TopicCommand$)
[yun@mini01 ~]$ kafka-topics.sh --alter --zookeeper mini01: --partitions --topic test01 # 增加分区数
WARNING: If partitions are increased for a topic that has a key, the partition logic or ordering of the messages will be affected
Adding partitions succeeded!
[yun@mini01 ~]$ kafka-topics.sh --describe --zookeeper mini01: --topic test01
Topic:test01 PartitionCount: ReplicationFactor: Configs:
Topic: test01 Partition: Leader: Replicas: Isr:
Topic: test01 Partition: Leader: Replicas: Isr:
Topic: test01 Partition: Leader: Replicas: Isr:
Topic: test01 Partition: Leader: Replicas: Isr:
Topic: test01 Partition: Leader: Replicas: Isr:
Topic: test01 Partition: Leader: Replicas: Isr:
Topic: test01 Partition: Leader: Replicas: Isr:

4. 删除topic

 # server.properties中设置delete.topic.enable=true 【当前版本默认就是true】否则只是标记删除或者直接重启
[yun@mini01 ~]$ kafka-topics.sh --delete --zookeeper mini01: --topic test
Topic test is marked for deletion.
Note: This will have no impact if delete.topic.enable is not set to true.
[yun@mini01 ~]$ kafka-topics.sh --list --zookeeper mini01: # 再次查看 只有 zhang,则表示真的删除了
zhang

5. 查看所有topic

 [yun@mini01 ~]$ kafka-topics.sh --list --zookeeper mini01:
__consumer_offsets
test
zhang

6. 查看某个Topic的详情

 [yun@mini01 ~]$ kafka-topics.sh --describe --zookeeper mini01: --topic zhang
Topic:zhang PartitionCount: ReplicationFactor: Configs:
Topic: zhang Partition: Leader: Replicas: ,, Isr: ,,
Topic: zhang Partition: Leader: Replicas: ,, Isr: ,,
Topic: zhang Partition: Leader: Replicas: ,, Isr: ,,
Topic: zhang Partition: Leader: Replicas: ,, Isr: ,,

7. 通过shell命令生产消息

7.1. 输入单条数据

 [yun@mini01 ~]$ kafka-console-producer.sh --broker-list mini01: --topic zhang
>
>
>
>
>
>
>
>
>

7.2. 批量导入数据

 [yun@mini01 zhangliang]$ kafka-console-producer.sh --broker-list mini01: --topic liang < .info

8. 通过shell命令消费消息

 # --from-beginning 从最开始读取
# kafka-console-consumer.sh --zookeeper mini01: --from-beginning --topic zhang # 老版本
[yun@mini01 ~]$ kafka-console-consumer.sh --bootstrap-server mini01: --from-beginning --topic zhang

9. 消费组消费

9.1. 创建topic

 [yun@mini01 ~]$ kafka-topics.sh --create --zookeeper mini01: --replication-factor  --partitions  --topic order
Created topic "order".
[yun@mini01 ~]$ kafka-topics.sh --list --zookeeper mini01: # 查看所有topic列表
__consumer_offsets
order
test
zhang
[yun@mini01 ~]$ kafka-topics.sh --describe --zookeeper mini01: --topic order # 查看topic详情
Topic:order PartitionCount: ReplicationFactor: Configs:
Topic: order Partition: Leader: Replicas: Isr:
Topic: order Partition: Leader: Replicas: Isr:
Topic: order Partition: Leader: Replicas: Isr:
Topic: order Partition: Leader: Replicas: Isr:

9.2. 生产消息

 [yun@mini01 ~]$ kafka-console-producer.sh --broker-list mini01: --topic order
>
>
>
>
>

9.3. 消费组消费消息

mini02机器上运行

 # --group  指定组
[yun@mini02 ~]$ kafka-console-consumer.sh --bootstrap-server mini01: --topic order --group order-group

mini03机器上运行

 # --group  指定组
[yun@mini03 ~]$ kafka-console-consumer.sh --bootstrap-server mini01: --topic order --group order-group # 新开一个窗口执行
[yun@mini03 ~]$ kafka-console-consumer.sh --bootstrap-server mini01: --topic order --group order-group

  表示order-group消费组有3个消费者,消费topic order的信息。

9.4. 消费组消费位置信息查看

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group order-group  # 查看消费情况

 TOPIC    PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG   CONSUMER-ID                                     HOST            CLIENT-ID
order consumer--2e9805db-e021--8c62-92f8691fbf20 /172.16.1.13 consumer-
order consumer--2e9805db-e021--8c62-92f8691fbf20 /172.16.1.13 consumer-
order consumer--9e65dcfb-246f--aaf7-3ee83532237f /172.16.1.13 consumer-
order consumer--ee17939d-1ffe-42c7--b19be8acea43 /172.16.1.12 consumer-
[yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group order-group --members --verbose CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT
consumer--9e65dcfb-246f--aaf7-3ee83532237f /172.16.1.13 consumer- order()
consumer--2e9805db-e021--8c62-92f8691fbf20 /172.16.1.13 consumer- order(,)
consumer--ee17939d-1ffe-42c7--b19be8acea43 /172.16.1.12 consumer- order()

10. 管理消费组

10.1. 查看所有消费组

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --list
console-consumer-
console-consumer-
console-consumer-
console-consumer-
console-consumer-
console-consumer-
console-consumer-

10.2. 查看消费组消费情况【消费位置】

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group console-consumer-
Consumer group 'console-consumer-26727' has no active members. TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
zhang - - -
zhang - - -
zhang - - -
zhang - - -
[yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group console-consumer- TOPIC PARTITION CURRENT-OFFSET LOG-END-OFFSET LAG CONSUMER-ID HOST CLIENT-ID
zhang consumer--17c812f0-116b-42a9-88d8-90d1a85949e1 /172.16.1.13 consumer-
zhang consumer--17c812f0-116b-42a9-88d8-90d1a85949e1 /172.16.1.13 consumer-
zhang consumer--17c812f0-116b-42a9-88d8-90d1a85949e1 /172.16.1.13 consumer-
zhang consumer--17c812f0-116b-42a9-88d8-90d1a85949e1 /172.16.1.13 consumer-

--members

 # --members 此选项提供使用者组中所有活动成员的列表。
[yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group console-consumer- --members CONSUMER-ID HOST CLIENT-ID #PARTITIONS
consumer--17c812f0-116b-42a9-88d8-90d1a85949e1 /172.16.1.13 consumer-

--verbose

 # --verbose 这个选项还提供了分配给每个成员的分区。
[yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group console-consumer- --members --verbose CONSUMER-ID HOST CLIENT-ID #PARTITIONS ASSIGNMENT
consumer--17c812f0-116b-42a9-88d8-90d1a85949e1 /172.16.1.13 consumer- zhang(,,,)

--state

 # --state  这个选项提供了有用的组级信息。
[yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --describe --group console-consumer- --state COORDINATOR (ID) ASSIGNMENT-STRATEGY STATE #MEMBERS
mini01: () range Stable

10.3. 删除一个或多个用户组

 [yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --list
console-consumer-
console-consumer-
console-consumer-
console-consumer-
console-consumer-
console-consumer-
console-consumer-
# 删除一个或多个组
[yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --delete --group console-consumer- --group console-consumer-
Deletion of requested consumer groups ('console-consumer-31713', 'console-consumer-11661') was successful.
[yun@mini01 ~]$ kafka-consumer-groups.sh --bootstrap-server mini01: --list
console-consumer-
console-consumer-
console-consumer-
console-consumer-
console-consumer-

11. 增加副本因子

 [yun@mini01 kafka_20180916]$ kafka-topics.sh --create --zookeeper mini01: --replication-factor  --partitions  --topic order
Created topic "order".
[yun@mini01 kafka_20180916]$ kafka-topics.sh --describe --zookeeper mini01: --topic order
Topic:order PartitionCount: ReplicationFactor: Configs:
Topic: order Partition: Leader: Replicas: Isr:
Topic: order Partition: Leader: Replicas: Isr:
Topic: order Partition: Leader: Replicas: Isr:
Topic: order Partition: Leader: Replicas: Isr:

要求:topic order 的副本数由1变为2, 之前每个分区在哪台机器上在上面已给出。

说明:part 0分布在群集0,1; part 1分布在集群1,2;part 2 分布在集群2,0;part 3分布在集群0,1。

11.1. 创建一个重新调整计划文件

 [yun@mini01 kafka_20180916]$ cat increase-replication-factor.json
{
"version":,
"partitions":[
{"topic": "order","partition": ,"replicas": [,]},
{"topic": "order","partition": ,"replicas": [,]},
{"topic": "order","partition": ,"replicas": [,]},
{"topic": "order","partition": ,"replicas": [,]}
]
}

11.2. 语句执行

 [yun@mini01 kafka_20180916]$ kafka-reassign-partitions.sh --zookeeper mini01: --reassignment-json-file increase-replication-factor.json --execute
Current partition replica assignment {"version":,"partitions":[{"topic":"order","partition":,"replicas":[],"log_dirs":["any"]},{"topic":"order","partition":,"replicas":[],"log_dirs":["any"]},{"topic":"order","partition":,"replicas":[],"log_dirs":["any"]},{"topic":"order","partition":,"replicas":[],"log_dirs":["any"]}]} Save this to use as the --reassignment-json-file option during rollback
Successfully started reassignment of partitions.

11.3. 查看是否执行成功

 [yun@mini01 kafka_20180916]$ kafka-reassign-partitions.sh --zookeeper mini01: --reassignment-json-file increase-replication-factor.json --verify
Status of partition reassignment:
Reassignment of partition order- completed successfully
Reassignment of partition order- completed successfully
Reassignment of partition order- completed successfully
Reassignment of partition order- completed successfully

11.4. 再次查看该topic详情

 [yun@mini01 kafka_20180916]$ kafka-topics.sh --describe --zookeeper mini01: --topic order  # 由下可见分配成功
Topic:order PartitionCount: ReplicationFactor: Configs:
Topic: order Partition: Leader: Replicas: , Isr: ,
Topic: order Partition: Leader: Replicas: , Isr: ,
Topic: order Partition: Leader: Replicas: , Isr: ,
Topic: order Partition: Leader: Replicas: , Isr: ,

12. 创建partitions时在broker的分配机制

 kafka-topics.sh --create --zookeeper mini01: --replication-factor  --partitions  --topic test01
kafka-topics.sh --create --zookeeper mini01: --replication-factor --partitions --topic test02

注意在各机器上partitions的分布

 mini01
test01-
test01-
test02-
test02-
test02- mini02
test01-
test01-
test02-
test02-
test02-
test02- mini03
test01-
test02-
test02-
test02-
test02-

kafka_2.11-2.0.0_常用操作的更多相关文章

  1. Hbase-2.0.0_02_常用操作

    主要是常用的hbase shell命令,包括表的创建与删除,表数据的增删查[hbase没有修改]:以及hbase的导出与导入. 参考教程:HBase教程 参考博客:hbase shell基础和常用命令 ...

  2. 『心善渊』Selenium3.0基础 — 11、Selenium对元素常用操作

    目录 1.Selenium对元素常用操作 2.Selenium对元素的其他操作 1.Selenium对元素常用操作 操作 说明 click() 单击元素 send_keys() 模拟输入 clear( ...

  3. Oracle 11gR2 11.2.0.1 ( 11.2.0.1的BUG?):ohasd不能正常启动:ioctl操作:npohasd的问题:【chmod a+wr /var/tmp/.oracle/npohasd】

    问题1:执行安装,编译成功后,执行asmca时,失败,无法成功创建后台相关服务 问题2:os系统重启后,ohasd无法正常启动,css服务失败 原因:11.2.0.1的BUG:/var/tmp/.or ...

  4. C# 字符串常用操作 分类: C# 2014-08-22 15:07 238人阅读 评论(0) 收藏

    string str1 = "C#操作字符串<几种常见方式>如下"; string str2 = "C#操作字符串";     //比较字符串 Co ...

  5. Python 基礎 - 字符串常用操作

    字符串常用操作 今天就介紹一下常用的字符串操作,都是以 Python3撰寫的 首字母變大寫 #!/usr/bin/env python3 # -*- coding:utf-8 -*- name = & ...

  6. select元素javascript常用操作 转

    /*------------------------------------------------------ *作者:xieyu @ 2007-08-14 *语言:JavaScript *说明:s ...

  7. guava常用操作

    Jack47 我思故我在 Google Java编程库Guava介绍 本系列想介绍下Java下开源的优秀编程库--Guava[ˈgwɑːvə].它包含了Google在Java项目中使用一些核心库,包含 ...

  8. [转]Mysql命令行常用操作

    Mysql命令行常用操作 一.从命令行登录MySQL数据库服务器 1.登录使用默认3306端口的MySQL /usr/local/mysql/bin/mysql -u root -p 2.通过TCP连 ...

  9. NSString&NSMutableString常用操作梳理(转)

    作者:弦苦 授权本站转载. 上一篇梳理了NSArray&NSMutableArray常用操作,这次来梳理一下Objective-C中每天都要用到的字符串处理类——NSString. Objec ...

随机推荐

  1. python对象属性管理(2):property管理属性

    使用Property管理属性 python提供了一种友好的getter.setter.deleter类方法的属性管理工具:property. property()是一个内置函数,它返回一个Proper ...

  2. Go基础系列:为select设置超时时间

    Go channel系列: channel入门 为select设置超时时间 nil channel用法示例 双层channel用法示例 指定goroutine的执行顺序 After() 谁也无法保证某 ...

  3. 动手实践Mybatis插件

    前言 Mybatis的插件开发过程的前提是必须要对Mybatis整个SQL执行过程十分熟悉,这样才能正确覆盖源码保证插件运行,总的来说Mybatis的插件式一种侵入式插件,使用时应该十分注意. 在之前 ...

  4. 机器学习笔记(5) KNN算法

    这篇其实应该作为机器学习的第一篇笔记的,但是在刚开始学习的时候,我还没有用博客记录笔记的打算.所以也就想到哪写到哪了. 你在网上搜索机器学习系列文章的话,大部分都是以KNN(k nearest nei ...

  5. 【转载】阿里云服务器为网站选配Https证书

    数字证书是一个经权威授权机构数字签名.包含公开密钥拥有者信息以及公开密钥的文件,是权威机构颁发给网站的可信凭证.最简单的证书包含一个公开密钥.证书名称以及证书授权中心的数字签名,只在特定的时间内有效. ...

  6. [nodejs] nodejs开发个人博客(五)分配数据

    使用回掉大坑进行取数据 能看明白的就看,看不明白的手动滑稽 /** * 首页控制器 */ var router=express.Router(); /*每页条数*/ var pageSize=5; r ...

  7. Docker命令行与守护进程如何交互?

    译者按: Docker是典型的C/S架构,其守护进程(daemon)与命令行(CLI)是通过REST API进行交互的. 原文: Understanding how the Docker Daemon ...

  8. 三问助你Debug

    译者按: Debug也要三省吾身! 原文: Three Questions About Each Bug You Find 译者: Fundebug 为了保证可读性,本文采用意译而非直译.另外,本文版 ...

  9. 使用CSS实现无滚动条滚动

    我们都知道,撸页面的时候当我们的内容超出了我们的div,往往会出现滚动条,影响美观. 尤其是当我们在做一些导航菜单的时候.滚动条一出现就破坏了UI效果. 我们不希望出现滚动条,也不希望超出去的内容被放 ...

  10. HTML5效果:Canvas 实现圆形进度条并显示数字百分比

    实现效果 1.首先创建html代码 <canvas id="canvas" width="500" height="500" styl ...