背景

在kafka中,正常情况下,同一个group.id下的不同消费者不会消费同样的partition,也即某个partition在任何时刻都只能被具有相同group.id的consumer中的一个消费。
也正是这个机制才能保证kafka的重要特性:

  • 1、可以通过增加partitions和consumer来提升吞吐量;
  • 2、保证同一份消息不会被消费多次。

在KafkaConsumer类中(官方API),消费者可以通过assign和subscribe两种方式指定要消费的topic-partition。具体的源码可以参考下文,

这两个接口貌似是完成相同的功能,但是还有细微的差别,初次使用的同学可能感到困惑,下面就详细介绍下两者的区别。

对比结果

  • KafkaConsumer.subscribe() : 为consumer自动分配partition,有内部算法保证topic-partition以最优的方式均匀分配给同group下的不同consumer。

  • KafkaConsumer.assign() : 为consumer手动、显示的指定需要消费的topic-partitions,不受group.id限制,相当与指定的group无效(this method does not use the consumer's group management)。

测试代码

public class KafkaManualAssignTest {
private static final Logger logger = LoggerFactory.getLogger(KafkaManualAssignTest.class); private static Properties props = new Properties();
private static KafkaConsumer<String, String> c1, c2; private static final String brokerList = "localhost:9092"; static {
props.put("bootstrap.servers", brokerList);
props.put("group.id", "assignTest");
props.put("auto.offset.reset", "earliest");
props.put("enable.auto.commit", "true");
props.put("session.timeout.ms", "30000");
props.put("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
props.put("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer"); c1 = new KafkaConsumer<String, String>(props);
c2 = new KafkaConsumer<String, String>(props);
} public static void main(String[] args) {
TopicPartition tp = new TopicPartition("topic", 0);
// 采用assign方式显示的为consumer指定需要消费的topic, 具有相同group.id的两个消费者
// 各自消费了一份数据, 出现了数据的重复消费
c1.assign(Arrays.asList(tp));
c2.assign(Arrays.asList(tp)); // 采用subscribe方式, 利用broker为consumer自动分配topic-partitions,
// 两个消费者各自消费一个partition, 数据互补, 无交叉.
// c1.subscribe(Arrays.asList("topic"));
// c2.subscribe(Arrays.asList("topic")); while (true) {
ConsumerRecords<String, String> msg1 = c1.poll(1000L);
if (msg1 != null) {
for (ConsumerRecord m1 : msg1) {
logger.info("m1 offset : {} , value : {}", m1.offset(), m1.value());
}
} logger.info("=====================");
ConsumerRecords<String, String> msg2 = c2.poll(1000L);
if (msg2 != null) {
for (ConsumerRecord m2 : msg2) {
logger.info("m2 offset : {} , value : {}", m2.offset(), m2.value());
}
} System.exit(0);
}
}
}
复制代码

官方api

官方关于subscribe的解释:

/**
* Subscribe to the given list of topics to get dynamically assigned partitions.
* <b>Topic subscriptions are not incremental. This list will replace the current
* assignment (if there is one).</b> It is not possible to combine topic subscription with group management
* with manual partition assignment through {@link #assign(Collection)}.
*
* If the given list of topics is empty, it is treated the same as {@link #unsubscribe()}.
*
* <p>
* This is a short-hand for {@link #subscribe(Collection, ConsumerRebalanceListener)}, which
* uses a no-op listener. If you need the ability to seek to particular offsets, you should prefer
* {@link #subscribe(Collection, ConsumerRebalanceListener)}, since group rebalances will cause partition offsets
* to be reset. You should also provide your own listener if you are doing your own offset
* management since the listener gives you an opportunity to commit offsets before a rebalance finishes.
*
* @param topics The list of topics to subscribe to
* @throws IllegalArgumentException If topics is null or contains null or empty elements
* @throws IllegalStateException If {@code subscribe()} is called previously with pattern, or assign is called
* previously (without a subsequent call to {@link #unsubscribe()}), or if not
* configured at-least one partition assignment strategy
*/
@Override
public void subscribe(Collection<String> topics) {
subscribe(topics, new NoOpConsumerRebalanceListener());
}
复制代码

官方关于assign的解释:

/**
* Manually assign a list of partitions to this consumer. This interface does not allow for incremental assignment
* and will replace the previous assignment (if there is one).
* <p>
* If the given list of topic partitions is empty, it is treated the same as {@link #unsubscribe()}.
* <p>
* Manual topic assignment through this method does not use the consumer's group management
* functionality. As such, there will be no rebalance operation triggered when group membership or cluster and topic
* metadata change. Note that it is not possible to use both manual partition assignment with {@link #assign(Collection)}
* and group assignment with {@link #subscribe(Collection, ConsumerRebalanceListener)}.
* <p>
* If auto-commit is enabled, an async commit (based on the old assignment) will be triggered before the new
* assignment replaces the old one.
*
* @param partitions The list of partitions to assign this consumer
* @throws IllegalArgumentException If partitions is null or contains null or empty topics
* @throws IllegalStateException If {@code subscribe()} is called previously with topics or pattern
* (without a subsequent call to {@link #unsubscribe()})
*/
@Override
public void assign(Collection<TopicPartition> partitions) {
acquireAndEnsureOpen();
try {
if (partitions == null) {
throw new IllegalArgumentException("Topic partition collection to assign to cannot be null");
} else if (partitions.isEmpty()) {
this.unsubscribe();
} else {
Set<String> topics = new HashSet<>();
for (TopicPartition tp : partitions) {
String topic = (tp != null) ? tp.topic() : null;
if (topic == null || topic.trim().isEmpty())
throw new IllegalArgumentException("Topic partitions to assign to cannot have null or empty topic");
topics.add(topic);
} // make sure the offsets of topic partitions the consumer is unsubscribing from
// are committed since there will be no following rebalance
this.coordinator.maybeAutoCommitOffsetsAsync(time.milliseconds()); log.debug("Subscribed to partition(s): {}", Utils.join(partitions, ", "));
this.subscriptions.assignFromUser(new HashSet<>(partitions));
metadata.setTopics(topics);
}
} finally {
release();
}
}
复制代码

建议

建议使用 subscribe() 函数来实现partition的分配。

除非各位同学清楚了解自己需要消费的topic-partitions(不是topic),而且能确定自己的消息全部在这些topic-partitions中,则可以使用assign。

KafkaConsumer assign VS subscribe的更多相关文章

  1. kafka consumer assign 和 subscribe模式差异分析

    转载请注明原创地址:http://www.cnblogs.com/dongxiao-yang/p/7200971.html 最近需要研究flink-connector-kafka的消费行为,发现fli ...

  2. 九 assign和subscribe

    1 subscribe:  自动安排分区, 通过group自动重新的负载均衡: 关于Group的实验: 如果auto commit = true, 重新启动进程,如果是同样的groupID,从上次co ...

  3. 利用Kafka的Assign模式实现超大群组(10万+)消息推送

    引言 IM即时通信场景下,最重要的一个能力就是推送:在线的直接通过长连接网关服务转发,离线的通过APNS或者极光等系统进行推送.   本文主要是针对在线用户推送场景来进行总结和探讨:如何利用Kafka ...

  4. 【Kafka源码】KafkaConsumer

    [TOC] KafkaConsumer是从kafka集群消费消息的客户端.这是kafka的高级消费者,而SimpleConsumer是kafka的低级消费者.何为高级?何为低级? 我们所谓的高级,就是 ...

  5. KafkaConsumer 简析

    使用方式 创建一个 KafkaConsumer 对象订阅主题并开始接收消息: Properties properties = new Properties(); properties.setPrope ...

  6. kafka消费者客户端(0.9.0.1API)

    转自:http://orchome.com/203 kafka客户端从kafka集群消费消息(记录).它会透明地处理kafka集群中服务器的故障.它获取集群内数据的分区,也和服务器进行交互,允许消费者 ...

  7. Kafka 0.10.0

    2.1 Producer API We encourage all new development to use the new Java producer. This client is produ ...

  8. Kafka学习-Producer和Customer

    在上一篇kafka入门的基础之上,本篇主要介绍Kafka的生产者和消费者. Kafka 生产者 kafka Producer发布消息记录到Kakfa集群.生产者是线程安全的,可以在多个线程之间共享生产 ...

  9. Kafka的CommitFailedException异常

    一.含义 CommitFailedException异常:位移提交失败时候抛出的异常.通常该异常被抛出时还会携带这样的一段话: Commit cannot be completed since the ...

随机推荐

  1. .NET Core项目部署到Linux(Centos7)(六)发布.NET Core 项目到Linux

    目录 1.前言 2.环境和软件的准备 3.创建.NET Core API项目 4.VMware Workstation虚拟机及Centos 7安装 5.Centos 7安装.NET Core环境 6. ...

  2. Mysql数据库错误代码大全

                                                    Mysql数据库错误代码大全 出现较多的一些网页代码提示的意思: 1016错误:文件无法打开,使用后台修 ...

  3. php 直接跳出嵌套循环

    break 结束当前 for,foreach,while,do-while 或者 switch 结构的执行. break 可以接受一个可选的数字参数来决定跳出几重循环. <?php $arr = ...

  4. Redis系列(五):Redis的过期键删除策略

    本篇博客是Redis系列的第5篇,主要讲解下Redis的过期键删除策略. 本系列的前4篇可以点击以下链接查看: Redis系列(一):Redis简介及环境安装 Redis系列(二):Redis的5种数 ...

  5. Scratch 第2课淘气男孩儿

    素材及视频下载 链接:https://pan.baidu.com/s/1qX0T2B_zczcLaCCpiRrsnA提取码:xfp8

  6. python-nmap 使用基础

    前言 python-nmap是一个Python库,可帮助您使用nmap端口扫描程序.它可以轻松操纵nmap扫描结果,将是一个完美的选择想要自动执行扫描任务的系统管理员的工具和报告. 它还支持nmap脚 ...

  7. python--模块、列表生成式、集合元祖列表

    一.导入模块的两种方式 1.直接使用import import 模块名 #调用 模块名.方法名() 2.使用from…import… from 模块名 import 方法名1,方法名2(from 模块 ...

  8. 「一闻秒懂」你了解goroutine和channel吗?

    开源库「go home」聚焦Go语言技术栈与面试题,以协助Gopher登上更大的舞台,欢迎go home~ 背景介绍 大家都知道进程是操作系统资源分配的基本单位,有独立的内存空间,线程可以共享同一个进 ...

  9. CentOS之crontab

    1.crontab介绍 功能说明:设置计时器. 语 法:crontab [-u <用户名称>][配置文件] 或 crontab [-u <用户名称>][-elr] 补充说明:c ...

  10. python 写一个生成大乐透号码的程序

    """ 写一个生成大乐透号码的程序 生成随机号码:大乐透分前区号码和后区号码, 前区号码是从01-35中无重复地取5个号码, 后区号码是从01-12中无重复地取2个号码, ...