kafka原理和实践（四）spring-kafka消费者源码
系列目录
kafka原理和实践（三）spring-kafka生产者源码
kafka原理和实践（四）spring-kafka消费者源码
==============正文分割线=====================
一、kafkaConsumer消费者模型
如上图所示，spring-kafka消费者模型主要流程：
1.容器启动，轮询执行消费。
2.kafkaConsumer拉取消息流程：
1）Fetcher请求获取器获取请求并存储在unset中
2）ConsumerNetworkClient网络客户端执行poll(),调用NetWlrikClient的send()方法从unset中获取ClientRequest请求转成RequestSend最终塞进Selector的KafkaChannel通道中，Seletcor.send()从kafka集群拉取待消费数据ConsumerRecords
3. 消费者监听器MessageListener.onMessage()执行用户自定义的实际消费业务逻辑。
一、kafkaConsumer构造
@SuppressWarnings("unchecked")
private KafkaConsumer(ConsumerConfig config,
Deserializer<K> keyDeserializer,
Deserializer<V> valueDeserializer) {
try {
log.debug("Starting the Kafka consumer");
this.requestTimeoutMs = config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG);
int sessionTimeOutMs = config.getInt(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG);
int fetchMaxWaitMs = config.getInt(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG);
if (this.requestTimeoutMs <= sessionTimeOutMs || this.requestTimeoutMs <= fetchMaxWaitMs)
throw new ConfigException(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG + " should be greater than " + ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG + " and " + ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG);
this.time = new SystemTime();
String clientId = config.getString(ConsumerConfig.CLIENT_ID_CONFIG);
if (clientId.length() <= 0)
clientId = "consumer-" + CONSUMER_CLIENT_ID_SEQUENCE.getAndIncrement();
this.clientId = clientId;
Map<String, String> metricsTags = new LinkedHashMap<>();
metricsTags.put("client-id", clientId);
MetricConfig metricConfig = new MetricConfig().samples(config.getInt(ConsumerConfig.METRICS_NUM_SAMPLES_CONFIG))
.timeWindow(config.getLong(ConsumerConfig.METRICS_SAMPLE_WINDOW_MS_CONFIG), TimeUnit.MILLISECONDS)
.tags(metricsTags);
List<MetricsReporter> reporters = config.getConfiguredInstances(ConsumerConfig.METRIC_REPORTER_CLASSES_CONFIG,
MetricsReporter.class);
reporters.add(new JmxReporter(JMX_PREFIX));
this.metrics = new Metrics(metricConfig, reporters, time);
this.retryBackoffMs = config.getLong(ConsumerConfig.RETRY_BACKOFF_MS_CONFIG);
// load interceptors and make sure they get clientId
Map<String, Object> userProvidedConfigs = config.originals();
userProvidedConfigs.put(ConsumerConfig.CLIENT_ID_CONFIG, clientId);
List<ConsumerInterceptor<K, V>> interceptorList = (List) (new ConsumerConfig(userProvidedConfigs)).getConfiguredInstances(ConsumerConfig.INTERCEPTOR_CLASSES_CONFIG,
ConsumerInterceptor.class);
this.interceptors = interceptorList.isEmpty() ? null : new ConsumerInterceptors<>(interceptorList);
if (keyDeserializer == null) {
this.keyDeserializer = config.getConfiguredInstance(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG,
Deserializer.class);
this.keyDeserializer.configure(config.originals(), true);
} else {
config.ignore(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG);
this.keyDeserializer = keyDeserializer;
}
if (valueDeserializer == null) {
this.valueDeserializer = config.getConfiguredInstance(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG,
Deserializer.class);
this.valueDeserializer.configure(config.originals(), false);
} else {
config.ignore(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG);
this.valueDeserializer = valueDeserializer;
}
ClusterResourceListeners clusterResourceListeners = configureClusterResourceListeners(keyDeserializer, valueDeserializer, reporters, interceptorList);
this.metadata = new Metadata(retryBackoffMs, config.getLong(ConsumerConfig.METADATA_MAX_AGE_CONFIG), false, clusterResourceListeners);
List<InetSocketAddress> addresses = ClientUtils.parseAndValidateAddresses(config.getList(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG));
this.metadata.update(Cluster.bootstrap(addresses), 0);
String metricGrpPrefix = "consumer";
ChannelBuilder channelBuilder = ClientUtils.createChannelBuilder(config.values());
NetworkClient netClient = new NetworkClient(
new Selector(config.getLong(ConsumerConfig.CONNECTIONS_MAX_IDLE_MS_CONFIG), metrics, time, metricGrpPrefix, channelBuilder),
this.metadata,
clientId,
100, // a fixed large enough value will suffice
config.getLong(ConsumerConfig.RECONNECT_BACKOFF_MS_CONFIG),
config.getInt(ConsumerConfig.SEND_BUFFER_CONFIG),
config.getInt(ConsumerConfig.RECEIVE_BUFFER_CONFIG),
config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG), time);
this.client = new ConsumerNetworkClient(netClient, metadata, time, retryBackoffMs,
config.getInt(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG));
OffsetResetStrategy offsetResetStrategy = OffsetResetStrategy.valueOf(config.getString(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG).toUpperCase(Locale.ROOT));
this.subscriptions = new SubscriptionState(offsetResetStrategy);
List<PartitionAssignor> assignors = config.getConfiguredInstances(
ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG,
PartitionAssignor.class);
this.coordinator = new ConsumerCoordinator(this.client,
config.getString(ConsumerConfig.GROUP_ID_CONFIG),
config.getInt(ConsumerConfig.MAX_POLL_INTERVAL_MS_CONFIG),
config.getInt(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG),
config.getInt(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG),
assignors,
this.metadata,
this.subscriptions,
metrics,
metricGrpPrefix,
this.time,
retryBackoffMs,
new ConsumerCoordinator.DefaultOffsetCommitCallback(),
config.getBoolean(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG),
config.getInt(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG),
this.interceptors,
config.getBoolean(ConsumerConfig.EXCLUDE_INTERNAL_TOPICS_CONFIG));
this.fetcher = new Fetcher<>(this.client,
config.getInt(ConsumerConfig.FETCH_MIN_BYTES_CONFIG),
config.getInt(ConsumerConfig.FETCH_MAX_BYTES_CONFIG),
config.getInt(ConsumerConfig.FETCH_MAX_WAIT_MS_CONFIG),
config.getInt(ConsumerConfig.MAX_PARTITION_FETCH_BYTES_CONFIG),
config.getInt(ConsumerConfig.MAX_POLL_RECORDS_CONFIG),
config.getBoolean(ConsumerConfig.CHECK_CRCS_CONFIG),
this.keyDeserializer,
this.valueDeserializer,
this.metadata,
this.subscriptions,
metrics,
metricGrpPrefix,
this.time,
this.retryBackoffMs);
config.logUnused();
AppInfoParser.registerAppInfo(JMX_PREFIX, clientId);
log.debug("Kafka consumer created");
} catch (Throwable t) {
// call close methods if internal objects are already constructed
// this is to prevent resource leak. see KAFKA-2121
close(true);
// now propagate the exception
throw new KafkaException("Failed to construct kafka consumer", t);
}
}
从KafkaConsumer构造函数来看，核心组件有：
1.Metadata：封装了元数据的一些逻辑的类。元数据仅保留一个主题的子集，随着时间的推移可以添加。当我们请求一个主题的元数据时，我们没有任何元数据会触发元数据更新。如果对元数据启用了主题过期，那么在更新之后，在过期时间间隔内未使用的任何主题都将从元数据刷新集中删除。
2.ConsumerNetworkClient：高等级消费者访问网络层，为请求Future任务提供基本支持。这个类是线程安全的，但是不提供响应回调的同步。这保证在调用它们时不会持有锁。
3.SubscriptionState：订阅的TopicPartition的offset状态维护
4.ConsumerCoordinator：消费者的协调者，负责partitiion的分配，reblance
5.Fetcher：从brokers上按照配置获取消息。
二、消费者容器启动流程
kafka消费者有两种常见的实现方式：
1.xml配置文件
2.基于注解实现
其实，不管哪种方式，本质只是生成Spring Bean的方式不同而已。我们就以xml的实现方式来追踪源码。
基于xml的总体配置如下：
1 <!-- 1.定义consumer的参数 -->
2 <bean id="consumerProperties" class="java.util.HashMap">
3 <constructor-arg>
4 <map>
5 <entry key="bootstrap.servers" value="${bootstrap.servers}" />
6 <entry key="group.id" value="${group.id}" />
7 <entry key="enable.auto.commit" value="${enable.auto.commit}" />
8 <entry key="session.timeout.ms" value="${session.timeout.ms}" />
9 <entry key="key.deserializer"
10 value="org.apache.kafka.common.serialization.StringDeserializer" />
11 <entry key="value.deserializer"
12 value="org.apache.kafka.common.serialization.StringDeserializer" />
13 </map>
14 </constructor-arg>
15 </bean>
16
17 <!-- 2.创建consumerFactory bean -->
18 <bean id="consumerFactory"
19 class="org.springframework.kafka.core.DefaultKafkaConsumerFactory" >
20 <constructor-arg>
21 <ref bean="consumerProperties" />
22 </constructor-arg>
23 </bean>
24
25 <!-- 3.定义消费实现类 -->
26 <bean id="kafkaConsumerService" class="xxx.service.impl.KafkaConsumerSerivceImpl" />
27
28 <!-- 4.消费者容器配置信息 -->
29 <bean id="containerProperties" class="org.springframework.kafka.listener.config.ContainerProperties">
30 <!-- topic -->
31 <constructor-arg name="topics">
32 <list>
33 <value>${kafka.consumer.topic.credit.for.lease}</value>
34 <value>${loan.application.feedback.topic}</value>
35 <value>${templar.agreement.feedback.topic}</value>
36 <value>${templar.aggrement.active.feedback.topic}</value>
37 <value>${templar.aggrement.agreementRepaid.topic}</value>
38 <value>${templar.aggrement.agreementWithhold.topic}</value>
39 <value>${templar.aggrement.agreementRepayRemind.topic}</value>
40 </list>
41 </constructor-arg>
42 <property name="messageListener" ref="kafkaConsumerService" />
43 </bean>
44 <!-- 5.消费者并发消息监听容器，执行doStart()方法 -->
45 <bean id="messageListenerContainer" class="org.springframework.kafka.listener.ConcurrentMessageListenerContainer" init-method="doStart" >
46 <constructor-arg ref="consumerFactory" />
47 <constructor-arg ref="containerProperties" />
48 <property name="concurrency" value="${concurrency}" />
49 </bean>
分为5个步骤：
2.1.定义消费参数bean
consumerProperties ,就是个map<key,value>
2.2.创建consumerFactory bean
DefaultKafkaConsumerFactory 实现了ConsumerFactory接口，提供创建消费者和判断是否自动提交2个方法。通过consumerProperties作为参数构造。
public interface ConsumerFactory<K, V> {
Consumer<K, V> createConsumer();
boolean isAutoCommit();
}
2.3.定义消费实现类
自定义一个类实现MessageListener接口，接口设计如下：
实现onMessage方法，去消费接收到的消息。两种方案：
1）MessageListener 消费完消息后自动提交offset(enable.auto.commit=true时),可提高效率，存在消费失败但移动了偏移量的风险。
2）AcknowledgingMessageListener 消费完消息后手动提交offset(enable.auto.commit=false时)效率降低，无消费失败但移动偏移量的风险。
2.4.监听容器配置信息
ContainerProperties：包含了一个监听容器的运行时配置信息，主要定义了监听的主题、分区、初始化偏移量，还有消息监听器。
public class ContainerProperties {
private static final int DEFAULT_SHUTDOWN_TIMEOUT = 10000;
private static final int DEFAULT_QUEUE_DEPTH = 1;
private static final int DEFAULT_PAUSE_AFTER = 10000;
/**
* Topic names.监听的主题字符串数组
*/
private final String[] topics;
/**
* Topic pattern.监听的主题模板
*/
private final Pattern topicPattern;
/**
* Topics/partitions/initial offsets.
*/
private final TopicPartitionInitialOffset[] topicPartitions;
/**
* 确认模式（自动确认属性为false时使用）
* <ul>
* <li>1.RECORD逐条确认: 每条消息被发送给监听者后确认</li>
* <li>2.BATCH批量确认: 当批量消息记录被消费者接收到并传送给监听器时确认</li>
* <li>3.TIME超时确认:当超过设置的超时时间毫秒数时确认(should be greater than
* {@code #setPollTimeout(long) pollTimeout}.</li>
* <li>4.COUNT计数确认: 当接收到指定数量之后确认</li>
* <li>5.MANUAL手动确认：由监听器负责确认（AcknowledgingMessageListener）</ul>
*/
private AbstractMessageListenerContainer.AckMode ackMode = AckMode.BATCH;
/**
* The number of outstanding record count after which offsets should be
* committed when {@link AckMode#COUNT} or {@link AckMode#COUNT_TIME} is being
* used.
*/
private int ackCount;
/**
* The time (ms) after which outstanding offsets should be committed when
* {@link AckMode#TIME} or {@link AckMode#COUNT_TIME} is being used. Should be
* larger than
*/
private long ackTime;
/**
* 消息监听器，必须是 MessageListener或者AcknowledgingMessageListener两者中的一个 * */
private Object messageListener;
/**
* The max time to block in the consumer waiting for records.
*/
private volatile long pollTimeout = 1000;
/**
* 线程执行器：轮询消费者
*/
private AsyncListenableTaskExecutor consumerTaskExecutor;
/**
* 线程执行器：调用监听器
*/
private AsyncListenableTaskExecutor listenerTaskExecutor;
/**
* 错误回调，当监听器抛出异常时
*/
private GenericErrorHandler<?> errorHandler;
/**
* When using Kafka group management and {@link #setPauseEnabled(boolean)} is
* true, the delay after which the consumer should be paused. Default 10000.
*/
private long pauseAfter = DEFAULT_PAUSE_AFTER;
/**
* When true, avoids rebalancing when this consumer is slow or throws a
* qualifying exception - pauses the consumer. Default: true.
* @see #pauseAfter
*/
private boolean pauseEnabled = true;
/**
* Set the queue depth for handoffs from the consumer thread to the listener
* thread. Default 1 (up to 2 in process).
*/
private int queueDepth = DEFAULT_QUEUE_DEPTH;
/**
* 停止容器超时时间 */
private long shutdownTimeout = DEFAULT_SHUTDOWN_TIMEOUT;
/**
* 用户定义的消费者再平衡监听器实现类 */
private ConsumerRebalanceListener consumerRebalanceListener;
/**
* 提交回调，默认记录日志。 */
private OffsetCommitCallback commitCallback;
/**
* Whether or not to call consumer.commitSync() or commitAsync() when the
* container is responsible for commits. Default true. See
* https://github.com/spring-projects/spring-kafka/issues/62 At the time of
* writing, async commits are not entirely reliable.
*/
private boolean syncCommits = true;
private boolean ackOnError = true;
private Long idleEventInterval;
public ContainerProperties(String... topics) {
Assert.notEmpty(topics, "An array of topicPartitions must be provided");
this.topics = Arrays.asList(topics).toArray(new String[topics.length]);
this.topicPattern = null;
this.topicPartitions = null;
}
public ContainerProperties(Pattern topicPattern) {
this.topics = null;
this.topicPattern = topicPattern;
this.topicPartitions = null;
}
public ContainerProperties(TopicPartitionInitialOffset... topicPartitions) {
this.topics = null;
this.topicPattern = null;
Assert.notEmpty(topicPartitions, "An array of topicPartitions must be provided");
this.topicPartitions = new LinkedHashSet<>(Arrays.asList(topicPartitions))
.toArray(new TopicPartitionInitialOffset[topicPartitions.length]);
}
...省略各种set、get
}
2.5.启动并发消息监听容器
核心类ConcurrentMessageListenerContainer,继承自抽象类AbstractMessageListenerContainer，类图如下：
看上图可知AbstractMessageListenerContainer有2个实现类分别对应单线程和多线程，建议采用多线程消费。下面分析一下主要ConcurrentMessageListenerContainer类，注意2个方法：
1.构造函数，入参：消费者工厂ConsumerFactory+容器配置ContainerProperties
2.doStart()：核心方法KafkaMessageListenerContainer的start()方法。源码如下：
public class ConcurrentMessageListenerContainer<K, V> extends AbstractMessageListenerContainer<K, V> {
private final ConsumerFactory<K, V> consumerFactory;
private final List<KafkaMessageListenerContainer<K, V>> containers = new ArrayList<>();
private int concurrency = 1;
/**
* Construct an instance with the supplied configuration properties.
* The topic partitions are distributed evenly across the delegate
* {@link KafkaMessageListenerContainer}s.
* @param consumerFactory the consumer factory.
* @param containerProperties the container properties.
*/
public ConcurrentMessageListenerContainer(ConsumerFactory<K, V> consumerFactory,
ContainerProperties containerProperties) {
super(containerProperties);
Assert.notNull(consumerFactory, "A ConsumerFactory must be provided");
this.consumerFactory = consumerFactory;
}
public int getConcurrency() {
return this.concurrency;
}
/**
* The maximum number of concurrent {@link KafkaMessageListenerContainer}s running.
* Messages from within the same partition will be processed sequentially.
* @param concurrency the concurrency.
*/
public void setConcurrency(int concurrency) {
Assert.isTrue(concurrency > 0, "concurrency must be greater than 0");
this.concurrency = concurrency;
}
/**
* Return the list of {@link KafkaMessageListenerContainer}s created by
* this container.
* @return the list of {@link KafkaMessageListenerContainer}s created by
* this container.
*/
public List<KafkaMessageListenerContainer<K, V>> getContainers() {
return Collections.unmodifiableList(this.containers);
}
/*
* Under lifecycle lock.
*/
@Override
protected void doStart() {
if (!isRunning()) {
ContainerProperties containerProperties = getContainerProperties();
TopicPartitionInitialOffset[] topicPartitions = containerProperties.getTopicPartitions();
if (topicPartitions != null//校验并发数>分区数，报错。
&& this.concurrency > topicPartitions.length) {
this.logger.warn("When specific partitions are provided, the concurrency must be less than or "
+ "equal to the number of partitions; reduced from " + this.concurrency + " to "
+ topicPartitions.length);
this.concurrency = topicPartitions.length;//并发数最大只能=分区数
}
setRunning(true);
//遍历创建监听器容器
for (int i = 0; i < this.concurrency; i++) {
KafkaMessageListenerContainer<K, V> container;
if (topicPartitions == null) {
container = new KafkaMessageListenerContainer<>(this.consumerFactory, containerProperties);
}
else {
container = new KafkaMessageListenerContainer<>(this.consumerFactory, containerProperties,
partitionSubset(containerProperties, i));
}
if (getBeanName() != null) {
container.setBeanName(getBeanName() + "-" + i);
}
if (getApplicationEventPublisher() != null) {
container.setApplicationEventPublisher(getApplicationEventPublisher());
}
container.setClientIdSuffix("-" + i);
container.start();//核心方法，启动容器
this.containers.add(container);
}
}
} ...省略
}
继续追踪，调用AbstractMessageListenerContainer的doStart(),值得注意的是start()和stop方法加了同一把锁，用于锁住生命周期。
private final Object lifecycleMonitor = new Object(); @Override
public final void start() {
synchronized (this.lifecycleMonitor) {
Assert.isTrue(
this.containerProperties.getMessageListener() instanceof KafkaDataListener,
"A " + KafkaDataListener.class.getName() + " implementation must be provided");
doStart();
}
} protected abstract void doStart();
最终调用的是KafkaMessageListenerContainer的doStart()
@Override
protected void doStart() {
if (isRunning()) {
return;
}
ContainerProperties containerProperties = getContainerProperties(); if (!this.consumerFactory.isAutoCommit()) {
AckMode ackMode = containerProperties.getAckMode();
if (ackMode.equals(AckMode.COUNT) || ackMode.equals(AckMode.COUNT_TIME)) {
Assert.state(containerProperties.getAckCount() > 0, "'ackCount' must be > 0");
}
if ((ackMode.equals(AckMode.TIME) || ackMode.equals(AckMode.COUNT_TIME))
&& containerProperties.getAckTime() == 0) {
containerProperties.setAckTime(5000);
}
} Object messageListener = containerProperties.getMessageListener();
Assert.state(messageListener != null, "A MessageListener is required");
if (messageListener instanceof GenericAcknowledgingMessageListener) {
this.acknowledgingMessageListener = (GenericAcknowledgingMessageListener<?>) messageListener;
}
else if (messageListener instanceof GenericMessageListener) {
this.listener = (GenericMessageListener<?>) messageListener;
}
else {
throw new IllegalStateException("messageListener must be 'MessageListener' "
+ "or 'AcknowledgingMessageListener', not " + messageListener.getClass().getName());
}
if (containerProperties.getConsumerTaskExecutor() == null) {
SimpleAsyncTaskExecutor consumerExecutor = new SimpleAsyncTaskExecutor(
(getBeanName() == null ? "" : getBeanName()) + "-C-");
containerProperties.setConsumerTaskExecutor(consumerExecutor);
}
if (containerProperties.getListenerTaskExecutor() == null) {
SimpleAsyncTaskExecutor listenerExecutor = new SimpleAsyncTaskExecutor(
(getBeanName() == null ? "" : getBeanName()) + "-L-");
containerProperties.setListenerTaskExecutor(listenerExecutor);
}//1.构建 监听消费者
this.listenerConsumer = new ListenerConsumer(this.listener, this.acknowledgingMessageListener);
setRunning(true);
//2.异步提交 监听消费者任务，返回Future并赋值。
this.listenerConsumerFuture = containerProperties
.getConsumerTaskExecutor()
.submitListenable(this.listenerConsumer);
}
doStart主要包含2个操作：构建内部类ListenerConsumer和提交 监听消费者任务，返回Future并赋值。
1.构建内部类ListenerConsumer
ListenerConsumer类图如下：
ListenerConsumer构造函数源码如下：
@SuppressWarnings("unchecked")
ListenerConsumer(GenericMessageListener<?> listener, GenericAcknowledgingMessageListener<?> ackListener) {
Assert.state(!this.isAnyManualAck || !this.autoCommit,
"Consumer cannot be configured for auto commit for ackMode " + this.containerProperties.getAckMode());
@SuppressWarnings("deprecation")
final Consumer<K, V> consumer =
KafkaMessageListenerContainer.this.consumerFactory instanceof
org.springframework.kafka.core.ClientIdSuffixAware
? ((org.springframework.kafka.core.ClientIdSuffixAware<K, V>) KafkaMessageListenerContainer
.this.consumerFactory)
.createConsumer(KafkaMessageListenerContainer.this.clientIdSuffix)
: KafkaMessageListenerContainer.this.consumerFactory.createConsumer();
this.theListener = listener == null ? ackListener : listener;
ConsumerRebalanceListener rebalanceListener = createRebalanceListener(consumer);
if (KafkaMessageListenerContainer.this.topicPartitions == null) {
if (this.containerProperties.getTopicPattern() != null) {
consumer.subscribe(this.containerProperties.getTopicPattern(), rebalanceListener);
}
else {
consumer.subscribe(Arrays.asList(this.containerProperties.getTopics()), rebalanceListener);
}
}
else {
List<TopicPartitionInitialOffset> topicPartitions =
Arrays.asList(KafkaMessageListenerContainer.this.topicPartitions);
this.definedPartitions = new HashMap<>(topicPartitions.size());
for (TopicPartitionInitialOffset topicPartition : topicPartitions) {
this.definedPartitions.put(topicPartition.topicPartition(),
new OffsetMetadata(topicPartition.initialOffset(), topicPartition.isRelativeToCurrent()));
}
consumer.assign(new ArrayList<>(this.definedPartitions.keySet()));
}
this.consumer = consumer;
GenericErrorHandler<?> errHandler = this.containerProperties.getGenericErrorHandler();
this.genericListener = listener;
//1.
if (this.theListener instanceof BatchAcknowledgingMessageListener) {
this.listener = null;
this.batchListener = null;
this.acknowledgingMessageListener = null;
this.batchAcknowledgingMessageListener = (BatchAcknowledgingMessageListener<K, V>) this.theListener;
this.isBatchListener = true;
}//2.
else if (this.theListener instanceof AcknowledgingMessageListener) {
this.listener = null;
this.acknowledgingMessageListener = (AcknowledgingMessageListener<K, V>) this.theListener;
this.batchListener = null;
this.batchAcknowledgingMessageListener = null;
this.isBatchListener = false;
}//3.
else if (this.theListener instanceof BatchMessageListener) {
this.listener = null;
this.batchListener = (BatchMessageListener<K, V>) this.theListener;
this.acknowledgingMessageListener = null;
this.batchAcknowledgingMessageListener = null;
this.isBatchListener = true;
}//4.
else if (this.theListener instanceof MessageListener) {
this.listener = (MessageListener<K, V>) this.theListener;
this.batchListener = null;
this.acknowledgingMessageListener = null;
this.batchAcknowledgingMessageListener = null;
this.isBatchListener = false;
}
else {
throw new IllegalArgumentException("Listener must be one of 'MessageListener', "
+ "'BatchMessageListener', 'AcknowledgingMessageListener', "
+ "'BatchAcknowledgingMessageListener', not " + this.theListener.getClass().getName());
}
if (this.isBatchListener) {
validateErrorHandler(true);
this.errorHandler = new LoggingErrorHandler();
this.batchErrorHandler = errHandler == null ? new BatchLoggingErrorHandler()
: (BatchErrorHandler) errHandler;
}
else {
validateErrorHandler(false);
this.errorHandler = errHandler == null ? new LoggingErrorHandler() : (ErrorHandler) errHandler;
this.batchErrorHandler = new BatchLoggingErrorHandler();
}
Assert.state(!this.isBatchListener || !this.isRecordAck, "Cannot use AckMode.RECORD with a batch listener");
}
1.定义消费者订阅topic或者指定分区
2.设置监听器，支持4种：
1）BatchAcknowledgingMessageListener批量需确认消息监听器
2）AcknowledgingMessageListener需确认消息监听器
3）BatchMessageListener批量消息监听器
4）MessageListener消息监听器（用的最多，一次消费一条）
2.提交 监听消费者任务（ListenerConsumer），返回Future并赋值。
这里我们看一下任务Runnable接口的run方法，分两种情况
1.如果自定义了分区,没必要再平衡分配分区了，直接回调
2.未指定分区，进入自旋消费
@Override
public void run() {
if (this.genericListener instanceof ConsumerSeekAware) {
((ConsumerSeekAware) this.genericListener).registerSeekCallback(this);
}
this.count = 0;
this.last = System.currentTimeMillis();
if (isRunning() && this.definedPartitions != null) {// 1.如果运行中且自定义了分区,没必要再平衡分配分区了，直接回调
initPartitionsIfNeeded();// 有需要就初始化分区
// 回调
if (!this.autoCommit) {
startInvoker();
}
}
long lastReceive = System.currentTimeMillis();
long lastAlertAt = lastReceive;
while (isRunning()) {//2.未指定分区，进入自旋消费
try {
if (!this.autoCommit) {
processCommits();// 如果手动提交，处理提交
}
processSeeks();// 重新定位偏移量，下一次消费时使用
if (this.logger.isTraceEnabled()) {
this.logger.trace("Polling (paused=" + this.paused + ")...");
}// 1）拉取消费记录
ConsumerRecords<K, V> records = this.consumer.poll(this.containerProperties.getPollTimeout());
if (records != null && this.logger.isDebugEnabled()) {
this.logger.debug("Received: " + records.count() + " records");
}
if (records != null && records.count() > 0) {
if (this.containerProperties.getIdleEventInterval() != null) {
lastReceive = System.currentTimeMillis();
}// 2）如果设置了自动提交，直接在当前线程执行
if (this.autoCommit) {
invokeListener(records);
}
else {// 3）否则发送消息进缓存队列
if (sendToListener(records)) {
if (this.assignedPartitions != null) {
// avoid group management rebalance due to a slow
// consumer
this.consumer.pause(this.assignedPartitions);
this.paused = true;
this.unsent = records;
}
}
}
}
else {
if (this.containerProperties.getIdleEventInterval() != null) {
long now = System.currentTimeMillis();
if (now > lastReceive + this.containerProperties.getIdleEventInterval()
&& now > lastAlertAt + this.containerProperties.getIdleEventInterval()) {
publishIdleContainerEvent(now - lastReceive);
lastAlertAt = now;
if (this.genericListener instanceof ConsumerSeekAware) {
seekPartitions(getAssignedPartitions(), true);
}
}
}
}
this.unsent = checkPause(this.unsent);
}
catch (WakeupException e) {
this.unsent = checkPause(this.unsent);
}
catch (Exception e) {
if (this.containerProperties.getGenericErrorHandler() != null) {
this.containerProperties.getGenericErrorHandler().handle(e, null);
}
else {
this.logger.error("Container exception", e);
}
}
}
if (this.listenerInvokerFuture != null) {
stopInvoker();
commitManualAcks();
}
try {
this.consumer.unsubscribe();
}
catch (WakeupException e) {
// No-op. Continue process
}
this.consumer.close();
if (this.logger.isInfoEnabled()) {
this.logger.info("Consumer stopped");
}
}
1.如果用户自定义了分区且非自动提交，那么开启异步线程执行ListenerInvoker任务，源码如下：
private void startInvoker() {
ListenerConsumer.this.invoker = new ListenerInvoker();
ListenerConsumer.this.listenerInvokerFuture = this.containerProperties.getListenerTaskExecutor()
.submit(ListenerConsumer.this.invoker);
}
执行ListenerInvoker的run方法，实际上就执行一遍，因为CountDownLatch初始化为1
private final class ListenerInvoker implements SchedulingAwareRunnable {
private final CountDownLatch exitLatch = new CountDownLatch(1);
private volatile boolean active = true;
private volatile Thread executingThread;
ListenerInvoker() {
super();
}
@Override
public void run() {
Assert.isTrue(this.active, "This instance is not active anymore");
if (ListenerConsumer.this.theListener instanceof ConsumerSeekAware) {
((ConsumerSeekAware) ListenerConsumer.this.theListener).registerSeekCallback(ListenerConsumer.this);
}
try {
this.executingThread = Thread.currentThread();
while (this.active) {
try {// 从阻塞队列LinkedBlockingQueue recordsToProcess中拉取 待消费记录
ConsumerRecords<K, V> records = ListenerConsumer.this.recordsToProcess.poll(1,
TimeUnit.SECONDS);
if (this.active) {
if (records != null) {
invokeListener(records);// 消费
}
else {
if (ListenerConsumer.this.logger.isTraceEnabled()) {
ListenerConsumer.this.logger.trace("No records to process");
}
}
}
}
catch (InterruptedException e) {
if (!this.active) {
Thread.currentThread().interrupt();
}
else {
ListenerConsumer.this.logger.debug("Interrupt ignored");
}
}
}
}
finally {
this.active = false;
this.exitLatch.countDown();
}
}
@Override
public boolean isLongLived() {
return true;
}
}
private void invokeListener(final ConsumerRecords<K, V> records) {
if (this.isBatchListener) {
invokeBatchListener(records);
}
else {
invokeRecordListener(records);
}
}
如上图，从阻塞队列中取得待消费记录，用迭代器iterator消费，根据自定义消费类型，用不同listener来执行onMessage方法（用户自定义MessageListener接口的onMessage方法，实现用户自己的消费业务逻辑）
private void invokeRecordListener(final ConsumerRecords<K, V> records) {
Iterator<ConsumerRecord<K, V>> iterator = records.iterator();
while (iterator.hasNext() && (this.autoCommit || (this.invoker != null && this.invoker.active))) {
final ConsumerRecord<K, V> record = iterator.next();
if (this.logger.isTraceEnabled()) {
this.logger.trace("Processing " + record);
}
try {
if (this.acknowledgingMessageListener != null) {
this.acknowledgingMessageListener.onMessage(record,// 终极核心方法，用户自定义的MessageListener接口的onMessage方法
this.isAnyManualAck
? new ConsumerAcknowledgment(record, this.isManualImmediateAck)
: null);
}
else {
this.listener.onMessage(record);// 终极核心方法，用户自定义的MessageListener接口的onMessage方法
}
if (!this.isAnyManualAck && !this.autoCommit) {
this.acks.add(record);
}
}
catch (Exception e) {
if (this.containerProperties.isAckOnError() && !this.autoCommit) {
this.acks.add(record);
}
try {
this.errorHandler.handle(e, record);
}
catch (Exception ee) {
this.logger.error("Error handler threw an exception", ee);
}
catch (Error er) { //NOSONAR
this.logger.error("Error handler threw an error", er);
throw er;
}
}
}
}
2.未指定分区，进入自旋
// 1）拉取消费记录
ConsumerRecords<K, V> records = this.consumer.poll(this.containerProperties.getPollTimeout());
2）如果设置了自动提交，直接在当前线程执行
invokeListener(records);
// 3）否则发送消息进缓存队列
sendToListener(records)
1）在每个轮询中，消费者将尝试使用最后一个被使用的偏移量作为起始偏移量，并按顺序提取。最后一个被消费的偏移量可以通过 seek(TopicPartition,long)或自动设置为最后一个被订阅的分区列表的偏移量获得。
@Override
public ConsumerRecords<K, V> poll(long timeout) {
acquire();
try {
if (timeout < 0)
throw new IllegalArgumentException("Timeout must not be negative"); if (this.subscriptions.hasNoSubscriptionOrUserAssignment())
throw new IllegalStateException("Consumer is not subscribed to any topics or assigned any partitions"); // poll for new data until the timeout expires
long start = time.milliseconds();
long remaining = timeout;
do {
Map<TopicPartition, List<ConsumerRecord<K, V>>> records = pollOnce(remaining);
if (!records.isEmpty()) {
fetcher.sendFetches();// 在返回所获取的记录之前，我们可以发送下一轮的fetches并避免阻塞等待它们的响应，以便在用户处理获取的记录时进行流水线操作。
client.pollNoWakeup();//由于已经更新了所使用的位置，所以我们不允许在返回所获取的记录之前触发wakeups或任何其他错误。 if (this.interceptors == null)
return new ConsumerRecords<>(records);
else// 如果存在消费者拦截器执行拦截
return this.interceptors.onConsume(new ConsumerRecords<>(records));
} long elapsed = time.milliseconds() - start;
remaining = timeout - elapsed;
} while (remaining > 0); return ConsumerRecords.empty();
} finally {
release();
}
}
pollOnce：
private Map<TopicPartition, List<ConsumerRecord<K, V>>> pollOnce(long timeout) {
coordinator.poll(time.milliseconds());
// 遍历所有的TopicPartition，如果有未知偏移量（分区的），那么更新。涉及coordinator刷新已提交分区偏移量+fetcher更新获取位置
if (!subscriptions.hasAllFetchPositions())
updateFetchPositions(this.subscriptions.missingFetchPositions());
// 返回已获取到的记录
Map<TopicPartition, List<ConsumerRecord<K, V>>> records = fetcher.fetchedRecords();
if (!records.isEmpty())
return records;
// 发送fetch请求
fetcher.sendFetches();
long now = time.milliseconds();
long pollTimeout = Math.min(coordinator.timeToNextPoll(now), timeout);
// 执行IO，拉取数据
client.poll(pollTimeout, now, new PollCondition() {
@Override
public boolean shouldBlock() {
// since a fetch might be completed by the background thread, we need this poll condition
// to ensure that we do not block unnecessarily in poll()
return !fetcher.hasCompletedFetches();
}
});
if (coordinator.needRejoin())
return Collections.emptyMap();
return fetcher.fetchedRecords();
}
好吧，再往下涉及到通信IO层了，这里不再多说。将来补全了kafka通信协议相关文章后再加上飞机票。
2）invokeListener和分支1一样，最终调用的是用户自定义的MessageListener接口的onMessage方法，不再重复。
3) sendToListener,这里塞进缓存队列LinkedBlockingQueue<ConsumerRecords<K, V>> recordsToProcess,塞进队列后，何时再消费？ListenerInvoker的run方法执行了recordsToProcess.poll进行了消费，
kafka原理和实践（四）spring-kafka消费者源码的更多相关文章
- kafka原理和实践（一）原理：10分钟入门
系列目录 kafka原理和实践(一)原理:10分钟入门 kafka原理和实践(二)spring-kafka简单实践 kafka原理和实践(三)spring-kafka生产者源码 kafka原理和实践( ...
- kafka原理和实践（二）spring-kafka简单实践
系列目录 kafka原理和实践(一)原理:10分钟入门 kafka原理和实践(二)spring-kafka简单实践 kafka原理和实践(三)spring-kafka生产者源码 kafka原理和实践( ...
- kafka原理和实践（六）总结升华
系列目录 kafka原理和实践(一)原理:10分钟入门 kafka原理和实践(二)spring-kafka简单实践 kafka原理和实践(三)spring-kafka生产者源码 kafka原理和实践( ...
- kafka原理和实践（三）spring-kafka生产者源码
系列目录 kafka原理和实践(一)原理:10分钟入门 kafka原理和实践(二)spring-kafka简单实践 kafka原理和实践(三)spring-kafka生产者源码 kafka原理和实践( ...
- kafka原理和实践（五）spring-kafka配置详解
系列目录 kafka原理和实践(一)原理:10分钟入门 kafka原理和实践(二)spring-kafka简单实践 kafka原理和实践(三)spring-kafka生产者源码 kafka原理和实践( ...
- Spring IOC 容器源码分析系列文章导读
1. 简介 Spring 是一个轻量级的企业级应用开发框架,于 2004 年由 Rod Johnson 发布了 1.0 版本.经过十几年的迭代,现在的 Spring 框架已经非常成熟了.Spring ...
- Kafka设计解析（四）Kafka Consumer设计解析
转载自 技术世界,原文链接 Kafka设计解析(四)- Kafka Consumer设计解析 目录 一.High Level Consumer 1. Consumer Group 2. High Le ...
- WebSocket原理与实践(四)--生成数据帧
WebSocket原理与实践(四)--生成数据帧 从服务器发往客户端的数据也是同样的数据帧,但是从服务器发送到客户端的数据帧不需要掩码的.我们自己需要去生成数据帧,解析数据帧的时候我们需要分片. 消息 ...
- Spring第四天，BeanPostProcessor源码分析，彻底搞懂IOC注入及注解优先级问题！
随机推荐
- [Struts2] <s:property>标签
调用类中的属性,通过get方法调用,而非其属性名调用. 比如A类中有属性a1,a1的get方法是getA1Method(); 相应Action类中的get方法是getA1Action(). 那么需要通 ...
- [数据结构] 大纲 - Stan Zhang 数据结构速通教程
* 注: 本文/本系列谢绝转载,如有转载,本人有权利追究相应责任. 2019年4月8日 P1.1 链表 Link:https://www.cnblogs.com/yosql473/p/10727471 ...
- LeetCode--019--删除链表的倒数第N个节点(java)
给定一个链表,删除链表的倒数第 n 个节点,并且返回链表的头结点. 示例: 给定一个链表: 1->2->3->4->5, 和 n = 2. 当删除了倒数第二个节点后,链表变为 ...
- 基于TCAM 的高速路由查找
摘要 随着路由器接口速率的提高,传统的软件路由查找机制已经不能满足要求.目前常见的硬件解决方案是采用TCAM实现关键词 TCAM,路由查找,最长前缀匹配. 1.引言 路由器转发IP 分组时,转发引擎需 ...
- 漏洞复现——Apache SSI远程命令执行
漏洞原理:当目标服务器开启了SSI与CGI支持,我们就可以上传shtml文件,利用<!--#exec cmd="id" -->语法执行命令. SSI:SSI(服务器端包 ...
- React文档（九）list和key
首先,我们回顾一下在js里如何转换数组. 给出下面的代码,我们使用map()函数来获取一个数组的numbers然后将值变成两倍大.我们分配新数组由map()返回: const numbers = [1 ...
- 第8天【文件系统挂载、ext文件系统及read命令、Linux RAID、lvm应用】
文件系统挂载与管理工具(01)_recv 文件系统管理: 将额外文件系统与根文件系统某现存的目录建立关联关系,进而使得此目录作为其他文件访问入口的行成为挂载: 解除此关联关系的过程 吧设备关联挂载点: ...
- Hexo-使用
hexo 写新文章 创建新的文章 ``` bash $ hexo new "Hexo-使用" ``` 生成md文件 ``` bash $ hexo generate ``` 写文章 ...
- mycat 入门使用例子
目的:有 user 和 t_order 两张数据表,表 user 的数据全部存放在 db1_zhang 中,表 t_order 的数据按 id 对 2 取模分别存放在 db1_zhang 和 db2_ ...
- python 二分法模板——牢记
class Solution: # @param nums: The integer array # @param target: Target number to find # @return th ...