问题描述

Azure Event Hubs -- Kafka 生产者发送消息存在延迟接收和丢失问题, 在客户端的日志中发现如下异常:

2023-06-05 02:00:20.467 [kafka-producer-thread | producer-1] ERROR com.deloitte.common.kafka.CommonKafkaProducer - messageId:9235f334-e39f-b429-227e-45cd30dd6486, topic:notify_topic
发送消息失败 org.springframework.kafka.core.KafkaProducerException: Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: The request timed out.
at org.springframework.kafka.core.KafkaTemplate.lambda$buildCallback$6(KafkaTemplate.java:690)
at org.apache.skywalking.apm.plugin.kafka.CallbackAdapter.onCompletion(CallbackAdapter.java:45)
at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer$1.onCompletion$original$dElInXX8(DefaultKafkaProducerFactory.java:1001)
at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer$1.onCompletion$original$dElInXX8$accessor$6jLL1TNr(DefaultKafkaProducerFactory.java)
at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer$1$auxiliary$ldSQQGBZ.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
at org.springframework.kafka.core.DefaultKafkaProducerFactory$CloseSafeProducer$1.onCompletion(DefaultKafkaProducerFactory.java)
at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion$original$PwZecSoL(KafkaProducer.java:1350)
at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion$original$PwZecSoL$accessor$5Ux1udg0(KafkaProducer.java)
at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback$auxiliary$a5oVYNi3.call(Unknown Source)
at org.apache.skywalking.apm.agent.core.plugin.interceptor.enhance.InstMethodsInter.intercept(InstMethodsInter.java:86)
at org.apache.kafka.clients.producer.KafkaProducer$InterceptorCallback.onCompletion(KafkaProducer.java)
at org.apache.kafka.clients.producer.internals.ProducerBatch.completeFutureAndFireCallbacks(ProducerBatch.java:273)
at org.apache.kafka.clients.producer.internals.ProducerBatch.done(ProducerBatch.java:234)
at org.apache.kafka.clients.producer.internals.ProducerBatch.completeExceptionally(ProducerBatch.java:198)
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:758)
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:743)
at org.apache.kafka.clients.producer.internals.Sender.failBatch(Sender.java:695)
at org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:634)
at org.apache.kafka.clients.producer.internals.Sender.lambda$null$1(Sender.java:575)
at java.base/java.util.ArrayList.forEach(ArrayList.java:1541)
at org.apache.kafka.clients.producer.internals.Sender.lambda$handleProduceResponse$2(Sender.java:562)
at java.base/java.lang.Iterable.forEach(Iterable.java:75)
at org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:562)
at org.apache.kafka.clients.producer.internals.Sender.lambda$sendProduceRequest$5(Sender.java:836)
at org.apache.kafka.clients.ClientResponse.onComplete(ClientResponse.java:109)
at org.apache.kafka.clients.NetworkClient.completeResponses(NetworkClient.java:574)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:566)
at org.apache.kafka.clients.producer.internals.Sender.runOnce(Sender.java:328)
at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:243)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.kafka.common.errors.TimeoutException: The request timed out.

当发送端收到 Failed to send; nested exception is org.apache.kafka.common.errors.TimeoutException: The request timed out. 日志后, 如何来分析这个问题呢?

问题回答

机器人回答一:

This error message indicates that the Kafka producer failed to send a message due to a timeout.

One possible reason for this error is that the transaction timeout is larger than the maximum value allowed by the broker.

You can check the broker configuration for `transaction.max.timeout.ms` and make sure that it is set to a value that is larger than the transaction timeout used by the producer.

You can also check the Kafka logs for more information about the error.

If you are using Flink, you can refer to the Hilo documentation on [Flink to HDInsightKafkaTransactionTimeoutIssue](https://supportability.visualstudio.com/870b0c89-5235-4a33-a0c6-44be26e284e7/_wiki/wikis/e9f84cb2-cd1b-4143-ac21-5bc63f3bdba0?pagePath=%2FHilo%2F[Hilo] TSGs%2F[Hilo] Flink TSGs%2F[Hilo] Flink to HDInsightKafkaTransactionTimeoutIssue) for more information on this issue. One or more links in the message.

机器人回答二:

The error message you provided indicates that the request timed out. This could be due to a network issue or a configuration issue. You can check the following: 1. Check the network connection between the producer and the Event Hub service. Make sure that the network is stable and that there are no issues with the firewall or network security groups. 2. Check the Kafka producer configuration. Make sure that the `request.timeout.ms` property is set to a value that is appropriate for your use case. You can also try increasing the value of this property to see if it resolves the issue. 3. Check the size of the message being sent. If the message size is greater than the maximum allowed size, the request may time out. You can check the maximum message size allowed by the Event Hub service and make sure that your message size is within the limit. 4. Check the Kafka logs for more information about the error. The logs may provide more details about the cause of the error. If you are still unable to resolve the issue, you can contact Azure support for further assistance. Message is interactive. Click to interact.

If you are not using Flink and are using the Kafka producer method to send messages to Event Hub, you can check the Kafka logs for more information about the error.

The error message you provided indicates that the request timed out. This could be due to a network issue or a configuration issue. You can check the following:

1. Check the network connection between the producer and the Event Hub service. Make sure that the network is stable and that there are no issues with the firewall or network security groups.

2. Check the Kafka producer configuration. Make sure that the `request.timeout.ms` property is set to a value that is appropriate for your use case. You can also try increasing the value of this property to see if it resolves the issue.

3. Check the size of the message being sent. If the message size is greater than the maximum allowed size, the request may time out. You can check the maximum message size allowed by the Event Hub service and make sure that your message size is within the limit.

4. Check the Kafka logs for more information about the error. The logs may provide more details about the cause of the error.

If you are still unable to resolve the issue, you can contact Azure support for further assistance.

【Azure 事件中心】Kafka 生产者发送消息失败,根据失败消息询问机器人得到的分析步骤的更多相关文章

  1. 【Azure 事件中心】在微软云中国区 (Mooncake) 上实验以Apache Kafka协议方式发送/接受Event Hubs消息 (Java版)

    问题描述 事件中心提供 Kafka 终结点,现有的基于 Kafka 的应用程序可将该终结点用作运行你自己的 Kafka 群集的替代方案. 事件中心可与许多现有 Kafka 应用程序配合使用.在Azur ...

  2. 【Azure 事件中心】为应用程序网关(Application Gateway with WAF) 配置诊断日志,发送到事件中心

    问题描述 在Application Gateway中,开启WAF(Web application firewall)后,现在需要把访问的日志输出到第三方分析代码中进行分析,如何来获取WAF的诊断日志呢 ...

  3. 【Azure 事件中心】Event Hub 无法连接,出现 Did not observe any item or terminal signal within 60000ms in 'flatMapMany' 的错误消息

    问题描述 使用Java SDK连接Azure Event Hub,一直出现 java.util.concurrent.TimeoutException 异常, 消息为:java.util.concur ...

  4. 【Azure 事件中心】EPH (EventProcessorHost) 消费端观察到多次Shutdown,LeaseLost的error信息,这是什么情况呢?

    问题详情 使用EPH获取Event Hub数据时,多次出现连接shutdown和LeaseLost的error  ,截取某一次的error log如: Time:2021-03-10 08:43:48 ...

  5. 【Azure 事件中心】Azure Event Hub 新功能尝试 -- 异地灾难恢复 (Geo-Disaster Recovery)

    问题描述 关于Event Hub(事件中心)的灾备方案,大多数就是新建另外一个备用的Event Hub,当主Event Hub出现不可用的情况时,就需要切换到备Event Hub上. 而在切换的过程中 ...

  6. 【Azure 事件中心】使用Azure AD认证方式创建Event Hub Consume Client + 自定义Event Position

    问题描述 当使用SDK连接到Azure Event Hub时,最常规的方式为使用连接字符串.这种做法参考官网文档就可成功完成代码:https://docs.azure.cn/zh-cn/event-h ...

  7. 【Azure 事件中心】 org.slf4j.Logger 收集 Event Hub SDK(Java) 输出日志并以文件形式保存

    问题描述 在使用Azure Event Hub的SDK时候,常规情况下,发现示例代码中并没有SDK内部的日志输出.因为在Java项目中,没有添加 SLF4J 依赖,已致于在启动时候有如下提示: SLF ...

  8. Kafka生产者发送消息的三种方式

    Kafka是一种分布式的基于发布/订阅的消息系统,它的高吞吐量.灵活的offset是其它消息系统所没有的. Kafka发送消息主要有三种方式: 1.发送并忘记 2.同步发送 3.异步发送+回调函数 下 ...

  9. kafka 生产者发送消息

    KafkaProducer 创建一个 KafkaThread 来运行 Sender.run 方法. 1. 发送消息的入口在 KafkaProducer#doSend 中,但其实是把消息加入到 batc ...

  10. 【Azure 事件中心】azure-spring-cloud-stream-binder-eventhubs客户端组件问题, 实践消息非顺序可达

    问题描述 查阅了Azure的官方文档( 将事件发送到特定分区: https://docs.azure.cn/zh-cn/event-hubs/event-hubs-availability-and-c ...

随机推荐

  1. lldb3.9.0 安装攻略

    Study From https://github.com/dotnet/diagnostics/blob/master/documentation/lldb/centos7/build-instal ...

  2. React Hooks 指北

    前言 这篇文章旨在总结 React Hooks 的使用技巧以及在使用过程中需要注意的问题,其中会附加一些问题产生的原因以及解决方式.但是请注意,文章中所给出的解决方式并不一定完全适用,解决问题的方案有 ...

  3. 【介绍一个工具】图形化界面查看一个 golang 二进制文件的汇编代码

    作者:张富春(ahfuzhang),转载时请注明作者和引用链接,谢谢! cnblogs博客 zhihu Github 公众号:一本正经的瞎扯 通常,可以通过命令行工具,来查看一个代码文件对应的汇编代码 ...

  4. git checkout switch restore

    前言 在 Git 术语中,"checkout"是在目标实体的不同版本之间切换的行为.该命令对三个不同的实体进行操作:文件.提交和分支.除了"checkout"的 ...

  5. pycharm alt+f7(查找)显示动态用法的结果过多(dynamic usages)

    在脚本语言中查找引用时,如果有同名函数,在动态用法那一栏会出现大量的结果,,如何缩小或者动态用法(dynamic usages)的结果呢? 在官网上也有提出了这个问题,但官方没有给出答案issue:P ...

  6. 【6】opencv采用映射技术实现鱼眼镜头校正和鱼眼镜头还原全景图。

    相关文章: [1]windows下安装OpenCV(4.3)+VS2017安装+opencv_contrib4.3.0配置 [2]Visual Studio 2017同时配置OpenCV2.4 以及O ...

  7. 记录一次py2编码带来的坑

    "中文" == u"中文" # PY2 False"中文" == u"中文" # PY3 True

  8. 从零开始教你手动搭建幻兽帕鲁私服( CentOS 版)

    哈喽大家好,我是咸鱼. 想必上网冲浪的小伙伴最近都被<幻兽帕鲁>这款游戏刷屏了. (文中图片均来自网络,侵删) 幻兽帕鲁是 Pocketpair 打造的一款开放世界的生存建造游戏.在游戏中 ...

  9. 穿透 SESSION 0 隔离

    注:这套程序在这里只是作为了解,博主本人并未测试,所有测试均来自文章最底层链接的原文章. 查看目标进程运行的SESSION位置 在实际开发过程中,可以通过Process Explorer 检查服务或程 ...

  10. delphi中的退出程序的确认问题

    在formclose中用if Application.MessageBox('你确认要退出吗?','请确认',MB_YesNo+MB_IconQuestion)=IDno then begin ... ...