Problem description:

There is error messge when run spark app using spark streaming Kafka version 0.10.0.1(https://spark.apache.org/docs/latest/streaming-kafka-0-10-integration.html) to send message to version 0.9 or 0.8 of Kafka broker.

2019-04-08 01:08:04,211 [ERROR] [kafka-producer-network-thread | producer-1]  Uncaught error in kafka producer I/O thread: 
org.apache.kafka.common.protocol.types.SchemaException: Error reading field 'brokers': Error reading field 'host': Error reading string of length 25394, only 543 bytes available
                   at org.apache.kafka.common.protocol.types.Schema.read(Schema.java:73)
                   at org.apache.kafka.clients.NetworkClient.parseResponse(NetworkClient.java:380)
                   at org.apache.kafka.clients.NetworkClient.handleCompletedReceives(NetworkClient.java:449)
                   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:269)
                   at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:229)
                   at org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)
                   at java.lang.Thread.run(Thread.java:745)

Root cause:

The Kafka broker version is still using version less than 0.10. Since there is big change after 0.10 that timestamp added in Schema. https://kafka.apache.org/documentation/

"

Soution:

Kafka broker should be upgrade and compatible to low version Schema format.

"1.5 Upgrading From Previous Versions

Upgrading from 0.8.x or 0.9.x to 0.10.0.0

0.10.0.0 has potential breaking changes (please review before upgrading) and possible performance impact following the upgrade. By following the recommended rolling upgrade plan below, you guarantee no downtime and no performance impact during and following the upgrade. 
Note: Because new protocols are introduced, it is important to upgrade your Kafka clusters before upgrading your clients.

Notes to clients with version 0.9.0.0: Due to a bug introduced in 0.9.0.0, clients that depend on ZooKeeper (old Scala high-level Consumer and MirrorMaker if used with the old consumer) will not work with 0.10.0.x brokers. Therefore, 0.9.0.0 clients should be upgraded to 0.9.0.1 beforebrokers are upgraded to 0.10.0.x. This step is not necessary for 0.8.X or 0.9.0.1 clients.

For a rolling upgrade:

  1. Update server.properties file on all brokers and add the following properties:Upgrade the brokers. This can be done a broker at a time by simply bringing it down, updating the code, and restarting it.

  2. Once the entire cluster is upgraded, bump the protocol version by editing inter.broker.protocol.version and setting it to 0.10.0.0. NOTE: You shouldn't touch log.message.format.version yet - this parameter should only change once all consumers have been upgraded to 0.10.0.0
  3. Restart the brokers one by one for the new protocol version to take effect.
  4. Once all consumers have been upgraded to 0.10.0, change log.message.format.version to 0.10.0 on each broker and restart them one by one.

Note: If you are willing to accept downtime, you can simply take all the brokers down, update the code and start all of them. They will start with the new protocol by default.

Note: Bumping the protocol version and restarting can be done any time after the brokers were upgraded. It does not have to be immediately after."

Kafka topic Schema version mismatch error - org.apache.kafka.common.protocol.types.SchemaException的更多相关文章

  1. kafka删除topic后再创建同名的topic报错(ERROR org.apache.kafka.common.errors.TopicExistsException)

    [hadoop@datanode3 logs]$ kafka-topics.sh --delete --zookeeper datanode1:2181 --topic firstTopic firs ...

  2. Error when sending message to topic test with key: null, value: 2 bytes with error: (org.apache.kafka.clients.producer.internals.ErrorLoggingCallback)

    windows下使用kafka遇到这个问题: Error when sending message to topic test with key: null, value: 2 bytes with ...

  3. libtool: Version mismatch error 解决

    在编译一个软件的时候,在 ./configure 和 make  之后可能会出现如下错误: libtool: Version mismatch error.  This is libtool 2.4. ...

  4. libtool: Version mismatch error. 解决方法

    在编译一个软件的时候,在 ./configure 和 make  之后可能会出现如下错误: libtool: Version mismatch error.  This is libtool 2.4. ...

  5. DataPipeline |《Apache Kafka实战》作者胡夕:Apache Kafka监控与调优

    胡夕 <Apache Kafka实战>作者,北航计算机硕士毕业,现任某互金公司计算平台总监,曾就职于IBM.搜狗.微博等公司.国内活跃的Kafka代码贡献者. 前言 虽然目前Apache ...

  6. 实践部署与使用apache kafka框架技术博文资料汇总

    前一篇Kafka框架设计来自英文原文(Kafka Architecture Design)的翻译及整理文章,非常有借鉴性,本文是从一个企业使用Kafka框架的角度来记录及整理的Kafka框架的技术资料 ...

  7. Apache Kafka® is a distributed streaming platform

    Kafka Connect简介 我们知道过去对于Kafka的定义是分布式,分区化的,带备份机制的日志提交服务.也就是一个分布式的消息队列,这也是他最常见的用法.但是Kafka不止于此,打开最新的官网. ...

  8. Spring for Apache Kafka

    官方文档详见:http://docs.spring.io/spring-kafka/docs/1.0.2.RELEASE/reference/htmlsingle/ Authors Gary Russ ...

  9. kafka的api操作(官网http://kafka.apache.org/documentation/#producerapi)

    Kafka API 简单用法 本篇会用到以下依赖:(本人包和这个不同,去maven里查找) <dependency><groupId>org.apache.kafka</ ...

随机推荐

  1. 【XSY2495】余数

    Input Output Input 3 4 Output 4 HINT 原式 =n*m-n除以i向下取整 用数论分块做就可以了 #include<bits/stdc++.h> #defi ...

  2. [考试反思]1011csp-s模拟测试68:守恒

    在RP守恒定律的持续作用下, 不出所料,这场稍炸 还有10分钟就是下一场了,但愿继续守恒? 改题太慢了,连写博的时间都没有了 然而最后还是在吃饭前彻彻底底改出来了 的确是个菜鸡 所以今天的题解只能先咕 ...

  3. postman发送json参数到后端controller报错415的解决方案

    警告 [http-nio-8080-exec-9] org.springframework.web.servlet.handler.AbstractHandlerExceptionResolver.l ...

  4. 精心整理(含图版)|你要的全拿走!(R数据分析,可视化,生信实战)

    本文首发于“生信补给站”公众号,https://mp.weixin.qq.com/s/ZEjaxDifNATeV8fO4krOIQ更多关于R语言,ggplot2绘图,生信分析的内容,敬请关注小号. 为 ...

  5. python、C++经典算法题:打印100以内的素数

    题目 打印100以内的素数 思路1 素数的特点: 素数一定是奇数 一个数如果是合数,那么它一定能够被2到这个合数的开平方内的某个素数整除(这个特点是提升效率的关键) 一个数如果不能被从2到它自身开平方 ...

  6. supervisord进程管理

    一:简介 supervisord是一个进程管理工具,提供web页面管理,能对进程进行自动重启等操作. 优点: - 可以将非后台运行程序后台运行 - 自动监控,重启进程 缺点: - 不能管理后台运行程序 ...

  7. 130道ASP.NET面试题(一)

    1 .简述 private,protected,public,internal修饰符的访问权限 答: private : 私有成员, 在类的内部才可以访问. protected : 保护成员,该类内部 ...

  8. 008.Kubernetes二进制部署Nginx实现高可用

    一 Nginx代理实现kube-apiserver高可用 1.1 Nginx实现高可用 基于 nginx 代理的 kube-apiserver 高可用方案. 控制节点的 kube-controller ...

  9. 对学习Python自然语言处理的一点儿感悟

    研究生阶段,如果一开始找不到知识点,那就结合导师的研究课题以及这个课题设计到的知识领域,扎好基础知识能力.以我研究的自然语言处理领域为例,如果一开始不知道研究点,且又要安排学习计划,那么我会在选择: ...

  10. 《计算机网络 自顶向下方法》 第3章 运输层 Part2

    待补充完善 TCP 相关基本点 1.面向连接 两个不同主机上的进程在通过 TCP 进行通信之前,必须先通过三次握手来建立 TCP 连接 2.全双工服务 即,如果一台主机上的进程 A 与另一台主机上的进 ...