Kafka管理与监控——broker宕机后无法消费问题

背景

因磁盘满了，导致kafka所有的服务器全部宕机了，然后重启kafka集群，服务是启动成功了，但有一些报错：

broker1：

broker2：

broker3：一直在刷以下错误信息

虽然报了这些错，但kafka正常启动了，通过命令测试了集群能正常生产和消费消息，但是看kafka-manager界面，出现副本未分配的异常情况：

检查消费这些主题的程序，果然是消费失败了，一直在刷如下异常信息：

注：图中IP的是broker3节点

截止到这里可以看出，broker3节点出问题了，导致消费者程序连接不上，但奇怪的话，通过命令创建主题测试，在broker3节点又能消费。

继续分析broker3的日志，报错原因：集群要求的副本数是2，但只找到1个。

于是查看相关主题的详细信息，发现确实ISR列表中是少了副本

猜测由于宕机后，有些节点落后leader太多，还没有追上来，所以脱离了ISR列表，于是等它自动追上来。

等到第2天一看，还是一样，没有追上来，于是决定重启kafka集群，发现有些分区的会自动扩展成2，出问题的那些分区还是没有。。。。

然后想通过重新分配分区指定副本，看能否让它自动恢复一下副本，通过以下命令进行处理：

bin/kafka-reassign-partitions.sh --zookeeper 10.0.xx.x:,10.0.xx.x:,10.0.xx.x: --reassignment-json-file reassign.json --execute

reassign.json文件内容：

{"version":, "partitions":[
 {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]},
  {"topic":"__consumer_offsets","partition":,"replicas":[,]}
]}`

重新分区指定副本的方法也不行，于是修改kafka配置，把集群要求的副本数改为1：

vi server.properties

重启kafka集群后，broker3不在就报错了，在重启消费都程序，也能正常连上kafka进行消费了。

总结：

kafka出现宕机后，副本脱离ISR列表（落后leader太多），按正常来说它会慢慢追上来后在自动重新加入ISR列表中，但我的等了20个小时后还没有，重启kafka集群后也没有恢复。导致服务启动有问题。

现在临时解决方案是调整成1，让它先跑一段时间后，看能否恢复回来，到时在设置成2。

问题：

1、原因尚未找到；

2、这样调整后，kafka会出现数据丢失的情况（出问题期间的数据都丢失了）。