kafka 小案例【二】 --kafka 设置多个消费着集群

这个配是我在http://www.cnblogs.com/zhangXingSheng/p/6646972.html 的基础上再添加的配置

设置多个消息集群

（1）复制两份配置文件

> cp config/server.properties config/server-.properties
> cp config/server.properties config/server-.properties

（2）编辑配置文件

server1.properties

broker.id=
listeners=PLAINTEXT://:9093
log.dir=/home/zhangxs/datainfo/developmentData/kafka/log/log1

server2.propreties

broker.id=
listeners=PLAINTEXT://:9094
log.dir=/home/zhangxs/datainfo/developmentData/kafka/log/log2

(3)启动这个两个消息服务

 bin/kafka-server-start.sh config/server-.properties
 bin/kafka-server-start.sh config/server-.properties

（4）创建一个新的topic

> bin/kafka-topics.sh --create --zookeeper  192.168.177.120: --replication-factor  --partitions  --topic test2

(5)启动2个消费者进程，来订阅这个topic test2

> bin/kafka-console-consumer.sh --bootstrap-server 192.168.177.120: --from-beginning --topic
 test2

(6)输入消息

linux

(7)显示结果

遇到的问题：

（2）在我启动生产者集群的时候，抛出的异常

kafka.common.InconsistentBrokerIdException: Configured broker.id  doesn't match
 stored broker.id  in meta.properties. If you moved your data, make sure your c
onfigured broker.id matches. If you intend to create a new broker, you should re
move all data in your data directories (log.dirs).
    at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:
)
    at kafka.Kafka$.main(Kafka.scala:)
    at kafka.Kafka.main(Kafka.scala)
[-- ::,] INFO shutting down (kafka.server.KafkaServer)
[-- ::,] INFO Terminate ZkClient event thread. (org.I0Itec.zkcl
ient.ZkEventThread)
[-- ::,] INFO Session: 0x15b1f53fc120002 closed (org.apache.zoo
keeper.ZooKeeper)
[-- ::,] INFO EventThread shut down for session: 0x15b1f53fc120
 (org.apache.zookeeper.ClientCnxn)
[-- ::,] INFO shut down completed (kafka.server.KafkaServer)
[-- ::,] FATAL Fatal error during KafkaServerStartable startup.
 Prepare to shutdown (kafka.server.KafkaServerStartable)
kafka.common.InconsistentBrokerIdException: Configured broker.id  doesn't match stored broker.id 0 in meta.properties. If you moved your data, make s
ure your configured broker.id matches. If you intend to create a new broker, you should remove all data in your data directories (log.dirs).
    at kafka.server.KafkaServer.getBrokerId(KafkaServer.scala:)
    at kafka.server.KafkaServer.startup(KafkaServer.scala:)
    at kafka.server.KafkaServerStartable.startup(KafkaServerStartable.scala:)
    at kafka.Kafka$.main(Kafka.scala:)
    at kafka.Kafka.main(Kafka.scala)
因为server1.properties的broker.id=，与server.properties配置的broker,id重复

这个是因为server1.properties配置文件的上面已经配置broker.id=0,我没有把这段配置注释掉，导致我的配置没有生效，与server.properties配置的broker,id重复。

（2）在我启动另一个生产者服务抛出异常

kafka.common.KafkaException: Found directory /home/zhangxs/datainfo/developmentData/kafka/log/log1, 'log1' is not in the form of topic-partition
If a directory does not contain Kafka topic data it should not exist in Kafka's log directory
    at kafka.log.Log$.exception$(Log.scala:)
    at kafka.log.Log$.parseTopicPartitionName(Log.scala:)
    at kafka.log.LogManager$$anonfun$loadLogs$$$anonfun$$$anonfun$apply$$$anonfun$apply$.apply$mcV$sp(LogManager.scala:)
    at kafka.utils.CoreUtils$$anon$.run(CoreUtils.scala:)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:)
    at java.util.concurrent.FutureTask.run(FutureTask.java:)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:)
    at java.lang.Thread.run(Thread.java:)
[-- ::,] INFO [Kafka Server ], shutting down (kafka.server.KafkaServer)
[-- ::,] INFO Terminate ZkClient event thread. (org.I0Itec.zkclient.ZkEventThread)
[-- ::,] INFO Session: 0x15b1f53fc120005 closed (org.apache.zookeeper.ZooKeeper)
[-- ::,] INFO EventThread shut down for session: 0x15b1f53fc120005 (org.apache.zookeeper.ClientCnxn)
[-- ::,] INFO [Kafka Server ], shut down completed (kafka.server.KafkaServer)
[-- ::,] FATAL Fatal error during KafkaServerStartable startup. Prepare to shutdown (kafka.server.KafkaServerStartable)
kafka.common.KafkaException: Found directory /home/zhangxs/datainfo/developmentData/kafka/log/log1, 'log1' is not in the form of topic-partition
If a directory does not contain Kafka topic data it should not exist in Kafka's log directory
    at kafka.log.Log$.exception$(Log.scala:)

这个是我配置日志目录有问题，我原目录配置是[/home/zhangxs/datainfo/developmentData/kafka/log],他自动找到这个目录的下一级目录log1.   log1是我其他生产者消息的日志目录，所以才抛出这个错误