Zookeeper+Kafka集群部署

Zookeeper+Kafka集群部署

主机规划：

10.200.3.85  　　Kafka+ZooKeeper

10.200.3.86  　　Kafka+ZooKeeper

10.200.3.87 　　 Kafka+ZooKeeper

软件下载地址：

#wget http://mirrors.hust.edu.cn/apache/zookeeper/zookeeper-3.4.10/zookeeper-3.4.10.tar.gz
#wget http://mirror.bit.edu.cn/apache/kafka/1.1.0/kafka_2.12-1.1.0.tgz

三台主机hosts文件一致：

# cat /etc/hosts
10.200.3.85 ZooKeeper-Kafka-
10.200.3.86 ZooKeeper-Kafka-
10.200.3.87 ZooKeeper-Kafka-

一、安装zookeeper

1.在master节点上操作：

[root@ZooKeeper-Kafka- src]# tar -zxvf zookeeper-3.4..tar.gz -C /usr/local/
[root@ZooKeeper-Kafka- src]# cd ..
[root@ZooKeeper-Kafka- local]# ln -s zookeeper-3.4. zookeeper
[root@ZooKeeper-Kafka- local]# cd zookeeper/conf/
[root@ZooKeeper-Kafka- conf]# cp zoo_sample.cfg zoo.cfg
tickTime=
initLimit=
syncLimit=
dataDir=/tmp/zookeeper
clientPort=
server.=ZooKeeper-Kafka-::
server.=ZooKeeper-Kafka-::
server.=ZooKeeper-Kafka-::

2.创建dataDir目录创建/tmp/zookeeper

# 在master节点上

[root@ZooKeeper-Kafka- conf]# mkdir /tmp/zookeeper
[root@ZooKeeper-Kafka- conf]# touch /tmp/zookeeper/myid
[root@ZooKeeper-Kafka- conf]# echo  > /tmp/zookeeper/myid

3.将zookeeper文件复制到另外两个节点：

[root@ZooKeeper-Kafka- local]# scp -r zookeeper-3.4./ 10.200.3.86:/usr/local/
[root@ZooKeeper-Kafka- local]# scp -r zookeeper-3.4./ 10.200.3.87:/usr/local/

4.在两个slave节点创建目录和文件

#ZooKeeper-Kafka-02节点:

[root@ZooKeeper-Kafka- local]# ln -s zookeeper-3.4. zookeeper
[root@ZooKeeper-Kafka- local]# mkdir /tmp/zookeeper
[root@ZooKeeper-Kafka- local]# touch /tmp/zookeeper/myid
[root@ZooKeeper-Kafka- local]# echo  > /tmp/zookeeper/myid

#ZooKeeper-Kafka-03节点

[root@ZooKeeper-Kafka- local]# ln -s zookeeper-3.4. zookeeper
[root@ZooKeeper-Kafka- local]# mkdir /tmp/zookeeper
[root@ZooKeeper-Kafka- local]# touch /tmp/zookeeper/myid
[root@ZooKeeper-Kafka- local]# echo  > /tmp/zookeeper/myid

5.分别在每个节点上启动 zookeeper测试：

[root@ZooKeeper-Kafka- zookeeper]# ./bin/zkServer.sh start
[root@ZooKeeper-Kafka- zookeeper]# ./bin/zkServer.sh start
[root@ZooKeeper-Kafka- zookeeper]# ./bin/zkServer.sh start

6.查看状态：

[root@ZooKeeper-Kafka- zookeeper]# ./bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: follower
[root@ZooKeeper-Kafka- zookeeper]# ./bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: leader
[root@ZooKeeper-Kafka- zookeeper]# ./bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper/bin/../conf/zoo.cfg
Mode: follower

至此zookeeper集群安装成功！！！

Kafka集群安装配置

1.安装：

[root@ZooKeeper-Kafka- src]# tar -zxvf kafka_2.-1.1..tgz -C /usr/local/
[root@ZooKeeper-Kafka- src]# cd ..
[root@ZooKeeper-Kafka- local]# ln -s kafka_2.-1.1. kafka

2.修改server.properties文件

# master为0
broker.id=
listeners=PLAINTEXT://ZooKeeper-Kafka-01:9092
advertised.listeners=PLAINTEXT://ZooKeeper-Kafka-01:9092
num.network.threads=
num.io.threads=
socket.send.buffer.bytes=
socket.receive.buffer.bytes=
socket.request.max.bytes=
log.dirs=/tmp/kafka-logs
num.partitions=
num.recovery.threads.per.data.dir=
offsets.topic.replication.factor=
transaction.state.log.replication.factor=
transaction.state.log.min.isr=
log.retention.hours=
log.segment.bytes=
log.retention.check.interval.ms=
# 连接
zookeeper.connect=ZooKeeper-Kafka-:,ZooKeeper-Kafka-:,ZooKeeper-Kafka-:
zookeeper.connection.timeout.ms=
group.initial.rebalance.delay.ms=
# 可删除topic
delete.topic.enable=true

3.将 kafka_2.12-1.1.0 文件夹复制到另外两个节点下

[root@ZooKeeper-Kafka- local]# scp -r kafka_2.-1.1./ 10.200.3.86:/usr/local/
[root@ZooKeeper-Kafka- local]# scp -r kafka_2.-1.1./ 10.200.3.87:/usr/local/

并修改每个节点对应的 server.properties 文件的 broker.id和listenrs：

[root@ZooKeeper-Kafka- config]# cat server.properties
broker.id=
listeners=PLAINTEXT://ZooKeeper-Kafka-02:9092
advertised.listeners=PLAINTEXT://ZooKeeper-Kafka-02:9092

[root@ZooKeeper-Kafka- config]# cat server.properties
broker.id=
listeners=PLAINTEXT://ZooKeeper-Kafka-03:9092
advertised.listeners=PLAINTEXT://ZooKeeper-Kafka-03:9092

4.启动服务

bin/kafka-server-start.sh config/server.properties &

Zookeeper+Kafka集群测试

创建topic：

[root@ZooKeeper-Kafka- kafka]# bin/kafka-topics.sh --create --zookeeper ZooKeeper-Kafka-:, ZooKeeper-Kafka-:, ZooKeeper-Kafka-: --replication-factor  --partitions  --topic test

显示topic：

[root@ZooKeeper-Kafka- kafka]# bin/kafka-topics.sh --describe --zookeeper ZooKeeper-Kafka-:, ZooKeeper-Kafka-:, ZooKeeper-Kafka-: --topic test

列出topic：

[root@ZooKeeper-Kafka- kafka]# bin/kafka-topics.sh --list --zookeeper ZooKeeper-Kafka-:, ZooKeeper-Kafka-:, ZooKeeper-Kafka-:
test

创建 producer(生产者);

# 在master节点上 测试生产消息

[root@ZooKeeper-Kafka- kafka]# bin/kafka-console-producer.sh --broker-list ZooKeeper-Kafka-: -topic test
>hello world
>[-- ::,] INFO Updated PartitionLeaderEpoch. New: {epoch:, offset:}, Current: {epoch:-, offset:-} for Partition: test-. Cache now contains  entries. (kafka.server.epoch.LeaderEpochFileCache)
this is example ...
>[-- ::,] INFO Updated PartitionLeaderEpoch. New: {epoch:, offset:}, Current: {epoch:-, offset:-} for Partition: test-. Cache now contains  entries. (kafka.server.epoch.LeaderEpochFileCache)
welcome to china
>[-- ::,] INFO Updated PartitionLeaderEpoch. New: {epoch:, offset:}, Current: {epoch:-, offset:-} for Partition: test-. Cache now contains  entries. (kafka.server.epoch.LeaderEpochFileCache)

创建 consumer(消费者):

# 在ZooKeeper-Kafka-02节点上 测试消费

[root@ZooKeeper-Kafka- kafka]# bin/kafka-console-consumer.sh --zookeeper ZooKeeper-Kafka-:, ZooKeeper-Kafka-:, ZooKeeper-Kafka-: -topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
this is example ...
hello world
[-- ::,] INFO Updated PartitionLeaderEpoch. New: {epoch:, offset:}, Current: {epoch:-, offset:-} for Partition: test-. Cache now contains  entries. (kafka.server.epoch.LeaderEpochFileCache)
welcome to china

#在ZooKeeper-Kafka-03节点上 测试消费

[root@ZooKeeper-Kafka- kafka]# bin/kafka-console-consumer.sh --zookeeper ZooKeeper-Kafka-:, ZooKeeper-Kafka-:, ZooKeeper-Kafka-: -topic test --from-beginning
Using the ConsoleConsumer with old consumer is deprecated and will be removed in a future major release. Consider using the new consumer by passing [bootstrap-server] instead of [zookeeper].
welcome to china
hello world
this is example ...

然后在 producer 里输入消息，consumer 中就会显示出同样的内容，表示消费成功

删除 topic 和关闭服务

[root@ZooKeeper-Kafka- kafka]# bin/kafka-topics.sh --delete --zookeeper ZooKeeper-Kafka-:, ZooKeeper-Kafka-:, ZooKeeper-Kafka-: --topic test

启动服务:

bin/kafka-server-start.sh config/server.properties &

停止服务：

bin/kafka-server-stop.sh

至此Zookeeper+Kafka集群配置成功.

参考文档：https://blog.csdn.net/jiecxy/article/details/52348120