快速搭建Zookeeper和Kafka环境
前言
由于项目需要涉及到zookeeper和Kafka的使用,快速做了一篇笔记,方便小伙伴们搭建环境。
zookeeper
官方定义
What is ZooKeeper?
ZooKeeper is a centralized service for maintaining configuration information, naming, providing distributed synchronization, and providing group services. All of these kinds of services are used in some form or another by distributed applications. Each time they are implemented there is a lot of work that goes into fixing the bugs and race conditions that are inevitable. Because of the difficulty of implementing these kinds of services, applications initially usually skimp on them, which make them brittle in the presence of change and difficult to manage. Even when done correctly, different implementations of these services lead to management complexity when the applications are deployed.
ZooKeeper是一个分布式的,开放源码的分布式应用程序协调服务,是Google的Chubby一个开源的实现,是Hadoop和Hbase的重要组件。它是一个为分布式应用提供一致性服务的软件,提供的功能包括:配置维护、域名服务、分布式同步、组服务等。
ZooKeeper的目标就是封装好复杂易出错的关键服务,将简单易用的接口和性能高效、功能稳定的系统提供给用户。
zookeeper常用端口号:2181
环境搭建
下载zookeeper:
在zookeeper-3.4.10目录下,新建文件夹,并命名,例如data。(路径为:D:\zookeeper-3.4.10\conf\data)
复制“zoo_sample.cfg”副本à并将副本重命名为“zoo.cfg,在任意文本编辑器(eg:记事本)中打开zoo.cfg,找到并编辑
dataDir=D:\\zookeeper-3.4.10\\data
windows的话,在cmd下运行:
D:\zookeeper-3.4.13\bin>zkServer.cmd
D:\zookeeper-3.4.13\bin>zkServer.cmd
D:\zookeeper-3.4.13\bin>call "C:\Program Files\Java\jdk1.8.0_181"\bin\java "-Dzookeeper.log.dir=D:\zookeeper-3.4.13\bin\.." "-Dzookeeper.root.logger=INFO,CONSOLE" -cp "D:\zookeeper-3.4.13\bin\..\build\classes;D:\zookeeper-3.4.13\bin\..\build\lib\*;D:\zookeeper-3.4.13\bin\..\*;D:\zookeeper-3.4.13\bin\..\lib\*;D:\zookeeper-3.4.13\bin\..\conf" org.apache.zookeeper.server.quorum.QuorumPeerMain "D:\zookeeper-3.4.13\bin\..\conf\zoo.cfg"
2019-03-18 22:07:46,267 [myid:] - INFO [main:QuorumPeerConfig@136] - Reading configuration from: D:\zookeeper-3.4.13\bin\..\conf\zoo.cfg
2019-03-18 22:07:46,276 [myid:] - INFO [main:DatadirCleanupManager@78] - autopurge.snapRetainCount set to 3
2019-03-18 22:07:46,276 [myid:] - INFO [main:DatadirCleanupManager@79] - autopurge.purgeInterval set to 0
2019-03-18 22:07:46,277 [myid:] - INFO [main:DatadirCleanupManager@101] - Purge task is not scheduled.
2019-03-18 22:07:46,280 [myid:] - WARN [main:QuorumPeerMain@116] - Either no config or no quorum defined in config, running in standalone mode
2019-03-18 22:07:46,397 [myid:] - INFO [main:QuorumPeerConfig@136] - Reading configuration from: D:\zookeeper-3.4.13\bin\..\conf\zoo.cfg
2019-03-18 22:07:46,398 [myid:] - INFO [main:ZooKeeperServerMain@98] - Starting server
2019-03-18 22:07:46,422 [myid:] - INFO [main:Environment@100] - Server environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 04:05 GMT
2019-03-18 22:07:46,422 [myid:] - INFO [main:Environment@100] - Server environment:host.name=DESKTOP-D5MA1P9
2019-03-18 22:07:46,422 [myid:] - INFO [main:Environment@100] - Server environment:java.version=1.8.0_181
2019-03-18 22:07:46,423 [myid:] - INFO [main:Environment@100] - Server environment:java.vendor=Oracle Corporation
2019-03-18 22:07:46,423 [myid:] - INFO [main:Environment@100] - Server environment:java.home=C:\Program Files\Java\jdk1.8.0_181\jre
2019-03-18 22:07:46,423 [myid:] - INFO [main:Environment@100] - Server environment:java.class.path=D:\zookeeper-3.4.13\bin\..\build\classes;D:\zookeeper-3.4.13\bin\..\build\lib\*;D:\zookeeper-3.4.13\bin\..\zookeeper-3.4.13.jar;D:\zookeeper-3.4.13\bin\..\lib\audience-annotations-0.5.0.jar;D:\zookeeper-3.4.13\bin\..\lib\jline-0.9.94.jar;D:\zookeeper-3.4.13\bin\..\lib\log4j-1.2.17.jar;D:\zookeeper-3.4.13\bin\..\lib\netty-3.10.6.Final.jar;D:\zookeeper-3.4.13\bin\..\lib\slf4j-api-1.7.25.jar;D:\zookeeper-3.4.13\bin\..\lib\slf4j-log4j12-1.7.25.jar;D:\zookeeper-3.4.13\bin\..\conf
2019-03-18 22:07:46,424 [myid:] - INFO [main:Environment@100] - Server environment:java.library.path=C:\Program Files\Java\jdk1.8.0_181\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\nodejs\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Git\cmd;C:\Program Files\Java\jre1.8.0_191\bin;C:\Program Files\Java\jdk1.8.0_181\bin;C:\Users\yzd\AppData\Local\Microsoft\WindowsApps;C:\Users\yzd\AppData\Roaming\npm;.
2019-03-18 22:07:46,425 [myid:] - INFO [main:Environment@100] - Server environment:java.io.tmpdir=C:\Users\yzd\AppData\Local\Temp\
2019-03-18 22:07:46,425 [myid:] - INFO [main:Environment@100] - Server environment:java.compiler=<NA>
2019-03-18 22:07:46,427 [myid:] - INFO [main:Environment@100] - Server environment:os.name=Windows 10
2019-03-18 22:07:46,427 [myid:] - INFO [main:Environment@100] - Server environment:os.arch=amd64
2019-03-18 22:07:46,427 [myid:] - INFO [main:Environment@100] - Server environment:os.version=10.0
2019-03-18 22:07:46,428 [myid:] - INFO [main:Environment@100] - Server environment:user.name=yzd
2019-03-18 22:07:46,428 [myid:] - INFO [main:Environment@100] - Server environment:user.home=C:\Users\yzd
2019-03-18 22:07:46,428 [myid:] - INFO [main:Environment@100] - Server environment:user.dir=D:\zookeeper-3.4.13\bin
2019-03-18 22:07:46,438 [myid:] - INFO [main:ZooKeeperServer@836] - tickTime set to 2000
2019-03-18 22:07:46,438 [myid:] - INFO [main:ZooKeeperServer@845] - minSessionTimeout set to -1
2019-03-18 22:07:46,439 [myid:] - INFO [main:ZooKeeperServer@854] - maxSessionTimeout set to -1
2019-03-18 22:07:46,647 [myid:] - INFO [main:ServerCnxnFactory@117] - Using org.apache.zookeeper.server.NIOServerCnxnFactory as server connection factory
2019-03-18 22:07:46,651 [myid:] - INFO [main:NIOServerCnxnFactory@89] - binding to port 0.0.0.0/0.0.0.0:2181
搭建成功
zookeeper和eureka比较
ZooKeeper、Eureka对比:
https://www.cnblogs.com/jieqing/p/8394001.html
CAP定理(C-数据一致性;A-服务可用性;P-服务对网络分区故障的容错性,这三个特性在任何分布式系统中不能同时满足,最多同时满足两个)
ZooKeeper基于CP,不保证高可用,如果zookeeper正在选主,或者Zookeeper集群中半数以上机器不可用,那么将无法获得数据。Eureka基于AP,能保证高可用,即使所有机器都挂了,也能拿到本地缓存的数据。作为注册中心,其实配置是不经常变动的,只有发版和机器出故障时会变。对于不经常变动的配置来说,CP是不合适的,而AP在遇到问题时可以用牺牲一致性来保证可用性,既返回旧数据,缓存数据。
所以理论上Eureka是更适合作注册中心。而现实环境中大部分项目可能会使用ZooKeeper,那是因为集群不够大,并且基本不会遇到用做注册中心的机器一半以上都挂了的情况。所以实际上也没什么大问题。
kfaka
Kafka is used for building real-time data pipelines and streaming apps. It is horizontally scalable, fault-tolerant, wicked fast, and runs in production in thousands of companies.
Kafka是由Apache软件基金会开发的一个开源流处理平台,由Scala和Java编写。Kafka是一种高吞吐量的分布式发布订阅消息系统,它可以处理消费者规模的网站中的所有动作流数据。 这种动作(网页浏览,搜索和其他用户的行动)是在现代网络上的许多社会功能的一个关键因素。 这些数据通常是由于吞吐量的要求而通过处理日志和日志聚合来解决。 对于像Hadoop一样的日志数据和离线分析系统,但又要求实时处理的限制,这是一个可行的解决方案。Kafka的目的是通过Hadoop的并行加载机制来统一线上和离线的消息处理,也是为了通过集群来提供实时的消息。
kafka包内自带整合好的zookeeper,所以如果要搭建kafka可以省去单独下载zookeeper的步骤。想起了欲练武功必先自宫的笑话,哈哈哈。
环境搭建
- 下载地址:http://kafka.apache.org/downloads.html
- 解压缩,建立一个空文件夹 logs. eg: D:\kafka_2.11-1.0.0\logs
- 进入config目录,编辑 server.properties文件(eg: 用“写字板”打开)。找到并编辑log.dirs= D:\kafka_2.11-1.0.0\logs
- 请确保在启动Kafka服务器前,Zookeeper实例已经准备好并开始运行。
- 在此处打开Powershell窗口,运行:.\bin\windows\kafka-server-start.bat .\config\server.properties
PS D:\kafka_2.12-2.1.1> .\bin\windows\kafka-server-start.bat .\config\server.properties
[2019-03-18 22:15:34,866] INFO Registered kafka:type=kafka.Log4jController MBean (kafka.utils.Log4jControllerRegistration$)
[2019-03-18 22:15:35,503] INFO starting (kafka.server.KafkaServer)
[2019-03-18 22:15:35,505] INFO Connecting to zookeeper on localhost:2181 (kafka.server.KafkaServer)
[2019-03-18 22:15:35,530] INFO [ZooKeeperClient] Initializing a new session to localhost:2181. (kafka.zookeeper.ZooKeeperClient)
[2019-03-18 22:15:35,536] INFO Client environment:zookeeper.version=3.4.13-2d71af4dbe22557fda74f9a9b4309b15a7487f03, built on 06/29/2018 00:39 GMT (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,536] INFO Client environment:host.name=DESKTOP-D5MA1P9 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,536] INFO Client environment:java.version=1.8.0_181 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,537] INFO Client environment:java.vendor=Oracle Corporation (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,537] INFO Client environment:java.home=C:\Program Files\Java\jdk1.8.0_181\jre (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,537] INFO Client environment:java.class.path=D:\kafka_2.12-2.1.1\libs\activation-1.1.1.jar;D:\kafka_2.12-2.1.1\libs\aopalliance-repackaged-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\argparse4j-0.7.0.jar;D:\kafka_2.12-2.1.1\libs\audience-annotations-0.5.0.jar;D:\kafka_2.12-2.1.1\libs\commons-lang3-3.8.1.jar;D:\kafka_2.12-2.1.1\libs\connect-api-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-basic-auth-extension-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-file-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-json-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-runtime-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\connect-transforms-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\guava-20.0.jar;D:\kafka_2.12-2.1.1\libs\hk2-api-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\hk2-locator-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\hk2-utils-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\jackson-annotations-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-core-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-databind-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-jaxrs-base-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-jaxrs-json-provider-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\jackson-module-jaxb-annotations-2.9.8.jar;D:\kafka_2.12-2.1.1\libs\javassist-3.22.0-CR2.jar;D:\kafka_2.12-2.1.1\libs\javax.annotation-api-1.2.jar;D:\kafka_2.12-2.1.1\libs\javax.inject-1.jar;D:\kafka_2.12-2.1.1\libs\javax.inject-2.5.0-b42.jar;D:\kafka_2.12-2.1.1\libs\javax.servlet-api-3.1.0.jar;D:\kafka_2.12-2.1.1\libs\javax.ws.rs-api-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\javax.ws.rs-api-2.1.jar;D:\kafka_2.12-2.1.1\libs\jaxb-api-2.3.0.jar;D:\kafka_2.12-2.1.1\libs\jersey-client-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-common-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-container-servlet-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-container-servlet-core-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-hk2-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-media-jaxb-2.27.jar;D:\kafka_2.12-2.1.1\libs\jersey-server-2.27.jar;D:\kafka_2.12-2.1.1\libs\jetty-client-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-continuation-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-http-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-io-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-security-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-server-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-servlet-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-servlets-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jetty-util-9.4.12.v20180830.jar;D:\kafka_2.12-2.1.1\libs\jopt-simple-5.0.4.jar;D:\kafka_2.12-2.1.1\libs\kafka-clients-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-log4j-appender-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-streams-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-streams-examples-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-streams-scala_2.12-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-streams-test-utils-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka-tools-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-javadoc.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-javadoc.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-scaladoc.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-scaladoc.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-sources.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-sources.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-test-sources.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-test-sources.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-test.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1-test.jar.asc;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1.jar;D:\kafka_2.12-2.1.1\libs\kafka_2.12-2.1.1.jar.asc;D:\kafka_2.12-2.1.1\libs\log4j-1.2.17.jar;D:\kafka_2.12-2.1.1\libs\lz4-java-1.5.0.jar;D:\kafka_2.12-2.1.1\libs\maven-artifact-3.6.0.jar;D:\kafka_2.12-2.1.1\libs\metrics-core-2.2.0.jar;D:\kafka_2.12-2.1.1\libs\osgi-resource-locator-1.0.1.jar;D:\kafka_2.12-2.1.1\libs\plexus-utils-3.1.0.jar;D:\kafka_2.12-2.1.1\libs\reflections-0.9.11.jar;D:\kafka_2.12-2.1.1\libs\rocksdbjni-5.14.2.jar;D:\kafka_2.12-2.1.1\libs\scala-library-2.12.7.jar;D:\kafka_2.12-2.1.1\libs\scala-logging_2.12-3.9.0.jar;D:\kafka_2.12-2.1.1\libs\scala-reflect-2.12.7.jar;D:\kafka_2.12-2.1.1\libs\slf4j-api-1.7.25.jar;D:\kafka_2.12-2.1.1\libs\slf4j-log4j12-1.7.25.jar;D:\kafka_2.12-2.1.1\libs\snappy-java-1.1.7.2.jar;D:\kafka_2.12-2.1.1\libs\validation-api-1.1.0.Final.jar;D:\kafka_2.12-2.1.1\libs\zkclient-0.11.jar;D:\kafka_2.12-2.1.1\libs\zookeeper-3.4.13.jar;D:\kafka_2.12-2.1.1\libs\zstd-jni-1.3.7-1.jar (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,538] INFO Client environment:java.library.path=C:\Program Files\Java\jdk1.8.0_181\bin;C:\WINDOWS\Sun\Java\bin;C:\WINDOWS\system32;C:\WINDOWS;C:\Program Files (x86)\Common Files\Oracle\Java\javapath;C:\WINDOWS\system32;C:\WINDOWS;C:\WINDOWS\System32\Wbem;C:\WINDOWS\System32\WindowsPowerShell\v1.0\;C:\WINDOWS\System32\OpenSSH\;C:\Program Files\nodejs\;C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR;C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common;C:\Program Files\Git\cmd;C:\Program Files\Java\jre1.8.0_191\bin;C:\Program Files\Java\jdk1.8.0_181\bin;C:\Users\yzd\AppData\Local\Microsoft\WindowsApps;C:\Users\yzd\AppData\Roaming\npm;. (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,539] INFO Client environment:java.io.tmpdir=C:\Users\yzd\AppData\Local\Temp\ (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,542] INFO Client environment:java.compiler=<NA> (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,542] INFO Client environment:os.name=Windows 10 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,543] INFO Client environment:os.arch=amd64 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,543] INFO Client environment:os.version=10.0 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,547] INFO Client environment:user.name=yzd (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,547] INFO Client environment:user.home=C:\Users\yzd (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,547] INFO Client environment:user.dir=D:\kafka_2.12-2.1.1 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,549] INFO Initiating client connection, connectString=localhost:2181 sessionTimeout=6000 watcher=kafka.zookeeper.ZooKeeperClient$ZooKeeperClientWatcher$@3e2e18f2 (org.apache.zookeeper.ZooKeeper)
[2019-03-18 22:15:35,570] INFO [ZooKeeperClient] Waiting until connected. (kafka.zookeeper.ZooKeeperClient)
[2019-03-18 22:15:35,570] INFO Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error) (org.apache.zookeeper.ClientCnxn)
[2019-03-18 22:15:35,574] INFO Socket connection established to localhost/0:0:0:0:0:0:0:1:2181, initiating session (org.apache.zookeeper.ClientCnxn)
[2019-03-18 22:15:35,598] INFO Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x1000ae3e3690000, negotiated timeout = 6000 (org.apache.zookeeper.ClientCnxn)
[2019-03-18 22:15:35,602] INFO [ZooKeeperClient] Connected. (kafka.zookeeper.ZooKeeperClient)
[2019-03-18 22:15:35,974] INFO Cluster ID = brvhJ2K3R4qZnGJ6CQhghA (kafka.server.KafkaServer)
[2019-03-18 22:15:35,980] WARN No meta.properties file under dir D:\kafka_2.12-2.1.1\logs\meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2019-03-18 22:15:36,049] INFO KafkaConfig values:
advertised.host.name = null
...(省略若干)
zookeeper.sync.time.ms = 2000
(kafka.server.KafkaConfig)
[2019-03-18 22:15:36,092] INFO [ThrottledChannelReaper-Fetch]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-03-18 22:15:36,092] INFO [ThrottledChannelReaper-Produce]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-03-18 22:15:36,094] INFO [ThrottledChannelReaper-Request]: Starting (kafka.server.ClientQuotaManager$ThrottledChannelReaper)
[2019-03-18 22:15:36,132] INFO Loading logs. (kafka.log.LogManager)
[2019-03-18 22:15:36,143] INFO Logs loading complete in 11 ms. (kafka.log.LogManager)
[2019-03-18 22:15:36,161] INFO Starting log cleanup with a period of 300000 ms. (kafka.log.LogManager)
[2019-03-18 22:15:36,166] INFO Starting log flusher with a default period of 9223372036854775807 ms. (kafka.log.LogManager)
[2019-03-18 22:15:36,486] INFO Awaiting socket connections on 0.0.0.0:9092. (kafka.network.Acceptor)
[2019-03-18 22:15:36,534] INFO [SocketServer brokerId=0] Started 1 acceptor threads (kafka.network.SocketServer)
[2019-03-18 22:15:36,562] INFO [ExpirationReaper-0-Produce]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,563] INFO [ExpirationReaper-0-DeleteRecords]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,563] INFO [ExpirationReaper-0-Fetch]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,581] INFO [LogDirFailureHandler]: Starting (kafka.server.ReplicaManager$LogDirFailureHandler)
[2019-03-18 22:15:36,607] INFO Creating /brokers/ids/0 (is it secure? false) (kafka.zk.KafkaZkClient)
[2019-03-18 22:15:36,612] INFO Result of znode creation at /brokers/ids/0 is: OK (kafka.zk.KafkaZkClient)
[2019-03-18 22:15:36,613] INFO Registered broker 0 at path /brokers/ids/0 with addresses: ArrayBuffer(EndPoint(DESKTOP-D5MA1P9,9092,ListenerName(PLAINTEXT),PLAINTEXT)) (kafka.zk.KafkaZkClient)
[2019-03-18 22:15:36,615] WARN No meta.properties file under dir D:\kafka_2.12-2.1.1\logs\meta.properties (kafka.server.BrokerMetadataCheckpoint)
[2019-03-18 22:15:36,682] INFO [ExpirationReaper-0-topic]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,684] INFO [ExpirationReaper-0-Heartbeat]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,685] INFO [ExpirationReaper-0-Rebalance]: Starting (kafka.server.DelayedOperationPurgatory$ExpiredOperationReaper)
[2019-03-18 22:15:36,689] INFO Successfully created /controller_epoch with initial epoch 0 (kafka.zk.KafkaZkClient)
[2019-03-18 22:15:36,714] INFO [GroupCoordinator 0]: Starting up. (kafka.coordinator.group.GroupCoordinator)
[2019-03-18 22:15:36,716] INFO [GroupCoordinator 0]: Startup complete. (kafka.coordinator.group.GroupCoordinator)
[2019-03-18 22:15:36,731] INFO [GroupMetadataManager brokerId=0] Removed 0 expired offsets in 15 milliseconds. (kafka.coordinator.group.GroupMetadataManager)
[2019-03-18 22:15:36,748] INFO [ProducerId Manager 0]: Acquired new producerId block (brokerId:0,blockStartProducerId:0,blockEndProducerId:999) by writing to Zk with path version 1 (kafka.coordinator.transaction.ProducerIdManager)
[2019-03-18 22:15:36,780] INFO [TransactionCoordinator id=0] Starting up. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-03-18 22:15:36,782] INFO [TransactionCoordinator id=0] Startup complete. (kafka.coordinator.transaction.TransactionCoordinator)
[2019-03-18 22:15:36,791] INFO [Transaction Marker Channel Manager 0]: Starting (kafka.coordinator.transaction.TransactionMarkerChannelManager)
[2019-03-18 22:15:36,875] INFO [/config/changes-event-process-thread]: Starting (kafka.common.ZkNodeChangeNotificationListener$ChangeEventProcessThread)
[2019-03-18 22:15:36,892] INFO [SocketServer brokerId=0] Started processors for 1 acceptors (kafka.network.SocketServer)
[2019-03-18 22:15:36,901] INFO Kafka version : 2.1.1 (org.apache.kafka.common.utils.AppInfoParser)
[2019-03-18 22:15:36,919] INFO Kafka commitId : 21234bee31165527 (org.apache.kafka.common.utils.AppInfoParser)
[2019-03-18 22:15:36,922] INFO [KafkaServer id=0] started (kafka.server.KafkaServer)
搭建成功。
测试kafka生产者消费者
开启zookeeper和kafka服务
创建topic:
> bin/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test
看一眼刚才创建的主题
> bin/kafka-topics.sh --list --zookeeper localhost:2181
test
- 发消息:接收者
bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
test
- 收消息:消费者
> bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a message
This is another message
参考
https://blog.csdn.net/tianmanchn/article/details/78943147
https://www.jianshu.com/p/ca60bc8c5673
关注我
我是蛮三刀把刀,硕士,后端开发工程师。主要关注后端开发,数据安全,爬虫等方向。微信:yangzd1102
Github个人主页:
原创博客主要内容
- Java知识点复习全手册
- Leetcode算法题解析
- 剑指offer算法题解析
- Python爬虫相关技术实战
- 后端开发相关技术实战
- SpringCloud实战
个人公众号:后端技术漫谈
如果文章对你有帮助,不妨收藏起来并转发给您的朋友们~
快速搭建Zookeeper和Kafka环境的更多相关文章
- Docker快速搭建Zookeeper和kafka集群
使用Docker快速搭建Zookeeper和kafka集群 镜像选择 Zookeeper和Kafka集群分别运行在不同的容器中zookeeper官方镜像,版本3.4kafka采用wurstmeiste ...
- 使用Docker快速搭建Zookeeper和kafka集群
使用Docker快速搭建Zookeeper和kafka集群 镜像选择 Zookeeper和Kafka集群分别运行在不同的容器中zookeeper官方镜像,版本3.4kafka采用wurstmeiste ...
- 快速搭建 Node.js 开发环境以及加速 npm
如何快速搭建 node 开发环境 npm 超慢 github 无法打开的问题 于是我觉得应该写一篇文章解答所有这些起步问题,让新同学也能顺顺利利入门. 快速搭建 Node.js 开发环境 如果你想长期 ...
- windows下9款一键快速搭建PHP本地运行环境的好工具(含php7.0环境)
推荐几款一键快速搭建PHP本地运行环境的好工具(含php7.0及apache,nigix,mysql) 首推phpstudy2016和wampServer3.0.6 理由支持php7.0 目前 ...
- 【转】使用nvm快速搭建 Node.js 开发环境
原文链接:http://www.cnblogs.com/shuoer/p/7802891.html 快速搭建 Node.js 开发环境 如果你想长期做 node 开发, 或者想快速更新 node 版本 ...
- 搭建zookeeper和Kafka集群
搭建zookeeper和Kafka集群: 本实验拥有3个节点,均为CentOS 7系统,分别对应IP为10.211.55.11.10.211.55.13.10.211.55.14,且均有相同用户名 ( ...
- docker-compose搭建zookeeper集群环境 CodingCode
docker-compose搭建zookeeper集群环境 使用docker-compose搭建zookeeper集群环境 zookeeper是一个集群环境,用来管理微服务架构下面的配置管理功能. 这 ...
- centos7 快速搭建redis集群环境
本文主要是记录一下快速搭建redis集群环境的方式. 环境简介:centos 7 + redis-3.2.4 本次用两个服务6个节点来搭建:192.168.116.120 和 192.168.1 ...
- 快速搭建appium自动测试环境
首先申明本文是基本于Python与Android来快速搭建Appium自动化测试环境: 主要分为以下几个步骤: 前提条件: 1)安装与配置python环境,打开 Python官网,找到"Do ...
- Linux快速搭建C/C++开发环境
导读:越来越多的程序员在Linux下进行C/C++的开发.本文以CentOS 7为例,教你快速搭建一个vi + gcc/g++ + Make + valgrind的开发环境. 本文字数:1500,阅读 ...
随机推荐
- mybatis案例程序
前置工作 导包(mysql-connector-java.mybatis) 实体类 Mapper层 1.接口 public interface BookMapper { public Book get ...
- #trie#A 区间异或
题目 给定一个长度为\(n\)的序列,询问有多少个\((l,r),1\leq l\leq r\leq n\)满足 \[xor_{l\leq j\leq r}a_j\geq k \] 分析 显然跑一次前 ...
- 解决 `remote: You must use a personal access token with 'api' scope for Git over HTTP.`
背景 在家远程办公的时候 git clone 报错: remote: HTTP Basic: Access denied remote: You must use a personal access ...
- 在Centos 8 服务器用tmux多开窗口
在 CentOS 服务器上使用 tmux 来多开窗口是一个高效的方式.tmux 是一个终端复用器,它允许你在一个终端窗口中打开多个终端会话,还可以在会话之间轻松切换,非常适合长时间运行程序或多任务操作 ...
- SQL 数据库语句- 创建和管理数据库
SQL CREATE DATABASE 语句 SQL CREATE DATABASE 语句用于创建一个新的 SQL 数据库. 语法 CREATE DATABASE 数据库名称; 示例 以下 SQL 语 ...
- skywalking入口
skywalking通过maven配置指定org.apache.skywalking.apm.agent.SkyWalkingAgent为premain-class,该类中包含了一个premain方法 ...
- 从模型到部署,教你如何用Python构建机器学习API服务
本文分享自华为云社区<Python构建机器学习API服务从模型到部署的完整指南>,作者: 柠檬味拥抱. 在当今数据驱动的世界中,机器学习模型在解决各种问题中扮演着重要角色.然而,将这些模型 ...
- Python从 requirements.txt 安装库
pip install -r requirements.txt
- Python将依赖包导出到requirements.txt文件
代码 # 查询环境中已经安装的库 pip list # 将所有依赖库导出到 requirements.txt 文件 pip freeze > requirements.txt
- 日志分析qsnctfwp
使用工具:http Logs Viewer 使用 http Logs Viewer 载入 access.log 按照 Status 排序 其中大量 Status 为404的日志不难推断出,这是在进行目 ...