配置单机kafka

关闭selinux,开启防火墙9092端口

1、关闭selinux

vi /etc/selinux/config

#SELINUX=enforcing #注释掉

#SELINUXTYPE=targeted #注释掉

SELINUX=disabled #增加

:wq! #保存退出

setenforce 0 #使配置立即生效

2、配置防火墙,开启9092端口号

vi /etc/sysconfig/iptables #编辑防火墙配置文件

# Firewall configuration written by system-config-firewall

# Manual customization of this file is not recommended.

*filter

:INPUT ACCEPT [0:0]

:FORWARD ACCEPT [0:0]

:OUTPUT ACCEPT [0:0]

-A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT

-A INPUT -p icmp -j ACCEPT

-A INPUT -i lo -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 22 -j ACCEPT

-A INPUT -m state --state NEW -m tcp -p tcp --dport 9092 -j ACCEPT

-A INPUT -j REJECT --reject-with icmp-host-prohibited

-A FORWARD -j REJECT --reject-with icmp-host-prohibited

COMMIT
:wq! #保存退出

service iptables restart #最后重启防火墙使配置生效

安装jdk

kafka运行需要JDK支持

下载地址:https://www.oracle.com/java/technologies/javase-downloads.html

历史版本:https://www.oracle.com/java/technologies/oracle-java-archive-downloads.html

这里稍微多说一句,下载地址偶尔是会变动的,上面的地址是截止到这片文章发布时的最新链接;但是一般Oracle都会做兼容(如果上面的链接失效了,点开后一般会重定向到对应的新地址),下载界面也可能会随着网站整体UI而更新,但都是大同小异,进到Oracle官网按照导航找到需要的JDK版本并不是很难

1、解压安装

shell> mkdir /usr/local/java
shell> cd /usr/local/java
shell> tar -zxvf /tools/jdk-8u241-linux-x64.tar.gz shell> ll
total 4
drwxr-xr-x 7 10143 10143 4096 Dec 11 18:39 jdk1.8.0_241

2、配置环境变量

shell> vim /etc/profile

文件末尾增加以下配置

# JDK
export JAVA_HOME=/usr/local/java/jdk1.8.0_241
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/dt.jar:$JAVA_HOME/lib/tools.jar

使配置生效

source /etc/profile

3、测试

shell> java -version
java version "1.8.0_241"
Java(TM) SE Runtime Environment (build 1.8.0_241-b07)
Java HotSpot(TM) 64-Bit Server VM (build 25.241-b07, mixed mode)

安装ZooKeeper

安装kafka前还需要安装Zookeeper,zookeeper是运行在java环境下的。

1、下载

官网:http://zookeeper.apache.org/releases.html

镜像:https://www.apache.org/dyn/closer.cgi/zookeeper

2、解压

tar -zxvf apache-zookeeper-3.5.8-bin.tar.gz -C /usr/local/

3、修改配置

shell> cd /usr/local/apache-zookeeper-3.5.8-bin/conf/
shell> cp zoo_sample.cfg zoo.cfg
shell> vim zoo.cfg
dataDir=/usr/local/apache-zookeeper-3.5.8-bin/data
dataLogDir=/usr/local/apache-zookeeper-3.5.8-bin/logs

4、启动&停止

# 启动

shell> bin/zkServer.sh start
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.5.3-beta/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED
# 查看状态

shell> bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.5.3-beta/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: standalone
# 停止

shell> bin/zkServer.sh stop
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.5.3-beta/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
# 重启

shell> bin/zkServer.sh restart
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.5.3-beta/bin/../conf/zoo.cfg
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.5.3-beta/bin/../conf/zoo.cfg
Stopping zookeeper ... STOPPED
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.5.3-beta/bin/../conf/zoo.cfg
Starting zookeeper ... STARTED

5、测试

# jps,查看QuorumPeerMain是否存在

shell> jps
19748 Jps
19159 QuorumPeerMain
# zkServer.sh status

shell> bin/zkServer.sh status
ZooKeeper JMX enabled by default
Using config: /usr/local/zookeeper-3.5.3-beta/bin/../conf/zoo.cfg
Client port found: 2181. Client address: localhost.
Mode: standalone
# zkClient

shell> bin/zkCli.sh -server localhost:2181
Connecting to localhost:2181
2020-03-26 23:41:43,303 [myid:] - INFO [main:Environment@109] - Client environment:zookeeper.version=3.5.3-beta-8ce24f9e675cbefffb8f21a47e06b42864475a60, built on 04/03/2017 16:19 GMT
2020-03-26 23:41:43,306 [myid:] - INFO [main:Environment@109] - Client environment:host.name=VM_2_24_centos
2020-03-26 23:41:43,306 [myid:] - INFO [main:Environment@109] - Client environment:java.version=1.8.0_241
2020-03-26 23:41:43,308 [myid:] - INFO [main:Environment@109] - Client environment:java.vendor=Oracle Corporation
2020-03-26 23:41:43,308 [myid:] - INFO [main:Environment@109] - Client environment:java.home=/usr/local/java/jdk1.8.0_241/jre
2020-03-26 23:41:43,308 [myid:] - INFO [main:Environment@109] - Client environment:java.class.path=/usr/local/zookeeper-3.5.3-beta/bin/../build/classes:/usr/local/zookeeper-3.5.3-beta/bin/../build/lib/*.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/slf4j-log4j12-1.7.5.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/slf4j-api-1.7.5.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/netty-3.10.5.Final.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/log4j-1.2.17.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/jline-2.11.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/jetty-util-9.2.18.v20160721.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/jetty-servlet-9.2.18.v20160721.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/jetty-server-9.2.18.v20160721.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/jetty-security-9.2.18.v20160721.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/jetty-io-9.2.18.v20160721.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/jetty-http-9.2.18.v20160721.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/javax.servlet-api-3.1.0.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/jackson-mapper-asl-1.9.11.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/jackson-core-asl-1.9.11.jar:/usr/local/zookeeper-3.5.3-beta/bin/../lib/commons-cli-1.2.jar:/usr/local/zookeeper-3.5.3-beta/bin/../zookeeper-3.5.3-beta.jar:/usr/local/zookeeper-3.5.3-beta/bin/../src/java/lib/*.jar:/usr/local/zookeeper-3.5.3-beta/bin/../conf:.:/usr/local/java/jdk1.8.0_241/lib/dt.jar:/usr/local/java/jdk1.8.0_241/lib/tools.jar
2020-03-26 23:41:43,308 [myid:] - INFO [main:Environment@109] - Client environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2020-03-26 23:41:43,308 [myid:] - INFO [main:Environment@109] - Client environment:java.io.tmpdir=/tmp
2020-03-26 23:41:43,308 [myid:] - INFO [main:Environment@109] - Client environment:java.compiler=<NA>
2020-03-26 23:41:43,308 [myid:] - INFO [main:Environment@109] - Client environment:os.name=Linux
2020-03-26 23:41:43,308 [myid:] - INFO [main:Environment@109] - Client environment:os.arch=amd64
2020-03-26 23:41:43,308 [myid:] - INFO [main:Environment@109] - Client environment:os.version=3.10.0-1062.9.1.el7.x86_64
2020-03-26 23:41:43,308 [myid:] - INFO [main:Environment@109] - Client environment:user.name=root
2020-03-26 23:41:43,309 [myid:] - INFO [main:Environment@109] - Client environment:user.home=/root
2020-03-26 23:41:43,309 [myid:] - INFO [main:Environment@109] - Client environment:user.dir=/usr/local/zookeeper-3.5.3-beta
2020-03-26 23:41:43,309 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.free=52MB
2020-03-26 23:41:43,310 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.max=228MB
2020-03-26 23:41:43,310 [myid:] - INFO [main:Environment@109] - Client environment:os.memory.total=57MB
2020-03-26 23:41:43,313 [myid:] - INFO [main:ZooKeeper@865] - Initiating client connection, connectString=localhost:2181 sessionTimeout=30000 watcher=org.apache.zookeeper.ZooKeeperMain$MyWatcher@5e8c92f4
2020-03-26 23:41:43,323 [myid:] - INFO [main:ClientCnxnSocket@236] - jute.maxbuffer value is 4194304 Bytes
Welcome to ZooKeeper!
2020-03-26 23:41:43,335 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1113] - Opening socket connection to server localhost/0:0:0:0:0:0:0:1:2181. Will not attempt to authenticate using SASL (unknown error)
JLine support is enabled
2020-03-26 23:41:43,401 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@948] - Socket connection established, initiating session, client: /0:0:0:0:0:0:0:1:42446, server: localhost/0:0:0:0:0:0:0:1:2181
2020-03-26 23:41:43,412 [myid:localhost:2181] - INFO [main-SendThread(localhost:2181):ClientCnxn$SendThread@1381] - Session establishment complete on server localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x10009d728340001, negotiated timeout = 30000 WATCHER:: WatchedEvent state:SyncConnected type:None path:null
[zk: localhost:2181(CONNECTED) 0] ls /
[zookeeper]

安装kafka

1、安装前准备

安装JDK

安装ZooKeeper

2、解压

shell> tar -zxvf kafka_2.12-2.2.2.tgz -C /usr/local/

3、配置

cd /usr/local/kafka_2.12-2.2.2/config/
vim server.properties

编辑以下内容

############################# Server Basics #############################
broker.id=0 # 集群的话,这个id会不同,也可以自定义
port=9092 # kafka的端口号,可以不配置此项,默认为9092
host.name=172.17.194.139 # 阿里云内网地址
advertised.host.name=123.56.162.92 # 阿里云外网映射地址 # 默认为/tmp/kafka-logs,可能会被删掉
log.dirs=/usr/local/kafka_2.12-2.2.2/logs # 监听地址,ifconfig查看到的本机真实地址,单节点不配置也可以
# 如果部署在阿里云服务器上,此处不能配置为公网ip,会直接启动不了kafka
# 阿里云服务器上需要配置为:listeners=PLAINTEXT://:9092,或者内网IP
listeners=PLAINTEXT://172.17.194.139:9092 # 打开注释,设置为对外可访问的地址,可以是主机名,也可以是ip地址,但一定是连接kafka的其他机器可以访问到的地址,简单来说就果是云主机并且直接通过外网访问kafka(一般不这么做,但是如果这么做了),需要设置成弹性ip或绑定的域名
# advertised.listeners=PLAINTEXT://tianranll.cn:9092
# 如果部署在本地虚拟机中,此处可以不开注释;
# 如果部署在云服务器中,需要配置此处ip为公网ip
advertised.listeners=PLAINTEXT://123.56.162.92:9092 # zookeeper地址及端口号
# 如果是集群,可能是这样:hostname1:port1,hostname2:port2,hostname3:port3
# 如果是带有chroot:hostname1:port1,hostname2:port2,hostname3:port3/chroot/path
zookeeper.connect=172.17.194.139:2181 # zookeeper如果和kafka装在同一台机器上,设置为阿里云的内网IP即可。

4、启动&停止

# 启动
cd /usr/local/kafka_2.12-2.2.2/
nohup bin/kafka-server-start.sh config/server.properties

如果使用kafka-manager,需要开启JMX_PORT

shell> JMX_PORT=9999 bin/kafka-server-start.sh -daemon config/server.properties
shell> bin/kafka-server-stop.sh

5、测试

1、jps,确认kafka是否存在(或者ps查看kafka相关进程)
2、zookeeper查看注册信息
shell> cd /usr/local/apache-zookeeper-3.5.8-bin/
shell> bin/zkCli.sh
[zk: localhost:2181(CONNECTED) 0] ls /brokers/ids
[0]
[zk: localhost:2181(CONNECTED) 1] get /brokers/ids/0
{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://VM_2_24_centos:9092"],"jmx_port":-1,"host":"VM_2_24_centos","timestamp":"1585396767598","port":9092,"version":4}
3、kafka命令
创建topic
shell> bin/kafka-topics.sh --create --bootstrap-server 123.56.162.92:9092 --replication-factor 1 --partitions 1 --topic test
查看topic
shell> bin/kafka-topics.sh --list --bootstrap-server 123.56.162.92:9092:9092
test
生产者发送消息
shell> bin/kafka-console-producer.sh --broker-list 123.56.162.92:9092:9092 --topic test
>Hello
>World
消费者消费消息
shell> bin/kafka-console-consumer.sh --bootstrap-server 123.56.162.92:9092 --topic test --from-beginning
Hello

创建启动脚本

cd /usr/local//usr/local/kafka_2.12-2.2.2
vim kafkastart.sh
#!/bin/sh

#启动zookeeper

/usr/local/apache-zookeeper-3.5.8-bin/bin/zkServer.sh start &

sleep 3 #等3秒后执行

#守护进程启动kafka
nohup /usr/local/kafka_2.12-2.2.2/bin/kafka-server-start.sh /usr/local/kafka_2.12-2.2.2/config/server.properties > /dev/null 2>&1 &

创建关闭脚本

vim kafkastop.sh
#!/bin/sh

#关闭zookeeper

/usr/local/apache-zookeeper-3.5.8-bin/bin/zkServer.sh stop &

sleep 3 #等3秒后执行

#关闭kafka

/usr/local/kafka_2.12-2.2.2/bin/kafka-server-stop.sh &

添加脚本执行权限

#添加脚本执行权限

chmod +x kafkastart.sh

chmod +x kafkastop.sh

设置脚本开机自动执行

vi /etc/rc.d/rc.local #编辑,在最后添加一行

sh /usr/local/kafka/kafkastart.sh & #设置开机自动在后台运行脚本

:wq! #保存退出

sh /usr/local/kafka/kafkastart.sh #启动kafka

sh /usr/local/kafka/kafkastop.sh #关闭kafka

配置单机Kafka的更多相关文章

  1. Centos 7 配置单机Hadoop

    Centos 7 配置单机Hadoop 2018年10月11日 09:48:13 GT_Stone 阅读数:82   系统镜像:CentuOS-7-x86_64-Everything-1708 Jav ...

  2. 在Centos 7上安装配置 Apche Kafka 分布式消息系统集群

    Apache Kafka是一种颇受欢迎的分布式消息代理系统,旨在有效地处理大量的实时数据.Kafka集群不仅具有高度可扩展性和容错性,而且与其他消息代理(如ActiveMQ和RabbitMQ)相比,还 ...

  3. MongoDB3.0.x版本用户授权配置(单机环境)

    MongoDB数据库默认情况下是没有做权限控制的,只要能够连接所开放的端口就能进行访问,而且拥有root级别的权限:对于生产环境而言是极不安全的,所以需要建立用户,进行授权控制. 单机环境下的用户授权 ...

  4. 分布式文件系统FastDFS安装与配置(单机)

    安装包如下:fastdfs-nginx-module_v1.16.tar.gzFastDFS_v5.05.tar.gzlibfastcommon-master.zipnginx-1.8.0.tar.g ...

  5. ZooKeeper 的安装和配置---单机和集群

    如题本文介绍的是ZooKeeper 的安装和配置过程,此过程非常简单,关键是如何应用(将放在下节及相关节中介绍). 单机安装.配置: 安装非常简单,只要获取到 Zookeeper 的压缩包并解压到某个 ...

  6. Windows下安装单机Kafka

    Zookeeper 解压后进入conf目录,复制出一个zoo.cfg,然后进入bin目录,直接运行zkServer.cmd Kafka 首先你得有一个安装包: 解压,目录结构: config目录里面是 ...

  7. zookeeper安装和配置(单机+伪集群+集群)

    #单机模式 解压到合适目录. 进入zookeeper目录下的conf子目录, 复制zoo_sample.cfg-->zoo.cfg(如果没有data和logs就新建):tickTime=2000 ...

  8. Winows上简单配置使用kafka(.net使用)

    一.kafka环境配置 1.jdk安装 安装文件:http://www.oracle.com/technetwork/java/javase/downloads/index.html 下载JDK安装完 ...

  9. zookeeper之一 安装和配置(单机+集群)

    这里我以zookeeper3.4.10.tar.gz来演示安装,安装到/usr/local/soft目录下. 一.单机版配置 1.安装和配置 #.下载 wget http://apache.fayea ...

随机推荐

  1. VUE 后台管理系统权限控制

    谈一谈VUE 后台管理系统权限控制 前端权限从本质上来讲, 就是控制视图层的展示,比如说是某个页面或者某个按钮,后端权限可以控制某个用户是否能够查询数据, 是否能够修改数据等操作,后端权限大部分是基于 ...

  2. 题目分享G 二代目

    题意:有n组数,每组包含两个数,问在每组只能取一个的前提下能组成的最长的从1开始的连续自然数有几个? 分析:刚学了差分约束系统,很容易往转换成图的方向去想 将他读入的这n组数当成边读入 很容易会得到一 ...

  3. 软件——IDEA中如何去掉警告虚线

    初次安装使用IDEA,总是能看到导入代码后,出现很多的波浪线,下划线和虚线,这是IDEA给我们的一些提示和警告,但是有时候我们并不需要,反而会让人看着很不爽,这里简单记录一下自己的调整方法,供其他的小 ...

  4. SpringBoot系列(十四)集成邮件发送服务及邮件发送的几种方式

    往期推荐 SpringBoot系列(一)idea新建Springboot项目 SpringBoot系列(二)入门知识 springBoot系列(三)配置文件详解 SpringBoot系列(四)web静 ...

  5. Centos7 网卡桥接

     一.在centos7主机创建用于虚拟化的网桥 1)增加 /etc/sysconfig/network-scripts/ifcfg-br0 DEVICE=br0BOOTPROTO=staticONBO ...

  6. [hdu5389 Zero Escape]数根的性质,DP

    题意:把n个数(1-9)放到A集合和B集合里面去,使得A集合里面的数的数根为a,B集合里面的数的数根为b,也可以只放在A或B任一个集合里面.求方法总数.比如A={2,4,5},则A的数根为[2+4+5 ...

  7. 管理环境一:venv

    初衷: 在工作的时候,我们会有很多个项目,每个项目使用的库的版本不一样,导致我们切换项目的时候会很麻烦. 比如:我有两个django项目,项目一使用的版本是 django 1.7 , 项目二使用的版本 ...

  8. 推荐一款 python 管理工具:anaconda

    1.jpg 2.jpg 3.jpg 4.jpg 5.jpg 6.jpg 7.jpg 8.jpg 9.jpg 10.jpg 11.jpg 12.jpg 13.jpg 14.jpg 15.jpg 16.j ...

  9. JavaWeb学习之JSP(三) EL表达式

    EL表达式 什么是EL表达式   EL,Expression Language,表达式语言,是一种在JSP页面中获取数据的简单方式,通过${变量名}的方式可以获取到值,需要注意的是EL只能从 page ...

  10. 我的.vimrc配置

    termux平台下,通过pkg update && pkg upgrade && pkg install vim来获得vim.可是,当我配置.vimrc时发现如果我每在 ...