一、场景

线上已经有kafka集群，服务运行稳定。但是因为产品升级，需要对kakfa做安全测试，也就是权限验证。

但是增加权限验证，会不会对性能有影响呢？影响大吗？不知道呀！

因此，本文就此来做一下对比！

测试环境

本文使用的是单机kafka测试，仅在一台服务器做测试。

使用的kafka版本为kafka_2.12-2.1.0，zk直接使用kafka自带的。

硬件配置为：1(核)-4G内容-40G硬盘

二、普通版安装

关于普通版的kafka，请参考链接：

https://www.cnblogs.com/xiao987334176/p/10075659.html

里面有关于 kafka_server_test 镜像如何制作！

三、acl版安装

这里使用的acl设置，仅仅使用write用户来测试。赋予它所有权限，下面会说到！

新建空目录

mkdir /opt/kafka_server_acl

consumer.config

security.protocol=SASL_PLAINTEXT

sasl.mechanism=PLAIN

group.id=test-group

dockerfile

FROM ubuntu:16.04

# 修改更新源为阿里云

ADD sources.list /etc/apt/sources.list

ADD kafka_2.-2.1..tgz /

ADD kafka_cluster_jaas.conf /

ADD consumer.config /kafka_2.-2.1./config/

ADD producer.config /kafka_2.-2.1./config/

#ADD reader_jaas.conf /kafka_2.-2.1./config/

ADD writer_jaas.conf /kafka_2.-2.1./config/

# 安装jdk

RUN apt-get update && apt-get install -y openjdk--jdk --allow-unauthenticated && apt-get clean all && \

    cd /kafka_2.-2.1. && \

    mv /kafka_cluster_jaas.conf config/ && \

    sed -i '$ s/^/#&/g' bin/kafka-server-start.sh && \

    sed -i '$ a\exec $base_dir/kafka-run-class.sh $EXTRA_ARGS -Djava.security.auth.login.config=/kafka_2.12-2.1.0/config/kafka_cluster_jaas.conf kafka.Kafka "$@"' bin/kafka-server-start.sh

EXPOSE

# 添加启动脚本

ADD run.sh .

RUN chmod  run.sh

ENTRYPOINT [ "/run.sh"]

kafka_cluster_jaas.conf

KafkaServer {

org.apache.kafka.common.security.plain.PlainLoginModule required

username="admin"

password="admin"

user_admin="admin"

user_reader="reader"

user_writer="writer";

};

producer.config

security.protocol=SASL_PLAINTEXT

sasl.mechanism=PLAIN

run.sh

#!/bin/bash

# 启动自带的zookeeper

cd /kafka_2.-2.1.

bin/zookeeper-server-start.sh config/zookeeper.properties &

# 启动kafka acl配置

echo "

authorizer.class.name=kafka.security.auth.SimpleAclAuthorizer

listeners=SASL_PLAINTEXT://:9092

security.inter.broker.protocol= SASL_PLAINTEXT

sasl.mechanism.inter.broker.protocol=PLAIN

sasl.enabled.mechanisms=PLAIN

super.users=User:admin

" >> /kafka_2.12-2.1.0/config/server.properties

# 进入工作目录

cd /kafka_2.-2.1.

# 生产者

# 最后一行注释掉,添加#

sed -i '$ s/^/#&/g' bin/kafka-console-producer.sh

# 最后一行添加内容

sed -i '$ a\exec $(dirname $0)/kafka-run-class.sh -Djava.security.auth.login.config=/kafka_2.12-2.1.0/config/writer_jaas.conf kafka.tools.ConsoleProducer "$@"' bin/kafka-console-producer.sh

# 消费者

sed -i '$ s/^/#&/g' bin/kafka-console-consumer.sh

sed -i '$ a\exec $(dirname $0)/kafka-run-class.sh -Djava.security.auth.login.config=/kafka_2.12-2.1.0/config/writer_jaas.conf kafka.tools.ConsoleConsumer "$@"' bin/kafka-console-consumer.sh

# 生产者压测脚本

sed -i '$ s/^/#&/g' bin/kafka-producer-perf-test.sh

sed -i '$ a\exec $(dirname $0)/kafka-run-class.sh -Djava.security.auth.login.config=/kafka_2.12-2.1.0/config/writer_jaas.conf org.apache.kafka.tools.ProducerPerformance "$@"' bin/kafka-producer-perf-test.sh

# 消费者压测脚本

sed -i '$ s/^/#&/g' bin/kafka-consumer-perf-test.sh

sed -i '$ a\exec $(dirname $0)/kafka-run-class.sh -Djava.security.auth.login.config=/kafka_2.12-2.1.0/config/writer_jaas.conf kafka.tools.ConsumerPerformance "$@"' bin/kafka-consumer-perf-test.sh

# 启动kafka

sleep

bin/kafka-server-start.sh config/server.properties

注意：默认的kafka压测脚本是针对于普通版的kafka。如果是已经开启了acl验证的kafkfa，运行会直接报错。

因此为了解决这个问题，有2个步骤：

1. 修改压测脚本，最后一行指定参数-Djava.security.auth.login.config

2. 运行脚本时，要增加参数。下文会说到，稍安勿躁！

sources.list

deb http://mirrors.aliyun.com/ubuntu/ xenial main restricted

deb http://mirrors.aliyun.com/ubuntu/ xenial-updates main restricted

deb http://mirrors.aliyun.com/ubuntu/ xenial universe

deb http://mirrors.aliyun.com/ubuntu/ xenial-updates universe

deb http://mirrors.aliyun.com/ubuntu/ xenial multiverse

deb http://mirrors.aliyun.com/ubuntu/ xenial-updates multiverse

deb http://mirrors.aliyun.com/ubuntu/ xenial-backports main restricted universe multiverse

deb http://mirrors.aliyun.com/ubuntu xenial-security main restricted

deb http://mirrors.aliyun.com/ubuntu xenial-security universe

deb http://mirrors.aliyun.com/ubuntu xenial-security multiverse

writer_jaas.conf

KafkaClient {

org.apache.kafka.common.security.plain.PlainLoginModule required

username="writer"

password="writer";

};

此时目录结构如下：

./

├── consumer.config

├── dockerfile

├── kafka_2.-2.1..tgz

├── kafka_cluster_jaas.conf

├── producer.config

├── run.sh

├── sources.list

└── writer_jaas.conf

生成镜像

docker build -t kafka_server_acl /opt/kafka_server_acl

启动容器

docker run -it kafka_server_acl

查看docker进程

root@ubuntu:~# docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS              PORTS               NAMES

4f4c33fd92fa        kafka_server_acl    "/run.sh"            seconds ago      Up  seconds       /tcp            eager_goldberg

进入容器

root@ubuntu:~# docker exec -it 4f4c33fd92fa /bin/bash

root@4f4c33fd92fa:/# cd /kafka_2.12-2.1.0/

设置权限

bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost: --add --allow-principal User:writer --operation All --topic=*

参数解释：

User:writer  表示用户writer

--operation All  操作权限为所有。一般操作权限分为2种，分别是Read和Write。所以All就包含了这2种权限。

--topic=* 表示所有topic

输出：

Adding ACLs for resource `Topic:LITERAL:*`:

     User:writer has Allow permission for operations: All from hosts: * 

Current ACLs for resource `Topic:LITERAL:*`:

     User:writer has Allow permission for operations: All from hosts: *

允许writer用户的所有组操作。一般在java代码中，组名是不固定的。有些是动态生成的，所以直接*

bin/kafka-acls.sh --authorizer kafka.security.auth.SimpleAclAuthorizer --authorizer-properties zookeeper.connect=localhost: --add --allow-principal User:writer --operation All -group=*

注意：topic和grop权限，不能一条命令设置，需要分2条命令执行才行！

输出：

Adding ACLs for resource `Group:LITERAL:*`:

     User:writer has Allow permission for operations: All from hosts: * 

Current ACLs for resource `Group:LITERAL:*`:

     User:writer has Allow permission for operations: All from hosts: *

创建测试topic，名为test

bin/kafka-topics.sh --create --zookeeper localhost: --replication-factor  --partitions  --topic test

进入生产者模式

bin/kafka-console-producer.sh --broker-list localhost: --topic test --producer.config config/producer.config

进入消费者模式

bin/kafka-console-consumer.sh --bootstrap-server localhost: --topic test --from-beginning --consumer.config config/consumer.config

在生产者模式输入234

>

消费者模式如果收到234，说明acl运行正常！

四、100万性能对比

普通版

先启动镜像

docker run -it kafka_server_test

查看docker进程

root@ubuntu:~# docker ps

CONTAINER ID        IMAGE               COMMAND             CREATED              STATUS              PORTS               NAMES

eb40b250c38e        kafka_server_test   "/run.sh"           About a minute ago   Up About a minute   /tcp            cocky_bhabha

进入容器

root@ubuntu:~# docker exec -it eb40b250c38e /bin/bash

root@eb40b250c38e:/# cd /kafka_2.12-2.1.0/

写入100万

bin/kafka-producer-perf-test.sh --topic test_perf --num-records  --record-size   --throughput  --producer-props bootstrap.servers=localhost:

输出：

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

 records sent, 0.2 records/sec (0.00 MB/sec), 6529.0 ms avg latency, 6529.0 max latency.

 records sent, 195.5 records/sec (0.19 MB/sec), 4129.0 ms avg latency, 5700.0 max latency.

 records sent, 505.2 records/sec (0.48 MB/sec), 7376.3 ms avg latency, 9462.0 max latency.

 records sent, 548.0 records/sec (0.52 MB/sec), 11849.3 ms avg latency, 14333.0 max latency.

 records sent, 799.5 records/sec (0.76 MB/sec), 16470.2 ms avg latency, 18490.0 max latency.

 records sent, 1623.0 records/sec (1.55 MB/sec), 20567.4 ms avg latency, 22227.0 max latency.

 records sent, 1821.4 records/sec (1.74 MB/sec), 22738.9 ms avg latency, 23278.0 max latency.

 records sent, 1792.0 records/sec (1.71 MB/sec), 23369.0 ms avg latency, 23889.0 max latency.

 records sent, 2298.0 records/sec (2.19 MB/sec), 19010.4 ms avg latency, 23768.0 max latency.

 records sent, 2155.4 records/sec (2.06 MB/sec), 16760.0 ms avg latency, 17435.0 max latency.

 records sent, 2214.9 records/sec (2.11 MB/sec), 15430.7 ms avg latency, 16299.0 max latency.

 records sent, 2669.8 records/sec (2.55 MB/sec), 14335.0 ms avg latency, 14934.0 max latency.

 records sent, 2360.1 records/sec (2.25 MB/sec), 13364.5 ms avg latency, 14022.0 max latency.

 records sent, 2801.1 records/sec (2.67 MB/sec), 12638.4 ms avg latency, 13613.0 max latency.

 records sent, 3672.2 records/sec (3.50 MB/sec), 11248.4 ms avg latency, 12355.0 max latency.

 records sent, 3619.3 records/sec (3.45 MB/sec), 9437.7 ms avg latency, 10156.0 max latency.

 records sent, 2846.6 records/sec (2.71 MB/sec), 7655.5 ms avg latency, 9229.0 max latency.

 records sent, 3673.6 records/sec (3.50 MB/sec), 4640.2 ms avg latency, 6955.0 max latency.

 records sent, 2879.2 records/sec (2.75 MB/sec), 1508.1 ms avg latency, 2735.0 max latency.

 records sent, 2346.7 records/sec (2.24 MB/sec), 176.9 ms avg latency, 836.0 max latency.

 records sent, 1967.7 records/sec (1.88 MB/sec), 64.5 ms avg latency, 416.0 max latency.

 records sent, 1998.0 records/sec (1.91 MB/sec), 43.7 ms avg latency, 512.0 max latency.

 records sent, 2002.0 records/sec (1.91 MB/sec), 11.7 ms avg latency, 296.0 max latency.

 records sent, 2035.6 records/sec (1.94 MB/sec), 118.2 ms avg latency, 730.0 max latency.

 records sent, 2001.0 records/sec (1.91 MB/sec), 47.0 ms avg latency, 423.0 max latency.

 records sent, 1964.6 records/sec (1.87 MB/sec), 88.8 ms avg latency, 421.0 max latency.

 records sent, 2036.4 records/sec (1.94 MB/sec), 39.3 ms avg latency, 418.0 max latency.

 records sent, 2041.9 records/sec (1.95 MB/sec), 11.3 ms avg latency, 197.0 max latency.

 records sent, 2006.3 records/sec (1.91 MB/sec), 22.2 ms avg latency, 329.0 max latency.

 records sent, 1995.7 records/sec (1.90 MB/sec), 7.3 ms avg latency, 193.0 max latency.

 records sent, 2036.8 records/sec (1.94 MB/sec), 1.8 ms avg latency, 36.0 max latency.

 records sent, 1967.2 records/sec (1.88 MB/sec), 5.9 ms avg latency, 189.0 max latency.

 records sent, 2001.6 records/sec (1.91 MB/sec), 3.1 ms avg latency, 109.0 max latency.

 records sent, 2033.6 records/sec (1.94 MB/sec), 0.6 ms avg latency, 78.0 max latency.

 records sent, 2003.6 records/sec (1.91 MB/sec), 0.7 ms avg latency, 89.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.4 ms avg latency, 67.0 max latency.

 records sent, 2001.2 records/sec (1.91 MB/sec), 0.5 ms avg latency, 70.0 max latency.

 records sent, 1973.4 records/sec (1.88 MB/sec), 0.4 ms avg latency, 48.0 max latency.

 records sent, 2028.0 records/sec (1.93 MB/sec), 0.7 ms avg latency, 63.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.4 ms avg latency, 39.0 max latency.

 records sent, 2000.8 records/sec (1.91 MB/sec), 0.8 ms avg latency, 72.0 max latency.

 records sent, 2000.8 records/sec (1.91 MB/sec), 0.5 ms avg latency, 61.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.5 ms avg latency, 36.0 max latency.

 records sent, 2002.0 records/sec (1.91 MB/sec), 0.7 ms avg latency, 71.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.5 ms avg latency, 86.0 max latency.

 records sent, 1999.8 records/sec (1.91 MB/sec), 0.5 ms avg latency, 44.0 max latency.

 records sent, 2001.0 records/sec (1.91 MB/sec), 0.5 ms avg latency, 84.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.5 ms avg latency, 42.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.5 ms avg latency, 66.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 1.1 ms avg latency, 84.0 max latency.

 records sent, 2026.4 records/sec (1.93 MB/sec), 0.6 ms avg latency, 82.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.8 ms avg latency, 72.0 max latency.

 records sent, 2002.4 records/sec (1.91 MB/sec), 0.6 ms avg latency, 53.0 max latency.

 records sent, 1998.8 records/sec (1.91 MB/sec), 0.5 ms avg latency, 25.0 max latency.

 records sent, 1979.8 records/sec (1.89 MB/sec), 0.7 ms avg latency, 87.0 max latency.

 records sent, 2023.2 records/sec (1.93 MB/sec), 1.3 ms avg latency, 89.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.8 ms avg latency, 88.0 max latency.

 records sent, 2001.6 records/sec (1.91 MB/sec), 0.6 ms avg latency, 58.0 max latency.

 records sent, 1999.6 records/sec (1.91 MB/sec), 0.8 ms avg latency, 53.0 max latency.

 records sent, 1976.4 records/sec (1.88 MB/sec), 0.5 ms avg latency, 53.0 max latency.

 records sent, 2025.0 records/sec (1.93 MB/sec), 3.6 ms avg latency, 184.0 max latency.

 records sent, 2000.2 records/sec (1.91 MB/sec), 3.5 ms avg latency, 183.0 max latency.

 records sent, 2001.4 records/sec (1.91 MB/sec), 1.0 ms avg latency, 84.0 max latency.

 records sent, 2001.0 records/sec (1.91 MB/sec), 1.0 ms avg latency, 88.0 max latency.

 records sent, 2001.6 records/sec (1.91 MB/sec), 2.8 ms avg latency, 161.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.5 ms avg latency, 88.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 72.0 max latency.

 records sent, 2000.8 records/sec (1.91 MB/sec), 0.7 ms avg latency, 59.0 max latency.

 records sent, 2000.8 records/sec (1.91 MB/sec), 0.5 ms avg latency, 54.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.9 ms avg latency, 54.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 1.3 ms avg latency, 101.0 max latency.

 records sent, 1999.2 records/sec (1.91 MB/sec), 0.8 ms avg latency, 77.0 max latency.

 records sent, 2001.6 records/sec (1.91 MB/sec), 0.8 ms avg latency, 84.0 max latency.

 records sent, 2031.8 records/sec (1.94 MB/sec), 2.9 ms avg latency, 177.0 max latency.

 records sent, 2001.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 76.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 1.0 ms avg latency, 77.0 max latency.

 records sent, 1999.6 records/sec (1.91 MB/sec), 1.1 ms avg latency, 100.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.7 ms avg latency, 63.0 max latency.

 records sent, 1984.0 records/sec (1.89 MB/sec), 0.9 ms avg latency, 86.0 max latency.

 records sent, 2016.4 records/sec (1.92 MB/sec), 1.0 ms avg latency, 88.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.7 ms avg latency, 76.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 1.1 ms avg latency, 82.0 max latency.

 records sent, 1986.4 records/sec (1.89 MB/sec), 0.7 ms avg latency, 70.0 max latency.

 records sent, 2014.8 records/sec (1.92 MB/sec), 0.9 ms avg latency, 70.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.8 ms avg latency, 76.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 1.5 ms avg latency, 104.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 1.0 ms avg latency, 89.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.9 ms avg latency, 89.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 1.2 ms avg latency, 99.0 max latency.

 records sent, 2000.8 records/sec (1.91 MB/sec), 0.8 ms avg latency, 80.0 max latency.

 records sent, 1999.6 records/sec (1.91 MB/sec), 6.0 ms avg latency, 262.0 max latency.

 records sent, 2000.8 records/sec (1.91 MB/sec), 0.8 ms avg latency, 73.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 1.3 ms avg latency, 77.0 max latency.

 records sent, 1999.6 records/sec (1.91 MB/sec), 0.7 ms avg latency, 80.0 max latency.

 records sent, 2001.6 records/sec (1.91 MB/sec), 0.7 ms avg latency, 50.0 max latency.

 records sent, 1975.2 records/sec (1.88 MB/sec), 0.9 ms avg latency, 83.0 max latency.

 records sent, 2026.8 records/sec (1.93 MB/sec), 0.7 ms avg latency, 77.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 1.2 ms avg latency, 71.0 max latency.

 records sent, 1999.548102 records/sec (1.91 MB/sec), 2456.76 ms avg latency, 23889.00 ms max latency,  ms 50th,  ms 95th,  ms 99th,  ms .9th.

消费100万

bin/kafka-consumer-perf-test.sh --broker-list localhost: --topic test_perf --fetch-size  --messages  --threads

如果出现以下提示：

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec

WARNING: Exiting before consuming the expected number of messages: timeout ( ms) exceeded. You can use the --timeout option to increase the timeout.

-- :::, -- :::, 0.0000, 0.0000, , 0.0000, , , 0.0000, 0.0000

表示超时了，请仔细检查参数是否正确！

正常输出：

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec

-- :::, -- :::, 953.6743, 26.1088, , 27377.0088, , , 26.1891, 27461.2110

为了公平起见，删除进程

docker rm eb40b250c38e -f

acl版

写入100万

注意要增加参数 --producer.config config/producer.config

bin/kafka-producer-perf-test.sh --producer.config config/producer.config --topic test_perf --num-records  --record-size   --throughput  --producer-props bootstrap.servers=localhost:

输出：

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

[-- ::,] WARN [Producer clientId=producer-] Error while fetching metadata with correlation id  : {test_perf=LEADER_NOT_AVAILABLE} (org.apache.kafka.clients.NetworkClient)

 records sent, 0.1 records/sec (0.00 MB/sec), 6959.0 ms avg latency, 6959.0 max latency.

 records sent, 183.2 records/sec (0.17 MB/sec), 3791.8 ms avg latency, 5484.0 max latency.

 records sent, 572.3 records/sec (0.55 MB/sec), 6858.4 ms avg latency, 8794.0 max latency.

 records sent, 548.1 records/sec (0.52 MB/sec), 11087.4 ms avg latency, 13351.0 max latency.

 records sent, 1160.7 records/sec (1.11 MB/sec), 15455.8 ms avg latency, 17528.0 max latency.

 records sent, 1831.7 records/sec (1.75 MB/sec), 19294.1 ms avg latency, 21259.0 max latency.

 records sent, 1754.0 records/sec (1.67 MB/sec), 21916.4 ms avg latency, 22293.0 max latency.

 records sent, 2480.0 records/sec (2.37 MB/sec), 20387.3 ms avg latency, 22414.0 max latency.

 records sent, 2785.4 records/sec (2.66 MB/sec), 15041.6 ms avg latency, 16889.0 max latency.

 records sent, 2144.4 records/sec (2.05 MB/sec), 12817.3 ms avg latency, 13446.0 max latency.

 records sent, 2629.6 records/sec (2.51 MB/sec), 13026.9 ms avg latency, 13467.0 max latency.

 records sent, 2297.6 records/sec (2.19 MB/sec), 13915.6 ms avg latency, 14426.0 max latency.

 records sent, 3499.6 records/sec (3.34 MB/sec), 12703.0 ms avg latency, 14420.0 max latency.

 records sent, 3530.5 records/sec (3.37 MB/sec), 10118.3 ms avg latency, 11597.0 max latency.

 records sent, 3415.1 records/sec (3.26 MB/sec), 6927.8 ms avg latency, 8714.0 max latency.

 records sent, 3464.7 records/sec (3.30 MB/sec), 3330.6 ms avg latency, 5185.0 max latency.

 records sent, 2580.7 records/sec (2.46 MB/sec), 297.0 ms avg latency, 1450.0 max latency.

 records sent, 1889.8 records/sec (1.80 MB/sec), 7.2 ms avg latency, 211.0 max latency.

 records sent, 2113.0 records/sec (2.02 MB/sec), 81.2 ms avg latency, 518.0 max latency.

 records sent, 1967.5 records/sec (1.88 MB/sec), 49.0 ms avg latency, 413.0 max latency.

 records sent, 1965.7 records/sec (1.87 MB/sec), 19.0 ms avg latency, 210.0 max latency.

 records sent, 1999.6 records/sec (1.91 MB/sec), 14.1 ms avg latency, 211.0 max latency.

 records sent, 2075.2 records/sec (1.98 MB/sec), 35.2 ms avg latency, 422.0 max latency.

 records sent, 1967.9 records/sec (1.88 MB/sec), 6.2 ms avg latency, 209.0 max latency.

 records sent, 2008.6 records/sec (1.92 MB/sec), 17.7 ms avg latency, 308.0 max latency.

 records sent, 2027.4 records/sec (1.93 MB/sec), 23.7 ms avg latency, 296.0 max latency.

 records sent, 2124.3 records/sec (2.03 MB/sec), 36.9 ms avg latency, 397.0 max latency.

 records sent, 2033.4 records/sec (1.94 MB/sec), 7.6 ms avg latency, 182.0 max latency.

 records sent, 1969.5 records/sec (1.88 MB/sec), 22.2 ms avg latency, 397.0 max latency.

 records sent, 1957.6 records/sec (1.87 MB/sec), 2.7 ms avg latency, 190.0 max latency.

 records sent, 2074.4 records/sec (1.98 MB/sec), 3.4 ms avg latency, 187.0 max latency.

 records sent, 1980.7 records/sec (1.89 MB/sec), 1.0 ms avg latency, 87.0 max latency.

 records sent, 2030.6 records/sec (1.94 MB/sec), 0.8 ms avg latency, 76.0 max latency.

 records sent, 1984.8 records/sec (1.89 MB/sec), 1.0 ms avg latency, 71.0 max latency.

 records sent, 2017.6 records/sec (1.92 MB/sec), 1.2 ms avg latency, 86.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.7 ms avg latency, 47.0 max latency.

 records sent, 1990.0 records/sec (1.90 MB/sec), 0.7 ms avg latency, 77.0 max latency.

 records sent, 2012.4 records/sec (1.92 MB/sec), 0.8 ms avg latency, 35.0 max latency.

 records sent, 2001.2 records/sec (1.91 MB/sec), 1.9 ms avg latency, 107.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.8 ms avg latency, 54.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.5 ms avg latency, 48.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 1.0 ms avg latency, 83.0 max latency.

 records sent, 2000.8 records/sec (1.91 MB/sec), 0.7 ms avg latency, 70.0 max latency.

 records sent, 2000.8 records/sec (1.91 MB/sec), 0.7 ms avg latency, 87.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.9 ms avg latency, 67.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.8 ms avg latency, 58.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.7 ms avg latency, 90.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.8 ms avg latency, 71.0 max latency.

 records sent, 2002.8 records/sec (1.91 MB/sec), 1.2 ms avg latency, 89.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.9 ms avg latency, 72.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.9 ms avg latency, 85.0 max latency.

 records sent, 2000.8 records/sec (1.91 MB/sec), 0.8 ms avg latency, 85.0 max latency.

 records sent, 2002.0 records/sec (1.91 MB/sec), 1.2 ms avg latency, 84.0 max latency.

 records sent, 1984.5 records/sec (1.89 MB/sec), 0.8 ms avg latency, 70.0 max latency.

 records sent, 2017.6 records/sec (1.92 MB/sec), 0.8 ms avg latency, 65.0 max latency.

 records sent, 1996.8 records/sec (1.90 MB/sec), 0.8 ms avg latency, 72.0 max latency.

 records sent, 2003.6 records/sec (1.91 MB/sec), 0.7 ms avg latency, 81.0 max latency.

 records sent, 2001.2 records/sec (1.91 MB/sec), 0.6 ms avg latency, 66.0 max latency.

 records sent, 1972.0 records/sec (1.88 MB/sec), 0.8 ms avg latency, 82.0 max latency.

 records sent, 2016.0 records/sec (1.92 MB/sec), 0.9 ms avg latency, 89.0 max latency.

 records sent, 1979.0 records/sec (1.89 MB/sec), 2.2 ms avg latency, 91.0 max latency.

 records sent, 2035.6 records/sec (1.94 MB/sec), 11.9 ms avg latency, 310.0 max latency.

 records sent, 1970.4 records/sec (1.88 MB/sec), 12.4 ms avg latency, 209.0 max latency.

 records sent, 1996.6 records/sec (1.90 MB/sec), 2.8 ms avg latency, 108.0 max latency.

 records sent, 2035.8 records/sec (1.94 MB/sec), 0.9 ms avg latency, 76.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.8 ms avg latency, 87.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.7 ms avg latency, 65.0 max latency.

 records sent, 2000.2 records/sec (1.91 MB/sec), 1.2 ms avg latency, 84.0 max latency.

 records sent, 2000.6 records/sec (1.91 MB/sec), 0.8 ms avg latency, 79.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.7 ms avg latency, 59.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 66.0 max latency.

 records sent, 1999.8 records/sec (1.91 MB/sec), 0.8 ms avg latency, 88.0 max latency.

 records sent, 2000.6 records/sec (1.91 MB/sec), 0.5 ms avg latency, 66.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 66.0 max latency.

 records sent, 1965.0 records/sec (1.87 MB/sec), 0.8 ms avg latency, 90.0 max latency.

 records sent, 2022.4 records/sec (1.93 MB/sec), 1.0 ms avg latency, 91.0 max latency.

 records sent, 2014.0 records/sec (1.92 MB/sec), 5.6 ms avg latency, 211.0 max latency.

 records sent, 2000.8 records/sec (1.91 MB/sec), 0.9 ms avg latency, 81.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 85.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 2.3 ms avg latency, 124.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.7 ms avg latency, 84.0 max latency.

 records sent, 2001.6 records/sec (1.91 MB/sec), 1.0 ms avg latency, 108.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.4 ms avg latency, 6.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.6 ms avg latency, 84.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 65.0 max latency.

 records sent, 1998.8 records/sec (1.91 MB/sec), 1.3 ms avg latency, 98.0 max latency.

 records sent, 2066.8 records/sec (1.97 MB/sec), 3.5 ms avg latency, 163.0 max latency.

 records sent, 1999.6 records/sec (1.91 MB/sec), 0.7 ms avg latency, 59.0 max latency.

 records sent, 1981.3 records/sec (1.89 MB/sec), 0.6 ms avg latency, 61.0 max latency.

 records sent, 2019.2 records/sec (1.93 MB/sec), 0.6 ms avg latency, 66.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.6 ms avg latency, 91.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.6 ms avg latency, 65.0 max latency.

 records sent, 1896.9 records/sec (1.81 MB/sec), 2.0 ms avg latency, 266.0 max latency.

 records sent, 2106.4 records/sec (2.01 MB/sec), 8.0 ms avg latency, 266.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 0.5 ms avg latency, 36.0 max latency.

 records sent, 2000.0 records/sec (1.91 MB/sec), 1.1 ms avg latency, 103.0 max latency.

 records sent, 2002.8 records/sec (1.91 MB/sec), 1.2 ms avg latency, 92.0 max latency.

 records sent, 2000.4 records/sec (1.91 MB/sec), 0.8 ms avg latency, 68.0 max latency.

 records sent, 1999.836013 records/sec (1.91 MB/sec), 2061.17 ms avg latency, 22414.00 ms max latency,  ms 50th,  ms 95th,  ms 99th,  ms .9th.

消费100万

注意要增加参数 --consumer.config config/consumer.config

bin/kafka-consumer-perf-test.sh --consumer.config config/consumer.config --broker-list localhost: --topic test_perf --fetch-size  --messages  --threads

输出：

start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, nMsg.sec, rebalance.time.ms, fetch.time.ms, fetch.MB.sec, fetch.nMsg.sec

-- :::, -- :::, 953.6743, 18.5154, , 19414.8368, , , 18.7414, 19651.7706

五、结果对比

先来看普通版的写入100万结果

 records sent, 1999.548102 records/sec (1.91 MB/sec), 2456.76 ms avg latency, 23889.00 ms max latency,  ms 50th,  ms 95th,  ms 99th,  ms .9th.

再来看acl版的结果

 records sent, 1999.836013 records/sec (1.91 MB/sec), 2061.17 ms avg latency, 22414.00 ms max latency,  ms 50th,  ms 95th,  ms 99th,  ms .9th.

可以看出，消息延迟时间，一个平均是2456.76 ms，一个是2061.17 ms。结果相差不大！

kafka基本版与kafka acl版性能对比(单机版)的更多相关文章

  1. 《Apache kafka实战》读书笔记-管理Kafka集群安全之ACL篇

    <Apache kafka实战>读书笔记-管理Kafka集群安全之ACL篇 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 想必大家能看到这篇博客的小伙伴,估计你对kaf ...

  2. Kafka相关内容总结（存储和性能）

    Kafka消息的存储 Kafka的设计基于一种非常简单的指导思想:不是要在内存中保存尽可能多的数据,在需要时将这些数据刷新(flush)到文件系统,而是要做完全相反的事情.所有数据都要立即写入文件系统 ...

  3. 转 Kafka、RabbitMQ、RocketMQ等消息中间件的对比 —— 消息发送性能和优势

    Kafka.RabbitMQ.RocketMQ等消息中间件的对比 —— 消息发送性能和优势 引言 分布式系统中,我们广泛运用消息中间件进行系统间的数据交换,便于异步解耦.现在开源的消息中间件有很多,前 ...

  4. Kafka 集群配置SASL+ACL

    一.简介 在Kafka0.9版本之前,Kafka集群时没有安全机制的.Kafka Client应用可以通过连接Zookeeper地址,例如zk1:2181:zk2:2181,zk3:2181等.来获取 ...

  5. Kafka学习笔记之Kafka性能测试方法及Benchmark报告

    0x00 概述 本文主要介绍了如何利用Kafka自带的性能测试脚本及Kafka Manager测试Kafka的性能,以及如何使用Kafka Manager监控Kafka的工作状态,最后给出了Kafka ...

  6. Kafka权威指南 读书笔记之（三）Kafka 生产者一一向 Kafka 写入数据

    不管是把 Kafka 作为消息队列.消息总线还是数据存储平台来使用 ,总是需要有一个可以往 Kafka 写入数据的生产者和一个从 Kafka 读取数据的消费者,或者一个兼具两种角色的应用程序. 开发者 ...

  7. kafka 基础知识梳理-kafka是一种高吞吐量的分布式发布订阅消息系统

    一.kafka 简介 今社会各种应用系统诸如商业.社交.搜索.浏览等像信息工厂一样不断的生产出各种信息,在大数据时代,我们面临如下几个挑战: 如何收集这些巨大的信息 如何分析它 如何及时做到如上两点 ...

  8. kafka学习2：kafka集群安装与配置

    在前一篇:kafka学习1:kafka安装 中,我们安装了单机版的Kafka,而在实际应用中,不可能是单机版的应用,必定是以集群的方式出现.本篇介绍Kafka集群的安装过程: 一.准备工作 1.开通Z ...

  9. 【kafka学习之四】kafka集群性能测试

    kafka集群的性能受限于JVM参数.服务器的硬件配置以及kafka的配置,因此需要对所要部署kafka的机器进行性能测试,根据测试结果,找出符合业务需求的最佳配置. 1.kafka broker j ...

随机推荐

  1. 【贪心/Trie】【CF1083B】 The Fair Nut and Strings

    Description 有 \(k\) 个长度为 \(n\) 的只含 \(a\) 或 \(b\) 字符串,并不知道它们具体是多少,只知道它们的字典序不小于字符串 \(A\),同时不大于字符串 \(B\ ...

  2. MySQL基本了解与使用

    MySQL的相关概念介绍 MySQL 为关系型数据库(Relational Database Management System), 这种所谓的"关系型"可以理解为"表格 ...

  3. L. Twice Equation ACM Nanning 2017

    https://nanti.jisuanke.com/t/19978 acm提交:类 Main 使用java:高精度 BigInteger import java.math.BigInteger; i ...

  4. C#线程篇---你所不知道的线程池（4）

    线程的创建和销毁都要耗费大量的时间,有什么更好的办法?用线程池! 太多的线程浪费内存资源,有什么更好的办法?用线程池! 太多线程有损性能,有什么更好的办法?用线程池!(⊙_⊙)? 线程池是什么?继前三 ...

  5. 野指针(Wild pointer)和悬垂指针(dangling pointer)

    详细参考如下: Dangling pointer(悬垂指针.迷途指针)和 Wild pointer(野指针) 迷途指针经常出现在混杂使用malloc() 和 free() 库调用: 当指针指向的内存释 ...

  6. python基础-异常（exception）处理

    python基础-异常(exception)处理 作者:尹正杰 版权声明:原创作品,谢绝转载!否则将追究法律责任. 程序中难免出现错误,而错误分成两种,即语法错误和逻辑错误.语法错误根本过不了pyth ...

  7. VS2010 中更改项目名称

    Time.vcxproj修改项目名称,主要是通过以下几个步骤就能完成: 修改解决方案的名称. 修改解决项目名称. 修改项目的程序集名称和默认命名空间 替换整个项目或解决方案中的出现该名称的地方. 修改 ...

  8. Python之路,Day2 - Python基础，列表，循环

    1.列表练习name0 = 'wuchao'name1 = 'jinxin'name2 = 'xiaohu'name3 = 'sanpang'name4 = 'ligang' names = &quo ...

  9. bzoj千题计划167：bzoj3527: [Zjoi2014]力

    http://www.lydsy.com/JudgeOnline/problem.php?id=3527 给出n个数qi,给出Fj的定义如下: 令Ei=Fi/qi,求Ei.      以n=4为例: ...

  10. Qt_扫雷游戏实现

    源代码下载(详细注释): 详细代码(注释): https://github.com/xiaocangtian/GameMine 链接: http://pan.baidu.com/s/1gf9Ux5h ...