[转帖]Pepper-Box - Kafka Load Generator
https://github.com/GSLabDev/pepper-box
Pepper-Box is kafka load generator plugin for jmeter. It allows to send kafka messages of type plain text(JSON, XML, CSV or any other custom format) as well as java serialized objects.
Getting Started
Pepper-Box includes four main components
- PepperBoxKafkaSampler : This is jmeter java sampler sends messages to kafka.
- Pepper-Box PlainText Config : This jmeter config element generates plaintext messages based on input schema template designed.
- Pepper-Box Serialized Config : This jmeter config element generates serialized object messages based on input class and its property configurations.
- PepperBoxLoadGenerator : This is standalone utility which can be used without jmeter.
Setup
Requirement
Pepper-Box uses Java 8 with java compiler API, hence on JMeter machine JDK 8 should be installed instead of JRE 8.
Install openjdk on Debian, Ubuntu, etc.,
sudo apt-get install openjdk-8-jdk
Install openjdk on Fedora, Oracle Linux, Red Hat Enterprise Linux, etc.,
su -c "yum install java-1.8.0-openjdk-devel"
For windows you can download oracle JDK 8 setup from here
Build Project
mvn clean install -Djmeter.version=3.0 -Dkafka.version=0.9.0.1
JMeter and Kafka version can be passed dynamically.
Once build is completed, copy jar file to JMETER_HOME/lib/ext directory.
PepperBoxKafkaSampler
This is java sampler hence in ThreadGroup add sampler as Java Request and select class as com.gslab.pepper.sampler.PepperBoxKafkaSampler
As you can see in above screen, you can configure producer properties and topic details.
- bootstrap.servers : broker-ip-1:port, broker-ip-2:port, broker-ip-3:port
- zookeeper.servers : zookeeper-ip-1:port, zookeeper-ip-2:port, zookeeper-ip-3:port. Note : Any one of bootstrap or zookeeper server detail is enough. if zookeeper servers are given then bootstrap.servers are retrieved dynamically from zookeeper servers.
- kafka.topic.name : Topic on which messages will be sent
- key.serializer : Key serializer (This is optional and can be kept as it is as we are not sending keyed messages).
- value.serializer : For plaintext config element value can be kept same as default but for serialized config element, value serializer can be "com.gslab.pepper.input.serialized.ObjectSerializer"
- compression.type : kafka producer compression type(none/gzip/snappy/lz4)
- batch.size : messages batch size(increased batch size with compression like lz4 gives better throughput)
- linger.ms : How much maximum time producer should wait till batch becomes full(should be 5-10 when increased batch size and compression is enabled)
- buffer.memory : Total buffer memory for producer.
- acks : Message sent acknowledgement, value can be (0/1/-1).
- send.buffer.bytes : The size of the TCP send buffer (SO_SNDBUF) to use when sending data. If the value is -1, the OS default will be used.
- receive.buffer.bytes : The size of the TCP receive buffer (SO_RCVBUF) to use when reading data. If the value is -1, the OS default will be used.
- security.protocol : kafka producer protocol. Valid values are: PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL.
- message.placeholder.key : Config element message variable name. This name should be same as message placeholder key in serialized/plaintext config element.
- kerberos.auth.enabled : YES/NO if it is disabled all below properties will be ignored
- java.security.auth.login.config : jaas.conf of kafka Kerberos
- java.security.krb5.conf : Kerberos server krb5.conf file
- sasl.kerberos.service.name : Kafka Kerberos service name
Above properties are added by default in sampler as those are more significant in terms of performance in most of the cases. But you can add other non listed kafka properties with prefix "_".
For example to enable SSL properties you can add below properties
_ssl.key.password
_ssl.keystore.location
_ssl.keystore.password
_ssl.keystore.type
_ssl.truststore.location
_ssl.truststore.password
_ssl.truststore.type
Note: These are just sample properties, SSL properties are already included in kafka sampler.
Pepper-Box PlainText Config
Pepper-Box PlainText Config is jmeter config element. It takes schema template is input and generates message for each sampler request.
You can add this config element using Thread group --> Add --> Config Element --> Pepper-Box PlainText Config
Input schema template can be in any format
JSON schema template
{
"messageId":{{SEQUENCE("messageId", 1, 1)}},
"messageBody":"{{RANDOM_ALPHA_NUMERIC("abcedefghijklmnopqrwxyzABCDEFGHIJKLMNOPQRWXYZ", 100)}}",
"messageCategory":"{{RANDOM_STRING("Finance", "Insurance", "Healthcare", "Shares")}}",
"messageStatus":"{{RANDOM_STRING("Accepted","Pending","Processing","Rejected")}}",
"messageTime":{{TIMESTAMP()}}
}
XML schema template
<message>
<messageId>{{SEQUENCE("messageId", 1, 1)}}</messageId>
<messageBody>{{RANDOM_ALPHA_NUMERIC("abcedefghijklmnopqrwxyzABCDEFGHIJKLMNOPQRWXYZ", 100)}}</messageBody>
<messageCategory>{{RANDOM_STRING("Finance", "Insurance", "Healthcare", "Shares")}}</messageCategory>
<messageStatus>{{RANDOM_STRING("Accepted","Pending","Processing","Rejected")}}</messageStatus>
<messageTime>{{TIMESTAMP()}}</messageTime>
</message>
Custom schema template
Hello {{FIRST_NAME()}}
This is sample message sending at {{DATE("dd/MM/yyyy HH:mm:ss")}}.
Thanks and Regards,
{{FIRST_NAME()}} {{LAST_NAME()}}
Pepper-Box Serialized Config
Java serialized objects can be sent to kafka using Pepper-Box Serialized Config Element. This config element can be added using Thread group --> Add --> Config Element --> Pepper-Box Serialized Config
Follow below steps to use this config element,
- Enter fully qualified name in
class name
section (e.g. com.gslab.pepper.Message in above screen). This class should be present in jmeter classpath folder(lib or lib/ext). You can copy jar containing required class to JMETER_HOME/lib/ext folder. - Click on load button which will populate all fields of given class with default values as
Ignore
means field value will not set. - Assign function expression to each field.
Example Class,
package com.gslab.pepper;
import java.io.Serializable;
public class Message implements Serializable{ private long messageId;
private String messageBody;
private String messageStatus;
private String messageCategory;
private long messageTime; public long getMessageId() {
return messageId;
} public void setMessageId(long messageId) {
this.messageId = messageId;
} public String getMessageBody() {
return messageBody;
} public void setMessageBody(String messageBody) {
this.messageBody = messageBody;
} public String getMessageStatus() {
return messageStatus;
} public void setMessageStatus(String messageStatus) {
this.messageStatus = messageStatus;
} public String getMessageCategory() {
return messageCategory;
} public void setMessageCategory(String messageCategory) {
this.messageCategory = messageCategory;
} public long getMessageTime() {
return messageTime;
} public void setMessageTime(long messageTime) {
this.messageTime = messageTime;
}
}
Please make sure that function return type and field data type should be compatible with each other.
PepperBoxLoadGenerator
PepperBoxLoadGenerator is console plaintext load generation utility.
Command,
java -cp pepper-box-1.0.jar com.gslab.pepper.PepperBoxLoadGenerator --schema-file <schema file absolute path> --producer-config-file <producer properties absoulte path> --throughput-per-producer <throughput rate per producer> --test-duration <test duration in seconds> --num-producers <number of producers>
Example
- Schema file
{
"messageId":{{SEQUENCE("messageId", 1, 1)}},
"messageBody":"{{RANDOM_ALPHA_NUMERIC("abcedefghijklmnopqrwxyzABCDEFGHIJKLMNOPQRWXYZ", 100)}}",
"messageCategory":"{{RANDOM_STRING("Finance", "Insurance", "Healthcare", "Shares")}}",
"messageStatus":"{{RANDOM_STRING("Accepted","Pending","Processing","Rejected")}}",
"messageTime":{{TIMESTAMP()}}
}
- producer properties file
bootstrap.servers=<Broker List>
zookeeper.servers=<Zookeeper List>
kafka.topic.name=<kafka topic>
key.serializer=org.apache.kafka.common.serialization.StringSerializer
value.serializer=org.apache.kafka.common.serialization.StringSerializer
acks=0
send.buffer.bytes=131072
receive.buffer.bytes=32768
batch.size=16384
linger.ms=0
buffer.memory=33554432
compression.type=none
security.protocol=PLAINTEXT
kerberos.auth.enabled=NO
java.security.auth.login.config=<JAAS File Location>
java.security.krb5.conf=<krb5.conf location>
sasl.kerberos.service.name=<Kerberos service name>
For schema file and producer properties file most of the features same as jmeter plain text config element.
We have also included pepper_box.jmx
jmeter sample test file which can be directly imported in jmeter.
Schema Template Functions
Pepper-Box provides various template functions for random data generation,
Function | Details | Example(For serialized use without {{ }} ) |
Returns |
---|---|---|---|
TIMESTAMP() | current time in long | {{TIMESTAMP()}} |
Long |
TIMESTAMP(startDate, endDate) | Random long date between two Dates | {{TIMESTAMP("01-05-1998 10:30:12","03-03-2017 12:12:12")}} |
Long |
DATE(format) | current date with given format | {{DATE("dd-MM-yyyy HH:mm:ss")}} |
String |
RANDOM_STRING(string1, string2, string3,...) | Random string among given | {{RANDOM_STRING("ONE","TWO","THREE","FOUR")}} |
String |
RANDOM_INT(int1, int2, int3,...) | Random integer among given | {{RANDOM_INT(1, 2, 3, 4)}} |
Integer |
RANDOM_FLOAT(float1, float2, float3,...) | Random float among given | {{RANDOM_FLOAT(1.1F ,2.1F, 3.1F, 4.1F)}} |
Float |
RANDOM_DOUBLE(double1, double2, double3,...) | Random double among given | {{RANDOM_DOUBLE(1.1, 2.1, 3.1, 4.1)}} |
Double |
RANDOM_LONG(long1, long2, long3,...) | Random long among given | {{RANDOM_LONG(1, 2, 3, 4)}} |
Long |
RANDOM_INT_RANGE(min, max) | Random integer among given | {{RANDOM_INT_RANGE(1,100)}} |
Integer |
RANDOM_FLOAT_RANGE(min, max) | Random float between min and max | {{RANDOM_FLOAT_RANGE(1.0F, 100.0F)}} |
Float |
RANDOM_FLOAT_RANGE(min, max) | Random double between min and max | {{RANDOM_FLOAT_RANGE(1.0, 100.0)}} |
Double |
RANDOM_LONG_RANGE(min, max) | Random long between min and max | {{RANDOM_LONG_RANGE(1,100)}} |
Long |
FIRST_NAME() | Random first name | {{FIRST_NAME()}} |
String |
LAST_NAME() | Random last name | {{LAST_NAME()}} |
String |
RANDOM_ALPHA_NUMERIC(charSet, length) | Random string of given length from given char set | {{RANDOM_ALPHA_NUMERIC("abcdefghijklmn", 10)}} |
String |
UUID() | Random UUID | {{UUID()}} |
String |
SEQUENCE(sequenceId, startValue, incrementBy) | Generates incremental sequence | {{SEQUENCE("messageId", 1, 1)}} |
Long |
PHONE() | Random 10 digit phone number | {{PHONE()}} |
String |
GENDER() | Random gender | {{GENDER()}} |
String |
BOOLEAN() | Random boolean | {{BOOLEAN()}} |
boolean |
EMAIL(domain) | Random email id for given domain | {{EMAIL("test.com")}} |
String |
USERNAME() | Random username | {{USERNAME()}} |
String |
IPV4() | Random IPV4 address | {{IPV4()}} |
String |
IPV6() | Random IPV6 address | {{IPV6()}} |
String |
Custom Functions
Apart from these functions, you can also add your own custom function in com.gslab.pepper.input.CustomFunctions
class. Please make sure that those are static functions.
Example
public static float AVG(float... floats){
int count = floats.length;
float sum = 0.0;
for (float number : floats){
sum += number;
}
return sum/count;
}
AVG function can be used in schema as shown below,
{{AVG(32.2, 34.5, 64.2)}}
Note: While writing custom functions, please try to keep data in memory or scale your function as much other functions otherwise your custom function itself becomes performance bottlneck. e.g. you need some record ids from RDBMS for some schema fields, instead of querying every time bring all ids inmemory and get random id from those ids.
You can also add manipulations on template functions, for example TIMESTAMP() function returns time in milliseconds but you can get time in seconds,
{{java.util.concurrent.TimeUnit.MILLISECONDS.toSeconds(TIMESTAMP())}}
Special Thanks!
We would like to special thanks to kafkameterand wrtting custom jmeter plugin blogpost which helped to understand writing custom plugins for JMeter.
We also like to thanks to InMemoryJavaCompiler which helped to understand in memory code compilation.
[转帖]Pepper-Box - Kafka Load Generator的更多相关文章
- win7下安装load generator
win7下只安装loadrunner load generator 1.点击setup.exe 2.选择“loadrunner 完整安装程序” 3.安装完成所需要的插件,然后重启 4.重新打开安装 ...
- 在 Linux 系统中安装Load Generator ,并在windows 调用方法
在 Linux 系统中安装Load Generator ,并在windows 调用 由于公司需要测试系统的最大用户承受能力,所以需要学习使用loadrunner.在安装的时候碰到了不少问题,所以写下此 ...
- loadrunner controller:设置多个load generator
下面讲一下如何使用多台电脑进行负载测试. 1) 打开load generator,如图所示默认已添加了我们本地的Generator: 2) 点击"Add. ...
- Load generator连接失败的解决办法!(转)
环境:1.loadrunner control 一台物理机(win2008r2) 2.loadrunner agent 两台物理机(win2008r2) 问题:loadrunner control 连 ...
- Linux下安装load generator步骤及问题解决
Linux下安装load generator步骤及问题解决 上一篇 / 下一篇 2014-08-06 18:33:00 / 个人分类:loadrunner相关 查看( 146 ) / 评论( 0 ) ...
- 在 Linux 系统中安装Load Generator ,并在windows 调用
原文地址:http://www.blogjava.net/qileilove/archive/2012/03/14/371861.html 由于公司需要测试系统的最大用户承受能力,所以需要学习使用lo ...
- 在liunx系统安装负载生成器(Load Generator)
#初始化系统 yum -y update yum -y install ntpdate ntpdate asia.pool.ntp.org yum -y install wget make cmake ...
- loadrunner load generator设置init人数
Load Generator中还有一个很重要的设置.就是用来设置init人数的,我们在运行脚本的时候会发现,在场景监控中,init默认不会超过50个人,也就是最大并发是50个人,我们想使用超过50个人 ...
- LoadRunner 2020 社区版本负载机(Load generator)Linux 安装教程
1.HP官方注册 下载 LoadRunner_2020_Edition_Standalone_Applications_Micro_Focus_LoadRunner_2020_Community_Ed ...
- [转帖]Docker save and load镜像保存
Docker save and load镜像保存 https://www.cnblogs.com/zhuochong/p/10064350.html docker save 和 load 以及 imp ...
随机推荐
- 0X01 位运算笔记
位运算,经常可以用来处理一些数学或动归方面的问题,通常会在数据范围较小的情况下使用. 为方便起见,一个 \(\mathrm{n}\) 位二进制数从右到左分别为第 \(\mathrm{0 \sim n ...
- POJ 2609 DP 双队列DP
POJ 2609 DP 双队列DP 题意 给出队列容量与一组数,并且我们有两个这样容量的空队列,要求我们按照次序把数装进两个队列中,输出最多能装多少以及每个数装进哪个队列. 思路 由于按照次序来装,不 ...
- linux中redis下载安装部署启动
下载安装部署 创建一个存放Redis的文件夹,下载安装包 mkdir redis 进入redis文件 cd redis 下载redis安装包 wget http://download.redis.io ...
- 一文掌握Ascend C孪生调试
本文分享自华为云社区<一文掌握Ascend C孪生调试>,作者:昇腾CANN. 1 What,什么是孪生调试 Ascend C提供孪生调试方法,即CPU域模拟NPU域的行为,相同的算子代码 ...
- 详解CCE服务:一站式告警配置和云原生日志视图
本文分享自华为云社区<新一代云原生可观测平台之CCE服务日志和告警篇>,作者:云容器大未来. 告警和日志是运维人员快速定位问题.恢复异常的主要手段.运维人员日常的工作模式往往是先接收告警信 ...
- 手把手教你写一个spring IOC容器
摘要:spring框架的基础核心和起点毫无疑问就是IOC,IOC作为spring容器提供的核心技术,成功完成了依赖的反转:从主类的对依赖的主动管理反转为了spring容器对依赖的全局控制.今天就带大家 ...
- Vue组件间的传值五大场景,你造吗?
摘要:组件是 vue.js最强大的功能之一,这五个组件间传值场景你了解吗? 本文分享自华为云社区<你了解Vue组件间传值五大场景吗?>,作者:北极光之夜. . 父组件向子组件传值: 比如有 ...
- Go 1.18 新特性:多模块工作区模式
摘要:在 Go 1.18 推出多模块工作区模式--Multi-Module Workspaces,用以支持模块的多个工作空间,我们来看看到底有什么特别. 本文分享自华为云社区<一起看看 Go 1 ...
- 鸿蒙轻内核源码分析:Newlib C
摘要:本文介绍了LiteOS-M内核Newlib C的实现,特别是文件系统和内存分配释放部分,最后介绍了Newlib钩子函数. 本文分享自华为云社区<鸿蒙轻内核M核源码分析系列二十 Newlib ...
- 破解数据匮乏现状:纵向联邦学习场景下的逻辑回归(LR)
摘要:主要介绍了华为云可信智能计算服务(TICS)采用的纵向联邦逻辑回归(LR)方案. 本文分享自华为云社区<纵向联邦学习场景下的逻辑回归(LR)>,作者: 汽水要加冰. 海量训练数据是人 ...