基于Java的支持可变QPS的http负载生成器，提供交互界面和RMI接口
Load generator
The load generator is a Java maven project which is implemented using httpclient+threadpool that works in open-loop [1], it has a web GUI for the realtime latency watch (e.g., 99th tail-latency, QPS, RPS, etc.). Meanwhile, the load generator provides RMI interface for external call and supports dynamically changeable request loads
- support thousands level concurrent request per second in single node
- support web GUI for real-time latency watch
- support dynamiclly changeable QPS in open-loop
- support RMI interface for external call
Code architecture
/LoadGen
|----/build/sdcloud.war # the executable war package
|----/src/main
|----/java/scs/
| |----/controller/* # MVC controller layer
| |----/pojo/* # entity bean layer
| +----/util
| |----/format/* # format time and data
| |----/loadGen
| | |----/loadDriver/* # generate request loads
| | |----/recordDriver/* # record request metrics
| | +----/strategy/*
| |----/respository/* # in-memory data storage
| |----/rmi/* # RMI service and interfaces
| +----/tools/* # some tools
|----/resources/* # configuration files
+----/webapp/* # GUI pages
Hardware environment
In our experiment environment, the configuration of nodes are shown as below:
| hostname | description | IP | role |
|---|---|---|---|
| tank-node1 | where the load generator is deployed | 192.168.3.110 | k8s master |
| tank-node3 | where the inference service is deployed | 192.168.3.130 | k8s slave |
Build load generator
The load generator is writen in Java, it can be deployed in container or host machine, and we need install Java JDK and apache tomcat before using it
Step 1: Install Java JDK
Download java jdk and install to /usr/local/java/
$ wget https://download.oracle.com/otn/java/jdk/8u231-b11/5b13a193868b4bf28bcb45c792fce896/jdk-8u231-linux-x64.tar.gz
$ tar -zxvf jdk-8u231-linux-x64.tar.gz /usr/local/java/
Modify the /etc/profile file
$ vi /erc/profile
Config Java environment variables, append the following content into the file
export JAVA_HOME=/usr/local/java/jdk1.8.0_231
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=$PATH:${JAVA_HOME}/bin
Enable the configuration
$ source /etc/profile
$ java -version
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b12, mixed mode)
Step 2: Install apache tomcat
Download apache tomcat and install to /usr/local/tomcat/
$ http://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.47/bin/apache-tomcat-8.5.47.tar.gz
$ tar -zxvf apache-tomcat-8.5.47.tar.gz /usr/local/tomcat/
Step 3: Deploy the load generator into tomcat
An executable war package has been provided in loadGen/build/, you can also build the source code uses
jaroreclipse IDE
Deploy the web package into tomcat webapp/
$ mv loadGen/build/sdcloud.war /usr/local/tomcat/apache-tomcat-8.5.47/webapp
$ /usr/local/tomcat/apache-tomcat-8.5.47/bin/startup.sh
Validate if the depolyment is successful
$ curl http://localhost:8080/sdcloud/
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
...
welcome to the load generator page!
...
</body>
The command will output the content of a welcome page, means the war package has been deployed successfully
Then modify the configuration file in loadGen and restart tomcat
$ vi LoadGen/src/main/resources/conf/sys.properties
Modify the content of sys.properties as below:
# URL of web inference service that can be accessed by http
imageClassifyBaseURL=http://192.168.3.130:31500/gpu
# node IP that deployed load generator
serverIp=192.168.3.110
# the port of RMI service provided by load generator
rmiPort=22222
# window size of latency recorder, which can be seen from GUI page
windowSize=60
# record interval of latency, default: 1000ms
recordInterval=1000
Then restart the tomcat
$ /usr/local/tomcat/apache-tomcat-8.5.47/bin/shutdown.sh
$ /usr/local/tomcat/apache-tomcat-8.5.47/bin/startup.sh
If error occured when restart like java.rmi.server.ExportException: Port already in use: 22222, we need to kill the process that uses this port and restart tomcat
$ netstat -apn |grep 22222
tcp6 1 0 202.113.8.12:36284 127.0.0.1:22222 USING 35487/java
$ kill 35487
$ /usr/local/tomcat/apache-tomcat-8.5.47/bin/startup.sh
Step 4: Test the load generator
Open the exlporer and visit url http://192.168.3.110:8080/sdcloud/
The GUI page is shown as below:
Type 1: Using the URL interfaces
We provide four URL interfaces to control the load generator as below:
| url interface | description | type | parameter |
|---|---|---|---|
startOnlineQuery.do?intensity=1&serviceId=0 |
start to generate the request load | GET | intensity: concurrent requests per second (RPS) serviceId: inference service index id |
goOnlineQuery.do?serviceId=0 |
visit the real-time latency page | GET | serviceId: inference service index id |
setIntensity.do?intensity=20&serviceId=0 |
change the RPS dynamically | GET | intensity: RPS serviceId: inference service index id |
stopOnlineQuery.do?serviceId=0 |
stop the load generator | GET | --- |
The index id of mobilenet-coco300-tf is 0, and the second inference service should be set to 1, ..., etc. The supported service in load generator is easy to scale
For examle, firstly, click the startOnlineQuery.do?intensity=1&serviceId=0 link to generate request loads, and you will see the page in a waiting state (circle loading), then after $windowSize seconds, click the goOnlineQuery.do?serviceId=0 link to watch the real-time latency as below.
Metrics in real-time latency watch page:
- realRPS: The concurrent requests number of last second from client, RPS (request per second)
- realQPS: The response number of last second in server, QPS (query per second)
- AvgRPS: The average concurrent RPS in
$windowsizetime scale - AvgQPS: The average QPS in
$windowsizetime scale - SR: The average service rate in
$windowsizetime scale,SR=AvgQPS/AvgRPS*100% - queryTime: The 99th tail-latency of the concurrent requests per second
- Avg99th: The average
queryTimesin$windowsizetime scale
When the load generator is running, click the setIntensity.do?intensity=N&serviceId=0 link to change the RPS, please replace N to the number of concurrent requests per second you want. Finally, click stopOnlineQuery.do?serviceId=0 to stop the load testing
Type 2: Using the RMI interfaces
We also provide the Java RMI interfaces for the remote funciton calls in users' external code without clicking URL links. Using RMI, you can control the load generator and collect metric data. The RMI interface file is in LoadGen/src/main/java/scs/util/rmi/LoadInterface.java, the interface functions are shown as below:
package scs.util.rmi;
import java.rmi.Remote;
import java.rmi.RemoteException;
/**
* RMI interface class, which is used to control the load generator
* The functions can be call by remote client code
* @author Yanan Yang
* @date 2019-11-11
* @address TANKLab, TianJin University, China
*/
public interface LoadInterface extends Remote{
public float getWindowAvgPerSec99thLatency(int serviceId) throws RemoteException; //return the value of Avg99th
public float getRealPerSec99thLatency(int serviceId) throws RemoteException; //return the value of queryTime
//public float getWindowSize95thRealLatency(int serviceId) throws RemoteException; //return the value of queryTime (95th), unused
//public float getLcCurLatency999thRealLatency(int serviceId) throws RemoteException; //return the value of queryTime (99.9th), unused
public int getRealQueryIntensity(int serviceId) throws RemoteException; //return the value of realQPS
public int getRealRequestIntensity(int serviceId) throws RemoteException; //return the value of realRPS
public float getWindowAvgServiceRate(int serviceId) throws RemoteException; //return the value of SR
public void execStartHttpLoader(int serviceId) throws RemoteException; //start load generator for serviceId
public void execStopHttpLoader(int serviceId) throws RemoteException; //stop load generator for serviceId
public int setIntensity(int intensity,int serviceId) throws RemoteException; //change the RPS dynamically
}
When tomcat starts, the server side will automatically setup the RMI service using serverIp and rmiPort in LoadGen/src/main/resources/conf/sys.properties, the RMI service function is shown as below:
package scs.util.rmi;
import java.net.MalformedURLException;
import java.rmi.Naming;
import java.rmi.RemoteException;
import java.rmi.registry.LocateRegistry;
public class RmiService {
private static RmiService loadService=null;
private RmiService(){}
public synchronized static RmiService getInstance() {
if (loadService == null) {
loadService = new RmiService();
}
return loadService;
}
public void setupService(String serverIp,int rmiPort) {
try {
System.setProperty("java.rmi.server.hostname",serverIp);
LocateRegistry.createRegistry(rmiPort);
LoadInterface load = new LoadInterfaceImpl();
Naming.rebind("rmi://"+serverIp+":"+rmiPort+"/load", load);
} catch (RemoteException e) {
e.printStackTrace();
} catch (MalformedURLException e) {}
}
}
The client side need to setup the RMI connection before controling the load generator, a stand connection function is shown as below:
private static void setupRmiConnection(){
try {
LoadInterface loader=(LoadInterface) Naming.lookup("rmi://192.168.3.110:22222/load");
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (RemoteException e) {
e.printStackTrace();
} catch (NotBoundException e) {
e.printStackTrace();
}
if(loader!=null){
System.out.println(Repository.loaderRmiUrl +"connection successed");
}else{
System.out.println(Repository.loaderRmiUrl +"connection failed");
}
}
More tutorial of RMI interface can be see from here
Performance evaluation of loadGen
We evaluate the performance of load generator tool and web inference service and show the results as below:
Performance testing of load generator
To aviod the performance bottleneck of web service interfering the testing results, we set the request url in LoadGen/src/main/resources/conf/sys.properties to an empty url (this url does nothing and just returns 'helloworld')
mageClassifyBaseURL=http://192.168.3.130:31500/helloworld
Then we test the concurrent ability of load generator and collect the latency data, the client and server are deployed individually on two nodes that connected with 1Gbps WLAN
Fig.1 depicts the 99th tail-latency collected by load generator with the workloads ranges from RPS=1 to RPS=2000, the requests are sent using multi-threads in open-loop, the worst 99th tail-latency < 250ms when the RPS=2000, which shows the low queue latency in load generator. Fig.2 shows the 99th tail-latency increases linearly with the RPS, this demonstrates the load generator is well designed and has a good performance of workload scalability. Fig.3 shows the CPU usage in server end with increasing RPS, which has a same trend with the tail-latency in Fig.1. The inference service consumes < 0.5 CPU core when RPS=400, while the CPU usage no more than 2 CPU cores when RPS=2000, it demonstrates the low overhead of guicorn+flask framework
Future work
The latest released version of load generator has satisfied our experiment needs, in the future, we plan to implement these functions as below:
- Distributed load generator
- Diversified output statistics (e.g., PDF, hist graph)
Bug report & Question
We have used the load generator for a long time in our work [2,3], and fixed many bugs that have been found. If you have some new findings, please contact us via Email: ynyang@tju.edu.cn
Reference
[1] Kasture H, Sanchez D. Tailbench: a benchmark suite and evaluation methodology for latency-critical applications[C]//2016 IEEE International Symposium on Workload Characterization (IISWC). IEEE, 2016: 1-10.
[2] Y. Yang, L. Zhao, Z. Li, L. Nie, P. Chen and K. Li. ElaX: Provisioning Resource Elastically for Containerized Online Cloud Services[C]//2019 IEEE 21st International Conference on High Performance Computing and Communications (HPCC). IEEE, 2019: 1987-1994.
[3] L. Zhao, Y. Yang, K. Zhang, etc. Rhythm: Component-distinguishable Workload
Deployment in Datacenters[C]//EuroSys 2020. ACM. Under review
基于Java的支持可变QPS的http负载生成器，提供交互界面和RMI接口的更多相关文章
- 【公开课】【阿里在线技术峰会】魏鹏：基于Java容器的多应用部署技术实践
对于公开课,可能目前用不上这些,但是往往能在以后想解决方案的时候帮助到我.以下是阿里对公开课的整理 摘要: 在首届阿里巴巴在线峰会上,阿里巴巴中间件技术部专家魏鹏为大家带来了题为<基于Java容 ...
- Spring Boot 2.2 正式发布，大幅性能提升 + Java 13 支持
之前 Spring Boot 2.2没能按时发布,是由于 Spring Framework 5.2 的发布受阻而推迟.这次随着 Spring Framework 5.2.0 成功发布之后,Spring ...
- 9个基于Java的搜索引擎框架
在这个信息相当繁杂的互联网时代,我们已经学会了如何利用搜索引擎这个强大的利器来找寻目标信息,比如你会在Google上搜索情人节如何讨女朋友欢心,你也会在百度上寻找正规的整容医疗机构(尽管有很大一部分广 ...
- 基于java平台的常用资源整理
这里整理了基于java平台的常用资源 翻译 from :akullpp | awesome-java 大家一起学习,共同进步. 如果大家觉得有用,就mark一下,赞一下,或评论一下,让更多的人知道.t ...
- 基于Java Netty框架构建高性能的部标808协议的GPS服务器
使用Java语言开发一个高质量和高性能的jt808 协议的GPS通信服务器,并不是一件简单容易的事情,开发出来一段程序和能够承受数十万台车载接入是两码事,除去开发部标808协议的固有复杂性和几个月长周 ...
- 基于Java Mina框架的部标808服务器设计和开发
在开发部标GPS平台中,部标808GPS服务器是系统的核心关键,决定了部标平台的稳定性和行那个.Linux服务器是首选,为了跨平台,开发语言选择Java自不待言. 我们为客户开发的部标服务器基于Min ...
- 这里整理了基于java平台的常用资源
这里整理了基于java平台的常用资源 翻译 from :akullpp | awesome-java 大家一起学习,共同进步. 如果大家觉得有用,就mark一下,赞一下,或评论一下,让更多的人知道.t ...
- memcached学习——常用命令+基于java客户端的3种简单实现（二）
常用命令: memcached设计的原则就是简单,所以支持的命令也不是特别多~ 1.查看memcached的状态,主要用于分析内存的使用状况.优化内存分配等 stats 查看memcached的运行状 ...
- 基于Java图片数据库Neo4j 3.0.0发布 全新的内部架构
基于Java图片数据库Neo4j 3.0.0发布 全新的内部架构 Neo4j 3.0.0 正式发布,这是 Neo4j 3.0 系列的第一个版本.此版本对内部架构进行了全新的设计;提供给开发者更强大的生 ...
随机推荐
- 安裝TA-Lib到想要罵髒話
收集下載股票交易指數的歷史數據,並計算主要的幾個指標來進行技術分析. 查找網上的資料,發現大家都在用TA-Lib來計算指標,於是在下載並匯入了數據後開始安裝這個庫. pip install TA-Li ...
- [LeetCode] 190. Reverse Bits 翻转二进制位
Reverse bits of a given 32 bits unsigned integer. For example, given input 43261596 (represented in ...
- 【miscellaneous】Winserver2012安装后无法进入桌面
原因分析: 自己操作卸载了.net framework,系统没有了图形界面(由完整模式Full变为了核心模式core),需要重新恢复.net framework4.5. 解决方法分析: 需要将核心模式 ...
- radio和checkbox的js勾选使用
Html: <table> <tr><th class="w1">党内职务</th><td colspan="3&q ...
- Response的应用
1．HttpServletResponse概述 service方法中的response的类型是ServletResponse,而doGet/doPost方法的response的类型是HttpServl ...
- Java开发笔记（一百三十八）JavaFX的箱子
前面介绍了JavaFX标签控件的用法,其中提到Label文本支持中文字体,那么它到底支持哪些中文字体呢?自然要看当前的操作系统都安装了哪些字体才行,对于中文的Windows系统,默认安装了黑体“Sim ...
- 函数的学习2——返回值&传递列表——参考Python编程从入门到实践
返回值 函数并非总是直接显示输出,相反,它可以处理一些数据,并返回一个或一组值.函数的返回值被称为返回值. 1. 简单的返回值 def get_formatted_name(first_name, l ...
- 初学者用pycharm创建一个django项目和一个app时需要注意的事项
如何新建一个djiango项目: 1.在pycharm中点击File,选择new project,点击djiango,在右面的Location中将untitile改为你的项目名,其余部分注意见下图: ...
- 44 容器（三）——ArrayList索引相关方法
方法都比较简单,这里列出来即可: add(index,ele) //忘制定下标插入元素 add(ele) addAll(Collection <C> c) 泛型必须与调用add的泛型保持一 ...
- MySQL练手小试题
创建表和数据 创建class表 create table class ( cid int(11) primary key auto_increment, caption varchar(32) not ...