基于Java的支持可变QPS的http负载生成器，提供交互界面和RMI接口
Load generator
The load generator is a Java maven project which is implemented using httpclient+threadpool that works in open-loop [1], it has a web GUI for the realtime latency watch (e.g., 99th tail-latency, QPS, RPS, etc.). Meanwhile, the load generator provides RMI interface for external call and supports dynamically changeable request loads
- support thousands level concurrent request per second in single node
- support web GUI for real-time latency watch
- support dynamiclly changeable QPS in open-loop
- support RMI interface for external call
Code architecture
/LoadGen
|----/build/sdcloud.war # the executable war package
|----/src/main
|----/java/scs/
| |----/controller/* # MVC controller layer
| |----/pojo/* # entity bean layer
| +----/util
| |----/format/* # format time and data
| |----/loadGen
| | |----/loadDriver/* # generate request loads
| | |----/recordDriver/* # record request metrics
| | +----/strategy/*
| |----/respository/* # in-memory data storage
| |----/rmi/* # RMI service and interfaces
| +----/tools/* # some tools
|----/resources/* # configuration files
+----/webapp/* # GUI pages
Hardware environment
In our experiment environment, the configuration of nodes are shown as below:
| hostname | description | IP | role |
|---|---|---|---|
| tank-node1 | where the load generator is deployed | 192.168.3.110 | k8s master |
| tank-node3 | where the inference service is deployed | 192.168.3.130 | k8s slave |
Build load generator
The load generator is writen in Java, it can be deployed in container or host machine, and we need install Java JDK and apache tomcat before using it
Step 1: Install Java JDK
Download java jdk and install to /usr/local/java/
$ wget https://download.oracle.com/otn/java/jdk/8u231-b11/5b13a193868b4bf28bcb45c792fce896/jdk-8u231-linux-x64.tar.gz
$ tar -zxvf jdk-8u231-linux-x64.tar.gz /usr/local/java/
Modify the /etc/profile file
$ vi /erc/profile
Config Java environment variables, append the following content into the file
export JAVA_HOME=/usr/local/java/jdk1.8.0_231
export JRE_HOME=${JAVA_HOME}/jre
export CLASSPATH=.:${JAVA_HOME}/lib:${JRE_HOME}/lib
export PATH=$PATH:${JAVA_HOME}/bin
Enable the configuration
$ source /etc/profile
$ java -version
java version "1.8.0_231"
Java(TM) SE Runtime Environment (build 1.8.0_231-b12)
Java HotSpot(TM) 64-Bit Server VM (build 25.231-b12, mixed mode)
Step 2: Install apache tomcat
Download apache tomcat and install to /usr/local/tomcat/
$ http://mirrors.tuna.tsinghua.edu.cn/apache/tomcat/tomcat-8/v8.5.47/bin/apache-tomcat-8.5.47.tar.gz
$ tar -zxvf apache-tomcat-8.5.47.tar.gz /usr/local/tomcat/
Step 3: Deploy the load generator into tomcat
An executable war package has been provided in loadGen/build/, you can also build the source code uses
jaroreclipse IDE
Deploy the web package into tomcat webapp/
$ mv loadGen/build/sdcloud.war /usr/local/tomcat/apache-tomcat-8.5.47/webapp
$ /usr/local/tomcat/apache-tomcat-8.5.47/bin/startup.sh
Validate if the depolyment is successful
$ curl http://localhost:8080/sdcloud/
<!DOCTYPE html PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN" "http://www.w3.org/TR/html4/loose.dtd">
<html>
...
welcome to the load generator page!
...
</body>
The command will output the content of a welcome page, means the war package has been deployed successfully
Then modify the configuration file in loadGen and restart tomcat
$ vi LoadGen/src/main/resources/conf/sys.properties
Modify the content of sys.properties as below:
# URL of web inference service that can be accessed by http
imageClassifyBaseURL=http://192.168.3.130:31500/gpu
# node IP that deployed load generator
serverIp=192.168.3.110
# the port of RMI service provided by load generator
rmiPort=22222
# window size of latency recorder, which can be seen from GUI page
windowSize=60
# record interval of latency, default: 1000ms
recordInterval=1000
Then restart the tomcat
$ /usr/local/tomcat/apache-tomcat-8.5.47/bin/shutdown.sh
$ /usr/local/tomcat/apache-tomcat-8.5.47/bin/startup.sh
If error occured when restart like java.rmi.server.ExportException: Port already in use: 22222, we need to kill the process that uses this port and restart tomcat
$ netstat -apn |grep 22222
tcp6 1 0 202.113.8.12:36284 127.0.0.1:22222 USING 35487/java
$ kill 35487
$ /usr/local/tomcat/apache-tomcat-8.5.47/bin/startup.sh
Step 4: Test the load generator
Open the exlporer and visit url http://192.168.3.110:8080/sdcloud/
The GUI page is shown as below:
Type 1: Using the URL interfaces
We provide four URL interfaces to control the load generator as below:
| url interface | description | type | parameter |
|---|---|---|---|
startOnlineQuery.do?intensity=1&serviceId=0 |
start to generate the request load | GET | intensity: concurrent requests per second (RPS) serviceId: inference service index id |
goOnlineQuery.do?serviceId=0 |
visit the real-time latency page | GET | serviceId: inference service index id |
setIntensity.do?intensity=20&serviceId=0 |
change the RPS dynamically | GET | intensity: RPS serviceId: inference service index id |
stopOnlineQuery.do?serviceId=0 |
stop the load generator | GET | --- |
The index id of mobilenet-coco300-tf is 0, and the second inference service should be set to 1, ..., etc. The supported service in load generator is easy to scale
For examle, firstly, click the startOnlineQuery.do?intensity=1&serviceId=0 link to generate request loads, and you will see the page in a waiting state (circle loading), then after $windowSize seconds, click the goOnlineQuery.do?serviceId=0 link to watch the real-time latency as below.
Metrics in real-time latency watch page:
- realRPS: The concurrent requests number of last second from client, RPS (request per second)
- realQPS: The response number of last second in server, QPS (query per second)
- AvgRPS: The average concurrent RPS in
$windowsizetime scale - AvgQPS: The average QPS in
$windowsizetime scale - SR: The average service rate in
$windowsizetime scale,SR=AvgQPS/AvgRPS*100% - queryTime: The 99th tail-latency of the concurrent requests per second
- Avg99th: The average
queryTimesin$windowsizetime scale
When the load generator is running, click the setIntensity.do?intensity=N&serviceId=0 link to change the RPS, please replace N to the number of concurrent requests per second you want. Finally, click stopOnlineQuery.do?serviceId=0 to stop the load testing
Type 2: Using the RMI interfaces
We also provide the Java RMI interfaces for the remote funciton calls in users' external code without clicking URL links. Using RMI, you can control the load generator and collect metric data. The RMI interface file is in LoadGen/src/main/java/scs/util/rmi/LoadInterface.java, the interface functions are shown as below:
package scs.util.rmi;
import java.rmi.Remote;
import java.rmi.RemoteException;
/**
* RMI interface class, which is used to control the load generator
* The functions can be call by remote client code
* @author Yanan Yang
* @date 2019-11-11
* @address TANKLab, TianJin University, China
*/
public interface LoadInterface extends Remote{
public float getWindowAvgPerSec99thLatency(int serviceId) throws RemoteException; //return the value of Avg99th
public float getRealPerSec99thLatency(int serviceId) throws RemoteException; //return the value of queryTime
//public float getWindowSize95thRealLatency(int serviceId) throws RemoteException; //return the value of queryTime (95th), unused
//public float getLcCurLatency999thRealLatency(int serviceId) throws RemoteException; //return the value of queryTime (99.9th), unused
public int getRealQueryIntensity(int serviceId) throws RemoteException; //return the value of realQPS
public int getRealRequestIntensity(int serviceId) throws RemoteException; //return the value of realRPS
public float getWindowAvgServiceRate(int serviceId) throws RemoteException; //return the value of SR
public void execStartHttpLoader(int serviceId) throws RemoteException; //start load generator for serviceId
public void execStopHttpLoader(int serviceId) throws RemoteException; //stop load generator for serviceId
public int setIntensity(int intensity,int serviceId) throws RemoteException; //change the RPS dynamically
}
When tomcat starts, the server side will automatically setup the RMI service using serverIp and rmiPort in LoadGen/src/main/resources/conf/sys.properties, the RMI service function is shown as below:
package scs.util.rmi;
import java.net.MalformedURLException;
import java.rmi.Naming;
import java.rmi.RemoteException;
import java.rmi.registry.LocateRegistry;
public class RmiService {
private static RmiService loadService=null;
private RmiService(){}
public synchronized static RmiService getInstance() {
if (loadService == null) {
loadService = new RmiService();
}
return loadService;
}
public void setupService(String serverIp,int rmiPort) {
try {
System.setProperty("java.rmi.server.hostname",serverIp);
LocateRegistry.createRegistry(rmiPort);
LoadInterface load = new LoadInterfaceImpl();
Naming.rebind("rmi://"+serverIp+":"+rmiPort+"/load", load);
} catch (RemoteException e) {
e.printStackTrace();
} catch (MalformedURLException e) {}
}
}
The client side need to setup the RMI connection before controling the load generator, a stand connection function is shown as below:
private static void setupRmiConnection(){
try {
LoadInterface loader=(LoadInterface) Naming.lookup("rmi://192.168.3.110:22222/load");
} catch (MalformedURLException e) {
e.printStackTrace();
} catch (RemoteException e) {
e.printStackTrace();
} catch (NotBoundException e) {
e.printStackTrace();
}
if(loader!=null){
System.out.println(Repository.loaderRmiUrl +"connection successed");
}else{
System.out.println(Repository.loaderRmiUrl +"connection failed");
}
}
More tutorial of RMI interface can be see from here
Performance evaluation of loadGen
We evaluate the performance of load generator tool and web inference service and show the results as below:
Performance testing of load generator
To aviod the performance bottleneck of web service interfering the testing results, we set the request url in LoadGen/src/main/resources/conf/sys.properties to an empty url (this url does nothing and just returns 'helloworld')
mageClassifyBaseURL=http://192.168.3.130:31500/helloworld
Then we test the concurrent ability of load generator and collect the latency data, the client and server are deployed individually on two nodes that connected with 1Gbps WLAN
Fig.1 depicts the 99th tail-latency collected by load generator with the workloads ranges from RPS=1 to RPS=2000, the requests are sent using multi-threads in open-loop, the worst 99th tail-latency < 250ms when the RPS=2000, which shows the low queue latency in load generator. Fig.2 shows the 99th tail-latency increases linearly with the RPS, this demonstrates the load generator is well designed and has a good performance of workload scalability. Fig.3 shows the CPU usage in server end with increasing RPS, which has a same trend with the tail-latency in Fig.1. The inference service consumes < 0.5 CPU core when RPS=400, while the CPU usage no more than 2 CPU cores when RPS=2000, it demonstrates the low overhead of guicorn+flask framework
Future work
The latest released version of load generator has satisfied our experiment needs, in the future, we plan to implement these functions as below:
- Distributed load generator
- Diversified output statistics (e.g., PDF, hist graph)
Bug report & Question
We have used the load generator for a long time in our work [2,3], and fixed many bugs that have been found. If you have some new findings, please contact us via Email: ynyang@tju.edu.cn
Reference
[1] Kasture H, Sanchez D. Tailbench: a benchmark suite and evaluation methodology for latency-critical applications[C]//2016 IEEE International Symposium on Workload Characterization (IISWC). IEEE, 2016: 1-10.
[2] Y. Yang, L. Zhao, Z. Li, L. Nie, P. Chen and K. Li. ElaX: Provisioning Resource Elastically for Containerized Online Cloud Services[C]//2019 IEEE 21st International Conference on High Performance Computing and Communications (HPCC). IEEE, 2019: 1987-1994.
[3] L. Zhao, Y. Yang, K. Zhang, etc. Rhythm: Component-distinguishable Workload
Deployment in Datacenters[C]//EuroSys 2020. ACM. Under review
基于Java的支持可变QPS的http负载生成器，提供交互界面和RMI接口的更多相关文章
- 【公开课】【阿里在线技术峰会】魏鹏：基于Java容器的多应用部署技术实践
对于公开课,可能目前用不上这些,但是往往能在以后想解决方案的时候帮助到我.以下是阿里对公开课的整理 摘要: 在首届阿里巴巴在线峰会上,阿里巴巴中间件技术部专家魏鹏为大家带来了题为<基于Java容 ...
- Spring Boot 2.2 正式发布，大幅性能提升 + Java 13 支持
之前 Spring Boot 2.2没能按时发布,是由于 Spring Framework 5.2 的发布受阻而推迟.这次随着 Spring Framework 5.2.0 成功发布之后,Spring ...
- 9个基于Java的搜索引擎框架
在这个信息相当繁杂的互联网时代,我们已经学会了如何利用搜索引擎这个强大的利器来找寻目标信息,比如你会在Google上搜索情人节如何讨女朋友欢心,你也会在百度上寻找正规的整容医疗机构(尽管有很大一部分广 ...
- 基于java平台的常用资源整理
这里整理了基于java平台的常用资源 翻译 from :akullpp | awesome-java 大家一起学习,共同进步. 如果大家觉得有用,就mark一下,赞一下,或评论一下,让更多的人知道.t ...
- 基于Java Netty框架构建高性能的部标808协议的GPS服务器
使用Java语言开发一个高质量和高性能的jt808 协议的GPS通信服务器,并不是一件简单容易的事情,开发出来一段程序和能够承受数十万台车载接入是两码事,除去开发部标808协议的固有复杂性和几个月长周 ...
- 基于Java Mina框架的部标808服务器设计和开发
在开发部标GPS平台中,部标808GPS服务器是系统的核心关键,决定了部标平台的稳定性和行那个.Linux服务器是首选,为了跨平台,开发语言选择Java自不待言. 我们为客户开发的部标服务器基于Min ...
- 这里整理了基于java平台的常用资源
这里整理了基于java平台的常用资源 翻译 from :akullpp | awesome-java 大家一起学习,共同进步. 如果大家觉得有用,就mark一下,赞一下,或评论一下,让更多的人知道.t ...
- memcached学习——常用命令+基于java客户端的3种简单实现（二）
常用命令: memcached设计的原则就是简单,所以支持的命令也不是特别多~ 1.查看memcached的状态,主要用于分析内存的使用状况.优化内存分配等 stats 查看memcached的运行状 ...
- 基于Java图片数据库Neo4j 3.0.0发布 全新的内部架构
基于Java图片数据库Neo4j 3.0.0发布 全新的内部架构 Neo4j 3.0.0 正式发布,这是 Neo4j 3.0 系列的第一个版本.此版本对内部架构进行了全新的设计;提供给开发者更强大的生 ...
随机推荐
- git用法汇总
使用了一年多的git命令了,昨晚竟然又出现了问题.虽然解决了,不过还是被罚了... 总结下自己常用的git命令和遇到的一些坑. 1)常用的命令 1. 从git远程分支clone代码: git clon ...
- （转）自动微分(Automatic Differentiation)简介——tensorflow核心原理
现代深度学习系统中(比如MXNet, TensorFlow等)都用到了一种技术——自动微分.在此之前,机器学习社区中很少发挥这个利器,一般都是用Backpropagation进行梯度求解,然后进行SG ...
- effictive-python笔记
第一章 用Pythonic方式来思考 1.确认自己所用的python版本(python3) 两个主流的python版本:python2(2020年就不维护) python3(推荐) 多种流行的pyth ...
- js set集合转数组 Array.from的使用方法
1.set集合转化Array数组 注意:这个可以使用过滤数组中的重复的元素 你可以先把数组转化为set集合 然后在把这个集合通过Array.from这个方法把集合在转化为数组 var set = n ...
- [LeetCode] 260. Single Number III 单独数 III
Given an array of numbers nums, in which exactly two elements appear only once and all the other ele ...
- 本地dev环境，运行时用node模块自动读取并整合文件
const http = require('http'); const fs = require('fs'); const path = require('path'); const glob = r ...
- Git设定不合并的文件
一个最简单的做法,通过添加.gitattributes文件来完成 1 在要被合并的分支中设置 git config --global merge.ours.driver true设置git配置项mer ...
- python爬虫scrapy（一）
一,准备scrapy依赖组件环境,按照以下顺序安装 .wheel pip install wheel .lxml http:.PyOpenssl https://pypi.python.org/pyp ...
- Java开发笔记（一百三十九）JavaFX的输入框
循着Swing的旧例,JavaFX仍然提供了三种文本输入框,分别是单行输入框TextField.密码输入框PasswordField.多行输入框TextArea.这些输入框都由抽象类TextInput ...
- GhostScript说明
关于ghostscript(以下简称gs).Gs是一个地下工作者,一般用户不熟悉它,因为它上不和用户直接打交道,下不直接接触打印机.但是在打印工作中它却扮演了极为重要的解色. 一般从用户常见文件如图片 ...