JMH 使用指南 - java 性能测试

JMH 篇

JMH，即Java Microbenchmark Harness 翻译：java 微基准测试 工具套件。
## 1.添加依赖
```
<dependency>
   <groupId>org.openjdk.jmh</groupId>
   <artifactId>jmh-core</artifactId>
   <version>1.19</version>
   <scope>provided</scope>
</dependency>
<dependency>
   <groupId>org.openjdk.jmh</groupId>
   <artifactId>jmh-generator-annprocess</artifactId>
   <version>1.19</version>
   <scope>provided</scope>
</dependency>
```
## 2.第一个例子
请参加： JMHFirstBenchmark.java

请参加(晋级)： SecondBenchmark.java
请参加(晋级)： ThirdBenchmark.java

## 3.常用注解说明 

###3.1 @BenchmarkMode(Mode.All)

Mode有：- Throughput: 整体吞吐量，例如“1秒内可以执行多少次调用” (thrpt,参加第5点)
- AverageTime: 调用的平均时间，例如“每次调用平均耗时xxx毫秒”。（avgt）
- SampleTime: 随机取样，最后输出取样结果的分布，例如“99%的调用在xxx毫秒以内，99.99%的调用在xxx毫秒以内”（simple）
- SingleShotTime: 以上模式都是默认一次 iteration 是 1s，唯有 SingleShotTime 是只运行一次。往往同时把 warmup 次数设为0，用于测试冷启动时的性能。（ss）

### 3.2 @OutputTimeUnit(TimeUnit.MILLISECONDS)

统计单位， 微秒、毫秒 、分、小时、天
### 3.3 @State

 可参：JMHFirstBenchmark.java

类注解，JMH测试类必须使用@State注解，State定义了一个类实例的生命周期，可以类比Spring Bean的Scope。由于JMH允许多线程同时执行测试，不同的选项含义如下：

```
Scope.Thread：默认的State，每个测试线程分配一个实例；
Scope.Benchmark：所有测试线程共享一个实例，用于测试有状态实例在多线程共享下的性能；
Scope.Group：每个线程组共享一个实例；
```

### 3.4 @Benchmark 
很重要的方法注解，表示该方法是需要进行 benchmark 的对象。和@test 注解一致
### 3.5 @Setup 

方法注解，会在执行 benchmark 之前被执行，正如其名，主要用于初始化。
### 3.6 @TearDown (Level)

方法注解，与@Setup 相对的，会在所有 benchmark 执行结束以后执行，主要用于资源的回收等。
   (Level)   用于控制 @Setup，@TearDown 的调用时机，默认是 Level.Trial。

          Trial：每个benchmark方法前后；
          Iteration：每个benchmark方法每次迭代前后；
          Invocation：每个benchmark方法每次调用前后，谨慎使用，需留意javadoc注释；  
### 3.7 @Param
@Param注解接收一个String数组 ，
可以用来指定某项参数的多种情况。特别适合用来测试一个函数在不同的参数输入的情况下的性能。
 可参：JMHFirstBenchmark.java

## 4 Options常用选项
### 4.1 include 

benchmark 所在的类的名字，这里可以使用正则表达式对所有类进行匹配。
参考：SecondBenchmark.java

### 4.2 fork 
JVM因为使用了profile-guided optimization而“臭名昭著”，这对于微基准测试来说十分不友好，因为不同测试方法的profile混杂在一起，“互相伤害”彼此的测试结果。对于每个@Benchmark方法使用一个独立的进程可以解决这个问题，这也是JMH的默认选项。注意不要设置为0，设置为n则会启动n个进程执行测试（似乎也没有太大意义）。
fork选项也可以通过方法注解以及启动参数来设置。

### 4.3 warmupIterations
预热次数，每次默认1秒。

### 4.4 measurementIterations 
实际测量的迭代次数，每次默认1秒。

### 4.5 Group 
方法注解，可以把多个 benchmark 定义为同一个 group，则它们会被同时执行，譬如用来模拟生产者－消费者读写速度不一致情况下的表现。

### 4.6 Threads 
每个fork进程使用多少条线程去执行你的测试方法，默认值是Runtime.getRuntime().availableProcessors()。 

## 5 输出结果  
```

# @BenchmarkMode(Mode.All)
# JMH version: 1.19
# VM version: JDK 1.7.0_80, VM 24.80-b11
# VM invoker: C:\Program Files\Java\jdk1.7.0_80\jre\bin\java.exe
# VM options: -javaagent:D:\Program Files\JetBrains\IntelliJ IDEA 2018.1\lib\idea_rt.jar=51664:D:\Program Files\JetBrains\IntelliJ IDEA 2018.1\bin -Dfile.encoding=UTF-8
# Warmup: 2 iterations, single-shot each
# Measurement: 2 iterations, single-shot each
# Timeout: 10 min per iteration
# Threads: 10 threads
# Benchmark mode: Single shot invocation time
# Benchmark: com.gemantic.wealth.yunmatong.service.jmh.SecondBenchmark.singleThreadBench
# Parameters: (length = 100000)

# Run progress: 99.98% complete, ETA 00:00:00
# Fork: 1 of 1
# Warmup Iteration   1: 34.641 ±(99.9%) 33.844 ms/op
# Warmup Iteration   2: 7.129 ±(99.9%) 9.238 ms/op
Iteration   1: 7.573 ±(99.9%) 4.581 ms/op
Iteration   2: 6.235 ±(99.9%) 4.150 ms/op

# Run complete. Total time: 00:00:36

Benchmark                                                    (length)    Mode     Cnt    Score    Error   Units
SecondBenchmark.multiThreadBench                               100000   thrpt       2  147.758           ops/ms    
SecondBenchmark.singleThreadBench                              100000   thrpt       2    0.983           ops/ms
SecondBenchmark.multiThreadBench                               100000    avgt       2    0.068            ms/op
SecondBenchmark.singleThreadBench                              100000    avgt       2   10.510            ms/op
SecondBenchmark.multiThreadBench                               100000  sample  295532    0.068 ±  0.001   ms/op
SecondBenchmark.multiThreadBench:multiThreadBench·p0.00        100000  sample            0.010            ms/op
SecondBenchmark.multiThreadBench:multiThreadBench·p0.50        100000  sample            0.066            ms/op
SecondBenchmark.multiThreadBench:multiThreadBench·p0.90        100000  sample            0.095            ms/op
SecondBenchmark.multiThreadBench:multiThreadBench·p0.95        100000  sample            0.104            ms/op
SecondBenchmark.multiThreadBench:multiThreadBench·p0.99        100000  sample            0.126            ms/op
SecondBenchmark.multiThreadBench:multiThreadBench·p0.999       100000  sample            0.172            ms/op
SecondBenchmark.multiThreadBench:multiThreadBench·p0.9999      100000  sample            1.729            ms/op
SecondBenchmark.multiThreadBench:multiThreadBench·p1.00        100000  sample            4.309            ms/op
SecondBenchmark.singleThreadBench                              100000  sample    2036   10.196 ±  0.581   ms/op
SecondBenchmark.singleThreadBench:singleThreadBench·p0.00      100000  sample            6.201            ms/op
SecondBenchmark.singleThreadBench:singleThreadBench·p0.50      100000  sample            8.020            ms/op
SecondBenchmark.singleThreadBench:singleThreadBench·p0.90      100000  sample           10.355            ms/op
SecondBenchmark.singleThreadBench:singleThreadBench·p0.95      100000  sample           38.443            ms/op
SecondBenchmark.singleThreadBench:singleThreadBench·p0.99      100000  sample           41.943            ms/op
SecondBenchmark.singleThreadBench:singleThreadBench·p0.999     100000  sample           73.498            ms/op
SecondBenchmark.singleThreadBench:singleThreadBench·p0.9999    100000  sample           74.973            ms/op
SecondBenchmark.singleThreadBench:singleThreadBench·p1.00      100000  sample           74.973            ms/op
SecondBenchmark.multiThreadBench                               100000      ss       2    0.223            ms/op
SecondBenchmark.singleThreadBench                              100000      ss       2    6.904            ms/op

```

6.第一个例子

package com.gemantic.wealth.yunmatong.service.jmh;

import lombok.extern.slf4j.Slf4j;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import org.openjdk.jmh.runner.options.TimeValue;

import java.util.concurrent.TimeUnit;

@Slf4j
@BenchmarkMode(Mode.AverageTime)// 测试方法平均执行时间
@OutputTimeUnit(TimeUnit.MICROSECONDS)// 输出结果的时间粒度为微秒
@State(Scope.Benchmark) // 每个测试线程一个实例
public class JMHFirstBenchmark {
    /*
     * Most of the time, you need to maintain some state while the benchmark is
     * running. Since JMH is heavily used to build concurrent benchmarks, we
     * opted for an explicit notion of state-bearing objects.
     *
     * Below are two state objects. Their class names are not essential, it
     * matters they are marked with @State. These objects will be instantiated
     * on demand, and reused during the entire benchmark trial.
     *
     * The important property is that state is always instantiated by one of
     * those benchmark threads which will then have the access to that state.
     * That means you can initialize the fields as if you do that in worker
     * threads (ThreadLocals are yours, etc).
     */

    @State(Scope.Benchmark)
    public static class BenchmarkState {
        volatile double x = Math.PI;
    }

    @State(Scope.Thread)
    public static class ThreadState {
        volatile double x = Math.PI;
    }

    @Benchmark
    public void measureUnshared(ThreadState state) {
        // All benchmark threads will call in this method.
        //
        // However, since ThreadState is the Scope.Thread, each thread
        // will have it's own copy of the state, and this benchmark
        // will measure unshared case.
        state.x++;
        try {
            Thread.sleep(500);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        log.info("measureUnshared:"+ state.x);
    }

    @Benchmark
    public void measureShared(BenchmarkState state) {
        // All benchmark threads will call in this method.
        //
        // Since BenchmarkState is the Scope.Benchmark, all threads
        // will share the state instance, and we will end up measuring
        // shared case.
        state.x++;
        try {
            Thread.sleep(500);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        log.info("measureShared:"+ state.x);
    }

    /*
     * ============================== HOW TO RUN THIS TEST: ====================================
     *
     * You are expected to see the drastic difference in shared and unshared cases,
     * because you either contend for single memory location, or not. This effect
     * is more articulated on large machines.
     *
     * You can run this test:
     *
     * a) Via the command line:
     *    $ mvn clean install
     *    $ java -jar target/benchmarks.jar JMHSample_03 -wi 5 -i 5 -t 4 -f 1
     *    (we requested 5 measurement/warmup iterations, with 4 threads, single fork)
     *
     * b) Via the Java API:
     *    (see the JMH homepage for possible caveats when running from IDE:
     *      http://openjdk.java.net/projects/code-tools/jmh/)
     */

    public static void main(String[] args) throws RunnerException {
        // 可以通过注解
        Options opt = new OptionsBuilder()
                .include(JMHFirstBenchmark.class.getSimpleName())
                .warmupIterations(3) // 预热3次
                .measurementIterations(2).measurementTime(TimeValue.valueOf("1s")) // 运行5次，每次10秒
                .threads(10) // 10线程并发
                .forks(2)
                .build();

        new Runner(opt).run();
    }

}

　　

第二个例子setup&TearDown&Param

package com.gemantic.wealth.yunmatong.service.jmh;

import com.gemantic.wealth.yunmatong.service.jmh.service.Calculator;
import com.gemantic.wealth.yunmatong.service.jmh.service.MultithreadCalculator;
import com.gemantic.wealth.yunmatong.service.jmh.service.SinglethreadCalculator;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;

import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.All)
@OutputTimeUnit(TimeUnit.MILLISECONDS)
@State(Scope.Benchmark)
public class SecondBenchmark {
    @Param({"100000"})
    private int length;

    private int[] numbers;
    private Calculator singleThreadCalc;
    private Calculator multiThreadCalc;

    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(SecondBenchmark.class.getSimpleName()) // .include("JMHF.*") 可支持正则
                .forks(0)
                .warmupIterations(2)
                .measurementIterations(2).threads(10)
                .build();

        new Runner(opt).run();
    }

    @Benchmark
    public long singleThreadBench() {
        return singleThreadCalc.sum(numbers);
    }

    @Benchmark
    public long multiThreadBench() {
        return multiThreadCalc.sum(numbers);
    }

    @Setup(Level.Trial)
    public void prepare() {
        int n = length;
        numbers =new int[n];
        for (int i=0;i<n;i++){
            numbers[i]=i;
        }
        singleThreadCalc = new SinglethreadCalculator();
        multiThreadCalc = new MultithreadCalculator(Runtime.getRuntime().availableProcessors());
    }

    @TearDown
    public void shutdown() {
        singleThreadCalc.shutdown();
        multiThreadCalc.shutdown();
    }
}

　　

第三个例子group

package com.gemantic.wealth.yunmatong.service.jmh;

import com.gemantic.wealth.yunmatong.service.jmh.service.Calculator;
import com.gemantic.wealth.yunmatong.service.jmh.service.MultithreadCalculator;
import com.gemantic.wealth.yunmatong.service.jmh.service.SinglethreadCalculator;
import org.openjdk.jmh.annotations.*;
import org.openjdk.jmh.runner.Runner;
import org.openjdk.jmh.runner.RunnerException;
import org.openjdk.jmh.runner.options.Options;
import org.openjdk.jmh.runner.options.OptionsBuilder;
import org.openjdk.jmh.runner.options.TimeValue;

import java.util.concurrent.TimeUnit;

@BenchmarkMode(Mode.AverageTime)
@OutputTimeUnit(TimeUnit.MICROSECONDS)
@State(Scope.Benchmark)
public class ThirdBenchmark {

    @State(Scope.Group)
    public static class BenchmarkState {
        volatile double x = Math.PI;
    }

    @Benchmark
    @Group("custom")
    @GroupThreads(10)
    public void read(BenchmarkState state) {
        state.x++;
        try {
            Thread.sleep(5);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        System.out.println("ThirdBenchmark.read: "+ state.x);
    }

    @Benchmark
    @Group("custom")
    public void book(BenchmarkState state) {
        state.x++;
        try {
            Thread.sleep(5);
        } catch (InterruptedException e) {
            e.printStackTrace();
        }
        System.out.println("ThirdBenchmark.book: "+ state.x);
    }

    public static void main(String[] args) throws RunnerException {
        Options opt = new OptionsBuilder()
                .include(ThirdBenchmark.class.getSimpleName()) // .include("JMHF.*") 可支持正则
                .forks(0)
                .warmupIterations(0)
                .measurementIterations(2).measurementTime(TimeValue.valueOf("10ms")).threads(5)
                .build();

        new Runner(opt).run();
    }
}