原文地址:http://architects.dzone.com/articles/how-analyze-java-thread-dumps

The Performance Zone is presented by AppDynamics. AppDynamics is a leaders in the APM space with massive cost reductions for users.

The content of this article was originally written by Tae Jin Gu on the Cubrid blog

When there is an obstacle, or when a Java based Web application is running much slower than expected, we need to use thread dumps. If thread dumps feel like very complicated to you, this article may help you very much. Here I will explain what threads are in Java, their types, how they are created, how to manage them, how you can dump threads from a running application, and finally how you can analyze them and determine the bottleneck or blocking threads. This article is a result of long experience in Java application debugging.

Java and Thread

A web server uses tens to hundreds of threads to process a large number of concurrent users. If two or more threads utilize the same resources, a contention between the threads is inevitable, and sometimes deadlock occurs.

Thread contention is a status in which one thread is waiting for a lock, held by another thread, to be lifted. Different threads frequently access shared resources on a web application. For example, to record a log, the thread trying to record the log must obtain a lock and access the shared resources.

Deadlock is a special type of thread contention, in which two or more threads are waiting for the other threads to complete their tasks in order to complete their own tasks.

Different issues can arise from thread contention. To analyze such issues, you need to use the thread dump. A thread dump will give you the information on the exact status of each thread.

Background Information for Java Threads

Thread Synchronization

A thread can be processed with other threads at the same time. In order to ensure compatibility when multiple threads are trying to use shared resources, one thread at a time should be allowed to access the shared resources by using thread synchronization.

Thread synchronization on Java can be done using monitor. Every Java object has a single monitor. The monitor can be owned by only one thread. For a thread to own a monitor that is owned by a different thread, it needs to wait in the wait queue until the other thread releases its monitor.

Thread Status

In order to analyze a thread dump, you need to know the status of threads. The statuses of threads are stated on java.lang.Thread.State.

Figure 1: Thread Status.

  • NEW: The thread is created but has not been processed yet.
  • RUNNABLE: The thread is occupying the CPU and processing a task. (It may be in WAITING status due to the OS's resource distribution.)
  • BLOCKED: The thread is waiting for a different thread to release its lock in order to get the monitor lock.
  • WAITING: The thread is waiting by using a wait, join or park method.
  • TIMED_WAITING: The thread is waiting by using a sleep, wait, join or park method. (The difference from WAITING is that the maximum waiting time is specified by the method parameter, and WAITING can be relieved by time as well as external changes.)

Thread Types

Java threads can be divided into two:

  1. daemon threads;
  2. and non-daemon threads.

Daemon threads stop working when there are no other non-daemon threads. Even if you do not create any threads, the Java application will create several threads by default. Most of them are daemon threads, mainly for processing tasks such as garbage collection or JMX.

A thread running the 'static void main(String[] args)’ method is created as a non-daemon thread, and when this thread stops working, all other daemon threads will stop as well. (The thread running this main method is called the VM thread in HotSpot VM.)

Getting a Thread Dump

We will introduce the three most commonly used methods. Note that there are many other ways to get a thread dump. A thread dump can only show the thread status at the time of measurement, so in order to see the change in thread status, it is recommended to extract them from 5 to 10 times with 5-second intervals.

Getting a Thread Dump Using jstack

In JDK 1.6 and higher, it is possible to get a thread dump on MS Windows using jstack.

Use PID via jps to check the PID of the currently running Java application process.

Use the extracted PID as the parameter of jstack to obtain a thread dump.

A Thread Dump Using jVisualVM

Generate a thread dump by using a program such as jVisualVM.

Figure 2:  A Thread Dump Using visualvm.

The task on the left indicates the list of currently running processes. Click on the process for which you want the information, and select the thread tab to check the thread information in real time. Click the Thread Dump button on the top right corner to get the thread dump file.

Generating in a Linux Terminal

Obtain the process pid by using ps -ef command to check the pid of the currently running Java process.

Use the extracted pid as the parameter of kill –SIGQUIT(3) to obtain a thread dump.

Thread Information from the Thread Dump File

  • Thread name: When using Java.lang.Thread class to generate a thread, the thread will be named Thread-(Number), whereas when using java.util.concurrent.ThreadFactory class, it will be named pool-(number)-thread-(number).
  • Priority: Represents the priority of the threads.
  • Thread ID: Represents the unique ID for the threads. (Some useful information, including the CPU usage or memory usage of the thread, can be obtained by using thread ID.)
  • Thread status: Represents the status of the threads.
  • Thread callstack: Represents the call stack information of the threads.

Thread Dump Patterns by Type

When Unable to Obtain a Lock (BLOCKED)

This is when the overall performance of the application slows down because a thread is occupying the lock and prevents other threads from obtaining it. In the following example, BLOCKED_TEST pool-1-thread-1 thread is running with <0x0000000780a000b0> lock, while BLOCKED_TEST pool-1-thread-2 and BLOCKED_TEST pool-1-thread-3 threads are waiting to obtain <0x0000000780a000b0> lock.

Figure 3: A thread blocking other threads.

"BLOCKED_TEST pool-1-thread-1" prio=6 tid=0x0000000006904800 nid=0x28f4 runnable [0x000000000785f000]
java.lang.Thread.State: RUNNABLE
at java.io.FileOutputStream.writeBytes(Native Method)
at java.io.FileOutputStream.write(FileOutputStream.java:282)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
- locked (a java.io.BufferedOutputStream)
at java.io.PrintStream.write(PrintStream.java:432)
- locked (a java.io.PrintStream)
at sun.nio.cs.StreamEncoder.writeBytes(StreamEncoder.java:202)
at sun.nio.cs.StreamEncoder.implFlushBuffer(StreamEncoder.java:272)
at sun.nio.cs.StreamEncoder.flushBuffer(StreamEncoder.java:85)
- locked (a java.io.OutputStreamWriter)
at java.io.OutputStreamWriter.flushBuffer(OutputStreamWriter.java:168)
at java.io.PrintStream.newLine(PrintStream.java:496)
- locked (a java.io.PrintStream)
at java.io.PrintStream.println(PrintStream.java:687)
- locked (a java.io.PrintStream)
at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:44)
- locked (a com.nbp.theplatform.threaddump.ThreadBlockedState)
at com.nbp.theplatform.threaddump.ThreadBlockedState$1.run(ThreadBlockedState.java:7)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers:
- (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "BLOCKED_TEST pool-1-thread-2" prio=6 tid=0x0000000007673800 nid=0x260c waiting for monitor entry [0x0000000008abf000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:43)
- waiting to lock (a com.nbp.theplatform.threaddump.ThreadBlockedState)
at com.nbp.theplatform.threaddump.ThreadBlockedState$2.run(ThreadBlockedState.java:26)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers:
- (a java.util.concurrent.locks.ReentrantLock$NonfairSync) "BLOCKED_TEST pool-1-thread-3" prio=6 tid=0x00000000074f5800 nid=0x1994 waiting for monitor entry [0x0000000008bbf000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.nbp.theplatform.threaddump.ThreadBlockedState.monitorLock(ThreadBlockedState.java:42)
- waiting to lock (a com.nbp.theplatform.threaddump.ThreadBlockedState)
at com.nbp.theplatform.threaddump.ThreadBlockedState$3.run(ThreadBlockedState.java:34)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662) Locked ownable synchronizers:
- (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

When in Deadlock Status

This is when thread A needs to obtain thread B's lock to continue its task, while thread B needs to obtain thread A's lock to continue its task. In the thread dump, you can see that DEADLOCK_TEST-1 thread has 0x00000007d58f5e48 lock, and is trying to obtain 0x00000007d58f5e60 lock. You can also see that DEADLOCK_TEST-2 thread has 0x00000007d58f5e60 lock, and is trying to obtain 0x00000007d58f5e78 lock. Also, DEADLOCK_TEST-3 thread has 0x00000007d58f5e78 lock, and is trying to obtain 0x00000007d58f5e48 lock. As you can see, each thread is waiting to obtain another thread's lock, and this status will not change until one thread discards its lock.

Figure 4: Threads in a Deadlock status.

"DEADLOCK_TEST-1" daemon prio=6 tid=0x000000000690f800 nid=0x1820 waiting for monitor entry [0x000000000805f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197)
- waiting to lock (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182)
- locked (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135) Locked ownable synchronizers:
- None "DEADLOCK_TEST-2" daemon prio=6 tid=0x0000000006858800 nid=0x17b8 waiting for monitor entry [0x000000000815f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197)
- waiting to lock (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182)
- locked (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135) Locked ownable synchronizers:
- None "DEADLOCK_TEST-3" daemon prio=6 tid=0x0000000006859000 nid=0x25dc waiting for monitor entry [0x000000000825f000]
java.lang.Thread.State: BLOCKED (on object monitor)
at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.goMonitorDeadlock(ThreadDeadLockState.java:197)
- waiting to lock (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.monitorOurLock(ThreadDeadLockState.java:182)
- locked (a com.nbp.theplatform.threaddump.ThreadDeadLockState$Monitor)
at com.nbp.theplatform.threaddump.ThreadDeadLockState$DeadlockThread.run(ThreadDeadLockState.java:135) Locked ownable synchronizers:
- None

When Continuously Waiting to Receive Messages from a Remote Server

The thread appears to be normal, since its state keeps showing as RUNNABLE. However, when you align the thread dumps chronologically, you can see that socketReadThread thread is waiting infinitely to read the socket.

Figure 5: Continuous Waiting Status.

When Waiting

The thread is maintaining WAIT status. In the thread dump, IoWaitThread thread keeps waiting to receive a message from LinkedBlockingQueue. If there continues to be no message for LinkedBlockingQueue, then the thread status will not change.

Figure 6: Waiting status.

 

When Thread Resources Cannot be Organized Normally

Unnecessary threads will pile up when thread resources cannot be organized normally. If this occurs, it is recommended to monitor the thread organization process or check the conditions for thread termination.

Figure 7: Unorganized Threads.

How to Solve Problems by Using Thread Dump

Example 1: When the CPU Usage is Abnormally High

1. Extract the thread that has the highest CPU usage.

From the application, find out which thread is using the CPU the most.

Acquire the Light Weight Process (LWP) that uses the CPU the most and convert its unique number (10039) into a hexadecimal number (0x2737).

2. After acquiring the thread dump, check the thread's action.

Extract the thread dump of an application with a PID of 10029, then find the thread with an nid of 0x2737.

Extract thread dumps several times every hour, and check the status change of the threads to determine the problem.

Example 2: When the Processing Performance is Abnormally Slow

After acquiring thread dumps several times, find the list of threads with BLOCKED status.

 

Acquire the list of threads with BLOCKED status after getting the thread dumps several times.

If the threads are BLOCKED, extract the threads related to the lock that the threads are trying to obtain.

Through the thread dump, you can confirm that the thread status stays BLOCKED because <0xe0375410> lock could not be obtained. This problem can be solved by analyzing stack trace from the thread currently holding the lock.

There are two reasons why the above pattern frequently appears in applications using DBMS. The first reason is inadequate configurations. Despite the fact that the threads are still working, they cannot show their best performance because the configurations for DBCP and the like are not adequate. If you extract thread dumps multiple times and compare them, you will often see that some of the threads that were BLOCKED previously are in a different state.

The second reason is the abnormal connection. When the connection with DBMS stays abnormal, the threads wait until the time is out. In this case, even after extracting the thread dumps several times and comparing them, you will see that the threads related to DBMS are still in a BLOCKED state. By adequately changing the values, such as the timeout value, you can shorten the time in which the problem occurs.

Coding for Easy Thread Dump

Naming Threads

When a thread is created using java.lang.Thread object, the thread will be named Thread-(Number). When a thread is created using java.util.concurrent.DefaultThreadFactory object, the thread will be named pool-(Number)-thread-(Number). When analyzing tens to thousands of threads for an application, if all the threads still have their default names, analyzing them becomes very difficult, because it is difficult to distinguish the threads to be analyzed.

Therefore, you are recommended to develop the habit of naming the threads whenever a new thread is created.

When you create a thread using java.lang.Thread, you can give the thread a custom name by using the creator parameter.

When you create a thread using java.util.concurrent.ThreadFactory, you can name it by generating your own ThreadFactory. If you do not need special functionalities, then you can use MyThreadFactory as described below:

Obtaining More Detailed Information by Using MBean

You can obtain ThreadInfo objects using MBean. You can also obtain more information that would be difficult to acquire via thread dumps, by using ThreadInfo.

 

You can acquire the amount of time that the threads WAITed or were BLOCKED by using the method in ThreadInfo, and by using this you can also obtain the list of threads that have been inactive for an abnormally long period of time.

In Conclusion

In this article I was concerned that for developers with a lot of experience in multi-thread programming, this material may be common knowledge, whereas for less experienced developers, I felt that I was skipping straight to thread dumps, without providing enough background information about the thread activities. This was because of my lack of knowledge, as I was not able to explain the thread activities in a clear yet concise manner. I sincerely hope that this article will prove helpful for many developers.

How to Analyze Java Thread Dumps--reference的更多相关文章

  1. How to Analyze Java Thread Dumps

    When there is an obstacle, or when a Java based Web application is running much slower than expected ...

  2. JAVA Thread Dumps 三部曲

    一.windows环境下方法 1:cmd下找到运行服务器容器的PID jps -v 例: C:\Users\Administrator>jps -v4856 Bootstrap -Djdk.tl ...

  3. 三个实例演示 Java Thread Dump 日志分析

    原文地址: http://www.cnblogs.com/zhengyun_ustc/archive/2013/01/06/dumpanalysis.html jstack Dump 日志文件中的线程 ...

  4. 三个实例演示 Java Thread Dump 日志分析(转)

    原文链接:http://www.cnblogs.com/zhengyun_ustc/archive/2013/01/06/dumpanalysis.html 转来当笔记^_^ jstack Dump ...

  5. 在 TDA 工具里看到 Java Thread State 的第一反应是

    转载:http://itindex.net/detail/43158-tda-%E5%B7%A5%E5%85%B7-java   使用 TDA 工具,看到大量 Java Thread State 的第 ...

  6. Java Thread dump 日志分析

    jstack Dump 日志文件中的线程状态 dump 文件里,值得关注的线程状态有: 死锁,Deadlock(重点关注)  执行中,Runnable 等待资源,Waiting on conditio ...

  7. 性能分析之-- JAVA Thread Dump 分析综述

    性能分析之-- JAVA Thread Dump 分析综述       一.Thread Dump介绍 1.1什么是Thread Dump? Thread Dump是非常有用的诊断Java应用问题的工 ...

  8. [转]Java Thread Dump 性能分析

    Java and Thread 一个 web 服务器使用几十到几百个线程来处理大量并发用户,如果一个或多个线程使用相同的资源,线程之间的竞争就不可避免了,并且有时候可能会发生死锁. Thread co ...

  9. Java Thread线程控制

    一.线程和进程 进程是处于运行中的程序,具有一定的独立能力,进程是系统进行资源分配和调度的一个独立单位. 进程特征: A.独立性:进程是系统中独立存在的实体,可以拥有自己独立的资源,每个进程都拥有自己 ...

随机推荐

  1. 一滴一点vim(学习+备忘)

    普通模式: h j k l 分别是左下上右方式移动: :w 保存修改 :q 推出 :wq 保存修改并退出 :q! 放弃修改强制推出 x 删除光标所在位置字符 i 在光标所以位置插入字符 删除类命令: ...

  2. FireFox、chrome通过插件使用IE内核,IE Tab v2

    fireFox 插件管理中  搜索IE tab  找到ieTab v2 安装后,添加网站规则就可以用. chrome 去ietab.net 下载 *.crx文件,  打开插件管理界面, 拖拽 *.cr ...

  3. Visual Studio中的一些较大的文件的作用

    1.sdf 这些是工程中的中间,用于预编译等作用,最终可执行文件是不需要的,默认情况下,删除后重新编译还会生成.如果不需要,在Visual Studio里进入如下设置: 进入"Tools & ...

  4. UESTC 250 windy数(数位DP)

    题意:题意:求区间[A,B]之间的,不含前导0,且相邻两数位之间相差至少为2的正整数有多少个. 分析:dp[i][j]表示,长度为i.以j为结尾的表示的个数,再加一个前导0判断即可 #include ...

  5. nginx log_format指令记录自定义响应头

    我们用的nginx有做过一些定制开发,为了调试方便,加了一些自定义的response header,那么如何把这个自定义头记录到日志中以便于观察呢? nginx log_format指令支持这种扩展, ...

  6. HIbernate学习笔记(九) hibernate事务并发处理与乐观悲观锁

    事务并发处理 一. 数据库的隔离级别:并发性作用. 1.   ReadUncommited(未提交读):没有提交就可以读取到数据(发出了Insert,但没有commit就可以读取到.)很少用 2.   ...

  7. homework01

    第一眼看到这个题目的时候就意识到这道题应该使用动态规划来解决,但因代码能力有限,因此从一维的问题开始解决,用C语言编写,代码如下: int maxsum(int *p,int size){ int i ...

  8. Qt 固定窗口【worldsing 笔记】

    w.setWindowFlags(Qt::WindowMinimizeButtonHint); //禁止最大化按钮 w.setFixedSize(1024,587);//固定窗口大小

  9. passport.js

    $(function(){ function isPlaceholder(){ var input = document.createElement('input'); return 'placeho ...

  10. My97datepicker设置后一个日期大于前一个日期

    <@e.text label="开始时间" name="mtpiStratTime" required="true" class=&q ...