tomcat服务无响应堆栈分析

tomcat服务突然无响应了，导出内存堆栈和线程堆栈，分析后发现是同步锁使用不合理导致的。

[root@prd-dtb-web-01 ~]#
[root@prd-dtb-web-01 ~]# jmap -heap 10472
Attaching to process ID 10472, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 25.111-b14

using thread-local object allocation.
Parallel GC with 2 thread(s)

Heap Configuration:
   MinHeapFreeRatio         = 0
   MaxHeapFreeRatio         = 100
   MaxHeapSize              = 1983905792 (1892.0MB)
   NewSize                  = 41943040 (40.0MB)
   MaxNewSize               = 661127168 (630.5MB)
   OldSize                  = 83886080 (80.0MB)
   NewRatio                 = 2
   SurvivorRatio            = 8
   MetaspaceSize            = 21807104 (20.796875MB)
   CompressedClassSpaceSize = 1073741824 (1024.0MB)
   MaxMetaspaceSize         = 17592186044415 MB
   G1HeapRegionSize         = 0 (0.0MB)

Heap Usage:
PS Young Generation
Eden Space:
   capacity = 487063552 (464.5MB)
   used     = 26314992 (25.095932006835938MB)
   free     = 460748560 (439.40406799316406MB)
   5.402784070362958% used
From Space:
   capacity = 72351744 (69.0MB)
   used     = 71945680 (68.61274719238281MB)
   free     = 406064 (0.3872528076171875MB)
   99.43876404693161% used
To Space:
   capacity = 84934656 (81.0MB)
   used     = 0 (0.0MB)
   free     = 84934656 (81.0MB)
   0.0% used
PS Old Generation
   capacity = 254279680 (242.5MB)
   used     = 136744120 (130.40935516357422MB)
   free     = 117535560 (112.09064483642578MB)
   53.77705367570071% used

36326 interned Strings occupying 4333960 bytes.
[root@prd-dtb-web-01 ~]# jmap -dump:file=dump_dtb  10472
Dumping heap to /root/dump_dtb ...
Heap dump file created

[root@prd-dtb-web-01 ~]# jstack 10472 > thread_dtb

使用Eclipse MemoryAnalyzer对内存堆栈的分析，发现线程已经占满了。

通过对线程堆栈文件内容的分析，发现大量线程都处于waiting to lock状态，进一步发现，对应代码使用了synchronized同步锁，一个线程内部访问数据库发生了超时，长时间占用了该锁，导致其它线程都处于等待状态。

...

"http-nio-8002-exec-26" #52 daemon prio=5 os_prio=0 tid=0x00007f951c01b000 nid=0x291e waiting for monitor entry [0x00007f9530dc9000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at cn.friendsure.tdtb.services.WeixinPayService.payed(WeixinPayService.java:273)
    - waiting to lock <0x000000008a9103b0> (a cn.friendsure.tdtb.services.WeixinPayService)
    at sun.reflect.GeneratedMethodAccessor264.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.springframework.web.bind.annotation.support.HandlerMethodInvoker.invokeHandlerMethod(HandlerMethodInvoker.java:175)
    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.invokeHandlerMethod(AnnotationMethodHandlerAdapter.java:446)
    at org.springframework.web.servlet.mvc.annotation.AnnotationMethodHandlerAdapter.handle(AnnotationMethodHandlerAdapter.java:434)

...

at cn.friendsure.tdtb.logics.OrderLogic.transferOrder(OrderLogic.java:531)
    - locked <0x000000008a9104e0> (a cn.friendsure.tdtb.logics.OrderLogic)
    at cn.friendsure.tdtb.services.WeixinPayService.transferOrder(WeixinPayService.java:478)
    at cn.friendsure.tdtb.services.WeixinPayService.payed(WeixinPayService.java:399)
    - locked <0x000000008a9103b0> (a cn.friendsure.tdtb.services.WeixinPayService)
    at sun.reflect.GeneratedMethodAccessor264.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)

...

【解决方案】

去掉不必要的同步锁。

【总结】

涉及IO的方法，尽量不要使用synchronized关键字，如果一定要用，要确保程序逻辑中有明确的超时控制机制，并且超时时间不要太长。