转载自:http://www.rigongyizu.com/jvm-max-threads/

有应用报出这样的异常“java.lang.OutOfMemoryError: unable to create new native thread”。甚至机器上执行shell命令也会报”-bash: fork: Resource temporarily unavailable”异常。机器上的其他应用如hadoop也会受影响:

1 2013-08-21 20:15:48,496 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread
2 java.lang.OutOfMemoryError: unable to create new native thread
3         at java.lang.Thread.start0(Native Method)
4         at java.lang.Thread.start(Thread.java:640)
5         at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
6 ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:524)
7         at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.
8 ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java:456)
9         at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java:128)
10         at org.apache.hadoop.yarn.event.AsyncDispatcher$1.run(AsyncDispatcher.java:77)
11         at java.lang.Thread.run(Thread.java:662)
12 2013-08-21 20:15:48,497 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye..

一看以为内存不够导致无法创建新的线程,但是观察机器上的内存还有空闲,猜测是哪个地方对线程创建有限制。

首先需要排除操作系统对线程创建数的限制,参考:《JVM中可生成的最大Thread数量》一文,设置操作系统可以支持创建10万个线程:

1 echo "100000" > /proc/sys/kernel/threads-max
2 echo "100000" > /proc/sys/kernel/pid_max     (默认32768
3 echo "200000" > /proc/sys/vm/max_map_count   (默认65530
4 ulimit -u unlimited   (设置max user processes的值)

当前测试环境为:

1 [admin@bufer108081.tbc ~]$ uname -a
2 Linux bufer108081.tbc 2.6.32-220.23.2.ali927.el5.x86_64 #1 SMP Mon Jan 28 14:57:06 CST 2013 x86_64 x86_64 x86_64 GNU/Linux
3 [admin@bufer108081.tbc ~]$ cat /etc/redhat-release
4 Red Hat Enterprise Linux Server release 5.7 (Tikanga)
5 [admin@bufer108081.tbc ~]$ java -version
6 java version "1.7.0_51"
7 Java(TM) SE Runtime Environment (build 1.7.0_51-b13)
8 OpenJDK (Alibaba) 64-Bit Server VM (build 24.45-b08-internal, mixed mode)
9 [admin@bufer108081.tbc ~]$ ulimit -a
10 core file size          (blocks, -c) 0
11 data seg size           (kbytes, -d) unlimited
12 scheduling priority             (-e) 0
13 file size               (blocks, -f) unlimited
14 pending signals                 (-i) 387068
15 max locked memory       (kbytes, -l) 64
16 max memory size         (kbytes, -m) unlimited
17 open files                      (-n) 131072
18 pipe size            (512 bytes, -p) 8
19 POSIX message queues     (bytes, -q) 819200
20 real-time priority              (-r) 0
21 stack size              (kbytes, -s) 10240
22 cpu time               (seconds, -t) unlimited
23 max user processes              (-u) unlimited
24 virtual memory          (kbytes, -v) unlimited
25 file locks                      (-x) unlimited
26 [admin@bufer108081.tbc ~/dev/baoniu]$ free -g
27              total       used       free     shared    buffers     cached
28 Mem:            47         31         15          0          3         25
29 -/+ buffers/cache:          3         44
30 Swap:            0          0          0

测试程序见本文最后面。测试结果:突破了网上所说的32000个线程数,成功创建了 10万个线程。 (由于/proc/sys/kernel/pid_max默认为32768,所以网上很多测试程序测试JVM只能创建32000个线程。)

1 [admin@bufer108081.tbc ~/dev/baoniu]$ java -Xss128k MaxThreadsMain
2 The stack size specified is too small, Specify at least 228k
3 Error: Could not create the Java Virtual Machine.
4 Error: A fatal exception has occurred. Program will exit.
5 [admin@bufer108081.tbc ~/dev/baoniu]$ java -Xss228k MaxThreadsMain
6 4,000 threads: Time to create 4,000 threads was 0.846 seconds
7 8,000 threads: Time to create 4,000 threads was 2.425 seconds
8 12,000 threads: Time to create 4,000 threads was 4.813 seconds
9 16,000 threads: Time to create 4,000 threads was 7.229 seconds
10 20,000 threads: Time to create 4,000 threads was 10.443 seconds
11 24,000 threads: Time to create 4,000 threads was 14.480 seconds
12 28,000 threads: Time to create 4,000 threads was 19.709 seconds
13 32,000 threads: Time to create 4,000 threads was 24.742 seconds
14 36,000 threads: Time to create 4,000 threads was 31.181 seconds
15 40,000 threads: Time to create 4,000 threads was 36.629 seconds
16 44,000 threads: Time to create 4,000 threads was 42.796 seconds
17 48,000 threads: Time to create 4,000 threads was 48.659 seconds
18 52,000 threads: Time to create 4,000 threads was 55.030 seconds
19 56,000 threads: Time to create 4,000 threads was 60.130 seconds
20 60,000 threads: Time to create 4,000 threads was 67.419 seconds
21 64,000 threads: Time to create 4,000 threads was 73.507 seconds
22 68,000 threads: Time to create 4,000 threads was 79.416 seconds
23 72,000 threads: Time to create 4,000 threads was 85.261 seconds
24 76,000 threads: Time to create 4,000 threads was 92.201 seconds
25 80,000 threads: Time to create 4,000 threads was 98.087 seconds
26 84,000 threads: Time to create 4,000 threads was 108.263 seconds
27 88,000 threads: Time to create 4,000 threads was 114.840 seconds
28 92,000 threads: Time to create 4,000 threads was 121.841 seconds
29 96,000 threads: Time to create 4,000 threads was 127.714 seconds
30 After creating 99,410 threads, java.lang.OutOfMemoryError: unable to create new native thread
31         at java.lang.Thread.start0(Native Method)
32         at java.lang.Thread.start(Thread.java:713)
33         at MaxThreadsMain.addThread(MaxThreadsMain.java:43)
34         at MaxThreadsMain.main(MaxThreadsMain.java:13)

创建9W多个线程后,进程占用内存:VIRT=40.5g RES=4.7g,用free -g查看系统还有9G的空闲(free)内存。


JVM最多能启动的线程数参照公式:

1 (MaxProcessMemory  - JVMMemory – ReservedOsMemory) / (ThreadStackSize) = Number of threads
  • MaxProcessMemory : 进程的最大寻址空间
  • JVMMemory : JVM内存
  • ReservedOsMemory : 保留的操作系统内存,如Native heap,JNI之类,一般100多M
  • ThreadStackSize : 线程栈的大小,jvm启动时由Xss指定

MaxProcessMemory:如32位的linux默认每个进程最多申请3G的地址空间,64位的操作系统可以支持到46位(64TB)的物理地址空间和47位(128T)的进程虚拟地址空间(linux 64位CPU内存限制)。

JVM内存:由Heap区和Perm区组成。通过-Xms和-Xmx可以指定heap区大小,通过-XX:PermSize和-XX:MaxPermSize指定perm区的大小(默认从32MB 到64MB,和JVM版本有关)。

线程栈ThreadStackSize:

Java程序中,每个线程都有自己的Stack Space。这个Stack Space的空间是独立分配的,与-Xmx和-Xms指定的堆大小无关。Stack Space用来做方法的递归调用时压入Stack Frame。所以当递归调用太深的时候,就有可能耗尽Stack Space,爆出StackOverflow的错误。对于32位JVM,缺省值为256KB,对于64位JVM,缺省值为512KB。最大值根据平台和特定机器配置的不同而不同。如果超过最大值,那么将报告java/lang/OutOfMemoryError消息。

可见,减少Xss指定的线程栈大小能够启动更多的线程,但是线程总数也受到系统空闲内存和操作系统的限制。

总结下影响Java线程数量的因素:

  • Java虚拟机本身:-Xms,-Xmx,-Xss;
  • 系统限制: /proc/sys/kernel/pid_max, /proc/sys/kernel/thread-max, max_user_process(ulimit -u), /proc/sys/vm/max_map_count。

ps: 最后发现是这台机器上有个应用代码问题创建了过多的线程,达到系统限制,而影响了YARN和其他应用。一般来说,单机线程数过多可以考虑使用线程池或者更多的服务器。


附测试程序:

1 import java.util.ArrayList;
2 import java.util.List;
3  
4 public class MaxThreadsMain {
5  
6   public static final int BATCH_SIZE = 4000;
7  
8   public static void main(String... args) throws InterruptedException {
9     List<Thread> threads = new ArrayList<Thread>();
10     try {
11       for (int i = 0; i <= 100 1000; i += BATCH_SIZE) {
12         long start = System.currentTimeMillis();
13         addThread(threads, BATCH_SIZE);
14         long end = System.currentTimeMillis();
15         Thread.sleep(1000);
16         long delay = end - start;
17         System.out.printf("%,d threads: Time to create %,d threads was %.3f seconds %n", threads.size(), BATCH_SIZE, delay / 1e3);
18       }
19     catch (Throwable e) {
20       System.err.printf("After creating %,d threads, ", threads.size());
21       e.printStackTrace();
22     }
23  
24   }
25  
26   private static void addThread(List<Thread> threads, int num) {
27     for (int i = 0; i < num; i++) {
28       Thread t = new Thread(new Runnable() {
29         @Override
30         public void run() {
31           try {
32             while (!Thread.interrupted()) {
33               Thread.sleep(1000);
34             }
35           catch (InterruptedException ignored) {
36             //
37           }
38         }
39       });
40       t.setDaemon(true);
41       t.setPriority(Thread.MIN_PRIORITY);
42       threads.add(t);
43       t.start();
44     }
45   }
46 }

 

附:

  1. MySQL Performance: Hitting Error "Can't Create A New Thread (Errno 11)" On A High Number Of Connections:
  2. Http://Dimitrik.Free.Fr/Blog/Archives/2010/11/Mysql-Performance-Hitting-Error-Cant-Create-A-New-Thread-Errno-11-On-A-High-Number-Of-Connections.Html
  3. $ Ulimit -U
  4. 1024
  5. It Explains 1000 Sessions Limitation ;-)
  6. Adding Few Lines More To My "/Etc/Security/Limits.Conf" File:
  7. # Cat /Etc/Security/Limits.Conf
  8. Mysql     Soft    Nofile  10240
  9. Mysql     Hard    Nofile  40960
  10. Mysql     Soft    Nproc   10240
  11. Mysql     Hard    Nproc   40960
  12. Fixed My Issue! :-))
  1. MySQL Performance: Hitting Error "Can't Create A New Thread (Errno 11)" On A High Number Of Connections:
  2. Http://Dimitrik.Free.Fr/Blog/Archives/2010/11/Mysql-Performance-Hitting-Error-Cant-Create-A-New-Thread-Errno-11-On-A-High-Number-Of-Connections.Html
  3.  
  4. $ Ulimit -U
  5. 1024
  6. It Explains 1000 Sessions Limitation ;-)
  7.  
  8. Adding Few Lines More To My "/Etc/Security/Limits.Conf" File:
  9.  
  10. # Cat /Etc/Security/Limits.Conf
  11. Mysql Soft Nofile 10240
  12. Mysql Hard Nofile 40960
  13. Mysql Soft Nproc 10240
  14. Mysql Hard Nproc 40960
  15. Fixed My Issue! :-))

JVM最多能创建多少个线程: unable to create new native thread的更多相关文章

  1. JVM虚拟机宕机_java.lang.OutOfMemoryError: unable to create new native thread

    原因:当前用户的系统最最大程序数数已达到最大值,使用ulimit -u可以看到是1024   解决办法:在当前用户下使用ulimit -u 65535 然后再执行jsp,一切ok     功能说明:控 ...

  2. JVM内存越多,能创建的线程越少,越容易发生java.lang.OutOfMemoryError: unable to create new native thread。

    一.认识问题: 首先我们通过下面这个 测试程序 来认识这个问题:运行的环境 (有必要说明一下,不同环境会有不同的结果):32位 Windows XP,Sun JDK 1.6.0_18, eclipse ...

  3. JVM最多能创建多少个线程:unabletocreatenewnativethread

    最近需要测试一个长连接服务器,数据上需要达到100W的长连接,测试的客户端,一个线程保持一个连接,发现linux服务器默认创建到3200多个线程的时候,就会报错这个错误“java.lang.OutOf ...

  4. JVM截至多少线程可以创建: unable to create new native thread

    最近的测试需要很长的连接server.这些数据需要达到100W长连接,试client.一个线程来保持连接.查找linuxserver创建者默认3200当多个线程.这个错误将得到"java.l ...

  5. 记一次tomcat线程创建异常调优:unable to create new native thread

    测试在进行一次性能测试的时候发现并发300个请求时出现了下面的异常: HTTP Status 500 - Handler processing failed; nested exception is ...

  6. 剥下“java.lang.OutOfMemoryError: unable to create new native thread”的外衣 创建线程数公式(MaxProcessMemory - JVMMemory – ReservedOsMemory)

    剥下“java.lang.OutOfMemoryError: unable to create new native thread”的外衣 星期一早上到了公司,据称产品环境抛出了最可爱的异常—OutO ...

  7. 线程ava.lang.OutOfMemoryError: unable to create new native thread

    近日开发遇到多线程的问题: java.lang.OutOfMemoryError: unable to create new native thread Exception in thread &qu ...

  8. 解决Unable to create new native thread

    两种类型的Out of Memory java.lang.OutOfMemoryError: Java heap space error 当JVM尝试在堆中分配对象,堆中空间不足时抛出.一般通过设定J ...

  9. java.lang.OutOfMemoryError: unable to create new native thread如何解决

    工作中碰到过这个问题好几次了,觉得有必要总结一下,所以有了这篇文章,这篇文章分为三个部分:认识问题.分析问题.解决问题. 一.认识问题: 首先我们通过下面这个 测试程序 来认识这个问题:运行的环境 ( ...

随机推荐

  1. Codeforces 757 D. Felicity's Big Secret Revealed 状压DP

    D. Felicity's Big Secret Revealed   The gym leaders were fascinated by the evolutions which took pla ...

  2. Servlet8

    一.Annotation 进行配置不需要 web.xml 新建new Servlet 时,不需要在web.xml 文件中生成Servlet的相关信息 import java.io.IOExceptio ...

  3. 清空sql 日志

    USE [master] GO ALTER DATABASE 库名 SET RECOVERY SIMPLE GO USE 库名 GO ,,TRUNCATEONLY) GO USE [master] G ...

  4. Ubuntu 搭建 LAMP 服务器

    /******************************************************************** * Ubuntu 搭建 LAMP 服务器 * 说明: * 想 ...

  5. linux下libpcap抓包分析

    一.首先下载libpcap包http://www.tcpdump.org/#latest-release 然后安装,安装完成后进入安装根目录的tests文件夹,编译运行findalldevstest. ...

  6. Phpspy 2011继续身份验证绕过漏洞

    Author: Tm3yShell7 官方目前下载已经修补上了 目前官方下载是2011.php, 文件名为2011ok.php的是带洞版本. 今天m0r5和我说phpspy2011 我都不知道2011 ...

  7. Android SDK更新以及ADT更新出现问题的解决办法(转载)

    转自:http://zyueqi.iteye.com/blog/1474323 问题描述 使用SDK Manager更新时出现问题Failed to fetch URL https://dl-ssl. ...

  8. HDU1175:连连看(搜索)

    传送门 题意 给定一个n*m的矩阵,询问q次,两个方块是否能被消掉,弯折次数不超过两次 分析 这题写了有一个下午,思路很简单,但是有很多trick,(唉),我还是太弱 trick 初始判断:1.两点不 ...

  9. OKEX websocket API 连接Python范例

    因为 websocket-client 新版的各种大脑降级设计 很多功能无法使用需要安装老版本websocket-client的包才能正常使用 pip3 install websocket-clien ...

  10. Activiti6.0教程 Eclipse安装Activiti Diagram插件(一)

    最近这段时间打算出一个Activiti6.0的详细教程,Activiti作为一个流行的开源工作流引擎,正在不断发展,其6.0版本以API形式提供服务,而之前版本基本都是要求我们的应用以JDK方式与其交 ...