转载自:http://www.rigongyizu.com/jvm-max-threads/
有应用报出这样的异常“java.lang.OutOfMemoryError: unable to create new native thread”。甚至机器上执行shell命令也会报”-bash: fork: Resource temporarily unavailable”异常。机器上的其他应用如hadoop也会受影响:
1 |
2013 - 08 - 21 20 : 15 : 48 , 496 FATAL org.apache.hadoop.yarn.event.AsyncDispatcher: Error in dispatcher thread |
2 |
java.lang.OutOfMemoryError: unable to create new native thread |
3 |
at java.lang.Thread.start0(Native Method) |
4 |
at java.lang.Thread.start(Thread.java: 640 ) |
5 |
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer. |
6 |
ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java: 524 ) |
7 |
at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer. |
8 |
ResourceLocalizationService$LocalizerTracker.handle(ResourceLocalizationService.java: 456 ) |
9 |
at org.apache.hadoop.yarn.event.AsyncDispatcher.dispatch(AsyncDispatcher.java: 128 ) |
10 |
at org.apache.hadoop.yarn.event.AsyncDispatcher$ 1 .run(AsyncDispatcher.java: 77 ) |
11 |
at java.lang.Thread.run(Thread.java: 662 ) |
12 |
2013 - 08 - 21 20 : 15 : 48 , 497 INFO org.apache.hadoop.yarn.event.AsyncDispatcher: Exiting, bbye.. |
一看以为内存不够导致无法创建新的线程,但是观察机器上的内存还有空闲,猜测是哪个地方对线程创建有限制。
首先需要排除操作系统对线程创建数的限制,参考:《JVM中可生成的最大Thread数量》一文,设置操作系统可以支持创建10万个线程:
1 |
echo "100000" > /proc/sys/kernel/threads-max |
2 |
echo "100000" > /proc/sys/kernel/pid_max (默认 32768 ) |
3 |
echo "200000" > /proc/sys/vm/max_map_count (默认 65530 ) |
4 |
ulimit -u unlimited (设置max user processes的值) |
当前测试环境为:
1 |
[admin@bufer108081.tbc ~]$ uname -a |
2 |
Linux bufer108081.tbc 2.6.32-220.23.2.ali927.el5.x86_64 #1 SMP Mon Jan 28 14:57:06 CST 2013 x86_64 x86_64 x86_64 GNU/Linux |
3 |
[admin@bufer108081.tbc ~]$ cat /etc/redhat-release |
4 |
Red Hat Enterprise Linux Server release 5.7 (Tikanga) |
5 |
[admin@bufer108081.tbc ~]$ java -version |
6 |
java version "1.7.0_51" |
7 |
Java(TM) SE Runtime Environment (build 1.7.0_51-b13) |
8 |
OpenJDK (Alibaba) 64-Bit Server VM (build 24.45-b08-internal, mixed mode) |
9 |
[admin@bufer108081.tbc ~]$ ulimit -a |
10 |
core file size (blocks, -c) 0 |
11 |
data seg size (kbytes, -d) unlimited |
12 |
scheduling priority (-e) 0 |
13 |
file size (blocks, -f) unlimited |
14 |
pending signals (-i) 387068 |
15 |
max locked memory (kbytes, -l) 64 |
16 |
max memory size (kbytes, -m) unlimited |
17 |
open files (-n) 131072 |
18 |
pipe size (512 bytes, -p) 8 |
19 |
POSIX message queues (bytes, -q) 819200 |
20 |
real- time priority (-r) 0 |
21 |
stack size (kbytes, -s) 10240 |
22 |
cpu time (seconds, -t) unlimited |
23 |
max user processes (-u) unlimited |
24 |
virtual memory (kbytes, - v ) unlimited |
25 |
file locks (-x) unlimited |
26 |
[admin@bufer108081.tbc ~/dev/baoniu]$ free -g |
27 |
total used free shared buffers cached |
29 |
-/+ buffers/cache: 3 44 |
测试程序见本文最后面。测试结果:突破了网上所说的32000个线程数,成功创建了 10万个线程。 (由于/proc/sys/kernel/pid_max默认为32768,所以网上很多测试程序测试JVM只能创建32000个线程。)
1 |
[admin@bufer108081.tbc ~/dev/baoniu]$ java -Xss128k MaxThreadsMain |
2 |
The stack size specified is too small, Specify at least 228k |
3 |
Error: Could not create the Java Virtual Machine. |
4 |
Error: A fatal exception has occurred. Program will exit . |
5 |
[admin@bufer108081.tbc ~/dev/baoniu]$ java -Xss228k MaxThreadsMain |
6 |
4,000 threads: Time to create 4,000 threads was 0.846 seconds |
7 |
8,000 threads: Time to create 4,000 threads was 2.425 seconds |
8 |
12,000 threads: Time to create 4,000 threads was 4.813 seconds |
9 |
16,000 threads: Time to create 4,000 threads was 7.229 seconds |
10 |
20,000 threads: Time to create 4,000 threads was 10.443 seconds |
11 |
24,000 threads: Time to create 4,000 threads was 14.480 seconds |
12 |
28,000 threads: Time to create 4,000 threads was 19.709 seconds |
13 |
32,000 threads: Time to create 4,000 threads was 24.742 seconds |
14 |
36,000 threads: Time to create 4,000 threads was 31.181 seconds |
15 |
40,000 threads: Time to create 4,000 threads was 36.629 seconds |
16 |
44,000 threads: Time to create 4,000 threads was 42.796 seconds |
17 |
48,000 threads: Time to create 4,000 threads was 48.659 seconds |
18 |
52,000 threads: Time to create 4,000 threads was 55.030 seconds |
19 |
56,000 threads: Time to create 4,000 threads was 60.130 seconds |
20 |
60,000 threads: Time to create 4,000 threads was 67.419 seconds |
21 |
64,000 threads: Time to create 4,000 threads was 73.507 seconds |
22 |
68,000 threads: Time to create 4,000 threads was 79.416 seconds |
23 |
72,000 threads: Time to create 4,000 threads was 85.261 seconds |
24 |
76,000 threads: Time to create 4,000 threads was 92.201 seconds |
25 |
80,000 threads: Time to create 4,000 threads was 98.087 seconds |
26 |
84,000 threads: Time to create 4,000 threads was 108.263 seconds |
27 |
88,000 threads: Time to create 4,000 threads was 114.840 seconds |
28 |
92,000 threads: Time to create 4,000 threads was 121.841 seconds |
29 |
96,000 threads: Time to create 4,000 threads was 127.714 seconds |
30 |
After creating 99,410 threads, java.lang.OutOfMemoryError: unable to create new native thread |
31 |
at java.lang.Thread.start0(Native Method) |
32 |
at java.lang.Thread.start(Thread.java:713) |
33 |
at MaxThreadsMain.addThread(MaxThreadsMain.java:43) |
34 |
at MaxThreadsMain.main(MaxThreadsMain.java:13) |
创建9W多个线程后,进程占用内存:VIRT=40.5g RES=4.7g,用free -g查看系统还有9G的空闲(free)内存。
JVM最多能启动的线程数参照公式:
1 |
(MaxProcessMemory - JVMMemory – ReservedOsMemory) / (ThreadStackSize) = Number of threads |
- MaxProcessMemory : 进程的最大寻址空间
- JVMMemory : JVM内存
- ReservedOsMemory : 保留的操作系统内存,如Native heap,JNI之类,一般100多M
- ThreadStackSize : 线程栈的大小,jvm启动时由Xss指定
MaxProcessMemory:如32位的linux默认每个进程最多申请3G的地址空间,64位的操作系统可以支持到46位(64TB)的物理地址空间和47位(128T)的进程虚拟地址空间(linux 64位CPU内存限制)。
JVM内存:由Heap区和Perm区组成。通过-Xms和-Xmx可以指定heap区大小,通过-XX:PermSize和-XX:MaxPermSize指定perm区的大小(默认从32MB 到64MB,和JVM版本有关)。
线程栈ThreadStackSize:
Java程序中,每个线程都有自己的Stack Space。这个Stack Space的空间是独立分配的,与-Xmx和-Xms指定的堆大小无关。Stack Space用来做方法的递归调用时压入Stack Frame。所以当递归调用太深的时候,就有可能耗尽Stack Space,爆出StackOverflow的错误。对于32位JVM,缺省值为256KB,对于64位JVM,缺省值为512KB。最大值根据平台和特定机器配置的不同而不同。如果超过最大值,那么将报告java/lang/OutOfMemoryError消息。
可见,减少Xss指定的线程栈大小能够启动更多的线程,但是线程总数也受到系统空闲内存和操作系统的限制。
总结下影响Java线程数量的因素:
- Java虚拟机本身:-Xms,-Xmx,-Xss;
- 系统限制: /proc/sys/kernel/pid_max, /proc/sys/kernel/thread-max, max_user_process(ulimit -u), /proc/sys/vm/max_map_count。
ps: 最后发现是这台机器上有个应用代码问题创建了过多的线程,达到系统限制,而影响了YARN和其他应用。一般来说,单机线程数过多可以考虑使用线程池或者更多的服务器。
附测试程序:
1 |
import java.util.ArrayList; |
4 |
public class MaxThreadsMain { |
6 |
public static final int BATCH_SIZE = 4000 ; |
8 |
public static void main(String... args) throws InterruptedException { |
9 |
List<Thread> threads = new ArrayList<Thread>(); |
11 |
for ( int i = 0 ; i <= 100 * 1000 ; i += BATCH_SIZE) { |
12 |
long start = System.currentTimeMillis(); |
13 |
addThread(threads, BATCH_SIZE); |
14 |
long end = System.currentTimeMillis(); |
16 |
long delay = end - start; |
17 |
System.out.printf( "%,d threads: Time to create %,d threads was %.3f seconds %n" , threads.size(), BATCH_SIZE, delay / 1e3); |
19 |
} catch (Throwable e) { |
20 |
System.err.printf( "After creating %,d threads, " , threads.size()); |
26 |
private static void addThread(List<Thread> threads, int num) { |
27 |
for ( int i = 0 ; i < num; i++) { |
28 |
Thread t = new Thread( new Runnable() { |
32 |
while (!Thread.interrupted()) { |
35 |
} catch (InterruptedException ignored) { |
41 |
t.setPriority(Thread.MIN_PRIORITY); |
附:
- MySQL Performance: Hitting Error "Can't Create A New Thread (Errno 11)" On A High Number Of Connections:
- Http://Dimitrik.Free.Fr/Blog/Archives/2010/11/Mysql-Performance-Hitting-Error-Cant-Create-A-New-Thread-Errno-11-On-A-High-Number-Of-Connections.Html
- $ Ulimit -U
- 1024
- It Explains 1000 Sessions Limitation ;-)
- Adding Few Lines More To My "/Etc/Security/Limits.Conf" File:
- # Cat /Etc/Security/Limits.Conf
- Mysql Soft Nofile 10240
- Mysql Hard Nofile 40960
- Mysql Soft Nproc 10240
- Mysql Hard Nproc 40960
- Fixed My Issue! :-))
MySQL Performance: Hitting Error "Can't Create A New Thread (Errno 11)" On A High Number Of Connections:
Http://Dimitrik.Free.Fr/Blog/Archives/2010/11/Mysql-Performance-Hitting-Error-Cant-Create-A-New-Thread-Errno-11-On-A-High-Number-Of-Connections.Html
$ Ulimit -U
1024
It Explains 1000 Sessions Limitation ;-)
Adding Few Lines More To My "/Etc/Security/Limits.Conf" File:
# Cat /Etc/Security/Limits.Conf
Mysql Soft Nofile 10240
Mysql Hard Nofile 40960
Mysql Soft Nproc 10240
Mysql Hard Nproc 40960
Fixed My Issue! :-))
- JVM虚拟机宕机_java.lang.OutOfMemoryError: unable to create new native thread
原因:当前用户的系统最最大程序数数已达到最大值,使用ulimit -u可以看到是1024 解决办法:在当前用户下使用ulimit -u 65535 然后再执行jsp,一切ok 功能说明:控 ...
- JVM内存越多,能创建的线程越少,越容易发生java.lang.OutOfMemoryError: unable to create new native thread。
一.认识问题: 首先我们通过下面这个 测试程序 来认识这个问题:运行的环境 (有必要说明一下,不同环境会有不同的结果):32位 Windows XP,Sun JDK 1.6.0_18, eclipse ...
- JVM最多能创建多少个线程:unabletocreatenewnativethread
最近需要测试一个长连接服务器,数据上需要达到100W的长连接,测试的客户端,一个线程保持一个连接,发现linux服务器默认创建到3200多个线程的时候,就会报错这个错误“java.lang.OutOf ...
- JVM截至多少线程可以创建: unable to create new native thread
最近的测试需要很长的连接server.这些数据需要达到100W长连接,试client.一个线程来保持连接.查找linuxserver创建者默认3200当多个线程.这个错误将得到"java.l ...
- 记一次tomcat线程创建异常调优:unable to create new native thread
测试在进行一次性能测试的时候发现并发300个请求时出现了下面的异常: HTTP Status 500 - Handler processing failed; nested exception is ...
- 剥下“java.lang.OutOfMemoryError: unable to create new native thread”的外衣 创建线程数公式(MaxProcessMemory - JVMMemory – ReservedOsMemory)
剥下“java.lang.OutOfMemoryError: unable to create new native thread”的外衣 星期一早上到了公司,据称产品环境抛出了最可爱的异常—OutO ...
- 线程ava.lang.OutOfMemoryError: unable to create new native thread
近日开发遇到多线程的问题: java.lang.OutOfMemoryError: unable to create new native thread Exception in thread &qu ...
- 解决Unable to create new native thread
两种类型的Out of Memory java.lang.OutOfMemoryError: Java heap space error 当JVM尝试在堆中分配对象,堆中空间不足时抛出.一般通过设定J ...
- java.lang.OutOfMemoryError: unable to create new native thread如何解决
工作中碰到过这个问题好几次了,觉得有必要总结一下,所以有了这篇文章,这篇文章分为三个部分:认识问题.分析问题.解决问题. 一.认识问题: 首先我们通过下面这个 测试程序 来认识这个问题:运行的环境 ( ...
随机推荐
- Installation error: INSTALL_FAILED_CPU_ABI_INCOMPATIBLE
解决方法: http://my.oschina.net/u/242764/blog/375909 当我们安装好Genymotion后,把Android运用部署到上面调试时,console 控制台会报错 ...
- Restrictions.or多个条件用法
两个条件或查询: Restrictions.or(Restrictions.in("username",list1),Restrictions.idEq(1)); 三个或多个条件查 ...
- CRM 配置 ADFS后,使用自定义STS遇到的问题总结
1 登录ADFS服务查看 ADFS日志 2 根据日志提示的错误,设置ADFS对应的属性 (Get-ADFSRelyingPartyTrust) | Set-ADFSRelyingPartyTrust ...
- lucene DocValues——本质是为通过docID查找某field的值 看图
Why DocValues? The standard way that Solr builds the index is with an inverted index. This style bui ...
- Java String 和 new String()的区别
Java String 和 new String()的区别 本文转自:http://www.cnblogs.com/heima-jieqi/archive/2012/04/10/2440086.htm ...
- jsp 中href链接有前缀
在jsp页面中,如果有一个链接写为:“href="www.hqvolunteer.org"”(链接到的是外网),这样发布项目了后,把鼠标移到上面显示的是:你的项目路径/www.hq ...
- EF通过反射追踪修改记录.适合记录变更系统
private static void IsUpdate<T>(T old, T current, string id) { Model.PerFileHistory history = ...
- Windows JDK 1.8降级为1.7方法
转自:https://www.aliyun.com/jiaocheng/890626.html 正常情况下,JDK的升降级只需更改环境变量即可 可是到了1.8版本,Oracle使了一点小手段, 在系统 ...
- linuxmint安装卡在探测文件系统,解决方法。
今天安装linuxmint17.3,一直卡在探测文件... 解决方法如下: 执行安装程序前,打开终端执行如下命令即可. sudo umount -l /isodevic
- asp.net调用oracle存储过程
oracle内的存储过程是通过游标返回结果集的 DataTable dt = new DataTable(); OracleParameter[] paras = ]; paras[] = new O ...