dbcp连接池不合理的锁导致连接耗尽

应用报错,表象来看是连接池爆满了。

org.springframework.transaction.CannotCreateTransactionException: Could not open JDBC Connection for transaction; nested exception is org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted
        at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:241) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]
        at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82) [dubbo-2.5.3.jar:2.5.3]
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) [na:1.6.0_33]
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) [na:1.6.0_33]
        at java.lang.Thread.run(Thread.java:662) [na:1.6.0_33]
Caused by: org.apache.commons.dbcp.SQLNestedException: Cannot get a connection, pool exhausted
       at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:103) ~[commons-dbcp.jar:1.2.1]
        at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540) ~[commons-dbcp.jar:1.2.1]
        at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74) ~[crmpub-jdbc-1.0.jar:1.0]
        at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203) ~[spring-jdbc-3.2.2.RELEASE.jar:3.2.2.RELEASE]
        ... 32 common frames omitted
Caused by: java.util.NoSuchElementException: Timeout waiting for idle object
        at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1174) ~[commons-pool-1.6.jar:1.6]
        at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74) ~[commons-dbcp.jar:1.2.1]
        at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95) ~[commons-dbcp.jar:1.2.1]
        ... 35 common frames omitted

应用堆栈，几十条线程都阻塞到dbcp池的PoolableConnectionFactory.makeObject方法了，此方法在等待0x0000000709a638a0对象锁，但0x0000000709a638a0对象一直被其中一条线程执行PoolableConnectionFactory.makeObject时锁了，而且这条线程获取到锁后还阻塞住了，所以导致后面几十条线程都BLOCKED了。

"DubboServerHandler-132.121.95.136:28101-thread-300" daemon prio=10 tid=0x00007f02b4136800 nid=0x4954 waiting for monitor entry [0x00007f0286018000]
   java.lang.Thread.State: BLOCKED (on object monitor)
    at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
    - waiting to lock <0x0000000709a638a0> (a org.apache.commons.dbcp.PoolableConnectionFactory)
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
    at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
    at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74)
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
    at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
    at com.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74)
    at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
    at com.eshore.crm.service.sysmgr.sca.impl.SysMgrServiceImpl$$EnhancerByCGLIB$$bfd71326.qryBulletinList(<generated>)
    at com.alibaba.dubbo.common.bytecode.Wrapper101.invokeMethod(Wrapper101.java)
    at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
    at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
    at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
    at com.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
    at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
    at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
    at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - <0x000000070a120f00> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

"DubboServerHandler-132.121.95.136:28101-thread-290" daemon prio=10 tid=0x00007f03203da800 nid=0x4948 runnable [0x00007f0286a22000]
   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
    at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
    at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
    - locked <0x0000000715a9ce70> (a com.mysql.jdbc.util.ReadAheadInputStream)
    at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3140)
    at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:597)
    at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085)
    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2494)
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527)
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309)
    - locked <0x00000007159d1898> (a com.mysql.jdbc.JDBC4Connection)
    at com.mysql.jdbc.ConnectionImpl.<init>(ConnectionImpl.java:834)
    at com.mysql.jdbc.JDBC4Connection.<init>(JDBC4Connection.java:46)
    at sun.reflect.GeneratedConstructorAccessor78.newInstance(Unknown Source)
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
    at java.lang.reflect.Constructor.newInstance(Constructor.java:513)
    at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
    at com.mysql.jdbc.ConnectionImpl.getInstance(ConnectionImpl.java:419)
    at com.mysql.jdbc.NonRegisteringDriver.connect(NonRegisteringDriver.java:344)
    at org.apache.commons.dbcp.DriverConnectionFactory.createConnection(DriverConnectionFactory.java:37)
    at org.apache.commons.dbcp.PoolableConnectionFactory.makeObject(PoolableConnectionFactory.java:290)
    - locked <0x0000000709a638a0> (a org.apache.commons.dbcp.PoolableConnectionFactory)
    at org.apache.commons.pool.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:1188)
    at org.apache.commons.dbcp.AbandonedObjectPool.borrowObject(AbandonedObjectPool.java:74)
    at org.apache.commons.dbcp.PoolingDataSource.getConnection(PoolingDataSource.java:95)
    at org.apache.commons.dbcp.BasicDataSource.getConnection(BasicDataSource.java:540)
    at com.eshore.crmpub.jdbc.datasource.MultiDataSource.getConnection(MultiDataSource.java:74)
    at org.springframework.jdbc.datasource.DataSourceTransactionManager.doBegin(DataSourceTransactionManager.java:203)
    at org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:372)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:417)
    at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:255)
    at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:94)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
    at org.springframework.aop.aspectj.MethodInvocationProceedingJoinPoint.proceed(MethodInvocationProceedingJoinPoint.java:80)
    at com.eshore.crm.service.common.pub.aspect.DataSourceRouteAspect.around(DataSourceRouteAspect.java:74)
    at sun.reflect.GeneratedMethodAccessor253.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
    at java.lang.reflect.Method.invoke(Method.java:597)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethodWithGivenArgs(AbstractAspectJAdvice.java:621)
    at org.springframework.aop.aspectj.AbstractAspectJAdvice.invokeAdviceMethod(AbstractAspectJAdvice.java:610)
    at org.springframework.aop.aspectj.AspectJAroundAdvice.invoke(AspectJAroundAdvice.java:65)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:161)
    at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
    at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172)
    at org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:631)
    at com.eshore.crm.service.cachemgr.sca.common.DictCacheServiceImpl$$EnhancerByCGLIB$$e27e8e25.getDict(<generated>)
    at com.alibaba.dubbo.common.bytecode.Wrapper93.invokeMethod(Wrapper93.java)
    at com.alibaba.dubbo.rpc.proxy.javassist.JavassistProxyFactory$1.doInvoke(JavassistProxyFactory.java:46)
    at com.alibaba.dubbo.rpc.proxy.AbstractProxyInvoker.invoke(AbstractProxyInvoker.java:72)
    at com.alibaba.dubbo.rpc.protocol.InvokerWrapper.invoke(InvokerWrapper.java:53)
    at com.eshore.crmpub.sca.dubbo.filter.BaseFilter.invoke(BaseFilter.java:90)
    at com.alibaba.dubbo.rpc.protocol.ProtocolFilterWrapper$1.invoke(ProtocolFilterWrapper.java:91)
    at com.alibaba.dubbo.rpc.protocol.dubbo.DubboProtocol$1.reply(DubboProtocol.java:108)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.handleRequest(HeaderExchangeHandler.java:84)
    at com.alibaba.dubbo.remoting.exchange.support.header.HeaderExchangeHandler.received(HeaderExchangeHandler.java:170)
    at com.alibaba.dubbo.remoting.transport.DecodeHandler.received(DecodeHandler.java:52)
    at com.alibaba.dubbo.remoting.transport.dispatcher.ChannelEventRunnable.run(ChannelEventRunnable.java:82)
    at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
    at java.lang.Thread.run(Thread.java:662)

   Locked ownable synchronizers:
    - <0x000000070a0f9218> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)

可以看到其实获取锁的线程是阻塞在这里了，做mysql握手包的时候一直在等待读，阻塞在网络IO了。

   java.lang.Thread.State: RUNNABLE
    at java.net.SocketInputStream.socketRead0(Native Method)
    at java.net.SocketInputStream.read(SocketInputStream.java:129)
    at com.mysql.jdbc.util.ReadAheadInputStream.fill(ReadAheadInputStream.java:112)
    at com.mysql.jdbc.util.ReadAheadInputStream.readFromUnderlyingStreamIfNecessary(ReadAheadInputStream.java:159)
    at com.mysql.jdbc.util.ReadAheadInputStream.read(ReadAheadInputStream.java:187)
    - locked <0x0000000715a9ce70> (a com.mysql.jdbc.util.ReadAheadInputStream)
    at com.mysql.jdbc.MysqlIO.readFully(MysqlIO.java:3140)
    at com.mysql.jdbc.MysqlIO.readPacket(MysqlIO.java:597)
    at com.mysql.jdbc.MysqlIO.doHandshake(MysqlIO.java:1085)
    at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2494)
    at com.mysql.jdbc.ConnectionImpl.connectOneTryOnly(ConnectionImpl.java:2527)
    at com.mysql.jdbc.ConnectionImpl.createNewIO(ConnectionImpl.java:2309)

这个是dbcp1.x版本在创建新的连接对象时会把整个连接工厂类对象锁了，一旦创建过程中发生阻塞就会导致整个池都死掉。这个问题很大。
解决方案，升级dbcp2.x版本或tomcat jdbc或druid连接池，推荐druid连接池，功能强大自带监控。

dbcp连接池不合理的锁导致连接耗尽的更多相关文章

解决Mysql连接池被关闭，hibernate尝试连接不能连接的问题。 (默认mysql连接池可以访问的时间为8小时，如果超过8小时没有连接，mysql会自动关闭连接池。系统发布第二天访问链接关闭问题。
解决Mysql连接池被关闭 ,hibernate尝试连接不能连接的问题. (默认MySQL连接池可以访问的时间为8小时,如果超过8小时没有连接,mysql会自动关闭连接池. 所以系统发布第二天访问会 ...
c3p0数据库连接池原创： Java之行 Java之行 5月8日一、连接池概述实际开发中“获得连接”或“释放资源”是非常消耗系统资源的两个过程
c3p0数据库连接池原创: Java之行 Java之行 5月8日一.连接池概述实际开发中“获得连接”或“释放资源”是非常消耗系统资源的两个过程 DB连接池HikariCP为什么如此快原创: D ...
使用druid连接池的超时回收机制排查连接泄露问题
在工程中使用了druid连接池,运行一段时间后系统出现异常: Caused by: org.springframework.jdbc.CannotGetJdbcConnectionException: ...
红眼技术博客 » redis连接池红眼技术博客 » redis连接池
红眼技术博客 » redis连接池 redis连接池
使用druid连接池的超时回收机制排查连接泄露
起因:系统连接池满了 Exception wait millis 60012, active 2000, maxActive 2000, creating 0 加配置排查: <!-- 超过时间限 ...
c3p0，dbcp与druid 三大连接池的区别[转]
说到druid,这个是在开源中国开源项目中看到的,说是比较好的数据连接池.于是乎就看看.扯淡就到这. 下面就讲讲用的比较多的数据库连接池.(其实我最先接触的是dbcp这个) 1)DBCP DBCP是一 ...
DBCP连接池使用问题
问题现象: 启动应用,访问无压力,一切正常,一段时间过后,应用访问异常. 问题分析: 1.web容器线程爆满,拒绝服务.由于应用并发量大,线程响应时间长增加,线程池连接数逐步递增直到爆满,导致应用拒绝 ...
jdbc基础 (五) 连接池与数据源 DBCP以及C3P0的使用
一.连接池的概念和使用在实际应用开发中,特别是在WEB应用系统中,如果JSP.Servlet或EJB使用JDBC直接访问数据库中的数据,每一次数据访问请求都必须经历建立数据库连接.打开数据库.存取数 ...
一次Mysql连接池卡死导致服务无响应问题分析(.Net Mysql.Data 6.9.9)
问题: 进程启动后,线程数迅速上升至最小线程数后,缓慢上升(线程池限制)到数千,然后由于线程过多,CPU飙升到90%. 对外表现为Api无响应或连接超时. 背景有些数据存在于另一个机房,通过内网专线 ...

随机推荐

[bzoj1187][HNOI2007]神奇游乐园
来自FallDream的博客,未经允许,请勿转载,谢谢, 经历了一段艰辛的旅程后,主人公小P乘坐飞艇返回.在返回的途中,小P发现在漫无边际的沙漠中,有一块狭长的绿地特别显眼.往下仔细一看,才发现这是一 ...
Spring源码分析(一)--BeanProcessor
一.何谓BeanProcessor BeanProcessor是SpringFramework里非常重要的核心接口之一,我先贴出一段源代码: /* * Copyright 2002-2015 the ...
[ Java学习基础 ] Java的继承与多态
看到自己写的东西(4.22的随笔[ Java学习基础 ] Java构造函数)第一次达到阅读100+的成就还是挺欣慰的,感谢大家的支持!希望以后能继续和大家共同学习,共同努力,一起进步!共勉! ---- ...
什么是Hash？
什么是Hash? Hash中文翻译为散列,又成为"哈希",是一类函数的统称,其特点是定义域无限,值域有限.把任意长度的输入(又叫做预映射, pre-image),通过散列算法,变换 ...
一日一练-JS toString 和valueOf 方法的联系与区别
子曰:类型转换中toString 和valueOf 的联系与区别分析首先是看看ES5 的规范是如何进行说明的在这里有几个基础知识点需要了解一下: [[Class]] [[Class]] 属于Obj ...
初体验GCP，【福利300$试用金】
1.https://cloud.google.com/free/ ,填写相应信息.需要信用卡,预扣1美元. 2.一波信息填写,成功. 3.激活终端 4.创建一个Python项目,选择部署地点. 5.部 ...
Object 类
Kafka(转载)
Kafka是由LinkedIn开发的一个分布式的消息系统,使用Scala编写,它以可水平扩展和高吞吐率而被广泛使用.目前越来越多的开源分布式处理系统如Cloudera.Apache Storm.Spa ...
Zookeeper 快速入门（上）
来源:holynull, blog.leanote.com/post/holynull/Zookeeper 如有好文章投稿,请点击 → 这里了解详情 Zookeeper是Hadoop分布式调度服务,用 ...
渗透测试环境DVWA搭建
一.DVWA介绍 DVWA(Damn Vulnerable Web Application)是一个用来进行安全脆弱性鉴定的PHP/MySQL Web应用,旨在为安全专业人员测试自己的专业技能和工具提供 ...

dbcp连接池不合理的锁导致连接耗尽

dbcp连接池不合理的锁导致连接耗尽的更多相关文章

随机推荐

热门专题