weblogic诊断案例-AdminServer平均1-2周崩溃

OS2台：RH5.5 64位

WEBLOGIC VERSION：9.2.3

JDK：1.5 64位

weblogic一个管理服务，4个受管（2台服务器做集群，每台服务器2个受管）。应用部署后通过一个月测试发现adminserver1-2周就崩溃一次。日志错误信息种类很多如下：

####<2014-6-25 下午05时23分35秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <weblogic.timers.TimerThread> <<WLS Kernel>> <> <> <1403688215418> <BEA-002911> <WorkManager weblogic.kernel.System failed to schedule a request due to weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
weblogic.utils.UnsyncCircularQueue$FullQueueException: Queue exceed maximum capacity of: '65536' elements
at weblogic.utils.UnsyncCircularQueue.expandQueue(UnsyncCircularQueue.java:72)
at weblogic.utils.UnsyncCircularQueue.put(UnsyncCircularQueue.java:94)
at weblogic.work.MinThreadsConstraint.add(MinThreadsConstraint.java:85)
at weblogic.work.RequestManager.addToPriorityQueue(RequestManager.java:261)
at weblogic.work.RequestManager.executeIt(RequestManager.java:235)
at weblogic.work.ServerWorkManagerImpl.schedule(ServerWorkManagerImpl.java:142)
at weblogic.timers.internal.TimerManagerImpl.execute(TimerManagerImpl.java:631)
at weblogic.timers.internal.TimerThread$Thread.run(TimerThread.java:285)
>
####<2014-6-26 上午11时44分05秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754245838> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=HomePage1.>
####<2014-6-26 上午11时44分16秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754256969> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpserver1%2CType%3DServer%22%29.>
####<2014-6-26 上午11时46分19秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754379930> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpims1%2CType%3DServer%22%29.>
####<2014-6-26 上午11时46分24秒 CST> <Warning> <netuix> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '8' for queue: 'weblogic.kernel.Default (self-tuning)'> <weblogic> <> <> <1403754384137> <BEA-423420> <Redirect is executed in begin or refresh action. Redirect url is /console/console.portal?_nfpb=true&_pageLabel=ServerConfigGeneralTabPage&handle=com.bea.console.handles.JMXHandle%28%22com.bea%3AName%3Ddxpserver1%2CType%3DServer%22%29.>
####<2014-6-28 上午07时14分45秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403910885281> <BEA-002911> <WorkManager direct failed to schedule a request due to java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分52秒 CST> <Error> <RMI> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403911072751> <BEA-080001> <Error in Dispatcher
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分57秒 CST> <Warning> <RMI> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911077647> <BEA-080004> <An error was thrown by rmi server: weblogic.jndi.internal.RootNamingNode.lookup(Ljava.lang.String;Ljava.util.Hashtable;)
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时18分08秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911088190> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时18分18秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911098656> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时32分51秒 CST> <Error> <Socket> <dxp1> <AdminServer> <ExecuteThread: '7' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403911971079> <BEA-000405> <Uncaught Throwable in processSockets
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时38分39秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912319851> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时40分17秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912417869> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时41分22秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '7' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912482057> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时41分51秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912511420> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space
>
####<2014-7-2 上午10时19分42秒 CST> <Notice> <Security> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267582796> <BEA-090082> <Security initializing using security realm myrealm.>
####<2014-7-2 上午10时19分45秒 CST> <Notice> <WebLogicServer> <dxp1> <AdminServer> <main> <<WLS Kernel>> <> <> <1404267585545> <BEA-000365> <Server state changed to STANDBY>
####<2014-7-2 上午10时19分45秒 CST> <Notice> <WebLogicServer> <dxp1> <AdminServer> <main> <<WLS Kernel>> <> <> <1404267585546> <BEA-000365> <Server state changed to STARTING>
####<2014-7-2 上午10时19分47秒 CST> <Notice> <Log Management> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '0' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267587400> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200004> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "638" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200007> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时54分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913260011> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "698" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@3da42689", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时54分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913260012> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "718" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-7-2 上午10时19分51秒 CST> <Notice> <Log Management> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267591935> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时46分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912782263> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@26f147c8", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时46分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912782267> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "624" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@2a9faf71", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时47分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912842270> <BEA-000337> <[STUCK] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "718" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@26f147c8", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-6-28 上午07时47分22秒 CST> <Error> <WebLogicServer> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403912842270> <BEA-000337> <[STUCK] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "684" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@2a9faf71", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
weblogic.server.channels.RemoteChannelServiceImpl.isShutdown(RemoteChannelServiceImpl.java:87)
weblogic.server.channels.RemoteChannelServiceImpl.access$200(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:101)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>
####<2014-7-2 上午10时19分50秒 CST> <Notice> <Log Management> <dxp1> <dxpims1> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1404267590942> <BEA-170027> <The server initialized the domain log broadcaster successfully. Log messages will now be broadcasted to the domain log.>
####<2014-6-28 上午07时17分55秒 CST> <Error> <WebLogicServer> <dxp1> <dxpserver1> <[ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911075803> <BEA-000337> <[STUCK] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "623" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@6c2416ed", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)
>

具体有：

1、

2、

####<2014-6-28 上午07时14分45秒 CST> <Critical> <WorkManager> <dxp1> <AdminServer> <ExecuteThread: '3' for queue: 'weblogic.socket.Muxer'> <<WLS Kernel>> <> <> <1403910885281> <BEA-002911> <WorkManager direct failed to schedule a request due to java.lang.OutOfMemoryError: Java heap space
java.lang.OutOfMemoryError: Java heap space
>
####<2014-6-28 上午07时17分42秒 CST> <Error> <Kernel> <dxp1> <AdminServer> <[ACTIVE] ExecuteThread: '6' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403911062026> <BEA-000802> <ExecuteRequest failed
java.lang.OutOfMemoryError: Java heap space.
java.lang.OutOfMemoryError: Java heap space

3、

####<2014-6-28 上午07时53分20秒 CST> <Error> <WebLogicServer> <dxp2> <dxpims2> <[ACTIVE] ExecuteThread: '3' for queue: 'weblogic.kernel.Default (self-tuning)'> <<WLS Kernel>> <> <> <1403913200007> <BEA-000337> <[STUCK] ExecuteThread: '2' for queue: 'weblogic.kernel.Default (self-tuning)' has been busy for "658" seconds working on the request "weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl@63de0bfc", which is more than the configured time (StuckThreadMaxTime) of "600" seconds. Stack trace:
java.net.SocketInputStream.socketRead0(Native Method)
java.net.SocketInputStream.read(SocketInputStream.java:129)
java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
java.io.BufferedInputStream.read(BufferedInputStream.java:235)
weblogic.net.http.MessageHeader.isHTTP(MessageHeader.java:220)
weblogic.net.http.MessageHeader.parseHeader(MessageHeader.java:143)
weblogic.net.http.HttpClient.parseHTTP(HttpClient.java:463)
weblogic.net.http.HttpURLConnection.getInputStream(HttpURLConnection.java:357)
weblogic.rjvm.http.HTTPClientJVMConnection.connect(HTTPClientJVMConnection.java:198)
weblogic.rjvm.http.HTTPClientJVMConnection.createConnection(HTTPClientJVMConnection.java:87)
weblogic.rjvm.ConnectionManager.createConnection(ConnectionManager.java:1753)
weblogic.rjvm.ConnectionManager.findOrCreateConnection(ConnectionManager.java:1410)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:448)
weblogic.rjvm.ConnectionManager.bootstrap(ConnectionManager.java:326)
weblogic.rjvm.RJVMManager.findOrCreateRemoteInternal(RJVMManager.java:261)
weblogic.rjvm.RJVMManager.findOrCreate(RJVMManager.java:204)
weblogic.rjvm.RJVMFinder.findOrCreateRemoteServer(RJVMFinder.java:226)
weblogic.rjvm.RJVMFinder.findOrCreate(RJVMFinder.java:189)
weblogic.rjvm.ServerURL.findOrCreateRJVM(ServerURL.java:154)
weblogic.jndi.WLInitialContextFactoryDelegate.getInitialReference(WLInitialContextFactoryDelegate.java:398)
weblogic.jndi.Environment.getInitialReference(Environment.java:237)
weblogic.server.channels.RemoteChannelServiceImpl.registerInternal(RemoteChannelServiceImpl.java:153)
weblogic.server.channels.RemoteChannelServiceImpl.access$300(RemoteChannelServiceImpl.java:46)
weblogic.server.channels.RemoteChannelServiceImpl$TimerListenerImpl.timerExpired(RemoteChannelServiceImpl.java:107)
weblogic.timers.internal.TimerImpl.run(TimerImpl.java:265)
weblogic.work.ServerWorkManagerImpl$WorkAdapterImpl.run(ServerWorkManagerImpl.java:518)
weblogic.work.ExecuteThread.execute(ExecuteThread.java:209)
weblogic.work.ExecuteThread.run(ExecuteThread.java:181)

针对这些错误做出联合整改方案具体如下：

1、针对weblogic.utils.UnsyncCircularQueue$FullQueueException通过官方文档查看为weblogic bug 需打补丁包：Patch 8179406

WebLogic - Admin Server Stops Responding With "weblogic.utils.UnsyncCircularQueue$FullQueueException" Exceptions (文档 ID 1228224.1)

打补丁步骤：

（1）修改/apps/weblogic/bea/weblogic92/common/bin/commEnv.sh,在文件开始处添加export CLASSPATH=/apps/weblogic/bea/weblogic92/Patches/p8179406.jar:$CLASSPATH
（2）将p8179406.jar补丁拷贝到apps/weblogic/bea/weblogic92/Patches目录下，然后重启WebLogic

2、针对错误3 点击Adminserver -> 协议 -> 常规 -> 启用隧道，然后重启服务

通过这两个解决方案目前观察2个月adminserver运行稳定。没有出现类似错误。

补丁下载： http://pan.baidu.com/s/1c0gr6M0

bug文档： http://pan.baidu.com/s/1c0H5r7y

可参考的连接：https://community.oracle.com/thread/694983?start=0&tstart=0

http://v-vampires.iteye.com/blog/861897

weblogic诊断案例-AdminServer平均1-2周崩溃的更多相关文章

Linux SendMail发送邮件失败诊断案例(三)
一Linux服务器突然发送不出邮件,检查了很多地方都没有发现异常,检查/var/log/maillog发现如下具体信息: Apr 12 00:36:04 mylinux sendmail[4685]: ...
深入浅出Oracle:DBA入门、进阶与诊断案例（读书笔记1）
一.数据库的启动和关闭 Oracle Server共有2部分组成:Instance和Database. Instance是指一组后台进程/线程和一块共享内存区域,而Database是指存储在磁盘上的一 ...
Linux sendmail发送邮件失败诊断案例(一)
在新服务器上测试sendmail发送邮件时,发现邮件发送不成功,检查日志文件发现如下错误(Notice:由于涉及公司服务器,邮箱等,故下面hostname.邮箱地址等信息使用xxx代替) tail - ...
Linux SendMail发送邮件失败诊断案例(二)
Linux上Sendmail经常由于一些配置问题,导致邮件发送失败,下面整理.收集了一些邮件发送失败.异常的案例. 案例1:在新服务器上测试sendmail发送邮件时,发现邮件发送不成功,检查/var ...
Netbeans8下 Weblogic EJB案例
1:接口 @Remote public interface XgmZzsNssb {} 2:实现 @Stateless(mappedName="XgmZzsNssbImpl") @ ...
Linux SendMail发送邮件失败诊断案例(四）
最近又碰到一起Linux下SendMail发送邮件失败的案例,邮件发送后,邮箱收不到具体邮件, 查看日志/var/log/maillog 发现有"DSN: User unknown" ...
深入浅出Oracle:DBA入门、进阶与诊断案例（读书笔记2)
第5章 Buffer Cache与Shared Pool原理 5.1 Buffer Cache原理 Buffer Cache是Oracle SGA中的一个重要部分,通常的数据访问和修改都需要通过Bu ...
转 Linux SendMail发送邮件失败诊断案例(四）
http://www.cnblogs.com/kerrycode/p/7826036.html
深入浅出Oracle：DBA入门、进阶与诊断案例 PDF 下载
网盘地址: 链接:https://pan.baidu.com/s/1tMFoNSUW7ICKOtmSQ5ZazA 提取码:dbnc

随机推荐

【转】ASP.NET中服务器控件Table动态生成表格及其属性介绍
下文所有内容转自开源中国:http://www.oschina.net/question/565065_86453#tags_nav ================================= ...
【转】svn服务器IP修改后,本地怎么跟新svn同步,svn relocate 操作
本文来源:http://www.kukaka.org/home/showonews/444 1.进入工作复本 cd ~/test 2.查看仓库地址(URL) svn info 路径: . URL ...
html之filedset标签
fieldset 元素可将表单内的相关元素分组,当一组表单元素放到 <fieldset> 标签内时,浏览器会以特殊方式来显示它们,它们可能有特殊的边界.3D 效果,或者甚至可创建一个子表单 ...
C＃-提取网页中的超链接
转载:http://www.wzsky.net/html/Program/net/26849.htmlusing System; using System.Xml; using System.Text ...
WCF学习心得------（二）设计和实现服务协定
设计和实现服务协定创建服务协定—WCF术语消息消息是一个独立的数据单元,它可能由几个部分组成,包括消息正文和消息头. 服务服务是一个构造,它公开一个或多个终结点,其中每个终结点都公开一个或多个 ...
java读取properties文件的内容
获得properties文件中某个key对应着的value // 路径名称 + properties的名称,不要“properties” private static final String BUN ...
TextBox(只允许输入字母或者数字)
实现如下: using System;using System.Collections.Generic;using System.Linq;using System.Text;using System ...
TMS320C54x系列DSP的CPU与外设——第2章 TMS320C54x DSP体系结构总体介绍
第2章 TMS320C54x DSP体系结构总体介绍本章介绍TMS320C54x DSP体系结构的概况,包括中央处理单元(CPU).存在器和片内外设. C54x DSP采用了高级的改进哈佛结构,用8 ...
Keepalived高可用软件的安装与配置
监听和替换多台服务器之间的来回切换一.安装tar zxvf keepalived-1.1.15.tar.gzcd keepalived-1.1.15./configure --prefix=/usr ...
Scala控制抽象
private def filesHere = (new java.io.File(".")).listFiles() def filesEnding(query: String) ...

weblogic诊断案例-AdminServer平均1-2周崩溃

weblogic诊断案例-AdminServer平均1-2周崩溃的更多相关文章

随机推荐

热门专题