ORA-27301: OS failure message: Not enough space

OS:HP-UNIX

ORA-27300: OS system dependent operation:fork failed with status: 12 

ORA-27301: OS failure message: Not enough space 

ORA-27302: failure occurred at: skgpspawn3

swapinfo -mt

发现swap已经100percent了。sqlplus无法连接到数据库中。应用这边的人找过来了，说他们应用无法连上数据库了。

查看系统日志：

more  /var/adm/syslog/syslog.log

发现系统频繁出现无法分配内存的错误

应用那边说今天早上他们把WebLogic中jdbc调大到了200，按说应该是没有问题，预计是jdbc没有释放，叫他们把连接数调整到原样，swap恢复到88percent。数据库恢复正常

看mos原文：

Troubleshooting Guide ORA-3136: WARNING Inbound Connection Timed Out (Doc ID 465043.1)

To Bottom

---

In this Document

Purpose

Troubleshooting Steps

References

---

APPLIES TO:

Oracle Net Services – Version 10.2.0.5 to 11.2.0.4 [Release 10.2 to 11.2]

Information in this document applies to any platform.

***Checked for relevance on 21-APR-2014***

PURPOSE

Troubleshooting guide for "ORA-3136  WARNING inbound connection timed out" seen in the alert log.

TROUBLESHOOTING STEPS

The "WARNING: inbound connection timed out (ORA-3136)" in the alert log indicates that the client was not able to complete  the  authentication process
within the period of time specified by the parameter SQLNET.INBOUND_CONNECT_TIMEOUT.

You might also see the errors ORA-12170 or TNS-12535 in the sqlnet.log that is generated on the server.

Check $ORACLE_HOME/network/log for this file.  This entry should contain client address from which the timeout originated and may be helpful in determining how to troubleshoot the issue.  Some applications or JDBC thin driver applications may not have these
details.  The sqlnet.log file is not generated  by default in 11g and newer.

From 10.2.0.1 onwards the default setting for the parameter SQLNET.INBOUND_CONNECT_TIMEOUT is 60 seconds.  If the client is not able to authenticate within 60 seconds, the warning would appear in the alert log and the client connection will be terminated.

Note: This timeout restriction was introduced to combat Denial of Service (DoS) attack whereby malicious clients attempt to flood database servers with connect requests that consumes
resources.

The following are the most likely reasons for this error -

1. Server gets a connection request from a malicious client which is not supposed to connect to the database. 
   In this case the error thrown would be the expected and desirable behavior. You can get the client address for which the error was thrown in the sqlnet.log file that is local to the database.
2. The server receives a valid client connection request but the client takes a long time to authenticate
   more than the default 60 seconds.
3. The DB server is heavily loaded due to which it cannot finish the client logon within the timeout specified.

To understand what is causing this issue, following checks can be done 

The default value of 60 seconds is good enough in most conditions for the database server to authenticate a client connection. If it is taking longer, then it's worth checking
the following items before implementing the workaround:

1. Check whether local connection on the database server is successful & quick.

2. If local connections are quick ,then check for underlying network delay with the help of your network administrator.

3. Check whether your Database performance has degraded in anyway.

4. Check alert log for any critical errors for eg, ORA-600 or ORA-7445 and get them  resolved first. 

These critical errors might have triggered the slowness of the database server.

It is often necessary to increase the values for INBOUND CONNECT TIMEOUT at  both the listener and the database in order to resolve this issue.    It is usually advisable to set the database (sqlnet.ora) value slightly higher than the listener (listener.ora).   
The authentication process is more demanding for the database than the listener.

To set these parameters to use values higher than the default of 60 seconds, follow these instructions and restart the listener.  There is no need to restart Oracle:

Edit the server side sqlnet.ora file and add this parameter:

SQLNET.INBOUND_CONNECT_TIMEOUT=<n>  Where <n> is the value in seconds.

E.g.:

SQLNET.INBOUND_CONNECT_TIMEOUT = 120

Edit the listener.ora file and add this parameter:

INBOUND_CONNECT_TIMEOUT_<listenername> = <n>  Again, where <n> is the timeout value in seconds.

For example if the listener name is LISTENER then use:

INBOUND_CONNECT_TIMEOUT_LISTENER = 110

From Oracle version 10.2.0.1 onwards the default value of INBOUND_CONNECT_TIMEOUT_<listenername> is 60 seconds. For previous releases it is zero or OFF by default.

How to check whether inbound timeout is active for the listener and database server:

For example,  INBOUND_CONNECT_TIMEOUT_<listener_name> =110

You can check whether the parameter is active or not by simply doing telnet to the listener port.

$ telnet <database server IP> <listener port> 

for eg.

$ telnet 123.23.23.23 1521

The telnet session should disconnect after 110 seconds which indicates that the inbound connection timeout for the listener is active.

Alternatively, check at the LSNRCTL prompt using:

LSNRCTL>set current_listener <listener_name>

LSNRCTL>show inbound_connect_timeout

To check whether database server SQLNET.INBOUND_CONNECT_TIMEOUT is active: 

Eg.

SQLNET.INBOUND_CONNECT_TIMEOUT=120

a. For Dedicated server setup, enable the support level sqlnet server tracing will show the timeout value as below:

niotns: Enabling CTO, value=120000 (milliseconds) <== 120 seconds

niotns: Not enabling dead connection detection.

niotns: listener bequeathed shadow coming to life…

b. For shared Server setup, 

$ telnet <database server IP> <dispatcher port>

Example.

$ telnet 123.23.23.23  51658

The telnet session should disconnect after 120 seconds which indicates that the sqlnet.inbound_connect_timeout is active.

If you have further questions  regarding this issue then please create a Service Request in My Oracle Support and provide the following information:

a. Client and matching server traces generated at support level.

Note 395525.1 How to Enable Oracle Net Client,Server,Listener,Kerberos and External procedure Tracing from Net Manager
(netmgr):

Note 374116.1 How to Match Oracle Net Client and Server Trace Files

b. Upload sqlnet.ora, listener.ora Sqlnet.log, & Alert_<sid>.log from database server

REFERENCES

NOTE:395525.1 - How to Enable Oracle
SQLNet Client , Server , Listener , Kerberos and External procedure Tracing from Net Manager

id=730066.1&parent=DOCUMENT&sourceId=465043.1" style="">NOTE:730066.1 - Troubleshooting ORA-3135/ORA-3136 Connection Timeouts Errors – Database Diagnostics

NOTE:345197.1 - Connections that Used to Work in Oracle 10gR1 Now Intermittently Fail with ORA-3113,ORA-3106 or ORA-3136
from 10.2 Onwards

id=374116.1&parent=DOCUMENT&sourceId=465043.1" style="">NOTE:374116.1 - How to Match Oracle Net Client and Server Trace Files