shutdown immediate时 hang住 (转载)

shutdown immediate 
经常关库时hang住,在alert中有

License high water mark = 4
All dispatchers and shared servers shutdown

多等一会会出现
SHUTDOWN: Active processes prevent shutdown operation

造成这个现象的原因是(也可能是em的原因,这篇与em无关):

之前的session没有断开，而后又使用了host切换到OS提示符下，导致数据库无法正常关闭

[oracle@localhost ~]$ sqlplus scott/tiger

SQL> show user
USER is "SCOTT"
SQL> !
[oracle@localhost ~]$ echo $$
3897
[oracle@localhost ~]$ sqlplus / as sysdba

SQL> select pid,spid,p.pname,p.program from v$process p,v$session s where p.addr=s.paddr and s.sid=userenv('sid');

PID SPID                     PNAME PROGRAM
---------- ------------------------ ----- ------------------------------------------------
        27 3911                           oracle@localhost.myrh6 (TNS V1-V3)

SQL> shutdown immediate

hang住

alert内容:

Sat Jan 04 17:41:53 2014
Shutting down instance (immediate)
Stopping background process SMCO
Shutting down instance: further logons disabled
Sat Jan 04 17:41:53 2014
Stopping background process CJQ0
Stopping background process QMNC
Stopping background process MMNL
Stopping background process MMON
License high water mark = 4
All dispatchers and shared servers shutdown

Sat Jan 04 17:46:58 2014
SHUTDOWN: Active processes prevent shutdown operation

Sat Jan 04 17:52:00 2014
SHUTDOWN: Active processes prevent shutdown operation

另一终端
$ pstree -p

├─sshd(1668)─┬─sshd(1838)───sshd(1840)─┬─bash(3067)───tail(3095)
│            │                         └─bash(3106)───rlwrap(3893)───sqlplus(3894)─┬─bash(3897)───sqlplus(3910)───oracle(3911)
│            │                                                                     └─oracle(3895)

[oracle@localhost ~]$ ps -ef |grep oracle
可以发现
oracle    3895  3894  0 17:34 ?        00:00:00 [oracle]

一个fg process变成了僵尸进程

[oracle@localhost ~]$ ps -o pid,ppid,state -p 3895 
  PID  PPID S
 3895  3894 Z

kill -9 3895 是杀不掉僵尸进程的

解决办法:

1.可以再等会,会自动结束
SQL> shutdown immediate

SQL> Error 45 initializing SQL*Plus
Internal error
Hangup
[oracle@localhost ~]$ [oracle@localhost ~]$

2.kill父进程,子进程(即使是zombie)也会一起结束
ps -Al 或 ps -o pid,ppid -p pid
ppid是3894

kill -9 3894

3.hang住的终端尝试ctrl+c,或ctrl+z,或直接关了, 重新sqlplus / as sysdba
shutdown abort
startup restrict 
shutdown normal

ctrl-c 是发送 SIGINT 信号，终止一个进程
ctrl-z 是发送 SIGSTOP信号，挂起一个进程

ctrl-d不发送信号. 如果ctrl-d时当前行没有输入字符, bash把它当作EOF字符, 结束/退出.
如果ctrl-d时当前行有输入, 删除光标后的字符. 这是readline的特性, 不发送信号.

/* 以下引用 http://blog.csdn.net/leshami/article/details/9466559 */

Oracle对此给出的说明是这不是一个Oracle bug，而是使用了一个非正常的数据库关闭顺序。
建议先断开所有连接再关闭数据库，如先关闭db control，具体见下面的Oracle 文档及示例。

Doc ID 416658.1

Shutdown Immediate Hangs / Active Processes Prevent Shutdown (Doc ID 416658.1)

Applies to: 
Oracle Server - Enterprise Edition - Version: 10.1.0.2 to 11.2.0.2 - Release: 10.1 to 11.2
Oracle Server - Enterprise Edition - Version: 10.1.0.2 to 11.2.0.2   [Release: 10.1 to 11.2]
Information in this document applies to any platform.

Symptoms
The 'shutdown immediate' command hangs or is very slow

The alert log lists messages like:

SHUTDOWN: waiting for active calls to complete.ACTIVE PROCESSES PREVENT SHUTDOWN OPERATION

Cause
This is not a bug.

If the DB Control repository is running on the database target against which shutdown immediate was attempted then an incorrect 
order of events seems used.

You should stop DB Control first to get rid of all connections between DB Control and the repository database and then shutdown
the database with 'shutdown immediate'.

Current database sessions may show:

SQL> select SID, USERNAME, PROGRAM from v$session;
SID   USERNAME               PROGRAM
----- ---------------------- ----------------------------------
  243 SYSTEM                 SQL Developer
  246 SYSMAN                 OMS
  247                        oracle@lgiora09 (q001)
  248                        oracle@lgiora09 (q000)
  251 DBSNMP                 emagent@lgiora09 (TNS V1-V3)
  252 SYSMAN                 OMS
  253 SYSMAN                 OMS
  254 DBSNMP                 emagent@lgiora09 (TNS V1-V3)
  255 SYSTEM                 java.exe
  256 SYSMAN                 OMS

Clearly OMS and OEM are connected (Oracle Enterprise Manager Grid Control or DBConsole) via users SYSMAN and DBSNMP.
These sessions should be de-activated (that is to log off any OEM, OMS, SYSMAN and DBSNMP) before the shutdown immediate is attempted.

Oracle Enterprise Manager, Grid Control, Dbconsole and agents keep doing internal processing.
This may include a few PLSQL notification procedures running on the database by database control like
BEGIN EMD_NOTIFICATION.QUEUE_READY(:1, :2, :3); END;

As per internal documentation of the shutdown immediate, if there are active calls then it would wait for all the active calls to finish.

Solution
To implement the solution:

1. Given OEM connections are active (SYSMAN and DBSNMP), de-activate these sessions, i.e. by stopping the agent/DBConsole

2. Then shutdown immediate as normal

- OR -

There may be processes still running and holding locks at the time a shutdown is issued. 
Sometimes these are failed jobs or transactions, which are effectively 'zombies', which are not able to receive a signal from Oracle.

If this occurs, the only way to shutdown the database is by doing:

sql> 
shutdown abort 
startup restrict 
shutdown normal

The startup does any necessary recovery and cleanup, so that a valid cold backup can be taken afterward.

If this issue occurs frequently, it would be a good practice to see if there are any active user processes running in v$session or
v$process before shutting down the instance.

If the problem persists, and no apparent user processes are active, you can set this event prior to issuing the shutdown command 
in order to see what is happening. This will dump a systemstate every 5 minutes while shutdown is hanging

SQL> 
connect / as sysdba 
alter session set events '10400 trace name context forever, level 1'

http://blog.itpub.net/28859270/viewspace-1067155/