library cache lock和cursor: pin S wait on X等待
1.现象:
客户10.2.0.4 RAC环境，出现大量的library cache lock和cursor: pin S wait on X等待，经分析是由于统计信息收集僵死导致的。
数据库在8点到9点期间，数据库两个节点都存在明显的cursor: pin S wait on X和library cache lock的等待：
TOP 5 EVENT:
|
Event |
Waits |
Time(s) |
Avg Wait(ms) |
% Total Call Time |
Wait Class |
|
cursor: pin S wait on X |
1,573,056 |
30,651 |
19 |
146.2 |
Concurrency |
|
library cache lock |
31,757 |
7,009 |
221 |
33.4 |
Concurrency |
|
CPU time |
6,416 |
30.6 |
|||
|
DFS lock handle |
12,381 |
2,979 |
241 |
14.2 |
Other |
|
latch: library cache |
1,646 |
1,974 |
1,199 |
9.4 |
Concurrency |
|
Event |
Waits |
Time(s) |
Avg Wait(ms) |
% Total Call Time |
Wait Class |
|
cursor: pin S wait on X |
17,586,451 |
342,437 |
19 |
812.1 |
Concurrency |
|
library cache lock |
63,657 |
30,153 |
474 |
71.5 |
Concurrency |
|
CPU time |
3,820 |
9.1 |
|||
|
db file sequential read |
241,761 |
1,094 |
5 |
2.6 |
User I/O |
|
inactive session |
两个节点的等待现象基本一致，而节点1上还存在明显的DFS lock handle等待事件。
通过分析ASH信息，发现library cache lock和cursor: pin S wait on X等待基本上都是6点之后才开始出现：
2.分析原因
- SELECT trunc(sample_time, 'hh24') TIME, COUNT(*)
- FROM WRH$_ACTIVE_SESSION_HISTORY ash, wrh$_event_name en
- WHERE ash.event_id = en.event_id
- AND sample_time>= to_timestamp('', 'yyyymmdd')
- AND event_name IN ('cursor: pin S wait on X', 'library cache lock')
- GROUP BY trunc(sample_time, 'hh24')
- ORDER BY 1;
查询事件发生的情况
而观察出现cursor: pin S wait on X和library cache lock等待会话的BLOCKING SESSION，发现大部分处于等待的会话被会话37锁定：
- SELECT to_char(sample_time, 'hh24:mi:ss.ff') TIME, session_id sid,
- decode(en.event_name, 'cursor: pin S wait on X', to_number(substr(ltrim(to_char(p2, '0xxxxxxxxxxxxxxx')), 1, 8), 'xxxxxxxx'), blocking_session) b_sid,
- en.event_name
- FROM WRH$_ACTIVE_SESSION_HISTORY ash, wrh$_event_name en
- WHERE ash.event_id = en.event_id(+)
- AND instance_number = 1
- AND sample_time >= to_timestamp('', 'yyyymmddhh24mi')
- AND sample_time <= to_timestamp('', 'yyyymmddhh24mi')
- ORDER BY 1;
查询堵塞事件
- TIME SID B_SID EVENT_NAME
- ------------------ ---------- ---------- ------------------------------
- 08:00:09.569 457 4294967291 latch: library cache
- 08:00:09.569 52 981 cursor: pin S wait ON X
- 08:00:09.569 168 73 library cache LOCK
- 08:00:09.569 1763 4294967291 SQL*Net message FROM client
- 08:00:09.569 123 4294967295 kksfbc child completion
- 08:00:09.569 476 4294967291 kst: async disk IO
- 08:00:09.569 500 4294967292 db file sequential READ
- 08:00:09.569 327 4294967291 SQL*Net message TO client
- 08:00:09.569 233 4294967291 db file scattered READ
- 08:00:09.569 73 73 DFS LOCK handle
- 08:00:09.569 1889 4294967291 gc CURRENT block 2-way
- 08:00:09.569 1969 981 cursor: pin S wait ON X
- 08:00:09.569 857 981 cursor: pin S wait ON X
- 08:00:09.569 812 168 cursor: pin S wait ON X
- 08:00:09.569 981 73 library cache LOCK
- 08:00:19.669 52 981 cursor: pin S wait ON X
- 08:00:19.669 812 168 cursor: pin S wait ON X
- 08:00:19.669 857 981 cursor: pin S wait ON X
- 08:00:19.669 1969 981 cursor: pin S wait ON X
- 08:00:19.669 233 4294967291 db file scattered READ
- 08:00:19.669 500 4294967292 db file sequential READ
- 08:00:19.669 1763 4294967291 db file sequential READ
- 08:00:19.669 476 4294967291 kst: async disk IO
- 08:00:19.669 457 4294967291 kksfbc child completion
- 08:00:19.669 73 73 DFS LOCK handle
- 08:00:19.669 654 4294967291 SQL*Net message TO client
- 08:00:19.669 168 73 library cache LOCK
- 08:00:19.669 981 73 library cache LOCK
- 08:00:29.769 1969 981 cursor: pin S wait ON X
- 08:00:29.769 52 981 cursor: pin S wait ON X
- 08:00:29.769 457 4294967291 kksfbc child completion
- 08:00:29.769 981 73 library cache LOCK
- 08:00:29.769 168 73 library cache LOCK
- 08:00:29.769 500 4294967292 db file sequential READ
- 08:00:29.769 73 73 DFS LOCK handle
- 08:00:29.769 654 4294967291 SQL*Net message TO client
- 08:00:29.769 327 4294967291 SQL*Net message TO client
- 08:00:29.769 1763 4294967291 db file sequential READ
- 08:00:29.769 857 981 cursor: pin S wait ON X
- 08:00:29.769 123 4294967295 latch: library cache
- 08:00:29.769 196 4294967295 latch: library cache
- 08:00:29.769 233 4294967295 latch: library cache
- 08:00:29.769 812 168 cursor: pin S wait ON X
- 08:00:29.769 476 4294967291 kst: async disk IO
- 08:00:39.879 476 4294967291 kst: async disk IO
- 08:00:39.879 658 4294967291 db file sequential READ
- 08:00:39.879 500 4294967292 db file sequential READ
- 08:00:39.879 1969 981 cursor: pin S wait ON X
- 08:00:39.879 857 981 cursor: pin S wait ON X
- 08:00:39.879 812 168 cursor: pin S wait ON X
- 08:00:39.879 52 981 cursor: pin S wait ON X
- 08:00:39.879 981 73 library cache LOCK
- 08:00:39.879 168 73 library cache LOCK
- 08:00:39.879 1849 4294967291 SQL*Net message TO client
- 08:00:39.879 327 4294967291 SQL*Net message TO client
- 08:00:39.879 235 4294967291 rdbms ipc reply
- 08:00:39.879 19 4294967291 rdbms ipc reply
- 08:00:39.879 73 73 DFS LOCK handle
- 08:00:39.879 123 4294967294 enq: WF - contention
- 08:00:49.987 476 4294967291 kst: async disk IO
- 08:00:49.987 500 4294967292 db file sequential READ
- 08:00:49.987 382 4294967291 gc cr block busy
- 08:00:49.987 1969 981 cursor: pin S wait ON X
- 08:00:49.987 857 981 cursor: pin S wait ON X
- 08:00:49.987 812 168 cursor: pin S wait ON X
- 08:00:49.987 235 4294967295 kksfbc child completion
- 08:00:49.987 658 4294967295 latch: library cache
- 08:00:49.987 19 4294967291 latch: library cache
- 08:00:49.987 981 73 library cache LOCK
- 08:00:49.987 168 73 library cache LOCK
- 08:00:49.987 1889 4294967291 SQL*Net message FROM client
- 08:00:49.987 73 73 DFS LOCK handle
- 08:00:49.987 52 981 cursor: pin S wait ON X
查询结果
可以判断，导致数据库实例HANG死的原因是因为SID为73的进程：
- SELECT to_char(sample_time, 'hh24:mi:ss'), program, action, event_name
- FROM wrh$_active_session_history ash, wrh$_event_name en
- WHERE session_id = 73
- AND ash.event_id = en.event_id(+)
- AND sample_time >= to_date('', 'yyyymmddhh24mi')
- AND sample_time <= to_date('', 'yyyymmddhh24mi')
- ORDER BY 1;
查询该会话运行的程序
- TO_CHAR( PROGRAM ACTION EVENT_NAME
- -------- ---------------------------------------- ------------------------------ ---------------------------
- 05:50:07 oracle@orcldb5 (J002) GATHER_STATS_JOB db file sequential READ
- 05:50:17 oracle@orcldb5 (J002) GATHER_STATS_JOB db file sequential READ
- 05:50:27 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:50:37 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:50:47 oracle@orcldb5 (J002) GATHER_STATS_JOB gc cr block 2-way
- 05:50:57 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:51:08 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:51:18 oracle@orcldb5 (J002) GATHER_STATS_JOB db file scattered READ
- 05:51:28 oracle@orcldb5 (J002) GATHER_STATS_JOB gc cr multi block request
- 05:51:38 oracle@orcldb5 (J002) GATHER_STATS_JOB direct path WRITE temp
- 05:51:48 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:51:58 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:52:08 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:52:18 oracle@orcldb5 (J002) GATHER_STATS_JOB db file scattered READ
- 05:52:28 oracle@orcldb5 (J002) GATHER_STATS_JOB db file sequential READ
- 05:52:38 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:52:49 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:52:59 oracle@orcldb5 (J002) GATHER_STATS_JOB db file scattered READ
- 05:53:09 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:53:19 oracle@orcldb5 (J002) GATHER_STATS_JOB db file scattered READ
- 05:53:29 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:53:39 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:53:49 oracle@orcldb5 (J002) GATHER_STATS_JOB gc CURRENT GRANT 2-way
- 05:53:59 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:54:09 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:54:20 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:54:30 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:54:40 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:54:50 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:55:00 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:55:10 oracle@orcldb5 (J002) GATHER_STATS_JOB db file sequential READ
- 05:55:20 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:55:30 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:55:40 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:55:51 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:56:01 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:56:11 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:56:21 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:56:31 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:56:41 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:56:51 oracle@orcldb5 (J002) GATHER_STATS_JOB gc CURRENT GRANT 2-way
- 05:57:01 oracle@orcldb5 (J002) GATHER_STATS_JOB library cache pin
- 05:57:11 oracle@orcldb5 (J002) GATHER_STATS_JOB DFS LOCK handle
- 05:57:21 oracle@orcldb5 (J002) GATHER_STATS_JOB DFS LOCK handle
- 05:57:32 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:57:42 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:57:52 oracle@orcldb5 (J002) GATHER_STATS_JOB DFS LOCK handle
- 05:58:02 oracle@orcldb5 (J002) GATHER_STATS_JOB db file sequential READ
- 05:58:12 oracle@orcldb5 (J002) GATHER_STATS_JOB db file sequential READ
- 05:58:22 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:58:32 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:58:42 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:58:52 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:59:03 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:59:13 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:59:23 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:59:33 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 05:59:43 oracle@orcldb5 (J002) GATHER_STATS_JOB db file sequential READ
- 05:59:53 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 06:00:03 oracle@orcldb5 (J002) GATHER_STATS_JOB DFS LOCK handle
- 06:00:13 oracle@orcldb5 (J002) GATHER_STATS_JOB latch: library cache
- 06:00:23 oracle@orcldb5 (J002) GATHER_STATS_JOB DFS LOCK handle
- 06:00:34 oracle@orcldb5 (J002) GATHER_STATS_JOB DFS LOCK handle
- 06:00:44 oracle@orcldb5 (J002) GATHER_STATS_JOB DFS LOCK handle
- 06:00:54 oracle@orcldb5 (J002) GATHER_STATS_JOB
查询结果
根据MOS文档，Bug 6011045 – DBMS_STATS causes deadlock between ‘cursor: pin S wait on X’ and ‘library cache lock’ [ID 6011045.8]，这个问题是RAC环境中Oracle收集统计信息的BUG，在自动收集数据字典信息时，可能会出现进程HANG死，并导致大量的library cache lock和cursor: pin S wait on X等待，此外还可能出现DFS lock handle以及row cache lock的等待。
这个BUG确认影响版本包括10.2.0.4和10.2.0.5，这个BUG Oracle在10.2.0.5.5 PSU中进行了修复。
library cache lock和cursor: pin S wait on X等待的更多相关文章
- 关于library cache lock和row cache lock产生的常见原因
这两个等待事件其实很少出现在top5列表中,一般都没什么印象,在此整理记录以便以后查阅. 常见的library cache lock产生的原因在<高级OWI与Oracle性能调查>这本书和 ...
- 如何使用event 10049分析定位library cache lock and library cache pin
Oracle Library Cache 的 lock 与 pin 说明 一. 相关的基本概念 之前整理了一篇blog,讲了Library Cache 的机制,参考: Oracle Library c ...
- impdp时卡住,DW等待library cache lock
同事反映impdp时在SCHEMA_REPORT/TYPE/TYPE_SPEC步骤卡住,1个多小时后也没有响应, 查下v$session: select program,sid, event,bloc ...
- Library cache lock/pin详解
Library cache lock/pin 一.概述 ---本文是网络资料加metalink 等整理得来一个实例中的library cache包括了不同类型对象的描述,如:游标,索引,表,视图,过程 ...
- 外键约束列并没有导致大量建筑指数library cache pin/library cache lock
外键约束列并没有导致大量建筑指数library cache pin/library cache lock 清除一个100大数据表超过一百万线,发现已经运行了几个小时: delete B001.T_B1 ...
- Library Cache: Lock, Pin and Load Lock
What is "Library cache lock" ? This event controls the concurrency between clients of the ...
- Library cache lock 故障解决一例
今天收到同事电话,说是数据库中一张名为acct_balance进行操作是奇慢,第一反映是不是扫行计划有问题,结果我错了,现将过程记录下来. 用pl/sql连上数据库情况:1.对acct_balance ...
- Performance tuning library cache lock & single-task message
My colleague suddenly encountered a problem today,a Database becomes very slow , and the a lot of se ...
- 深入理解shared pool共享池之library cache的library cache lock系列四
本文了解下等待事件library cache lock,进一步理解library cache,之前的文章请见: 深入理解shared pool共享池之library cache的library ca ...
随机推荐
- 配置IIS应用程序池
IIS 6的核心在于工作进程隔离模式,而应用程序池则是定义工作进程如何进行工作,因此,可以说应用程序池是整个IIS 6的核心. 和IIS 5中只能使用单个应用程序池不同,工作在工作进程隔离模式的IIS ...
- 关于SIGPIPE导致的程序退出
http://www.cppblog.com/elva/archive/2008/09/10/61544.html 收集一些网上的资料,以便参考: http://blog.chinaunix.net/ ...
- Centos环境下部署游戏服务器－Eclipse
一直在想这篇文章该不该写,因为这篇文章更像是教你如何使用一个ide这种文章,毫无价值可言.但思来想去还是应给写.上篇文章主要说了编译原理和过程,这篇文章就是理论联系实际的典范.并且很多工程师一辈子都不 ...
- 为什么要关闭360云盘：新来的美工嫌我们logo太丑，所以就决定关闭了。这个理由怎么样
新来的美工嫌我们logo太丑,所以就决定关闭了.这个理由怎么样曾经拥有的不要忘记:不能得到的更要珍惜:属于自己的不要放弃:已经失去的留作回忆.我刚来~~~嘿嘿~~ 久经考验的,忠诚的国际宅男主义战士, ...
- 两个C++对象是否相等，要程序员自己下定义，通常是覆盖==操作符
我曾经好多年对Java的==和equals的区别和联系搞不清楚,后来搞清楚了,笔记在这里: http://www.cnblogs.com/findumars/p/3240761.htmlhttp:// ...
- nasm fasm yasm 还是masm、gas
留个爪,稍后学习 选择编译器nasm?fasm?yasm?还是masm.gas或其他? 前面三个是免费开源的汇编编译器,总体上来讲都使用Intel的语法.yasm是在nasm的基础上开发的,与nasm ...
- Android Calendar获取年月日时分秒毫秒
开始使用new Date()测试,并用通过date.getMonth(),和date.getDay()获取,不过后来发现这两个访求是jdk1.1版本的,现在已经不用了,而且结果也不正确. ; int ...
- spring.net异常处理
接下来我们看一下SpringNetAop层的内容:此层主要是实现一些AOP的代码,以及特性和一个统一调用spring.net的类:记录日志我们使用Log4Net来实现: 1:Aspects 文件夹里存 ...
- opencv实现KNN手写数字的识别
人工智能是当下很热门的话题,手写识别是一个典型的应用.为了进一步了解这个领域,我阅读了大量的论文,并借助opencv完成了对28x28的数字图片(预处理后的二值图像)的识别任务. 预处理一张图片: 首 ...
- [HDOJ3466]Proud Merchants（贪心+01背包）
题目链接:http://acm.hdu.edu.cn/showproblem.php?pid=3466 n个商人,每个商人有一个物品,物品有价格p.价值v还有一个交易限制q.q的意义是假如你现在拥有的 ...