性能问题，AWR High Event enq: US - contention
1.1问题现象
应用反馈业务执行SQL响应超时，需要数据库排除DB是否存在问题，创建AWR观察到top event 新增enq: US - contention ???
1.2问题分析
1） DB Time指标
2020-11-12 17:30 18:00 DB time 3310.19
2020-11-12 18:00 18:30 DB time 15883.77
2020-11-12 18:30 19:00 DB time 19463.52
2020-11-12 19:00 19:30 DB time 16001.57 2） AWR报告
Top 10 Foreground Events by Total Wait Time
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Tota Wait % DB
Event Waits Time Avg(ms) time Wait Class
------------------------------ ------------ ---- ------- ------ ----------
latch: row cache objects 41,636,274 669. 16 57.4 Concurrenc
enq: US - contention 20,130,960 291. 14 25.0 Other
latch: ges resource hash list 11,265,725 55.8 5 4.8 Other
DB CPU 33.2 2.8
buffer busy waits 3,502,842 2301 1 .2 Concurrenc
log file sync 1,831,042 1774 1 .2 Commit
library cache: mutex X 301,887 1170 4 .1 Concurrenc
latch: cache buffers chains 1,222,834 924. 1 .1 Concurrenc
latch: enqueue hash chains 365,389 222. 1 .0 Other
latch: undo global data 505,480 116. 0 .0 Other Statistic Name Time (s) % of DB Time
------------------------------------------ ------------------ ------------
sql execute elapsed time 1,165,594.2 99.8
PL/SQL execution elapsed time 611,956.6 52.4
DB CPU 33,234.1 2.8
parse time elapsed 20.2 .0
connection management call elapsed time 17.0 .0
repeated bind elapsed time 3.2 .0
hard parse elapsed time 0.0 .0
sequence load elapsed time 0.0 .0
hard parse (sharing criteria) elapsed time 0.0 .0
failed parse elapsed time 0.0 .0
DB time 1,167,811.4
background elapsed time 1,515.8
background cpu time 513.7 -> Ordered by statistic name Statistic Total per Second per Trans
-------------------------------- ------------------ -------------- -------------
db block changes 97,449,293 54,125.4 47.7
execute count 14,890,901 8,270.7 7.3
gc cr block receive time 1,277 0.7 0.0
gc cr blocks received 28,544 15.9 0.0
gc current block receive time 24,427 13.6 0.0
gc current blocks received 94,906 52.7 0.1
logons cumulative 2,788 1.6 0.0
opened cursors cumulative 3,649,945 2,027.3 1.8
parse count (total) 2,649,940 1,471.8 1.3
parse time elapsed 1,781 1.0 0.0
physical reads 18,385 10.2 0.0
physical writes 2,808,767 1,560.1 1.4
redo size 23,166,050,528 12,866,904.3 11,345.2
session cursor cache hits 2,292,253 1,273.2 1.1
session logical reads 2,394,937,749 1,330,198.0 1,172.9
user calls 8,341,907 4,633.3 4.1
user commits 1,911,204 1,061.5 0.9
user rollbacks 130,726 72.6 0.1
workarea executions - optimal 206,244 114.6 0.1
------------------------------------------------------ 3）新出现的等待事件是什么？
enq: US - contention 参考如下文档
How to correct performance issues with enq: US - contention related to undo segments (Doc ID 1332738.1)
The wait event "enq: US Contention" is associated with contention on the latch in the row cache (dc_rollback_seg). Enqueue US - Contention can become a bottle-neck for performance if workload dictates that a lot of offlined undo segments must be onlined over a short period of time. The latch on the row cache can be unable to keep up with the workload.
This can happen for a number of reasons and some scenarios are legitimate workload demands.
Solution: ensure that peaks in onlined undo segments do not happen (see workaround #2). That is not always feasible.
从里面我们能掌握的信息有，UNDO的争用, 理想化的解决方法是避免undo段的使用出现峰值？？？ 呵呵。
4）UNDO表空间的使用情况查询
SQL> select status,sum(bytes)/1024/1024/1024,TABLESPACE_NAME from dba_undo_extents group by status,TABLESPACE_NAME;
STATUS SUM(BYTES)/1024/1024/1024 TABLESPACE_NAME
--------- ------------------------- ------------------------------
EXPIRED .194824219 UNDOTBS1
EXPIRED 2.67224121 UNDOTBS2
UNEXPIRED 29.1513672 UNDOTBS1
ACTIVE 7.76220703 UNDOTBS1
UNEXPIRED 10.0617676 UNDOTBS2
GROUP_NUMBER NAME TOTAL_GB FREE_GB FREE_PERCENT STATE TYPE CARE
------------ -------------------- ---------- ---------- ------------ --------------- ------ -----
1 ARCH 500.00 499.87 99.97 CONNECTED EXTERN
2 DATA 500.00 49.74 9.95 CONNECTED EXTERN *
3 OCR 30.00 29.10 96.99 MOUNTED NORMAL
4 REDO_LOG 100.00 85.87 85.87 CONNECTED EXTERN
SQL> show parameter undo
NAME TYPE VALUE
------------------------------------ ----------- ------------------------------
_gc_undo_affinity boolean FALSE
_undo_autotune boolean FALSE
undo_management string AUTO
undo_retention integer 10800
undo_tablespace string UNDOTBS1
SQL> select 10800/60/60 from dual;
10800/60/60
-----------
3
1.3问题处理
根据查询到的UNDO情况，我们可以发现UNDOTBS1 第一个文件自扩展，当前分配7G,最大可扩展31.9xxG; UNDOTBS2 第一个文件自扩展，当前分配0.02G,最大可扩展31.9XXXG;
根据UNDO EXTENT状态可以发现，UNDOTBS1 30G处于UNEXPIRED 未过期状态，7G处于正在使用；只有少量数量处于已过期状态；
UNDOTBS2 10G处于UNEXPIRED 未过期状态，0G处于正在使用；2.6G已过期状态；
由于UNDOTBS1 有2个UNDO文件，第一个DB建好，自扩展，当前7，最大31.9G； 第二个DB建后，DBA新增的已完全分配，非自动扩展30G；
本次就是由于UNDOTBS1还有扩展的可能性，因此新的UNDO申请需要对UNDOTBS1的第一个自扩展文件进行扩展时，出现严重的争用；
扩容！！！ SQL> alter tablespace UNDOTBS1 add datafile '+DATA' size 30g;
SQL> alter tablespace UNDOTBS2 add datafile '+DATA' size 10g;
业务反馈突然好了！
收尾
select file_id,TABLESPACE_NAME,bytes/1024/1024/1024,MAXbytes/1024/1024/1024 from dba_data_files where tablespace_name in('UNDOTBS1','UNDOTBS2');
FILE_ID TABLESPACE_NAME BYTES/1024/1024/1024 MAXBYTES/1024/1024/1024
---------- ------------------------------ -------------------- -----------------------
3 UNDOTBS1 7.14355469 31.9999847
5 UNDOTBS2 .024414063 31.9999847
6 UNDOTBS1 30 0
7 UNDOTBS2 30 0
18 UNDOTBS1 30 0
19 UNDOTBS2 10 0
6 rows selected.
select file_name from dba_data_files where file_id in(3,5);
SQL> alter database datafile 3,5 autoextend off;
SQL> alter database datafile '+DATA/wglcta/datafile/undotbs1.258.1047648231' resize 10g;
SQL> alter database datafile '+DATA/wglcta/datafile/undotbs2.264.1047648343' resize 10g;
select file_id,TABLESPACE_NAME,bytes/1024/1024/1024,AUTOEXTENSIBLE,MAXbytes/1024/1024/1024 from dba_data_files where tablespace_name in('UNDOTBS1','UNDOTBS2');
FILE_ID TABLESPACE_NAME BYTES/1024/1024/1024 AUT MAXBYTES/1024/1024/1024
---------- ------------------------------ -------------------- --- -----------------------
3 UNDOTBS1 10 NO 0
5 UNDOTBS2 10 NO 0
6 UNDOTBS1 30 NO 0
7 UNDOTBS2 30 NO 0
18 UNDOTBS1 30 NO 0
19 UNDOTBS2 10 NO 0
GROUP_NUMBER NAME TOTAL_GB FREE_GB FREE_PERCENT STATE TYPE CARE
------------ -------------------- ---------- ---------- ------------ --------------- ------ -----
1 ARCH 500.00 499.87 99.97 CONNECTED EXTERN
2 DATA 500.00 26.89 5.38 CONNECTED EXTERN *
1.4事后追踪
1） 确认UNDOTBS1的使用情况，最近几个小时包括问题时段
SQL> select ts#,name from v$tablespace where name='UNDOTBS1';
TS# NAME
---------- --------------------
2 UNDOTBS1 select * from
DBA_HIST_TBSPC_SPACE_USAGE where TABLESPACE_ID=2 order by RTIME;
SNAP_ID DBID TABLESPACE_ID TABLESPACE_SIZE TABLESPACE_MAXSIZE TABLESPACE_USEDSIZE RTIME
---------- ---------- ------------- --------------- ------------------ ------------------- -------------------------
4728 476729316 2 3942400 8126462 1344368 11/12/2020 01:30:01
4729 476729316 2 3942400 8126462 1147240 11/12/2020 02:00:04
4730 476729316 2 3942400 8126462 1001072 11/12/2020 02:30:10
4731 476729316 2 3942400 8126462 923368 11/12/2020 03:00:12
4732 476729316 2 3942400 8126462 959200 11/12/2020 03:30:01
4733 476729316 2 3942400 8126462 973392 11/12/2020 04:00:02
4734 476729316 2 3942400 8126462 990800 11/12/2020 04:30:03
4735 476729316 2 3942400 8126462 1031128 11/12/2020 05:00:04
4736 476729316 2 3942400 8126462 1087312 11/12/2020 05:30:06
4737 476729316 2 3942400 8126462 1168968 11/12/2020 06:00:07
4738 476729316 2 3942400 8126462 1315032 11/12/2020 06:30:08
4739 476729316 2 3942400 8126462 1544672 11/12/2020 07:00:10
4740 476729316 2 3942400 8126462 1795568 11/12/2020 07:30:11
4741 476729316 2 3942400 8126462 2033392 11/12/2020 08:00:12
4742 476729316 2 3942400 8126462 2305024 11/12/2020 08:30:14
4743 476729316 2 3942400 8126462 2605192 11/12/2020 09:00:15
4744 476729316 2 3942400 8126462 2872568 11/12/2020 09:30:16
4745 476729316 2 3942400 8126462 3098480 11/12/2020 10:00:18
4746 476729316 2 3942400 8126462 3246192 11/12/2020 10:30:06
4747 476729316 2 3942400 8126462 3414576 11/12/2020 11:00:11
4748 476729316 2 3942400 8126462 3577952 11/12/2020 11:30:14
4749 476729316 2 3942400 8126462 3580104 11/12/2020 12:00:17
4750 476729316 2 3942400 8126462 3458504 11/12/2020 12:30:19
4751 476729316 2 3942400 8126462 3300168 11/12/2020 13:00:22
4752 476729316 2 3942400 8126462 3024576 11/12/2020 13:30:28
4753 476729316 2 3942400 8126462 2831064 11/12/2020 14:00:29
4754 476729316 2 3942400 8126462 2698656 11/12/2020 14:30:31
4755 476729316 2 3942400 8126462 2644232 11/12/2020 15:00:32
4756 476729316 2 3942400 8126462 2669832 11/12/2020 15:30:37
4757 476729316 2 3942400 8126462 2780168 11/12/2020 16:00:38
4758 476729316 2 3942400 8126462 2911240 11/12/2020 16:30:40
4759 476729316 2 3942400 8126462 2985720 11/12/2020 17:00:41
4760 476729316 2 3942400 8126462 3577632 11/12/2020 17:30:42
4761 476729316 2 3942400 8126462 3697616 11/12/2020 18:00:44
4762 476729316 2 3942400 8126462 3812664 11/12/2020 18:30:44
4763 476729316 2 4071680 8126462 4055680 11/12/2020 19:01:35
4764 476729316 2 8800640 12058622 4935560 11/12/2020 19:30:12
695 rows selected. 参考
https://yq.aliyun.com/articles/283841 我们查询ASH视图看看当时的情况： SELECT D.SQL_ID,CHR(BITAND(P1, -16777216) / 16777215) ||
CHR(BITAND(P1, 16711680) / 65535) "Lock",
BITAND(P1, 65535) "Mode", COUNT(1),COUNT(DISTINCT d.session_id )
FROM DBA_HIST_ACTIVE_SESS_HISTORY D
WHERE D.SAMPLE_TIME BETWEEN TO_DATE('2020-11-12 16:00:00', 'YYYY-MM-DD HH24:MI:SS') AND
TO_DATE('2020-11-12 19:00:00', 'YYYY-MM-DD HH24:MI:SS')
AND D.EVENT = 'enq: US - contention'
GROUP BY D.SQL_ID,(CHR(BITAND(P1, -16777216) / 16777215) ||
CHR(BITAND(P1, 16711680) / 65535)),(BITAND(P1, 65535)); SQL_ID Lock Mode COUNT(1) COUNT(DISTINCTD.SESSION_ID)
------------- -------- ---------- ---------- ---------------------------
q US 6 5472 1161
c US 6 33943 1340select distinct sql_id,sql_text from v$sql where sql_id in('xx')
order by 1;
2）AWR对比报告，看看能否观察一些其它信息
AWR对比报告
@?/rdbms/admin/awrddrpt.sql
Snapshot Set Begin Snap Id Begin Snap Time End Snap Id End Snap Time Avg Active Users Elapsed Time (min) DB time (min)
------------ -------------- ------------------------- ------------ ------------------------- --------------------------
1st 4761 12-Nov-20 18:00:43 (Thu) 4762 12-Nov-20 18:30:00 (Thu) 542.2 29.3 15,883.8
2nd 4760 12-Nov-20 17:30:41 (Thu) 4761 12-Nov-20 18:00:43 (Thu) 113.0 30.0 3,310.2
~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~~~~~~~~~~~~~~~~~~~~~~~~~ %Diff: -79.7% 2.5% -79.2%
How to correct performance issues with enq: US - contention related to undo segments (Doc ID 1332738.1)
IF: Undo Wait Event - Enq: US - contention (Doc ID 1951680.1)
Update to latest version or apply the Patch 14226599 (included in 11.2.0.4 patchset and above) and Patch 17720709 (included in 12.2 and above)
Setting _ROLLBACK_SEGMENT_COUNT to a high number to keep undo segments online:
ALTER SYSTEM SET "_rollback_segment_count"=<n>;
Note: In databases with high query activity, particularly parallel query and a high setting for _ROLLBACK_SEGMENT_COUNT, you can expect to see wait contention on the row cache for DC_ROLLBACK_SEGS. It is highly recommended in these environments where setting _ROLLBACK_SEGMENT_COUNT to a high value (10s of thousands and higher) apply the patch for Bug:14226599 (as mentioned in solution 1). This will increase the hash buckets on the DC_ROLLBACK_SEGS row cache to help alleviate latch contention.
A fix to Bug:7291739 is to set a new hidden parameter, _HIGHTHRESHOLD_UNDORETENTION to set a high threshold for undo retention completely distinct from maxquerylen:
ALTER SYSTEM SET "_highthreshold_undoretention"=<n>;
性能问题，AWR High Event enq: US - contention的更多相关文章
- Analyzing 'enq: HW - contention' Wait Event (Doc ID 740075.1)
Analyzing 'enq: HW - contention' Wait Event (Doc ID 740075.1) In this Document Symptoms Cause ...
- enq: TM - contention一例
今天下午,有台服务器出现异常,响应特别慢,io等待奇高,awr top 5事件如下: 经回查ash,找到了造成这些事件的sql语句,如下: select * from v$active_session ...
- 【等待事件】序列等待事件总结（enq: SQ - contention、row cache lock、DFS lock handle和enq: SV - contention）
[等待事件]序列等待事件总结(enq: SQ - contention.row cache lock.DFS lock handle和enq: SV - contention) 1 BLOG文档结 ...
- Oracle 性能之 Enq: CF - contention
Oracle 性能之 Enq: CF - contention Table of Contents 1. 原因 2. 解决问题 2.1. 针对持有锁进程类型处理 2.1.1. 查看持有锁会话的进程类型 ...
- 【故障处理】队列等待之enq IV - contention案例
[故障处理]队列等待之enq IV - contention案例 1.1 BLOG文档结构图 1.2 前言部分 1.2.1 导读和注意事项 各位技术爱好者,看完本文后,你可以掌握如下的技能,也 ...
- 等待事件：enq: HW - contention和enq: TM - contention
今天生成了生产库前几日的AWR报告,发现等待事件中出现了一个陌生的event--enq: HW - contention,google一下是ASSM(Auto Segment Space Manage ...
- enq: SQ - contention
--每分钟操作 SQL> select sql_id, mi, count(mi) 2 from (select event, sql_id, to_char(sample_time ...
- oracle 11g enq: JI – contention等待事件
最近使用物化视图同步的环境在大量刷新的时候频繁出现enq: JI – contention等待事件,经查: JI enqueue is acquired in exclusive mode on th ...
- undo表空间居高不下和enq: US - contention
这几天遇到一个错误,我也不知道算不算错误吧,因为没有报错,只是在那突然的短短2分钟内表的操作突然降低了,导致了该软件重新启动.查看alert日志没有报错,而是在ASH里找到了TOP SQL框有一个这样 ...
随机推荐
- git 查看本地分支和切换本地分支的命令
查看本地分支,和当前所在的分支 git branch -vv git checkout developer 切换到developer分支
- WGS-84 to Web mercator
function mercator_encrypt (wgsLat, wgsLon) { var x = wgsLon * 20037508.34 / 180.; var y = Math.l ...
- kafka生产者 消费者
publisher.php <?php $rk = new RdKafka\Producer(); $rk->addBrokers("192.168.33.50"); ...
- 【转】Python3 正则表达式特殊符号及用法（详细列表）
转载自鱼c论坛:https://fishc.com.cn/forum.php?mod=viewthread&tid=57691&extra=page%3D1%26filter%3Dty ...
- Linux 动态库加载
动态库运行时搜索顺序 1.LD_PRELOAD LD_PRELOAD是一个环境变量,用于动态库加载,动态库加载的优先级最高: 2.-wl,-rpath 编译目标代码时指定的动态库搜索路径(指的是用-w ...
- PHP实现Bitmap的探索 - GMP扩展使用
原文地址:https://blog.fanscore.cn/p/22/ 一.背景 公司当前有一个用户群的系统,核心功能是根据不同的条件组去不同的业务线中get符合条件的uid列表,然后存到redis中 ...
- 联赛模拟测试22 B. 分组配对 倍增+二分
题目描述 分析 首先,容易发现一个小组内的最优配对方式(能得到最大综合实力的方式) 一定是实力值最大的男生和最大的女生配对,次大的和次大的配对,以此类推. 但是每次新插入一个值时,需要用 \(nlog ...
- cp: cannot stat: filepath Permission denied
在执行 cp -r frompath topath时,报错cp: cannot stat: frompath Permission denied. 百度,google都没有找到解决方案,无意中发现,原 ...
- 欧拉计划第十一题--java实现
先做个说明,里面所有的题都是我自己写的解题报告,由于我的能力有限,是个刚学java的小白,有很多不足的地方,还望各位大佬不奢赐教,谢谢! Largest product in a grid In th ...
- 如何分析、排查、解决Redis变慢问题？
关于如何分析.排查.解决Redis变慢问题,根据实践总结了一些清单如下: 1.使用复杂度过高的命令(例如SORT/SUION/ZUNIONSTORE/KEYS),或一次查询全量数据(例如LRANGE ...