[Oracle][DATAGUARD] LOGICAL STANDBY环境里，有些SEQUENCE无法应用，导致Primary和Standby无法同期
今天遇到了一个客户，问题是这样的，客户构筑了一个RACtoRAC的 LOGICAL STANDBY环境。
并用EM在监视同期情况，发现EM页面上55115和55116这两个SEQUENCE一直在应用。
首先向客户要了下列的资料
Primary端
Script to Collect Data Guard Primary Site Diagnostic Information for Version 10g and Above (Including RAC). (Doc ID 1577401.1)
Standby端
SRDC - Collect Logical Standby Database Information (Doc ID 1910065.1)
=============================
Logical Standby Database (from all Instances if RAC is used)
ALERT.LOG covering at least the Time since the last Instance Startup and the Problem
Output from Note: 1577407.1 - Script to Collect Data Guard Logical Standby Diagnostic Information for Version 10g and above (Including RAC)
Output from Note: 1296954.1 - How to monitor the progress of the logical standby
TNSNAMES.ORA, LISTENER.ORA
Latest LSPx- and ASxx-Tracefiles from Background Dump-Directory
=============================
拿到资料后，先看Standby端的DB alert log
1. 是不是Redo没有传到Standby端
[root@standby ~]# grep 'RFS LogMiner: Registered logfile.*thread_1' alert_mykartes.log ★从Standby端抽取传输信息
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55109.346.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55110.347.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55108.351.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55112.338.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55111.339.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55113.340.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55114.354.955743503] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567] to LogMiner session id [2] ★55116之后的数字也出现了
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55117.391.955749633] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55118.393.955749695] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55119.399.955750553] to LogMiner session id [2] ★RFS进程接收REDO看起来很正常
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55120.406.955751745] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55121.415.955753049] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55122.422.955754083] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55123.428.955755081] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55124.439.955756675] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_27/thread_1_seq_55125.442.955756975] to LogMiner session id [2]
2. 55115和55116，这两个SEQUENCE应用出了什么错么
[root@yan1 ~]# grep 'LOGMINER: Begin mining logfile for session 2 thread 1 sequence 5511' alert_mykartes.log
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55110, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55110.347.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55111, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55111.339.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55112, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55112.338.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55113, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55113.340.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55114, +DG1/mykartes/onlinelog/group_10.315.955743237
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/onlinelog/group_9.314.955743237
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55111, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55111.339.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55112, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55112.338.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55113, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55113.340.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55114, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55114.354.955743503
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421 ★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567 ★55116
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421 ★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567 ★55116
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421 ★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567 ★55116
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421 ★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567 ★55116
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421 ★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567 ★55116
我了个去，为啥这两个SEQUENCE会应用这么多次！
看起来好像是LSP进程一直想要应用他们，但就是应用不了。
那就只能从 Doc ID 1910065.1 的资料里找线索了。
Doc ID 1910065.1 的脚本执行结果被客户放在了 new_dg_lsby_diag_mykartes_20170928_1729.html 里面
3.Check 一下REDO的应用状况
new_dg_lsby_diag_mykartes_20170928_1729.html 的部分信息
===========================
SQL> SELECT thread#, sequence#, first_change#, next_change#, dict_begin beg, dict_end end, TO_CHAR(timestamp, 'hh:mi:ss') timestamp, (CASE WHEN l.next_change# < p.read_scn THEN 'YES' WHEN l.first_change# < p.applied_scn THEN 'CURRENT' ELSE 'NO' END) applied FROM dba_logstdby_log l, dba_logstdby_progress p ORDER BY thread#, first_change#;
THREAD# SEQUENCE# FIRST_CHANGE# NEXT_CHANGE# BEG END TIMESTAM APPLIED
1 55107 3734503196 3734503595 NO NO 08:17:43 YES
1 55108 3734503595 3734504891 YES NO 08:17:43 YES
1 55109 3734504891 3734506783 NO YES 08:17:43 YES
1 55110 3734506783 3734506876 NO NO 08:17:43 YES
1 55111 3734506876 3734507595 NO NO 08:17:43 YES
1 55112 3734507595 3734508534 NO NO 08:17:43 YES
1 55113 3734508534 3734509236 NO NO 08:17:43 YES
1 55114 3734509236 3734509452 NO NO 08:18:22 YES
1 55115 3734509452 3734532403 NO NO 08:33:41 CURRENT ★55115在应用中
1 55116 3734532403 3734613007 NO NO 09:09:27 CURRENT ★55116在应用中
1 55117 3734613007 3734706486 NO NO 10:00:32 NO
1 55118 3734706486 3734805339 NO NO 10:01:35 NO
===========================
有戏啊，看来就是这两个家伙的应用hang了。
4.Check 一下REDO应用时有没有什么Error
new_dg_lsby_diag_mykartes_20170928_1729.html により抜粋
===========================
SQL> -- Verify that log apply services on the standby are currently running. ★
SQL> -- If the query against V$LOGSTDBY returns no rows then logical apply is not running.
SQL>
SQL> column status format a50 wrap
SQL> column type format a11
SQL> set numwidth 15
SQL>
SQL> SELECT type, status, high_scn FROM v$logstdby;
TYPE STATUS HIGH_SCN
COORDINATOR ORA-16116: 使用できる作業はありません 3734535493
ANALYZER ORA-16116: 使用できる作業はありません 3734559504
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535506 ★ORA-16113
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535504 ★ORA-16113
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535499 ★ORA-16113
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535494 ★ORA-16113
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535511 ★ORA-16113
READER ORA-16127: 追加のトランザクションが適用されるまで 待機して停止しました。 3734560999
BUILDER ORA-16127: 追加のトランザクションが適用されるまで 待機して停止しました。 3734559504
PREPARER ORA-16127: 追加のトランザクションが適用されるまで 待機して停止しました。 3734559503
===========================
丁……
好多ORA-16113啊！
5.从MOS寻找关于ORA-16113的文档
[参考情報]
ORA-16113: applying change to table or sequence "SYS"."AUD$" (Doc ID 1964125.1)
===========================
Symptoms ★事象
Archives stoped applying on logical standby server
Checking apply with this query we find it is stuck on the SYS.AUD$ table
SELECT type, status, high_scn FROM v$logstdby;
TYPE STATUS HIGH_SCN
COORDINATOR ORA-16116: no work available 4157595068
ANALYZER ORA-16116: no work available 4157607708
APPLIER ORA-16123: transaction 22 31 76123 is waiting for commit approval 4157595083
APPLIER ORA-16123: transaction 8 25 71375 is waiting for c ommit approval 4157595075
APPLIER ORA-16123: transaction 2 27 68682 is waiting for c ommit approval 4157595077
APPLIER ORA-16113: applying change to table or sequence "SYS"."AUD$" ★很像啊
＜省略＞
Cause ★原因
Although SYS is an INTERNAL SCHEMA there are some objects that are propagated to the logical standby because they support user data.
These are some: AUD$, SEQ$, and FGA$
What usually happens is the SYS.AUD$ table gets too big and then it is too slow to be updated in the logical standby. ★SYS.AUD$ table肥大化的话，就有可能发生哦
You can skip the object or reorg it in the primary.
Solution ★解決方法
＜省略＞
If not using broker
ALTER DATABASE STOP LOGICAL STANDBY APPLY; ★
SQL> exec dbms_logstdby.skip('DML','SYS','AUD$'); ★
SQL> exec dbms_logstdby.skip('SCHEMA_DDL','SYS','AUD$'); ★
ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATE; ★
===========================
这不就是她嘛！！！！
让客户实验了一下上面的解决方法，客户的现象就解决了。
LOGICAL STANDBY的CASE已经不多，客户更多的还是使用PHYSICAL STANDBY。
遇到LOGICAL STANDBY的时候发现，可以参照下面的MOS文档，个人感觉很有帮助。
Troubleshooting Logical Standby (Doc ID 215020.1)
[Oracle][DATAGUARD] LOGICAL STANDBY环境里，有些SEQUENCE无法应用，导致Primary和Standby无法同期的更多相关文章
- [Oracle][DATAGUARD] PHYSICAL STANDBY环境里，使用CATALOG管理Primary和Standby
1.先使用控制文件构筑好PHYSICAL STANDBY环境(Primary:Single 11.2.0.4,Standby Single 11.2.0.4) 2.构筑好Catalog用的服务器(Ca ...
- [Oracle][DATAGUARD] PHYSICAL STANDBY环境里，11.2.0.4 , 也可以使用Pfile来运行Primary和Standby（虽然很少有人用）
####Primary#### [oracle@primary ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on 金 ...
- Oracle DataGuard 升级 [11.2.0.1 -> 11.2.0.4]
Oracle DataGuard 升级 [11.2.0.1 -> 11.2.0.4] Primary: 11.2.0.1 单机,Site A. Standby: 11.2.0.1 单机,Site ...
- Oracle Dataguard之物理standby的基本配置
尽管网上有很多Oracle Dataguard的配置教程,但不难发现,很多采用的是rman duplicate这种方法,尽管此种方法较为简便.但在某种程度上,却也误导了初学者,虽说也能配置成功,但只知 ...
- Oracle Dataguard Standby Redo Log的两个实验
在Data Guard环境中,Standby Redo Log是一个比较特殊的日志类型.从最新的DG安装指导中,都推荐在Primary和Standby端,都配置Standby Redo Log. 简单 ...
- Oracle DataGuard 物理Standby 搭建(上)
物理standby database 环境搭建 Arch asysnc Oracle Dataguard host IP Oracle_sid DB_unique_name FAL_server FA ...
- Oracle DataGuard 物理Standby 搭建(下)
主备库切换 Switchover 一般SWITCHOVER切换都是计划中的切换,特点是在切换后,不会丢失任何的数据,而且这个过程是可逆的,整个DATA GUARD环境不会被破坏,原来DATA GUAR ...
- Oracle Dataguard之switchover
Oracle Dataguard的角色转换包含两类:Switchover和Failover.Switchover指主备之间角色转换,主库降为备库,备库升级为主库.而failover则是指主库出现问题时 ...
- Oracle DataGuard数据备份方案详解
Oracle DataGuard是一种数据库级别的HA方案,最主要功能是冗灾.数据保护.故障恢复等. 在生产数据库的"事务一致性"时,使用生产库的物理全备份(或物理COPY)创建备 ...
随机推荐
- 单元测试系列之十：Sonar 常用代码规则整理(二)
摘要:帮助公司部署了一套sonar平台,经过一段时间运行,发现有一些问题出现频率很高,因此有必要将这些问题进行整理总结和分析,避免再次出现类似问题. 作者原创技术文章,转载请注明出处 ＝＝＝＝＝＝＝＝ ...
- 用redis构建分布式锁
单实例的实现 从2.6.12版本开始,redis为SET命令增加了一系列选项: EX seconds – 设置键key的过期时间,单位时秒 PX milliseconds – 设置键key的过期时间, ...
- laravel文件上传
一.视图文件代码 <td> <input type="file" name="brand_logo" id="logo" ...
- _spellmod_base
技能基础修改(进去可能会改动) 可以配合数据库的spell_dbc在线制作无补丁技能. 具体效果查询DBC表 `spellid` int(11) NOT NULL DEFAULT '0', `Effe ...
- glom初级教程
1.glom介绍 通常对于字典和json的提取我们都是使用如下方式 >>> data = {'a': {'b': {'c': 'd'}}} >>> data['a' ...
- mysql自增长主键，删除数据后，将主键顺序重新排序
用数据库的时候,难免会删除数据,会发现设置的主键增长不是按照正常顺序排列,中间有断隔比如这样. 以我这个情况举例 处理方法的原理:删除原有的自增ID,重新建立新的自增ID. ALTER TABLE ` ...
- sparkSQL脚本更改问题
相应的pom依赖文件 <dependencies> <!-- <dependency> <groupId>org.apache.storm</group ...
- HBase Block Cache（块缓存）
Block Cache HBase提供了两种不同的BlockCache实现,用于缓存从HDFS读出的数据.这两种分别为: 默认的,存在于堆内存的(on-heap)LruBlockCache 存在堆外内 ...
- pgRouting新增扩展
环境依赖:postgresql cgal boost perl 环境变量: boost环境变量 CGAL环境变量 postgresql环境变量 1.新建C++ 空项目 2,添加common引用,更改配 ...
- Caffe on Mac OS X 10.11
在Mac环境安装Caffe环境(CPU_ONLY) http://blog.csdn.net/xidiancoder/article/details/52081519 有问题 http://blo ...