[Oracle][DATAGUARD] LOGICAL STANDBY环境里，有些SEQUENCE无法应用，导致Primary和Standby无法同期
今天遇到了一个客户，问题是这样的，客户构筑了一个RACtoRAC的 LOGICAL STANDBY环境。
并用EM在监视同期情况，发现EM页面上55115和55116这两个SEQUENCE一直在应用。
首先向客户要了下列的资料
Primary端
Script to Collect Data Guard Primary Site Diagnostic Information for Version 10g and Above (Including RAC). (Doc ID 1577401.1)
Standby端
SRDC - Collect Logical Standby Database Information (Doc ID 1910065.1)
=============================
Logical Standby Database (from all Instances if RAC is used)
ALERT.LOG covering at least the Time since the last Instance Startup and the Problem
Output from Note: 1577407.1 - Script to Collect Data Guard Logical Standby Diagnostic Information for Version 10g and above (Including RAC)
Output from Note: 1296954.1 - How to monitor the progress of the logical standby
TNSNAMES.ORA, LISTENER.ORA
Latest LSPx- and ASxx-Tracefiles from Background Dump-Directory
=============================
拿到资料后，先看Standby端的DB alert log
1. 是不是Redo没有传到Standby端
[root@standby ~]# grep 'RFS LogMiner: Registered logfile.*thread_1' alert_mykartes.log ★从Standby端抽取传输信息
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55109.346.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55110.347.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55108.351.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55112.338.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55111.339.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55113.340.955743463] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55114.354.955743503] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421] to LogMiner session id [2]
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567] to LogMiner session id [2] ★55116之后的数字也出现了
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55117.391.955749633] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55118.393.955749695] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55119.399.955750553] to LogMiner session id [2] ★RFS进程接收REDO看起来很正常
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55120.406.955751745] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55121.415.955753049] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55122.422.955754083] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55123.428.955755081] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55124.439.955756675] to LogMiner session id [2] ★
RFS LogMiner: Registered logfile [+DG1/mykartes/foreign_archivelog/mykarte/2017_09_27/thread_1_seq_55125.442.955756975] to LogMiner session id [2]
2. 55115和55116，这两个SEQUENCE应用出了什么错么
[root@yan1 ~]# grep 'LOGMINER: Begin mining logfile for session 2 thread 1 sequence 5511' alert_mykartes.log
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55110, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55110.347.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55111, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55111.339.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55112, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55112.338.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55113, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55113.340.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55114, +DG1/mykartes/onlinelog/group_10.315.955743237
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/onlinelog/group_9.314.955743237
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55111, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55111.339.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55112, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55112.338.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55113, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55113.340.955743463
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55114, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55114.354.955743503
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421 ★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567 ★55116
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421 ★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567 ★55116
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421 ★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567 ★55116
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421 ★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567 ★55116
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55115, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55115.361.955744421 ★55115
LOGMINER: Begin mining logfile for session 2 thread 1 sequence 55116, +DG1/mykartes/foreign_archivelog/mykarte/2017_09_26/thread_1_seq_55116.373.955746567 ★55116
我了个去，为啥这两个SEQUENCE会应用这么多次！
看起来好像是LSP进程一直想要应用他们，但就是应用不了。
那就只能从 Doc ID 1910065.1 的资料里找线索了。
Doc ID 1910065.1 的脚本执行结果被客户放在了 new_dg_lsby_diag_mykartes_20170928_1729.html 里面
3.Check 一下REDO的应用状况
new_dg_lsby_diag_mykartes_20170928_1729.html 的部分信息
===========================
SQL> SELECT thread#, sequence#, first_change#, next_change#, dict_begin beg, dict_end end, TO_CHAR(timestamp, 'hh:mi:ss') timestamp, (CASE WHEN l.next_change# < p.read_scn THEN 'YES' WHEN l.first_change# < p.applied_scn THEN 'CURRENT' ELSE 'NO' END) applied FROM dba_logstdby_log l, dba_logstdby_progress p ORDER BY thread#, first_change#;
THREAD# SEQUENCE# FIRST_CHANGE# NEXT_CHANGE# BEG END TIMESTAM APPLIED
1 55107 3734503196 3734503595 NO NO 08:17:43 YES
1 55108 3734503595 3734504891 YES NO 08:17:43 YES
1 55109 3734504891 3734506783 NO YES 08:17:43 YES
1 55110 3734506783 3734506876 NO NO 08:17:43 YES
1 55111 3734506876 3734507595 NO NO 08:17:43 YES
1 55112 3734507595 3734508534 NO NO 08:17:43 YES
1 55113 3734508534 3734509236 NO NO 08:17:43 YES
1 55114 3734509236 3734509452 NO NO 08:18:22 YES
1 55115 3734509452 3734532403 NO NO 08:33:41 CURRENT ★55115在应用中
1 55116 3734532403 3734613007 NO NO 09:09:27 CURRENT ★55116在应用中
1 55117 3734613007 3734706486 NO NO 10:00:32 NO
1 55118 3734706486 3734805339 NO NO 10:01:35 NO
===========================
有戏啊，看来就是这两个家伙的应用hang了。
4.Check 一下REDO应用时有没有什么Error
new_dg_lsby_diag_mykartes_20170928_1729.html により抜粋
===========================
SQL> -- Verify that log apply services on the standby are currently running. ★
SQL> -- If the query against V$LOGSTDBY returns no rows then logical apply is not running.
SQL>
SQL> column status format a50 wrap
SQL> column type format a11
SQL> set numwidth 15
SQL>
SQL> SELECT type, status, high_scn FROM v$logstdby;
TYPE STATUS HIGH_SCN
COORDINATOR ORA-16116: 使用できる作業はありません 3734535493
ANALYZER ORA-16116: 使用できる作業はありません 3734559504
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535506 ★ORA-16113
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535504 ★ORA-16113
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535499 ★ORA-16113
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535494 ★ORA-16113
APPLIER ORA-16113: 表またはシーケンス"SYS"."AUD$"に変更を 適用しています 3734535511 ★ORA-16113
READER ORA-16127: 追加のトランザクションが適用されるまで 待機して停止しました。 3734560999
BUILDER ORA-16127: 追加のトランザクションが適用されるまで 待機して停止しました。 3734559504
PREPARER ORA-16127: 追加のトランザクションが適用されるまで 待機して停止しました。 3734559503
===========================
丁……
好多ORA-16113啊！
5.从MOS寻找关于ORA-16113的文档
[参考情報]
ORA-16113: applying change to table or sequence "SYS"."AUD$" (Doc ID 1964125.1)
===========================
Symptoms ★事象
Archives stoped applying on logical standby server
Checking apply with this query we find it is stuck on the SYS.AUD$ table
SELECT type, status, high_scn FROM v$logstdby;
TYPE STATUS HIGH_SCN
COORDINATOR ORA-16116: no work available 4157595068
ANALYZER ORA-16116: no work available 4157607708
APPLIER ORA-16123: transaction 22 31 76123 is waiting for commit approval 4157595083
APPLIER ORA-16123: transaction 8 25 71375 is waiting for c ommit approval 4157595075
APPLIER ORA-16123: transaction 2 27 68682 is waiting for c ommit approval 4157595077
APPLIER ORA-16113: applying change to table or sequence "SYS"."AUD$" ★很像啊
＜省略＞
Cause ★原因
Although SYS is an INTERNAL SCHEMA there are some objects that are propagated to the logical standby because they support user data.
These are some: AUD$, SEQ$, and FGA$
What usually happens is the SYS.AUD$ table gets too big and then it is too slow to be updated in the logical standby. ★SYS.AUD$ table肥大化的话，就有可能发生哦
You can skip the object or reorg it in the primary.
Solution ★解決方法
＜省略＞
If not using broker
ALTER DATABASE STOP LOGICAL STANDBY APPLY; ★
SQL> exec dbms_logstdby.skip('DML','SYS','AUD$'); ★
SQL> exec dbms_logstdby.skip('SCHEMA_DDL','SYS','AUD$'); ★
ALTER DATABASE START LOGICAL STANDBY APPLY IMMEDIATE; ★
===========================
这不就是她嘛！！！！
让客户实验了一下上面的解决方法，客户的现象就解决了。
LOGICAL STANDBY的CASE已经不多，客户更多的还是使用PHYSICAL STANDBY。
遇到LOGICAL STANDBY的时候发现，可以参照下面的MOS文档，个人感觉很有帮助。
Troubleshooting Logical Standby (Doc ID 215020.1)
[Oracle][DATAGUARD] LOGICAL STANDBY环境里，有些SEQUENCE无法应用，导致Primary和Standby无法同期的更多相关文章
- [Oracle][DATAGUARD] PHYSICAL STANDBY环境里，使用CATALOG管理Primary和Standby
1.先使用控制文件构筑好PHYSICAL STANDBY环境(Primary:Single 11.2.0.4,Standby Single 11.2.0.4) 2.构筑好Catalog用的服务器(Ca ...
- [Oracle][DATAGUARD] PHYSICAL STANDBY环境里，11.2.0.4 , 也可以使用Pfile来运行Primary和Standby（虽然很少有人用）
####Primary#### [oracle@primary ~]$ sqlplus / as sysdba SQL*Plus: Release 11.2.0.4.0 Production on 金 ...
- Oracle DataGuard 升级 [11.2.0.1 -> 11.2.0.4]
Oracle DataGuard 升级 [11.2.0.1 -> 11.2.0.4] Primary: 11.2.0.1 单机,Site A. Standby: 11.2.0.1 单机,Site ...
- Oracle Dataguard之物理standby的基本配置
尽管网上有很多Oracle Dataguard的配置教程,但不难发现,很多采用的是rman duplicate这种方法,尽管此种方法较为简便.但在某种程度上,却也误导了初学者,虽说也能配置成功,但只知 ...
- Oracle Dataguard Standby Redo Log的两个实验
在Data Guard环境中,Standby Redo Log是一个比较特殊的日志类型.从最新的DG安装指导中,都推荐在Primary和Standby端,都配置Standby Redo Log. 简单 ...
- Oracle DataGuard 物理Standby 搭建(上)
物理standby database 环境搭建 Arch asysnc Oracle Dataguard host IP Oracle_sid DB_unique_name FAL_server FA ...
- Oracle DataGuard 物理Standby 搭建(下)
主备库切换 Switchover 一般SWITCHOVER切换都是计划中的切换,特点是在切换后,不会丢失任何的数据,而且这个过程是可逆的,整个DATA GUARD环境不会被破坏,原来DATA GUAR ...
- Oracle Dataguard之switchover
Oracle Dataguard的角色转换包含两类:Switchover和Failover.Switchover指主备之间角色转换,主库降为备库,备库升级为主库.而failover则是指主库出现问题时 ...
- Oracle DataGuard数据备份方案详解
Oracle DataGuard是一种数据库级别的HA方案,最主要功能是冗灾.数据保护.故障恢复等. 在生产数据库的"事务一致性"时,使用生产库的物理全备份(或物理COPY)创建备 ...
随机推荐
- vue 学习链接地址
使用Vue.js构建Web应用程序 http://www.jdon.com/48545# 一步步带你做vue后台管理框架(一)——介绍框架 http://www.cnblogs.com/herozho ...
- Linux 最小系统制作
Linux 最小系统制作 一.制作工具Busybox 在制作文件系统的时候,我们需要使用“Busybox 工具”,即为附件压缩包“busybox-1.21.1.tar.bz2”.“BusyBox 工具 ...
- TCP/IP学习
1.TCP/IP网络包括两部分 ①传输协议 ②网络协议
- Vue学习(一)Vue目录结构
安装教程网上一大把,可以自己搜索.记录下学习过程. 认识下Vue的目录结构,取自:https://www.cnblogs.com/dragonir/p/8711761.html vue 文件目录结构详 ...
- git遇到error: RPC failed; curl 18 transfer closed with outstanding read data remaining fatal: The remote end hung up unexpectedly fatal: early EOF fatal: index-pack failed failed怎么办?
答: 将clone地址中的https://替换成git://即可解决 如: 将https://git.openwrt.org/project/luci.git修改为git://git.openwrt. ...
- Java解析XML文件的常用方法介绍
XML是一个可扩展标记语言.很多时候我们需要进行数据交换,同时也存在跨平台使用,XML文件对这些需求提供了很好的帮助! 对于Java来说,XML常见的用途就是保存数据和配置,这就涉及了对XML文件的增 ...
- 2018-2019-2 网络对抗技术 20165303 Exp4 恶意代码分析
实践目标 1.1是监控你自己系统的运行状态,看有没有可疑的程序在运行. 1.2是分析一个恶意软件,就分析Exp2或Exp3中生成后门软件:分析工具尽量使用原生指令或sysinternals,systr ...
- java网络编程小白教程
1 网络编程 1.1 网络 把多台终端(计算机)通过物理线路连接起来,形成网络.便于交换数据.共享信息.组成更强大的逻辑体. 1.1.1 网络通信三要素 [1]IP地址:唯一标识网络上的每一台计算机 ...
- Linux基础命令ls
目录处理命令:ls -a 显示所有文件,包括隐藏文件 --all -l h 详细信息显示 --long --human -d 查看目录属性 - -i 查看文件唯一编号 -表示文件 d表示目录 l ...
- og协议-有利于SNS网站分享
一丶前言 全球快递toWhere官网发现使用og协议,并且支付宝和淘宝活动源码也会添加og协议,查阅资料弄清og协议是什么,此刻附上og协议官方文档 一丶什么是og协议 Open Graph通讯协定( ...