本文转自 惜纷飞 大师。

模拟基表事务未提交数据库crash,undo丢失恢复异常恢复，运维DBA反映Oracle数据库无法启动报错ORA-01157 ORA-01110，分析原因为Oracle数据库坏块导致

对于基表特别是bootstrap$中的表,如果进行了dml操作,但是没有提交,这个时候数据库crash,而且undo异常.对于类似的场景,都是很多数据库恢复的人都有畏惧,因为_corrupted_rollback_segments参数不能起作用.本实验dml修改seq$表,最终却引起了undo$和I_UNDO1异常(他们都在bootstrap$中)

1.模拟seq$表事务未提交

SQL> select * from v$version; 

BANNER

--------------------------------------------------------------------------------

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 - Production

PL/SQL Release 11.2.0.3.0 - Production

CORE    11.2.0.3.0      Production

TNS for Linux: Version 11.2.0.3.0 - Production

NLSRTL Version 11.2.0.3.0 - Production

SQL> select sysdate "www.oracleplus.com" from dual;

www.oracleplus

------------

31-AUG-13

SQL> CREATE SEQUENCE seq_oracleplus INCREMENT BY 1 START WITH 1 CACHE 3;

Sequence created.

SQL> select object_id from dba_objects where object_name='SEQ_oracleplus';

 OBJECT_ID

----------

     77282

SQL> SELECT MINVALUE,HIGHWATER,INCREMENT$, CYCLE#,ORDER$,CACHE from seq$ where obj#=77282;

  MINVALUE  HIGHWATER INCREMENT$     CYCLE#     ORDER$      CACHE

---------- ---------- ---------- ---------- ---------- ----------

         1          1          1          0          0          3

SQL> update seq$ set  CYCLE#=3 where obj#=77282;

1 row updated.

--另外一个会话abort数据库

SQL> shutdown abort;

ORACLE instance shut down.

--人工rm掉undotbs01.dbf文件

--启动数据库

SQL> startup

ORACLE instance started.

Total System Global Area  175775744 bytes

Fixed Size                  1343668 bytes

Variable Size             117444428 bytes

Database Buffers           50331648 bytes

Redo Buffers                6656000 bytes

Database mounted.

ORA-01157: cannot identify/lock data file 3 - see DBWR trace file

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

--离线数据文件

SQL> alter database datafile 3 offline;

Database altered.

--尝试open数据库

SQL> alter database open;

alter database open

*

ERROR at line 1:

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

Process ID: 17864

Session ID: 125 Serial number: 5

alert日志

SMON: enabling cache recovery

Mon Aug 26 17:08:49 2013

ARC3 started with pid=25, OS id=17902

ARC1: Archival started

ARC2: Archival started

ARC1: Becoming the 'no FAL' ARCH

ARC1: Becoming the 'no SRL' ARCH

ARC2: Becoming the heartbeat ARCH

Archived Log entry 13 added for thread 1 sequence 13 ID 0xfb920b77 dest 1:

Errors in file /u01/oracle/diag/rdbms/ora11g/ora11g/trace/ora11g_ora_17864.trc:

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

Errors in file /u01/oracle/diag/rdbms/ora11g/ora11g/trace/ora11g_ora_17864.trc:

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

Error 704 happened during db open, shutting down database

USER (ospid: 17864): terminating the instance due to error 704

Instance terminated by USER, pid = 17864

ORA-1092 signalled during: alter database open...

opiodr aborting process unknown ospid (17864) as a result of ORA-1092

做10046跟踪

SQL> conn / as sysdba

Connected to an idle instance.

SQL> startup nomount;

ORACLE instance started.

Total System Global Area  175775744 bytes

Fixed Size                  1343668 bytes

Variable Size             117444428 bytes

Database Buffers           50331648 bytes

Redo Buffers                6656000 bytes

SQL> alter database mount;

Database altered.

SQL> alter session set events  '10046 trace name context forever,level 12';

Session altered.

SQL> alter session set db_file_multiblocK_read_count=1;

Session altered.

SQL> alter database open;

alter database open

*

ERROR at line 1:

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

Process ID: 17979

Session ID: 125 Serial number: 3

trace文件内容

PARSING IN CURSOR #3064524800 len=142 dep=3 uid=0 oct=3 lid=0 tim=1377508386082495 hv=361892850 ad='265c66d0' sqlid='7bd391hat42zk'

select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1

END OF STMT

PARSE #3064524800:c=2999,e=2162,p=0,cr=0,cu=0,mis=1,r=0,dep=3,og=3,plh=0,tim=1377508386082492

BINDS #3064524800:

 Bind#0

  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00

  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0

  kxsbbbfp=b6a8ebd0  bln=22  avl=02  flg=05

  value=27

EXEC #3064524800:c=1999,e=2030,p=0,cr=0,cu=0,mis=1,r=0,dep=3,og=3,plh=906473769,tim=1377508386084833

WAIT #3064524800: nam='db file sequential read' ela= 65 file#=1 block#=321 blocks=1 obj#=34 tim=1377508386085083

WAIT #3064524800: nam='db file sequential read' ela= 21 file#=1 block#=225 blocks=1 obj#=15 tim=1377508386085208

FETCH #3064524800:c=0,e=375,p=2,cr=2,cu=0,mis=0,r=1,dep=3,og=3,plh=906473769,tim=1377508386085319

STAT #3064524800 id=1 cnt=1 pid=0 pos=1 obj=15 op='TABLE ACCESS BY INDEX ROWID UNDO$ (cr=2 pr=2 pw=0 time=362 us)'

STAT #3064524800 id=2 cnt=1 pid=1 pos=1 obj=34 op='INDEX UNIQUE SCAN I_UNDO1 (cr=1 pr=1 pw=0 time=223 us)'

CLOSE #3064524800:c=0,e=12,dep=3,type=0,tim=1377508386085657

WAIT #3064511164: nam='control file sequential read' ela= 33 file#=0 block#=1 blocks=1 obj#=15 tim=1377508386085816

WAIT #3064511164: nam='control file sequential read' ela= 18 file#=0 block#=15 blocks=1 obj#=15 tim=1377508386085889

WAIT #3064511164: nam='control file sequential read' ela= 18 file#=0 block#=17 blocks=1 obj#=15 tim=1377508386085994

WAIT #3064511164: nam='control file sequential read' ela= 18 file#=0 block#=23 blocks=1 obj#=15 tim=1377508386086058

WAIT #3064511164: nam='control file sequential read' ela= 17 file#=0 block#=1 blocks=1 obj#=15 tim=1377508386086135

WAIT #3064511164: nam='control file sequential read' ela= 15 file#=0 block#=15 blocks=1 obj#=15 tim=1377508386086196

WAIT #3064511164: nam='control file sequential read' ela= 15 file#=0 block#=17 blocks=1 obj#=15 tim=1377508386086255

WAIT #3064511164: nam='control file sequential read' ela= 18 file#=0 block#=32 blocks=1 obj#=15 tim=1377508386086317

DDE rules only execution for: ORA 1110

----- START Event Driven Actions Dump ----

---- END Event Driven Actions Dump ----

----- START DDE Actions Dump -----

Executing SYNC actions

----- START DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (Async) -----

Successfully dispatched

----- END DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (SUCCESS, 0 csec) -----

Executing ASYNC actions

----- END DDE Actions Dump (total 0 csec) -----

FETCH #3064511164:c=8998,e=9239,p=5,cr=6,cu=0,mis=0,r=0,dep=2,og=4,plh=2853959010,tim=1377508386088336

STAT #3064511164 id=1 cnt=0 pid=0 pos=1 obj=18 op='TABLE ACCESS BY INDEX ROWID OBJ$ (cr=0 pr=0 pw=0 time=82 us)'

STAT #3064511164 id=2 cnt=1 pid=1 pos=1 obj=37 op='INDEX RANGE SCAN I_OBJ2 (cr=3 pr=3 pw=0 time=1166 us)'

=====================

PARSE ERROR #3064512008:len=60 dep=1 uid=0 oct=3 lid=0 tim=1377508386088729 err=604

SELECT NULL FROM PROPS$ WHERE NAME='BOOTSTRAP_UPGRADE_ERROR'

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

_corrupted_rollback_segments无法打开数据库

SQL> show parameter _corrupted_rollback_segments;

NAME                                 TYPE        VALUE

------------------------------------ ----------- ------------------------------

_corrupted_rollback_segments         string      _SYSSMU22_1366820418$, _SYSSMU

                                                 23_2165404374$, _SYSSMU24_1842

                                                 376483$, _SYSSMU25_2373003110$

                                                 , _SYSSMU26_3635237253$, _SYSS

                                                 MU27_1730246918$, _SYSSMU28_31

                                                 59173804$, _SYSSMU29_141006113

                                                 4$, _SYSSMU30_2299069988$, _SY

                                                 SSMU31_290584066$

SQL> startup pfile='/tmp/pfile' mount

ORACLE instance started.

Total System Global Area  175775744 bytes

Fixed Size                  1343668 bytes

Variable Size             117444428 bytes

Database Buffers           50331648 bytes

Redo Buffers                6656000 bytes

Database mounted.

SQL> alter database open;

alter database open

*

ERROR at line 1:

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

Process ID: 17979

Session ID: 125 Serial number: 3

dump block225

Block header dump:  0x004000e1

 Object id on Block? Y

 seg/obj: 0xf  csc: 0xb8c.3c0c297b  itc: 1  flg: O  typ: 1 - DATA

     fsl: 0  fnx: 0x0 ver: 0x01

 Itl           Xid                  Uba        Flag  Lck        Scn/Fsc

0x01   0x0000.001.00000034  0x00400221.0034.08  --U-    1  fsc 0x0000.3c0c297c

bdba: 0x004000e1

data_block_dump,data header at 0xb6b8a644

bbed 提交225事务

[oracle@oracleplus ~]$ bbed password=blockedit filename='/u01/oracle/oradata/ora11g/system01.dbf' blocksize=8192

BBED: Release 2.0.0.0.0 - Limited Production on Mon Aug 26 17:31:09 2013

Copyright (c) 1982, 2007, Oracle.  All rights reserved.

************* !!! For Oracle Internal Use only !!! ***************

BBED> set block 225

        BLOCK#          225

BBED> map

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 225                                   Dba:0x00000000

------------------------------------------------------------

 KTB Data Block (Table/Cluster)

 struct kcbh, 20 bytes                      @0       

 struct ktbbh, 48 bytes                     @20      

 struct kdbh, 14 bytes                      @68      

 struct kdbt[1], 4 bytes                    @82      

 sb2 kdbr[32]                               @86      

 ub1 freespace[59]                          @150     

 ub1 rowdata[7979]                          @209     

 ub4 tailchk                                @8188    

BBED> p ktbbh

struct ktbbh, 48 bytes                      @20

   ub1 ktbbhtyp                             @20       0x01 (KDDBTDATA)

   union ktbbhsid, 4 bytes                  @24

      ub4 ktbbhsg1                          @24       0x0000000f

      ub4 ktbbhod1                          @24       0x0000000f

   struct ktbbhcsc, 8 bytes                 @28

      ub4 kscnbas                           @28       0x3c0c297b

      ub2 kscnwrp                           @32       0x0b8c

   b2 ktbbhict                              @36      -2047

   ub1 ktbbhflg                             @38       0x03 (KTBFONFL)

   ub1 ktbbhfsl                             @39       0x00

   ub4 ktbbhfnx                             @40       0x00000000

   struct ktbbhitl[0], 24 bytes             @44

      struct ktbitxid, 8 bytes              @44

         ub2 kxidusn                        @44       0x0000

         ub2 kxidslt                        @46       0x0001

         ub4 kxidsqn                        @48       0x00000034

      struct ktbituba, 8 bytes              @52

         ub4 kubadba                        @52       0x00400221

         ub2 kubaseq                        @56       0x0034

         ub1 kubarec                        @58       0x08

      ub2 ktbitflg                          @60       0x2001 (KTBFUPB)

      union _ktbitun, 2 bytes               @62

         b2 _ktbitfsc                       @62       0

         ub2 _ktbitwrp                      @62       0x0000

      ub4 ktbitbas                          @64       0x3c0c297c

BBED> set mode edit

        MODE            Edit

BBED> m /x 0180 offset 60

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 225              Offsets:   60 to   91           Dba:0x00000000

------------------------------------------------------------------------

 01800000 7c290c3c 00012000 ffff5200 8d003017 30170000 20008e1f 4106f805 

 <32 bytes per line>

BBED> sum apply

Check value for File 0, Block 225:

current = 0x55db, required = 0x55db

BBED> verify

DBVERIFY - Verification starting

FILE = /u01/oracle/oradata/ora11g/system01.dbf

BLOCK = 225

Block Checking: DBA = 4194529, Block Type = KTB-managed data block

data header at 0xb6773244

kdbchk: row locked by non-existent transaction

        table=0   slot=20

        lockid=1   ktbbhitc=1

Block 225 failed with check code 6101

DBVERIFY - Verification complete

Total Blocks Examined         : 1

Total Blocks Processed (Data) : 1

Total Blocks Failing   (Data) : 1

Total Blocks Processed (Index): 0

Total Blocks Failing   (Index): 0

Total Blocks Empty            : 0

Total Blocks Marked Corrupt   : 0

Total Blocks Influx           : 0

BBED> p *kdbr[20]

rowdata[608]

------------

ub1 rowdata[608]                            @817      0x2c

BBED> dump

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 225              Offsets:  817 to  848           Dba:0x00000000

------------------------------------------------------------------------

 2c011102 c115145f 53595353 4d553230 5f343035 34303430 31322402 c10202c1 

 <32 bytes per line>

BBED> m /x 002c

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 225              Offsets:  817 to  848           Dba:0x00000000

------------------------------------------------------------------------

 002c1102 c115145f 53595353 4d553230 5f343035 34303430 31322402 c10202c1 

 <32 bytes per line>

BBED> sum apply

Check value for File 0, Block 225:

current = 0x79f6, required = 0x79f6

BBED> verify

DBVERIFY - Verification starting

FILE = /u01/oracle/oradata/ora11g/system01.dbf

BLOCK = 225

Block Checking: DBA = 4194529, Block Type = KTB-managed data block

data header at 0xb6773244

kdbchk: row locked by non-existent transaction

        table=0   slot=20

        lockid=44   ktbbhitc=1

Block 225 failed with check code 6101

DBVERIFY - Verification complete

Total Blocks Examined         : 1

Total Blocks Processed (Data) : 1

Total Blocks Failing   (Data) : 1

Total Blocks Processed (Index): 0

Total Blocks Failing   (Index): 0

Total Blocks Empty            : 0

Total Blocks Marked Corrupt   : 0

Total Blocks Influx           : 0

BBED> p *kdbr[20]

rowdata[608]

------------

ub1 rowdata[608]                            @817      0x00

BBED> d

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 225              Offsets:  817 to  848           Dba:0x00000000

------------------------------------------------------------------------

 002c1102 c115145f 53595353 4d553230 5f343035 34303430 31322402 c10202c1 

 <32 bytes per line>

BBED> m /x 2c00

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 225              Offsets:  817 to  848           Dba:0x00000000

------------------------------------------------------------------------

 2c001102 c115145f 53595353 4d553230 5f343035 34303430 31322402 c10202c1 

 <32 bytes per line>

BBED> sum apply

Check value for File 0, Block 225:

current = 0x55da, required = 0x55da

BBED> verify

DBVERIFY - Verification starting

FILE = /u01/oracle/oradata/ora11g/system01.dbf

BLOCK = 225

DBVERIFY - Verification complete

Total Blocks Examined         : 1

Total Blocks Processed (Data) : 1

Total Blocks Failing   (Data) : 0

Total Blocks Processed (Index): 0

Total Blocks Failing   (Index): 0

Total Blocks Empty            : 0

Total Blocks Marked Corrupt   : 0

Total Blocks Influx           : 0

dump 321 block

SQL> alter system dump datafile '/u01/oracle/oradata/ora11g/system01.dbf' block 321;

System altered.

SQL> select value from v$diag_info where name='Default Trace File';

VALUE

--------------------------------------------------------------------------------

/u01/oracle/diag/rdbms/ora11g/ora11g/trace/ora11g_ora_19704.trc

--trace文件

Block header dump:  0x00400141

 Object id on Block? Y

 seg/obj: 0x22  csc: 0xb8c.3c0c28ec  itc: 2  flg: -  typ: 2 - INDEX

     fsl: 0  fnx: 0x0 ver: 0x01

 Itl           Xid                  Uba         Flag  Lck        Scn/Fsc

0x01   0x0000.000.00000000  0x00000000.0000.00  ----    0  fsc 0x0000.00000000

0x02   0x0000.038.00000033  0x0040021e.0033.3d  --U-    1  fsc 0x0000.3c0c28ed

bbed提交321 block 事务

BBED> set block 321

        BLOCK#          321

BBED> map

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 321                                   Dba:0x00000000

------------------------------------------------------------

 KTB Data Block (Index Leaf)

 struct kcbh, 20 bytes                      @0       

 struct ktbbh, 72 bytes                     @20      

 struct kdxle, 32 bytes                     @92      

 b2 kd_off[32]                              @124     

 ub1 freespace[7581]                        @188     

 ub1 rowdata[351]                           @7769    

 ub4 tailchk                                @8188    

BBED> p ktbbh

struct ktbbh, 72 bytes                      @20

   ub1 ktbbhtyp                             @20       0x02 (KDDBTINDEX)

   union ktbbhsid, 4 bytes                  @24

      ub4 ktbbhsg1                          @24       0x00000022

      ub4 ktbbhod1                          @24       0x00000022

   struct ktbbhcsc, 8 bytes                 @28

      ub4 kscnbas                           @28       0x3c0c28ec

      ub2 kscnwrp                           @32       0x0b8c

   b2 ktbbhict                              @36      -2046

   ub1 ktbbhflg                             @38       0x02 (NONE)

   ub1 ktbbhfsl                             @39       0x00

   ub4 ktbbhfnx                             @40       0x00000000

   struct ktbbhitl[0], 24 bytes             @44

      struct ktbitxid, 8 bytes              @44

         ub2 kxidusn                        @44       0x0000

         ub2 kxidslt                        @46       0x0000

         ub4 kxidsqn                        @48       0x00000000

      struct ktbituba, 8 bytes              @52

         ub4 kubadba                        @52       0x00000000

         ub2 kubaseq                        @56       0x0000

         ub1 kubarec                        @58       0x00

      ub2 ktbitflg                          @60       0x0000 (NONE)

      union _ktbitun, 2 bytes               @62

         b2 _ktbitfsc                       @62       0

         ub2 _ktbitwrp                      @62       0x0000

      ub4 ktbitbas                          @64       0x00000000

   struct ktbbhitl[1], 24 bytes             @68

      struct ktbitxid, 8 bytes              @68

         ub2 kxidusn                        @68       0x0000

         ub2 kxidslt                        @70       0x0038

         ub4 kxidsqn                        @72       0x00000033

      struct ktbituba, 8 bytes              @76

         ub4 kubadba                        @76       0x0040021e

         ub2 kubaseq                        @80       0x0033

         ub1 kubarec                        @82       0x3d

      ub2 ktbitflg                          @84       0x2001 (KTBFUPB)

      union _ktbitun, 2 bytes               @86

         b2 _ktbitfsc                       @86       0

         ub2 _ktbitwrp                      @86       0x0000

      ub4 ktbitbas                          @88       0x3c0c28ed

BBED> m /x 0180 offset 84

Warning: contents of previous BIFILE will be lost. Proceed? (Y/N) y

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 321              Offsets:   84 to  595           Dba:0x00000000

------------------------------------------------------------------------

 01800000 ed280c3c 00008001 00000000 20006400 011e9d1d 00000000 00000000

 00000000 06000000 601f0000 561f4b1f 401f351f 2a1f1f1f 141f091f fe1ef31e

 e81edd1e d21ec71e bc1eb11e a61e9b1e 901e851e 7a1e6f1e 641e591e 4e1e431e

 381e2d1e 221e171e 0c1e011e 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

BBED> sum apply

Check value for File 0, Block 321:

current = 0x990e, required = 0x990e

BBED> verify

DBVERIFY - Verification starting

FILE = /u01/oracle/oradata/ora11g/system01.dbf

BLOCK = 321

Block Checking: DBA = 4194625, Block Type = KTB-managed data block

**** actual rows locked by itl 2  = 1 != # in trans. header = 0

---- end index block validation

Block 321 failed with check code 6401

DBVERIFY - Verification complete

Total Blocks Examined         : 1

Total Blocks Processed (Data) : 0

Total Blocks Failing   (Data) : 0

Total Blocks Processed (Index): 1

Total Blocks Failing   (Index): 1

Total Blocks Empty            : 0

Total Blocks Marked Corrupt   : 0

Total Blocks Influx           : 0

BBED> p kdxle

struct kdxle, 32 bytes                      @92

   struct kdxlexco, 16 bytes                @92

      ub1 kdxcolev                          @92       0x00

      ub1 kdxcolok                          @93       0x00

      ub1 kdxcoopc                          @94       0x80

      ub1 kdxconco                          @95       0x01

      ub4 kdxcosdc                          @96       0x00000000

      sb2 kdxconro                          @100      32

      b2 kdxcofbo                           @102      100

      b2 kdxcofeo                           @104      7681

      b2 kdxcoavs                           @106      7581

   b2 kdxlespl                              @108      0

   sb2 kdxlende                             @110      0

   ub4 kdxlenxt                             @112      0x00000000

   ub4 kdxleprv                             @116      0x00000000

   ub1 kdxledsz                             @120      0x06

   ub1 kdxleunuse                           @121      0x00

BBED> m /x 00 offset 95

 File: /u01/oracle/oradata/ora11g/system01.dbf (0)

 Block: 321              Offsets:   95 to  606           Dba:0x00000000

------------------------------------------------------------------------

 00000000 00200064 00011e9d 1d000000 00000000 00000000 00060000 00601f00

 00561f4b 1f401f35 1f2a1f1f 1f141f09 1ffe1ef3 1ee81edd 1ed21ec7 1ebc1eb1

 1ea61e9b 1e901e85 1e7a1e6f 1e641e59 1e4e1e43 1e381e2d 1e221e17 1e0c1e01

 1e000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000 

 <32 bytes per line>

BBED> sum apply

Check value for File 0, Block 321:

current = 0x380e, required = 0x380e

BBED> verify

DBVERIFY - Verification starting

FILE = /u01/oracle/oradata/ora11g/system01.dbf

BLOCK = 321

DBVERIFY - Verification complete

Total Blocks Examined         : 1

Total Blocks Processed (Data) : 0

Total Blocks Failing   (Data) : 0

Total Blocks Processed (Index): 1

Total Blocks Failing   (Index): 0

Total Blocks Empty            : 0

Total Blocks Marked Corrupt   : 0

Total Blocks Influx           : 0

bbed提交事务open依旧报错

SQL> alter database open;

alter database open

*

ERROR at line 1:

ORA-01092: ORACLE instance terminated. Disconnection forced

ORA-00704: bootstrap process failure

ORA-00604: error occurred at recursive SQL level 2

ORA-00376: file 3 cannot be read at this time

ORA-01110: data file 3: '/u01/oracle/oradata/ora11g/undotbs01.dbf'

Process ID: 20229

Session ID: 125 Serial number: 5

做10046的trace文件错误依旧

=====================

PARSING IN CURSOR #3065238148 len=142 dep=3 uid=0 oct=3 lid=0 tim=1377519352653391 hv=361892850 ad='269a8064' sqlid='7bd391hat42zk'

select /*+ rule */ name,file#,block#,status$,user#,undosqn,xactsqn,scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1

END OF STMT

PARSE #3065238148:c=2999,e=3152,p=0,cr=0,cu=0,mis=1,r=0,dep=3,og=3,plh=0,tim=1377519352653388

BINDS #3065238148:

 Bind#0

  oacdty=02 mxl=22(22) mxlc=00 mal=00 scl=00 pre=00

  oacflg=08 fl2=0001 frm=00 csi=00 siz=24 off=0

  kxsbbbfp=b6b32268  bln=22  avl=02  flg=05

  value=27

EXEC #3065238148:c=2999,e=3182,p=0,cr=0,cu=0,mis=1,r=0,dep=3,og=3,plh=906473769,tim=1377519352657030

WAIT #3065238148: nam='db file sequential read' ela= 169 file#=1 block#=321 blocks=1 obj#=34 tim=1377519352657400

WAIT #3065238148: nam='db file sequential read' ela= 25 file#=1 block#=225 blocks=1 obj#=15 tim=1377519352657546

FETCH #3065238148:c=1000,e=578,p=2,cr=2,cu=0,mis=0,r=1,dep=3,og=3,plh=906473769,tim=1377519352657738

STAT #3065238148 id=1 cnt=1 pid=0 pos=1 obj=15 op='TABLE ACCESS BY INDEX ROWID UNDO$ (cr=2 pr=2 pw=0 time=538 us)'

STAT #3065238148 id=2 cnt=1 pid=1 pos=1 obj=34 op='INDEX UNIQUE SCAN I_UNDO1 (cr=1 pr=1 pw=0 time=338 us)'

CLOSE #3065238148:c=0,e=13,dep=3,type=0,tim=1377519352657942

WAIT #3065201888: nam='control file sequential read' ela= 111 file#=0 block#=1 blocks=1 obj#=15 tim=1377519352658170

WAIT #3065201888: nam='control file sequential read' ela= 19 file#=0 block#=16 blocks=1 obj#=15 tim=1377519352658269

WAIT #3065201888: nam='control file sequential read' ela= 18 file#=0 block#=18 blocks=1 obj#=15 tim=1377519352658329

WAIT #3065201888: nam='control file sequential read' ela= 19 file#=0 block#=24 blocks=1 obj#=15 tim=1377519352658391

WAIT #3065201888: nam='control file sequential read' ela= 17 file#=0 block#=1 blocks=1 obj#=15 tim=1377519352658469

WAIT #3065201888: nam='control file sequential read' ela= 16 file#=0 block#=16 blocks=1 obj#=15 tim=1377519352658528

WAIT #3065201888: nam='control file sequential read' ela= 28 file#=0 block#=18 blocks=1 obj#=15 tim=1377519352658787

WAIT #3065201888: nam='control file sequential read' ela= 20 file#=0 block#=32 blocks=1 obj#=15 tim=1377519352658883

DDE rules only execution for: ORA 1110

----- START Event Driven Actions Dump ----

---- END Event Driven Actions Dump ----

----- START DDE Actions Dump -----

Executing SYNC actions

----- START DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (Async) -----

Successfully dispatched

----- END DDE Action: 'DB_STRUCTURE_INTEGRITY_CHECK' (SUCCESS, 0 csec) -----

Executing ASYNC actions

----- END DDE Actions Dump (total 0 csec) -----

FETCH #3065201888:c=11998,e=12032,p=5,cr=6,cu=0,mis=0,r=0,dep=2,og=4,plh=2853959010,tim=1377519352661044

STAT #3065201888 id=1 cnt=0 pid=0 pos=1 obj=18 op='TABLE ACCESS BY INDEX ROWID OBJ$ (cr=0 pr=0 pw=0 time=19 us)'

STAT #3065201888 id=2 cnt=1 pid=1 pos=1 obj=37 op='INDEX RANGE SCAN I_OBJ2 (cr=3 pr=3 pw=0 time=1001 us)'

通过观察每次的10046文件,发现总是在查询select /*+ rule */ name,file#, block#,status$,user#,undosqn,xactsqn, scnbas,scnwrp,DECODE(inst#,0,NULL,inst#),ts#,spare1 from undo$ where us#=:1并且总是us#=27号回滚段上过不去,通过使用bbed或者dul修改数据文件,让数据库启动时候不再扫描27号回滚段(主要是修改undo$.status$=1,block位置可以通过10046确定,也可以通过odu来确定,亦或者找个相同版本的库查看)

继续open数据库

SQL> startup pfile='/tmp/pfile' mount

ORACLE instance started.

Total System Global Area  175775744 bytes

Fixed Size                  1343668 bytes

Variable Size             117444428 bytes

Database Buffers           50331648 bytes

Redo Buffers                6656000 bytes

Database mounted.

SQL> alter database open;

Database altered.

删除有问题undo,重建新undo

SQL> drop tablespace undotbs1;

drop tablespace undotbs1

*

ERROR at line 1:

ORA-01561: failed to remove all objects in the tablespace specified

SQL> !oerr ora 01561

01561, 00000, "failed to remove all objects in the tablespace specified"

// *Cause: Failed to remove all objects when dropping a tablespace

// *Action: Retry the drop tablespace until all objects are dropped

SQL> drop tablespace undotbs1 including contents;

drop tablespace undotbs1 including contents

*

ERROR at line 1:

ORA-01561: failed to remove all objects in the tablespace specified

SQL> drop tablespace undotbs1 including contents and datafiles;

drop tablespace undotbs1 including contents and datafiles

*

ERROR at line 1:

ORA-01561: failed to remove all objects in the tablespace specified

-------------------------------------------------

因为前面使用了bbed/dul之类的工具，修改了undo$表，

导致undo$中的标记27号回滚段与seg$中不一致

解决方法使得修改undo$.status$为3或者2

-------------------------------------------------

SQL> update undo$ set status$=3 where us#=27;

1 row updated.

SQL> commit;

Commit complete.

SQL> drop tablespace undotbs1 ;

drop tablespace undotbs1

*

ERROR at line 1:

ORA-01561: failed to remove all objects in the tablespace specified

SQL> shutdown abort;

ORACLE instance shut down.

SQL> startup pfile='/tmp/pfile'      这个地方可以用 _allow_resetlogs_corruption 这个参数打开

ORACLE instance started.

Total System Global Area  175775744 bytes

Fixed Size                  1343668 bytes

Variable Size             117444428 bytes

Database Buffers           50331648 bytes

Redo Buffers                6656000 bytes

Database mounted.

Database opened.

SQL> drop tablespace undotbs1 ;

Tablespace dropped.

SQL> create undo tablespace undotbs1 datafile '/u01/oracle/oradata/ora11g/undotbs1.dbf' size 10M;

Tablespace created.

undo丢失恢复异常恢复，运维DBA反映Oracle数据库无法启动报错ORA-01157 ORA-01110，分析原因为Oracle数据库坏块导致的更多相关文章

  1. 【转】运维DBA的4大纪律9项注意

    朋友们调侃说,运维是个把脑袋别在裤腰带上的活,更有人说,运维是个把脑袋别在他人裤腰带上的活,苦劳没人认,有锅就有得背! 测试的同学说,“吃瓜群众很难感知运维背后的付出,倒是出了事情更能体现我们的专业性 ...

  2. 运维DBA要不要学python

    运维DBA要不要学python 我个人认为是:要 现在python在运维数据库的工作中主要用在 1.编写一些运维脚本 2.编写运维管理平台 3.研究互联网大厂的运维脚本/工具并应有 特别是运维开源数据 ...

  3. docker启动报错iptables failed: -重建docker0网络恢复

    # docker启动报错 [root@localhost mysqlconf]# docker run -d -p 8080:8080 --link zookeeper:zookeeper -e du ...

  4. 一、linux-mysql 运维DBA介绍

    一.DBA数据库管理人员需要在整个架构中解决数据库的压力,前端业务通过扩展,加机器就可以很好的解决,但是存储.数据库就不是很好的可以进行扩展,数据也是分配不均的,所以,1)通过在数据库前面添加Memc ...

  5. LINUX运维实战案例之文件已删除但空间不释放问题的分析与解决办法

    1.错误现象 运维的监控系统发来通知,报告一台服务器空间满了,登陆服务器查看,根分区确实没有空间了,如下图所示: 这里首先说明一下服务器的一些删除策略,由于Linux没有回收站功能,我们的线上服务器所 ...

  6. 网站运维之 使用IIS日志分析器1.03.exe进行IIS服务器日志分析

    引言 对于网站运维是一个比较要细心有耐心的工作,当一个网站从开发到上线后,后期的维护也很关键,特别是对于引流的网站来说更是至关重要. 对于网站运维的内容大致可以分为: SEO流量监控方面:风险防控:访 ...

  7. 恢复云数据库MySQL的备份文件到自建数据库遇到的报错

    报错信息 : 恢复云数据库MySQL的备份文件到自建数据库,自建数据库版本5.6.36. 按照阿里云文档操作,启动数据库报错 -- :: [ERROR] /application/mysql/bin/ ...

  8. NBU恢复数据库数据文件报错RMAN-06091

    RMAN-06091: no channel allocated for maintenance (of an appropriate type) 一.错误信息 报错信息如下 Starting res ...

  9. 启动mysql错误解决方案，学会查看错误日志：mysql.sock丢失，mysqld_safe启动报错

    本人还是个菜鸟,下面是我的经验之谈,能解决一些问题,有不对的地方,敬请斧正. 我的是CentOS6.3+MySQL5.1.57. 重启了一次服务器后,使用> mysql -u root -p登陆 ...

随机推荐

  1. vue2 设置网页title的问题

    好东东,没个标题看着多难受 看到1文章  http://blog.csdn.net/qishuixian/article/details/72912368 推荐使用 vue-wechat-title插 ...

  2. windows 获取用户的Sid的方法

    正常获取: whoami /user 如果要获取其他用户的SID就显得力不从心了,我们可以使用微软提供的系统工具 Sysinternals Suite 下载地址:https://docs.micros ...

  3. 剑指offer（java版）【转】

    面试题 2 :实现单例模式 1. 饿汉式单例类 public class SingletonClass { private static final SingletonClass instance=n ...

  4. spring4笔记----使用装配注入合作者Bean的三种方式

    no :不自动装配 byName :id(name)与setter方法去set前缀,并小写首字母后同名的Bean完成注入,如有多个匹配则抛异常 byType :spring容器找全部bean,如果找到 ...

  5. kali2016.2（debian）快速安装mysql5.7.17

    糊里糊涂的删除了kali原本的mysql5.6.27版本,原本的mysql与很多软件关联在一起,每次安装都失败,后来把相关的都卸载了(悲催的浪费了一天) 下载地址  debian mysql下载地址 ...

  6. [20190401]那个更快的疑问.txt

    [20190401]那个更快的疑问.txt --//前一阵子,做了11g于10g下,单表单条记录唯一索引扫描的测试,摘要如下:--//参考链接:http://blog.itpub.net/267265 ...

  7. Django基础笔记

    1.准备工作 .虚拟环境设置 python3 pip install virtualenv python -m venv env(虚拟环境文件名) env\Scripts\activate pip i ...

  8. c/c++ 标准容器 vector的内存空间是如何自动增长的

    c/c++ 标准容器 vector的内存空间是如何自动增长的 vector,string,deque的内存存储机制:在一个连续的内存空间存储,所以才支持下标操作. vector的课题:由于容器的大小是 ...

  9. Hibernate 5 入门指南-基于类注解

    首先创建hibernate.cfg.xml配置文件并做简单的配置 <hibernate-configuration>    <session-factory>        & ...

  10. 关于激活Windows10专业版2018长期服务版

    之前重装了一次系统,偷懒用了小白一键重装,装好之后显示的是Windows10专业版2018长期服务版,当时也没想太多就放着用了. 然后 ,这几天一直提示  “你的windows许可证即将过期” ,就按 ...