ASM丢失disk header导致ORA-15032、ORA-15040、ORA-15042 Diskgroup无法mount

SQL> select * from v$version;

BANNER

——————————————————————————–

Oracle Database 11g Enterprise Edition Release 11.2.0.3.0 – 64bit Production

PL/SQL Release 11.2.0.3.0 – Production

CORE 11.2.0.3.0 Production

TNS for Linux: Version 11.2.0.3.0 – Production

NLSRTL Version 11.2.0.3.0 – Production

SQL> alter diskgroup datadg mount;

alter diskgroup datadg mount

*

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15040: diskgroup is incomplete

ORA-15042: ASM disk “5” is missing from group number “1”

ERROR: alter diskgroup datadg mount

Wed Mar 13 07:42:03 2013

SQL> alter diskgroup datadg mount 

NOTE: cache registered group DATADG number=1 incarn=0xccb845cd

NOTE: cache began mount (first) of group DATADG number=1 incarn=0xccb845cd

NOTE: Assigning number (1,2) to disk (/dev/asm-diskg)

NOTE: Assigning number (1,1) to disk (/dev/asm-diskf)

NOTE: Assigning number (1,0) to disk (/dev/asm-diske)

Wed Mar 13 16:42:09 2013

NOTE: GMON heartbeating for grp 1

GMON querying group 1 at 20 for pid 27, osid 5439

NOTE: Assigning number (1,5) to disk ()

GMON querying group 1 at 21 for pid 27, osid 5439

NOTE: cache dismounting (clean) group 1/0xCCB845CD (DATADG) 

NOTE: messaging CKPT to quiesce pins Unix process pid: 5439, image: oracle@vmac1 (TNS V1-V3)

NOTE: dbwr not being msg’d to dismount

NOTE: lgwr not being msg’d to dismount

NOTE: cache dismounted group 1/0xCCB845CD (DATADG) 

NOTE: cache ending mount (fail) of group DATADG number=1 incarn=0xccb845cd

NOTE: cache deleting context for group DATADG 1/0xccb845cd

GMON dismounting group 1 at 22 for pid 27, osid 5439

NOTE: Disk in mode 0x8 marked for de-assignment

NOTE: Disk in mode 0x8 marked for de-assignment

NOTE: Disk in mode 0x8 marked for de-assignment

NOTE: Disk in mode 0x8 marked for de-assignment

ERROR: diskgroup DATADG was not mounted

ORA-15032: not all alterations performed

ORA-15040: diskgroup is incomplete

ORA-15042: ASM disk “5” is missing from group number “1” 

ERROR: alter diskgroup datadg mount

Wed Mar 13 16:42:10 2013

ASM Health Checker found 1 new failures

 

[grid@vmac1 ~]$ kfed read /dev/asm-diskh

kfbh.endian: 0 ; 0x000: 0x00

kfbh.hard: 0 ; 0x001: 0x00

kfbh.type: 0 ; 0x002: KFBTYP_INVALID

kfbh.datfmt: 0 ; 0x003: 0x00

kfbh.block.blk: 0 ; 0x004: blk=0

kfbh.block.obj: 0 ; 0x008: file=0

kfbh.check: 0 ; 0x00c: 0x00000000

kfbh.fcn.base: 0 ; 0x010: 0x00000000

kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

kfbh.spare1: 0 ; 0x018: 0x00000000

kfbh.spare2: 0 ; 0x01c: 0x00000000

7FA1DA233400 00000000 00000000 00000000 00000000 [................]

Repeat 255 times

KFED-00322: Invalid content encountered during block traversal: [kfbtTraverseBlock][Invalid OSM block type][][0]

col path for a20

set linesize 200 pagesize 1400

select path,header_status,state from v$asm_disk;

PATH HEADER_STATUS STATE

——————– ———————————— ————————

/dev/asm-diskh CANDIDATE NORMAL

/dev/asm-diskg MEMBER NORMAL

/dev/asm-diskf MEMBER NORMAL

/dev/asm-diske MEMBER NORMAL

/dev/asm-diskc MEMBER NORMAL

/dev/asm-diskd MEMBER NORMAL

/dev/asm-diskb MEMBER NORMAL

7 rows selected.

[grid@vmac1 ~]$ kfed repair /dev/asm-diskh

KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]

 

[grid@vmac1 ~]$ kfed repair /dev/asm-diskh ausz=1048576

KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]

 

关闭ASM实例

 

 

优先备份 问题ASM的header

dd if=<bad disk> of=<file> bs=4096 count=1

 

 

3. 检查现有磁盘找出拥有file 1 block 1的

[grid@vmac1 ~]$ kfed read /dev/asm-diske |grep f1b1

kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002

[grid@vmac1 ~]$ 

[grid@vmac1 ~]$ kfed read /dev/asm-diskf |grep f1b1 

kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000

[grid@vmac1 ~]$ kfed read /dev/asm-diskg |grep f1b1 

kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000

 

这里asm-diske上出现了f1b1非零值。则其拥有file 1 block 1，能够通过检查第二个au的类型是否是KFBTYP_LISTHEAD来确认

[grid@vmac1 ~]$ kfed read /dev/asm-diske aun=2|grep kfbh.type

kfbh.type: 5 ; 0x002: KFBTYP_LISTHEAD

 

 

若丢失的磁盘包括了”file 1 block 1 F1B1″则扫描 该磁盘上全部的 AU直到找到KFBTYP_LISTHEAD ， 假设找不到LISTHEAD 那么别无选择仅仅能重建diskgroup。

 

从版本号11.1.0.7開始(10g是从10.2.0.5開始，所以尽量别用10.2.0.5之前的版本号上的ASM)。当每个I/O写提交向ASM disk header(AU 0 blocknum 0)，都会拷贝到 AU 1中，最后第二个块。基于不同的AU size，该块的位置不同

Allocation Unit Size Block Number on AU 1

1048576 254

4194304 1022

8388608 2046

16777216 4094

 

首先使用kfed 验证该位置是否有正确的disk header，否则手动找到合适的header。 表1如參考：

[grid@vmac1 ~]$ kfed read /dev/asm-diske ausz=1048576 aun=1 blkn=254|less 

kfbh.endian: 1 ; 0x000: 0x01

kfbh.hard: 130 ; 0x001: 0x82

kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD

kfbh.datfmt: 1 ; 0x003: 0x01

kfbh.block.blk: 254 ; 0x004: blk=254

kfbh.block.obj: 2147483648 ; 0x008: disk=0

kfbh.check: 2086475720 ; 0x00c: 0x7c5d17c8

kfbh.fcn.base: 31322 ; 0x010: 0x00007a5a

kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

kfbh.spare1: 0 ; 0x018: 0x00000000

之后使用kfed repair命令修复disk header

[grid@vmac1 ~]$ kfed repair /dev/asm-diskh ausz=1048576

 

 

假设disk header的自己主动备份也丢失了。那么会报例如以下错误

KFED-00320: Invalid block num1 = [0], num2 = [1], error = [endian_kfbh]

 

若上述kfed repair无效则须要通过 手动恢复disk header的办法了：

 

对于版本号10.2.0.5之前没有kred repair可用的情况：

 

找一个与问题disk 在同一个diskgroup中的，不包括f1b1 的好的disk header。 比如这里的asm-diskf

[grid@vmac1 ~]$ kfed read /dev/asm-diskf |grep f1b1 

kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000

 

使用kfed read <device name> > fix.txt 命令保存其头部

[grid@vmac1 ~]$ kfed read /dev/asm-diskf > fix.txt

编辑fix.txt 改动kfdhdb.dsknum 、kfdhdb.dskname 、 kfdhdb.fgname 三个信息：

同一时候參考alert.log中的信息：

[grid@vmac1 trace]$ grep “cache opening” alert_+ASM1.log

NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb

NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc

NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd

NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb

NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc

NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd

NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb

NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc

NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd

NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb

NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc

NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd

NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb

NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc

NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd

NOTE: cache opening disk 0 of grp 1: SYSTEDG_0000 path:/dev/asm-diskb

NOTE: cache opening disk 1 of grp 1: SYSTEDG_0001 path:/dev/asm-diskc

NOTE: cache opening disk 2 of grp 1: SYSTEDG_0002 path:/dev/asm-diskd

NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/asm-diske

NOTE: cache opening disk 1 of grp 2: DATADG_0001 path:/dev/asm-diskf

NOTE: cache opening disk 2 of grp 2: DATADG_0002 path:/dev/asm-diskg

NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/asm-diske

NOTE: cache opening disk 1 of grp 2: DATADG_0001 path:/dev/asm-diskf

NOTE: cache opening disk 2 of grp 2: DATADG_0002 path:/dev/asm-diskg

NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/asm-diske

NOTE: cache opening disk 1 of grp 1: DATADG_0001 path:/dev/asm-diskf

NOTE: cache opening disk 2 of grp 1: DATADG_0002 path:/dev/asm-diskg

NOTE: cache opening disk 0 of grp 2: SYSTEDG_0000 path:/dev/asm-diskb

NOTE: cache opening disk 1 of grp 2: SYSTEDG_0001 path:/dev/asm-diskc

NOTE: cache opening disk 2 of grp 2: SYSTEDG_0002 path:/dev/asm-diskd

NOTE: cache opening disk 5 of grp 1: DATADG_0005 path:/dev/asm-diskh

 

 

原fix.txt中的内容：

[grid@vmac1 ~]$ egrep “dsknum|grptyp|hdrsts|dskname|grpname|fgname” fix.txt 

kfdhdb.dsknum: 1 ; 0x024: 0x0001

kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL

kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER

kfdhdb.dskname: DATADG_0001 ; 0x028: length=11

kfdhdb.grpname: DATADG ; 0x048: length=6

kfdhdb.fgname: DATADG_0001 ; 0x068: length=11

 

改动后：

[grid@vmac1 ~]$ egrep “dsknum|grptyp|hdrsts|dskname|grpname|fgname” fix.txt 

kfdhdb.dsknum: 5 ; 0x024: 0x0005

kfdhdb.grptyp: 1 ; 0x026: KFDGTP_EXTERNAL

kfdhdb.hdrsts: 3 ; 0x027: KFDHDR_MEMBER

kfdhdb.dskname: DATADG_0005 ; 0x028: length=11

kfdhdb.grpname: DATADG ; 0x048: length=6

kfdhdb.fgname: DATADG_0005 ; 0x068: length=11

 

还须要改动 kfbh.block.obj

[grid@vmac1 ~]$ grep kfbh.block.obj fix.txt

kfbh.block.obj: 2147483649 ; 0x008: disk=1

2147483649==》0x80000001

0x80000001 最后一位是ASM DISK NUMBER , 应当等于kfdhdb.dsknum ， 在这里为 0x80000005 ==》 2147483653

[grid@vmac1 ~]$ grep kfbh.block.obj fix.txt

kfbh.block.obj: 2147483653 ; 0x008: disk=5

 

若使用windows平台上的ASMLIB则还要麻烦地改动kfdhdb.driver.reserved[0]， 所幸非常少人会用Windows上的ASMLIB

 

 

 

接着查看aunum=2 blknum=2来查找 disk directory， kfed read 拥有 f1b1的disk的 aunum=2 blknum=位置:

kfed read <device name> aunum=2 blknum=2 | more

[grid@vmac1 ~]$ kfed read /dev/asm-diske|grep f1b1

kfdhdb.f1b1locn: 2 ; 0x0d4: 0x00000002

 

[grid@vmac1 ~]$ kfed read /dev/asm-diske aunum=2 blknum=2|more

kfbh.endian: 1 ; 0x000: 0x01

kfbh.hard: 130 ; 0x001: 0x82

kfbh.type: 4 ; 0x002: KFBTYP_FILEDIR

kfbh.datfmt: 1 ; 0x003: 0x01

kfbh.block.blk: 2 ; 0x004: blk=2

kfbh.block.obj: 1 ; 0x008: file=1

kfbh.check: 322527999 ; 0x00c: 0x133962ff

kfbh.fcn.base: 0 ; 0x010: 0x00000000

kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

kfbh.spare1: 0 ; 0x018: 0x00000000

kfbh.spare2: 0 ; 0x01c: 0x00000000

kfffdb.node.incarn: 1 ; 0x000: A=1 NUMM=0x0

kfffdb.node.frlist.number: 4294967295 ; 0x004: 0xffffffff

…………………….

kfffde[0].xptr.au: 3 ; 0x4a0: 0x00000003

kfffde[0].xptr.disk: 0 ; 0x4a4: 0x0000

kfffde[0].xptr.flags: 0 ; 0x4a6: L=0 E=0 D=0 S=0

kfffde[0].xptr.chk: 41 ; 0x4a7: 0x29

kfffde[1].xptr.au: 4294967295 ; 0x4a8: 0xffffffff

kfffde[1].xptr.disk: 65535 ; 0x4ac: 0xffff

kfffde[1].xptr.flags: 0 ; 0x4ae: L=0 E=0 D=0 S=0

kfffde[1].xptr.chk: 42 ; 0x4af: 0x2a

==》disk directory 位于 disk=0 的 aunum=3

NOTE: cache opening disk 0 of grp 2: DATADG_0000 path:/dev/asm-diske ==> 还是 asm-diske

[grid@vmac1 ~]$ kfed read /dev/asm-diske aunum=3 blknum=0|more

kfbh.endian: 1 ; 0x000: 0x01

kfbh.hard: 130 ; 0x001: 0x82

kfbh.type: 6 ; 0x002: KFBTYP_DISKDIR

kfbh.datfmt: 1 ; 0x003: 0x01

kfbh.block.blk: 0 ; 0x004: blk=0

kfbh.block.obj: 2 ; 0x008: file=2

kfbh.check: 389127513 ; 0x00c: 0x17319d59

kfbh.fcn.base: 31299 ; 0x010: 0x00007a43

kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

kfbh.spare1: 0 ; 0x018: 0x00000000

kfbh.spare2: 0 ; 0x01c: 0x00000000

kffdnd.bnode.incarn: 1 ; 0x000: A=1 NUMM=0x0

kffdnd.bnode.frlist.number: 4294967295 ; 0x004: 0xffffffff

kffdnd.bnode.frlist.incarn: 0 ; 0x008: A=0 NUMM=0x0

kffdnd.overfl.number: 4294967295 ; 0x00c: 0xffffffff

kffdnd.overfl.incarn: 0 ; 0x010: A=0 NUMM=0x0

kffdnd.parent.number: 0 ; 0x014: 0x00000000

kffdnd.parent.incarn: 1 ; 0x018: A=1 NUMM=0x0

kffdnd.fstblk.number: 0 ; 0x01c: 0x00000000

kffdnd.fstblk.incarn: 1 ; 0x020: A=1 NUMM=0x0

kfddde[0].entry.incarn: 1 ; 0x024: A=1 NUMM=0x0

kfddde[0].entry.hash: 0 ; 0x028: 0x00000000

kfddde[0].entry.refer.number:4294967295 ; 0x02c: 0xffffffff

kfddde[0].entry.refer.incarn: 0 ; 0x030: A=0 NUMM=0x0

………………………………………………….

kfddde 结构为 disk directory结构，仅有kfddde[0].entry.incarn A=1的记录是已分配的记录，若A=0则说明该记录被删除。

[grid@vmac1 ~]$ grep “kfddde\[5\]” disk.txt 

kfddde[5].entry.incarn: 1 ; 0x8e4: A=1 NUMM=0x0

kfddde[5].entry.hash: 5 ; 0x8e8: 0x00000005

kfddde[5].entry.refer.number:4294967295 ; 0x8ec: 0xffffffff

kfddde[5].entry.refer.incarn: 0 ; 0x8f0: A=0 NUMM=0x0

kfddde[5].dsknum: 5 ; 0x8f4: 0x0005

kfddde[5].state: 2 ; 0x8f6: KFDSTA_NORMAL

kfddde[5].ddchgfl: 132 ; 0x8f7: 0x84

kfddde[5].dskname: DATADG_0005 ; 0x8f8: length=11

kfddde[5].fgname: DATADG_0005 ; 0x918: length=11

kfddde[5].crestmp.hi: 32984459 ; 0x938: HOUR=0xb DAYS=0xc MNTH=0x3 YEAR=0x7dd

kfddde[5].crestmp.lo: 2470649856 ; 0x93c: USEC=0x0 MSEC=0xc8 SECS=0x34 MINS=0x24

kfddde[5].failstmp.hi: 0 ; 0x940: HOUR=0x0 DAYS=0x0 MNTH=0x0 YEAR=0x0

kfddde[5].failstmp.lo: 0 ; 0x944: USEC=0x0 MSEC=0x0 SECS=0x0 MINS=0x0

kfddde[5].timer: 0 ; 0x948: 0x00000000

kfddde[5].size: 5120 ; 0x94c: 0x00001400

kfddde[5].srRloc.super.hiStart: 0 ; 0x950: 0x00000000

kfddde[5].srRloc.super.loStart: 0 ; 0x954: 0x00000000

kfddde[5].srRloc.super.length: 0 ; 0x958: 0x00000000

kfddde[5].srRloc.incarn: 0 ; 0x95c: 0x00000000

kfddde[5].dskrprtm: 0 ; 0x960: 0x00000000

kfddde[5].zones[0].start: 0 ; 0x964: 0x00000000

kfddde[5].zones[0].size: 5120 ; 0x968: 0x00001400

kfddde[5].zones[0].used: 2 ; 0x96c: 0x00000002

 

回到编辑fix.txt上来， 调整 crestmp.hi 和 crestmp.lo 匹配上面显示的信息，若已经匹配则无需改动。

原本

[grid@vmac1 ~]$ egrep “hi|lo” fix.txt 

kfbh.block.blk: 0 ; 0x004: blk=0

kfbh.block.obj: 2147483653 ; 0x008: disk=5

kfdhdb.crestmp.hi: 32983191 ; 0x0a8: HOUR=0x17 DAYS=0x4 MNTH=0x2 YEAR=0x7dd

kfdhdb.crestmp.lo: 2328519680 ; 0x0ac: USEC=0x0 MSEC=0x299 SECS=0x2c MINS=0x22

kfdhdb.mntstmp.hi: 32984468 ; 0x0b0: HOUR=0x14 DAYS=0xc MNTH=0x3 YEAR=0x7dd

kfdhdb.mntstmp.lo: 1231840256 ; 0x0b4: USEC=0x0 MSEC=0x319 SECS=0x16 MINS=0x12

kfdhdb.fstlocn: 1 ; 0x0cc: 0x00000001

kfdhdb.altlocn: 2 ; 0x0d0: 0x00000002

kfdhdb.f1b1locn: 0 ; 0x0d4: 0x00000000

kfdhdb.grpstmp.hi: 32983191 ; 0x0e4: HOUR=0x17 DAYS=0x4 MNTH=0x2 YEAR=0x7dd

kfdhdb.grpstmp.lo: 2328331264 ; 0x0e8: USEC=0x0 MSEC=0x1e1 SECS=0x2c MINS=0x22

改动后

kfdhdb.crestmp.hi: 32984459 ; 0x938: HOUR=0xb DAYS=0xc MNTH=0x3 YEAR=0x7dd

kfdhdb.crestmp.lo: 2470649856 ; 0x93c: USEC=0x0 MSEC=0xc8 SECS=0x34 MINS=0x24

kfdhdb.mntstmp.hi: 32984468 ; 0x0b0: HOUR=0x14 DAYS=0xc MNTH=0x3 YEAR=0x7dd

kfdhdb.mntstmp.lo: 1231840256 ; 0x0b4: USEC=0x0 MSEC=0x319 SECS=0x16 MINS=0x12

 

之后使用kfed merge命令合并disk header

kfed merge <device name> text=fix.txt

[grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt

 

若使用ASMLIb，则使用例如以下命令修复header中的asmlib信息

 

/etc/init.d/oracleasm force-renamedisk /dev/sdbg1 <ASMLIB Disk Name>

/etc/init.d/oracleasm scandisks

/etc/init.d/oracleasm listdisks

 

 

之后启动ASM实例到nomount

SQL> startup nomount;

SQL> col path for a20

SQL> set linesize 200 pagesize 1400

SQL> select path,header_status,state from v$asm_disk;

PATH HEADER_STATUS STATE

——————– ———————————— ————————

/dev/asm-diskh MEMBER NORMAL

/dev/asm-diskg MEMBER NORMAL

/dev/asm-diskf MEMBER NORMAL

/dev/asm-diske MEMBER NORMAL

/dev/asm-diskc MEMBER NORMAL

/dev/asm-diskd MEMBER NORMAL

/dev/asm-diskb MEMBER NORMAL

7 rows selected.

检查头部信息是否为member

 

 

 

[grid@vmac1 ~]$ kfed read /dev/asm-diskh

kfbh.endian: 1 ; 0x000: 0x01

kfbh.hard: 130 ; 0x001: 0x82

kfbh.type: 1 ; 0x002: KFBTYP_DISKHEAD

kfbh.datfmt: 1 ; 0x003: 0x01

kfbh.block.blk: 0 ; 0x004: blk=0

kfbh.block.obj: 2147483653 ; 0x008: disk=5

kfbh.check: 3412972861 ; 0x00c: 0xcb6dd53d

kfbh.fcn.base: 0 ; 0x010: 0x00000000

kfbh.fcn.wrap: 0 ; 0x014: 0x00000000

kfbh.spare1: 0 ; 0x018: 0x00000000

kfbh.spare2: 0 ; 0x01c: 0x00000000

kfdhdb.driver.provstr: ORCLDISK ; 0x000: length=8

kfdhdb.driver.reserved[0]: 0 ; 0x008: 0x00000000

kfdhdb.driver.reserved[1]: 0 ; 0x00c: 0x00000000

kfdhdb.driver.reserved[2]: 0 ; 0x010: 0x00000000

kfdhdb.driver.reserved[3]: 0 ; 0x014: 0x00000000

kfdhdb.driver.reserved[4]: 0 ; 0x018: 0x00000000

 

之后 alter diskgroup 问题DG mount;

若上述步骤无问题则能够正常mount dg;

SQL> 

SQL> alter diskgroup datadg mount;

Diskgroup altered.

NOTE: cache registered group DATADG number=1 incarn=0x01c845f0

NOTE: cache began mount (first) of group DATADG number=1 incarn=0x01c845f0

NOTE: Assigning number (1,5) to disk (/dev/asm-diskh)

NOTE: Assigning number (1,2) to disk (/dev/asm-diskg)

NOTE: Assigning number (1,1) to disk (/dev/asm-diskf)

NOTE: Assigning number (1,0) to disk (/dev/asm-diske)

Wed Mar 13 19:39:49 2013

NOTE: GMON heartbeating for grp 1

GMON querying group 1 at 56 for pid 27, osid 8690

NOTE: cache opening disk 0 of grp 1: DATADG_0000 path:/dev/asm-diske

NOTE: F1X0 found on disk 0 au 2 fcn 0.31322

NOTE: cache opening disk 1 of grp 1: DATADG_0001 path:/dev/asm-diskf

NOTE: cache opening disk 2 of grp 1: DATADG_0002 path:/dev/asm-diskg

NOTE: cache opening disk 5 of grp 1: DATADG_0005 path:/dev/asm-diskh

NOTE: cache mounting (first) external redundancy group 1/0x01C845F0 (DATADG)

Wed Mar 13 19:39:49 2013

* allocate domain 1, invalid = TRUE 

kjbdomatt send to inst 2

Wed Mar 13 19:39:49 2013

NOTE: attached to recovery domain 1

NOTE: starting recovery of thread=1 ckpt=11.2351 group=1 (DATADG)

NOTE: advancing ckpt for group 1 (DATADG) thread=1 ckpt=11.2351

NOTE: cache recovered group 1 to fcn 0.33763

NOTE: redo buffer size is 256 blocks (1053184 bytes)

Wed Mar 13 19:39:49 2013

NOTE: LGWR attempting to mount thread 1 for diskgroup 1 (DATADG)

NOTE: LGWR found thread 1 closed at ABA 11.2350

NOTE: LGWR mounted thread 1 for diskgroup 1 (DATADG)

NOTE: LGWR opening thread 1 at fcn 0.33763 ABA 12.2351

NOTE: cache mounting group 1/0x01C845F0 (DATADG) succeeded

NOTE: cache ending mount (success) of group DATADG number=1 incarn=0x01c845f0

GMON querying group 1 at 57 for pid 18, osid 2911

Wed Mar 13 19:39:49 2013

NOTE: Instance updated compatible.asm to 11.2.0.0.0 for grp 1

SUCCESS: diskgroup DATADG was mounted

SUCCESS: alter diskgroup datadg mount

Wed Mar 13 19:39:49 2013

NOTE: diskgroup resource ora.DATADG.dg is online

NOTE: diskgroup resource ora.DATADG.dg is updated

Wed Mar 13 19:39:59 2013

NOTE: client PROD1:PROD registered, osid 10169, mbr 0x1

Wed Mar 13 19:40:11 2013

NOTE: ASM client PROD1:PROD disconnected unexpectedly.

NOTE: check client alert log.

NOTE: Trace records dumped in trace file /g01/orabase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_10169.trc

 

 

 

 

注意 以上修复仅针对 asm header丢失磁盘头大约20MB(不一定)空间的数据， 若很多其它则仍可能无法mount diskgroup

若file numbe=4 Continuing Operations Directory (COD) – ASM file number 4也丢失了 那么一般非常难mount了。在以下的样例中是前22MB丢失时即便kfed merge也无力回天了。

SELECT x.xnum_kffxp “Extent”,

x.au_kffxp “AU”,

x.disk_kffxp “Disk #”,

d.name “Disk name”

FROM x$kffxp x, v$asm_disk_stat d

WHERE x.group_kffxp=d.group_number

and x.disk_kffxp=d.disk_number

and x.group_kffxp=1

and x.number_kffxp=4

ORDER BY 1, 2;

Extent AU Disk # Disk name

———- ———- ———- ——————————————————————————————

0 21 5 DATADG_0005

1 16 1 DATADG_0001

2 33 2 DATADG_0002

3 34 0 DATADG_0000

4 22 5 DATADG_0005

5 34 2 DATADG_0002

6 35 0 DATADG_0000

7 33 1 DATADG_0001

SQL> alter diskgroup datadg dismount;

Diskgroup altered.

[grid@vmac1 ~]$ dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=20

20+0 records in

20+0 records out

20971520 bytes (21 MB) copied, 0.0165823 s, 1.3 GB/s

 

[grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt

SQL> alter diskgroup datadg mount;

Diskgroup altered.

SQL> alter diskgroup datadg mount;

Diskgroup altered.

SQL> alter diskgroup datadg dismount;

Diskgroup altered.

[grid@vmac1 ~]$ dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=21

21+0 records in

21+0 records out

22020096 bytes (22 MB) copied, 0.0182842 s, 1.2 GB/s

[grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt

SQL> alter diskgroup datadg mount;

Diskgroup altered.

 

[grid@vmac1 ~]$ dd if=/dev/zero of=/dev/asm-diskh bs=1024k count=22

22+0 records in

22+0 records out

23068672 bytes (23 MB) copied, 0.0312157 s, 739 MB/s

[grid@vmac1 ~]$ kfed merge /dev/asm-diskh text=fix.txt

SQL> alter diskgroup datadg mount;

alter diskgroup datadg mount

*

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15130: diskgroup “DATADG” is being dismounted

ORA-15066: offlining disk “DATADG_0005″ in group “DATADG” may result in a data loss

ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]

 

Errors in file /g01/orabase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8690.trc:

ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]

ERROR: cache failed to read group=1(DATADG) fn=4 blk=0 from disk(s): 5(DATADG_0005)

ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]

NOTE: cache initiating offline of disk 5 group DATADG

NOTE: process _user8690_+asm1 (8690) initiating offline of disk 5.3915953639 (DATADG_0005) with mask 0x7e in group 1

WARNING: Disk 5 (DATADG_0005) in group 1 in mode 0x7f is now being taken offline on ASM inst 1

NOTE: initiating PST update: grp = 1, dsk = 5/0xe968b5e7, mask = 0x6a, op = clear

GMON updating disk modes for group 1 at 108 for pid 27, osid 8690

ERROR: Disk 5 cannot be offlined, since diskgroup has external redundancy.

ERROR: too many offline disks in PST (grp 1)

WARNING: Offline of disk 5 (DATADG_0005) in group 1 and mode 0x7f failed on ASM inst 1

Wed Mar 13 20:00:56 2013

NOTE: halting all I/Os to diskgroup 1 (DATADG)

System State dumped to trace file /g01/orabase/diag/asm/+asm/+ASM1/trace/+ASM1_ora_8690.trc

NOTE: AMDU dump of disk group DATADG created at /g01/orabase/diag/asm/+asm/+ASM1/trace

ERROR: ORA-15130 signalled during mount of diskgroup DATADG

NOTE: cache dismounting (clean) group 1/0xEB784617 (DATADG) 

NOTE: messaging CKPT to quiesce pins Unix process pid: 8690, image: oracle@vmac1 (TNS V1-V3)

NOTE: LGWR doing non-clean dismount of group 1 (DATADG)

NOTE: LGWR sync ABA=18.2360 last written ABA 18.2360

kjbdomdet send to inst 2

detach from dom 1, sending detach message to inst 2

Wed Mar 13 20:00:57 2013

List of instances:

1 2

Dirty detach reconfiguration started (new ddet inc 1, cluster inc 12)

Global Resource Directory partially frozen for dirty detach

* dirty detach – domain 1 invalid = TRUE 

0 GCS resources traversed, 0 cancelled

Dirty Detach Reconfiguration complete

freeing rdom 1

WARNING: dirty detached from domain 1

NOTE: cache dismounted group 1/0xEB784617 (DATADG) 

NOTE: cache ending mount (fail) of group DATADG number=1 incarn=0xeb784617

NOTE: cache deleting context for group DATADG 1/0xeb784617

GMON dismounting group 1 at 109 for pid 27, osid 8690

NOTE: Disk in mode 0x8 marked for de-assignment

NOTE: Disk in mode 0x8 marked for de-assignment

NOTE: Disk in mode 0x8 marked for de-assignment

NOTE: Disk in mode 0x8 marked for de-assignment

ERROR: diskgroup DATADG was not mounted

ORA-15032: not all alterations performed

ORA-15130: diskgroup “DATADG” is being dismounted

ORA-15066: offlining disk “DATADG_0005″ in group “DATADG” may result in a data loss

ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]

ERROR: alter diskgroup datadg mount

SQL> alter diskgroup datadg mount;

alter diskgroup datadg mount

*

ERROR at line 1:

ORA-15032: not all alterations performed

ORA-15130: diskgroup “DATADG” is being dismounted

ORA-15066: offlining disk “DATADG_0005″ in group “DATADG” may result in a data loss

ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 != 1]

ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 !=1]

ORA-15196: invalid ASM block header [kfc.c:26077] [endian_kfbh] [4] [0] [0 !=1]

[grid@vmac1 trace]$ dd if=/dev/asm-diske of=/dev/asm-diskh bs=4096 skip=3 seek=3 count=1

1+0 records in

1+0 records out

4096 bytes (4.1 kB) copied, 0.000617397 s, 6.6 MB/s

 

[grid@vmac1 trace]$ dd if=/dev/asm-diske of=/dev/asm-diskh bs=4096 skip=4 seek=4 count=1

kfffde[0].xptr.au: 21 ; 0x4a0: 0x00000015

kfffde[0].xptr.disk: 5 ; 0x4a4: 0x0005

kfffde[0].xptr.flags: 0 ; 0x4a6: L=0 E=0 D=0 S=0

kfffde[0].xptr.chk: 58 ; 0x4a7: 0x3a

kfffde[1].xptr.au: 16 ; 0x4a8: 0x00000010

 

[grid@vmac1 trace]$ dd if=/dev/asm-diske of=/dev/asm-diskh bs=1048576 skip=21 seek=21 count=1

1+0 records in

1+0 records out

1048576 bytes (1.0 MB) copied, 0.00296742 s, 353 MB/s

 

ORA-15066: offlining disk “DATADG_0005″ in group “DATADG” may result in a data loss

ORA-15196: invalid ASM block header [kfc.c:26077] [obj_kfbl] [4] [0] [3 != 4]

ORA-15196: invalid ASM block header [kfc.c:26077] [obj_kfbl] [4] [0] [3 != 4]

ERROR: alter diskgroup datadg mount force