为RAC私有网络配置网卡Bonding

在RAC的安装部署过程中。并不不过简单的安装完毕了事。整个安装过程要考虑可能出现的单点问题，当中比較重要的是私有网络。

私有网络是RAC节点间通信的通道。包含节点间的网络心跳信息、Cache fusion传递数据块都须要通过私有网络。而非常多的私有网络都不过一块单独的网卡连接上交换机就完毕了，更有甚者，直接使用server间网卡互连的方式配置私有网络。这样的部署方式简单，但RAC投入使用后风险非常大，存在诸多单点如网卡、网线、交换机口、交换机。

差点儿每一个组件发生问题都会导致RAC
split，所以建议为私有网络配置双网卡bonding。

以下是我的配置步骤：

环境：

OS：CentOS release 6.4 (Final)

Oracle：11.2.0.4 RAC

个 em1,em2,em3,em4，当前em1作为公有网卡，em3作为私有网卡已经启用了。em2和em4闲置。

配置bond模块并载入（在2个节点运行）：

编辑/etc/modprobe.d/bonding.conf增加内容：

[root@node2 ~]# vi /etc/modprobe.d/bonding.conf

alias bond0 bonding

[root@node2 ~]# modprobe -a bond0

验证：

[root@node2 ~]#  lsmod  |grep bond

bonding               127331  0

8021q                  25317  1 bonding

ipv6                  321422  274 bonding,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6

编辑网卡配置文件。编辑成例如以下内容：

节点一：

Ifcfg-em2:

DEVICE=em2

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

SLAVE=yes

Ifcfg-em4:

DEVICE=em4

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

SLAVE=yes

Ifcfg-bond0:

DEVICE=bond0

MASTER=yes

BOOTPROTO=node

ONBOOT=yes

BONDING_OPTS="mode=1 miimon=100"

IPADDR=10.10.10.105

PREFIX=24

GATEWAY=10.10.10.1

节点二：

ifcfg-em2:

DEVICE=em2

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

SLAVE=yes

ifcfg-em4:

DEVICE=em4

BOOTPROTO=none

ONBOOT=yes

MASTER=bond0

SLAVE=yes

Ifcfg-bond0:

DEVICE=bond0

MASTER=yes

BOOTPROTO=node

ONBOOT=yes

BONDING_OPTS="mode=1 miimon=100"

IPADDR=10.10.10.106

PREFIX=24

GATEWAY=10.10.10.1

我这里使用的是mode=1的主备网卡模式，平时仅仅激活一块网卡，一旦主网卡发生问题，会切换链路到备网卡，其它也能够考虑4,6两种mode。

改动完了配置文件之后。分别在2个节点启动bond0：ifup bond0。

此时能够看到：

[root@node1 ~]# ifconfig

bond0     Link encap:Ethernet  HWaddr C8:1F:66:FB:6F:CB

inet addr:10.10.10.105  Bcast:10.10.10.255  Mask:255.255.255.0

inet6 addr: fe80::ca1f:66ff:fefb:6fcb/64 Scope:Link

UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1

RX packets:9844809 errors:0 dropped:0 overruns:0 frame:0

TX packets:7731078 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:0

RX bytes:9097132073 (8.4 GiB)  TX bytes:6133004979 (5.7 GiB)

em2       Link encap:Ethernet  HWaddr C8:1F:66:FB:6F:CB

UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

RX packets:9792915 errors:0 dropped:0 overruns:0 frame:0

TX packets:7731078 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:9088278883 (8.4 GiB)  TX bytes:6133004979 (5.7 GiB)

Interrupt:38

em4       Link encap:Ethernet  HWaddr C8:1F:66:FB:6F:CB

UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1

RX packets:51894 errors:0 dropped:0 overruns:0 frame:0

TX packets:0 errors:0 dropped:0 overruns:0 carrier:0

collisions:0 txqueuelen:1000

RX bytes:8853190 (8.4 MiB)  TX bytes:0 (0.0 b)

Interrupt:36

网卡的bonding已经配置成功了。

測试验证

此时能够測试分别断掉em2 em4。在一个节点长ping还有一个节点的私有IP。并结合/proc/net/bonding/bond0的信息观察primary
slave的变化，能够发现当down一个网卡时ping不会中断。

Bond0配置好之后，接下来一步就是把配置成RAC的私有网卡。

为了避免配置失败，首先要备份好原来的配置文件。

以grid用户在2个节点对$GRID_HOME/ grid/gpnp/noden/profiles/peer/profile.xml文件运行备份：

cd /u01/app/11.2.0/grid/gpnp/noden/profiles/peer

cp  profile.xml  profile.xml.bk

[root@node2 peer]# ls

pending.xml  profile_orig.xml  profile.xml  profile.xml.bk、

查看眼下的私有网络配置：

node2-> oifcfg getif

em1  192.168.10.0  global  public

em3  10.10.10.0  global  cluster_interconnect

先加入新的私有网络，在任一节点运行就可以：

node1-> oifcfg setif -global bond0/10.10.10.0:cluster_interconnect

这一步在运行时可能会报错：

node1-> oifcfg setif -global bond0/10.10.10.0:cluster_interconnect

PRIF-33: Failed to set or delete interface because hosts could not be discovered

CRS-02307: No GPnP services on requested remote hosts.

PRIF-32: Error in checking for profile availability for host node2

CRS-02306: GPnP service on host "node2" not found.

这是由于gpnpd服务异常导致的。

解决方法：能够Kill掉gpnpd进程。GI会自己主动又一次启动gpnpd服务。

在2个节点运行：

[root@node2 ~]# ps -ef| grep gpnp

grid      4927     1  0 Sep22 ?        00:26:38 /u01/app/11.2.0/grid/bin/gpnpd.bin

grid     48568 46762  0 17:26 pts/3    00:00:00 tail -f /u01/app/11.2.0/grid/log/node2/gpnpd/gpnpd.log

root     48648 48623  0 17:26 pts/4    00:00:00 grep gpnp

[root@node2 ~]# kill -9 4927

[root@node2 ~]#

參考gpnpd.log

加入私有网络之后，我们依照例如以下步骤将原来的私有网络删除：

首先停止并disable掉crs。

以root用户在2个节点分别运行下面命令：

停止crs

crsctl stop crs

禁用crs

crsctl disable crs

改动hosts文件，将私有IP地址改为新地址。

2个节点分别运行：

ping node1-priv

ping node2-priv

再启动crs。

[root@node2 ~]# crsctl enable crs

CRS-4622: Oracle High Availability Services autostart is enabled.

[root@node2 ~]# crsctl start crs

删除原来的私有网络：

node2-> oifcfg delif -global em3/10.10.10.0:cluster_interconnect

检查验证，配置成功了。

node2-> oifcfg getif

em1  192.168.10.0  global  public

bond0  10.10.10.0  global  cluster_interconnect

node2->

以下做一个測试验证下bonding的效果：

ifdown掉em2，此时messages会出现日志信息：

Oct 25 22:00:32 node1 kernel: bonding: bond0: Removing slave em2

Oct 25 22:00:32 node1 kernel: bonding: bond0: Warning: the permanent HWaddr of em2 - c8:1f:66:fb:6f:cb - is still in use by bond0. Set the HWaddr of em2 to a different address to avoid conflicts.

Oct 25 22:00:32 node1 kernel: bonding: bond0: releasing active interface em2

Oct 25 22:00:32 node1 kernel: bonding: bond0: making interface em4 the new active one.

此时bond0自己主动切换到em4上，所以这个时候ping私有IP是没有问题的。

查看/proc/net/bonding/bond0发现active的slave已经变成em4了：

[root@node1 ~]# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: em4

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

Slave Interface: em4

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 3

Permanent HW addr: c8:1f:66:fb:6f:cd

Slave queue ID: 0

[root@node1 ~]#

crsd.log ocssd.log中没有报错信息。css仍然每隔5秒发送一次网络心跳。

2014-10-25 22:00:32.975: [    CSSD][656893696]clssnmSendingThread: sent 5 status msgs to all nodes

2014-10-25 22:00:37.977: [    CSSD][656893696]clssnmSendingThread: sending status msg to all nodes

2014-10-25 22:00:37.977: [    CSSD][656893696]clssnmSendingThread: sent 5 status msgs to all nodes

2014-10-25 22:00:42.978: [    CSSD][656893696]clssnmSendingThread: sending status msg to all nodes

这说明bonding确实做到了保护私有网络的单点故障。

此时再down掉em4：

[root@node1 ~]# ifdown em4

Em4down掉之后，私有IP无法从Node2上ping通了，此时bond0也down掉：

[root@node1 ~]# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: None

MII Status: down

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

此时messages信息：

Oct 25 22:02:23 node1 kernel: bonding: bond0: Removing slave em4

Oct 25 22:02:23 node1 kernel: bonding: bond0: releasing active interface em4

ocssd.log日志显示2秒后css已经检測到了私有网络故障：

2014-10-25 22:02:25.573: [GIPCHGEN][1744828160] gipchaInterfaceFail: marking interface failing 0x7f025c00c0a0 { host '', haName 'c617-7010-b72d-6c39', local (nil), ip '10.10.10.105:46469', subnet '10.10.10.0',
mask '255.255.255.0', mac 'c8-1f-66-fb-6f-cb', ifname 'bond0', numRef 1, numFail 0, idxBoot 0, flags 0xd }

2014-10-25 22:02:25.661: [GIPCHGEN][1951459072] gipchaInterfaceFail: marking interface failing 0x7f025c023d90 { host 'node2', haName 'ba2c-9227-ca29-8a21', local 0x7f025c00c0a0, ip '10.10.10.106:32369', subnet
'10.10.10.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 2, flags 0x6 }

2014-10-25 22:02:27.663: [GIPCHTHR][1951459072] gipchaWorkerCreateInterface: created remote interface for node 'node2', haName 'ba2c-9227-ca29-8a21', inf 'udp://10.10.10.106:32369'

而且发现本server的私有网络连不上了

2014-10-25 22:02:27.572: [GIPCHDEM][536868608] gipchaDaemonInfRequest: sent local interfaceRequest,  hctx 0xc99250 [0000000000000010] { gipchaContext : host 'node1', name 'CSS_node-cluster', luid 'e2a491a6-00000000',
numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd

2014-10-25 22:02:31.012: [    CSSD][656893696]clssnmSendingThread: sending status msg to all nodes

2014-10-25 22:02:31.012: [GIPCHALO][669509376] gipchaLowerProcessNode:
no valid interfaces found to node for 5530 ms, node 0x7f0c18065e40 { host 'node2', haName 'CSS_node-cluster', srcLuid e2a491a6-d8e24a48, dstLuid ebce4a7f-2b4e4348 numInf 0, contigSeq 317864, lastAck 317717, lastValidAck 317864,
sendSeq [317718 : 317729], createTime 4294073210, sentRegister 1, localMonitor 1, flags 0x2408 }

2014-10-25 22:02:31.012: [    CSSD][656893696]clssnmSendingThread: sent 5 status msgs to all nodes

2014-10-25 22:02:33.573: [GIPCHDEM][536868608] gipchaDaemonInfRequest: sent local interfaceRequest,  hctx 0xc99250 [0000000000000010] { gipchaContext : host 'node1', name 'CSS_node-cluster', luid 'e2a491a6-00000000',
numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd

2014-10-25 22:02:36.013: [    CSSD][656893696]clssnmSendingThread: sending status msg to all nodes

2014-10-25 22:02:36.013: [    CSSD][656893696]clssnmSendingThread: sent 5 status msgs to all nodes

2014-10-25 22:02:37.014: [GIPCHALO][669509376] gipchaLowerProcessNode: no valid interfaces found to node for 11530 ms, node 0x7f0c18065e40 { host 'node2', haName 'CSS_node-cluster', srcLuid e2a491a6-d8e24a48,
dstLuid ebce4a7f-2b4e4348 numInf 0, contigSeq 317864, lastAck 317717, lastValidAck 317864, sendSeq [317718 : 317741], createTime 4294073210, sentRegister 1, localMonitor 1, flags 0x2408 }

显示节点二的网络心跳失败，14.34秒之后会将节点二移出集群：

2014-10-25 22:02:39.010: [    CSSD][658470656]clssnmPollingThread: node node2 (2) at 50% heartbeat fatal, removal in 14.340 seconds

2014-10-25 22:02:39.010: [    CSSD][658470656]clssnmPollingThread: node node2 (2) is impending reconfig, flag 2228230, misstime 15660

网络心跳失败容忍多长时间由css中的Misscount决定，RAC默认配置是30秒：

node2-> crsctl get  css  misscount

CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.

此时開始报has a disk HB, but no network HB：

2014-10-25 22:02:53.012: [    CSSD][664778496]clssnmvDHBValidateNcopy: node 2, node2, has a disk HB, but no network HB, DHB has rcfg 309527290, wrtcnt, 3048511, LATS 212362014, lastSeqNo 3048510, uniqueness
1414032467, timestamp 1414245773/212369384

2014-10-25 22:02:53.192: [    CSSD][667932416]clssnmvDiskPing: Writing with status 0x3, timestamp 1414245773/212362194

2014-10-25 22:02:53.352: [    CSSD][658470656]clssnmPollingThread: Removal started for node node2 (2), flags 0x220006, state 3, wt4c 0

2014-10-25 22:02:53.352: [    CSSD][658470656]clssnmMarkNodeForRemoval: node 2, node2 marked for removal

2014-10-25 22:02:53.352: [    CSSD][658470656]clssnmDiscHelper: node2, node(2) connection failed, endp (0x6b7), probe(0x7f0c00000000), ninf->endp 0x6b7

2014-10-25 22:02:53.352: [    CSSD][658470656]clssnmDiscHelper: node 2 clean up, endp (0x6b7), init state 5, cur state 5

2014-10-25 22:02:53.352: [GIPCXCPT][658470656] gipcInternalDissociate: obj 0x7f0bf00084c0 [00000000000006b7] { gipcEndpoint : localAddr 'gipcha://node1:c5bc-f486-c390-b48', remoteAddr 'gipcha://node2:nm2_node-cluster/b370-8934-efb4-3f2',
numPend 1, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 1, wobj 0x7f0bf000a3f0, sendp (nil)flags 0x38606, usrFlags 0x0 } not associated with any container, ret gipcretFail (1)

@

之后node2被踢出：

2014-10-25 22:02:53.353: [    CSSD][655316736]clssnmNeedConfReq: No configuration to change

2014-10-25 22:02:53.353: [    CSSD][655316736]clssnmDoSyncUpdate: Terminating node 2, node2, misstime(30000) state(5)

2014-10-25 22:02:53.353: [    CSSD][655316736]clssnmDoSyncUpdate: Wait for 0 vote ack(s)

2014-10-25 22:02:53.353: [    CSSD][655316736]clssnmCheckDskInfo: Checking disk info...

2014-10-25 22:02:53.353: [    CSSD][655316736]clssnmCheckSplit: Node 2, node2, is alive, DHB (1414245773, 212369384) more than disk timeout of 27000 after the last NHB (1414245743, 212339734)

2014-10-25 22:02:53.353: [    CSSD][655316736]clssnmCheckDskInfo: My cohort: 1

2014-10-25 22:02:53.353: [    CSSD][891287296]clssgmQueueGrockEvent: groupName(crs_version) count(3) master(0) event(2), incarn 3, mbrc 3, to member 2, events 0x0, state 0x0

2014-10-25 22:02:53.353: [    CSSD][655316736]clssnmRemove: Start

2014-10-25 22:02:53.353: [    CSSD][655316736](:CSSNM00007:)clssnmrRemoveNode: Evicting node 2, node2, from the cluster in incarnation 309527290, node birth incarnation 309527289, death incarnation 309527290,
stateflags 0x224000 uniqueness value 1414032467

2014-10-25 22:02:53.353: [    CSSD][891287296]clssgmQueueGrockEvent: groupName(IGSSZGDBsszgdb) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0

2014-10-25 22:02:53.353: [ default][655316736]kgzf_gen_node_reid2: generated reid cid=6d207e372096ef48ff1031c3298552d5,icin=309527289,nmn=2,lnid=309527289,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node
hdr=0xfece0100

Node2被隔离：

2014-10-25 22:02:53.353: [    CSSD][655316736]clssnmrFenceSage: Fenced node node2, number 2, with EXADATA, handle 0

2014-10-25 22:02:53.353: [    CSSD][655316736]clssnmSendShutdown: req to node 2, kill time 212362354

2014-10-25 22:02:53.353: [    CSSD][891287296]clssgmQueueGrockEvent: groupName(CRF-) count(4) master(0) event(2), incarn 4, mbrc 4, to member 2, events 0x38, state 0x0

2014-10-25 22:02:53.353: [    CSSD][655316736]clssnmsendmsg: not connected to node 2

2014-10-25 22:02:53.353: [    CSSD][655316736]clssnmSendShutdown: Send to node 2 failed

2014-10-25 22:02:53.353: [    CSSD][655316736]clssnmWaitOnEvictions: Start

防止node2写数据文件：

2014-10-25 22:02:53.354: [    CSSD][664778496]clssnmvDiskEvict: Kill block write, file /dev/asm_datafile flags 0x00010004, kill block unique 1414032467, stamp 212362354/212362354

crsd.log显示原来执行的Node2上的资源被切换到node1上执行：

2014-10-25 22:03:00.946: [   CRSPE][870311680]{1:61167:7502} CRS-2676: Start of 'ora.node2.vip' on 'node1' succeeded

2014-10-25 22:03:00.946: [   CRSPE][870311680]{1:61167:7502} CRS-2676: Start of 'ora.node2.vip' on 'node1' succeeded

2014-10-25 22:03:02.358: [   CRSPE][870311680]{1:61167:7502} CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node1' succeeded

而且此时Node2上的crs已经无法通信：

node2-> crs_stat -t

CRS-0184: Cannot communicate with the CRS daemon.

将em4 up：

ifup em4

[root@node1 ~]# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: None

MII Status: down

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

Slave Interface: em4

MII Status: down

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: c8:1f:66:fb:6f:cd

Slave queue ID: 0

查看messages：

Oct 25 22:03:43 node1 kernel: bonding: bond0: setting mode to active-backup (1).

Oct 25 22:03:43 node1 kernel: bonding: bond0: Setting MII monitoring interval to 100.

Oct 25 22:03:43 node1 kernel: bonding: bond0: Adding slave em4.

Oct 25 22:03:44 node1 kernel: bonding: bond0: enslaving em4 as a backup interface with a down link.

Oct 25 22:03:47 node1 kernel: tg3 0000:02:00.1: em4: Link is up at 1000 Mbps, full duplex

Oct 25 22:03:47 node1 kernel: tg3 0000:02:00.1: em4: Flow control is on for TX and on for RX

Oct 25 22:03:47 node1 kernel: tg3 0000:02:00.1: em4: EEE is disabled

手动启动em4之后，em4被自己主动设置为Slave Interface，由于之前bond0已经被down，所以须要手动启动bond0

ifup bond0

[root@node1 ~]# ifup bond0

[root@node1 ~]# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: em4

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

Slave Interface: em4

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: c8:1f:66:fb:6f:cd

Slave queue ID: 0

Slave Interface: em2

MII Status: down

Speed: Unknown

Duplex: Unknown

Link Failure Count: 0

Permanent HW addr: c8:1f:66:fb:6f:cb

Slave queue ID: 0

此时messages显示bond0已经就绪：

Oct 25 22:05:25 node1 kernel: bonding: bond0: Adding slave em2.

Oct 25 22:05:26 node1 kernel: bonding: bond0: enslaving em2 as a backup interface with a down link.

Oct 25 22:05:26 node1 kernel: ADDRCONF(NETDEV_UP): bond0: link is not ready

Oct 25 22:05:26 node1 kernel: 8021q: adding VLAN 0 to HW filter on device bond0

Oct 25 22:05:26 node1 kernel: bond0: link status definitely up for interface em4, 1000 Mbps full duplex.

Oct 25 22:05:26 node1 kernel: bonding: bond0: making interface em4 the new active one.

Oct 25 22:05:26 node1 kernel: bonding: bond0: first active interface up!

Oct 25 22:05:26 node1 kernel: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready

Ocssd.log也显示能够连接到node2的信息：

2014-10-25 22:05:28.026: [GIPCHGEN][536868608] gipchaNodeAddInterface: adding interface information for inf 0x7f0c14222be0 { host '', haName 'CSS_node-cluster', local (nil), ip '10.10.10.105', subnet '10.10.10.0',
mask '255.255.255.0', mac 'c8-1f-66-fb-6f-cd', ifname 'bond0', numRef 0, numFail 0, idxBoot 0, flags 0x1841 }

2014-10-25 22:05:28.254: [GIPCHTHR][669509376] gipchaWorkerCreateInterface: created local interface for node 'node1', haName 'CSS_node-cluster', inf 'udp://10.10.10.105:60625'

2014-10-25 22:05:28.255: [GIPCHTHR][669509376] gipchaWorkerCreateInterface: created local bootstrap multicast interface for node 'node1', haName 'CSS_node-cluster', inf 'mcast://224.0.0.251:42424/10.10.10.105'

2014-10-25 22:05:30.560: [    CSSD][653739776]clssnmSendConnAck: connected to node 2, node2, con (0xa111d5), state 0

2014-10-25 22:05:30.560: [    CSSD][653739776]clssnmCompleteConnProtocol: node node2, 2, uniqueness 1414245779, msg uniqueness 1414245779, endp 0xa111d5 probendp 0x7f0c00000000 endp 0xa111d5

2014-10-25 22:05:31.465: [    CSSD][653739776]clssnmHandleJoin: node 2 JOINING, state 0->1 ninfendp 0x7f0c00a111d5

2014-10-25 22:05:31.940: [    CSSD][664778496]clssnmvReadDskHeartbeat: Reading DHBs to get the latest info for node(2/node2), LATSvalid(0), nodeInfoDHB uniqueness(1414032467)

2014-10-25 22:05:31.940: [    CSSD][664778496]clssnmvDHBValidateNcopy: Setting LATS valid due to uniqueness change for node(node2) number(2), nodeInfoDHB(1414032467), readInfo(1414245779)

2014-10-25 22:05:31.940: [    CSSD][664778496]clssnmvDHBValidateNcopy: Saving DHB uniqueness for node node2, number 2 latestInfo(1414245779), readInfo(1414245779), nodeInfoDHB(1414032467)

2014-10-25 22:05:32.366: [    CSSD][655316736]clssnmDoSyncUpdate: Initiating sync 309527291

2014-10-25 22:05:32.366: [    CSSD][655316736]clssscCompareSwapEventValue: changed NMReconfigInProgress  val 1, from -1, changes 7

2014-10-25 22:05:32.366: [    CSSD][655316736]clssnmDoSyncUpdate: local disk timeout set to 200000 ms, remote disk timeout set to 200000

2014-10-25 22:05:32.366: [    CSSD][655316736]clssnmDoSyncUpdate: new values for local disk timeout and remote disk timeout will take effect when the sync is completed.

2014-10-25 22:05:32.366: [    CSSD][655316736]clssnmDoSyncUpdate: Starting cluster reconfig with incarnation 309527291

2014-10-25 22:05:32.366: [    CSSD][655316736]clssnmSetupAckWait: Ack message type (11)

2014-10-25 22:05:32.366: [    CSSD][655316736]clssnmSetupAckWait: node(1) is ALIVE

2014-10-25 22:05:32.366: [    CSSD][655316736]clssnmSetupAckWait: node(2) is ALIVE

2014-10-25 22:05:32.369: [    CSSD][655316736]clssnmDoSyncUpdate: node(2) is transitioning from joining state to active state

2014-10-25 22:05:32.370: [    CSSD][653739776]clssnmHandleUpdate: NODE 1 (node1) IS ACTIVE MEMBER OF CLUSTER

2014-10-25 22:05:32.370: [    CSSD][653739776]clssnmHandleUpdate: NODE 2 (node2) IS ACTIVE MEMBER OF CLUSTER

2014-10-25 22:05:32.370: [    CSSD][891287296]clssgmSuspendAllGrocks: done

crsd.log显示将节点1上执行的节点二资源停止并在node2上又一次启动：

2014-10-25 22:06:05.097: [   CRSPE][870311680]{2:25913:2} CRS-2677: Stop of 'ora.node2.vip' on 'node1' succeeded

2014-10-25 22:06:05.101: [   CRSPE][870311680]{2:25913:2} CRS-2672: Attempting to start 'ora.node2.vip' on 'node2'

此时资源又又一次飘回node2了：

node2-> crs_stat -t

Name           Type           Target    State     Host

------------------------------------------------------------

ora....ER.lsnr ora....er.type ONLINE    ONLINE    node1

ora....N1.lsnr ora....er.type ONLINE    ONLINE    node2

ora....N2.lsnr ora....er.type ONLINE    ONLINE    node1

ora....N3.lsnr ora....er.type ONLINE    ONLINE    node1

ora.OCR.dg     ora....up.type ONLINE    ONLINE    node1

ora.TEMP.dg    ora....up.type ONLINE    ONLINE    node1

ora.UNDO.dg    ora....up.type ONLINE    ONLINE    node1

ora.asm        ora.asm.type   ONLINE    ONLINE    node1

ora.cvu        ora.cvu.type   ONLINE    ONLINE    node1

ora.gsd        ora.gsd.type   OFFLINE   OFFLINE

ora....network ora....rk.type ONLINE    ONLINE    node1

ora....SM1.asm application    ONLINE    ONLINE    node1

ora....E1.lsnr application    ONLINE    ONLINE    node1

ora.node1.gsd  application    OFFLINE   OFFLINE

ora.node1.ons  application    ONLINE    ONLINE    node1

ora.node1.vip  ora....t1.type ONLINE    ONLINE    node1

ora....SM2.asm application    ONLINE    ONLINE    node2

ora....E2.lsnr application    ONLINE    ONLINE    node2

ora.node2.gsd  application    OFFLINE   OFFLINE

ora.node2.ons  application    ONLINE    ONLINE    node2

ora.node2.vip  ora....t1.type ONLINE    ONLINE    node2

ora.oc4j       ora.oc4j.type  ONLINE    ONLINE    node1

ora.ons        ora.ons.type   ONLINE    ONLINE    node1

ora.scan1.vip  ora....ip.type ONLINE    ONLINE    node2

ora.scan2.vip  ora....ip.type ONLINE    ONLINE    node1

ora.scan3.vip  ora....ip.type ONLINE    ONLINE    node1

ora.sszgdb.db  ora....se.type ONLINE    ONLINE    node1

我们我们能够将bond0的还有一个slave interface启用起来：

[root@node1 ~]# ifup em2

[root@node1 ~]# cat /proc/net/bonding/bond0

Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)

Bonding Mode: fault-tolerance (active-backup)

Primary Slave: None

Currently Active Slave: em4

MII Status: up

MII Polling Interval (ms): 100

Up Delay (ms): 0

Down Delay (ms): 0

Slave Interface: em4

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: c8:1f:66:fb:6f:cd

Slave queue ID: 0

Slave Interface: em2

MII Status: up

Speed: 1000 Mbps

Duplex: full

Link Failure Count: 0

Permanent HW addr: c8:1f:66:fb:6f:cb

Slave queue ID: 0

Messages：

Oct 25 22:05:29 node1 kernel: tg3 0000:01:00.1: em2: Link is up at 1000 Mbps, full duplex

Oct 25 22:05:29 node1 kernel: tg3 0000:01:00.1: em2: Flow control is on for TX and on for RX

Oct 25 22:05:29 node1 kernel: tg3 0000:01:00.1: em2: EEE is disabled

Oct 25 22:05:29 node1 kernel: bond0: link status definitely up for interface em2, 1000 Mbps full duplex.

能够看到em2已经就绪，私有网络又恢复到拥有2个网卡保驾护航的状态。