为RAC私有网络配置网卡Bonding
在RAC的安装部署过程中。并不不过简单的安装完毕了事。整个安装过程要考虑可能出现的单点问题，当中比較重要的是私有网络。
私有网络是RAC节点间通信的通道。包含节点间的网络心跳信息、Cache fusion传递数据块都须要通过私有网络。而非常多的私有网络都不过一块单独的网卡连接上交换机就完毕了，更有甚者，直接使用server间网卡互连的方式配置私有网络。这样的部署方式简单，但RAC投入使用后风险非常大，存在诸多单点如网卡、网线、交换机口、交换机。
差点儿每一个组件发生问题都会导致RAC
split，所以建议为私有网络配置双网卡bonding。
以下是我的配置步骤：
环境：
OS：CentOS release 6.4 (Final)
Oracle：11.2.0.4 RAC
个 em1,em2,em3,em4，当前em1作为公有网卡，em3作为私有网卡已经启用了。em2和em4闲置。
配置bond模块并载入（在2个节点运行）：
编辑/etc/modprobe.d/bonding.conf增加内容：
[root@node2 ~]# vi /etc/modprobe.d/bonding.conf
alias bond0 bonding
[root@node2 ~]# modprobe -a bond0
验证：
[root@node2 ~]# lsmod |grep bond
bonding 127331 0
8021q 25317 1 bonding
ipv6 321422 274 bonding,ip6t_REJECT,nf_conntrack_ipv6,nf_defrag_ipv6
编辑网卡配置文件。编辑成例如以下内容：
节点一：
Ifcfg-em2:
DEVICE=em2
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
Ifcfg-em4:
DEVICE=em4
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
Ifcfg-bond0:
DEVICE=bond0
MASTER=yes
BOOTPROTO=node
ONBOOT=yes
BONDING_OPTS="mode=1 miimon=100"
IPADDR=10.10.10.105
PREFIX=24
GATEWAY=10.10.10.1
节点二：
ifcfg-em2:
DEVICE=em2
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
ifcfg-em4:
DEVICE=em4
BOOTPROTO=none
ONBOOT=yes
MASTER=bond0
SLAVE=yes
Ifcfg-bond0:
DEVICE=bond0
MASTER=yes
BOOTPROTO=node
ONBOOT=yes
BONDING_OPTS="mode=1 miimon=100"
IPADDR=10.10.10.106
PREFIX=24
GATEWAY=10.10.10.1
我这里使用的是mode=1的主备网卡模式，平时仅仅激活一块网卡，一旦主网卡发生问题，会切换链路到备网卡，其它也能够考虑4,6两种mode。
改动完了配置文件之后。分别在2个节点启动bond0：ifup bond0。
此时能够看到：
[root@node1 ~]# ifconfig
bond0 Link encap:Ethernet HWaddr C8:1F:66:FB:6F:CB
inet addr:10.10.10.105 Bcast:10.10.10.255 Mask:255.255.255.0
inet6 addr: fe80::ca1f:66ff:fefb:6fcb/64 Scope:Link
UP BROADCAST RUNNING MASTER MULTICAST MTU:1500 Metric:1
RX packets:9844809 errors:0 dropped:0 overruns:0 frame:0
TX packets:7731078 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:0
RX bytes:9097132073 (8.4 GiB) TX bytes:6133004979 (5.7 GiB)
em2 Link encap:Ethernet HWaddr C8:1F:66:FB:6F:CB
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:9792915 errors:0 dropped:0 overruns:0 frame:0
TX packets:7731078 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:9088278883 (8.4 GiB) TX bytes:6133004979 (5.7 GiB)
Interrupt:38
em4 Link encap:Ethernet HWaddr C8:1F:66:FB:6F:CB
UP BROADCAST RUNNING SLAVE MULTICAST MTU:1500 Metric:1
RX packets:51894 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:8853190 (8.4 MiB) TX bytes:0 (0.0 b)
Interrupt:36
网卡的bonding已经配置成功了。
測试验证
此时能够測试分别断掉em2 em4。在一个节点长ping还有一个节点的私有IP。并结合/proc/net/bonding/bond0的信息观察primary
slave的变化，能够发现当down一个网卡时ping不会中断。
Bond0配置好之后，接下来一步就是把配置成RAC的私有网卡。
为了避免配置失败，首先要备份好原来的配置文件。
以grid用户在2个节点对$GRID_HOME/ grid/gpnp/noden/profiles/peer/profile.xml文件运行备份：
cd /u01/app/11.2.0/grid/gpnp/noden/profiles/peer
cp profile.xml profile.xml.bk
[root@node2 peer]# ls
pending.xml profile_orig.xml profile.xml profile.xml.bk、
查看眼下的私有网络配置：
node2-> oifcfg getif
em1 192.168.10.0 global public
em3 10.10.10.0 global cluster_interconnect
先加入新的私有网络，在任一节点运行就可以：
node1-> oifcfg setif -global bond0/10.10.10.0:cluster_interconnect
这一步在运行时可能会报错：
node1-> oifcfg setif -global bond0/10.10.10.0:cluster_interconnect
PRIF-33: Failed to set or delete interface because hosts could not be discovered
CRS-02307: No GPnP services on requested remote hosts.
PRIF-32: Error in checking for profile availability for host node2
CRS-02306: GPnP service on host "node2" not found.
这是由于gpnpd服务异常导致的。
解决方法：能够Kill掉gpnpd进程。GI会自己主动又一次启动gpnpd服务。
在2个节点运行：
[root@node2 ~]# ps -ef| grep gpnp
grid 4927 1 0 Sep22 ? 00:26:38 /u01/app/11.2.0/grid/bin/gpnpd.bin
grid 48568 46762 0 17:26 pts/3 00:00:00 tail -f /u01/app/11.2.0/grid/log/node2/gpnpd/gpnpd.log
root 48648 48623 0 17:26 pts/4 00:00:00 grep gpnp
[root@node2 ~]# kill -9 4927
[root@node2 ~]#
參考gpnpd.log
加入私有网络之后，我们依照例如以下步骤将原来的私有网络删除：
首先停止并disable掉crs。
以root用户在2个节点分别运行下面命令：
停止crs
crsctl stop crs
禁用crs
crsctl disable crs
改动hosts文件，将私有IP地址改为新地址。
2个节点分别运行：
ping node1-priv
ping node2-priv
再启动crs。
[root@node2 ~]# crsctl enable crs
CRS-4622: Oracle High Availability Services autostart is enabled.
[root@node2 ~]# crsctl start crs
删除原来的私有网络：
node2-> oifcfg delif -global em3/10.10.10.0:cluster_interconnect
检查验证，配置成功了。
node2-> oifcfg getif
em1 192.168.10.0 global public
bond0 10.10.10.0 global cluster_interconnect
node2->
以下做一个測试验证下bonding的效果：
ifdown掉em2，此时messages会出现日志信息：
Oct 25 22:00:32 node1 kernel: bonding: bond0: Removing slave em2
Oct 25 22:00:32 node1 kernel: bonding: bond0: Warning: the permanent HWaddr of em2 - c8:1f:66:fb:6f:cb - is still in use by bond0. Set the HWaddr of em2 to a different address to avoid conflicts.
Oct 25 22:00:32 node1 kernel: bonding: bond0: releasing active interface em2
Oct 25 22:00:32 node1 kernel: bonding: bond0: making interface em4 the new active one.
此时bond0自己主动切换到em4上，所以这个时候ping私有IP是没有问题的。
查看/proc/net/bonding/bond0发现active的slave已经变成em4了：
[root@node1 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: em4
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: em4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 3
Permanent HW addr: c8:1f:66:fb:6f:cd
Slave queue ID: 0
[root@node1 ~]#
crsd.log ocssd.log中没有报错信息。css仍然每隔5秒发送一次网络心跳。
2014-10-25 22:00:32.975: [ CSSD][656893696]clssnmSendingThread: sent 5 status msgs to all nodes
2014-10-25 22:00:37.977: [ CSSD][656893696]clssnmSendingThread: sending status msg to all nodes
2014-10-25 22:00:37.977: [ CSSD][656893696]clssnmSendingThread: sent 5 status msgs to all nodes
2014-10-25 22:00:42.978: [ CSSD][656893696]clssnmSendingThread: sending status msg to all nodes
这说明bonding确实做到了保护私有网络的单点故障。
此时再down掉em4：
[root@node1 ~]# ifdown em4
Em4down掉之后，私有IP无法从Node2上ping通了，此时bond0也down掉：
[root@node1 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: None
MII Status: down
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
此时messages信息：
Oct 25 22:02:23 node1 kernel: bonding: bond0: Removing slave em4
Oct 25 22:02:23 node1 kernel: bonding: bond0: releasing active interface em4
ocssd.log日志显示2秒后css已经检測到了私有网络故障：
2014-10-25 22:02:25.573: [GIPCHGEN][1744828160] gipchaInterfaceFail: marking interface failing 0x7f025c00c0a0 { host '', haName 'c617-7010-b72d-6c39', local (nil), ip '10.10.10.105:46469', subnet '10.10.10.0',
mask '255.255.255.0', mac 'c8-1f-66-fb-6f-cb', ifname 'bond0', numRef 1, numFail 0, idxBoot 0, flags 0xd }
2014-10-25 22:02:25.661: [GIPCHGEN][1951459072] gipchaInterfaceFail: marking interface failing 0x7f025c023d90 { host 'node2', haName 'ba2c-9227-ca29-8a21', local 0x7f025c00c0a0, ip '10.10.10.106:32369', subnet
'10.10.10.0', mask '255.255.255.0', mac '', ifname '', numRef 0, numFail 0, idxBoot 2, flags 0x6 }
2014-10-25 22:02:27.663: [GIPCHTHR][1951459072] gipchaWorkerCreateInterface: created remote interface for node 'node2', haName 'ba2c-9227-ca29-8a21', inf 'udp://10.10.10.106:32369'
而且发现本server的私有网络连不上了
2014-10-25 22:02:27.572: [GIPCHDEM][536868608] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0xc99250 [0000000000000010] { gipchaContext : host 'node1', name 'CSS_node-cluster', luid 'e2a491a6-00000000',
numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2014-10-25 22:02:31.012: [ CSSD][656893696]clssnmSendingThread: sending status msg to all nodes
2014-10-25 22:02:31.012: [GIPCHALO][669509376] gipchaLowerProcessNode:
no valid interfaces found to node for 5530 ms, node 0x7f0c18065e40 { host 'node2', haName 'CSS_node-cluster', srcLuid e2a491a6-d8e24a48, dstLuid ebce4a7f-2b4e4348 numInf 0, contigSeq 317864, lastAck 317717, lastValidAck 317864,
sendSeq [317718 : 317729], createTime 4294073210, sentRegister 1, localMonitor 1, flags 0x2408 }
2014-10-25 22:02:31.012: [ CSSD][656893696]clssnmSendingThread: sent 5 status msgs to all nodes
2014-10-25 22:02:33.573: [GIPCHDEM][536868608] gipchaDaemonInfRequest: sent local interfaceRequest, hctx 0xc99250 [0000000000000010] { gipchaContext : host 'node1', name 'CSS_node-cluster', luid 'e2a491a6-00000000',
numNode 1, numInf 0, usrFlags 0x0, flags 0x63 } to gipcd
2014-10-25 22:02:36.013: [ CSSD][656893696]clssnmSendingThread: sending status msg to all nodes
2014-10-25 22:02:36.013: [ CSSD][656893696]clssnmSendingThread: sent 5 status msgs to all nodes
2014-10-25 22:02:37.014: [GIPCHALO][669509376] gipchaLowerProcessNode: no valid interfaces found to node for 11530 ms, node 0x7f0c18065e40 { host 'node2', haName 'CSS_node-cluster', srcLuid e2a491a6-d8e24a48,
dstLuid ebce4a7f-2b4e4348 numInf 0, contigSeq 317864, lastAck 317717, lastValidAck 317864, sendSeq [317718 : 317741], createTime 4294073210, sentRegister 1, localMonitor 1, flags 0x2408 }
显示节点二的网络心跳失败，14.34秒之后会将节点二移出集群：
2014-10-25 22:02:39.010: [ CSSD][658470656]clssnmPollingThread: node node2 (2) at 50% heartbeat fatal, removal in 14.340 seconds
2014-10-25 22:02:39.010: [ CSSD][658470656]clssnmPollingThread: node node2 (2) is impending reconfig, flag 2228230, misstime 15660
网络心跳失败容忍多长时间由css中的Misscount决定，RAC默认配置是30秒：
node2-> crsctl get css misscount
CRS-4678: Successful get misscount 30 for Cluster Synchronization Services.
此时開始报has a disk HB, but no network HB：
2014-10-25 22:02:53.012: [ CSSD][664778496]clssnmvDHBValidateNcopy: node 2, node2, has a disk HB, but no network HB, DHB has rcfg 309527290, wrtcnt, 3048511, LATS 212362014, lastSeqNo 3048510, uniqueness
1414032467, timestamp 1414245773/212369384
2014-10-25 22:02:53.192: [ CSSD][667932416]clssnmvDiskPing: Writing with status 0x3, timestamp 1414245773/212362194
2014-10-25 22:02:53.352: [ CSSD][658470656]clssnmPollingThread: Removal started for node node2 (2), flags 0x220006, state 3, wt4c 0
2014-10-25 22:02:53.352: [ CSSD][658470656]clssnmMarkNodeForRemoval: node 2, node2 marked for removal
2014-10-25 22:02:53.352: [ CSSD][658470656]clssnmDiscHelper: node2, node(2) connection failed, endp (0x6b7), probe(0x7f0c00000000), ninf->endp 0x6b7
2014-10-25 22:02:53.352: [ CSSD][658470656]clssnmDiscHelper: node 2 clean up, endp (0x6b7), init state 5, cur state 5
2014-10-25 22:02:53.352: [GIPCXCPT][658470656] gipcInternalDissociate: obj 0x7f0bf00084c0 [00000000000006b7] { gipcEndpoint : localAddr 'gipcha://node1:c5bc-f486-c390-b48', remoteAddr 'gipcha://node2:nm2_node-cluster/b370-8934-efb4-3f2',
numPend 1, numReady 0, numDone 0, numDead 0, numTransfer 0, objFlags 0x0, pidPeer 0, readyRef (nil), ready 1, wobj 0x7f0bf000a3f0, sendp (nil)flags 0x38606, usrFlags 0x0 } not associated with any container, ret gipcretFail (1)
@
之后node2被踢出：
2014-10-25 22:02:53.353: [ CSSD][655316736]clssnmNeedConfReq: No configuration to change
2014-10-25 22:02:53.353: [ CSSD][655316736]clssnmDoSyncUpdate: Terminating node 2, node2, misstime(30000) state(5)
2014-10-25 22:02:53.353: [ CSSD][655316736]clssnmDoSyncUpdate: Wait for 0 vote ack(s)
2014-10-25 22:02:53.353: [ CSSD][655316736]clssnmCheckDskInfo: Checking disk info...
2014-10-25 22:02:53.353: [ CSSD][655316736]clssnmCheckSplit: Node 2, node2, is alive, DHB (1414245773, 212369384) more than disk timeout of 27000 after the last NHB (1414245743, 212339734)
2014-10-25 22:02:53.353: [ CSSD][655316736]clssnmCheckDskInfo: My cohort: 1
2014-10-25 22:02:53.353: [ CSSD][891287296]clssgmQueueGrockEvent: groupName(crs_version) count(3) master(0) event(2), incarn 3, mbrc 3, to member 2, events 0x0, state 0x0
2014-10-25 22:02:53.353: [ CSSD][655316736]clssnmRemove: Start
2014-10-25 22:02:53.353: [ CSSD][655316736](:CSSNM00007:)clssnmrRemoveNode: Evicting node 2, node2, from the cluster in incarnation 309527290, node birth incarnation 309527289, death incarnation 309527290,
stateflags 0x224000 uniqueness value 1414032467
2014-10-25 22:02:53.353: [ CSSD][891287296]clssgmQueueGrockEvent: groupName(IGSSZGDBsszgdb) count(2) master(1) event(2), incarn 2, mbrc 2, to member 1, events 0x0, state 0x0
2014-10-25 22:02:53.353: [ default][655316736]kgzf_gen_node_reid2: generated reid cid=6d207e372096ef48ff1031c3298552d5,icin=309527289,nmn=2,lnid=309527289,gid=0,gin=0,gmn=0,umemid=0,opid=0,opsn=0,lvl=node
hdr=0xfece0100
Node2被隔离：
2014-10-25 22:02:53.353: [ CSSD][655316736]clssnmrFenceSage: Fenced node node2, number 2, with EXADATA, handle 0
2014-10-25 22:02:53.353: [ CSSD][655316736]clssnmSendShutdown: req to node 2, kill time 212362354
2014-10-25 22:02:53.353: [ CSSD][891287296]clssgmQueueGrockEvent: groupName(CRF-) count(4) master(0) event(2), incarn 4, mbrc 4, to member 2, events 0x38, state 0x0
2014-10-25 22:02:53.353: [ CSSD][655316736]clssnmsendmsg: not connected to node 2
2014-10-25 22:02:53.353: [ CSSD][655316736]clssnmSendShutdown: Send to node 2 failed
2014-10-25 22:02:53.353: [ CSSD][655316736]clssnmWaitOnEvictions: Start
防止node2写数据文件：
2014-10-25 22:02:53.354: [ CSSD][664778496]clssnmvDiskEvict: Kill block write, file /dev/asm_datafile flags 0x00010004, kill block unique 1414032467, stamp 212362354/212362354
crsd.log显示原来执行的Node2上的资源被切换到node1上执行：
2014-10-25 22:03:00.946: [ CRSPE][870311680]{1:61167:7502} CRS-2676: Start of 'ora.node2.vip' on 'node1' succeeded
2014-10-25 22:03:00.946: [ CRSPE][870311680]{1:61167:7502} CRS-2676: Start of 'ora.node2.vip' on 'node1' succeeded
2014-10-25 22:03:02.358: [ CRSPE][870311680]{1:61167:7502} CRS-2676: Start of 'ora.LISTENER_SCAN2.lsnr' on 'node1' succeeded
而且此时Node2上的crs已经无法通信：
node2-> crs_stat -t
CRS-0184: Cannot communicate with the CRS daemon.
将em4 up：
ifup em4
[root@node1 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: None
MII Status: down
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: em4
MII Status: down
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: c8:1f:66:fb:6f:cd
Slave queue ID: 0
查看messages：
Oct 25 22:03:43 node1 kernel: bonding: bond0: setting mode to active-backup (1).
Oct 25 22:03:43 node1 kernel: bonding: bond0: Setting MII monitoring interval to 100.
Oct 25 22:03:43 node1 kernel: bonding: bond0: Adding slave em4.
Oct 25 22:03:44 node1 kernel: bonding: bond0: enslaving em4 as a backup interface with a down link.
Oct 25 22:03:47 node1 kernel: tg3 0000:02:00.1: em4: Link is up at 1000 Mbps, full duplex
Oct 25 22:03:47 node1 kernel: tg3 0000:02:00.1: em4: Flow control is on for TX and on for RX
Oct 25 22:03:47 node1 kernel: tg3 0000:02:00.1: em4: EEE is disabled
手动启动em4之后，em4被自己主动设置为Slave Interface，由于之前bond0已经被down，所以须要手动启动bond0
ifup bond0
[root@node1 ~]# ifup bond0
[root@node1 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: em4
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: em4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: c8:1f:66:fb:6f:cd
Slave queue ID: 0
Slave Interface: em2
MII Status: down
Speed: Unknown
Duplex: Unknown
Link Failure Count: 0
Permanent HW addr: c8:1f:66:fb:6f:cb
Slave queue ID: 0
此时messages显示bond0已经就绪：
Oct 25 22:05:25 node1 kernel: bonding: bond0: Adding slave em2.
Oct 25 22:05:26 node1 kernel: bonding: bond0: enslaving em2 as a backup interface with a down link.
Oct 25 22:05:26 node1 kernel: ADDRCONF(NETDEV_UP): bond0: link is not ready
Oct 25 22:05:26 node1 kernel: 8021q: adding VLAN 0 to HW filter on device bond0
Oct 25 22:05:26 node1 kernel: bond0: link status definitely up for interface em4, 1000 Mbps full duplex.
Oct 25 22:05:26 node1 kernel: bonding: bond0: making interface em4 the new active one.
Oct 25 22:05:26 node1 kernel: bonding: bond0: first active interface up!
Oct 25 22:05:26 node1 kernel: ADDRCONF(NETDEV_CHANGE): bond0: link becomes ready
Ocssd.log也显示能够连接到node2的信息：
2014-10-25 22:05:28.026: [GIPCHGEN][536868608] gipchaNodeAddInterface: adding interface information for inf 0x7f0c14222be0 { host '', haName 'CSS_node-cluster', local (nil), ip '10.10.10.105', subnet '10.10.10.0',
mask '255.255.255.0', mac 'c8-1f-66-fb-6f-cd', ifname 'bond0', numRef 0, numFail 0, idxBoot 0, flags 0x1841 }
2014-10-25 22:05:28.254: [GIPCHTHR][669509376] gipchaWorkerCreateInterface: created local interface for node 'node1', haName 'CSS_node-cluster', inf 'udp://10.10.10.105:60625'
2014-10-25 22:05:28.255: [GIPCHTHR][669509376] gipchaWorkerCreateInterface: created local bootstrap multicast interface for node 'node1', haName 'CSS_node-cluster', inf 'mcast://224.0.0.251:42424/10.10.10.105'
2014-10-25 22:05:30.560: [ CSSD][653739776]clssnmSendConnAck: connected to node 2, node2, con (0xa111d5), state 0
2014-10-25 22:05:30.560: [ CSSD][653739776]clssnmCompleteConnProtocol: node node2, 2, uniqueness 1414245779, msg uniqueness 1414245779, endp 0xa111d5 probendp 0x7f0c00000000 endp 0xa111d5
2014-10-25 22:05:31.465: [ CSSD][653739776]clssnmHandleJoin: node 2 JOINING, state 0->1 ninfendp 0x7f0c00a111d5
2014-10-25 22:05:31.940: [ CSSD][664778496]clssnmvReadDskHeartbeat: Reading DHBs to get the latest info for node(2/node2), LATSvalid(0), nodeInfoDHB uniqueness(1414032467)
2014-10-25 22:05:31.940: [ CSSD][664778496]clssnmvDHBValidateNcopy: Setting LATS valid due to uniqueness change for node(node2) number(2), nodeInfoDHB(1414032467), readInfo(1414245779)
2014-10-25 22:05:31.940: [ CSSD][664778496]clssnmvDHBValidateNcopy: Saving DHB uniqueness for node node2, number 2 latestInfo(1414245779), readInfo(1414245779), nodeInfoDHB(1414032467)
2014-10-25 22:05:32.366: [ CSSD][655316736]clssnmDoSyncUpdate: Initiating sync 309527291
2014-10-25 22:05:32.366: [ CSSD][655316736]clssscCompareSwapEventValue: changed NMReconfigInProgress val 1, from -1, changes 7
2014-10-25 22:05:32.366: [ CSSD][655316736]clssnmDoSyncUpdate: local disk timeout set to 200000 ms, remote disk timeout set to 200000
2014-10-25 22:05:32.366: [ CSSD][655316736]clssnmDoSyncUpdate: new values for local disk timeout and remote disk timeout will take effect when the sync is completed.
2014-10-25 22:05:32.366: [ CSSD][655316736]clssnmDoSyncUpdate: Starting cluster reconfig with incarnation 309527291
2014-10-25 22:05:32.366: [ CSSD][655316736]clssnmSetupAckWait: Ack message type (11)
2014-10-25 22:05:32.366: [ CSSD][655316736]clssnmSetupAckWait: node(1) is ALIVE
2014-10-25 22:05:32.366: [ CSSD][655316736]clssnmSetupAckWait: node(2) is ALIVE
2014-10-25 22:05:32.369: [ CSSD][655316736]clssnmDoSyncUpdate: node(2) is transitioning from joining state to active state
2014-10-25 22:05:32.370: [ CSSD][653739776]clssnmHandleUpdate: NODE 1 (node1) IS ACTIVE MEMBER OF CLUSTER
2014-10-25 22:05:32.370: [ CSSD][653739776]clssnmHandleUpdate: NODE 2 (node2) IS ACTIVE MEMBER OF CLUSTER
2014-10-25 22:05:32.370: [ CSSD][891287296]clssgmSuspendAllGrocks: done
crsd.log显示将节点1上执行的节点二资源停止并在node2上又一次启动：
2014-10-25 22:06:05.097: [ CRSPE][870311680]{2:25913:2} CRS-2677: Stop of 'ora.node2.vip' on 'node1' succeeded
2014-10-25 22:06:05.101: [ CRSPE][870311680]{2:25913:2} CRS-2672: Attempting to start 'ora.node2.vip' on 'node2'
此时资源又又一次飘回node2了：
node2-> crs_stat -t
Name Type Target State Host
------------------------------------------------------------
ora....ER.lsnr ora....er.type ONLINE ONLINE node1
ora....N1.lsnr ora....er.type ONLINE ONLINE node2
ora....N2.lsnr ora....er.type ONLINE ONLINE node1
ora....N3.lsnr ora....er.type ONLINE ONLINE node1
ora.OCR.dg ora....up.type ONLINE ONLINE node1
ora.TEMP.dg ora....up.type ONLINE ONLINE node1
ora.UNDO.dg ora....up.type ONLINE ONLINE node1
ora.asm ora.asm.type ONLINE ONLINE node1
ora.cvu ora.cvu.type ONLINE ONLINE node1
ora.gsd ora.gsd.type OFFLINE OFFLINE
ora....network ora....rk.type ONLINE ONLINE node1
ora....SM1.asm application ONLINE ONLINE node1
ora....E1.lsnr application ONLINE ONLINE node1
ora.node1.gsd application OFFLINE OFFLINE
ora.node1.ons application ONLINE ONLINE node1
ora.node1.vip ora....t1.type ONLINE ONLINE node1
ora....SM2.asm application ONLINE ONLINE node2
ora....E2.lsnr application ONLINE ONLINE node2
ora.node2.gsd application OFFLINE OFFLINE
ora.node2.ons application ONLINE ONLINE node2
ora.node2.vip ora....t1.type ONLINE ONLINE node2
ora.oc4j ora.oc4j.type ONLINE ONLINE node1
ora.ons ora.ons.type ONLINE ONLINE node1
ora.scan1.vip ora....ip.type ONLINE ONLINE node2
ora.scan2.vip ora....ip.type ONLINE ONLINE node1
ora.scan3.vip ora....ip.type ONLINE ONLINE node1
ora.sszgdb.db ora....se.type ONLINE ONLINE node1
我们我们能够将bond0的还有一个slave interface启用起来：
[root@node1 ~]# ifup em2
[root@node1 ~]# cat /proc/net/bonding/bond0
Ethernet Channel Bonding Driver: v3.6.0 (September 26, 2009)
Bonding Mode: fault-tolerance (active-backup)
Primary Slave: None
Currently Active Slave: em4
MII Status: up
MII Polling Interval (ms): 100
Up Delay (ms): 0
Down Delay (ms): 0
Slave Interface: em4
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: c8:1f:66:fb:6f:cd
Slave queue ID: 0
Slave Interface: em2
MII Status: up
Speed: 1000 Mbps
Duplex: full
Link Failure Count: 0
Permanent HW addr: c8:1f:66:fb:6f:cb
Slave queue ID: 0
Messages：
Oct 25 22:05:29 node1 kernel: tg3 0000:01:00.1: em2: Link is up at 1000 Mbps, full duplex
Oct 25 22:05:29 node1 kernel: tg3 0000:01:00.1: em2: Flow control is on for TX and on for RX
Oct 25 22:05:29 node1 kernel: tg3 0000:01:00.1: em2: EEE is disabled
Oct 25 22:05:29 node1 kernel: bond0: link status definitely up for interface em2, 1000 Mbps full duplex.
能够看到em2已经就绪，私有网络又恢复到拥有2个网卡保驾护航的状态。
为RAC私有网络配置网卡Bonding的更多相关文章
- linux网络基础-网卡bonding技术
1.bondingbonding(绑定)是一种linux系统下的网卡绑定技术,可以把服务器上n个物理网卡在系统内部抽象(绑定)成一个逻辑上的网卡,实现本地网卡的冗余,带宽扩容和负载均衡.在应用部署中是 ...
- oracle 11g rac R2 for linux change（public,vip）IP ,hostname (oracle 11g rac R2 修改公有，虚拟，私有IP，网卡）
结构如下: 说明: 节点IP,主机名的修改步骤来自ORACLE support文档: 如何修改集群的公网信息(包括 VIP) (文档 ID 1674442.1) 实验环境情况如下: 实验 节点名称 数 ...
- 网卡bonding模式 - bond0、1、4配置
网卡bonding模式 - bond0.1.4配置 网卡bonding简介 网卡绑定就是把多张物理网卡通过软件虚拟成一个虚拟的网卡,配置完毕后,所有的物理网卡的ip和mac将会变成相同的.多网卡同时工 ...
- XenServer网卡Bonding
在给XenServer配置网卡bonding时,需要在所有节点都添加到集群之后再进行,这也是来自Citrix的建议:"Citrix recommends never joining a ho ...
- OpenStack 网络服务 Neutron 私有网络构建（十九）
本章内容基于之前提供者网络构建的基础上进行改动,之前文章参考如下: Openstack 网络服务 Neutron介绍和控制节点部署 (九) Openstack 网络服务 Neutron计算节点部署(十 ...
- 在VMware中配置网卡之NAT模式
为什么要在VMware中配置网卡? 因为在远程连接服务器时,需要虚拟机连接网络 虚拟机网络配置的三种模式:桥接模式,NAT模式,主机模式 NAT模式也称之为网络转换模式,两层路由: 第一层路由:物理机 ...
- centOS7网络配置（nmcli，bonding，网络组）
关于网络接口命名 CentOS 6之前,网络接口使用连续号码命名: eth0. eth1等,当增加或删除网卡时,名称可能会发生变化.CentOS 7使用基于硬件,设备拓扑和设置类型命名. 网卡命名机制 ...
- linux下网卡bonding配置（转）
linux下网卡bonding配置 章节 bonding技术 centos7配置bonding centos6配置bonding 一.bonding技术 bonding(绑定)是一种linux系统 ...
- linux下网卡bonding配置
linux下网卡bonding配置 章节 bonding技术 centos7配置bonding centos6配置bonding 一.bonding技术 bonding(绑定)是一种linux系统 ...
随机推荐
- Vue之数据绑定
<!DOCTYPE html> <html lang="en"> <head> <meta charset="UTF-8&quo ...
- CF666E Forensic Examination SAM+倍增，线段树和并
题面: 给你一个串S以及一个字符串数组T[1..m],q次询问,每次问S的子串S[p_l..p_r]在T[l..r]中的哪个串里的出现次数最多,并输出出现次数.如有多解输出最靠前的那一个. 分析: 第 ...
- mac rar文件解压缩
在下载文件时经常遇到RAR格式的压缩文件, 之前从APP Store下载了免费的解压软件, 但是总觉着不好用, 广告信息很多. 好用的软件都要花钱, 所以找到了命令行解决的办法. 步骤如下: 首先需要 ...
- vue 单独页面定时器 离开页面销毁定时器
data: { return { timer: null } }, created() { this.timer = setInterval(....); }, beforeDestroy() { i ...
- python str.format 中文对齐的细节问题
写了一个练手的爬虫...在输出的时候出现了让人很不愉♂悦的问题 像这样: 令人十分难受啊! #----------------------------------------------------- ...
- 零、常用的Mysql数据库操作语句大全
零.用户管理: 1.新建用户: >CREATE USER name IDENTIFIED BY 'ssapdrow'; 2.更改密码: >SET PASSWORD FOR name=PAS ...
- selenium启动chrome出错处理：Message: 'chromedriver' executable needs to be in PATH
- HDU-2647 Reward ，逆拓排。
Reward 发工资,以前看过这题,做没做忘了(应该是没做). 很明显的拓排.但数据范围这么大,吓得我当时就不敢动手.后来找题解发现还是相当于两层循环(are you kidding me?)当时卡在 ...
- 【转】关于大型网站技术演进的思考（二十）--网站静态化处理—web前端优化—中（12）
Web前端很多优化原则都是从如何提升网络通讯效率的角度提出的,但是这些原则使用的时候还是有很多陷阱在里面,如果我们不能深入理解这些优化原则背后所隐藏的技术原理,很有可能掉进这些陷阱里,最终没有达到最佳 ...
- 将登录等信息保存到session中和退出session
JShop简介:jshop是一套使用Java语言开发的B2C网店系统,致力于为个人和中小企业提供免费.好用的网店系统. 项目主页:http://git.oschina.net/dinguangx/js ...