Drbd双机环境安装配置

一、环境准备

1) 操作系统：ubuntu-14.04.1 x64

2) Ubuntu1 192.168.5.179 /dev/sdb1 主节点

Ubuntu2 192.168.5.178 /dev/sdb1 备节点

3) 在两台机器各加一块新的硬盘：

1) ll /dev/sd*

2) ll /dev/sdb*

3) brw-rw---- 1 root disk 8, 16 Jun 24 13:37 /dev/sdb

4) brw-rw---- 1 root disk 8, 17 Jun 24 13:37 /dev/sdb1

注：，对新加的磁盘分区，sdb1分区未格式化

二、安装DRBD软件

1) 安装drbd软件：

（drbd共有两部分组成：内核模块和用户空间的管理工具。其中drbd内核模块代码已经整合进Linux内核2.6.33以后的版本中，因此内核版本高于此版本的话，只需要安装管理工具drbd-utils即可）

1. root@controller1:/# apt-get install -y --force-yes drbd8-utils

2. Reading package lists... Done

3. Building dependency tree

4. ....

2) 查看drbd状态

1. root@controller1:/# service drbd status

2. drbd driver loaded OK; device status:

3. version: 8.4.3 (api:1/proto:86-101)

4. srcversion: F97798065516C94BE0F27DC

5. m:res  cs  ro  ds  p  mounted  fstype

• drbdadm –version命令查看drbd8-mod 和 drbd8-utils 的deb包版本（可能存在版本问题）
• lsmod | grep drbd命令查看drbd模块加载情况
• modprobe drbd命令重新加载drbd
• drbd属于两节点，需要在另外一节点安装一样的软件

（注：如果出现drbd没有正常加载的情况，重启机器试试，或者更新系统软件试试。又或者需要编译系统内核安装）

三、配置DRBD资源

1. root@controller1:/# cd /etc/drbd.d/

2. root@controller1:/etc/drbd.d# ls

3. global_common.conf  mysql.res

• 进入drbd配置目录，可以看到global_common.conf配置文件，上述mysql.res为笔者后来创建文件
• 创建Drbd设备资源配置文件，创建并且编辑文件

root@controller1:/etc/drbd.d# touch mysql.res
root@controller1:/etc/drbd.d# vim mysql.res

资源配置格式，如下文实际文件

1. resource mysql {

2.     device minor 0;

3.     disk "/dev/sdb5";

4.     meta-disk internal;

5.     on controller1 {

6.         address ipv4 192.168.2.11:7700;

7.     }

8.     on controller2 {

9.         address ipv4 192.168.2.12:7700;

10.     }

11. }

编辑格式：

1. resource 资源名 {

2.     device minor 0;    设备号

3.     disk "/dev/sdb5";  底层实际使用的设备，两个节点可以有差异

4.     meta-disk internal;

5. 节点信息，controller1，controller2为两节点hostname

6.     on controller1 {

7.         address ipv4 192.168.2.11:7700;

8.     }

9.     on controller2 {

10.         address ipv4 192.168.2.12:7700;

11.     }

12. }

四、启动设备资源

1) 当两个节点的配置文件都创建完毕后

1. root@controller1:/etc/drbd.d# drbdadm create-md mysql

2. You want me to create a v08 style flexible-size internal meta data block.

3. There appears to be a v08 flexible-size internal meta data block

4. already in place on /dev/sdb5 at byte offset 5368705024

5. Do you really want to overwrite the existing v08 meta-data?

6. [need to type 'yes' to confirm] yes

7.

8. Writing meta data...

9. md_offset 5368705024

10. al_offset 5368672256

11. bm_offset 5368508416

12.

13. Found xfs filesystem

14.      5242684 kB data area apparently used

15.      5242684 kB left usable by current configuration

16.

17. Even though it looks like this would place the new meta data into

18. unused space, you still need to confirm, as this is only a guess.

19.

20. Do you want to proceed?

21. [need to type 'yes' to confirm] yes

22.

23. initializing activity log

24. NOT initializing bitmap

25. New drbd meta data block successfully created.

• 注：使用drbdadm create-md创建DRBD元数据的时候需要使用-c指定配置文件，否则会提示“no resources defined!”。

root-shell> drbdadm –c /etc/drbd.conf create-md all

• Command 'drbdmeta /dev/drbd1 v08 /dev/sda2 internal create-md' terminated with exit code 40

drbdadm aborting

提示出错

解决办法：使用 dd 指令将一些资料塞到 /dev/sdb 后再执行 drbdadm create-md ha 指令即可顺利执行dd if=/de v/zero of=/dev/sdb1 bs=1M count=100

• 错误信息:
  执行指令 drbdadm create-md ha 时出现如下错误信息：
  open(/dev/sdb1) failed: No such file or directory
  Command 'drbdmeta 0 v08 /dev/hdb1 internal create-md' terminated with exit coolpre 20
  drbdadm create-md ha: exited with coolpre 20
  解决方法:
  因为忘了执行 fdisk /dev/sdb 指令建立分割区所造成，使用fdisk命令将 /dev/sdb 建立分割区后指令即可正常执行（无需格式化分区）

2) 启动设备

1. root@controller1:/etc/drbd.d# drbdadm up mysql

2. root@controller1:/etc/drbd.d# drbd-overview

3.   0:mysql/0  WFConnection Secondary/Unknown Inconsistent/DUnknown C r----s

3) 强制为主节点，只需要在一个节点上操作

1. root@controller1:~# drbdadm -- --force primary mysql

2. root@controller1:~# drbd-overview

3.   0:mysql/0  WFConnection Primary/Unknown UpToDate/Outdated C r----s

4) drbd-overview 可以查看drbd设备状态，另外drbd设备符

1. root@controller1:/dev/drbd# ls

2. by-disk  by-res

3. root@controller1:/dev/drbd# ls -al by-disk/

4. total 0

5. drwxr-xr-x 2 root root 60 Jul 29 13:20 .

6. drwxr-xr-x 4 root root 80 Jul 29 13:20 ..

7. lrwxrwxrwx 1 root root 11 Jul 29 13:22 sdb5 -> ../../drbd0

8. root@controller1:/dev/drbd# ls -al by-res/

9. total 0

10. drwxr-xr-x 3 root root 60 Jul 29 13:20 .

11. drwxr-xr-x 4 root root 80 Jul 29 13:20 ..

12. drwxr-xr-x 2 root root 60 Jul 29 13:20 mysql

13. root@controller1:/dev/drbd# ls -al by-res/mysql/

14. total 0

15. drwxr-xr-x 2 root root 60 Jul 29 13:20 .

16. drwxr-xr-x 3 root root 60 Jul 29 13:20 ..

17. lrwxrwxrwx 1 root root 14 Jul 29 13:22 0 -> ../../../drbd0

5) 将/dev/drbd0格式化并挂载

在主节点192.168.8.5上执行下面的命令（只能在主节点执行格式化挂载）

mkfs.ext3 /dev/drbd0
mkdir /mnt/drbd0
mount /dev/drbd0 /mnt/drbd0

Ø mount提示错误”mount: block device /dev/drbd0 is write-protected, mounting read-only mount: Wrong medium type“,是因为mount只能在Primary一端使用。

Ø 在执行“mkfs.ext3 /dev/drbd0”的时候，可能提示错误“mkfs.ext3: Wrong medium type while trying to determine filesystem size”。同上

6) 此时另外一个节点处于outdated状态，只需要在另外节点创建并启用，两者即会同步

1. root@controller2:~# drbdadm create-md mysql

2. You want me to create a v08 style flexible-size internal meta data block.

3. There appears to be a v08 flexible-size internal meta data block

4. already in place on /dev/sda5 at byte offset 5368705024

5. Do you really want to overwrite the existing v08 meta-data?

6. [need to type 'yes' to confirm] yes

7.

8. .......

9. New drbd meta data block successfully created.

10. root@controller2:~# drbd-overview

11.   0:mysql/0  Unconfigured . . . .

12. root@controller2:~# drbdadm up mysql

13. root@controller2:~# drbd-overview

14.   0:mysql/0  SyncTarget Secondary/Primary Inconsistent/UpToDate C r-----

15.         [>....................] sync'ed:  0.7% (5088/5116)Mfinish: 0:02:42 speed: 31,744 (31,744) want: 30,800 K/sec

7) Drbd脑裂行为解决（非必须）

如上我们看到节点2上数据处于standalone状态，即发生了脑裂。【因为该环境之前本身就有数据】

此时可以通过，使得次节点数据不可靠，让其主动同主节点同步

1. root@controller2:~# drbdadm secondary mysql

2. root@controller2:~# drbdadm  -- --discard-my-data connect mysql

3. root@controller1:/# drbdadm connect mysql

如果此种情况节点2，仍然是StandAlone，则可以完全重建节点2。

1. root@controller2:~# drbdadm detach mysql

2. root@controller2:~# drbdadm down mysql

3. root@controller2:~# drbdadm create-md mysql

4. You want me to create a v08 style flexible-size internal meta data block.

5. There appears to be a v08 flexible-size internal meta data block

6. already in place on /dev/sda5 at byte offset 5368705024

7. Do you really want to overwrite the existing v08 meta-data?

8. [need to type 'yes' to confirm]

9. ........

10. root@controller2:~# drbdadm up mysql

11. root@controller2:~# drbd-overview

12.   0:mysql/0  SyncTarget Secondary/Primary Inconsistent/UpToDate C r-----

13.         [>....................] sync'ed:  1.5% (5044/5116)Mfinish: 0:01:07 speed: 75,776 (75,776) want: 91,600 K/sec

经过一段时间同步后，同步完成。

1. root@controller1:/# drbd-overview

2.   0:mysql/0  SyncSource Primary/Secondary UpToDate/Inconsistent C r----- /mnt xfs 5.0G 112M 4.9G 3%

3.         [================>...] sync'ed: 86.2% (708/5116)Mfinish: 0:00:13 speed: 54,484 (54,420) K/sec

两个节点同步时，一个状态为SyncSource一个状态为SyncTarget

1. root@controller1:/# drbd-overview

2.   0:mysql/0  Connected Primary/Secondary UpToDate/UpToDate C r----- /mnt xfs 5.0G 112M 4.9G 3%

同步完成状态为Connected

8) 测试同步

在主节点192.168.8.5上执行下面的命令

cd /xserver-storage

echo "a file created in server5" > testfile

在备份节点192.168.8.6上执行下面的命令

mkdir /xserver-storage

mount /dev/drbd0 /xserver-storage

#mount 会出错，因为mount只能在Primary一端使用

在主节点192.168.8.5上执行下面的命令变成备份节点

umount /xserver-storage

drbdadm secondary all

在备份节点192.168.8.6上执行下面的命令变为主节点，可以看到192.168.8.5同步过来的内容

drbdadm primary all

mount /dev/drbd0 /xserver-storage

less /xserver-storage/testfile

可以查看到文件的内容为"a file created in server5"

在192.168.8.6新建一个文件

echo "a file created in server6" > testfile2

再将192.168.8.5变为主节点后mount可以看到testfile2的内容也同步了。

五、配置BRDB常见问题

问题1、’ha’ ignored, since this host (node2.centos.bz) is not mentioned with an ‘on’ keyword.？
错误信息:
执行指令 drbdadm create-md ha 时出现如下错误信息：
'ha' ignored, since this host (node2.centos.bz) is not mentioned with an 'on' keyword.
解决方法:
因为在 drbd 设定 drbd.conf 中 on 本来写的是 node1、node2 而以，将node1和node2分别改为node1.centos.bz,node2.centos.bz。
问题2、drbdadm create-md ha: exited with coolpre 20？
错误信息:
执行指令 drbdadm create-md ha 时出现如下错误信息：
open(/dev/hdb1) failed: No such file or directory
Command 'drbdmeta 0 v08 /dev/hdb1 internal create-md' terminated with exit coolpre 20
drbdadm create-md ha: exited with coolpre 20
解决方法:
因为忘了执行 fdisk /dev/hdb 指令建立分割区所造成，如下将 /dev/hdb 建立分割区后指令即可正常执行

# fdisk /dev/hdb //准备为 hdb 建立分割区

The number of cylinders for this disk is set to 20805.

There is nothing wrong with that, but this is larger than 1024,

and could in certain setups cause problems with:

1) software that runs at boot time (e.g., old versions of LILO)

2) booting and partitioning software from other OSs

(e.g., DOS FDISK, OS/2 FDISK)

Command (m for help): n //键入 n 表示要建立分割区

Command action

e extended

p primary partition (1-4)

p //键入 p 表示建立主要分割区

Partition number (1-4): 1 //键入 1 为此主要分割区代号

First cylinder (1-20805, default 1): //开始磁柱值，按下 enter 即可

Using default value 1

Last cylinder or +size or +sizeM or +sizeK (1-20805, default 20805): //结束磁柱值，按下 enter 即可

Using default value 20805

Command (m for help): w //键入 w 表示确定执行刚才设定

The partition table has been altered!

Calling ioctl() to re-read partition table.

Syncing disks.

[root@node1 yum.repos.d]# partprobe //使刚才的 partition table 变更生效

问题3、drbdadm create-md ha: exited with coolpre 40？
错误信息:
执行指令 drbdadm create-md ha 时出现如下错误信息:

Device size would be truncated, which

would corrupt data and result in

'access beyond end of device' errors.

You need to either

* use external meta data (recommended)

* shrink that filesystem first

* zero out the device (destroy the filesystem)

Operation refused.

Command 'drbdmeta 0 v08 /dev/hdb1 internal create-md' terminated with exit coolpre 40

drbdadm create-md ha: exited with coolpre 40

解决方法:
使用 dd 指令将一些资料塞到 /dev/hdb 后再执行 drbdadm create-md ha 指令即可顺利执行
# dd if=/dev/zero of=/dev/hdb1 bs=1M count=100
问题4、DRBD 状态始终是 Secondary/Unknown？
错误信息:
Node1、Node2 主机启动 DRBD 后状态始终是 Secondary/Unknown

#service drbd status

drbd driver loaded OK; device status:

version: 8.3.8 (api:88/proto:86-94)

GIT-hash: d78846e52224fd00562f7c225bcc25b2d422321d build by mockbuild@builder10.centos.org, 2010-06-04 08:04:16

m:res cs ro ds p mounted fstype

0:ha WFConnection Secondary/Unknown Inconsistent/DUnknown C

解决方法：
1、Node1、Node2 没有打开相对应的 Port，请开启相对应的 Port 或先把 IPTables 服务关闭即可。
2、可能发生了脑裂行为，一般出现在ha切换时，解决方法：
在一节点执行:
drbdadm secondary resource
drbdadm connect –discard-my-data resource
另一节点执行：
drbdadm connect resource
问题5、1: Failure: (104) Can not open backing device
错误信息：
执行drbdadm up r0时出现：
1: Failure: (104) Can not open backing device.
Command 'drbdsetup attach 1 /dev/sdb1 /dev/sdb1 internal' terminated with exit pre 10
解决方法：
可能因为你挂载了/dev/sdb1,执行umount /dev/sdb1即可。