CentOS配置multipath

可以通过2种方式查看HBA的WWN信息：

1. 查看sys文件系统

查看HBA卡型号：
[root@localhost ~]# lspci  | grep -i fibre
13:00.0 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
13:00.1 Fibre Channel: QLogic Corp. ISP2532-based 8Gb Fibre Channel to PCI Express HBA (rev 02)
查看HBA卡WWN信息：
[root@localhost ~]# ls -l /sys/class/fc_host/
total
drwxr-xr-x  root root  Jun   : host3
drwxr-xr-x  root root  Jun   : host4
[root@localhost ~]# cat /sys/class/fc_host/host3/port_name
0x21000024ff462814

2. 采用systool

systool所在软件包：

yum -y install sysfsutils

systool -c fc_host -v
-c <class_name>  /  -b <bus_name> /  -m <module_name>
[root@localhost ~]# systool -v -c fc_host
Class = "fc_host"
  Class Device = "host3"
  Class Device path = "/sys/class/fc_host/host3"
    fabric_name         = "0x1000000533746b8f"
    issue_lip           = <store method only>
    node_name           = "0x20000024ff462814"
    port_id             = "0x011300"
    port_name           = "0x21000024ff462814"
    port_state          = "Online"
    port_type           = "NPort (fabric via point-to-point)"
    speed               = "8 Gbit"
    supported_classes   = "Class 3"
    supported_speeds    = "1 Gbit, 2 Gbit, 4 Gbit, 8 Gbit"
    symbolic_name       = "QLE2562 FW:v5.03.16 DVR:v8.03.07.03.05.07-k"
    system_hostname     = ""
    tgtid_bind_type     = "wwpn (World Wide Port Name)"
    uevent              = <store method only>
    Device = "host3"
    Device path = "/sys/devices/pci0000:00/0000:00:07.0/0000:13:00.0/host3"
      ct                  =
      edc                 = <store method only>
      els                 =
      fw_dump             =
      nvram               = "ISP "
      optrom_ctl          = <store method only>
      optrom              =
      reset               = <store method only>
      sfp                 = ""
      uevent              = <store method only>
      vpd                 = "‚("

  Class Device = "host4"
  Class Device path = "/sys/class/fc_host/host4"
    fabric_name         = "0x10000005337638b9"
    issue_lip           = <store method only>
    node_name           = "0x20000024ff462815"
    port_id             = "0x021300"
    port_name           = "0x21000024ff462815"
    port_state          = "Online"
    port_type           = "NPort (fabric via point-to-point)"
    speed               = "8 Gbit"
    supported_classes   = "Class 3"
    supported_speeds    = "1 Gbit, 2 Gbit, 4 Gbit, 8 Gbit"
    symbolic_name       = "QLE2562 FW:v5.03.16 DVR:v8.03.07.03.05.07-k"
    system_hostname     = ""
    tgtid_bind_type     = "wwpn (World Wide Port Name)"
    uevent              = <store method only>
    Device = "host4"
    Device path = "/sys/devices/pci0000:00/0000:00:07.0/0000:13:00.1/host4"
      ct                  =
      edc                 = <store method only>
      els                 =
      fw_dump             =
      nvram               = "ISP "
      optrom_ctl          = <store method only>
      optrom              =
      reset               = <store method only>
      sfp                 = ""
      uevent              = <store method only>
      vpd                 = "‚("

Multipath相关软件介绍

1. device-mapper-multipath

　　即multipath-tools，主要提供multipathd和multipath等工具和 multipath.conf等配置文件。这些工具通过device mapper的ioctr的接口创建和配置multipath，设备创建的多路径设备映射会在/dev /mapper中。

2. device-mapper

　　主要包括两大部分：内核部分和用户部分。内核部分主要由device mapper核心（dm.ko）和一些target driver（md-multipath.ko）。核心完成设备的映射，而target根据映射关系和自身特点具体处理从mappered device 下来的i/o。同时，在核心部分，提供了一个接口，用户通过ioctr可和内核部分通信，以指导内核驱动的行为，比如如何创建mappered device，这些divece的属性等。linux device mapper的用户空间部分主要包括device-mapper这个包。其中包括dmsetup工具和一些帮助创建和配置mappered device的库。这些库主要抽象，封装了与ioctr通信的接口，以便方便创建和配置mappered device。multipath-tool的程序中就需要调用这些库

3. dm-multipath.ko和dm.ko：dm.ko是device mapper驱动。它是实现multipath的基础。dm-multipath其实是dm的一个target驱动。

https://www.cnblogs.com/jianqiang2010/archive/2011/09/01/2162574.html

AIX

http://blog.163.com/herod_xiao/blog/static/871883992011819112227689/

1. 检查multipath模块，如果没有相关模块就说明没有安装相关软件包

lsmod | grep dm_multipath

yum -y install device-mapper device-mapper-multipath

2. 加载相关模块

modprobe dm-multipath

modprobe dm-round-robin

3. 生成multipath配置文件，并配置

/sbin/mpathconf --enable

配置multipath.conf文件

4. 启动multipath服务

service multipathd start

chkcofig multipathd on

[root@tyhr ~]# multipath -ll
mpathc (36005076801820709e000000000000087) dm- IBM,
size=300G features='1 queue_if_no_path' hwhandler='' wp=rw
`-+- policy='round-robin 0' prio= status=active
  |- ::: sdc :  active ready running
  |- ::: sde :  active ready running
  |- ::: sdg :  active ready running
  `- ::: sdi : active ready running
mpathb (36005076801820709e000000000000086) dm- IBM,
size=100G features='1 queue_if_no_path' hwhandler='' wp=rw
`-+- policy='round-robin 0' prio= status=active
  |- ::: sdb :  active ready running
  |- ::: sdd :  active ready running
  |- ::: sdf :  active ready running
  `- ::: sdh : active ready running

如果完成配置后，不能使用multipath -ll查看到任何多路径信息，就需要手工重新扫描HBA信息

一般新增LUN映射给主机后，系统无法直接更新挂载的存储盘，需要重新扫描。

# cd /sys/class/scsi_host/host4/
# echo '- - - ' > scan
或
echo "" > /sys/class/fc_host/host/issue_lip  //某些存储或系统没有scan文件，可以通issue_lip文件识别

RHEL5.x 操作系统，新增multipath后，能够识别磁盘，但是通过fdisk分区后，不能显示分区信息（/dev/mapper/dm-app 磁盘不显示/dev/mapper/dm-appp1）。partprob无报错但是没有效果。fdisk w保存后有报错：

WARNING: Re-reading the partition table failed with error 22: Invalid argument.

Run "kpartx -a" after FDISK is completed to add all partition mappings on the
newly-created multipath device

[root@TYEDI1 host4]# ls -l /dev/mapper/dm-*
brw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app
brw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch
brw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db

[root@TYEDI1 host4]# kpartx -a /dev/mapper/dm-app 

[root@TYEDI1 host4]# ls -l /dev/mapper/dm-*
brw-rw---- 1 root disk 253, 0 Jan 21 22:59 /dev/mapper/dm-app
brw-rw---- 1 root disk 253, 3 Jan 21 23:01 /dev/mapper/dm-app1
brw-rw---- 1 root disk 253, 2 Jan 21 22:35 /dev/mapper/dm-arch
brw-rw---- 1 root disk 253, 1 Jan 21 22:52 /dev/mapper/dm-db

　　

其他配置：

1. udev绑定权限

RHEL6.5，对于multipath的权限，手工去修改几秒后会变回root。所以需要使用udev去绑定好权限。

1)搜索对应的配置文件模板：
[root@jyrac1 ~]# find / -name -*
/usr/share/doc/device-mapper-1.02./-dm-permissions.rules\
2)根据模板新增12-dm-permissions.rules文件在/etc/udev/rules.d/下面：
vi /etc/udev/rules.d/-dm-permissions.rules
# MULTIPATH DEVICES
#
# Set permissions for all multipath devices
ENV{DM_UUID}=="mpath-?*", OWNER:="grid", GROUP:="asmadmin", MODE:=""

# Set permissions for first two partitions created on a multipath device (and detected by kpartx)
# ENV{DM_UUID}=="part[1-2]-mpath-?*", OWNER:="root", GROUP:="root", MODE:=""
3)查看多路径对应的底层dm设备：
[root@jyrac2 rules.d]# ls -lh /dev/dm*
brw-rw----  root disk ,  Jul  : /dev/dm-
brw-rw----  root disk ,  Jul  : /dev/dm-
brw-rw----  root disk ,  Jul  : /dev/dm-
brw-rw----  root disk ,  Jul  : /dev/dm-

4)启动start_udev
[root@jyrac2 rules.d]# start_udev
Starting udev: [  OK  ]

5)查看权限：
[root@jyrac2 rules.d]# ls -lh /dev/dm*
brw-rw----  root disk     ,  Jul  : /dev/dm-
brw-rw----  root disk     ,  Jul  : /dev/dm-
brw-rw----  grid asmadmin ,  Jul  : /dev/dm-
brw-rw----  grid asmadmin ,  Jul  : /dev/dm-

6)等30s后再查，权限固定：
[root@jyrac2 rules.d]# ls -lh /dev/dm*
brw-rw----  root disk     ,  Jul  : /dev/dm-
brw-rw----  root disk     ,  Jul  : /dev/dm-
brw-rw----  grid asmadmin ,  Jul  : /dev/dm-
brw-rw----  grid asmadmin ,  Jul  : /dev/dm-

可以使用lsscsi查看识别的磁盘信息：

[root@tyhr ~]#  lsscsi
[:::]    cd/dvd  TSSTcorp DVD-ROM TS-L333H ID03  /dev/sr0
[:::]    cd/dvd  IMM      Virtual CD/DVD     /dev/sr1
[:::]    disk    IBM-ESXS MBF2300RC        SB27  -
[:::]    disk    IBM-ESXS MBF2300RC        SB27  -
[:::]    disk    LSILOGIC Logical Volume     /dev/sda
[:::]    disk    IBM                     /dev/sdb
[:::]    disk    IBM                     /dev/sdc
[:::]    disk    IBM                     /dev/sdd
[:::]    disk    IBM                     /dev/sde
[:::]    disk    IBM                     /dev/sdf
[:::]    disk    IBM                     /dev/sdg
[:::]    disk    IBM                     /dev/sdh
[:::]    disk    IBM                     /dev/sdi 

1. multipath常用操作：

multipath -r （修改multipath.conf配置文件之后重新加载）
multipath -ll （查看多路径状态）
multipath -v2 （格式化路径，检测路径，合并路径）
multipath -v3 （查看多路径详情blacklist、whitelist和设备wwid）
multipath -F （删除现有链路）

2. multipath文件详细介绍

接下来的工作就是要编辑/etc/multipath.conf的配置文件

multipath.conf主要包括blacklist、multipaths、devices三部份的配置

blacklist配置

blacklist {

　　devnode "^sda"

}

Multipaths部分配置multipaths和devices两部份的配置。

multipaths {

　　multipath {

　　　　wwid **************** 　　　　　　#此值multipath -v3可以看到

　　　　alias iscsi-dm0 　　　　　　　　#映射后的别名,可以随便取

　　　　path_grouping_policy multibus 　　#路径组策略

　　　　path_checker tur 　　　　　　　　#决定路径状态的方法

　　　　path_selector "round-robin 0" 　　#选择那条路径进行下一个IO操作的方法

　　}

}

Devices部分配置

devices {

　　device {

　　　　vendor "iSCSI-Enterprise" 　　　　　　　　　　#厂商名称

　　　　product "Virtual disk" 　　　　　　　　　　　　#产品型号

　　　　path_grouping_policy multibus 　　　　　　　　#默认的路径组策略

　　　　getuid_callout "/sbin/scsi_id -g -u -s /block/%n"　#获得唯一设备号使用的默认程序

　　　　prio_callout      "/sbin/acs_prio_alua %d" 　　　  #获取有限级数值使用的默认程序

　　　　path_checker readsector0 　　　　　　　　　　#决定路径状态的方法

　　　　path_selector "round-robin 0" 　　#选择那条路径进行下一个IO操作的方法

　　　　failback        immediate 　　　　#故障恢复的模式

   　　　no_path_retry      queue 　　　　#在disable queue之前系统尝试使用失效路径的次数的数值

  　　　　rr_min_io       100 　　　　　　#在当前的用户组中，在切换到另外一条路径之前的IO请求的数目

　　}

}

相关参数的标准文档的介绍：

Attribute	Description
wwid	Specifies the WWID of the multipath device to which the multipath attributes apply. This parameter is mandatory for this section of themultipath.conf file.
alias	Specifies the symbolic name for the multipath device to which themultipath attributes apply. If you are using user_friendly_names, do not set this value tompathn; this may conflict with an automatically assigned user friendly name and give you incorrect device node names.
path_grouping_policy	Specifies the default path grouping policy to apply to unspecified multipaths. Possible values include: failover = 1 path per priority group multibus = all valid paths in 1 priority group group_by_serial = 1 priority group per detected serial number group_by_prio = 1 priority group per path priority value group_by_node_name = 1 priority group per target node name
path_selector	Specifies the default algorithm to use in determining what path to use for the next I/O operation. Possible values include: round-robin 0: Loop through every path in the path group, sending the same amount of I/O to each. queue-length 0: Send the next bunch of I/O down the path with the least number of outstanding I/O requests. service-time 0: Send the next bunch of I/O down the path with the shortest estimated service time, which is determined by dividing the total size of the outstanding I/O to each path by its relative throughput.
failback	Manages path group failback. A value of immediate specifies immediate failback to the highest priority path group that contains active paths. A value of manual specifies that there should not be immediate failback but that failback can happen only with operator intervention. A value of followover specifies that automatic failback should be performed when the first path of a path group becomes active. This keeps a node from automatically failing back when another node requested the failover. A numeric value greater than zero specifies deferred failback, expressed in seconds.
prio	Specifies the default function to call to obtain a path priority value. For example, the ALUA bits in SPC-3 provide an exploitableprio value. Possible values include: const: Set a priority of 1 to all paths. emc: Generate the path priority for EMC arrays. alua: Generate the path priority based on the SCSI-3 ALUA settings. tpg_pref: Generate the path priority based on the SCSI-3 ALUA settings, using the preferred port bit. ontap: Generate the path priority for NetApp arrays. rdac: Generate the path priority for LSI/Engenio RDAC controller. hp_sw: Generate the path priority for Compaq/HP controller in active/standby mode. hds: Generate the path priority for Hitachi HDS Modular storage arrays.
no_path_retry	A numeric value for this attribute specifies the number of times the system should attempt to use a failed path before disabling queueing. A value of fail indicates immediate failure, without queueing. A value of queue indicates that queueing should not stop until the path is fixed.
rr_min_io	Specifies the number of I/O requests to route to a path before switching to the next path in the current path group. This setting is only for systems running kernels older that 2.6.31. Newer systems should userr_min_io_rq. The default value is 1000.
rr_min_io_rq	Specifies the number of I/O requests to route to a path before switching to the next path in the current path group, using request-based device-mapper-multipath. This setting should be used on systems running current kernels. On systems running kernels older than 2.6.31, use rr_min_io. The default value is 1.
rr_weight	If set to priorities, then instead of sending rr_min_io requests to a path before callingpath_selector to choose the next path, the number of requests to send is determined byrr_min_io times the path's priority, as determined by the prio function. If set touniform, all path weights are equal.
flush_on_last_del	If set to yes, then multipath will disable queueing when the last path to a device has been deleted.

multipath与oracle ASM结合

https://www.cnblogs.com/jyzhao/p/7208620.html

已有配置文件：

defaults {
    polling_interval
    failback            immediate
    no_path_retry
    rr_min_io
    path_checker        tur
    user_friendly_names yes
}
devnode_blacklist {
        devnode "sda"
}
devices {
    device {
        vendor                   "IBM"
        product                  ""
        path_grouping_policy     group_by_prio
        prio_callout             "/sbin/mpath_prio_alua /dev/%n"
    }
    device {
        vendor                   "IBM"
        product                  ""
        path_grouping_policy     group_by_prio
        prio_callout             "/sbin/mpath_prio_alua /dev/%n"
    }
    device {
        vendor                   "IBM"
        product                  ""
        path_grouping_policy     group_by_serial
    }
    device {
        vendor                   "IBM"
        product                  ""
        path_grouping_policy     group_by_serial
    }
}
multipaths {
        multipath {
                wwid    36005076801820709e000000000000043
                alias   dm-vote01
        }
}