Multipath
========
The connection from the server through Host Bus Adapter (HBA) to storage controller is referred as a path. When multiple paths exists to a storage device(LUN) on a storage subsystem, it is referred as multipath connectivity. It is a enterprise level storage capability. Main purpose of multipath connectivity are;
* To provide redundant access to the storage devices, i.e to have access to the storage device when one or more of the components in a path fail.
* The increased throughput by way of load balancing.
A simple example of multipath could be: 2 HBAs connected to a switch to which the storage controllers are connected. In this case the storage controller can be accessed from either of the HBAs and hence we have multipath connectivity.
In Linux, a SCSI device is configured for a LUN seen on each path. i.e, if a LUN has 4 paths, then one will see four SCSI devices getting configured for the same storage LUN. Doing I/O to a LUN in a such an environment is unmanageable,
* applications/administrators do not know which SCSI device to use
* all applications consistently using the same device even on path failure
* always using the storage device specific preferred path
* spreading I/O between multiple valid paths
Device mapper Multipathing
=====================
Device mapper multipathing (DM-Multipath) allows you to configure multiple I/O paths between server nodes and storage arrays into a single device. Multipathing aggregates the I/O paths, creating a new device that consists of the aggregated paths.
Storage devices with 2 controller can be configured,
Active/Active: Means that both the controllers can process I/Os.
Active/Passive: Means that one of the controllers(active) can process I/Os, and the other one(passive) is in a standby mode. I/Os to the passive controller will fail.
Steps to configure DM-Multipath
=======================
First we need to check whether all the hardware connections were performed successfully,i.e connection from host to SAN switch through Fiber Channel HBA and connection from SAN switch to storage. Steps listed here are for Active/Passive configured SAN Storage.
1. Check the following rpm for multipath drivers installled.
#rpm –qa | grep device-mapperdevice-mapper-1.02.28-2.el5device-mapper-event-1.02.28-2.el5device-mapper-multipath-0.4.7-23.el5device-mapper-1.02.28-2.el5If the above packages are not installed then install it using “rpm -ivh”. All the above packages we will get it from Enterprise Linux OS install CD. Also you can download from rpm.pbone.net
2. Check devices list in /proc/partitions
# cat /proc/partitionsmajor minor #blocks name8 0 291991552 sda8 1 200781 sda18 2 51199155 sda28 3 8193150 sda38 4 1 sda48 5 232396258 sda58 16 104857600 sdb8 32 104857600 sdc4. Get the unique scsi id (WWID) for the available devices
# scsi_id -g -u -s /block/sda3600605b00293c32014c002630a328feb# scsi_id -g -u -s /block/sdb3600605b00293e20014bf1ca41c462c45# scsi_id -g -u -s /block/sdc3600605b00293e20014bf1ca41c462c45#Device sdb and sdc are having same SCSI id which means they both points to same LUN in SAN storage.
Note: when obtaining unique scsi id we have use /block/sdb not with /dev/sdb
5. Check available devices listed in OS using fdisk
# fdisk -lDisk /dev/sda: 298.9 GB, 298999349248 bytes255 heads, 63 sectors/track, 36351 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System/dev/sda1 * 1 25 200781 83 Linux/dev/sda2 26 6399 51199155 83 Linux/dev/sda3 6400 7419 8193150 82 Linux swap / Solaris/dev/sda4 7420 36351 232396290 5 Extended/dev/sda5 7420 36351 232396258+ 83 LinuxDisk /dev/sdb: 107.3 GB, 107374182400 bytes255 heads, 63 sectors/track, 13054 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDisk /dev/sdb doesn’t contain a valid partition tableIgnore primary device /dev/sda, it is a local hard disk which has OS and other applications.
Note that device /dev/sdb is listed in fdisk but not /dev/sdc thats because I/O to the passive device here /dev/sdc fails.
5. Check for the multipath modules are loaded by issuing the ‘lsmod’ command
#lsmod | grep dm[\-_]rounddm_round_robin 36801 1dm_multipath 52433 2 dm_round_robin#If they are not loaded, load them using ‘modprobe’ command:
# modprobe dm-multipath# modprobe dm-round-robinps#6. Update /etc/multipath.conf
* Set user_friendly_names=yes, which provides user friendly name to the device other wise DM will name the device with WWID as default like /dev/mapper/3600605b00293e20014bf1ca41c462c45
* Blacklist is the list of devices which will not allow DM to configure multipath for example hard disks. To include only specified devices here SAN LUNs for the multipathing, blacklist all the devices and in blacklist_exceptions add the device required for multipathing.
blacklist { wwid “*” }blacklist_exceptions { wwid 3600605b00293e20014bf1ca41c462c45 }* Include different devices in multipathing group and provide alias name. If alias name is not provided DM will use the default user friendly naming convention /dev/mapper/mpathn where n is number from 0. In this example we have provided alias name as “oracle”.
multipaths { multipath { wwid 3600605b00293e20014bf1ca41c462c45 #<— for sdb and sdc alias oracle
} }7. Set chkconfig to start the mutipath daemon automatically on reboot.
# chkconfig –list multipathdmultipathd 0:off 1:off 2:off 3:off 4:off 5:off 6:off#If the service is disable as above, enable it:
# chkconfig multipathd on# chkconfig –list multipathdmultipathd 0:off 1:off 2:on 3:on 4:on 5:on 6:off#8. Start the multipath daemon service:
# service multipathd startStarting multipathd daemon: [ OK ]#9. Check the multipath configurations,
# multipath -llsdc: checker msg is “readsector0 checker reports path is down”oracle (3600605b00293e20014bf1ca41c462c45) dm-0 SUN,SUN_6180[size=100G][features=0][hwhandler=0][rw]\_ round-robin 0 [prio=1][active]\_ 7:0:1:0 sdb 8:48 [active][ready]\_ round-robin 0 [prio=0][enabled]\_ 7:0:0:0 sdc 8:32 [active][ghost]#Note that sdb is active/ready and sdc is active/ghost. status active/ghots points that it is passive connection.
10. Verify DM-Multipathing device in fdisk, in fdisk the DM devices will be listed like /dev/dm-n where n is number from 0.
# fdisk –lDisk /dev/sda: 298.9 GB, 298999349248 bytes255 heads, 63 sectors/track, 36351 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDevice Boot Start End Blocks Id System/dev/sda1 * 1 25 200781 83 Linux/dev/sda2 26 6399 51199155 83 Linux/dev/sda3 6400 7419 8193150 82 Linux swap / Solaris/dev/sda4 7420 36351 232396290 5 Extended/dev/sda5 7420 36351 232396258+ 83 LinuxDisk /dev/sdb: 107.3 GB, 107374182400 bytes255 heads, 63 sectors/track, 13054 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDisk /dev/sdb doesn’t contain a valid partition tableDisk /dev/dm-0: 107.3 GB, 107374182400 bytes255 heads, 63 sectors/track, 13054 cylindersUnits = cylinders of 16065 * 512 = 8225280 bytesDisk /dev/dm-0 doesn’t contain a valid partition table11. Check user friendly named devices for DM in /dev/
# ls -lrt /dev/mapper/*crw——- 1 root root 10, 62 Feb 5 01:46 /dev/mapper/controlbrw-rw—- 1 root disk 253, 0 Feb 5 01:46 /dev/mapper/oracle12. Use /dev/mapper/
for any file system activities like partition using fdisk, creating file system and to mount the device. Also make sure in /etc/fstab this user friendly named device is used for auto mount.
Example:
1. To partition,
#fdisk /dev/mapper/oracle
3. To create file syste, After creating partition the device name for the partition will be /dev/mapper/pn where n is number from 1,
#mkfs -t ext3 /dev/mapper/oraclep5
2. To mount the device
#mount -t ext3 /dev/mapper/oraclep5 /data
References
=========
1. Red Hat Enterprise Linux 6 DM Multipath - DM Multipath Configuration and Administration
2. http://sourceware.org/lvm2/wiki/MultipathUsageGuide#active_passive
3. Configuration and Use of Device Mapper Multipathing on Oracle Enterprise Linux (OEL) [ID 555603.1]