Storage Management in Linux

Storage Management in Linux
Liju Gopinath - RHCA, RHCSS, RHCDS, RHCVA, RHCSA, RHCE.

Subject Matter Expert - Linux
NAS Vs. SAN
NAS Vs SAN
NAS provides file level access
NAS appears to the client OS as a file server (the client can map network
drives to shares on that server)
SAN provides block level access
Whereas a disk available through a SAN still appears to the client OS as a disk,
visible in disk and volume management utilities (along with client's local
disks), and available to be formatted with a file system and mounted.

[ Presented by Liju Gopinath ]



iSCSI
iSCSI
iSCSI is Internet SCSI (Small Computer System Interface), an Internet
Protocol (IP)-based storage networking standard for linking data
storage facilities by carrying SCSI commands over IP networks.
We can mount block devices (disks) across an IP network to local
system and then use them like any other block device. iSCSI is a
client-server protocol.
By default iSCSI will listen to the port 3260

iSCSI Target & Initiator

The server-side is referred to as the 'target'
The client-side is referred to as the 'initiator'

Configuring iSCSI server

Install and enable the iSCSI target services.
#
#
#
#
yum -y install scsi-target-utils

service tgtd start
service tgtd stop
chkconfig tgtd on

10
/etc/tgt/targets.conf
<target iqn.2014-01.com.example:vmrh5.target1>
backing-store /dev/VG00/disk1
</target>

11
Configuring iSCSI client

Install iSCSI client package
# yum y install iscsi-initiator-utils
Start iSCSI services in the client

#
#
#
#
service iscsi start

service iscsid start
chkconfig iscsi on
chkconfig iscsid on

12
Discover the target LUNs

Discover the target LUNs exported by server using following
command.
It will provide iqn name with of LUN associated with given ip
address
# iscsiadm -m discovery -t sendtargets -p 192.168.0.9

13
iSCSI Discovery Configuration

The default iSCSI configuration file is /etc/iscsi/iscsid.conf.
This file contains iSCSI settings used by iscsid and iscsiadm.
Node records in /var/lib/iscsi/nodes:
When logging into a target, iscsiadm uses the settings in this file.
Discovery records in /var/lib/iscsi/send_targets:
When performing discovery to same destination, iscsiadm uses the
settings in this file.

14
Connecting to iSCSI target

To connect iSCSI target, we need to mention iscsi servers ip address
and iqn name.
# iscsiadm -m node -T iqn.2014-01.com.example:vmrh5.target1 p
192.168.0.9 l
To disconnect the iSCSI Target, you will need to log out:

# iscsiadm -m node -T iqn.2014-01.com.example:vmrh5.target1 -p
192.168.0.9 -u

15
Create File System and Mount iSCSI LUN

Create filesystem
# mkfs.ext3 /dev/sdb
/etc/fstab:
/dev/sdb /iscsi_test ext3 _netdev 0 0
Use _netdev parameter while adding entry in/etc/fstab, which is

used to prevent the system from attempting to mount the file
systems until the network, has been enabled on the system.

16
Logical Volume Manager
What is LVM?
Logical volume management provides a higher-level view of the
disk storage on a computer system than the traditional view of disks
and partitions.
This gives the system administrator much more flexibility in
allocating storage to applications and users.
Storage volumes created under the control of the logical volume
manager can be resized and moved around almost at will, although
this may need some upgrading of file system tools.
The LVM is structured in three elements:
Volumes: physical volume, logical volumes and volume groups

Extents: physical and logical extents
Device mapper: the Linux kernel module

18
Is your system ready for LVM2?

Verify whether LVM2 package is installed.
# lsmod | grep dm_mod
dm_mod
81500 11
dm_mirror,dm_log
Load the Device-mapper module if not loaded already.
# modprobe dm_mod

19

20

21
Physical Extents (PE ) and Logical Extents (LE)

22
Each PV consists of a number of fixed-size physical extents (PEs);

similarly, each LV consists of a number of fixed-size logical extents
(LEs).
LEs and PEs are always the same size, the default in LVM 2 is 4 MB.
An LV is created by mapping logical extents to physical extents, so
that references to logical block numbers are resolved to physical
block numbers.
When the logical volume is created a mapping is defined between
logical extents and physical extents,
e.g. logical extent 1 could map onto physical extent 51 of PV01,
data written to the first 4 MB of the logical volume in fact be
written to the 51st extent of PV01.

23
Logical Extend and Physical Extend mapping.

Through these different types of logical-to-physical mappings, LVM
can achieve four important advantages over raw physical partitions:
Logical volumes can be resized while they are mounted and accessible by the
database or file system, removing the downtime associated with adding or
deleting storage from a Linux server.
Data from one (potentially faulty or damaged) physical device may be
relocated to another device that is newer, faster or more resilient, while the
original volume remains online and accessible.
Logical volumes can be constructed by aggregating physical devices to
increase performance (via disk striping) or redundancy (via disk mirroring and
I/O multipathing).
Logical volume snapshots can be created to represent the exact state of the
volume at a certain point-in-time, allowing accurate backups to proceed
simultaneously with regular system operation.

24
Mapping modes (linear/striped)

Linear mapping will assign a range of PE's to an area of an LV in
order e.g.,
LE 1 - 99 map to PV01 and LE 100 - 347 map onto PV02.
Striped mapping will interleave the chunks of the logical extents
across a number of physical volumes eg.,
1st chunk of LE[1] -> PV01[1],
2nd chunk of LE[1] -> PV02[1],
3rd chunk of LE[1] -> PV03[1],
4th chunk of LE[1] -> PV01[2],

25
LVM Metadata
The configuration details of a volume group are referred to as the
metadata.
By default, an identical copy of the metadata is maintained in every
metadata area in every physical volume within the volume group.
LVM volume group metadata is stored as ASCII
By default, the pvcreate command places the physical volume label
in the 2nd 512-byte sector.
This label can optionally be placed in any of the first four sectors,
since the LVM tools that scan for a physical volume label check the
first 4 sectors.
The volume group metadata contains:
Information about how and when it was created

Information about the volume group:
26
Metadata backups and archives are automatically created on every

volume group and logical volume configuration change unless
disabled in the lvm.conf file.
By default, the metadata backup is stored in the /etc/lvm/backup
directory and the metadata archives are stored in the
/etc/lvm/archive directory.
You can manually back up the metadata to the /etc/lvm/backup file
with the vgcfgbackup command.
You can restore metadata with the vgcfgrestore command.

27
LVM snapshot
Snapshot creates exact copy of a logical volume, frozen at some
point in time.
Useful when taking backup.
LVM1 has read-only snapshots.
In LVM2, snapshots are read/write by default.
The snapshot is really nothing more than asking the LVM to give
you a "pointer" to the current state of the filesystem and to write
changes made after the snapshot to a designated area.
LVM snapshots are your basic "copy on write" snapshot solution.
# lvcreate --size 100M --snapshot --name snapshot /dev/VG01/lvol1

28
copy on write
As soon as you create a snapshot, LVM creates a pool of blocks.
This pool also contains a full copy of the LVM metadata of the
volume.
When writes happen to the main volume such as updating an
inode, the block being overwritten is copied to this new pool and
the new block is written to the main volume. This is the 'copy-onwrite'.
Because of this, the more data that gets changed between when a
snapshot was taken and the current state of the main volume, the
more space will get consumed by that snapshot pool.

29
device-mapper
The Device Mapper is a kernel driver that provides a framework for
volume management.
It provides a generic way of creating mapped devices, which may be
used as logical volumes.
It does not specifically know about volume groups or metadata
formats.
In addition to LVM, Device-Mapper multipath and the dmraid
command use the Device Mapper.
The user interface is the dmsetup command.
LVM logical volumes are activated using the Device Mapper.
Each logical volume is translated into a mapped device.
The Device Mapper supports a variety of mapping targets, including
linear mapping, striped mapping, and error mapping.
30
device-mapper files
If VG0 and LV0 are the volume group and logical volume, the
Device mapper would create the following device nodes in the
/dev filesystem:
/dev/mapper/VG0-LV0
/dev/VG0/LV0 (is a link to the above)
/dev/mapper/VG0-LV1
/dev/VG0/LV1 (is a link to the above)

31
The LVM Configuration Files

Central configuration file read by the tools.
/etc/lvm/lvm.conf
Device name filter cache file (configurable).
/etc/lvm/.cache
Directory for automatic volume group metadata backups

(configurable).
/etc/lvm/backup/
Directory for automatic volume group metadata archives

(configurable).
/etc/lvm/archive/
In single-host configuration, lock files to prevent parallel tool runs

from corrupting the metadata; in a cluster, cluster-wide DLM is
used.
/var/lock/lvm/

32
LVM Locking
Non clustered LVM
# grep locking_type /etc/lvm/lvm.conf
locking_type = 1
For CLVM
# grep locking_type /etc/lvm/lvm.conf
locking_type = 3

33
LVM commands
pvcreate
vgcreate
lvcreate
pvdisplay
vgdisplay
lvdisplay
pvscan
vgscan
lvscan
vgextend
lvextend

34
Physical Volume
# pvcreate /dev/sda
/dev/sdb
OR if you are using multipath
# pvcreate /dev/mapper/mpatha /dev/mapper/mpathb

# pvscan
# pvdisplay

35
Volume Group
# vgcreate VG01 /dev/sda /dev/sdb
# vgscan
# vgdisplay
# pvcreate /dev/sdc
# vgextend VG01 /dev/sdc

36
Logical Volume
#
#
#
#
#
#
lvcreate -l 10G -n LOGVOL01 VG01

lvscan
lvdisplay
mkfs.ext3 /dev/VG01/LOGVOL01
lvextend -L100 /dev/VG01/LOGVOL01
lvextend -L+100 /dev/VG01/LOGVOL01

37
pvdisplay
# pvdisplay
--- Physical volume --PV Name
/dev/sda2
VG Name
VG00
PV Size
14.75 GiB / not usable 5.00 MiB
Allocatable
yes (but full)
PE Size
32.00 MiB
Total PE
472
Free PE
0
Allocated PE
472
PV UUID
4sGBhJ-rtRF-2x6A-OTdA-DFS8-0zP9-eZfk2c

38
vgdisplay
# vgdisplay vg01
--- Volume group --VG Name
System ID
Format
Metadata Areas
Metadata Sequence No
VG Access
VG Status
MAX LV
Cur LV
Open LV
Max PV
Cur PV
Act PV
VG Size
PE Size
Total PE
Alloc PE / Size
Free PE / Size
VG UUID
vg01
lvm2
2
1
read/write
resizable
0
0
0
0
2
2
19.99 GiB
4.00 MiB
5118
0 / 0
5118 / 19.99 GiB
e13j0O-b3VG-C4MS-CwZW-lDbw-weQt-Q7G02R

39
lvdisplay
# lvdisplay |more
--- Logical volume --LV Name
/dev/vg02/lvol1
VG Name
vg02
LV UUID
jDmh8L-0pjd-fxbj-yypH-cMRJ-G7x5-Xcdj2Y
LV Write Access
read/write
LV Status
available
# open
0
LV Size
4.00 GiB
Current LE
1024
Segments
1
Allocation
inherit
Read ahead sectors
auto
- currently set to
256
Block device
253:4

40
PVS, VGS, LVS

[root@vmrh5cn01 ~]#
PV
VG
/dev/sda
/dev/sdb
/dev/hda2 VG00
pvs
Fmt
lvm2
lvm2
lvm2
Attr
aaa-
PSize PFree
10.00G 10.00G
10.00G 10.00G
14.88G
0
[root@vmrh5cn01 ~]# vgs

VG
#PV #LV #SN Attr
VSize VFree
VG00
0 wz--n- 14.88G
1
3
0
VG01
2
0
0 wz--n- 19.99G 19.99G
[root@vmrh5cn01 ~]# lvs

LV
VG
Attr
LSize Origin Snap%
home VG00 -wi-ao 2.00G
root VG00 -wi-ao 10.88G
swap VG00 -wi-ao 2.00G
lvol1 VG01 -wi-a- 1.00G
Move Log Copy%

Convert
41
dmsetup
The dmsetup command is a command line wrapper for
communication with the Device Mapper.
For general system information about LVM devices, you may find
the info, ls, status, and deps options of the dmsetup command to
be useful, as shown below.

42
Device Information
[root@vmrh5 ~]# dmsetup info /dev/mapper/VG01-lvol1
Name: VG01-lvol1
State: ACTIVE
Read Ahead: 256
Tables present: LIVE
Open count: 1
Event number: 0
Major, minor: 252, 8
Number of targets: 1
UUID: LVMAbvIw8a9BeN19rTvSIfkT9wpPu7bnhqH78OyjJPRdUOLJwx0dQLwpSeSIVcVmAad

43
[root@vmrh5 ~]# dmsetup ls

mpath2 (252, 5)
mpath1 (252, 4)
VG00-home (252, 1)
mpath0 (252, 3)
VG00-swap (252, 2)
VG00-root (252, 0)
VG01-lvol1 (252, 8)
[root@vmrh5 ~]# dmsetup ls --target linear
VG00-home (252, 1)
VG00-swap (252, 2)
VG00-root (252, 0)
VG01-lvol1 (252, 8)

44
Fibre Channel
Fibre Channel
FC is more likely to be used as a SCSI network. The simplest use of
Fibre Channel is to replace external SCSI-attached storage units on a
single server.
This involves one HBA, some FC-storage units and perhaps an FC
hub for making the cabling simpler.

46

47

48

49

50

51
WWID
The World Wide Identifier (WWID) can be used in reliably
identifying devices.
It is a persistent, system-independent ID that the SCSI Standard
requires from all SCSI devices.
The WWID identifier is guaranteed to be unique for every storage
device, and independent of the path that is used to access the
device.
WWNN (World Wide Node Number)
WWPN (World Wide Port Number)

52
H:B:T:L Values
/sys/class/scsi_host/host/device/targetH:B:T:L/H:B:T:L/
Where:
H is the host number of the HBA

B is the bus number (normally 0)
T is the target number
L is the LUN number

53
Scanning LUN
Scanning new LUNs without reboot.
# echo "- - -" > /sys/class/scsi_host/hostX/scan
# echo "1" > /sys/class/fc_host/hostX/issue_lip

54
device-mapper-multipathing
device-mapper-multipathing
The connection from the server through the HBA to the storage
controller is referred as a path.
When multiple paths exist to a storage device (LUN) on a storage
subsystem, it is referred as multipath connectivity.
Main purpose of multipath connectivity is to provide redundant
access to the storage devices.
Another advantage of multipathing is the increased throughput by
way of load balancing.

56
DM-MP consists of 4 components

DM MP kernel module - Kernel module that is responsible for
making the multipathing decisions in normal and failure situations.
multipath command - User space tool that allows the user with
initial configuration, listing and deletion of multipathed devices.
multipathd daemon - User space daemon that constantly monitors
the paths. It marks a path as failed when it finds the path faulty and
if all the paths in a priority group are faulty then it switches to the
next enable priority group. It keeps checking the failed path, once
the failed path comes alive, based on the failback policy, it can
activate the path. It automatically creates device mapper entries
when a new device comes into existence.
kpartx - User space command that creates device mapper entries
for all the partitions in a multipathed disk/LUN. When the multipath
command is invoked, this command automatically gets invoked. For
DOS based partitions this command need to be run manually.
57
Active/Passive Multipath Configuration

58
Active / Active Multipath Configuration

59

60

61
/etc/multipath.conf
[root@vmrh5 ~]# egrep -v '^#|^$' /etc/multipath.conf
defaults {
find_multipaths yes
user_friendly_names yes
path_grouping_policy
multibus
path_selector
"service-time 0"
}
blacklist {
}

62
# multipath -ll
mpath0 (1IET
00010001) dm-4 IET,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 4:0:0:1 sdb 8:16 active ready running
`- 5:0:0:1 sdf 8:80 active ready running
mpath1 (1IET
00010002) dm-3 IET,VIRTUAL-DISK
size=10G features='0' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
|- 4:0:0:2 sdc 8:32 active ready running
`- 5:0:0:2 sdg 8:96 active ready running
Here we have two multipath devices, mpath0, and mpath1.

The device mpath1 includes sdc & sdg.
These two sd devices, each correspond to a path.
As shown above the system has two paths to the iSCSI storage so two sd
devices are listed.
The IO policy is sent to round-robin which means that IO is sent
down each of the two paths in a round-robin fashion.

63
Standard output of multipath command

[root@vmrh5 ~]# multipath -ll
mpath2 (1IET_00010003) dm-5 IET,VIRTUAL-DISK
[size=10G][features=0][hwhandler=0][rw]
\_ round-robin 0 [prio=2][enabled]
\_ 1:0:0:3 sdc 8:32 [active][ready]
\_ 2:0:0:3 sdg 8:96 [active][ready]
\_ 1:0:0:2 sdb 8:16 [active][ready]
\_ 2:0:0:2 sdf 8:80 [active][ready]
\_ 1:0:0:1 sda 8:0
[active][ready]
\_ 2:0:0:1 sde 8:64 [active][ready]
\_ 1:0:0:4 sdd 8:48 [active][ready]
\_ 2:0:0:4 sdh 8:112 [active][ready]

64
mpatha (1IET
00010002) dm-3 IET, VIRTUAL-DISK
------ ----------------- ---- --- ------------|
|
|
|
|------->
|
|
|
|------------------>
|
|
|----------------------->
|
|--------------------------------->
|------------------------------------------------->
Product
Vendor
sysfs name
WWID of the device
User defined Alias name
size=10G
features=0
hwhandler=0
--------------------------------|
|
|----> Hardware Handler, if any
|
|-----------------------------> Features supported
|-----------------------------------------------> Size of the DM device
Path Group:
|-+- round-robin 0 prio=1
active
-------------- ----------|
|
|
|--------------------->
|
|
|------------------------------>
|
|---------------------------------------------->
|--------------------------------------------------->
Path
Path
Path
Path
group state
group priority
selector and repeat count
group level
First path on Path Group 1:

`- 4:0:0:2 sdd 8:32
active ready
-------- --- --------- ----|
|
|
|
|-----------------> Physical Path state
|
|
|
|------------------------> DM Path state
|
|
|--------------------------------> Major, minor numbers
|
|-------------------------------------> Linux device name
|--------------------------> SCSI information: host, channel, scsi_id and LUN

65
Finding scsi id of a disk

[root@vmrh5 ~]# /lib/udev/scsi_id --whitelisted --device=/dev/sdc
1IET
00010002
[root@vmrh5 ~]# /lib/udev/scsi_id --whitelisted --device=/dev/sdg
1IET
00010002

66
Finding HBA details

# systool -c scsi_host -v | egrep -i fw_version | driver_version
| model_name | model_desc
# lspci | grep -i fibre
# ll /sys/class/scsi_host
# ll /sys/class/fc_host/
# cat /proc/scsi/qla**
# systool -m qla2xxx v
# systool -c fc_host v
# cat /proc/scsi/scsi
67
# modinfo qla2xxx
# modinfo lpfc
# lspci | grep -i emulex
# cat /sys/class/scsi_host/host0/fwrev
# cat /sys/class/scsi_host/host0/node_name
# cat /sys/class/scsi_host/host0/port_name
# cat /sys/class/scsi_host/host0/lpfc_drvr_version
# cat /sys/class/scsi_host/host0/serialnum
# cat /sys/class/scsi_host/host0/speed
# cat /sys/class/scsi_host/host0/state
68
Reference
http://sources.redhat.com/lvm2/wiki/MultipathUsageGuide
http://www.tldp.org/HOWTO/LVM-HOWTO/index.html
http://www.ibm.com/developerworks/library/l-lvm2/l-lvm2-pdf.pdf
http://access.redhat.com/knowledge/docs/Red_Hat_Enterprise_Linux/
http://www.redhat.com/magazine/009jul05/features/lvm2/
http://www.tldp.org/HOWTO/LVM-HOWTO/
http://www.sourceware.org/lvm2/

69

Storage Management in Linux

Încărcat de

Informații document

Descriere originală:

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Storage Management in Linux

Încărcat de

Drepturi de autor:

Formate disponibile

Storage Management in Linux

Liju Gopinath - RHCA, RHCSS, RHCDS, RHCVA, RHCSA, RHCE.

NAS Vs. SAN

SAN provides block level access