LVM and VxVM

an Introduction

By Bill Hassell
With acknowledgements to
David Totsch and
Chris Wong

1
LVM - Introduction

LVM Basics
HFS and VxFS filesystem

2
 LVM - Disk Layout
LIF Header LVM Record

Physical Volume Bad Block

Volume Group Reserve Area Directory
Descriptor Area
Volume Group
Boot Data Duplicate
Information
Status Area Reserve Area
Mirror Volume Group
Consistency
Records
Reserve Area
Redundant LIF Volume
Information
Bad Block Pool
Physical
Extents
3
LVM Allocation
Linear Pvol 1

Lvol 3

1 4

2 5

3 6

5 Pvol 2

6 1

7 2

8 3

6 4
LVM Allocation
Mirroring Pvol 1

Lvol 3

1 4

2 5

3 6

5 Pvol 1

6 1

6
5
 LVM Allocation
Striping Pvol 1
1
2
3
4
Lvol
5
6
1

Pvol 2
2
1
2
3
3
4
4
5
6
5

Pvol 3
6
1
2
3
4
5
6

6
 LVM Allocation
"Chunking"
Pvol 1 Pvol 4
1 1
2 2
Lvol
3 3
4 4
1
5 5
6 6
2

Pvol 2 Pvol 5
3
1 1
2 2
4
3 3
4 4
5
5 5
6 6
6

Pvol 3 Pvol 6
1 1
2 2
3 3
4 4
5 5
6 6
7
 LVM Allocation
fragmentation
before interim after
Pvol 1 Pvol 1 Pvol 1
lvol 1 lvol 1 lvol 1
lvol 1 lvol 1 lvol 1

lvol 2 lvol 4-e

lvol 2 lvol 4-f

lvol 2 lvol 4-g

lvol 3 lvol 3 lvol 3
lvol 3 lvol 3 lvol 3
lvol 3 lvol 3 lvol 3

lvol 4 lvol 4 lvol 4-a

lvol 4 lvol 4 lvol 4-b
lvol 4 lvol 4 lvol 4-c
lvol 4 lvol 4 lvol 4-d

8
 LVM Performance
• Mirroring
– recovery policy
– cache
• Extent Size
– Is bigger really better?
• Number of Disks

9
 LVM Convenience
• Mirrors
– Splitting
– Merging
• vgexport/vgimport
• pvmove (!)

10
 LVM Don'ts
• Span disks w/o mirroring
• Leave /etc/lvmtab alone
• vgscan is a last resort
• backup w/o running vgcfgbackup

11
 File System Types
• UFS a.k.a. HFS
• VXFS Veritas Journaled File System
• CDFS CD-ROM File System
• NFS Network File System

12
 Boot Block

DATA

DATA
SuperBlock
Boot Block
Boot Block INODES
SuperBlock
SuperBlock
INODES
INODES
DATA
HFS Layout On Disk

DATA

DATA

13
 Fragmentation
O 3k
X 9k
Z 18k
O O O X Z Z
Z Z Z Z Z Z Z Z
X X X X X X X X
Z Z Z Z Z Z Z Z 14
 Inode
File Size
File Type • Direct - 96KB File
Mode
UID
• Single - 16.8MB
GID • Double - 34GB
Link Count
Time Stamps • Triple - 70TB
12 Direct

Single Indirect • 59 Char Fast SymLink

Double Indirect
Triple Indirect
15
 Directory
Directory name inode
. 78 this inode
.. 22 parent directory
.profile 128
resume 101
junk 0 "Empty Entry"
bin 88 Directory?
myStuff 269

16
 How HFS Works
SB

• Files stay in cylinder groups, where

possible.
• Flat performance until > 80% full.
• Most accommodating to many small files.

17
 JFS (a.k.a. VxFS)
• Based on version from Veritas Software,
Inc.
• Journaling
• Quick recovery after failure
• Extent based allocation
• Mount options balance
performance/integrity

18
 VXFS Layout

OLT

OLT
8Kb Super
Intent Log
Block

Extent
Map
Allocation Unit

AU
Data/Inode Extents
Header

Data/Inode Extents

19
 File System Integrity
• Use proper shutdown procedures!
• What fsck(1M) does
– hfs
– vxfs
• bcheckrc

20
 Mounting File Systems
• Methodology for adding a new file system
• MountPoint management
• Mounting for security
– nosuid (!)
– ro (?)
• What about the device files?

21
 How JFS Allocation Works

• Files allocated into extents.

• Extents are contiguous blocks.
• Extents not related to LVM extents.
• File expansion by growing extent,
• If not, allocate new extent

22
 JFS Extents
Extent Attributes
• Adjacent disk blocks treated as a unit
• Represented by starting block and count
• Objective: Instead of Many Small I/Os, Perform Fewer Large I/Os

File Growth
• Objective is to keep file contiguous on disk
• When file grows, JFS attempts to increase size of last extent
• Otherwise, double size of current extent and relocate data
• Otherwise, allocate a new extent

Note: JFS Extents are unrelated to LVM Extents

23
 Extent Performance
Extent Read Ahead
• Fetch more current extent than immediately
needed
• Does NOT involve posting extra I/O
operations

Extent Write Behinds

• Writes go to buffer cache to be flushed later
• When flushed, look for other I/O request to
merge

24
 File System
Consistency Check (fsck)

HFS JFS

• Full Integrity • Back-out unfinished

Check transactions
• Visits all blocks • Must explicitly
request full check

25
 Journaled File System
Base Features
JFS Features Beyond HFS

•Faster file system recovery

•Panic avoidance in JFS code
•Improved integrity and reliability
•Better performance due to using extents
•HP’s strategic file system direction

26
 Creating JFS Manually
•lvcreate -L 100 -n lvol2 /dev/vg04
•mkdir /test
•newfs -F vxfs /dev/vg04/rlvol2
•mount /dev/vg04/lvol2 /test

Add to /etc/fstab file:

•/dev/vg04/lvol2 /test vxfs defaults 0 2

27
 Base JFS Mount Options
• Logging
– Full Logging
– Delayed Logging
– Temporary logging
• Cache
– closesync
• Others
– Block Clear
– No Data In Log

28
 Resizing Base JFS
File Systems
• Extending
– lvextend(1m)
– unmount file system
– extendfs(1m)
• Reducing
– backup
– lvreduce(1m)
– newfs
– restore

29
 Online JFS Features
On Mounted File Systems, you can:
• Defragment (reorganize)
– Remove unused spaces from directories
– Make small files contiguous
– Consolidate free blocks
• Resize file system while mounted
• Take a snapshot
• Modify extent attributes

30
 Reporting Fragmentation
• fsadm -D directory
• fsadm -E extents

31
•
Defragmentation
-d directory
• -e extent
• -v verbose
• -p number of passes
• -s summarize each pass
• -t time limit
• -a how old is old
• -l how big is big

32
 Defragmenation Guidelines
• As often as necessary
• Defragment directories first
– or both at the same time

33
 Extending A File System
Extend the logical volume
•lvextend -L 440 /dev/vg00/lvol8

Extend the JFS filesystem

•fsadm -F vxfs -b 450560 /var

The sector count = newLVsize * 1024

34
 Reducing A File System
Backup the filesystem
•tar cvf /dev/rmt/0m /test
Perform/view directory & extent reorganization
•fsadm -F vxfs -d -D -e -E /test
Resize the JFS filesystem
fsadm -F vxfs -b <newsize> /test
Reduce the size of the logical volume
•lvreduce -L 1000 /dev/vg04/lvol2

CAUTION: fsadm(1m) may fail to reduce!

35
 The HP-UX Journal File System

The HP-UX Journal File System (JFS) is an extent-based

journaling file system that offers:

•Fast file system recovery •Online disk defragmentation

•Online administration •Online reorganization
•Online backup •Enhanced application interface
•Online resizing •Enhanced mount options

36
 Network File Systems
• Client
– biod
• Server
– /etc/exports
– nfsd
• Performance
– nfsstat
– number of daemons

37
 NFS Misc.
• Mount options
– hard / soft
– ro
– never allow remote root access!
• nfs vs. ftp/rcp
• automount(1M)

38
Veritas Volume Manager
(VxVM)
– benefits of HP VERITAS Manager 3.1
– major features HP VERITAS Volume
Manager 3.1 (VxVM)
– Volume Manager Objects
– disks and disk groups

39
 Volume Manager History
– Pre-10.0: Disk sections or partitions on 800’s
– Whole disk with data (+ optional swap) on 700’s
– HP-UX 9.0 = LVM on Series 800 only
– HP-UX 8.07 = Software Disk Striping (SDS) for
the Series 700 for 2 Gbyte disk or smaller
– HP-UX 10.0+, LVM added to 700
– HP-UX 11.11+, VxVM added to 800 only

40
 Present and Future
• Present:
– HP-UX 11i, first release of Veritas VxVM 3.1.
• HP Workstations are not supported.
• VxVM 3.1 cannot control the root/boot disk.
– HP-UX 11.20 (IA-64) will support only VxVM .
• VxVM will then be able to control the root/boot disk.
• Future:
– Next major release - can choose either LVM or
VxVM to be the root/boot disk.

41
HP VERITAS Volume Manager 3.1

– HP VERITAS Volume Manager 3.1 (VxVM)

provides online disk management for HP-UX.
– Provides features not available with LVM.
• Java-based admin GUI and command line interface
• RAID-5
• Support for up to 32 mirrors
• Striped mirrors
• Dynamic Multi-pathing for I/O load balancing

42
 HP VERITAS Volume Manager
3.1

• Consists of two products:

– Base HP VERITAS Volume Manager
(B7961AA)
• basic volume manager features
• included with HP-UX 11i Application Release
– HP VERITAS Volume Manager (B9116A)
• full set of enhanced volume manager capabilities
including mirroring
• requires purchase of an additional license

43
 Base HP VERITAS Volume
Manager Overview
• Base product features:
– Java-based admin GUI
– Striping (RAID 0)
– Concatenation
– Path failover support (active/passive)
– Online resizing of volumes
– Task monitor

44
 VxVM Overview
•VxVM includes all of the Base product features plus:
– Load-balancing -- DynamicMultiPathing (active/active)
– Hot-Relocation/unrelocation
– Mirroring (RAID-1)
– Supports up to 32 mirrors
– Mirrored Stripes (RAID 1+0)
– Striped Mirrors (RAID 0+1)
– RAID-5
– Online data migration
– Online relayout
45
 Coexistence with LVM
– VxVM coexists with LVM.
– VxVM cannot be used to control root/boot
disk (use LVM)
– Both LVM and VxVM utilities are aware of
each other, and will not overwrite disks that
are being managed by the other
– Although VxVM is targeted toward new
installations, a conversion utility:
vxvmconvert is provided for converting LVM
volume groups to VxVM disk groups.
46
 Limitations of VxVM
• There are limitations with the first release of
VxVM on HP-UX 11i:
– Cannot use VxVM to control the root/boot disk. LVM
must be the chosen volume manager for root/boot disk.
– Disk monitor integrated with EMS framework is not
available for disk being managed by VxVM.
– VxVM does not support HP Process Resource Manager
(PRM).

47
 Supported Migration Scenarios
• VxVM 3.1 is the first HP Release of the Veritas
Volume Manager on HP-UX.
– No migration path from Veritas Volume Manager
3.0 on HP-UX 11.00 has been tested or is supported.
• Product was released by Veritas, and not HP
– VxVM 3.0 on HP-UX 11.00 users that want to
migrate to 11i VxVM 3.1: Backup their data.
• Upgrade to HP-UX 11i
• Install HP VERITAS Volume Manager 3.1.

48
 HP or Veritas VxVM
– VxVM 3.0 – Contact Veritas for support
– HP/VeritasVxVM 3.1 on 11i+ supported by HP Response
Center.
– # swlist -l product | grep vxvm
HP Supported:
HPvxvm 3.1 HP VERITAS Volume Manager
Purchased from VERITAS:
VRTSvxvm 3.X VERITAS Volume Manager
– To verify that all files belong to HPvxvm are installed:
# swverify -v HPvxvm

49
 What is VxVM?
• VxVM operates as a subsystem between
HP-UX and data management systems.
– manage physical disks as logical volumes
called volumes.
– Volumes can span more than one physical disk.
– provides enhanced recovery, data availability,
performance and storage configuration options.

50
 VxVM Objects
•VxVM uses two objects to do storage management.
– Physical Objects
• Physical disks
• Partitions
– Virtual objects
• VM Disks
• Subdisks
• Plexes
• Volumes
• Disk Groups

51
 Conceptual Comparison
VxVM Term LVM Term
•VxVM Disk •Physical Volume
•Subdisk •Physical Extent
•Volume •Logical Volume
•Disk Group •Volume Group
•Private Region •PVRA/BDRA/VGRA
•Free Space •Unused Physical Extent
•Plexes •Mirrors
•Dirty Region Logging •Mirror Write Cache

•Dynamic Multi-Pathing •PV links

52
 VxVM Objects Overview
Disk Group

Volume
plex01 plex02
0 0
dg04-01
dg01-03 40
dg03-02
70
dg02-03
130 130

VM Disk VM Disk VM Disk VM Disk

dg03 dg04
dg01 dg02

40m
dg04-01
60m dg02-03 30m
dg03-02
130m dg01-03

53
 VM Disk
• When a physical disk is placed under VxVM, a
Volume Manager disk (VM Disk) is assigned to
the physical disk.
– A VM Disk has two regions.
• Private Region
– Area where VM internal configuration information is stored
• Public Region
– Area where storage space is allocated from
– Each VM Disk has a unique Disk Media Name.
• Default name that is assigned by VxVM is disk##.
• The disk will now be referred to by its Disk Media Name
instead of its physical address.

54
 Example of a VM Disk
Disk under VxVM control

VM Disk: disk01 Physical Disk

disk01

Private Region
c0t4d0
Public Region

55
 Subdisks
– A subdisk is a set of contiguous disk blocks.
– Each subdisk represents a specific portion of a VM disk
which is mapped to a specific region of a physical disk.
– Allocated space from the VM Disk public region
– Default name of a subdisk is disk##-##

VM Disk: disk01 VM Disk: disk01 with one subdisk

disk01 disk01

disk01-03 subdisk

56
 Subdisks Example
VM Disk: disk01
Subdisks

disk01
disk01-01

disk01-01
disk01-02
disk01-02 Public Region

disk01-03 disk01-03

57
 Plex
– Plex consists of one or more subdisks located
on one or more physical disks
– Plex also can be called a mirror (although it is
one copy of the data).
– Plexes have a variety of layouts:
• Concatenation
• Striping (RAID-0)
• RAID-5

58
 Plex Example
VM Disk: disk01
Plex: vol01-01
disk01
disk01-01
disk01-01
Subdisks
disk01-02
disk01-02

59
 Plex Layout - Concatenation
• Concatenation
– Maps data into a linear manner onto one or
more subdisks in a plex.
– Space is allocated from the first subdisk from
beginning to end. Then space is allocated from
the remaining subdisks in a similar fashion.
– Concatenation using subdisks that reside on
more than one VM disk is called spanning

60
 Plex Layout - Striping
• Striping (RAID-0)
– Maps data so that data is interleaved among two
or more physical disks.
– Striped plex contains two or more subdisk
spread out over two or more physical disks.
– Data is allocated alternately and evenly to the
subdisks of a striped plex.

61
 Plex Layout - RAID-5
• RAID-5
– Same as Striping but one additional column of
data is used for parity.
– Provides data redundancy by using Parity.
– Requires additional license.

62
 Plex Example
disk01
Plex01
disk01-01

disk01-02 0
30m
disk01-03 disk01-03
disk01-04 30m
30 m
disk02-02
disk02
70m
100m
disk02-01
70m
disk02-02

disk2-03
disk2-04 63
 Volume
– Volume is a virtual disk device, but does not have
the physical limitations of a physical disk device.
– Volume consists of one or more plexes, each
holding a copy of the selected data in the volume.
– Volume can consist of 32 plexes, each of which
contains one or more subdisks.
• Never use more than 31 plexes, VxVM uses one plex
for online (automatic or temporary) operations.
– When a volume has two or more plexes , it is a
mirrored volume.
64
Volume Example

Volume vol01

plex vol01-01

disk01-01 subdisk

65
Mirrored Volume Example

Mirrored Volume vol06

plex vol06-01 plex vol06-02

disk02-01
disk01-01
disk04-03

disk03-02

66
 Disk Group
• A disk group is a collection of VM disks that share
a common configuration
– A disk group configuration is a set of records with
detailed information about related VxVM objects, their
attributes and their connections.
– A disk group and its components can be moved as a
unit from one host machine to another.
• Note: The rootdg (root disk group) cannot be removed
without disrupting volume manager service to the entire
system.
67
 Disk Group Example
Disk Group

Volume
plex01 plex02
0 0
dg04-01
dg01-03 40
dg03-02 70
dg02-03
130 130

VM Disk VM Disk VM Disk VM Disk

dg03 dg04
dg01 dg02

40m
dg04-01
60m dg02-03 30m
dg03-02
130m dg01-03

68
 VxVM Installation Process
Overview
• Prior to installation:
– Check software & hardware requirements.
– Check disk space required to install.
– Prepare for installing on a system with LVM
Volume Groups
• Install VxVM:
– Non-interactively with swinstall or
– Interactively with swinstall

69
 Software Requirements
• Software:
– Requires HP-UX 11i
– VxVM is on the HP-UX 11i Application
Release.
• It is not on the HP-UX 11i Core Media.

70
 Hardware Requirements
• Supported Platforms:
– N-class
– V-class
– L-class
– D-class 32/64 bit
– R-class
– K-class 32/64 bit

71
 Hardware Requirements (cont)
• Supported Peripherals:
– HP SureStore E Disk System FC10
– HP SureStore E Disk System SC10
– HP SureStore E Disk Array XP256
– HP SureStore Disk Array FC60
– EMC Symmetrix
– High Availability Storage System (HASS)
A3311A/12A
– HP SureStore E Disk Array 12H
– HP High Availability Fibre Channel Disk Array
Model 30/FC
– HP High Availability SCSI Disk Array Model 2072
 Disk Space and Memory
Required
Additional disk space required:
File System Kbytes Needed
/ 21000
/stand 2600
/usr 11000
/var 800
/opt 25000
Memory:
Minimum: 128 MB, recommended: 256 MB
•
73
Preparing to install on LVM system
• Prior to installing VxVM on an LVM system
– Review the contents of all the disks on the
system.
– Select at least one disk to be placed under
VxVM control.
– Decide whether to put any existing LVM disks
under VxVM control.
– Decommission any unused disks under LVM.

74
 VxVM Product
• VxVM consists of two products:
– B7961AA - Base HP VERITAS Volume
Manager
• Free of charge with the HP-UX 11i Application
Release.
– B9116AA - HP VERITAS Volume Manager
• The full product requires an additional purchase to
obtain the codeword required to unlock the software.

75
 Installing VxVM

• VxVM can be installed two ways:

– Non-interactively with swinstall
– Interactively with swinstall

• After installing VxVM, the system will

reboot.
76
 VxVM Initialization
•After installing VxVM, run vxinstall to initialize it.
•vxinstall does the following:
– Runs, displays the license information and prompts for key.
– Sets up the initial VxVM disk group, rootdg, and populates it
– Starts the configuration, notification, and relocation daemons
used by VxVM.
•vxinstall is run only once per system
– Used to setup rootdg if it does not exist.
– Exception is special troubleshooting situations.

77
First VxVM Disk Group: rootdg
• VxVM cannot be used until rootdg is set up
and at least one disk is assigned.
– One disk must remain under rootdg at all times
while VxVM is running.
– Despite its name, rootdg will not be the system
root disk, as VxVM cannot be used to control
the root/boot disk.
– LVM will control the root/boot disk.

78
 Verify VxVM has initialized

• After the installation is complete, run the

following command to verify that the key VxVM
processes are running:
ps -ef | grep vx
output should include the following VxVM processes:
• vxiod
• vxnotify
• vxrelocd
• vxconfigd

79
 VxVM Utilities
• VxVM provides the following user
interfaces that can be used to manage disks:
– VMSA - graphical user interface
– vxdiskadm - interactive menu-based interface
– CLI - Command-Line Interface

80
 Setting up the Storage
Administrator GUI
• The Storage Administrator, VMSA, is the Java
based interface for VxVM.
– Installed as part of the Base HP VxVM.
– After initializing VxVM with vxinstall, you must set
up VMSA.
– VMSA consists of a server and a client:
• The server (vmsa_server) must be installed and run on a HP-
UX 11i machine with VxVM.
• The client (vmsa) is the graphical user interface. It must be
installed and run on an HP-UX 11i machine that supports the
Java 1.1 Runtime Environment.

81
 Starting VMSA
– Log in as root.
– Create and export VMSAHOME variable #
export VMSAHOME=/opt/HPvmsa
– Update PATH variable to include $VMSAHOME/bin

– It may be helpful to include these variables in your

.profile.
– To start the server run:
# vmsa_server &
– To start the client
#xhost + hostname
# vmsa &

82
 Customizing Security
– By default, only root can run VMSA.
– The system can be set up to allow other users to
run VMSA by:
• Adding a group name vrtsadm to /etc/group
• Specifying users, including root, as vrtsadm
members.
– VMSA can also be configured in read-only
mode.
• This mode is enabled via the properties file.
/opt/HPvmsa/vmsa/properties
83
 VMSA commands
– Query the run state of the server:
• # vmsa_server -q

– Kill the Server:

• # vmsa_server -k

– Show the VMSA version number:

• # vmsa_server -v
84
 VMSA GUI
Toolbar

Grid
Object tree

Command
launcher

85
 Useful CLI commands

– vxassist - Create and manage volumes in a

single step.
– vxprint - List VxVM information.
– vxdg - Create and manage disk groups.
– vxdisk - Create and manage VM Disks.
– Some CLI commands can be found in the
/etc/vx/bin directory.
• Add this directory to $PATH

86
 Removing VxVM
• Before removing VxVM software using
swremove, you must:
– Do a full backup of your data.
– Move data out of VxVM control.
– Stop VxVM.

87
 Summary
• VxVM is an LVM-like tool
• Much more powerful and complex
• Much more versatility
– Striping
– Mirroring (up to 31, not just 3)
– Stripe and mirror
– RAID 5 data protection
– And more
88