Veritas Dynamic Multi-Pathing

Optimized, Centrally Managed Multi-Pathing

Ashish Yajnik
Principal Product Manager

Storage & Availability Mgmt Group

Dynamic Multi-Pathing
1

Datacenter Multi-Pathing Trends & Challenges

Multi-Vendor storage & server environments increasingly a norm Single tools across physical & virtual env critical for efficiency

Configurations are continually scaling up & scaling out

Bridging System & Storage Team Communication is a challenge

Dynamic Multi-Pathing

SYMANTEC VISION 2011

DMP - Optimized Multi-Vendor Multi-Pathing Solution

One interface for all OS, storage & architecture
Enables multi-vendor environment & standardization Industrys broadest hardware compatibility list ~1000 arrays supported

Enterprise grade performance

Best in class throughput Proactive path management Automated & reliable recovery

Bridges storage & server boundaries

Enables informed decision making Reduces risk of configuration errors

Managed by Veritas Operations Manager

Single-pane-of-glass datacenter wide path management
Dynamic Multi-Pathing SYMANTEC VISION 2011
3

Agenda
Heterogeneity
Standardize across physical & virtual environments Leverage ANY Storage Standardize across any OS & any Storage Management Stack

Advanced Usability & Management

Bridge the Storage & Server Information Gap Centrally manage multi-pathing with Veritas Operations Manager (VOM) Identify bottleneck with I/O Stat tool

Maximize Availability & Performance

Proactive Path Failover & Reliable Failback Effective Load Balancing
Dynamic Multi-Pathing SYMANTEC VISION 2011
4

Heterogeneity
Standardize across physical & virtual environments Leverage ANY Storage Standardize across ANY OS & Storage Management Stacks

Advanced Usability & Management

Bridge the Storage & Server Information Gap Centrally manage multi-pathing with Veritas Operations Manager (VOM) Identify bottleneck with I/O Stat tool

Maximize Availability & Performance

Proactive Path Failover & Reliable Failback Effective Load Balancing
Dynamic Multi-Pathing SYMANTEC VISION 2011
5

Exploiting OS Native Stacks: Introducing DMP Standalone Product

Enable DMP EVERYWHERE
Clustering File System Volume Manger Native MP Veritas Cluster Server File System Volume Manger

Any OS Stack

Veritas Cluster Server Veritas Storage Foundation

DMP

DMP

Simpler for You to Standardize on DMP

Leverage DMPs extensive array support Standardize on single multi-pathing across ALL storage management stacks DMP reporting and management through VOM Seamless migration from native multi-pathing to DMP
SYMANTEC VISION 2011
6

Dynamic Multi-Pathing

Exploiting OS Native Stacks: DMP Standalone SCL and HCL Overview

Native FS Native LVM
Native MultiPathing

Any OS Stack

Native FS

VCS Native FS Native LVM

Native MultiPathing

VCS Native FS Native LVM DMP

Native LVM DMP

DMP Single node Native Stack

OS Version
AIX 5.3 & 6.1
RHEL4/5, SLES10, SLES11 Solaris 9, 10 HP-UX 11.31 Windows 2003, 2008

DMP Native Stack with Clustering

Clustering
VCS
VCS VCS VCS VCS, MSCS

Architecture
PowerPC
Intel/AMD X64 (no 32 bit) SUN Sparc & X86_64 IA & PA X86, X64, IA64

Storage Management Stack

AIX LVM, ASM
Linux LVM, ASM SUN ZFS, ASM HP-UX LVM Windows LDM

Enabled by DMP Native Support feature Oracle ASM & RAC supported with DMP Standalone Leverages DMPs existing HCL 1000+ arrays supported
Dynamic Multi-Pathing SYMANTEC VISION 2011
7

Exploiting ANY Storage: Eliminate Vendor Lock-In With Dynamic Multi-Pathing

Locked-In to Storage Vendors Multi-Vendor Benefits

Any Storage

EMC PowerPath

HDLM

Storage vendor lock-in results in 30-100% price premiums on storage

Enabling Multi-Vendor environments saves on storage cost

DMX

Clariion

USP-V

Free to Choose the Most Cost Effective Storage

Point tools are complex to manage or dont support all environments

Standardize on DMP across heterogeneous OS & Storage solutions

Dynamic Multipathing (DMP)

DMX

Clariion

HDS AMS

IBM XIV

Dynamic Multi-Pathing

SYMANTEC VISION 2011

Exploiting ANY Storage: Modular Architecture to Optimize to Any Storage

Challenge for Optimization
Different Array modes are hard to support

Any Storage

Array Support Library (ASL)

HW specific plug in to DDL Automate DMP driver configuration Discover array specific attributes

ASL1

ASL2

ANY FS ANY VM
APM2

DDL

vxdmp SCSI Drivers HBA Drivers

APM1

Array Policy Module (APM)

HW specific plug in to DMP

Dynamically loadable kernel module

Allows array-specific integration

Allows DMP to support 50+ array families & 1000+ array models
Dynamic Multi-Pathing SYMANTEC VISION 2011

DISK

Exploiting Virtualization: Enhanced VIOS Support with DMP

DMP Enhances VIOS
Enable standardization of DMP on AIX LPAR

Virtual Support

Virtualized AIX Server

LPAR
DMP

DMP resides inside the VIOS

Ability to run DMP inside LPAR as well
DMP

Leverage extensive DMP HCL Deep Visibility & Reporting

DMP in LPAR gets deep storage visibility Enables consistency of device names Informed decision making Centrally managed by VOM

Deep Storage Visibility

DMP DMP

VIO Server

VIO Server

Dynamic Multi-Pathing

SYMANTEC VISION 2011

10

Summary of DMPs Heterogeneous Support

OS & Native Stack Support
All Major Unix/Linux Support Windows LDM Support Oracle ASM support with DMP Standalone

Virtualization Multi-Pathing in LDOMs, LPARs & KVM Support

Adding value in VIOS environments

Utilize DMP across Physical & Virtual environments

Extensive Storage Support

Dynamic Multi-Pathing

All Tier1, Tier2 & major Tier3 arrays supported 1000~ arrays part of extensive HCL

SYMANTEC VISION 2011

11

Heterogeneity
Standardize across physical & virtual Leverage ANY Storage Standardize across any OS

Advanced Usability & Management

Bridge the Storage & Server Information Gap Centrally manage multi-pathing with Veritas Operations Manager (VOM) Identify bottleneck with I/O Stat tool

Maximize Availability & Performance

Proactive Path Failover & Reliable Failback Effective Load Balancing
Dynamic Multi-Pathing SYMANTEC VISION 2011
12

Bridging the Storage & System Information Gap Mapping OS Devices to SAN Devices
c0t5d65 c3t5d65 EMC0_19 EMC0_19 EMC0_0AF4

Manageability

EMC0_0AF4

c1t5d65 c0t5d65 c3t5d65 c4t5d65 c0t5d65

c1t5d65 c3t5d65 c4t5d65 c0t5d65

c1t5d65 c3t5d65 c4t5d65

LUN

LUN

LUN

WITHOUT DMP

LSN: D600172E015B

AVID: 0AF4

Challenges Same LUN, different name across servers OS names do not map to storage terminology
Dynamic Multi-Pathing

LSN sorting algorithm

AVID naming Consistent names across servers Meaningful names for storage teams

Consistent names across servers

SYMANTEC VISION 2011

13

Customized Device Naming

#> vxgetdmpnames > /etc/vx/device.custom #> vxddladm assign names file=/etc/vx/device.custom
Ability to change device names to user-specified names
$>cat /etc/vx/device.custom Vxdisk list output Admin can decide the names for the devices enclosure vendor=DGC product=DISK serial=CK200070400359 name=CLARiion

DEVICE TYPE DISK GROUP STATUS Framework for a file containing user-specified names to be uploaded to hosts

dmpnode serial=600601606D121B007C778BC48EDBDB11 name=MyApp-lun1

MyApp-lun1 auto:cdsdisk online Ability to clear the user-specified names with a CLI dmpnode serial=600601606D121B008FB6E0CA8EDBDB11 name=CLARiion-lun2
dmpnode serial=600601606D121B0091B6E0CA8EDBDB11-name=CLARiion-lun3 CLARiion-lun2 auto:cdsdisk online User-specified names will be made persistent dmpnode serial=600601606D121B0093B6E0CA8EDBDB11 name=CLARiion-oralun1 CLARiion-lun3 auto:cdsdisk online dmpnode serial=600601606D121B00BFAB99C28FDBDB11 name=CLARiion-oralun2

CLARiion-oralun1 auto:cdsdisk

oradg

online

CLARiion-oralun2 auto:cdsdisk
MyApp-lun1
EMC Clariion LUN

oradg

online

CLARiion-lun2
EMC Clariion LUN

CLARiion-oralun1 CLARiion-oralun2
EMC Clariion LUN SYMANTEC VISION 2011 EMC Clariion LUN
14

Manageability

Bridging Storage & Server Team Information Gap

DMP Discovers Device Characteristics
App Team
My application is on tier 1 servers, what tier of storage is it using? SSD, SATA, SAS, FC?

5
My mission critical file system is being protected or not? RAID5, RAID10?

Server Team

Is my critical data being replicated underneath or not?

Raid Levels RAID5, RAID10 Replication Attributes & Snapshots Thin Or Standard Devices Tier Information SSD, SATA, FC, SAS

#vxdisk e list DEVICE TYPE DISK USP0_af14 auto USP0_53 USP0_af15 auto USP0_54 USP0_af16 auto USP0_55
Dynamic Multi-Pathing

GROUP DFDG2 DFDG2 DFDG2

STATUS online online online

OS_NATIVE_NAME ATTR c0t50060482CAE053ECd25s2 PVOL,SSD c0t50060482CAE053ECd26s2 SATA,RAID_1 c0t50060482CAE053ECd27s2 FC,RAID_10

SYMANTEC VISION 2011
15

Proactively address risks to prevent downtime

Manageability

Reduce errors in your environment & increase HA

Manage risks in your environment with proactive reporting & alerts Customized health check reports for different violations
Dynamic Multi-Pathing SYMANTEC VISION 2011
16

Manageability

Better environment control with DMP I/O Statistics

New options in vxdmpadm provides:
Filter out 0 entries vxdmpadm z iostat show <selection-criteria> Displayed queued (-q) and errored (-e) I/Os vxdmpadm q iostat show <selection-criteria> Per second IO statistics vxdmpadm s iostat show <selection-criteria> Display output in units that make sense for normal admins vxdmpadm u iostat show <selection-criteria>

Aggregate statistics for capacity management

HBA port level statistics Groupby option to get enclosure, controller, and disk level statistics vxdmpadm iostat show groupby=ctlr|dmpnode|enclosure <selection criteria>

5.1 CLI Scalability & Accuracy Improvements

Filtering & grouping of data in the kernel Reduces burden on CLI thus improving reporting accuracy
Dynamic Multi-Pathing SYMANTEC VISION 2011
17

Manageability

Better environment control with DMP I/O Statistics

$>vxdmpadm -u m s iostat show groupby=ctlr interval=30 cpu usage = 878us per cpu memory = 32768b OPERATIONS/SEC BLOCKS/SEC CTLRNAME READS WRITES READS WRITES c0 33200 11600 207.25m 248.13m c1 33400 9040 233.24m 229.73m c2 17600 5040 61.53m 64.62m c3 18080 5680 98.45m 99.48m $>vxdmpadm -u m -s iostat show groupby=enclosure ctlr=c1 interval=30 cpu usage = 1396us per cpu memory = 32768b OPERATIONS/SEC BLOCKS/SEC ENCLOSURENAME READS WRITES READS WRITES emc_clariion0 17200 4000 118.97m 73.65m ams_wms0 12000 4400 82.07m 80.01m eva4k6k0 4000 4000 30.76m 73.65m

Dynamic Multi-Pathing

SYMANTEC VISION 2011

18

Heterogeneity
Standardize across physical & virtual Leverage ANY Storage Standardize across any OS

Advanced Usability & Management

Bridge the Storage & Server Information Gap Centrally manage multi-pathing with Veritas Operations Manager (VOM) Identify bottleneck with I/O Stat tool

Maximize Availability & Performance

Proactive Path Failover & Reliable Failback Effective Load Balancing
Dynamic Multi-Pathing SYMANTEC VISION 2011
19

Maximize Performance

Optimize Application I/O Throughput

MinimumQueue (MinQ) I/O Policy
Smallest HBA queue depth path chosen Typical work load & SAN configurations

Adaptive I/O Policy

Paths with lowest response time are used End-to-End bottleneck identification

HBA - A

HBA - B

Adaptive-MinQ
Combines best of MinQ & Adaptive

SP-A

SP-B

Balanced
LUN segmented in equal size partitions, Each partition associated with a path

Round-Robin, Single-Active, Priority

Standard I/O policies for native multi-pathing
Dynamic Multi-Pathing SYMANTEC VISION 2011
20

Maximize Availability

Proactive Path Management: Smarter & Faster I/O Error Analysis

Device Discovery Layer (DDL) VxFS/ANY FS VxVM/ ANY VM/ASM DMP
Regular IO Error Analysis

Multi-threaded for faster discovery

DMP Device Driver (VxDMP)

Multi-threaded core processes

Normal I/Os go through SCSI layer

Chance for SCSI to recover transient errors

OS SCSI Drivers HBA Driver

DMP Error Analysis bypasses SCSI layer SCSI Bypass Benefits:

Faster error analysis
Asynchronous interface More detailed information of I/O outcome

Dynamic Multi-Pathing

SYMANTEC VISION 2011

21

Maximize Availability

Proactive Path Management

DMP Proactive Path Management Capabilities
Avoids build-up of I/O on failing paths by proactive methods

Results in quicker I/O error analysis and speedier failover

Throttling
If I/O is backed up on the device for 10 seconds, mark the path Suspect

HBA API Notifications

When Port logout notification happens, mark the associated paths Suspect

Sub-Path Failover Group Protocol

Collectively taken an action by inquiring about few paths Good for scale out configurations and results in speedier analysis
Dynamic Multi-Pathing SYMANTEC VISION 2011
22

Maximize Availability

Proactive Path Management: Maximize Uptime in Scaled Up Config

Total Throughput Impact of a Fabric Failure 4096 LUN Paths Fail at Once, 2048 of them are Primary Paths with Live I/O DMP Minimum Queue
6000.00

5000.00

4000.00

KB/sec

3000.00

2000.00

Tier1 - ALUA 1024 LUNs, 8 Paths per LUN

1000.00

Failure of Fabric A DMP Fails over of 4096 LUN-paths in 40 seconds

0.00

4 ports per controller, each acting as primary port for 128 LUNs. 1024 LUNs total (128 * 8).

Workload: multiple vxbench, single striped volume, ensuring that all 4096 primary paths carry traffic.
Each Fabric contains 2 switches. Fabric A failed by disabling all Inter Switch Links
SYMANTEC VISION 2011
23

0: 00 :0 0 0: 00 :2 0 0: 00 :4 0 0: 01 :0 0 0: 01 :2 0 0: 01 :4 0 0: 02 :0 0 0: 02 :2 0 0: 02 :4 0 3: 00 :0 0 0: 03 :2 0 0: 03 :4 0 0: 04 :0 0 0: 04 :2 1 0: 04 :4 0 0: 05 :0 0 0: 05 :2 0 0: 05 :4 0 0: 06 :0 0 0: 06 :2 0 0: 06 :4 0 0: 07 :0 0 0: 07 :2 0 0: 07 :4 0 0: 08 :0 0 0: 08 :2 0 0: 08 :4 0 0: 09 :0 0 0: 09 :2 0 0: 09 :4 0 0: 10 :0 0
Run tim e (hh:m m :ss)

Maximize Availability

Real-Time & Reliable Path Recovery

DMP Path Recovery is in Kernel for reliability DMP regularly checks the state of failed and idle paths
Successful SCSI inquiry will cause DMP to bring the path back online

SCSI inquiry failure will cause DMP to disable the path

Path checking is triggered:

By fabric events: a port login event triggers path recovery
Utilizes Path Probing mechanism: Every 5 minutes (tunable)

Dynamic Multi-Pathing

SYMANTEC VISION 2011

24

Maximize Availability

Nimble Recovery of Paths LIPP - Low Impact Path Probing

Ensure real time accurate path state awareness
Optimized utilization of system resources

Utilizes Sub-path Failover Groups (SFG)

LIPP groups paths into logical entities Health of few paths determines the health of the rest in the group Reduces CPU overhead & avoids flooding of SAN with probes

Tuning LIPP
Tunable (dmp_low_impact_path_probe) to turn on/off LIPP Ability to change number of probes sent to idle or disabled paths

Dynamic Multi-Pathing

SYMANTEC VISION 2011

25

DMP - Optimized Multi-Vendor Multi-Pathing Solution

One interface for all OS, storage & architecture
Enables multi-vendor environment & standardization Industrys broadest hardware compatibility list Utilize DMP to enable more arrays

Enterprise grade performance

Best in class throughput Proactive path management Utilize faster failover & effective load balancing

Bridges storage & server boundaries

Enables informed decision making Utilize VOM & DMP for deep visibility

Managed by Veritas Operations Manager

Download VOM & utilize it for storage/path management
Dynamic Multi-Pathing SYMANTEC VISION 2011
26

Thank you!
Ashish Yajnik - Ashish_Yajnik@Symantec.com Chad Bersche Chad_Bersche@symantec.com

Copyright 2011 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice.

Dynamic Multi-Pathing

27