BEST PRACTICES FOR SUN STORAGE 6000 SERIES AND 2500 SERIES ARRAYS

VMware STORAGE VMotion

White Paper March 2009

Sun Microsystems, Inc.

Table of Contents
Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 VMware Storage VMotion Basics . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 What is Moved by VMware Storage VMotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . How VMware Storage VMotion Works . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . VMware Storage VMotion Requirements . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . VMware Storage VMotion Best Practices . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 1 2 5 6

VMware Storage VMotion Uses . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Storage Tiering with Sun Storage Arrays . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7 Sun Storage Array Performance and Scalability Management . . . . . . . . . . . . . . . . 8 Storage Maintenance . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9 Sun Storage 6000 and 2500 series architecture guidelines for VMware Storage VMotion . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10 References . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

1 VMware Storage VMotion

Sun Microsystems, Inc.

Introduction
VMware Storage VMotion, allows the user to perform live online migration of a running Virtual Machines disks, or disk files from ESX Servers perspective, from internal disk drives on a server, within and across storage arrays without the need for downtime while maintaining transaction integrity. The key difference between traditional live storage migration and Storage VMotion is that Storage VMotion can perform live online migration of the entire operating system, including Swap or page files and any virtual machine related configuration and log files, from one storage device to another. The purpose of this document is to provide guidelines and best practices for using VMware Storage VMotion with Sun Storage 6000 series, including the new ST6580 and ST6780, and the Sun Storage 2500 series storage arrays. Although all Fibre Channel or iSCSI capable Sun Microsystems Storage arrays listed in the VMware Storage/SAN Compatibility Guide are supported with VMware Storage VMotion. The guide is updated on a regular basis with new storage arrays added so it is highly recommended that the online version of the guide be referred to prior to providing any architectural recommendations.

VMware Storage VMotion Basics

Storage VMotion was introduced in VMware Virtual Infrastructure 3 release with ESX Server 3.5 and vCenter 2.5, initially supporting Fibre Channel storage. Release of ESX Server 3.5 update 1 added support for iSCSI storage.

What is Moved by VMware Storage VMotion

Virtual machines home directory Configuration file (vmx) Logs Swap file (vswp) Any snapshots of the virtual machine Any other files related to the virtual machine Virtual machine disks or luns (vmdk) The operating system disk Data disk(s)

The disks can be moved independently as needed or required. The OS disk can be moved without having to move data disks or luns and any and all disks can be placed on any disk/lun configuration on the particular ESX server.

2 VMware Storage VMotion

Sun Microsystems, Inc.

How VMware Storage VMotion Works

VMware Storage VMotion requires VMware vCenter 2.5 or later, VMware VMotion license enabled on the host and the VMware provided Remote Command-Line Interface application or Virtual Appliance which can be downloaded from VMware.com. There are VMware vCenter plug-ins available online which allow for a GUI-based Storage VMotion capability but none are officially support by VMware. For more details, please refer to the VMware VI3 Basic System Administration Guide and Remote Command-Line Interface Installation and Reference Guide. VMware Storage VMotion consists of six steps outlined below:

Step 1. Copy virtual machine home directory to new location. This step uses NFC copier to copy the files over to the new location

Step 2. Self-VMotion to new virtual machine home. A self-VMotion uses the same process as a normal VMotion operation. This step enables the use of the new swap file and reopen other copied files such as the virtual machine config file and other such files.

3 VMware Storage VMotion

Sun Microsystems, Inc.

Step 3. Take a disk-only snapshot. This creates a child disk which will now be used to maintain ALL changes being made to the Virtual Machine data.

Step 4. Copy virtual machine disk to destination (read-only).

4 VMware Storage VMotion

Sun Microsystems, Inc.

Step 5. Consolidate child disk created in step 4 into copied disk.

Step 6. Delete original virtual machine home and disks and enable read/write on new copied disk.

5 VMware Storage VMotion

Sun Microsystems, Inc.

VMware Storage VMotion Requirements

VMware vCenter with VMware VMotion license installed VMware VMotion must be enabled and properly configured on the ESX server (Refer to VMware Basic Administration Guide) The virtual machine must not be using any device preventing VMotion (e.g. floppy disk, CD-ROM) A physical network link for VMotion is not required for Storage VMotion The ESX server must have sufficient resources to perform all of the steps outlined above: At least twice the amount of memory and CPU resources to accommodate a second instance of the virtual machine on which Storage VMotion is to be performed (see Steps 1 and 2 above). For example, if the virtual machine uses 1 vCPU and 500 MB of memory, there needs to 1 vCPU and 500 MB readily available beyond what is being used by all running virtual machines to be able to perform Storage VMotion operation on this particular virtual machine. This doubling of memory and CPU resources is temporary while Storage VMotion steps above are taking place but required. At least twice the amount of storage needs to be available at the source storage to allow for a snapshot to be taken prior to copying the virtual machine disk to the destination storage (see Steps 3 to 5 above) Virtual machines disks must be snapshot eligible as a snapshot is taken in Step 3 to create a child disk prior to copying the disk to the new location. Without this capability Storage VMotion will not work. When using RDMs, RDMs in Virtual Compatibility Mode are supported with Storage VMotion as these are snapshot capable. Snapshots can not be taken for RDMs in Physical Compatibility Mode (refer to VMware Fibre Channel SAN Configuration Guide and VMware VM vs. Physical Server I/O Performance Comparison document for more information on RDMs). The ESX server must have access to both the source and destination storage or datastore A maximum of four concurrent Storage VMotion migrations can occur simultaneously on the same datastore Only Fibre Channel and iSCSI storage arrays are supported Virtual machines requiring Storage VMotion capability must not have network connection to any internal virtual switches

6 VMware Storage VMotion

Sun Microsystems, Inc.

Figure 1. Data on RDM luns do not move as part of Storage VMotion.

NOTE: Storage VMotion of an RDM does not move data from one physical raw lun to another. It only moves the mapping file over to the location where the virtual machine config file (vmx) is moved. To move data from Lun 1 physically, a different technology such as RVM can be used. (See Figure 1).

Virtual machines requiring Storage VMotion capability must not have any cluster relationship with another virtual machine as is the case when using a clustering technology such as Microsoft Cluster Server or Sun Cluster at the virtual machine level Simultaneous Storage VMotion and VMotion on the same virtual machine is not allowed. A virtual machines datastores can be moved to a different storage first using Storage VMotion and the virtual machine can be moved to a different ESX server second or vice-versa.

VMware Storage VMotion Best Practices

Although not required, it is best practice to use the VMware Remote CLI Virtual Appliance to perform Storage VMotion tasks via command line. There are GUI-based VMware vCenter plug-ins available but are not supported by VMware.com. A lot of pre-planning need to take place prior to any Storage VMotion operation Plan for Storage VMotion during initial design and architecture of the ESX servers, the SAN and the storage arrays Free memory, CPU and storage space in source LUN need to be available Ideally, use different host bus adapter ports for the source and destination storage on the ESX server for maximum performance

7 VMware Storage VMotion

Sun Microsystems, Inc.

If moving virtual machine disks on datastores residing on the same storage array, use different target ports on the array for the source and destination luns for maximum performance Coordination between storage, server and VMware administrators is key as all components need to work seemlessly for a successful Storage VMotion operation It is strongly recommended that Storage VMotion be performed during off peak hours for the specific virtual machine in order to avoid any noticeable performance impact

VMware Storage VMotion Uses

Storage Tiering with Sun Storage Storage Arrays
The most compelling case for Storage Storage VMotion is made when the need to migrate from legacy to latest array technologies arises in a virtualized infrastructure without a considerable impact to availability. Especially if there is no array-to-array replication available, as may be the case with customers migrating from a non-Sun storage array to the new Sun Storage 6780 or 6580 array for instance. There are many alternative methods to migrate data from one storage to another, some of which are listed below: Use of remote replication technologies within the arrays such as Remote Volume Mirroring for Sun Storage 6000 series arrays or TrueCopy for Sun Storage 9000 series arrays. However, this technology requires additional feature licenses to be enabled on the arrays, which the customer may not need beyond the scope of migrating the data from older to newer arrays. And, the remote replication technology works only within the same series of arrays so replication between Sun Storage 6000 series and Sun Storage 9000 series is not available without the use of additional software.
NOTE: Storage VMotion is not a technology replacement for Disaster Recovery implementations requiring data services such as Remote Replication to remote sites. Remote Replication of LUNs is still required for Storage Level data replication to remote sites and for maintaining data redundancy at the same site.

Snap Shot technologies within an array can be used to migrate data from one lun to another due to any number of reasons such as a new raid layout, for example moving from RAID-5 to RAID-6. These snapshot technologies such as ShadowImage for Sun Storage 9000 series or SnapShot for Sun Storage 6000 or 2500 array series also may require a license feature as is the case with ShadowImage for Sun Storage 9000 series and require further configuration tasks to setup a mirror, sync the luns, break the mirror and then perform further administration tasks at the ESX server level for the secondary lun to be mapped to the same ESX server. Storage VMotion is also not a technology replacement for In-Storage Replication Data Services. Storage VMotion does not create a copy of the data, rather migrates virtual machine and its associated disk files between a given set of LUNs.

8 VMware Storage VMotion

Sun Microsystems, Inc.

Stop all I/O to luns to be migrated, create a full backup and perform a restore to the destination array Possible issues include unnecessary and prolonged downtime, errors during backups and/or restores, possible overloading of the backup infrastructure which may take resources away from regular day-to-day backup requirements or having to create or add new backup resources such as tape drives, etc. To avoid all of these and many other much more complicated, task-oriented methods to perform a simple lun migration from one array to another or a full migration of all ESX server luns from one array to anther or to a different set of luns with in the same array, the use of Storage VMotion is the obvious choice as it maintains uptime for the virtual machines in any situation without having go through complex tasks of setting up remote replication, backup and recovery or an internal snapshot technology. Virtual machine disks resident on an iSCSI array can be migrated to Fibre Channel or vice-versa. For instance, a Test Virtual Machine moving into a Production status can simply be migrated from a test Sun Storage 2510 to a new Fibre Channel Sun Storage 6780 Production array using Storage VMotion.

Sun Storage Array Performance and Scalability Management

VMware Storage VMotion can be used to move virtual machine disks in order to achieve optimal performance and scalability for established SAN environments as well. Consider an ESX host where all of the virtual machines currently reside on the internal disk storage. All of the virtual machines are exposed to down time due to the server being the single point of failure. If the customer decides to scale their current configuration to two ESX servers and shared SAN and/or iSCSI storage, the configuration becomes much more scalable avoiding single points of failures at the server layer. (See Figure 2).

Figure 2. VMs residing on internal drives of an ESX server are exposed to SPoF. Shared Storage offers enhanced redundancy and availability.

9 VMware Storage VMotion

Sun Microsystems, Inc.

The configuration can be further enhanced by adding arrays which would then allow for complete redundancy at all levels. (See Figure 3). Virtual machine disks can be moved seamlessly from internal server disks to the single Sun Storage 6140 and then can be further scaled and divided between the Sun Storage 6140 and the Sun Storage 6780 based on the degree of data and virtual machine criticality. Data requiring high performance and redundancy can be moved independently to the higher performance more scalable ST6780 and other data and virtual machine disks can stay on the ST6140. All of this data migration can be performed using VMware Storage VMotion.

Figure 3. Low $$/GB storage should be used for non-essential data and High $$/GB, high performing storage for critical data.

Storage Maintenance
Another use of Storage VMotion could be storage maintenance which may have an impact on production I/O. Storage space reclamation is a major factor and another compelling reason for use of Storage VMotion.

10 VMware Storage VMotion

Sun Microsystems, Inc.

Sun Storage 6000 and 2500 Series Architecture Guidelines for VMware Storage VMotion
When architecting a storage solution for VMware, it is important to consider all aspects of the software including Storage VMotion, which can have a considerable impact on storage I/O performance when active. (See Figure 4).

Figure 4. If possible, isolate Storage VMotion I/O traffic from Production I/O by using different Initiators/Targets.

Given the illustration above, some important best practices to consider: It is strongly recommended to separate Storage VMotion traffic from Production I/O traffic at the HBA level to the Storage array port to avoid any possible performance bottle necks Another reason to design the solution as such is so data moves from Source Lun to Destination Lun using a completely different I/O path which will allow Storage VMotion to complete the process quicker than if the same path was to be used to moved hence possible creating a bottleneck. The label Storage VMotion path and Production path is generic in terms of the illustration above.

11 VMware Storage VMotion

Sun Microsystems, Inc.

Sun Storage Common Array Manager software can be used to change Preferred Controller for each Lun or Volume on the array which allows the I/O load to be spread across controllers. (See Figure 5).

Figure 5. Avoid moving VMware datastores from a lun within the same volume on an array. For example, in the illustration above, moving a datastore from Lun 2 to Lun 3 is not recommended as it will put unnecessary I/O stress on that volume. The given move also defeats the purpose of using Storage VMotion as a tool to disperse data effectively and decrease performance bottlenecks unless there was a specific reason to perform this task such as the need for more space at the lun level. For the Sun Storage 2510 array, it is strongly recommended to use dedicated host network interface and array port to avoid severe I/O bottlenecks when moving datastores from or to the array. Since the array ports run at 1 GigE speeds, moving datastores from a Fibre Channel array such as the Sun Storage 2540 could result in very slow Storage VMotion operation. Since Storage VMotion only currently supports Fibre Channel and iSCSI storage arrays, Sun Storage 2530 array is not supported with Storage VMotion Follow all Sun Storage and VMware best practices and guidelines to configure the ST6000 series and ST2500 series arrays for use with VMware ESX servers

12 VMware Storage VMotion

Sun Microsystems, Inc.

References
Sun Storage 6580 and 6780 Release Notes http://dlc.sun.com/pdf/820-5776-11/820-5776-11.pdf Sun Storage 6580 and 6780 Hardware Installation Guide http://dlc.sun.com/pdf/820-5773-10/820-5773-10.pdf Sun Storage 2500 Series Array Hardware Installation Guide http://dlc.sun.com/pdf/820-0015-12/820-0015-12.pdf Sun Storage 6140 Array Hardware Installation Guide http://dlc.sun.com/pdf/819-7497-11/819-7497-11.pdf Sun Storage Common Array Manager Software Installation Guide http://dlc.sun.com/pdf/820-5747-10/820-5747-10.pdf VMware VI3 and Virtual Infrastructure Basic System Administration Guide www.VMware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_admin_guide.pdf VMware Storage/SAN Compatibility Guide www.VMware.com/resources/compatibility/pdf/vi_san_guide.pdf VMware Remote Command-Line Interface Installation and Reference Guide www.VMware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_rcli.pdf VMware Fibre Channel SAN Configuration Guide www.VMware.com/pdf/vi3_35/esx_3/r35u2/vi3_35_25_u2_san_cfg.pdf

VMware Storage VMotion

On the Web sun.com

Sun Microsystems, Inc. 4150 Network Circle, Santa Clara, CA 95054 USA Phone 1-650-960-1300 or 1-800-555-9SUN (9786) Web sun.com
2009 Sun Microsystems, Inc. All rights reserved. Sun, Sun Microsystems, and the Sun logo are trademarks or registered trademarks of Sun Microsystems, Inc. or its subsidiaries in the United States and other countries. Information subject to change without notice. SunWIN #559704 Lit. #SYWP14884-0 04/09