NETAPP TECHNICAL REPORT

Oracle VM and NetApp Storage Best Practices Guide

Preetom Goswami, NetApp Antonio Jose Rodrigues Neto, NetApp September 2008 | TR-3712

EXECUTIVE SUMMARY
NetApps innovative technologies enable organizations to extract benefits out of their virtual infrastructures by seamlessly integrating advanced virtualized storage alongside the virtual servers. NetApp provides industry-leading solutions in the areas of data protection, thin storage provisioning, data deduplication, file-based backups, instantaneous virtual machine (VM) backup and restores, and instantaneous VM cloning for testing, application development, and training purposes. This technical report reviews the best practices for implementing an Oracle VM virtual infrastructure with NetApp storage systems. Scenarios corresponding to all three storage protocolsNFS, iSCSI and FC are covered along with an Oracle VMNetApp deployment case study.

TABLE OF CONTENTS
1 INTRODUCTION ......................................................................................................................... 3
1.1 1.2 1.3 PURPOSE .....................................................................................................................................................3 SCOPE ..........................................................................................................................................................3 INTENDED AUDIENCE AND ASSUMPTIONS.............................................................................................3

BACKGROUND .......................................................................................................................... 3
2.1 2.2 2.3 ADVANTAGES OF VIRTUALIZATION.........................................................................................................3 ORACLE VM FOR ENTERPRISE READY VIRTUALIZATION ....................................................................4 NETAPP STORAGE SOLUTIONS FOR ENTERPRISE READY VIRTUALIZATION ..................................5

CONFIGURATION AND SETUP ................................................................................................ 5

3.1 3.2 3.3 3.4 3.5 3.6 STORAGE CONFIGURATION AND SETUP ................................................................................................5 NETAPP STORAGE CONFIGURATIONS FOR ORACLE VM...................................................................15 IMPORTING VIRTUAL MACHINES CREATED ON SHARED STORAGE TO ORACLE VM MANAGER37 EXAMPLE OF LIVE MIGRATION WITH SHARED STORAGE..................................................................40 USING NETAPP THIN-PROVISIONING AND DEDUPLICATION..............................................................45 USING SNAPSHOT AND FLEXCLONE .....................................................................................................49

4 CASE STUDYORACLE VM IN A DATA CENTER ENVIRONMENT WITH NETAPP STORAGE .................................................................................................................................. 52 5 6 7 CONCLUSION .......................................................................................................................... 53 ACKNOWLEDGEMENTS......................................................................................................... 53 CAVEATS ................................................................................................................................. 53

Oracle VM and NetApp StorageBest Practices Guide

1
1.1

INTRODUCTION
PURPOSE

This technical report provides the best practices and step-by-step guide for deploying Oracle VM virtualization software with shared storage solutions from NetApp.

1.2

SCOPE

The scope of this TR is focussed on the successful deployment of Oracle VM virtualization software with NetApp storage (NFS, iSCSI and FC SAN). It does not cover the comprehensive list of features available with Oracle VM virtualization software and cannot be used as replacement of Oracle VMs User or Administrative manuals.

1.3

INTENDED AUDIENCE AND ASSUMPTIONS

This document is intended for system and storage administrators, product management, and IT infrastructure Managers familiar with concepts pertaining to Oracle VM Server v2.1 and NetApp Data ONTAP 7G.

2
2.1

BACKGROUND
ADVANTAGES OF VIRTUALIZATION

Virtualization is playing a key role in the ongoing transformation of the organizations' IT departments. IT needs to respond rapidly to ever-changing business demands. To do so, an extremely flexible IT infrastructure is needed. This is where virtualization steps in. Oracles virtualization strategy is an extension of the Oracle Grid strategy. Oracle Grid seamlessly leverages the latest advancements in virtualization to further enhance the value of Grid infrastructure. It enables customers to dynamically share resources such as servers, storage and network among different enterprise applications. By doing so, virtualization is fast becoming the key enabler for a dynamic IT. With the advent of the new generations of commodity servers supporting higher processor core counts and more powerful processors, servers deployed in enterprise data centers are well below optimal levels. This under-utilization symptom represents a major opportunity for improvement. The combination of server and storage virtualization deployed in the Oracle Grid environment increase both server and storage utilization to maximize data center efficiency, and at the same time lower cost of infrastructure and operations. Advanced server and storage virtualization significantly add value to an existing Oracle Grid infrastructure by improving management efficiency and delivering higher Service Level Agreements (SLA). This new generation of Oracle Grid will increasingly utilize the advanced server virtualization and the latest storage virtualization technologies to: Increase server consolidation Reduce power and space requirements Reduce complexity Achieve greater software isolation and platform uniformity Provide better support for legacy applications Simplify data management tasks Provide capacity on demand

Oracle VM and NetApp StorageBest Practices Guide

2.2

ORACLE VM FOR ENTERPRISE READY VIRTUALIZATION

In November 2007, Oracle introduced Oracle VMa Xen architecture based server virtualization software that fully supports both Oracle and non-Oracle applications. Oracle VM offers scalable, highly efficient, and low-cost server virtualization. Consisting of open source server software and an integrated browser-based management console, Oracle VM provides an easy-to-use GUI for creating and managing virtual server pools running on x86 and x86-64-based systems across an enterprise. Users can create and manage virtual machines (VMs) that exist on the same physical server but behave like independent physical servers. Each virtual machine created with Oracle VM has its own virtual CPUs, network interfaces, storage, and operating system. With Oracle VM, users have an easy-to-use browser based tool for creating, cloning, sharing, configuring, booting, and migrating VMs. Oracle VM is fully server-centric and designed for data center use. With Oracle VM, it is easy to load balance to make sure that resources are fully utilized, and move live virtual machines from one physical server to another without reconfiguration or experiencing downtime. A number of versions of Linux and Windows are supported as guest operating systems on either 32-bit or 64-bit server platforms. Oracle VM uses native Linux device drivers so that users do not have to wait for the latest hardware to be supported by the virtualization solution layer. Oracle VM supports two types of virtual machines: Hardware virtualized: The guest operating system does not need to be modified. It is available only on Intel VT and AMD SVM CPUs. Para virtualized: The guest operating system is recompiled for the virtual environment for optimal performance.

Oracle VM consists of two components: Oracle VM Manager: Provides a standard Application Development Framework (ADF) web application to manage Oracle VM-based virtual machines. It also provides an API for Oracle VM Server. Oracle VM Server: Provides a virtualization environment designed to provide a self-contained, secure, server based platform for running virtualized guests. Oracle VM Agent is included for communication with Oracle VM Manager.

Figure 1) Oracle VM architecture.

Oracle VM and NetApp StorageBest Practices Guide

2.3

NETAPP STORAGE SOLUTIONS FOR ENTERPRISE READY VIRTUALIZATION

Server virtualization is only one half of an Oracle VM Infrastructure. Virtualized storage is required to complete an Oracle VM environmentin fact, the benefits of server virtualization are fully realized only when deployed with a shared storage solution. Unified storage solutions from NetApp perfectly complement the manageability, utilization, and cost-saving benefits of Oracle VM. NetApp solutions enable powerful thin provisioning, simplified data management, scalable and consistent I/O performance for all IT protocols across NAS (NFS) and SAN (Fibre Channel and iSCSI) in a single pool of storage. Key benefits and features include: Supports SAN (Fibre Channel and iSCSI), or NAS Scales non-disruptively to 100s of TB Easily installed, configured, managed, and maintained Rapid backup and recovery with zero penalty Snapshot copies Simple, cost effective replication for disaster recovery Instant space efficient data clones for provisioning and testing Dynamically expand and contract storage volumes as needed Data deduplication to reduce capacity requirements

NetApp storage solutions offers these powerful data management and data protection capabilities, which allows the Oracle VM to lower costs while meeting their capacity, utilization, and performance requirements. MultiStore (a licensable feature) is another storage virtualization approach provided by NetApp, which perfectly complements the server consolidation achieved through Oracle VM. MultiStore subdivides a NetApp physical storage system into multiple logical domains or virtual storage server partitions known as vFilers, each with its own unique identity, management, and IP address. Diverse applications running on different virtual machines consolidated into the same physical server and common storage systems can be isolated and secured by creating separate vFilers to store the application data. MultiStore also enables vFilers to transparently migrate to different physical systems without requiring reconfiguration of the client application servers and the mount points used for accessing data. For more details regarding NetApp MultiStore, see: http://media.netapp.com/documents/tr-3462.pdf

3
3.1

CONFIGURATION AND SETUP

STORAGE CONFIGURATION AND SETUP

This section of the TR provides a high level overview of the NetApp components and features that should be considered when deploying Oracle VM server virtualization on NetApp storage solutions. 3.1.1 NETAPP SHARED STORAGE SYSTEM CONFIGURATION

In an Oracle VM environment, the availability and performance of the shared storage infrastructure is more critical than that of the individual servers running the virtualized server environment. It is therefore vital to consider the required level of availability and performance when selecting and designing the storage solution for the virtualized server environment. NetApp offers a comprehensive set of software and hardware solutions to address the most stringent requirements for availability and performance of large, scalable Oracle VM environments. ACTIVE-ACTIVE NETAPP CONTROLLERS When designing a shared storage environment for Oracle VM it is essential that the solution be highly available. NetApp has an active-active controller design to provide data availability in business-critical environments like Oracle VM. Active-Active controllers provide simple, automatic, and transparent failover to deliver enterprise class availability. Providing the highest level of availability of the shared storage is critical since all servers depend on it. For more details, see http://www.netapp.com/us/products/platform-os/active-active.html.

Oracle VM and NetApp StorageBest Practices Guide

MULTIPATH HA Multipath HA storage configuration further enhances the resiliency and performance of activeactive controller configurations. Although cluster failover software provides high availability by providing fault tolerance in the event of controller failure, storage triggered events often result in unneeded failovers or prevent successful takeovers. Multipath HA storage enhances storage resiliency by reducing unnecessary takeover by a partner node due to a storage fault, thus improving overall system availability and promoting higher performance consistency. Multipath HA provides added protection against various storage faults, including HBA or port failure, controller-toshelf cable failure, shelf module failure, dual inter-shelf cable failure, and secondary path failure. Multipath HA helps provide consistent performance in active-active configurations by providing larger aggregate storage loop bandwidth. For more details, see http://media.netapp.com/documents/tr-3437.pdf RAID DATA PROTECTION A challenge of any server virtualization effort is the increased risk of the consolidated platform failure. As physical servers are converted to virtual machines and multiple virtual machines are consolidated into a single physical server, the impact of a failure to the single platform could be catastrophic. When focusing on storage availability, many levels of redundancy are available for deployments, including purchasing physical servers with multiple storage interconnects or HBAs, deploying redundant storage networking and network paths, and leveraging storage arrays with redundant controllers. A deployed storage design that meets all of these criteria can be considered to eliminate all single points of failure. The reality is that data protection requirements in a virtual infrastructure (like Oracle VM) are greater than those in a traditional physical server infrastructure. Data protection is a paramount feature of shared storage devices. RAID-DP is an advanced RAID technology that is provided as the default RAID level on all NetApp storage systems. RAID-DP protects against the simultaneous loss of two drives in a single RAID group. It is very economical to deploy; the overhead with default RAID groups is a mere 12.5%. This level of resiliency and storage efficiency makes data residing on RAID-DP safer than data residing on RAID 5 and more cost effective than RAID 10. NetApp recommends using RAID-DP on all RAID groups that store Oracle VM data. For more information about NetApps deduplication technology, see http://media.netapp.com/documents/tr3505.pdf REMOTE LAN MANAGEMENT CARD The Remote LAN Management (RLM) card provides a secure out-of-band access to the storage controllers, which can be used regardless of the state of the controllers. The RLM offers a number of remote management capabilities for NetApp controllers including remote access, monitoring, troubleshooting, logging, and alerting features. The RLM also extends the AutoSupport capabilities of the NetApp controllers by sending alerts or "down-filer" notifications via an AutoSupport message when the controller goes down, regardless of whether the controller can send AutoSupport messages. These AutoSupport messages also provide proactive alerts to NetApp to help provide faster service. For more details on RLM, see http://now.netapp.com/NOW/download/tools/rlm_fw/info.shtml TOP RESILIENCY BEST PRACTICES NetApp recommends the following configuration options for best-in-class resiliency: Use RAID-DP, the NetApp high-performance implementation of RAID 6, for better data protection. Use multipath HA with active-active storage configurations to improve overall system availability as well as promote higher performance consistency. Use the default RAID group size (16) when creating aggregates. Allow Data ONTAP to select the disks automatically when creating aggregates or volumes. Use the latest storage controller, shelf, and disk firmware, and Data ONTAP general deployment release available from the NOW (NetApp on the Web) website. Maintain at least two hot spares for each type of disk drive in the storage system to take advantage of Maintenance Center.

Oracle VM and NetApp StorageBest Practices Guide

Maintenance Center (MC) software is a part of the NetApp suite of proactive, self-healing storage resiliency tools. MC provides configurable in-place disk drive diagnostics to determine the health of suspect disk drives. If Data ONTAP disk health monitoring determines that a disk drive has surpassed an error threshold, Rapid RAID Recovery is initiated to a hot spare. Afterward, the suspect disk can be placed into MC, where it undergoes a series of diagnostic tests. Maintenance Center consists of Storage Health Monitor (SHM), NetApp Health Triggers, and NetApp Drive Self Tests software to promote drive self-healing and preventive or corrective maintenance. Do not put SAN LUNs or user data into the root volume. Replicate data with SnapMirror or SnapVault for disaster recovery (DR) protection. Replicate to remote locations to increase data protection levels. Use an active-active storage controller configuration (clustered failover) to eliminate single points of failure (SPOFs). Deploy SyncMirror for the highest level of storage resiliency.

For more information on storage resiliency, refer to the following TRs: NetApp TR 3437: Storage Best Practices and Resiliency Guide NetApp TR 3450: Active-Active Controller Overview and Best Practices Guidelines STORAGE NETWORKING CONSIDERATIONS

3.1.2

When designing a network infrastructure (FC or IP), it should have no single point of failure. A highly available solution includes having two or more FC or IP network switches, two or more HBAs or network interface cards (NICs) per host and two or more target ports or NICs per storage controller. In addition, if using Fibre Channel, two fabrics are required to have a truly redundant architecture. For more information on designing and deploying a FC/iSCSI solution, see the FC/iSCSI Configuration Guide: http://now.netapp.com/NOW/knowledge/docs/san/fcp_iscsi_config/QuickRef/fc_iscsi_config_guide. pdf 3.1.2.1 FIBRE CHANNEL STORAGE NETWORKING NetApp recommends that the storage controllers have two or more target ports configured to two separate fabrics to make sure that there are multiple paths available to the Oracle VM Servers. In addition, the best practice is to have at least two FC HBA ports available for storage connectivity paths on the Oracle VM Server. FIBRE CHANNEL MULTIPATHING OPTION NetApp clustered storage systems have an option known as cluster failover mode (cfmode) that defines how Fibre Channel ports behave during failover in an active-active configuration. Selecting the right cfmode is critical to ensuring your LUNs are accessible and optimizing your storage system's performance in the event of a failover. If you are deploying storage solutions that provide storage for an Oracle VM environment, NetApp strongly recommends that the cfmode be set to Single System Image (SSI) as it provides LUNs accessibility across all storage ports. NetApp also strongly recommends using Data ONTAP version 7.3 or higher. To verify the current cfmode using the NetApp console, complete the following steps: Step 1 2 Action Log into the NetApp console via either SSH, Telnet, or Console connection. Type: fcp show cfmode If cfmode needs to be changed to SSI, type: priv set advanced

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Type: fcp set cfmode <mode type> For more information about the different cfmodes available and the impact of changing a cfmode, see section 8 in the Data ONTAP Block Management Guide. IP STORAGE NETWORKING

3.1.2.2

NETAPP VIRTUAL INTERFACES A virtual network interface (VIF) is a mechanism that supports aggregation of network interfaces into one logical interface unit. Once created, a VIF is indistinguishable from a physical network interface. VIFs are used to provide fault tolerance of the network connection and in some cases higher throughput to the storage device. Multimode VIFs are compliant with IEEE 802.3ad. In a multimode VIF, all physical connections in the VIF are simultaneously active and can carry traffic. This mode requires that all of the interfaces are connected to a switch that supports trunking or aggregation over multiple port connections. The switch must be configured to understand that all of the port connections share a common MAC address and are part of a single logical interface. The following figure is an example of a multimode VIF. Interfaces e0, e1, e2, and e3 are part of the MultiTrunk1 multimode VIF. All four interfaces in the MultiTrunk1 multimode VIF are active.

Figure 2) Multimode VIF.

In a single-mode VIF, only one of the physical connections is active at a time. If the storage controller detects a fault in the active connection, a standby connection is activated. No configuration is necessary on the switch to use a single-mode VIF, and the physical interfaces that constitute the VIF do not have to connect to the same switch. Note: IP load balancing is not supported on single-mode VIFs. Figure 3 illustrates an example of a single-mode VIF. In this figure, e0 and e1 are part of the SingleTrunk1 single-mode VIF. If the active interface, e0, fails, the standby e1 interface takes over and maintains the connection to the switch.

Oracle VM and NetApp StorageBest Practices Guide

Figure 3) Single-mode VIF.

It is also possible to create second-level single or multimode VIFs. By using second-level VIFs it is possible to take advantage of both the link aggregation features of a multimode VIF and the failover capability of a single-mode VIF. In the configuration shown in figure 4, two multimode VIFs are created, each connected to a different switch. A single-mode VIF is then created composed of the two multimode VIFs. In normal operation, traffic flows over only one of the multimode VIFs; but in the event of an interface or switch failure, the storage controller moves the network traffic to the other multimode VIF. For more information on the different types of VIFs, see the Data ONTAP Network Management Guide available on NOW.

Switch1

Switch2

SingleTrunk1

e0

e1

e2 e3 MultiTrunk2

MultiTrunk1

Figure 4) Second-level single or multimode VIFs.

ISCSI TRAFFIC SECURITY NetApp storage controllers also allow the restriction of the iSCSI protocol to specific interfaces and/or VLAN tags. These simple configuration settings have an enormous effect on the security and availability of IP-based host disks. ETHERNET SWITCH CONNECTIVITY An IP storage infrastructure provides the flexibility to connect to storage in different configurations, depending on the needs of the environment. A basic architecture can provide a single nonredundant link to a physical disk, suitable for storing ISO images, or various backups. A redundant architecture, suitable for most production environments, has multiple links, providing failover for switches and network interfaces. Link-aggregated and load-balanced environments make use of multiple switches and interfaces simultaneously to provide failover and additional overall throughput

Oracle VM and NetApp StorageBest Practices Guide

for the environment. Some ethernet switch models support stacking, where multiple switches are linked by a high-speed connection to allow greater bandwidth between individual switches. A subset of the stackable switch models support cross-stack EtherChannel trunks, where interfaces on different physical switches in the stack are combined into an 802.3ad EtherChannel trunk that spans the stack. The advantage of cross-stack EtherChannel trunks is that they can eliminate the need for additional passive links that are accessed only during failure scenarios in some configurations. 3.1.3 ORACLE VM SHARED STORAGE OPTIONS

The guest operating systems running inside the virtual machines hosted on the OVM Server view the disk storage allocated while creating the Virtual Machine (for example. using the virt-install command) as a single virtual hard disk, analogous to a physical hard disk. This appears as hda and can be partitioned and managed in the guest exactly as if it were regular physical hardware. Three types of shared storage options are available for an OVM Server configuration: Network Attached Storage using NFS iSCSI SAN

Fibre channel SAN NetApp NFS shared storage gives unmatched flexibility to a virtual infrastructure deployed with Oracle VM Server. The files corresponding to the virtual disks of the virtual machines are thin provisioned by default and also deduplicated (if deduplication license is enabled). This leads to a very high utilization of storage as well as a drastic reduction in the total amount of storage required. Oracle VM Server supports either a software-based iSCSI initiator or a supported iSCSI HBA. Similarly it supports Fibre Channel SANs using the supported FC HBA. The iSCSI or FC shared storage of OVM Server requires configuration of the Oracle Cluster File System (OCFS2) for use as a shared virtual disk for migration. Note: Each virtual disk file has its own I/O queue to the NetApp storage system for all of these storage solutions. Figures 5-8 illustrates the architectural diagrams to provide an overview of these storage options.

10

Oracle VM and NetApp StorageBest Practices Guide

Figure 5) OVM server with NFS shared storage.

11

Oracle VM and NetApp StorageBest Practices Guide

Figure 6) OVM server with iSCSI shared storage.

12

Oracle VM and NetApp StorageBest Practices Guide

Figure 7) OVM server with iSCSI shared storage (iSCSI HBA).

13

Oracle VM and NetApp StorageBest Practices Guide

Figure 8) OVM server with FC shared storage.

Oracle VM Server Repositories: A repository can be used for live migration of guests and also for local storage in an Oracle VM Server. A list of the Oracle VM Server repositories is available at /etc/ovs/repositories. A configuration file for mounting repositories is available at /etc/ovs/repositories.options. Repositories can be added by using the ovs-makerepo script, and can be removed using the ovsofflinerepo d script. Repositories are managed by Oracle VM Agent. Sections 3.2.2section 3.2.4 describes how to create OVM Server repositories for various shared storage options like NFS, iSCSI and FC. Section 3.3, describes how the virtual machines created on these repositories can be easily imported to the OVM Manager.

14

Oracle VM and NetApp StorageBest Practices Guide

3.2

NETAPP STORAGE CONFIGURATIONS FOR ORACLE VM

This section covers the best practices for configuring and provisioning the various available shared storage options (NFS, iSCSI, FC) for the OVM Server with a NetApp storage system. 3.2.1 CONFIGURING AN AGGREGATE

To configure an aggregate, complete the following steps.

NetApp recommends using a separate aggregate for the Oracle VM Server. Step Action From NetApp FilerView, click Aggregate> Add. Click Next. Result: The Aggregate Wizard appears.

Enter an appropriate aggregate name and select the Double Parity check box. (RAID-DP is the recommended RAID level for any NetApp storage). Click Next.

15

Oracle VM and NetApp StorageBest Practices Guide

Step

Action In the RAID Parameters window, select the default RAID Group size to 16. Click Next.

In the Disk Selection Method window, select the Automatic option and click Next.

In the Disk Type window, from the drop-down, select Any Type and click Next.

16

Oracle VM and NetApp StorageBest Practices Guide

Step

Action In the Number of Disks window, from the drop-down list, select 3. You require at least three disks.

Click Commit to create the aggregate.

17

Oracle VM and NetApp StorageBest Practices Guide

3.2.2 CONFIGURING SHARED STORAGE USING NFS NetApp recommends creating a separate volume for the for the aggregate created in section 3.2.1 for exporting into the OVM Server as the NFS shared storage for virtual machines. Step Action From NetApp FilerView, click Volume> Add. Result: The Volume Wizard appears.

Select the Flexible option and click Next.

18

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Enter an appropriate name for the volume and click Next.

Select the aggregate created in section 3.2.1 as the Containing Aggregate and select Volume as the Space Guarantee. Click Next.

19

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Set the size of the volume and click Next. For Snapshot Reserve, NetApp recommends that all volumes be configured with 0% and that the default Snapshot schedule be disabled.

Click Commit to create the volume. From NetApp FilerView, click NFS > Manage Exports. Click the /vol/OVM_NFS_STORAGE volume created. Click Next. Result: NFS Export Wizard appears.

20

Oracle VM and NetApp StorageBest Practices Guide

Step

Action The OVM Server is required to have root access to the volume exported so that it can manage the volume as shared storage for the virtual machines. In the NFS Export Wizard, select the Root Access check box and click Next.

Accept the defaults, and click Next.

21

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Accept the defaults, and click Next.

10

In the Root Access window, select the hosts and click Add. Click Next.

11

Enter the IP address (or hostname) of the OVM Server and click OK. If all OVM Server hosts on a particular subnet should be given access to the NFS storage, then enter the subnet (in CIDR format). Otherwise enter the IP address (or hostname) of each OVM Server.

12

22

Oracle VM and NetApp StorageBest Practices Guide

Step

Action In the NFS Export Wizard-Security window, select Unix Style and click Next.

12

Click Commit for changes to take effect.

13

23

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Log in to the OVM Server as root.

14

Create a directory path in OVM Server to mount the NFS export by typing: 15 mkdir p /OVS/netapp/nfsdatastore Add the following entry in the /etc/fstab file to mount the exported NFS path to the directory created above: 10.73.69.199:/vol/OVM NFS STORAGE /OVS/netapp/nfsdatastore nfs rw,bg,intr,hard,timeo=600,wsize=32768,rsize=32768,nfsvers=3,tcp 0 0

16

24

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Mount the directory path to access the NFS storage: mount /OVS/netapp/nfsdatastore

17

Now this mounted NFS path can be used as the shared storage for hosting virtual machines. For example, while creating virtual machines using the virt-install command, this mounted NFS path can be specified when prompted: What would you like to use as the disk (path)?

Refer to the following step-by-step guide from Oracle to learn more about creating virtual machines using the virt-install command:

http://www.oracle.com/technology/obe/ovm/ovmel5pvmdb11gr1/ovmel5pvmdb11gr1.htsm
Note: Instead of mounting the NFS share directly, it can also be mounted as an OVM Server repository. This is the recommended method. Use the ovs-makerepo script for this purpose.

25

Oracle VM and NetApp StorageBest Practices Guide

The newly created repository is shown in the etc/ovs/repositories file and is available for use. The virtual machines are created using the virt-install command shown earlier in the running_pool directory inside the repository. Virtual machines created inside the running_pool directory of the repository (or in the /OVS/running_pool directory) can be directly imported to the OVM Manager (see section 3.3).

26

Oracle VM and NetApp StorageBest Practices Guide

3.2.3

CONFIGURING SHARED STORAGE USING ISCSI

This section describes the provisioning of iSCSI shared storage to the OVM Server. Unlike, NFS storage, the iSCSI shared storage of an OVM Server requires configuring OCFS2 for use as a shared virtual disk for migration. Note: Only the software initiator based iSCSI storage provisioning is described below. Step Action Install the software iSCSI initiator in the OVM Server. Start the iSCSI service. Make a note of the initiator ID.

Create a flexible volume where the iSCSI LUN is to be created and click Next.

27

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Enter an appropriate name for the volume and click Next.

Choose the aggregate containing the volume and click Next.

28

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Set the size of the volume and click Next. For Snapshot Reserve, NetApp recommends that all volumes be configured with 0% and that the default Snapshot schedule be disabled.

6 7

Click Commit to create the volume. Open the LUN Wizard to create the iSCSI LUN. Result: The LUN Wizard Welcome screen appears. In this example, we created a LUN named LUN1 inside the OVM_ISCSI volume. Specify the size of the LUN. Choose LUN Protocol type as Linux. Clear the Space Reserved check box. This is to enable NetApp Thin Provisioning of LUNs. For details, see section 3.5. Click Next.

29

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Create and add a new initiator group and click Next.

Add initiators corresponding to each OVM Server in the server pool to the new initiator group so that they can use this LUN as shared storage. These initiators are obtained from OVM Servers as shown in step 1.

10

30

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Specify a unique LUN ID and click Next.

11

Click Commit to commit changes to create the LUN.

12

31

Oracle VM and NetApp StorageBest Practices Guide

Step

Action From the OVM Server, discover the iSCSI LUN.

13

Result: The new LUN is visible after restarting the iSCSI services.(/dev/sdc)

32

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Result: The new device is now available for use.

Format the new device using fdisk and create a partition.

14

33

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Result: The new partition is now listed.

As stated previously, unlike NFS, for iSCSI LUNs, OCFS2 must be configured so that the shared iSCSI storage can be used in live migration. Create an OCFS2 configuration filed named /etc/ocfs2/cluster.conf in each node of the server pool.

15

Configure and start the OCFS2 Cluster service: 16 service o2cb status service o2cb load service o2cb online service o2cb start service o2cb configure NetApp recommends the following values: O2CB_HEARTBEAT_THRESHOLD=81, O2CB_IDLE_TIMEOUT_MS=60000, O2CB_KEEPALIVE_DELAY_MS=4000, O2CB_RECONNECT_DELAY_MS=4000

34

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Format the shared virtual disk from any server in the cluster.

17

Mount the shared virtual disk from all of the servers in the cluster, by typing: mount /dev/sdb1 /OVS/netapp/iscsi t ocfs2 The /etc/fstab file can also be modified to mount the shared virtual disk at boot time.

18

35

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Instead of directly mounting the newly created device as OCFS2 file system, you can also create an OVM Server repository out of the device using the ovs-makerepo script. This way the virtual machines created on the repository can be directly imported to the Oracle VM Manager as shown in section 3.3

19

3.2.4 CONFIGURING SHARED STORAGE USING FIBRE CHANNEL The process of provisioning FC shared storage from a NetApp storage system to an Oracle VM Server is very similar to that of iSCSI shared storage. And similar to iSCSI shared storage, FC shared storage too requires configuration of OCFS2. The only difference in the process of FC shared storage provisioning is that when creating initiator groups, instead of providing ISCSI Qualified Name (IQN), provide the FCP World Wide Port Names (WWPN).

36

Oracle VM and NetApp StorageBest Practices Guide

3.3

IMPORTING VIRTUAL MACHINES CREATED ON SHARED STORAGE TO ORACLE VM MANAGER

This section describes how a virtual machine created on shared storage can be managed through the Oracle VM Manager. NFS is used as an example of shared storage. Step Action Consider a Virtual Machine named vm1 created on NFS shared storage as shown in section 3.2.2.

Install and Setup the Oracle VM Manager 2 An Oracle Enterprise Linux (OEL) machine is used to install the OVM Manager. OVM manager can be accessed from: http://<IP Address of OVM Manager>:8888/OVS Since OVM manager searches the virtual machines inside /OVS/running_pool or the /OVS/<repository>/running_pool directories only, the vm1 virtual machine created on NFS shared storage mounted on /OVS/netapp/nfsdatastore directory cannot directly be imported in to OVM Manager. o o One way to avoid this is to mount the shared storage in the /OVS/running_pool directory itself so that OVM Manager can see the virtual machines. Recommended way is to mount the NFS shared storage as an OVM Server repository; instead of mounting it directly. This way. the virtual machines reside inside the /OVS/<Repository>/running_pool directory and can directly be imported to the OVM Manager. The only additional task that needs to be performed in this method is to create a vm.cfg file inside the /OVS/<repository>/running_pool directory, as shown in the steps below. Otherwise, follow the procedure given below to import the virtual machine vm1 created in /OVS/netapp/nfsdatastore directory into the OVM Manager.

37

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Create a running_pool directory in /OVS if it is not there yet.

Create a directory for the virtual machine vm1 inside OVS/running_pool. In the directory just created: Create a soft link for the virtual machine image file in the shared storage path. Create a config file called vm.cfg.

38

Oracle VM and NetApp StorageBest Practices Guide

Step

Action The basic contents of the vm.cfg file can be as viewed.

In the OVM Manager, click Resources > Virtual Machine Images >Import >Select from Server Pool (Discover and Register). From the drop-down list, select the Server Pool name and Virtual Machine Image (vm1) and click Confirm.

39

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Result: Now the virtual machine vm1 can be managed and controlled through the OVM Manager. In section 3.4, this setup is further extended for live migration of virtual machines with NFS shared storage.

3.4

EXAMPLE OF LIVE MIGRATION WITH SHARED STORAGE

For live migration of virtual machines between two OVM Servers, the servers need to share common storage where the virtual machines are stored. In this section, NFS is used as the shared storage between the two OVM Servers. 1. Add two OVM Servers in the OVM Manager Server Pool. ovmserver10.73.69.106 ovmserver210.73.69.105

Both these OVM Serversovmserver and ovmserver2 share common NFS storage for storing the virtual machines. (10.73.68.199:/vol/OVM_NFS_STORAGE in the example below) 2. Mount on ovmserver.

40

Oracle VM and NetApp StorageBest Practices Guide

3.

Mount on ovmserver2.

Currently the virtual machine vm1 to be moved is running on ovmserver. To successfully migrate through OVM Manager, ovmserver2 must have corresponding directories in /OVS/running_pool.

41

Oracle VM and NetApp StorageBest Practices Guide

LIVE MIGRATION BY OVM MANAGER 1. Select the VM to be migrated. 2. From More Action, select Live Migration from the drop down list.

3.

Select the destination server.

4.

Click Confirm.

42

Oracle VM and NetApp StorageBest Practices Guide

The virtual machine is moved to the destination OVM Server. (Intermediate State is migrating).

5.

Check in the ovmserver2 to determine if the migration completed successfully.

43

Oracle VM and NetApp StorageBest Practices Guide

LIVE MIGRATION USING CLI This section describes how to migrate the virtual machine through CLI instead of OVM Manager. 1. Run: xm migrate <vmname> <destination-OVM-Server> --live

2.

Confirm if the virtual machine vm1 successfully migrated to ovmserver.

Migration done through the CLI is also visible to the OVM Manager. The host server location of the VM will change automatically to the new server once the migration triggered through the CLI is complete.

44

Oracle VM and NetApp StorageBest Practices Guide

3.5

USING NETAPP THIN-PROVISIONING AND DEDUPLICATION

DEDUPLICATION One of the most important features of Oracle VM is its ability to rapidly deploy new virtual machines from stored VM templates. A VM template includes a VM configuration file and one or more virtual disk files, which include an operating system, common applications, and patch files or system updates. Deploying from templates saves administrative time by copying the configuration and virtual disk files and registering this second copy as an independent VM. By design, this process introduces duplicate data for each new VM deployed. Figure 9 shows an example of typical storage consumption in a normal Oracle VM deployment:

45

Oracle VM and NetApp StorageBest Practices Guide

Figure 9) Storage consumption in an Oracle VM deployment with Traditional Storage Array

Deduplication technology from NetApp assists Oracle VM deployments to eliminate duplicate data in their environment, enabling greater storage utilization on the production environment. NetApp deduplication technology enables multiple virtual machines in an Oracle VM environment to share the same physical blocks on storage. Deduplication can be seamlessly introduced into a virtual infrastructure without having to make any changes to Oracle VM administration, practices, or tasks. Figure 10 shows an example of the impact of deduplication on storage consumption in an Oracle VM deployment. Deduplication is enabled on a volume, and the amount of data deduplication realized is based on the commonality of the data stored in a deduplication-enabled volume. For the largest storage savings, NetApp recommends grouping similar operating systems and similar applications into the same deduplication-enabled volume. DEDUPLICATION CONSIDERATIONS WITH ISCSI AND FC LUNS Storage savings are apparent if deduplication is enabled while provisioning LUNs. However, the default behavior of a LUN is to reserve an amount of storage equal to the provisioned LUN. This design means that although the storage array reduces the total amount of capacity consumed, any gains made with deduplication are, for the most part, unrecognizable, because the space reserved for LUNs is not reduced.

46

Oracle VM and NetApp StorageBest Practices Guide

Figure 10) Storage consumption in an Oracle VM deployment with NetApp FAS

To benefit from the storage savings of deduplication with LUNs, a LUN must be thin provisioned. NetApp Thin Provisioning is covered later in this section. In addition, although deduplication reduces the amount of consumed storage, this benefit is not seen directly by the Oracle VM administrative team, because their view of the storage is at a LUN level, and LUNs always represent their provisioned capacity, whether they are traditional or thin provisioned. Deduplication considerations with NFS Unlike with LUNs, when deduplication is enabled with NFS, the storage savings are both immediately available and also recognizable by the Oracle VM administrative team. No special considerations are required for its usage. For NetApp deduplication best practices, including scheduling and performance considerations, see http://media.netapp.com/documents/tr-3505.pdf NETAPP THIN PROVISIONING Oracle VM provides an excellent means to increase the hardware utilization of physical servers. By increasing hardware utilization the amount of hardware in a data center can be reduced, thus lowering the cost of data center operations. In a typical Oracle VM environment the process of migrating physical servers to virtual machines does not reduce the amount of data stored or the amount of storage provisioned. By default, server virtualization does not have any impact on improving storage utilization (and in many cases, it may have the opposite effect). In traditional storage provisioning, storage is already allocated and assigned to a server, or in the case of an Oracle VM, a virtual machine. It is also common practice for server administrators to overprovision storage to avoid running out of storage and the associated application downtime when expanding the provisioned storage. Although no system can be run at 100% storage utilization, there are methods of storage virtualization that allow administrators to oversubscribe storage in the same manner as with server resources (such as CPU, memory, networking, and so on). This form of storage virtualization is referred to as thin provisioning.

47

Oracle VM and NetApp StorageBest Practices Guide

Thin provisioning provides storage on demand; traditional provisioning pre-allocates storage. The value of thin-provisioned storage is that storage is treated as a shared resource pool and is consumed only as each individual VM requires it. This sharing increases the total utilization rate of storage by eliminating the unused but provisioned areas of storage that are associated with traditional storage. The drawback of thin provisioning and oversubscribing storage is that (without the addition of physical storage) if every VM requires its maximum possible storage at the same time, there will not be enough storage to satisfy the requests. It is important to note that the benefits of NetApp Thin Provisioning can be realized across all kinds of shared storage repositories in an Oracle VM environment this is, NFS, iSCSI or FC. If using NFS storage, NetApp flexible volumes are thin provisioned by default. No extra configuration steps are necessary. However, when using iSCSI or FC storage, make sure that the Space Reserved check box in the LUN wizard is not selected (see section 4.2.3). NetApp recommends that when enabling NetApp thin provisioning, to configure the storage management policies on the volumes that contain the thin-provisioned LUNs. These policies aid in providing the thinprovisioned LUNs with storage capacity as they require it. The important policies areautomatic sizing of a volume, automatic Snapshot copy deletion, and LUN fractional reserve. Volume Auto Size is a policy-based space management feature in Data ONTAP that allows a volume to grow in defined increments up to a predefined limit if the volume is nearly full. For Oracle VM environments, NetApp recommends setting this value to ON. Doing so requires setting the maximum volume and increment size options. To enable these options, do as follows: Step 1 2 Action Log in to the NetApp console. Set volume autosize policy by typing: vol autosize <vol-name> [-m <size>[k|m|g|t]] [-i <size>[k|m|g|t]] on

Snapshot Auto Delete is a policy-based space-management feature that automatically deletes the oldest Snapshot copies on a volume when that volume is nearly full. For Oracle VM environments, NetApp recommends setting this value to delete Snapshot copies at 5% of available space. In addition, you should set the volume option to have the system attempt to grow the volume before deleting Snapshot copies. To enable these options, do as follows: Step 1 2 Action Log in to the NetApp console. Set Snapshot autodelete policy by typing: snap autodelete <vol-name> commitment try trigger volume target_free_space 5 delete_order oldest_first Set volume autodelete policy by typing: 3 vol options <vol-name> try_first volume_grow

LUN Fractional Reserve is a policy that is required when you use NetApp Snapshot copies on volumes that contain Oracle VM LUNs. This policy defines the amount of additional space reserved to guarantee LUN writes if a volume becomes 100% full. For Oracle VM environments where Volume Auto Size and Snapshot Auto Delete are in use NetApp recommends setting this value to 0%. Otherwise, leave this setting at its default of 100%. To enable this option, do as follows:

48

Oracle VM and NetApp StorageBest Practices Guide

Step 1 2

Action Log in to the NetApp console. Set volume Snapshot fractional reserve by typing: vol options <vol-name> fractional_reserve 0

3.6

USING SNAPSHOT AND FLEXCLONE

NetApp Snapshot technology can be used to backup and restore the virtual machines stored on NFS, iSCSI or FC shared storage. Similarly NetApp FlexClone technology can be used to rapidly provision virtual machines. The following steps describes how to take Snapshot copies and then provision new virtual machines using FlexClone for a NFS shared storage repository. Step Action Consider an NFS volume from a NetApp FAS system has been mounted as a repository in the OVM Server. 1 10.73.68.199:/vol/OVM_NFS on /OVS/8C37E8B6199B4D559A2B5BC649E25820 type nfs (rw,addr=10.73.68.199) Take a Snapshot copy of the volume in the NetApp FAS system using FilerView or CLI. The Snapshot copy used in this example is named as OVMNFSSNAPSHOT. For recovering the virtual machines on this volume, you can mount this Snapshot copy and recover the individual VM images. Additionally the Snapshot technology can be seamlessly integrated with NetApp SnapMirror solution for providing disaster recovery solutions

49

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Create a FlexClone volume of this volume and provision new virtual machines from the base virtual machines already present in the parent volume (OVM_NFS).

Enter the name of the Snapshot copy from where the FlexClone volume will be created.

50

Oracle VM and NetApp StorageBest Practices Guide

Step

Action Click Commit to commit the changes to create the FlexClone volume.

Mount the newly created /vol/OVMNFSFLEX volume (FlexClone) on a server and delete the .xxxrepo directory. Unmount the volume from the server. Mount the FlexClone as an OVM Server repository using the ovs-makerepo script.

/usr/lib/ovs/ovs-makerepo 10.73.68.199:/vol/OVMNFSFLEX

1 FLEXCLONE

Before using this repository for provisioning new virtual machines, the following changes must be made: 1. Rename the virtual machines in the FlexClone volume. a. b. Change the name of the VM directory name inside /OVS/<repository>/running_pool Modify the vm.cfg file inside running_pool as follows: i. Change the VM name. 8 ii. Modify the disk path. iii. 2. 3. Delete the vif information, if any.

Create temporary directories in /OVS/running_pool corresponding to each new VM directory inside the FlexClone volume. Create soft links from the temporary directories to the vm.cfg file of the virtual machine directories inside the FlexClone volume.

Then the virtual machines can be imported to the OVM manager as shown in the section 3.3.

51

Oracle VM and NetApp StorageBest Practices Guide

CASE STUDYORACLE VM IN A DATA CENTER ENVIRONMENT WITH NETAPP STORAGE

Oracle On Demand, Oracles software as a service business, introduced server virtualization within Oracle On Demand Managed Services Grid using Oracle VM. In addition to the guaranteed consolidation benefits, the use of Oracle VM in the Oracle On Demand Data Center also provides additional benefits like capacity on demand, rapid provisioning, and high availability of virtual machines. NetApp storage solutions play a major role in achieving these benefits. Use of NetApp storage with Oracle VM in the Oracle On Demand environment delivers significant benefits like: Capacity on Demand: The hardware resources allocated to a particular application can be scaled without disruption. Rapid provisioning: Downtime resulting from hardware failures is nearly eliminated. Virtual machines can be restarted rapidly in the event of hardware failure. Decreased impact of necessary hardware maintenance: When maintenance is necessary, virtual machines can simply be migrated to another physical server, eliminating downtime.

NetApp storage when used with Oracle VM also helps guarantee that the transition to the virtualized environment from the physical environment is seamless and does not require any major process changes. That is, the strategies and processes used for backup and restore, disaster recovery, software upgrade, and so on, used in the data center does not go through a complete makeover because the underlying infrastructure has been transformed to a virtual environment. Some of the innovative technologies from NetApp that are exploited by Oracle in deploying Oracle VM in the Oracle On Demand data center are: FlexClone: NetApp FlexClone technology is used to rapidly provision virtual machines and storage for test and development environments in OOD. NetApp FlexClone technology creates true clonesinstantly replicated data volumes and data sets, without requiring additional storage space. For information on using FlexClone with Oracle VM, see section 3.6. Over provisioning of Storage through Thin Provisioning: For non-production environments, OOD is using NetApp Thin Provisioning technology to over provision storage to the virtual machines. For details on thin provisioning technology in the context of Oracle VM, see section 3.5. Deduplication: NetApp deduplication is successfully being used in the production environment. Both databases and operating systems, contained in the NetApp storage, undergo the deduplication process to achieve very high levels of storage savings. For more details about NetApp deduplication in Oracle VM environment, see section 3.5. Online backup/restore: Oracle On Demand online backup and restore strategy of Oracle VM virtual machine images is based on NetApp Snapshot technology. For more details about NetApp Snapshot technology in an Oracle VM environment, see section 3.6.

For more details about the Oracle VM implementation in Oracle On Demand, see the following white paper from Oracle: http://www.oracle.com/ondemand/collateral/virtualization-oracle-vm-wp.pdf

52

Oracle VM and NetApp StorageBest Practices Guide

CONCLUSION

With the introduction of Oracle VM, Oracle is now the only software vendor to combine the benefits of server clustering and server virtualization technologies, delivering integrated clustering, virtualization, storage, and management for Grid Computing. This report intends to provide detailed guidance on how best to implement Oracle VM Server Virtualization solutions on NetApp storage. NetApp has been on the forefront of solving complex business problems with its innovative technology and end-to-end solutions approach. This report is not intended to be a definitive implementation or solutions guide. Additional expertise may be required to solve specific deployments. Please contact your local NetApp sales representative to speak with one of our Oracle VM solutions experts. Please forward any errors, omissions, differences, new discoveries, and comments about this paper to preetom@netapp.com

ACKNOWLEDGEMENTS

Steven SchuettingerTechnical Alliance Manager, NetApp Lynne ThiemeSenior Product Manager, NetApp Uday ShetSenior Manager, NetApp Mani KannanReference Architect, NetApp Laurent SandroliniOracle Corp. Michele RestaOracle Corp. Deborah BerumenOracle Corp.

CAVEATS

These practices are only recommendations, not requirements. Not adhering to these recommendations does not affect the support from NetApp for your systems and solutions. Also, not all recommendations apply to every scenario. NetApp believes that customers will benefit from thinking through these recommendations before making any implementation decisions.

www.netapp.com

2008 NetApp. All rights reserved. Specifications are subject to change without notice. NetApp, the NetApp logo, Go further, faster, Data ONTAP, FlexClone, RAID-DP, SnapMirror, MultiStore, vFiler, NOW, SnapVault, SyncMirror, and FilerView, are trademarks or registered trademarks of NetApp, Inc. in the United States and/or other countries. Oracle is a registered trademark of Oracle Corporation. UNIX is a registered trademark of The Open Group. Linux is a registered trademark of Linus Torvalds. Windows is a registered trademark of Microsoft Corporation. Oracle, Oracle On Demand, and Oracle VM are registered trademarks of Oracle, Inc. Intel is a registered trademark of the Intel Corporation. All other brands or products are trademarks or registered trademarks of their respective holders and should be treated as such.

53

Oracle VM and NetApp StorageBest Practices Guide