MIGRATING FROM SINGLE INSTANCE TO RAC AND UPGRADING: A DATA GUARD SUCCESS STORY

Fuad Arshad @fuadar

Walgreen Company

Me, Myself & I

More than 15 years of Oracle experience Worked in textiles, telecom & retail/healthcare

Currently work for Walgreens in the Enterprise Shared Services area

Areas of Interest: backup & recovery, high availability, MAA, performance tuning

Socially Networkable

Twitter @fuadar

Linkedin http://www.linkedin.com/in/fuadar
G+ +Fuad Arshad Blog http://www.fuadarshad.com

To be My Walgreens for everyone in America, and the first choice for health and daily living.

$67B 2010 sales

Fortune: Most Admired Companies List for 17 years Fast Company: Named as 6th most innovative health care company for leadership in health care services

244,000 employees
7,800 locations

In 36th consecutive
year of record sales

Computerworld: CIO Tim Theriault named to 2012 Premier 100 IT Leaders list

Agenda

Problem Analysis Decision Strategy

Migration
Things to look out for!

Credits

Community Access

RAC Attack IOUG RAC SIG IOUG

Ask Questions

Blogs / Blogs /Blogs .

http://www.orana.info http://blogs.oracle.com http://www.oaktable.net

Freelists Oracle L Stack Overflow Quora

MOS Communities
Oracle Forums

The Problem
It was very early, and we were still like beta or alpha stage, and so we started receiving a ton of download. The server became overloaded, and that's when I realized that this had a huge market.

Shawn Fanning (Napster)

Paper #371

Our Problem

The application was growing and adding features Customer access slowdown was reported

Paper #371

Our Problem

System built in the early 2000s to handle the new

wave of technology

More work as newer features were rolled out Architecture & design not intended to handle the
rate of change

Database is always the bottleneck

Paper #371 10

Analysis

Active/Passive servers Oracle database version close to desupport Old OS version Old SAN infrastructure CPU and disk I/O issues
Result -----> IT WAS OLD
Paper #371 11

Analysis

Active/Passive Servers Oracle Database version close to desupport Old OS version Old SAN infrastructure

Result ---------------- IT WAS OLD

Paper #371 12

Paper #371

13

Decision

Lets migrate to RAC Can it be done yesterday? Can we do it with zero downtime? The application team doesnt want to change
anything
Paper #371 14

Strategy

Export/Import Transportable Table spaces Disk based replication GoldenGate Data Guard - The Choice
Paper #371 15

Pre-Migration

Build of servers Build of network RAC starter kits

Paper #371 16

Pre-Migration

The strategy was to build a RAC database from a single instance environment Oracle provided instructions on how to prep the single instance environment so that it is RAC-aware. This includes running catclust scripts and adding undo tablespaces for the number of nodes that the new cluster will have

Paper #371

17

Pre-Migration

The server builds need to have all the patches based on the OS that is being used

Oracle provides a variety of starter kits that can be used to validate the configuration
It is also important that the network be configured and available based on best practices 11.2.0.2 and above have added requirements like SCAN that need to be accounted for
Paper #371 18

Pre-Migration

RAC Check is important tool to ensure that the environment is built based on best practices

RAC Check can be downloaded at MOS # 1268927.1 and is updated frequently

OS settings including kernel parameters and patches CRS/Grid Infrastructure RDBMS memory and database parameters ASM parameters and statistics

Paper #371

19

Pre-Migration (Primary)
Data Guard setup steps need to be followed which include :

Enable supplemental logging & force logging on primary Add additional REDO threads (Based on # of RAC nodes) Add additional UNDO tablespaces (Based on # of RAC nodes) Backup the primary database Create a PFILE from the primary database Copy TNSNAMES and LISTENER files into a staging directory Configure the primary database for Data Guard
Paper #371 20

Pre-Migration

RAC Check is important tool to ensure that the environment is built based on best practices

RAC Check can be downloaded at MOS # 1268927.1 and is updated frequently

OS settings including kernel parameters and patches CRS/Grid Infrastructure RDBMS memory and database parameters ASM parameters and statistics

Paper #371

21

Pre-Migration (Standby RAC)

Data Guard setup steps need to be followed which include:
NFS mount or copy backup and auxiliary files Configure Oracle Net which includes setting up TNSNAMES and LISTENER for 11.2.0.2 (This will be done in the Grid infrastructure home) Modify the TNSNAMES.ORA to include the primary as well as the standby tns entries Create a tns entry for the listener and make sure it is referenced in the REMOTE_LISTENER (REMOTE_LISTENER specifies the global list of listeners in a RAC configuration) Make sure the listener alias referenced by the LOCAL_LISTENER initialization parameter is in each standby tnsnames.ora file Ensure that the standby server has the same software version and patches installed as the primary 22 Paper #371

Pre-Migration (Standby RAC, cont.)

Data Guard setup steps need to be followed which include:

Ensure that a password file is configured on both the primary and the standby Copy the PFILE to the $ORACLE_HOME/dbs in the standby node Add DB_UNIQUE_NAME and CLUSTER_DATABASE parameters in the PFILE Modify all the other parameters outlined in the white paper Connect to the standby and create an SPFILE in the ASM diskgroup Startup the proposed standby environment in NOMOUNT mode Duplicate the database using DUPLICATE TARGET DATABASE FOR STANDBY Setup Server Control (SRVCTL) to register the instances
Paper #371 23

Pre-Migration (Standby RAC, cont.)

Data Guard setup steps need to be followed which include:

Add additional standby REDO logs (The number of standby redo logs should be one more than the number of online REDO logs per thread and should be the same size) Enable Managed Recovery Mode on the standby database Enable archive destination on the primary to start shipping logs to the standby Validate the transfer Setup Data Guard Broker Validate the configuration and ensure that logs are being applied Bring up the other nodes in the RAC environment
Paper #371 24

Migration
Migrating is simple after the Data Guard database is setup

Ensure all ARCHIVELOGS have been applied Put the primary database in restricted mode and switch ARCHIVELOGS After all the outstanding logs have been applied, shutdown REDO apply on the standby Validate SCN across the primary and the standby

Two options from here:

- Switchover & upgrade - Shutdown & upgrade

**We Chose Shutdown & Upgrade**

Paper #371 25

Migration

All standby RAC nodes can be up if using DBUA

Ensure pre-checks are done

DBUA is brutal if you dont remove the obsolete parameters DST changes are important If manually upgrading DBMS_DST needs to be run to upgrade DST

Paper #371

26

Things to Look Out For

Ensure Environments Are Validated

RDA should be run and all issues fixed RAC check will give you a detailed overview of the system Ensure all starter kits and the RAC manuals are consulted: the result can be a good best practices evolving document Use services to better manage how the application connects and load balance your system Private network is key with RAC: follow Oracles best practices. XA changes in 11.2.0.2 & can cause issues: important to understand them Compatible parameter testing is important. Monitoring the environment after go-live is very important
Paper #371 27

Credits
Id like to thank the following people for helping me with the presentation and accompanying white paper:

Matthew Gallagher, Architect -- Walgreens Brice Lahl, Manager DB Architecture -- Walgreens Daniel Morgan, ACE Director -- Drugstore.com

A special shout out to Arup Nanda & Alex Gorbachev for encouraging me to present and Leighton Smith for reviewing my session
Also special credit to the RAC Assurance Team at Oracle for helping is get through this migration and subsequent issues. Some content in the whitepaper and presentation has been republished with permission from the Oracle RAC Assurance Team whose goal had been to proactively provide input to customers with new and existing CRS, RAC, ASM and MAA implementations. Paper #371 28

Questions
Please fill out evaluations
Paper #371 29