Real Application Cluster database upgradtion from 10.2.0.1 to 10.2.0.4 Database Version: 10.2.0.

1 Operating System: Ret Hat linux 4 As & Vmware 5 Storage Type : ASM & RAW Storage Pre-Request for upgrade:
Before upgrading any database we need take complete cold backup for database. Also we need to take backup of ORACLE_HOME, ORACLE_BASE & ORACLE_CRS_HOME In my case I taken ORACLE_HOME & ORACLE_CRS_HOME backups using (for all nodes) cd /u01/before_patch_backup tar -cvf oracle_base_jul19.tar $ORACLE_BASE tar -cvf oracle_home_jul19.tar $ORACLE_HOME tar -cvf oracle_crs_home_jul19.tar $ORACLE_CRS_HOME Form storage level we need take backups. I used dd command for taking OCR & VOTING disk backups. Also using I have taken complete RMAN backup for my database & using dd command I taken RAW devices backups (I mean ASM)

PS: Make sure; before taking any backups you need to sotp your cluster dont forgot. Using the below command you down your cluster. [root@linux1 bin]# ./crsctl stop crs Stopping resources. Successfully stopped CRS resources Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@linux1 bin]# ps root 27831 27206 [root@linux1 bin]# ps root 27833 27206 -ef | grep pmon 0 08:26 pts/2 -ef | grep lsn 0 08:26 pts/2 00:00:00 grep pmon 00:00:00 grep lsn

[root@linux2 bin]# ./crsctl stop crs Stopping resources. Successfully stopped CRS resources Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [oracle@linux2 oracle 20613 [oracle@linux2 [oracle@linux2 oracle 20638 /]$ ps 19349 /]$ /]$ ps 19349 -ef | grep pmon 0 05:54 pts/1 -ef | grep lsn 0 05:54 pts/1 00:00:00 grep pmon 00:00:00 grep lsn

PS: In my test environment I dont have separte ASM home; Im using ORACLE_HOME itslef. If you have ASM home then you may need take backup of ASM home also.

Cluster Upgration: Now Im going to upgrade my cluster database. Please carefully gothrow below screen shots & SQL Scripts. Step1:

1. Down 10.2.0.4 patch set (Patch Nr: 6810189) 2. Extract your patch 3. From oracle user you need to exeucte runInstaller then you will get below screen shot.

Step2: Please confirm your CRS Home. Because first of all you need to upgrade cluster only (Not Oracle database or ASM Home)

Step3: This is TWO Nodes cluster; thats why you will get your cluster nodes names list.

Step4:

This is pre-request checking steps; in case any thing wrong here you will come to know all the details.

Step5:

Once successfully completed your pre-request then continue the patch installation on cluster.

Step6 After succesfully installation you may need execute the below script from root user (in all nodes you need follow same steps) 5

/u01/app/oracle/product/10.2.0/crs/bin/crsctl stop crs /u01/app/oracle/product/10.2.0/crs/install/root102.sh

Form Node1
[root@linux1 bin]# ./crsctl stop crs Stopping resources. Error while stopping resources. Possible cause: CRSD is down. Stopping CSSD. Unable to communicate with the CSS daemon.

[root@linux1 bin]# cd .. [root@linux1 crs]# cd install [root@linux1 install]# ./root102.sh Creating pre-patch directory for saving pre-patch clusterware files Completed patching clusterware files to /u01/app/oracle/product/10.2.0/crs Relinking some shared libraries. Relinking of patched files is complete. WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root WARNING: directory '/u01/app/oracle/product' is not owned by root WARNING: directory '/u01/app/oracle' is not owned by root WARNING: directory '/u01/app' is not owned by root WARNING: directory '/u01' is not owned by root Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. Startup will be queued to init within 30 seconds. Starting up the CRS daemons. Waiting for the patched CRS daemons to start. This may take a while on some systems. . . . . 10204 patch successfully applied. clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 1: linux1 linux1-priv linux1 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully

From Node2

[root@linux2 bin]# ./crsctl stop crs Stopping resources. This could take several minutes. Successfully stopped CRS resources.

Stopping CSSD. Shutting down CSS daemon. Shutdown request successfully issued. [root@linux2 bin]# cd ../install [root@linux2 install]# ./root102.sh >> rootexecteation2.txt & [1] 10282 [root@linux2 install]# tail -f rootexecteation2.txt WARNING: directory '/u01/app/oracle/product/10.2.0' is not owned by root WARNING: directory '/u01/app/oracle/product' is not owned by root WARNING: directory '/u01/app/oracle' is not owned by root WARNING: directory '/u01/app' is not owned by root WARNING: directory '/u01' is not owned by root Preparing to recopy patched init and RC scripts. Recopying init and RC scripts. Startup will be queued to init within 30 seconds. /etc/profile: line 55: ulimit: open files: cannot modify limit: Operation not permitted /home/oracle/.bash_profile: line 30: ulimit: open files: cannot modify limit: Operation not permitted Starting up the CRS daemons. Waiting for the patched CRS daemons to start. This may take a while on some systems. . . 10204 patch successfully applied. clscfg: EXISTING configuration version 3 detected. clscfg: version 3 is 10G Release 2. Successfully accumulated necessary OCR keys. Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897. node <nodenumber>: <nodename> <private interconnect name> <hostname> node 2: linux2 linux2-priv linux2 Creating OCR keys for user 'root', privgrp 'root'.. Operation successful. clscfg -upgrade completed successfully

Cluster Status
[root@linux2 install]# cd ../bin

[root@linux2 bin]# ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.devdb.db application ONLINE OFFLINE ora....b1.inst application OFFLINE OFFLINE ora....b2.inst application ONLINE OFFLINE ora...._taf.cs application OFFLINE OFFLINE ora....db1.srv application OFFLINE OFFLINE ora....db2.srv application OFFLINE OFFLINE ora....SM1.asm application OFFLINE OFFLINE ora....X1.lsnr application OFFLINE OFFLINE ora.linux1.gsd application ONLINE ONLINE linux1 ora.linux1.ons application ONLINE ONLINE linux1 ora.linux1.vip application ONLINE ONLINE linux1 ora....SM2.asm application ONLINE ONLINE linux2 ora....X2.lsnr application ONLINE ONLINE linux2 ora.linux2.gsd application ONLINE OFFLINE ora.linux2.ons application ONLINE ONLINE linux2 ora.linux2.vip application ONLINE ONLINE linux2

PS: As per the above cluster status. (Which are highlighted red colors should be online form youre all nodes)

I am having some problem in Node2 thats why GSD down in node2.

Oracle Database Upgrade & ASM Upgration I dont have separate ASM home. Im keeping ASM in ORACLE_HOME itself; So Now going to continue my database & asm upgrade. 1. From oracle user you need to exeucte runInstaller then you will get below screen shot. 2. Please make sure your ORACLE_HOME in your path. 3. Please check any listener running from all nodes; (Else its will create problem) 4. Please check any database/asm instance running (If yes; pls shutdown those instance) 9

For example:
[oracle@linux1 bin]$ ps -ef | oracle 11993 22392 0 13:16 [oracle@linux1 bin]$ ps -ef | oracle 5155 1 0 11:06 grep pmon pts/2 00:00:00 grep pmon grep lsn ? 00:00:08

/u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_LINUX1 -inherit oracle 12000 22392 0 13:16 pts/2 00:00:00 grep lsn [oracle@linux1 bin]$ [oracle@linux1 bin]$ lsnrctl stop LISTENER_LINUX1 LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 19-JUL-2009 13:16:42

10

Copyright (c) 1991, 2005, Oracle. Connecting to

All rights reserved.

(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux1-vip.gen.com) (PORT=1521)(IP=FIRST))) The command completed successfully [oracle@linux2 bin]$ ps -ef | grep LISTENER_LINUX2 oracle 12726 1 0 13:29 ? 00:00:01 /u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_LINUX2 -inherit oracle 14894 8929 0 13:36 pts/1 00:00:00 grep LISTENER_LINUX2 [oracle@linux2 bin]$ lsnrctl stop LISTENER_LINUX2 LSNRCTL for Linux: Version 10.2.0.1.0 - Production on 19-JUL-2009 13:36:35 Copyright (c) 1991, 2005, Oracle. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux2-vip.gen.com) (PORT=1521)(IP=FIRST))) The command completed successfully [oracle@linux2 bin]$ [oracle@linux2 bin]$ ps -ef | grep pmon oracle 14993 8929 9 13:36 pts/1 00:00:00 grep pmon [oracle@linux2 bin]$ ps -ef | grep lsn oracle 15002 8929 0 13:36 pts/1 00:00:00 grep lsn [oracle@linux2 bin]$ All rights reserved.

Before upgrade cluster status & Listener status

[root@linux1 bin]# ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.devdb.db application OFFLINE OFFLINE ora....b1.inst application OFFLINE OFFLINE ora....b2.inst application OFFLINE OFFLINE ora...._taf.cs application OFFLINE OFFLINE ora....db1.srv application OFFLINE OFFLINE ora....db2.srv application OFFLINE OFFLINE ora....SM1.asm application ONLINE ONLINE linux1 ora....X1.lsnr application ONLINE ONLINE linux1 ora.linux1.gsd application ONLINE ONLINE linux1 ora.linux1.ons application ONLINE ONLINE linux1 ora.linux1.vip application ONLINE ONLINE linux1 ora....SM2.asm application ONLINE ONLINE linux2 ora....X2.lsnr application ONLINE ONLINE linux2 ora.linux2.gsd application ONLINE ONLINE linux2

11

ora.linux2.ons application ora.linux2.vip application

ONLINE ONLINE

ONLINE ONLINE

linux2 linux2

[oracle@linux1 bin]$ ps -ef | grep lsn oracle 8117 1 0 03:17 ? 00:00:02 /u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_LINUX1 -inherit oracle 12039 10732 0 03:28 pts/3 00:00:00 grep lsn [oracle@linux1 bin]$ [oracle@linux1 bin]$ lsnrctl stop LISTENER_LINUX1 LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 25-JUL-2009 03:28:18 Copyright (c) 1991, 2007, Oracle. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux1-vip.gen.com) (PORT=1521)(IP=FIRST))) The command completed successfully [oracle@linux1 bin]$ [oracle@linux1 bin]$ ps -ef | grep lsn oracle 12085 10732 0 03:28 pts/3 00:00:00 grep lsn [oracle@linux1 bin]$ [oracle@linux1 bin]$ ps -ef | grep pmon oracle 8024 1 0 03:17 ? 00:00:03 asm_pmon_+ASM1 oracle 12162 10732 0 03:28 pts/3 00:00:00 grep pmon [oracle@linux1 bin]$ [oracle@linux2 ~]$ cd /u01/app/oracle/product/10.2.0/crs/bin/ [oracle@linux2 bin]$ ps -ef | grep lsn oracle 7134 1 0 03:19 ? 00:00:01 /u01/app/oracle/product/10.2.0/db_1/bin/tnslsnr LISTENER_LINUX2 -inherit oracle 10678 10552 0 03:29 pts/2 00:00:00 grep lsn [oracle@linux2 bin]$ [oracle@linux2 bin]$ lsnrctl stop LISTENER_LINUX2 LSNRCTL for Linux: Version 10.2.0.4.0 - Production on 25-JUL-2009 03:29:45 Copyright (c) 1991, 2007, Oracle. Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=linux2-vip.gen.com) (PORT=1521)(IP=FIRST))) The command completed successfully [oracle@linux2 bin]$ [oracle@linux2 bin]$ ps -ef | grep pmon oracle 10784 1 21 03:29 ? 00:00:00 asm_pmon_+ASM2 oracle 10806 10552 0 03:30 pts/2 00:00:00 grep pmon All rights reserved. All rights reserved.

Step2: 12

This is TWO Nodes cluster; thats why you will get your cluster nodes names list.

Step3 If you want register you CSI Nr then you can otherwise ignore this slide 13

Step4

14

This is pre-request checking steps; in case any thing wrong here you will come to know all the details.

15

Step5 Once successfully completed your pre-request then continue the patch installation on cluster.

16

Step6: Now you need to execute below script for all nodes (Using root user) /u01/app/oracle/product/10.2.0/db_1/root.sh

[root@linux1 bin]# /u01/app/oracle/product/10.2.0/db_1/root.sh

17

Running Oracle10 root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/oracle/product/10.2.0/db_1 Enter the full pathname of the local bin directory: [/usr/local/bin]: The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying dbhome to /usr/local/bin ... The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying oraenv to /usr/local/bin ... The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying coraenv to /usr/local/bin ... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed.

[root@linux2 bin]# /u01/app/oracle/product/10.2.0/db_1/root.sh Running Oracle10 root.sh script... The following environment variables are set as: ORACLE_OWNER= oracle ORACLE_HOME= /u01/app/oracle/product/10.2.0/db_1 Enter the full pathname of the local bin directory: [/usr/local/bin]: The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying dbhome to /usr/local/bin ... The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying oraenv to /usr/local/bin ... The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n) [n]: y Copying coraenv to /usr/local/bin ... Entries will be added to the /etc/oratab file as needed by Database Configuration Assistant when a database is created Finished running generic part of root.sh script. Now product-specific root actions will be performed.

[oracle@linux1 bin]$ . oraenv ORACLE_SID = [devdb1] ? devdb [oracle@linux1 bin]$ export ORACLE_SID=devdb1 [oracle@linux1 bin]$ sqlplus '/as sysdba' SQL*Plus: Release 10.2.0.4.0 - Production on Sat Jul 25 03:38:20 2009

18

Copyright (c) 1982, 2007, Oracle. Connected to an idle instance. SQL> startup nomount ORACLE instance started. Total System Global Area Fixed Size Variable Size Database Buffers Redo Buffers SQL> SQL> show parameter clus 285212672 1267044 100666012 176160768 7118848

All Rights Reserved.

bytes bytes bytes bytes bytes

NAME ----------------------------------------------------------------cluster_database cluster_database_instances cluster_interconnects

TYPE VALUE ----------boolean integer string TRUE 2

SQL> alter system set cluster_database=FALSE scope=spfile; System altered. SQL> shutdown ORA-01507: database not mounted ORACLE instance shut down. SQL> SQL> exit Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Production With the Partitioning, Real Application Clusters, OLAP, Data Mining and Real Application Testing options [oracle@linux1 bin]$ sqlplus '/as sysdba' SQL*Plus: Release 10.2.0.4.0 - Production on Sat Jul 25 03:40:20 2009 Copyright (c) 1982, 2007, Oracle. Connected to an idle instance. SQL> startup upgrade ORACLE instance started. Total System Global Area Fixed Size Variable Size Database Buffers Redo Buffers Database mounted. Database opened. SQL> 285212672 1267044 100666012 176160768 7118848 bytes bytes bytes bytes bytes All Rights Reserved.

19

SQL> show parameter cluster NAME ----------------------------------------------------------------cluster_database cluster_database_instances cluster_interconnects SQL> TYPE VALUE ----------boolean integer string FALSE 1

SQL> SQL> spool /u01/patch.log SQL> SQL> @$ORACLE_HOME/rdbms/admin/catupgrd.sql ... .. ...... No errors. No errors. TIMESTAMP ------------------------------------------------------------------------------COMP_TIMESTAMP RUL 2009-07-25 11:54:47 DBUA_TIMESTAMP RUL VALID 2009-07-25 11:54:47

TIMESTAMP ------------------------------------------------------------------------------COMP_TIMESTAMP UPGRD_END 2009-07-25 11:54:50 . Oracle Database 10.2 Upgrade Status Utility 07-25-2009 11:54:50 . Component Status Version HH:MM:SS Oracle Database Server VALID 10.2.0.4.0 01:23:50 JServer JAVA Virtual Machine VALID 10.2.0.4.0 00:15:05 Oracle XDK VALID 10.2.0.4.0 00:02:21 Oracle Database Java Packages VALID 10.2.0.4.0 00:02:39 Oracle Text VALID 10.2.0.4.0 00:01:48 Oracle XML Database VALID 10.2.0.4.0 00:11:57 Oracle Real Application Clusters VALID 10.2.0.4.0 00:00:22

20

Oracle Workspace Manager VALID 10.2.0.4.3 00:04:23 Oracle Data Mining VALID 10.2.0.4.0 00:03:23 OLAP Analytic Workspace VALID 10.2.0.4.0 00:03:25 OLAP Catalog VALID 10.2.0.4.0 00:08:07 Oracle OLAP API VALID 10.2.0.4.0 00:06:20 Oracle interMedia VALID 10.2.0.4.0 00:31:09 Spatial VALID 10.2.0.4.0 00:13:33 Oracle Expression Filter VALID 10.2.0.4.0 00:02:04 Oracle Enterprise Manager VALID 10.2.0.4.0 00:12:57 Oracle Rule Manager VALID 10.2.0.4.0 00:01:11 . Total Upgrade Time: 03:24:44 DOC>################################################################### #### DOC>################################################################### #### DOC> DOC> The above PL/SQL lists the SERVER components in the upgraded DOC> database, along with their current version and status. DOC> DOC> Please review the status and version columns and look for DOC> any errors in the spool log file. If there are errors in the spool DOC> file, or any components are not VALID or not the current version, DOC> consult the Oracle Database Upgrade Guide for troubleshooting DOC> recommendations. DOC> DOC> Next shutdown immediate, restart for normal operation, and then DOC> run utlrp.sql to recompile any invalid application objects. DOC> DOC>################################################################### #### DOC>################################################################### #### DOC>#

SQL> shutdown immediate Database closed. Database dismounted. ORACLE instance shut down. SQL> SQL> startup ORACLE instance started. Total System Global Area Fixed Size 285212672 bytes 1267044 bytes

21

Variable Size Database Buffers Redo Buffers Database mounted. Database opened.

146803356 bytes 130023424 bytes 7118848 bytes

SQL> @$ORACLE_HOME/rdbms/admin/utlrp.sql TIMESTAMP ------------------------------------------------------------------------------COMP_TIMESTAMP UTLRP_BGN 2009-07-25 12:12:32 DOC> The following PL/SQL block invokes UTL_RECOMP to recompile invalid . ... ....... ERRORS DURING RECOMPILATION --------------------------0 PL/SQL procedure successfully completed. SQL> show parameter cluster NAME TYPE VALUE ------------------------------------ ---------------------------------------cluster_database boolean FALSE cluster_database_instances integer 1 cluster_interconnects string SQL> SQL> alter system set cluster_database=TRUE scope=spfile; System altered. SQL> alter system set cluster_database=TRUE scope=spfile; System altered. SQL> shutdown Database closed. Database dismounted. ORACLE instance shut down. SQL> SQL> startup ORACLE instance started. Total System Global Area Fixed Size Variable Size Database Buffers Redo Buffers Database mounted. Database opened. 285212672 1267044 163580572 113246208 7118848 bytes bytes bytes bytes bytes

22

SQL> SQL> show parameter cluster NAME ----------------------------------------------------------------cluster_database cluster_database_instances cluster_interconnects TYPE VALUE ----------boolean integer string TRUE 2

[oracle@linux1 bin]$ ./srvctl start instance -d devdb -i devdb1 [oracle@linux1 bin]$ ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.devdb.db application ONLINE ONLINE linux2 ora....b1.inst application ONLINE ONLINE linux1 ora....b2.inst application ONLINE OFFLINE ora...._taf.cs application OFFLINE OFFLINE ora....db1.srv application OFFLINE OFFLINE ora....db2.srv application OFFLINE OFFLINE ora....SM1.asm application ONLINE ONLINE linux1 ora....X1.lsnr application ONLINE ONLINE linux1 ora.linux1.gsd application ONLINE ONLINE linux1 ora.linux1.ons application ONLINE ONLINE linux1 ora.linux1.vip application ONLINE ONLINE linux1 ora....SM2.asm application ONLINE ONLINE linux2 ora....X2.lsnr application ONLINE ONLINE linux2 ora.linux2.gsd application ONLINE ONLINE linux2 ora.linux2.ons application ONLINE ONLINE linux2 ora.linux2.vip application ONLINE ONLINE linux2 [oracle@linux2 bin]$ ./srvctl start instance -d devdb -i devdb2 [oracle@linux1 bin]$ ./crs_stat -t Name Type Target State Host -----------------------------------------------------------ora.devdb.db application ONLINE ONLINE linux2 ora....b1.inst application ONLINE ONLINE linux1 ora....b2.inst application ONLINE ONLINE linux2 ora...._taf.cs application OFFLINE OFFLINE ora....db1.srv application OFFLINE OFFLINE ora....db2.srv application OFFLINE OFFLINE ora....SM1.asm application ONLINE ONLINE linux1 ora....X1.lsnr application ONLINE ONLINE linux1 ora.linux1.gsd application ONLINE ONLINE linux1 ora.linux1.ons application ONLINE ONLINE linux1 ora.linux1.vip application ONLINE ONLINE linux1 ora....SM2.asm application ONLINE ONLINE linux2 ora....X2.lsnr application ONLINE ONLINE linux2 ora.linux2.gsd application ONLINE ONLINE linux2 ora.linux2.ons application ONLINE ONLINE linux2 ora.linux2.vip application ONLINE ONLINE linux2 [oracle@linux2 bin]$ . oraenv ORACLE_SID = [devdb] ? [oracle@linux2 bin]$ sqlplus "/as sysdba"

23

SQL*Plus: Release 10.2.0.4.0 - Production on Sat Jul 25 12:45:02 2009 Copyright (c) 1982, 2007, Oracle. Connected to an idle instance. SQL> conn sys@devdb1 as sysdba Enter password: Connected. SQL> SQL> select * from v$version where rownum=1; BANNER ---------------------------------------------------------------Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Prod SQL> conn sys@devdb2 as sysdba Enter password: Connected. SQL> select * from v$version where rownum=1; BANNER ---------------------------------------------------------------Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - Prod All Rights Reserved.

24