Sunteți pe pagina 1din 416

Oracle Database 11g: RAC

Administration

m y
Student Guide
d e
ca
e A
c l
r a
O ly
l & On
D50311GC30
Edition 3.0
n a e
October 2010

t e r U s
D69173

I n
c l e
r a
O
Authors Copyright 2010, Oracle and/or it affiliates. All rights reserved.

James Womack Disclaimer


James Spiller This document contains proprietary information and is protected by copyright and
other intellectual property laws. You may copy and print this document solely for your
own use in an Oracle training course. The document may not be modified or altered in
Technical Contributors any way. Except where your use constitutes "fair use" under copyright law, you may
not use, share, download, upload, copy, print, display, perform, reproduce, publish,
David Brower license, post, transmit, or distribute this document in whole or in part without the
Jean-Francois Verrier express authorization of Oracle.
Mark Fuller
The information contained in this document is subject to change without notice. If you
Mike Leatherman find any problems in the document, please report them in writing to: Oracle University,
Barb Lundhild 500 Oracle Parkway, Redwood Shores, California 94065 USA. This document is not
warranted to be error-free.
S. Matt Taylor
Rick Wessman Restricted Rights Notice

If this documentation is delivered to the United States Government or anyone using


Technical Reviewers the documentation on behalf of the United States Government, the following notice is
applicable:
Christopher Andrews
Christian Bauwens U.S. GOVERNMENT RIGHTS
The U.S. Governments rights to use, modify, reproduce, release, perform, display, or
Michael Cebulla disclose these training materials are restricted by the terms of the applicable Oracle
Jonathan Creighton license agreement and/or the applicable U.S. Government contract.
Al Flournoy
Trademark Notice
Andy Fortunak
Joel Goodman Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names

y
may be trademarks of their respective owners.
Michael Hazel
Pete Jones
Jerry Lee
Markus Michalewicz
e m
Peter Sharman
a d
Ranbir Singh
Linda Smalley
Janet Stern
A c
Richard Strohm
Branislav Valny
c l e
Doug Williams

r a
Publisher
O ly
Sujatha Nagendra

l & On
n a e
t e r U s
I n
c l e
r a
O
Contents

11 Real Application Clusters Database Installation


Objectives 11-2
Installing the Oracle Database Software 11-3
Creating the Cluster Database 11-7
Select Database Type 11-8
Database Identification 11-9
Cluster Database Management Options 11-10
Passwords for Database Schema Owners 11-11
Database File Locations 11-12
Recovery Configuration 11-13
Database Content 11-14
Initialization Parameters 11-15
Database Storage Options 11-16
Create the Database 11-17
m y
Monitor Progress 11-18
d e
Postinstallation Tasks 11-19
Check Managed Targets 11-20
c a
Background Processes Specific to Oracle RAC 11-21

e
Single Instance to RAC Conversion 11-23 A
c l
Issues for Converting Single Instance Databases to Oracle RAC 11-24

Conversion Steps 11-26 r a


Single-Instance Conversion Using the DBCA 11-25

O ly
Single-Instance Conversion Using rconfig 11-29
Quiz 11-31
Summary 11-33
l & On
n a e
Practice 11 Overview 11-34

t e r U s
12 Oracle RAC Administration

I n
Objectives 12-2
Cluster Database Home Page 12-3

c l e
Cluster Database Instance Home Page 12-5

r a Cluster Home Page 12-6


Configuration Section 12-7

O Topology Viewer 12-9

iii
Enterprise Manager Alerts and RAC 12-10
Enterprise Manager Metrics and RAC 12-11
Enterprise Manager Alert History and RAC 12-13
Enterprise Manager Blackouts and RAC 12-14
Redo Log Files and RAC 12-15
Automatic Undo Management and RAC 12-17
Starting and Stopping RAC Instances 12-18
Starting and Stopping RAC Instances with SQL*Plus 12-19
Starting and Stopping RAC Instances with srvctl 12-20
Switch Between the Automatic and Manual Policies 12-21
RAC Initialization Parameter Files 12-22
SPFILE Parameter Values and RAC 12-23
EM and SPFILE Parameter Values 12-24
RAC Initialization Parameters 12-26
Parameters That Require Identical Settings 12-28
Parameters That Require Unique Settings 12-29
Quiescing RAC Databases 12-30
Terminating Sessions on a Specific Instance 12-31
How SQL*Plus Commands Affect Instances 12-32
Transparent Data Encryption and Wallets in RAC 12-33
m y
Quiz 12-34
d e
Summary 12-36
Practice 12 Overview 12-37
ca
13 Managing Backup and Recovery for RAC
e A
Objectives 13-2
c l
r
Media Recovery in Oracle RAC 13-4a
Protecting Against Media Failure 13-3

Parallel Recovery in RAC 13-5


O ly
l & On
Archived Log File Configurations 13-6
RAC and the Fast Recovery Area 13-7

a e
RAC Backup and Recovery Using EM 13-8
n
t e r U s
Configure RAC Recovery Settings with EM 13-9
Archived Redo File Conventions in RAC 13-10

I n
Configure RAC Backup Settings with EM 13-11
Oracle Recovery Manager 13-12

l e
Configure RMAN Snapshot Control File Location 13-13
c
Configure Control File and SPFILE Autobackup 13-14

r a
Crosschecking on Multiple RAC Clusters Nodes 13-15

O Channel Connections to Cluster Instances 13-16

iv
RMAN Channel Support for the Grid 13-17
RMAN Default Autolocation 13-18
Distribution of Backups 13-19
One Local Drive Shared Storage Backup Scheme 13-20
Multiple Drives Shared Storage Backup Scheme 13-21
Restoring and Recovering 13-22
Quiz 13-23
Summary 13-25
Practice 13 Overview 13-26

14 RAC Database Monitoring and Tuning


Objectives 14-2
CPU and Wait Time Tuning Dimensions 14-3
RAC-Specific Tuning 14-4
RAC and Instance or Crash Recovery 14-5
Instance Recovery and Database Availability 14-7
Instance Recovery and RAC 14-8
Analyzing Cache Fusion Impact in RAC 14-10
Typical Latencies for RAC Operations 14-11
Wait Events for RAC 14-12
m y
Wait Event Views 14-13
Global Cache Wait Events: Overview 14-14
d e
2-way Block Request: Example 14-16
ca
3-way Block Request: Example 14-17
2-way Grant: Example 14-18
e A
c l
Global Enqueue Waits: Overview 14-19

r a
Session and System Statistics 14-20
Most Common RAC Tuning Tips 14-21

O ly
Index Block Contention: Considerations 14-23

l & On
Oracle Sequences and Index Contention 14-24
Undo Block Considerations 14-25

a e
High-Water Mark Considerations 14-26
n
t e r U s
Concurrent Cross-Instance Calls: Considerations 14-27
Monitoring RAC Database and Cluster Performance 14-28

I n
Cluster Database Performance Page 14-29
Determining Cluster Host Load Average 14-30

c l e
Determining Global Cache Block Access Latency 14-31
Determining Average Active Sessions 14-32

r a Determining Database Throughput 14-33

O Accessing the Cluster Cache Coherency Page 14-35

v
Viewing the Cluster Interconnects Page 14-37
Viewing the Database Locks Page 14-39
AWR Snapshots in RAC 14-40
AWR Reports and RAC: Overview 14-41
Active Session History Reports for RAC 14-43
Automatic Database Diagnostic Monitor for RAC 14-45
What Does ADDM Diagnose for RAC? 14-47
EM Support for ADDM for RAC 14-48
Quiz 14-49
Summary 14-51
Practice 14 Overview 14-52

15 Services
Objectives 15-2
Oracle Services 15-3
Services for Policy- and Administrator-Managed Databases 15-4
Default Service Connections 15-5
Create Service with Enterprise Manager 15-6
Create Services with SRVCTL 15-7
Manage Services with Enterprise Manager 15-8
m y
Managing Services with EM 15-9
Manage Services with srvctl 15-10
d e
Use Services with Client Applications 15-11
ca
Services and Connection Load Balancing 15-12

e
Services and Transparent Application Failover 15-13 A
l
Use Services with the Resource Manager 15-14
c
r a
Services and Resource Manager with EM 15-15
Use Services with the Scheduler 15-16

O ly
Services and the Scheduler with EM 15-17

l & On
Using Distributed Transactions with RAC 15-19
Distributed Transactions and Services 15-20

a e
Service Thresholds and Alerts 15-22
n
t e r U s
Services and Thresholds Alerts: Example 15-23
Service Aggregation and Tracing 15-24

I n
Top Services Performance Page 15-25
Service Aggregation Configuration 15-26

l e
Service, Module, and Action Monitoring 15-27

c
Service Performance Views 15-28

r a
Quiz 15-29

O Summary 15-31
Practice 15: Overview 15-32

vi
16 Design for High Availability
Objectives 16-2
Causes of Unplanned Down Time 16-3
Causes of Planned Down Time 16-4
Oracles Solution to Down Time 16-5
RAC and Data Guard Complementarity 16-6
Maximum Availability Architecture 16-7
RAC and Data Guard Topologies 16-8
RAC and Data Guard Architecture 16-9
Data Guard Broker (DGB) and Oracle Clusterware (OC) Integration 16-11
Fast-Start Failover: Overview 16-12
Data Guard Broker Configuration Files 16-14
Real-Time Query Physical Standby Database 16-15
Hardware Assisted Resilient Data 16-16
Database High Availability: Best Practices 16-17
How Many ASM Disk Groups Per Database? 16-18
Which RAID Configuration for High Availability? 16-19
Should You Use ASM Mirroring Protection? 16-20
What Type of Striping Works Best? 16-21
ASM Striping Only 16-22
m y
Hardware RAIDStriped LUNs 16-23
Hardware RAIDStriped LUNs HA 16-24
d e
Disk I/O Design Summary 16-25
c a
Extended RAC: Overview 16-26
Extended RAC Connectivity 16-27
e A
l
Extended RAC Disk Mirroring 16-28
c
r a
Achieving Quorum with Extended RAC 16-29
Additional Data Guard Benefits 16-30

O ly
Using a Test Environment 16-31
Quiz 16-32
Summary 16-33
l & On
n a e
t e r U s
Appendix A: Practices and Solutions

I n
B DHCP and DNS Configuration For GNS
Objectives B-2

c l e
GNS Overview B-3
DHCP Service B-4

r a DHCP Configuration Example B-5

O DNS Concepts B-7


DNS Forwarding For GNS B-9

vii
DNS Configuration: Example B-11
DNS Configuration: Detail B-13
Summary B-17

C High Availability of Connections


Objectives C-2
Types of Workload Distribution C-3
Client-Side Load Balancing C-4
Client-Side, Connect-Time Failover C-5
Server-Side Load Balancing C-7
Runtime Connection Load Balancing and Connection Pools C-9
Fast Application Notification: Overview C-11
Fast Application Notification: Benefits C-12
FAN Events C-13
FAN Event Status C-14
FAN Event Reasons C-15
FAN Event Format C-16
Load Balancing Advisory: FAN Event C-17
Implementation of Server-Side Callouts C-18
Server-Side Callout Parse: Example C-19
m y
Server-Side Callout Filter: Example C-20
Configuring the Server-Side ONS C-21
d e
Optionally Configure the Client-Side ONS C-22
ca
JDBC Fast Connection Failover: Overview C-23
Using Oracle Streams Advanced Queuing for FAN C-24
e A
JDBC/ODP.NET FCF Benefits C-25
c l
Load Balancing Advisory C-26

r a
Runtime Connection Load Balancing and Connection Pools C-27
Monitor LBA FAN Events C-28
O ly
l & On
Transparent Application Failover: Overview C-29
TAF Basic Configuration Without FAN: Example C-30

a e
TAF Basic Configuration with FAN: Example C-31
n
t e
TAF Verification C-33 r U s
TAF Preconnect Configuration: Example C-32

Summary C-35 I n
FAN Connection Pools and TAF Considerations C-34

c l e
r a
O
viii
D Oracle RAC One Node
The Omotion Utility D-3
Oracle RAC One Node and OVM D-4
Creating and Managing RAC One Node D-5

E Cloning Oracle Clusterware


Objectives E-2
What Is Cloning? E-3
Benefits of Cloning Oracle Clusterware E-4
Creating a Cluster by Cloning Oracle Clusterware E-5
Preparing the Oracle Clusterware Home for Cloning E-6
Cloning to Create a New Oracle Clusterware Environment E-9
The clone.pl Script E-11
The clone.pl Environment Variables E-12
The clone.pl Command Options E-13
Cloning to Create a New Oracle Clusterware Environment E-14
Log Files Generated During Cloning E-18
Cloning to Extend Oracle Clusterware to More Nodes E-20
Quiz E-25
Summary E-27
m y
d e
F Clusterware Concepts
Objectives F-2
c a
Oracle Grid Infrastructure F-3
What Is a Cluster? F-4
e A
What Is Clusterware? F-5
c l
Oracle Clusterware F-6
r a
Oracle Clusterware Architecture and Services F-7

O ly
Oracle Clusterware Networking F-8

l & On
Oracle Clusterware Startup F-10
Oracle Clusterware Process Architecture F-11
a e
Grid Plug and Play F-13
n
t e r
GPnP Domain F-14
GPnP Components F-15
U s
I n
GPnP Profile F-16
Grid Naming Service F-17

c l e
Single Client Access Name F-18
GPnP Architecture Overview F-20

r a
O
ix
How GPnP Works: Cluster Node Startup F-22
How GPnP Works: Client Database Connections F-23
Summary F-24

G RAC Concepts
Objectives G-2
Benefits of Using RAC G-3
Clusters and Scalability G-4
Levels of Scalability G-5
Scaleup and Speedup G-6
Speedup/Scaleup and Workloads G-7
I/O Throughput Balanced: Example G-8
Performance of Typical Components G-9
Necessity of Global Resources G-10
Global Resources Coordination G-11
Global Cache Coordination: Example G-12
Write to Disk Coordination: Example G-13
Dynamic Reconfiguration G-14
Object Affinity and Dynamic Remastering G-15
Global Dynamic Performance Views G-16
m y
Efficient Internode Row-Level Locking G-17
Parallel Execution with RAC G-18
d e
Summary G-19
ca
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
x
Real Application Clusters Database
Installation

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Objectives

After completing this lesson, you should be able to:


Install the Oracle database software
Create a cluster database
Perform post-database-creation tasks
Perform a single instance to RAC conversion

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 2
Installing the Oracle Database Software
$ /stage/database/Disk1/runInstaller

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Installing the Oracle Database Software
e A
l
The Oracle Universal Installer (OUI) is used to install the Oracle Database 11g Release 2
c
r a
(11.2) software. Start the OUI by executing the runInstaller command from the root
directory of the Oracle Database 11g Release 2 CD-ROM or from the software staging
O ly
location. You can use the Specify Oracle Direct Connect Identification window to specify an

l & On
email address to receive security updates directly from Oracle Support as they occur.
Alternatively, you can elect to opt out of these alerts. If you want to receive them, supply your
a e
email address and your Oracle Support password, and click Next.
n
t e r U s
The Select installation option window enables you to create and configure a database, install
database software only, or upgrade an existing database. Select the Install database software

I n
only option and click Next.

c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 3
Installing the Oracle Database Software

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Installing the Oracle Database Software (continued)
e A
l
In the Grid Installation Options window, select Real Application Clusters database
c
r a
installation and select all nodes in your cluster on which the software should be installed.
Click Next to continue. In the Select product languages window, select your desired
O ly
languages from the Available Languages list and click the right arrow to promote the selected

l & On
languages to the Selected Languages list. Click Next to continue.
In the Select database edition window (not shown in the slide above), you select whether to

n a e
install the Enterprise Edition or the Standard Edition. Select the Enterprise Edition option and
click Next to continue.

t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 4
Installing the Oracle Database Software

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Installing the Oracle Database Software (continued)
e A
l
In the Specify Installation Location window, provide a value for ORACLE_BASE if you have
c
r a
not yet already done so. The default ORACLE_BASE location is /u01/app/oracle,
provided the RDBMS software is being installed by the oracle account. The Software
O ly
Location section of the window enables you to specify a value for the ORACLE_HOME

l & On
location. The default ORACLE_HOME location is
/u01/app/oracle/product/11.2.0/dbhome_1. Accept the suggested path or enter
a e
your own location. After entering the information, review it for accuracy, and click the Next
n
t e r U s
button to continue. In the Privileged operating system groups window, select the operating
system group that will act as the OSDBA group. Next, select the group that will act as the

n
OSOPER group. Click Next to continue.
I
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 5
Installing the Oracle Database Software

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Installing the Oracle Database Software (continued)
e A
l
The Perform prerequisite checks window verifies the operating system requirements that
c
Certified operating system check
r a
must be met for the installation to be successful. These requirements include:

O ly
Kernel parameters as required by the Oracle Database software

l & On
Required operating system packages and correct revisions
After each successful check, the Status for that check will indicate Succeeded. Any tests that

n a e
fail are also reported here. If any tests fail, click the Fix & Check Again button. The Installer

t e r U s
will generate fix-up scripts to correct the system deficiencies if possible. Execute the scripts as
directed by the Installer. The tests will be run again after completing the script executions.

I n
When all tests have succeeded, click the Next button to continue. In the Summary window,
review the Global settings and click Finish. At the end of the installation, the OUI will display
l e
another window, prompting you to run the root.sh scripts on the nodes you chose for the
c
r a
installation. Follow the instructions to run the scripts. When finished, click the OK button to
close the Execute Configuration Scripts window and return to the Finish screen. Click Close to
Ocomplete the installation and close the OUI.

Oracle Database 11g: RAC Administration 11 - 6


Creating the Cluster Database

$ cd /u01/app/oracle/product/11.2.0/dbhome_1/bin
$ ./dbca

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Creating the Cluster Database
e A
l
To create the cluster database, change directory to $ORACLE_HOME/bin on the installing
c
r a
node and execute the database configuration assistant (DBCA) utility as shown below:
$ cd /u01/app/oracle/product/11.2.0/dbhome_1/bin
$ ./dbca
O ly
l & On
The Welcome window appears first. You must select the type of database that you want to
install. Select the Oracle Real Application Clusters database option, and then click Next.
a e
The Operations window appears. For a first-time installation, you have only two choices: the
n
t e r U s
first option enables you to create a database and the other option enables you to manage
database creation templates. Select the Create a Database option, and then click Next to
continue.
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 7
Select Database Type

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Select Database Type
e A
l
The Database Templates window appears next. The DBCA tool provides several predefined
c
r a
database types to choose from, depending on your needs. The templates include:
General Purpose or Transaction Processing
Custom Database
O ly
Data Warehouse

l & On
In the example in the slide above, the General Purpose or Transaction Processing option is

n a e
chosen. Click the Next button to continue.

t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 8
Database Identification

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Database Identification
e A
l
In the Database Identification window, you must choose between an administrator-managed
c
and a policy-managed cluster database.
r a
Administrator-managed RAC databases specify a list of cluster nodes where RAC instances
O ly
will run. Services may also be specified and associated with preferred and alternative nodes.

& On
There is an explicit association between database services, instances, and cluster nodes.
l
n a e
Policy-based management, a new feature in this release, breaks the explicit association
between services, instances, and cluster nodes. Policy-based management introduces the

e r s
concept of server pools, which are logical divisions of a cluster that are dynamically allocated
t U
based on relative importance. Database services are associated with server pools and RAC

I n
instances are automatically started to satisfy the service to server pool associations. You

c l e
specify in which server pool the database resource will run and the number of instances needed
(cardinality). Oracle Clusterware is responsible for placing the database resource on a server.

r a
Server pools are logical divisions of a cluster into pools of servers which are allocated to host
databases or other applications. Server pools are managed using crsctl and srvctl
O
commands. Names must be unique within the resources defined for the cluster.
You must also choose the global database name and the nodes on which to create the cluster
database. The global database name can be up to 30 characters in length and must begin with
an alphabetical character. When you have finished, click Next to continue.
Oracle Database 11g: RAC Administration 11 - 9
Cluster Database Management Options

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cluster Database Management Options
e A
l
The Management Options window is displayed. For small cluster environments, you may
c
r a
choose to manage your cluster with Enterprise Manager Database Control. To do this, select
the Configure Enterprise Manager check box. If you have Grid Control installed somewhere
O ly
on your network, you can select the Use Grid Control for Database Management option. If

l & On
you select Enterprise Manager with the Grid Control option and the DBCA discovers agents
running on the local node, you can select the preferred agent from a list. Grid Control can
a e
simplify database management in large, enterprise deployments.
n
t e r U s
You can also configure Database Control to send email notifications when alerts occur. If you
want to configure this, you must supply a Simple Mail Transfer Protocol (SMTP) or outgoing

I n
mail server and an email address. You can also enable daily backups here. You must supply a
backup start time as well as operating system user credentials for this option.

c l e
If you want to use Grid Control to manage your database, but have not yet installed and

r a
configured a Grid Control server, do not click either of the management methods. When you
have made your choices, click the Next button to continue.
O
Oracle Database 11g: RAC Administration 11 - 10
Passwords for Database Schema Owners

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Passwords for Database Schema Owners
e A
l
The Database Credentials window appears next. You must supply passwords for the user
c
r a
accounts created by the DBCA when configuring your database. You can use the same
password for all of these privileged accounts by selecting the Use the Same Administrative
O ly
Password for All Accounts option. Enter your password in the Password field, and then enter

l & On
it again in the Confirm Password field.
Alternatively, you may choose to set different passwords for the privileged users. To do this,

n a e
select the Use Different Administrative Passwords option, enter your password in the

t e r U s
Password field, and then enter it again in the Confirm Password field. Repeat this for each user
listed in the User Name column. Click the Next button to continue.

I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 11
Database File Locations

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Database File Locations
e A
l
In the Database File Locations window, you must indicate where the database files are to be
c
r a
stored. You can choose your storage type from the drop-down list. Detected (and supported)
shared storage types are available here. You can choose to use a standard template for file
O ly
locations, one common location, or Oracle Managed Files (OMF). This cluster database uses

l & On
ASM and Oracle Managed Files. Therefore, select the Use Oracle-Managed Files option, and
enter the disk group name in the Database Area field. Alternatively, you can click the Browse
a e
button to indicate the location where the database files are to be created. When you have made
n
t e r U s
your choices, click the Next button to continue.
Note: Step numbers in the title bar can be different, depending on the choices made during the
installation.
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 12
Recovery Configuration

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Recovery Configuration
e A
l
In the Recovery Configuration window, you can select redo log archiving by selecting Enable
c
r a
Archiving. If you are using ASM or cluster file system storages, you can also select the Flash
Recovery Area size in the Recovery Configuration window. The size of the area defaults to
O ly
2048 megabytes, but you can change this figure if it is not suitable for your requirements. If

l & On
you are using ASM and a single disk group, the flash recovery area defaults to the ASM Disk
Group. If more than one disk group has been created, you can specify it here. If you use a
a e
cluster file system, the flash recovery area defaults to
n
t e r U s
$ORACLE_BASE/flash_recovery_area. You may also define your own variables for
the file locations if you plan to use the Database Storage window to define individual file

cluster database. I n
locations. Select the Enable Archiving check box to enable archiving immediately for the new

l e
Note: Flash Recovery Area has been renamed Fast Recovery Area in Oracle Database 11g
c
r a
Release 2 (11.2); however, some of the tools and utilities continue to use the previous name.
When you have completed your entries, click Next, and the Database Content window is
O
displayed.

Oracle Database 11g: RAC Administration 11 - 13


Database Content

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Database Content
e A
l
In the Database Content window, you can choose to install the Sample Schemas included with
c
r a
the database distribution. On the Custom Scripts tabbed page, you can choose to run your own
scripts as part of the database creation process. When you have finished, click the Next button
to continue to the next window.
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 14
Initialization Parameters

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Initialization Parameters
e A
l
In the Initialization Parameters window, you can set important database parameters. The
c
parameters are grouped under four tabs:
Memory
r a
Sizing
Character Sets O ly
Connection Mode
l & On
a e
On the Memory tabbed page, you can set parameters that deal with memory allocation,
n
t e r
including shared pool, buffer cache, Java pool, large pool, and PGA size. Automatic Memory
s
Management is the preferred memory management method and can be selected here. On the
U
I n
Sizing tabbed page, you can adjust the database block size. Note that the default is 8 kilobytes.
In addition, you can set the number of processes that can connect simultaneously to the
database.
c l e
By clicking the Character Sets tab, you can change the database character set. You can also
r a
select the default language and the date format. On the Connection Mode tabbed page, you can

O
choose the connection type that clients use to connect to the database. The default type is
Dedicated Server Mode. If you want to use Oracle Shared Server, click the Shared Server
Mode button. If you want to review the parameters that are not found on the four tabs, click
the All Initialization Parameters button. Click the Use Automatic Memory Management button
and click the Next button to continue.
Oracle Database 11g: RAC Administration 11 - 15
Database Storage Options

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Database Storage Options
e A
l
The Database Storage window provides full control over all aspects of database storage,
c
r
management are under your control here. a
including tablespaces, data files, and log members. Size, location, and all aspects of extent

O ly
When you have finished, click the Next button to continue to the next page.

l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 16
Create the Database

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Create the Database
e A
l
The Creation Options window appears. You can choose to create the database, or save your
c
r a
DBCA session as a database creation script by selecting the corresponding check box. Select
the Create Database check box, and then click the Finish button. The DBCA displays the
O ly
Summary screen, giving you a last chance to review all options, parameters, and so on that

l & On
have been chosen for your database creation.
Review the summary data. When you are ready to proceed, close the Summary window by
clicking the OK button.
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 17
Monitor Progress

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Monitor Progress
e A
l
The Progress Monitor window appears next. In addition to informing you about how fast the
c
r a
database creation is taking place, it also informs you about the specific tasks being performed
by the DBCA in real time. When the database creation progress reaches 100 percent, the
O ly
DBCA displays a dialog box announcing the completion of the creation process. It also directs

l & On
you to the installation log file location, parameter file location, and Enterprise Manager URL.
By clicking the Password Management button, you can manage the database accounts created
by the DBCA.
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 18
Postinstallation Tasks

Back up the root.sh script.


Download and install the required patch updates.
Verify the cluster database configuration.
$ srvctl config database -d orcl
Database unique name: ORCL
Database name: ORCL
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/orcl/spfileorcl.ora
Domain: example.com
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: orcl
Database instances: orcl1,orcl2, orcl3
Disk Groups: DATA,FRA
Services:
m y
e
Database is administrator managed

Copyright 2010, Oracle and/or its affiliates. All rights reserved.


a d
Postinstallation Tasks A c
l e
After the cluster database has been successfully created, run the following command to verify
c
the Oracle Cluster Registry configuration in your newly installed RAC environment:

r a
$ srvctl config database -d db_name

O ly
It is also recommended that you back up the root.sh script after you complete an
installation.
$ cd $ORACLE_HOME
l & On
$ cp root.sh root.sh.bak

n a e
r s
If you install other products in the same Oracle Home directory, the OUI updates the contents

e
of the existing root.sh script during the installation. If you require information contained in
t U
the original root.sh script, you can recover it from the root.sh file copy.
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 19
Check Managed Targets
https://host01.example.com:1158/em

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Check Managed Targets
e A
l
Another postinstallation task that you should perform if you are using Enterprise Manager
c
r a
Database Control or Grid Control is to check whether all the managed nodes and their
managed resources are properly registered and available. Open a browser and enter the address
O ly
for your Database Control console. Click the Targets tab to verify that all the targets appear
here.

l & On
Note: If Enterprise Manager Database Control has not been started on your node, you can start
a e
it with the following commands logged in as the oracle user, or the account that owns the
n
t e r
database software installation:

U s
$ export ORACLE_UNQNAME=orcl

I n
$ emctl start dbconsole

c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 20
Background Processes Specific to Oracle RAC

ACMS: Atomic Control file to Memory Service


GTX[0-j]: Global Transaction Process
LMON: Global Enqueue Service Monitor
LMD: Global Enqueue Service Daemon
LMS: Global Cache Service Process
LCK0: Instance Enqueue Process
RMSn: Oracle RAC Management Processes
RSMN: Remote Slave Monitor

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Background Processes Specific to Oracle RAC
e A
l
An Oracle RAC database has the same processes and memory structures as a single-instance
c
r a
Oracle database and additional process and memory structures that are specific to Oracle RAC.
The global cache service and global enqueue service processes, and the global resource
O ly
directory (GRD) collaborate to enable cache fusion. The Oracle RAC processes and their
identifiers are as follows:

l & On
Atomic Control file to Memory Service (ACMS): In a RAC environment, the ACMS
a e
per-instance process is an agent that contributes to ensuring a distributed SGA memory
n
t e r U s
update is either globally committed on success or globally aborted if a failure occurs.
Global Transaction Process (GTX[0-j]): The GTX[0-j] processes provide

I n
transparent support for XA global transactions in a RAC environment. The database
automatically tunes the number of these processes based on the workload of XA global

l e
transactions.
c
Global Enqueue Service Monitor (LMON): The LMON process monitors global

r a enqueues and resources across the cluster and performs global enqueue recovery

O operations.
Global Enqueue Service Daemon (LMD): The LMD process manages incoming remote
resource requests within each instance.

Oracle Database 11g: RAC Administration 11 - 21


Background Processes Specific to Oracle RAC (continued)
Global Cache Service Process (LMS): The LMS process maintains records of the data
file statuses and each cached block by recording information in the GRD.
The LMS process also controls the flow of messages to remote instances and manages
global data block access and transmits block images between the buffer caches of
different instances. This processing is part of the cache fusion feature.
LCK0: Instance Enqueue Process The LCK0 process manages noncache fusion
resource requests such as library and row cache requests.
RMSn: Oracle RAC Management Processes The RMSn processes perform
manageability tasks for Oracle RAC. Tasks accomplished by an RMSn process include
creation of resources related to Oracle RAC when new instances are added to the
clusters.
RSMN: Remote Slave Monitor The RSMN process manages background slave process
creation and communication on remote instances. These background slave processes
perform tasks on behalf of a coordinating process running in another instance.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 22
Single Instance to RAC Conversion

Single-instance databases can be


converted to RAC using:
DBCA
Enterprise Manager
Single-instance
RCONFIG utility database
DBCA automates most of the
conversion tasks.
Before conversion, ensure that:
Your hardware and operating RAC database
system are supported
Your cluster nodes have access to
shared storage

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Single Instance to RAC Conversion
e A
l
You can use the Database Configuration Assistant (DBCA) to convert single-instance Oracle
c
r a
databases to RAC. The DBCA automates the configuration of the control file attributes,
creates the undo tablespaces and the redo logs, and makes the initialization parameter file
O ly
entries for cluster-enabled environments. It also configures Oracle Net Services, Oracle

l & On
Clusterware resources, and the configuration for RAC database management for use by Oracle
Enterprise Manager or the srvctl utility.

n a e
Before you use the DBCA to convert a single-instance database to a RAC database, ensure that

t e r
your system meets the conditions:

U s
It is a supported hardware and operating system configuration.

I n
It has shared storage. A supported Cluster File System, NFS mount, or ASM is available
and accessible from all nodes.
l e
Your applications have no design characteristics that preclude their use with cluster
c
r adatabase processing.
You can also use Enterprise Manager and the rconfig utility to perform the single instance
O
to RAC conversion.

Oracle Database 11g: RAC Administration 11 - 23


Issues for Converting Single Instance Databases
to Oracle RAC
Backup procedures should be available before conversion
takes place.
Archiving in Oracle RAC environments requires a thread
number in the archive file format.
The archived logs from all instances of an Oracle RAC
database are required for media recovery.
By default, all database files are migrated to Oracle
Managed Files (OMF).

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Issues for Converting Single Instance Databases to Oracle RAC
e A
l
Whatever method you choose to use for the conversion, note the following administrative
c
r a
considerations before converting single-instance databases to Oracle RAC:
Backup procedures should be available before converting from a single-instance Oracle
O ly
Database to Oracle RAC. For archiving with Oracle RAC environments, the archive file

l & On
format requires a thread number.
The archived logs from all instances of an Oracle RAC database are required for media
a e
recovery. Because of this, if you archive to a file and you do not use a cluster file system,
n
t e r U s
or some other means to provide shared file systems, then you require a method of
accessing the archive logs from all nodes on which the cluster database has instances.

I n
By default, all database files are migrated to Oracle Managed Files (OMF). This feature
simplifies table space creation, ensures data file location consistency and compliance

l e
with OFA rules, and reduces human error with data file management.
c
r a
O
Oracle Database 11g: RAC Administration 11 - 24
Single-Instance Conversion Using the DBCA

Conversion steps for a single-instance database on


nonclustered hardware:
1. Back up the original single-instance database.
2. Complete the Oracle Clusterware installation.
3. Validate the cluster.
4. Copy the preconfigured database image.
5. Install the Oracle Database 11g software with RAC.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Single-Instance Conversion Using DBCA
e A
l
To convert from a single-instance Oracle database that is on a noncluster computer to a RAC
c
database, perform the following steps:
r a
1. Back up the original single-instance database.
O ly
2. Complete the Oracle Clusterware installation.
3. Validate the cluster.

l & On
4. Copy the preconfigured database image.
a e
5. Install the Oracle Database 11g software with RAC.
n
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 25
Conversion Steps
1. Back up the original single-instance database.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Conversion Steps
e A
l
1. Back Up the Original Single-Instance Database.
c
following procedure: r a
Use the DBCA to create a preconfigured image of your single-instance database by using the

O ly
1. Navigate to the bin directory in $ORACLE_HOME, and start the DBCA.

l & On
2. In the Welcome window, click Next.
3. In the Operations window, select Manage Templates, and click Next.

n a e
4. In the Template Management window, select Create a database template and From an

t e r U s
existing database (structure as well as data), and click Next.
5. In the Source Database window, enter the database name in the Database instance field,
and click Next.
I n
6. In the Template Properties window, enter a template name in the Name field. By default,
l e
the template files are generated in $ORACLE_HOME/assistants/dbca/templates.
c
r aEnter a description of the file in the Description field, and change the template file
location in the Template data file field if you want. When you have finished, click Next.
O 7. In the Location of Database Related Files window, select Maintain the file locations,
so that you can restore the database to the current directory structure, and click Finish.
The DBCA generates two files: a database structure file (template_name.ctl) and a
database preconfigured image file (template_name.dfb).

Oracle Database 11g: RAC Administration 11 - 26


Conversion Steps

2. Perform the preinstallation steps.


Tasks include kernel parameter configuration, hardware
setup, network configuration, and shared storage setup
3. Set up and validate the cluster.
Create a cluster with the required number of nodes
according to your hardware vendors documentation.
Validate cluster components before installation.
Install Oracle Clusterware.
Validate the completed cluster installation by using cluvfy.
4. Copy the preconfigured database image.
The database structure *.ctl file
The preconfigured database image *.dfb file
y

5. Install the Oracle Database 11g Release 2 software with


RAC.
em
Copyright 2010, Oracle and/or its affiliates. All rights reserved.
a d
Conversion Steps (continued) A c
2. Perform the Preinstallation Steps.
c l e
r a
Complete the Oracle Clusterware installation, as described in the lesson titled Grid
Infrastructure Installation or refer to the Oracle Grid Infrastructure Installation Guide for
your platform.
O ly
3. Set Up and Validate the Cluster.
l & On
Form a cluster with the required number of nodes according to your business needs. When you

n a e
have configured all the nodes in your cluster, validate cluster components by using the Cluster

t e r U s
Verification Utility, and then install Oracle Clusterware. When the clusterware is installed,
validate the completed cluster installation and configuration using the Cluster Verification
Utility.
I n
c l e
4. Copy the Preconfigured Database Image.
This includes copying the database structure *.ctl file and the database preconfigured image

r a
*.dfb file that the DBCA created in step one (Back Up the Original Single-Instance
Database) to a temporary location on the node in the cluster from which you plan to run the
O
DBCA.

Oracle Database 11g: RAC Administration 11 - 27


Conversion Steps (continued)
5. Install the Oracle Database 11g Release 2 software with RAC.
1. Run the OUI to perform an Oracle database installation with RAC. Select Cluster
Installation Mode and select the nodes to include in your RAC database.
2. In the OUI Database Configuration Types window, select Advanced install. After
installing the software, the OUI runs postinstallation tools such as NETCA, DBCA, and
so on.
3. In the DBCA Template Selection window, use the template that you copied to a
temporary location in the Copy the Preconfigured Database Image step. Use the
browse option to select the template location.
4. After creating the RAC database, the DBCA displays the Password Management window
in which you must change the passwords for database privileged users. When the DBCA
exits, the conversion is complete.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 28
Single-Instance Conversion Using rconfig

1. Locate the appropriate xml file located in the


$ORACLE_HOME/assistants/rconfig/sampleXMLs
directory.
2. Modify the ConvertToRAC_AdminManaged.xml or
ConvertToRAC_PolicyManaged.xml file as required
for your system.
3. Save the file under a different name.

$ cd $ORACLE_HOME/assistants/rconfig/sampleXMLs
$ vi ConvertToRAC_PolicyManaged.xml
... Saved as my_rac_conversion.xml
$ rconfig my_rac_conversion.xml

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Single-Instance Conversion Using rconfig
e A
c l
You can use the rconfig command-line utility to convert a single-instance database to RAC.
To use this feature, perform the following steps:
r a
1. Go to the $ORACLE_HOME/assistants/rconfig/sampleXMLs directory as the

O ly
oracle user and open the ConvertToRAC_AdminManaged.xml or the
ConvertToRAC_PolicyManaged.xml file (depending on your desired management

l & On
style) using a text editor, such as vi.
2. Review the XML file, and modify the parameters as required for your system. The XML

n a e
sample file contains comment lines that provide instructions about how to configure the

t e r U s
file. When you have completed making changes, save the file with the syntax
filename.xml. Make a note of the name you select.

I n
3. Assuming that you save your XML file as my_rac_conversion.xml, navigate to the
$ORACLE_HOME/bin directory, and use the following syntax to run the rconfig

l
command:

c e
$ ./rconfig my_rac_conversion.xml

r a
Note: The Convert verify option in the xml file has three options:
Convert verify="YES": rconfig performs checks to ensure that the
O prerequisites for single-instance to RAC conversion have been met before it starts
conversion.

Oracle Database 11g: RAC Administration 11 - 29


Single-Instance Conversion Using rconfig (continued)
Convert verify="NO": rconfig does not perform prerequisite checks, and starts
conversion.
Convert verify="ONLY": rconfig performs only prerequisite checks; it does
not start conversion after completing prerequisite checks.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 30
Quiz

The RAC database software installation is initiated by executing


runInstaller from the root directory of the Oracle Database
11g Release 2 CD-ROM or from the software staging location.
1. True
2. False

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 1
e A
The statement is true.
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 31
Quiz

A single instance database can be converted to a RAC


database by using (choose the correct options):
1. rconfig
2. netca
3. dbca

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 1,3
e A
Choices a and c are correct.
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 32
Summary

In this lesson, you should have learned how to:


Install the Oracle database software
Create a cluster database
Perform post-database-creation tasks
Perform a single instance to RAC conversion

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 33
Practice 11 Overview

This practice covers the following topics:


Installing the Oracle database software
Creating a RAC database

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 11 - 34
Oracle RAC Administration

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Objectives

After completing this lesson, you should be able to:


Use Enterprise Manager Cluster Database pages
Define redo log files in a RAC environment
Define undo tablespaces in a RAC environment
Start and stop RAC databases and instances
Modify initialization parameters in a RAC environment

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 2
Cluster Database Home Page

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cluster Database Home Page
e A
l
The Cluster Database Home page serves as a crossroad for managing and monitoring all
c
r a
aspects of your RAC database. From this page, you can access the other main cluster database
tabs: Performance, Availability, Server, Schema, Data Movement, Software and Support, and
Topology.
O ly
l & On
On this page, you find General, High Availability, Space Summary, and Diagnostic Summary
sections for information that pertains to your cluster database as a whole. The number of

n a e
instances is displayed for the RAC database, in addition to the status. A RAC database is

t e r U s
considered to be up if at least one instance has the database open. You can access the Cluster
Home page by clicking the Cluster link in the General section of the page.

I n
Other items of interest include the date of the last RMAN backup, archiving information,

c l e
space utilization, and an alert summary. By clicking the link next to the Flashback Database
Logging label, you can open the Recovery Settings page from where you can change various

r a
recovery parameters.
The Alerts table shows all the recent alerts that are open. Click the alert message in the
O
Message column for more information about the alert. When an alert is triggered, the name of
the metric for which the alert was triggered is displayed in the Name column.

Oracle Database 11g: RAC Administration 12 - 3


Cluster Database Home Page

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cluster Database Home Page (continued)
e A
l
The Related Alerts table provides information about alerts for related targets, such as Listeners
c
r
and the time the alert was last checked. a
and Hosts, and contains details about the message, the time the alert was triggered, the value,

O ly
The Policy Trend Overview page (accessed by clicking the Compliance Score link) provides a

l & On
comprehensive view about a group or targets containing other targets with regard to
compliance over a period of time. Using the tables and graphs, you can easily watch for trends
in progress and changes.
n a e
e r s
The Security At a Glance page shows an overview of the security health of the enterprise for
t U
the targets or specific groups. This helps you to focus on security issues by showing statistics

I n
about security policy violations and noting critical security patches that have not been applied.

l e
The Job Activity table displays a report of the job executions that shows the scheduled,

c
running, suspended, and problem (stopped/failed) executions for all Enterprise Manager jobs

r a
on the cluster database.

O
The Instances table lists the instances for the cluster database, their availability, alerts, policy
violations, performance findings, and related ASM Instance. Click an instance name to go to
the Home page for that instance. Click the links in the table to get more information about a
particular alert, advice, or metric.

Oracle Database 11g: RAC Administration 12 - 4


Cluster Database Instance Home Page

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cluster Database Instance Home Page
e A
l
The Cluster Database Instance Home page enables you to view the current state of the instance
c
r a
by displaying a series of metrics that portray its overall health. This page provides a launch
point for the performance, administration, and maintenance of the instance environment.
O ly
You can access the Cluster Database Instance Home page by clicking one of the instance

l & On
names from the Instances section of the Cluster Database Home page. This page has basically
the same sections as the Cluster Database Home page.

n a e
The difference is that tasks and monitored activities from these pages apply primarily to a

e r s
specific instance. For example, clicking the Shutdown button on this page shuts down only this
t U
one instance. However, clicking the Shutdown button on the Cluster Database Home page

I n
gives you the option of shutting down all or specific instances.

l e
By scrolling down on this page, you see the Alerts, Related Alerts, Policy Violations, Jobs

c
Activity, and Related Links sections. These provide information similar to that provided in the

r a
same sections on the Cluster Database Home page.

O
Oracle Database 11g: RAC Administration 12 - 5
Cluster Home Page

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cluster Home Page
e A
l
The slide shows you the Cluster Home page, which can be accessed by clicking the Cluster tab
c
r a
located in the top-right corner of Enterprise Manager. Even if the database is down, the cluster
page is available to manage resources. The cluster is represented as a composite target
O ly
composed of nodes and cluster databases. An overall summary of the cluster is provided here.

l & On
The Cluster Home page displays several sections, including General, Configuration,
Diagnostic Summary, Cluster Databases, Alerts, and Hosts. The General section provides a
a e
quick view of the status of the cluster, providing basic information such as current Status,
n
t e r U s
Availability, Up nodes, Clusterware Version, and Oracle Home.
The Configuration section enables you to view the operating systems (including hosts and OS

I n
patches) and hardware (including hardware configuration and hosts) for the cluster.

c l e
The Cluster Databases table displays the cluster databases (optionally associated with
corresponding services) associated with this cluster, their availability, and any alerts on those

r a
databases. The Alerts table provides information about any alerts that have been issued along
with the severity rating of each.
O
The Hosts table (not shown in the screenshot) displays the hosts for the cluster, availability,
corresponding alerts, CPU and memory utilization percentage, and total input/output (I/O) per
second.

Oracle Database 11g: RAC Administration 12 - 6


Configuration Section

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Configuration Section
e A
l
The Cluster Home page is invaluable for locating configuration-specific data. Locate the
c
r a
Configuration section on the Cluster Home page. The View drop-down list enables you to
inspect hardware and operating system overview information.
O ly
Click the Hosts link, and then click the Hardware Details link of the host that you want. On the

l & On
Hardware Details page, you find detailed information about your CPU, disk controllers,
network adapters, and so on. This information can be very useful when determining the Linux
patches for your platform.
n a e
e r s
Click History to access the hardware history information for the host.
t U
I n
Some hardware information is not available, depending on the hardware platform.
Note: The Local Disk Capacity (GB) field shows the disk space that is physically attached

l e
(local) to the host. This value does not include disk space that may be available to the host
c
through networked file systems.

r a
O
Oracle Database 11g: RAC Administration 12 - 7
Configuration Section

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Configuration Section (continued)
e A
l
The Operating System Details General page displays the operating system details for a host,
c
including:
r a
General information, such as the distributor version and the maximum swap space of the
operating system
O ly
l & On
Information about operating system properties
The Source column displays where Enterprise Manager obtained the value for each operating
system property.
n a e
e r s
To see a list of changes to the operating system properties, click History.
t U
I n
The Operating System Details File Systems page displays information about one or more file
systems for the selected hosts:

l e
Name of the file system on the host

c
Type of mounted file systemfor example, ufs or nfs

r a
Directory where the file system is mounted
The mount options for the file systemfor example, ro, nosuid, or nobrowse
O
The Operating System Details Packages page displays information about the operating system
packages that have been installed on a host.

Oracle Database 11g: RAC Administration 12 - 8


Topology Viewer

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Topology Viewer
e A
l
The Oracle Enterprise Manager Topology Viewer enables you to visually see the relationships
c
r a
between target types for each host of your cluster database. You can zoom in or out, pan, and
see selection details. These views can also be used to launch various administration functions.
O ly
The Topology Viewer populates icons on the basis of your system configuration. If a listener is

types are:
l & On
serving an instance, a line connects the listener icon and the instance icon. Possible target

Interface
n a e
Listener
ASM Instance
t e r U s
I
Database Instance
n
c l e
If the Show Configuration Details option is not selected, the topology shows the monitoring
view of the environment, which includes general information such as alerts and overall status.

r a
If you select the Show Configuration Details option, additional details are shown in the
Selection Details window, which are valid for any topology view. For instance, the Listener
O
component would also show the machine name and port number.
You can click an icon and then right-click to display a menu of available actions.

Oracle Database 11g: RAC Administration 12 - 9


Enterprise Manager Alerts and RAC

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Enterprise Manager Alerts and RAC
e A
l
You can use Enterprise Manager to administer alerts for RAC environments. Enterprise
c
r a
Manager distinguishes between database- and instance-level alerts in RAC environments.
Enterprise Manager also responds to metrics from across the entire RAC database and
O ly
publishes alerts when thresholds are exceeded. Enterprise Manager interprets both predefined

l & On
and customized metrics. You can also copy customized metrics from one cluster database
instance to another, or from one RAC database to another. A recent alert summary can be

n a e
found on the Database Control Home page. Notice that alerts are sorted by relative time and
target name.

t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 10
Enterprise Manager Metrics and RAC

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Enterprise Manager Metrics and RAC
e A
l
Alert thresholds for instance-level alerts, such as archive log alerts, can be set at the instance
c
r a
target level. This enables you to receive alerts for the specific instance if performance exceeds
your threshold. You can also configure alerts at the database level, such as setting alerts for
O ly
tablespaces. This enables you to avoid receiving duplicate alerts at each instance.

l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 11
Enterprise Manager Metrics and RAC

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Enterprise Manager Metrics and RAC (continued)
e A
l
It is also possible to view the metric across the cluster in a comparative or overlay fashion. To
c
r a
view this information, click the Compare Targets link at the bottom of the corresponding
metric page. When the Compare Targets page appears, choose the instance targets that you
O ly
want to compare by selecting them and then clicking the Move button. If you want to compare

the OK button to continue.


l & On
the metric data from all targets, click the Move All button. After making your selections, click

n a e
The Metric summary page appears next. Depending on your needs, you can accept the default

t e r U s
timeline of 24 hours or select a more suitable value from the View Data drop-down list. If you
want to add a comment regarding the event for future reference, enter a comment in the

I n
Comment for Most Recent Alert field, and then click the Add Comment button.

c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 12
Enterprise Manager Alert History and RAC

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Enterprise Manager Alert History and RAC
e A
l
In a RAC environment, you can see a summary of the alert history for each participating
c
r a
instance directly from the Cluster Database Home page. The drill-down process is shown in
the slide. You click the Alert History link in the Related Links section of the Cluster Database
O ly
Home page. This takes you to the Alert History page on which you can see the summary for

l & On
both instances in the example. Click on the Alert History chart for the instance to go directly
to the Alert history Page for that instance. From there, you can access a corresponding alert
a e
page by choosing the alert of your choice.
n
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 13
Enterprise Manager Blackouts and RAC

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Enterprise Manager Blackouts and RAC
e A
l
You can use Enterprise Manager to define blackouts for all managed targets of your RAC
c
r a
database to prevent alerts from being recorded. Blackouts are useful when performing
scheduled or unscheduled maintenance or other tasks that might trigger extraneous or
O ly
unwanted events. You can define blackouts for an entire cluster database or for specific cluster
database instances.

l & On
To create a blackout event, click the Setup link at the top of any Enterprise Manager page.

n a e
Then click the Blackouts link on the left. The Blackouts page appears.

e r s
Click the Create button. The Create Blackout: Properties page appears. You must enter a name
t U
or tag in the Name field. If you want, you can also enter a descriptive comment in the

I n
Comments field. This is optional. Enter a reason for the blackout in the Reason field.

l e
In the Targets area of the Properties page, you must choose a target Type from the drop-down

c
list. In the example in the slide, the entire cluster database RDBB is chosen. Click the cluster

r a
database in the Available Targets list, and then click the Move button to move your choice to
the Selected Targets list. Click the Next button to continue.
O
The Member Targets page appears next. Expand the Selected Composite Targets tree and
ensure that all targets that must be included appear in the list. Continue and define your
schedule as you normally would.

Oracle Database 11g: RAC Administration 12 - 14


Redo Log Files and RAC

Node1 Node2
RAC01 RAC02

Shared storage

Group 1 SPFILE
Group 4

Group 2 RAC01.THREAD=1
Group 5
RAC02.THREAD=2
Group 3
Thread 2
Thread 1

ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 4;


ALTER DATABASE ADD LOGFILE THREAD 2 GROUP 5;

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Redo Log Files and RAC
e A
l
If you increase the cardinality of a server pool in a policy-managed database and a new server
c
r a
is allocated to the server pool, then Oracle Clusterware starts an instance on the new server if
you have Oracle Managed Files (OMF) enabled. If the instance starts and there is no thread or
O ly
redo log file available, then Oracle Clusterware automatically enables a thread of redo and

file system.
l & On
allocates the associated redo log files and undo if the database uses Oracle ASM or any cluster

n a e
You should create redo log groups only if you are using administrator-managed databases. For

t e r U s
administrator-managed databases, each instance has its own online redo log groups. Create
these redo log groups and establish group members. To add a redo log group to a specific

I n
instance, specify the INSTANCE clause in the ALTER DATABASE ADD LOGFILE statement. If
you do not specify the instance when adding the redo log group, the redo log group is added to
l e
the instance to which you are currently connected. Each instance must have at least two groups
c
r a
of redo log files. You must allocate the redo log groups before enabling a new instance with
the ALTER DATABASE ENABLE INSTANCE instance_name command. When the current
O
group fills, an instance begins writing to the next log file group. If your database is in
ARCHIVELOG mode, each instance must save full online log groups as archived redo log files
that are tracked in the control file.
Note: You can use Enterprise Manager to administer redo log groups in a RAC environment.

Oracle Database 11g: RAC Administration 12 - 15


Redo Log Files and RAC

Node1 Node2
RAC01 RAC02

Shared storage

Group 1 SPFILE
Group 4

Group 2 RAC01.THREAD=1
Group 5
RAC02.THREAD=2
Group 3
Thread 2
Thread 1

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Redo Log Files and RAC (continued)
e A
l
If you increase the cardinality of a server pool in a policy-managed database and a new server
c
r a
is allocated to the server pool, then Oracle Clusterware starts an instance on the new server if
you have Oracle Managed Files (OMF) enabled. If the instance starts and there is no thread or
O ly
redo log file available, then Oracle Clusterware automatically enables a thread of redo and

file system.
l & On
allocates the associated redo log files and undo if the database uses Oracle ASM or any cluster

n a e
You should create redo log groups only if you are using administrator-managed databases. For

t e r U s
administrator-managed databases, each instance has its own online redo log groups. Create
these redo log groups and establish group members. To add a redo log group to a specific

I n
instance, specify the INSTANCE clause in the ALTER DATABASE ADD LOGFILE statement. If
you do not specify the instance when adding the redo log group, the redo log group is added to
l e
the instance to which you are currently connected. Each instance must have at least two groups
c
r a
of redo log files. You must allocate the redo log groups before enabling a new instance with
the ALTER DATABASE ENABLE INSTANCE instance_name command. When the current
O
group fills, an instance begins writing to the next log file group. If your database is in
ARCHIVELOG mode, each instance must save full online log groups as archived redo log files
that are tracked in the control file.
Note: You can use Enterprise Manager to administer redo log groups in a RAC environment.

Oracle Database 11g: RAC Administration 12 - 16


Automatic Undo Management and RAC

Pending Node1 Node2


offline
RAC01 RAC02

Consistent reads
Transaction recovery

Shared storage

SPFILE
undotbs3
RAC01.UNDO_TABLESPACE=undotbs1
RAC02.UNDO_TABLESPACE=undotbs2
undotbs1 undotbs2

ALTER SYSTEM SET UNDO_TABLESPACE=undotbs3 SID='RAC01';


m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Automatic Undo Management in RAC
e A
l
The Oracle database automatically manages undo segments within a specific undo tablespace
c
r a
that is assigned to an instance. Under normal circumstances, only the instance assigned to the
undo tablespace can modify the contents of that tablespace. However, all instances can always
O ly
read all undo blocks for consistent-read purposes. Also, any instance can update any undo

l & On
tablespace during transaction recovery, as long as that undo tablespace is not currently used by
another instance for undo generation or transaction recovery.

n a e
You assign undo tablespaces in your RAC database by specifying a different value for the

t e r U s
UNDO_TABLESPACE parameter for each instance in your SPFILE or individual PFILEs. If you do
not set the UNDO_TABLESPACE parameter, each instance uses the first available undo tablespace.

I n
For policy-managed databases, Oracle automatically allocates the undo tablespace when the
instance starts if you have OMF enabled.

c l e
You can dynamically switch undo tablespace assignments by executing the ALTER SYSTEM

r a
SET UNDO_TABLESPACE statement. You can run this command from any instance. In the
example above, the previously used undo tablespace assigned to the RAC01 instance remains
O
assigned to it until RAC01s last active transaction commits. The pending offline tablespace
may be unavailable for other instances until all transactions against that tablespace are
committed. You cannot simultaneously use Automatic Undo Management (AUM) and manual
undo management in a RAC database. It is highly recommended that you use the AUM mode.
Oracle Database 11g: RAC Administration 12 - 17
Starting and Stopping RAC Instances

Multiple instances can open the same database


simultaneously.
Shutting down one instance does not interfere with other
running instances.
SHUTDOWN TRANSACTIONAL LOCAL does not wait for
other instances transactions to finish.
RAC instances can be started and stopped by using:
Enterprise Manager
The Server Control (srvctl) utility
SQL*Plus
Shutting down a RAC database means shutting down all
instances accessing the database.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Starting and Stopping RAC Instances
e A
l
In a RAC environment, multiple instances can have the same RAC database open at the same
c
instances.
r a
time. Also, shutting down one instance does not interfere with the operation of other running

O ly
The procedures for starting up and shutting down RAC instances are identical to the

& On
procedures used in single-instance Oracle, with the following exception:
l
n a e
The SHUTDOWN TRANSACTIONAL command with the LOCAL option is useful to shut down an
instance after all active transactions on the instance have either committed or rolled back.

t e r U s
Transactions on other instances do not block this operation. If you omit the LOCAL option, this
operation waits until transactions on all other instances that started before the shutdown are

I n
issued either a COMMIT or a ROLLBACK.

l e
You can start up and shut down instances by using Enterprise Manager, SQL*Plus, or Server

c
Control (srvctl). Both Enterprise Manager and srvctl provide options to start up and shut

r a
down all the instances of a RAC database with a single step.

O
Shutting down a RAC database mounted or opened by multiple instances means that you need
to shut down every instance accessing that RAC database. However, having only one instance
opening the RAC database is enough to declare the RAC database open.

Oracle Database 11g: RAC Administration 12 - 18


Starting and Stopping
RAC Instances with SQL*Plus
[host01] $ echo $ORACLE_SID
orcl1
sqlplus / as sysdba
SQL> startup
SQL> shutdown

[host02] $ echo $ORACLE_SID


orcl2
sqlplus / as sysdba
SQL> startup
SQL> shutdown

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Starting and Stopping RAC Instances with SQL*Plus
e A
l
If you want to start up or shut down just one instance, and you are connected to your local
c
(SID) for the local instance.
r a
node, then you must first ensure that your current environment includes the system identifier

O ly
To start up or shut down your local instance, initiate a SQL*Plus session connected as SYSDBA

& On
or SYSOPER, and then issue the required command (for example, STARTUP).
l
n a e
You can start multiple instances from a single SQL*Plus session on one node by way of
Oracle Net Services. To achieve this, you must connect to each instance by using a Net

t e r U s
Services connection string, typically an instance-specific alias from your tnsnames.ora file.
For example, you can use a SQL*Plus session on a local node to shut down two instances on

I n
remote nodes by connecting to each using the instances individual alias name.

l e
It is not possible to start up or shut down more than one instance at a time in SQL*Plus, so you

c
cannot start or stop all the instances for a cluster database with a single SQL*Plus command.

r a
To verify that instances are running, on any node, look at V$ACTIVE_INSTANCES.

O
Note: SQL*Plus is integrated with Oracle Clusterware to make sure that corresponding
resources are correctly handled when starting up and shutting down instances via SQL*Plus.

Oracle Database 11g: RAC Administration 12 - 19


Starting and Stopping
RAC Instances with srvctl

start/stop syntax:
srvctl start|stop instance -d <db_name> -i <inst_name_list>
[-o open|mount|nomount|normal|transactional|immediate|abort>]

srvctl start|stop database -d <db_name>


[-o open|mount|nomount|normal|transactional|immediate|abort>]

Examples:
$ srvctl start instance -d orcl -i orcl1,orcl2

$ srvctl stop instance -d orcl -i orcl1,orcl2

$ srvctl start database -d orcl -o open

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Starting and Stopping RAC Instances with srvctl
e A
l
The srvctl start database command starts a cluster database, its enabled instances, and
c
r a
services. The srvctl stop database command stops a database, its instances, and its services.
The srvctl start instance command starts instances of a cluster database. This command
O ly
also starts all enabled and nonrunning services that have the listed instances either as preferred
or as available instances.
l & On
The srvctl stop instance command stops instances, and all enabled and running services

n a e
that have these instances as either preferred or available instances.

t e r U s
You must disable an object that you intend to keep stopped after you issue a srvctl stop
command, otherwise Oracle Clusterware can restart it as a result of another planned operation.

I n
srvctl does not support concurrent executions of commands on the same object. Therefore,

c l e
run only one srvctl command at a time for each database, service, or other object. In order to
use the START or STOP options of the SRVCTL command, your service must be an Oracle

r a
Clusterwareenabled, nonrunning service.
Note: For more information, refer to the Oracle Clusterware and Oracle Real Application
O
Clusters Administration and Deployment Guide.

Oracle Database 11g: RAC Administration 12 - 20


Switch Between the Automatic
and Manual Policies
$ srvctl config database -d orcl -a
Database unique name: orcl
Database name: orcl
Oracle home: /u01/app/oracle/product/11.2.0/dbhome_1
Oracle user: oracle
Spfile: +DATA/orcl/spfileorcl.ora
Domain:
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: orcl
Database instances: orcl1,orcl2
Disk Groups: DATA, FRA
Services:
Database is enabled

y
Database is administrator managed

srvctl modify database -d orcl -y MANUAL;


em
Copyright 2010, Oracle and/or its affiliates. All rights reserved.
a d
Switch Between the Automatic and Manual Policies A c
l e
By default, Oracle Clusterware is configured to start the VIP, listener, instance, ASM,
c
r a
database services, and other resources during system boot. It is possible to modify some
resources to have their profile parameter AUTO_START set to the value 2. This means that after

O ly
node reboot, or when Oracle Clusterware is started, resources with AUTO_START=2 need to be
started manually via srvctl. This is designed to assist in troubleshooting and system

l & On
maintenance. When changing resource profiles through srvctl, the command tool

n a e
automatically modifies the profile attributes of other dependent resources given the current
prebuilt dependencies. The command to accomplish this is:

e r s
srvctl modify database -d <dbname> -y AUTOMATIC|MANUAL

t U
To implement Oracle Clusterware and Real Application Clusters, it is best to have Oracle
I n
Clusterware start the defined Oracle Clusterware resources during system boot, which is the

l e
default. The first example in the slide uses the srvctl config database command to display
the current policy for the orcl database. As you can see, it is currently set to its default:
c
r a
AUTOMATIC. The second statement uses the srvctl modify database command to change
the current policy to MANUAL for the orcl database. When you add a new database by using the

O
srvctl add database command, by default, that database is placed under the control of
Oracle Clusterware using the AUTOMATIC policy. However, you can use the following
statement to directly set the policy to MANUAL: srvctl add database -d orcl -y MANUAL.
Note: You can also use this procedure to configure your system to prevent Oracle Clusterware
from auto-restarting failed database instances more than once.
Oracle Database 11g: RAC Administration 12 - 21
RAC Initialization Parameter Files

An SPFILE is created if you use the DBCA.


The SPFILE must be created in an ASM disk group or a
cluster file system file.
All instances use the same SPFILE.
If the database is created manually, create an SPFILE
from a PFILE.

Node1 Node2
RAC01 RAC02
initRAC01.ora initRAC02.ora
SPFILE= SPFILE=
SPFILE

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
RAC Initialization Parameter Files
e A
l
When you create the database, DBCA creates an SPFILE in the file location that you specify.
c
r a
This location can be an ASM disk group or a cluster file system file. If you manually create
your database, it is recommended to create an SPFILE from a PFILE.
O ly
All instances in the cluster database use the same SPFILE. Because the SPFILE is a binary file,

& On
do not edit it. Change the SPFILE settings by using EM or ALTER SYSTEM SQL statements.
l
n a e
RAC uses a traditional PFILE only if an SPFILE does not exist or if you specify PFILE in your
STARTUP command. Using SPFILE simplifies administration, maintaining parameter settings

e r s
consistent, and guarantees parameter settings persistence across database shutdown and
t U
startup. In addition, you can configure RMAN to back up your SPFILE.

I n
c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 22
SPFILE Parameter Values and RAC

You can change parameter settings using the ALTER


SYSTEM SET command from any instance:
ALTER SYSTEM SET <dpname> SCOPE=MEMORY sid='<sid|*>';
SPFILE entries such as:
*.<pname> apply to all instances
<sid>.<pname> apply only to <sid>
<sid>.<pname> takes precedence over *.<pname>
Use current or future *.<dpname> settings for <sid>:
ALTER SYSTEM RESET <dpname> SCOPE=MEMORY sid='<sid>';

Remove an entry from your SPFILE:


ALTER SYSTEM RESET <dpname> SCOPE=SPFILE sid='<sid|*>';
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
SPFILE Parameter Values and RAC
e A
l
You can modify the value of your initialization parameters by using the ALTER SYSTEM SET
c
r a
command. This is the same as with a single-instance database except that you have the
possibility to specify the SID clause in addition to the SCOPE clause.
O ly
By using the SID clause, you can specify the SID of the instance where the value takes effect.

l & On
Specify SID='*' if you want to change the value of the parameter for all instances. Specify
SID='sid' if you want to change the value of the parameter only for the instance sid. This

n a e
setting takes precedence over previous and subsequent ALTER SYSTEM SET statements that

t e r U
you do not specify the SID clause.s
specify SID='*'. If the instances are started up with an SPFILE, then SID='*' is the default if

I n
If you specify an instance other than the current instance, then a message is sent to that

c l e
instance to change the parameter value in its memory if you are not using the SPFILE scope.
The combination of SCOPE=MEMORY and SID='sid' of the ALTER SYSTEM RESET command

r a
allows you to override the precedence of a currently used <sid>.<dparam> entry. This allows
for the current *.<dparam> entry to be used, or for the next created *.<dparam> entry to be
O
taken into account on that particular SID.
Using the last example, you can remove a line from your SPFILE.
Note: When you start an instance with an SPFILE, the default for SQL*Plus is SCOPE=BOTH.

Oracle Database 11g: RAC Administration 12 - 23


EM and SPFILE Parameter Values

SCOPE=MEMORY

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
EM and SPFILE Parameter Values
e A
l
You can access the Initialization Parameters page by clicking the Initialization Parameters link
c
on the Cluster Database Server page.
r a
The Current tabbed page shows you the values currently used by the initialization parameters
O ly
of all the instances accessing the RAC database. You can filter the Initialization Parameters

Name field.
l & On
page to show only those parameters that meet the criteria of the filter that you entered in the

n a e
The Instance column shows the instances for which the parameter has the value listed in the

t e r U
instances of the cluster database.
s
table. An asterisk (*) indicates that the parameter has the same value for all remaining

I n
Choose a parameter from the Select column and perform one of the following steps:

l e
Click Add to add the selected parameter to a different instance. Enter a new instance

c
name and value in the newly created row in the table.

r a
Click Reset to reset the value of the selected parameter. Note that you can reset only
those parameters that do not have an asterisk in the Instance column. The value of the
O selected column is reset to the value of the remaining instances.
Note: For both Add and Reset buttons, the ALTER SYSTEM command uses SCOPE=MEMORY.

Oracle Database 11g: RAC Administration 12 - 24


EM and SPFILE Parameter Values

SCOPE=BOTH

SCOPE=SPFILE

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
EM and SPFILE Parameter Values (continued)
e A
l
The SPFile tabbed page displays the current values stored in your SPFILE.
c
r a
As on the Current tabbed page, you can add or reset parameters. However, if you select the
Apply changes in SPFILE mode check box, the ALTER SYSTEM command uses
O ly
SCOPE=BOTH. If this check box is not selected, SCOPE=SPFILE is used.

& On
Click Apply to accept and generate your changes.
l
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 25
RAC Initialization Parameters

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
RAC Initialization Parameters
e A
l
CLUSTER_DATABASE: Enables a database to be started in cluster mode. Set this to TRUE.
c
r a
CLUSTER_DATABASE_INSTANCES: Sets the number of instances in your RAC
environment. A proper setting for this parameter can improve memory use.
O ly
CLUSTER_INTERCONNECTS: Specifies the cluster interconnect when there is more than one

l & On
interconnect. Refer to your Oracle platformspecific documentation for the use of this
parameter, its syntax, and its behavior. You typically do not need to set the
a e
CLUSTER_INTERCONNECTS parameter. For example, do not set this parameter for the
n
t e r
following common configurations:

U s
If you have only one cluster interconnect

I n
If the default cluster interconnect meets the bandwidth requirements of your RAC
database, which is typically the case

l e
If NIC bonding is being used for the interconnect
c
When OIFCFGs global configuration can specify the right cluster interconnects. It only

r a needs to be specified as an override for OIFCFG.

O
DB_NAME: If you set a value for DB_NAME in instance-specific parameter files, the setting
must be identical for all instances.
DISPATCHERS: Set this parameter to enable a shared-server configuration, that is, a server
that is configured to allow many user processes to share very few server processes.

Oracle Database 11g: RAC Administration 12 - 26


RAC Initialization Parameters (continued)
With shared-server configurations, many user processes connect to a dispatcher. The
DISPATCHERS parameter may contain many attributes. Oracle recommends that you
configure at least the PROTOCOL and LISTENER attributes.
PROTOCOL specifies the network protocol for which the dispatcher process generates a
listening end point. LISTENER specifies an alias name for the Oracle Net Services listeners.
Set the alias to a name that is resolved through a naming method, such as a tnsnames.ora
file.
Other parameters that can affect RAC database configurations include:
ASM_PREFERRED_READ_FAILURE_GROUPS: Specifies a set of disks to be the preferred
disks from which to read mirror data copies. The values that you set for this parameter
are instance specific and need not be the same on all instances.
GCS_SERVER_PROCESSES: This static parameter specifies the initial number of server
processes for an Oracle RAC instances Global Cache Service (GCS). The GCS
processes manage the routing of interinstance traffic among Oracle RAC instances. The
default number of GCS server processes is calculated based on system resources. For
systems with one to three CPUs, there are two GCS server processes (LMSn). For
systems with four to fifteen CPUs, there are two GCS server processes (LMSn). For
systems with sixteen more CPUs, the number of GCS server processes equals (the
number of CPUs divided by 32) + 2, dropping any fractions. You can set this parameter
to different values on different instances.
m y
INSTANCE_NAME: The instances SID. The SID identifies the instances shared
d e
memory on a host. Any alphanumeric characters can be used. The value for this

number during the creation of the database when using DBCA. c a


parameter is automatically set to the database unique name followed by an incrementing

A
INSTANCE_NUMBER: An Oracle RAC parameter that specifies a unique number that
e
c l
maps the instance to one free list group for each database object. This parameter must be
set for every instance in the cluster. It is automatically defined during the creation of the
database when using DBCA.
r a
O ly
REMOTE_LISTENER: This parameter resolves to a SCAN:port, resolving to a SCAN
listener. Oracle Database 11g release 2 (11.2) and later instances only register with

l & On
SCAN listeners as remote listeners. Upgraded databases register with SCAN listeners as

a e
remote listeners, and also continue to register with all node listeners.
n
t e r
LOCAL_LISTENER: Specifies a network name that resolves to an address or address
s
list of Oracle Net local listeners (that is, listeners that are running on the same machine
U
I n
as this instance). The address or address list is specified in the TNSNAMES.ORA file or
other address repository as configured for your system.

c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 27
Parameters That Require Identical Settings

ACTIVE_INSTANCE_COUNT
ARCHIVE_LAG_TARGET
COMPATIBLE
CLUSTER_DATABASE/CLUSTER_DATABASE_INSTANCES
CONTROL_FILES
DB_BLOCK_SIZE
DB_DOMAIN
DB_FILES
DB_NAME
DB_RECOVERY_FILE_DEST/DB_RECOVERY_FILE_DEST_SIZE
DB_UNIQUE_NAME
PARALLEL_EXECUTION_MESSAGE_SIZE
REMOTE_LOGIN_PASSWORD_FILE
RESULT_CACHE_MAX_SIZE
UNDO_MANAGEMENT
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Parameters That Require Identical Settings
e A
l
Certain initialization parameters that are critical at database creation or that affect certain
c
r a
database operations must have the same value for every instance in RAC. Specify these
parameter values in the SPFILE, or within each init_dbname.ora file on each instance. In the
O ly
above list, each parameter must have the same value on all instances.

l & On
Note: The setting for DML_LOCKS and RESULT_CACHE_MAX_SIZE must be identical on every
instance only if set to zero. Disabling the result cache on some instances may lead to incorrect
results.
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 28
Parameters That Require Unique Settings
Instance settings:
INSTANCE_NAME
INSTANCE_NUMBER
UNDO_TABLESPACE
CLUSTER_INTERCONNECTS
ASM_PREFERRED_READ_FAILURE_GROUPS

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Parameters That Require Unique Settings
e A
l
The Oracle server uses the INSTANCE_NUMBER parameter to distinguish among instances at
c
r a
startup. The Oracle server uses the THREAD number to assign redo log groups to specific
instances. To simplify administration, use the same number for both the THREAD and
INSTANCE_NUMBER parameters.
O ly
l & On
If you specify UNDO_TABLESPACE with Automatic Undo Management enabled, set this
parameter to a unique undo tablespace name for each instance.

n a e
Using the ASM_PREFERRED_READ_FAILURE_GROUPS initialization parameter, you can specify a

e r s
list of preferred read failure group names. The disks in those failure groups become the
t U
preferred read disks. Thus, every node can read from its local disks. The setting for this

I n
parameter is instance specific, and the values need not be the same on all instances.

c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 29
Quiescing RAC Databases

Use the ALTER SYSTEM QUIESCE RESTRICTED


statement from a single instance:
SQL> ALTER SYSTEM QUIESCE RESTRICTED;
You must have the Database Resource Manager feature
activated to issue the statement above.
The database cannot be opened by other instances after
the ALTER SYSTEM QUIESCE statement starts.
The ALTER SYSTEM QUIESCE RESTRICTED and ALTER
SYSTEM UNQUIESCE statements affect all instances in a
RAC environment.
Cold backups cannot be taken when the database is in a
quiesced state.
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Quiescing RAC Databases
e A
l
To quiesce a RAC database, use the ALTER SYSTEM QUIESCE RESTRICTED statement from one
c
r a
instance. It is not possible to open the database from any instance while the database is in the
process of being quiesced from another instance. After all the non-DBA sessions become

O ly
inactive, the ALTER SYSTEM QUIESCE RESTRICTED statement executes and the database is
considered to be quiesced. In a RAC environment, this statement affects all instances.

l & On
To issue the ALTER SYSTEM QUIESCE RESTRICTED statement in a RAC environment, you
a e
must have the Database Resource Manager feature activated, and it must have been activated
n
t e r
since instance startup for all instances in the cluster database. It is through the Database
s
Resource Manager that non-DBA sessions are prevented from becoming active. The following
U
I n
conditions apply to RAC:
If you had issued the ALTER SYSTEM QUIESCE RESTRICTED statement, but the Oracle

l e
server has not finished processing it, then you cannot open the database.

c
You cannot open the database if it is already in a quiesced state.

r a
The ALTER SYSTEM QUIESCE RESTRICTED and ALTER SYSTEM UNQUIESCE statements
affect all instances in a RAC environment, not just the instance that issues the command.
O
Cold backups cannot be taken when the database is in a quiesced state because the Oracle
background processes may still perform updates for internal purposes even when the database
is in a quiesced state. Also, the file headers of online data files continue to appear as if they are
being accessed. They do not look the same as if a clean shutdown were done.

Oracle Database 11g: RAC Administration 12 - 30


Terminating Sessions on a Specific Instance

SQL> SELECT SID, SERIAL#, INST_ID


2 FROM GV$SESSION WHERE USERNAME='JMW';
SID SERIAL# INST_ID
---------- ---------- ----------
140 3340 2
SQL> ALTER SYSTEM KILL SESSION '140,3340,@2';
System altered.
SQL>

ALTER SYSTEM KILL SESSION '140,3340,@2'


*
ERROR at line 1:
ORA-00031: session marked for kill
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Terminating Sessions on a Specific Instance
e A
l
Starting with Oracle Database 11g Release 1, you can use the ALTER SYSTEM KILL SESSION
c
r a
statement to terminate a session on a specific instance.
The slide above illustrates this by terminating a session started on a different instance than the
O ly
one used to terminate the problematic session.

& On
If the session is performing some activity that must be completed, such as waiting for a reply
l
n a e
from a remote database or rolling back a transaction, then Oracle Database waits for this
activity to complete, marks the session as terminated, and then returns control to you. If the

e r s
waiting lasts a minute, Oracle Database marks the session to be terminated and returns control
t U
to you with a message that the session is marked to be terminated. The PMON background

I n
process then marks the session as terminated when the activity is complete.

l e
Note: You can also use the IMMEDIATE clause at the end of the ALTER SYSTEM command to

c
immediately terminate the session without waiting for outstanding activity to complete.

r a
O
Oracle Database 11g: RAC Administration 12 - 31
How SQL*Plus Commands Affect Instances

SQL*Plus Command Associated Instance


ARCHIVE LOG Generally affects the current instance
CONNECT Affects the default instance if no instance is specified
in the CONNECT command
HOST Affects the node running the SQL*Plus session
RECOVER Does not affect any particular instance, but rather the
database
SHOW PARAMETER and Show the current instance parameter and SGA
SHOW SGA information
STARTUP and Affect the current instance
SHUTDOWN
SHOW INSTANCE Displays information about the current instance

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
How SQL*Plus Commands Affect Instances
e A
l
Most SQL statements affect the current instance. You can use SQL*Plus to start and stop
c
r a
instances in the RAC database. You do not need to run SQL*Plus commands as root on
UNIX-based systems or as Administrator on Windows-based systems. You need only the
O ly
proper database account with the privileges that you normally use for single-instance Oracle

instances:
l & On
database administration. The following are some examples of how SQL*Plus commands affect

a e
The ALTER SYSTEM SET CHECKPOINT LOCAL statement affects only the instance
n
t e r U s
to which you are currently connected, rather than the default instance or all instances.
ALTER SYSTEM CHECKPOINT LOCAL affects the current instance.

I n
ALTER SYSTEM CHECKPOINT or ALTER SYSTEM CHECKPOINT GLOBAL
affects all instances in the cluster database.

l e
ALTER SYSTEM SWITCH LOGFILE affects only the current instance.
c
To force a global log switch, use the ALTER SYSTEM ARCHIVE LOG CURRENT

r a statement.

O The INSTANCE option of ALTER SYSTEM ARCHIVE LOG enables you to archive
each online redo log file for a specific instance.

Oracle Database 11g: RAC Administration 12 - 32


Transparent Data Encryption and Wallets in RAC

One wallet shared by all instances on shared storage:


No additional administration is required.
One copy of the wallet on each local storage:
Local copies need to be synchronized each time master key
is changed.

ALTER SYSTEM SET ENCRYPTION KEY 1

Wallet Wallet Wallet


Manual
copy
Master keys Master key Master key

2
Node1 Node2

Manual copy
Noden

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Transparent Data Encryption and Wallets in RAC
e A
l
Wallets used by RAC instances for Transparent Database Encryption may be a local copy of a
c
of the nodes can access.
r a
common wallet shared by multiple nodes, or a shared copy residing on shared storage that all

O ly
A deployment with a single wallet on a shared disk requires no additional configuration to use
Transparent Data Encryption.
l & On
n a e
If you want to use local copies, you must copy the wallet and make it available to all of the
other nodes after initial configuration. For systems using Transparent Data Encryption with

e r s
encrypted wallets, you can use any standard file transport protocol. For systems using
t U
Transparent Data Encryption with obfuscated wallets, file transport through a secured channel

I n
is recommended. The wallet must reside in the directory specified by the setting for the

c l e
WALLET_LOCATION or ENCRYPTION_WALLET_LOCATION parameter in sqlnet.ora. The
local copies of the wallet need not be synchronized for the duration of Transparent Data

r a
Encryption usage until the server key is rekeyed through the ALTER SYSTEM SET KEY SQL
statement. Each time you run the ALTER SYSTEM SET KEY statement at a database instance,
O
you must again copy the wallet residing on that node and make it available to all of the other
nodes. To avoid unnecessary administrative overhead, reserve rekeying for exceptional cases
where you are certain that the server master key is compromised and that not rekeying it would
cause a serious security problem.
Oracle Database 11g: RAC Administration 12 - 33
Quiz

If an instance starts in a policy-managed RAC environment and


no thread or redo log file is available, then Oracle Clusterware
automatically enables a thread of redo and allocates the redo
log files and undo if the database uses Oracle ASM or any
cluster file system and OMF is enabled.
1. True
2. False

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 1
e A
The statement is true.
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 34
Quiz

Which of the following statements is not true:


1. Multiple instances can open the same database
simultaneously.
2. Shutting down one instance does not interfere with other
running instances.
3. SHUTDOWN TRANSACTIONAL LOCAL will wait for other
instances transactions to finish.
4. Shutting down a RAC database means shutting down all
instances accessing the database.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 3
e A
Statement 3 is not true.
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 35
Summary

In this lesson, you should have learned how to:


Use Enterprise Manager Cluster Database pages
Define redo log files in a RAC environment
Define undo tablespaces in a RAC environment
Start and stop RAC databases and instances
Modify initialization parameters in a RAC environment

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 36
Practice 12 Overview

This practice covers the following topics:


Using operating system and password file authenticated
connections
Using Oracle Database authenticated connections
Stopping a complete ORACLE_HOME component stack

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 12 - 37
m y
d e
ca
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Managing Backup and Recovery for RAC

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Objectives

After completing this lesson, you should be able to configure


the following:
The RAC database to use ARCHIVELOG mode and the fast
recovery area
RMAN for the RAC environment

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 13 - 2
Protecting Against Media Failure

Archived Archived
log files log files

Database
Mirrored backups
disks

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Protecting Against Media Failure
e A
l
Although RAC provides you with methods to avoid or to reduce down time due to a failure of
c
r a
one or more (but not all) of your instances, you must still protect the database itself, which is
shared by all the instances. This means that you need to consider disk backup and recovery
O ly
strategies for your cluster database just as you would for a nonclustered database.

l & On
To minimize the potential loss of data due to disk failures, you may want to use disk mirroring
technology (available from your server or disk vendor). As in nonclustered databases, you can

n a e
have more than one mirror if your vendor allows it, to help reduce the potential for data loss

t e r U s
and to provide you with alternative backup strategies. For example, with your database in
ARCHIVELOG mode and with three copies of your disks, you can remove one mirror copy and

I n
perform your backup from it while the two remaining mirror copies continue to protect
ongoing disk activity. To do this correctly, you must first put the tablespaces into backup mode
l e
and then, if required by your cluster or disk vendor, temporarily halt disk operations by issuing
c
r a
the ALTER SYSTEM SUSPEND command. After the statement completes, you can break the
mirror and then resume normal operations by executing the ALTER SYSTEM RESUME
O
command and taking the tablespaces out of backup mode.

Oracle Database 11g: RAC Administration 13 - 3


Media Recovery in Oracle RAC

Media recovery must be user initiated through a client


application.
In these situations, use RMAN to restore backups of the
data files and then recover the database.
RMAN media recovery procedures for RAC do not differ
substantially from those for single-instance environments.
The node that performs the recovery must be able to
restore all of the required data files.
That node must also be able to either read all required
archived redo logs on disk or restore them from backups.
When recovering a database with encrypted tablespaces,
the Oracle Wallet must be opened after database mount
and before you open the database.
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Media Recovery in Oracle RAC
e A
l
Media recovery must be user initiated through a client application, whereas instance recovery
c
r a
is automatically performed by the database. In these situations, use RMAN to restore backups
of the data files and then recover the database. The procedures for RMAN media recovery in
O ly
Oracle RAC environments do not differ substantially from the media recovery procedures for
single-instance environments.

l & On
The node that performs the recovery must be able to restore all of the required data files. That

n a e
node must also be able to either read all the required archived redo logs on disk or be able to

t e r U s
restore them from backups. Each instance generates its own archive logs that are copies of its
dedicated redo log group threads. It is recommended that Automatic Storage Management

I n
(ASM) or a cluster file system be used to consolidate these files.

c l e
When recovering a database with encrypted tablespaces (for example, after a SHUTDOWN
ABORT or a catastrophic error that brings down the database instance), you must open the

r a
Oracle Wallet after database mount and before you open the database, so the recovery process
can decrypt data blocks and redo.
O
Oracle Database 11g: RAC Administration 13 - 4
Parallel Recovery in RAC

Oracle Database automatically selects the optimum degree


of parallelism for:
Instance recovery
Crash recovery
Archived redo logs are applied using an optimal number of
parallel processes based on the availability of CPUs.
With RMAN's RESTORE and RECOVER commands, the
following three stages of recovery can use parallelism:
Restoring data files
Applying incremental backups
Applying archived redo logs
To disable parallel instance and crash recovery, set the
RECOVERY_PARALLELISM parameter to 0.
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Parallel Recovery in RAC
e A
l
Oracle Database automatically selects the optimum degree of parallelism for instance and
c
r a
crash recovery. Oracle Database applies archived redo logs using an optimal number of
parallel processes based on the availability of CPUs. With RMANs RESTORE and RECOVER
O ly
commands, Oracle Database automatically uses parallelism for the following three stages of
recovery:

l & On
Restoring Data Files: When restoring data files, the number of channels you allocate in
a e
the RMAN recover script effectively sets the parallelism that RMAN uses. For example,
n
t e r U s
if you allocate five channels, you can have up to five parallel streams restoring data files.
Applying Incremental Backups: Similarly, when you are applying incremental

I n
backups, the number of channels you allocate determines the potential parallelism.
Applying Archived Redo Logs with RMAN: Oracle Database automatically selects the

l e
optimum degree of parallelism based on available CPU resources.
c
r a
To disable parallel instance and crash recovery on a system with multiple CPUs, set the
RECOVERY_PARALLELISM parameter to 0.
O
Use the NOPARALLEL clause of the RMAN RECOVER command or ALTER DATABASE
RECOVER statement to force the RAC database to use nonparallel media recovery.

Oracle Database 11g: RAC Administration 13 - 5


Archived Log File Configurations

Shared storage scheme: Local archive with NFS


Archive logs from each scheme: Each instance can
instance are written to the
same file location.
read mounted archive
destinations of all instances.
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Archived Log File Configurations
e A
l
During backup and recovery operations involving archived log files, the Oracle server
c
r a
determines the file destinations and names from the control file. If you use RMAN, the
archived log file path names can also be stored in the optional recovery catalog. However, the
O ly
archived log file path names do not include the node name, so RMAN expects to find the files

l & On
it needs on the nodes where the channels are allocated.
If you use a supported shared storage scheme, your instances can all write to the same archive

n a e
log destination. Backup and recovery of the archive logs are easy because all logs are located
in the same directory.

t e r U s
If a shared storage location is not available, Oracle recommends that local archive log

I n
destinations be created for each instance with NFS-read mount points to all other instances.

c l e
This is known as the local archive with network file system (NFS) scheme. During backup,
you can either back up the archive logs from each host or select one host to perform the

r a
backup for all archive logs. During recovery, one instance may access the logs from any host
without having to first copy them to the local destination. The LOG_ARCHIVE_FORMAT
O
parameter supports the %t variable that embeds the unique thread number into the name of the
archive logs so that each node generates unique names.
Using either scheme, you may want to provide a second archive destination to avoid single
points of failure.
Oracle Database 11g: RAC Administration 13 - 6
RAC and the Fast Recovery Area

Fast
recovery
area

Cluster file system Certified NFS


directory

ASM
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
RAC and the Fast Recovery Area
e A
l
To use a fast recovery area in RAC, you must place it on an ASM disk group, a cluster file
c
r a
system, or on a shared directory that is configured through certified NFS for each RAC
instance. That is, the fast recovery area must be shared among all the instances of a RAC
O ly
database. In addition, set the DB_RECOVERY_FILE_DEST parameter to the same value on
all instances.

l & On
Oracle Enterprise Manager enables you to set up a fast recovery area. To use this feature:

n a e
1. From the Cluster Database Home page, click the Maintenance tab.

t e r U s
2. Under the Backup/Recovery options list, click Configure Recovery Settings.
3. Specify your requirements in the Flash Recovery Area section of the page.

I n
Note: For Oracle Database 11g Release 2 (11.2), the flash recovery area has been renamed fast

pages.
c l e
recovery area. Oracle Enterprise Manager, however, still uses the older vocabulary on its Web

r a
O
Oracle Database 11g: RAC Administration 13 - 7
RAC Backup and Recovery Using EM

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
RAC Backup and Recovery Using EM
e A
l
You can access the Cluster Database backup and recoveryrelated tasks by clicking the
c
r a
Availability tab on the Cluster Database Home page. On the Availability tabbed page, you can
perform a range of backup and recovery operations using RMAN, such as scheduling backups,
O ly
performing recovery when necessary, and configuring backup and recovery settings. Also,

& On
there are links related to Oracle Secure Backup and Service management.

l
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 13 - 8
Configure RAC Recovery Settings with EM

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Configure RAC Recovery Settings with EM
e A
l
You can use Enterprise Manager to configure important recovery settings for your cluster
c
r a
database. On the Database Home page, click the Availability tab, and then click the Recovery
Settings link. From here, you can ensure that your database is in ARCHIVELOG mode and
configure flash recovery settings.
O ly
l & On
With a RAC database, if the Archive Log Destination setting is not the same for all instances,
the field appears blank, with a message indicating that instances have different settings for this

n a e
field. In this case, entering a location in this field sets the archive log location for all instances

t e r
Initialization Parameters page.
U s
of database. You can assign instance-specific values for an archive log destination by using the

I n
Note: You can run the ALTER DATABASE SQL statement to change the archiving mode in

c l e
RAC as long as the database is mounted by the local instance but not open in any instances.
You do not need to modify parameter settings to run this statement. Set the initialization

r a
parameters DB_RECOVERY_FILE_DEST and DB_RECOVERY_FILE_DEST_SIZE to the
same values on all instances to configure a flash recovery area in a RAC environment.
O
Oracle Database 11g: RAC Administration 13 - 9
Archived Redo File Conventions in RAC

Variable Description Example

%t Thread number, not padded log_1

%T Thread number, left-zero-padded log_0001

%s Log sequence number, not padded log_251

%S Log sequence number, left-zero-padded log_0000000251

%r Resetlogs identifier log_23452345

%R Padded resetlogs identifier log_0023452345

%t_%s_%r Using multiple variables log_1_251_23452345

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Archived Redo File Conventions in RAC
e A
l
For any archived redo log configuration, uniquely identify the archived redo logs with the
c
r a
LOG_ARCHIVE_FORMAT parameter. The format of this parameter is operating systemspecific
and it can include text strings, one or more variables, and a file name extension.
O ly
All of the thread parameters, in either uppercase or lowercase, are mandatory for RAC. This

l & On
enables the Oracle database to create unique names for archive logs across the incarnation.
This requirement is in effect when the COMPATIBLE parameter is set to 10.0 or greater. Use
a e
the %R or %r parameter to include the resetlogs identifier to avoid overwriting the logs from a
n
t e r U s
previous incarnation. If you do not specify a log format, the default is operating system
specific and includes %t, %s, and %r.

I n
As an example, if the instance associated with redo thread number 1 sets

as:
c l e
LOG_ARCHIVE_FORMAT to log_%t_%s_%r.arc, then its archived redo log files are named

r a
log_1_1000_23435343.arc
log_1_1001_23452345.arc

O
log_1_1002_23452345.arc
...

Oracle Database 11g: RAC Administration 13 - 10


Configure RAC Backup Settings with EM

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Configure RAC Backup Settings with EM
e A
l
Persistent backup settings can be configured using Enterprise Manager. On the Database
c
r a
Control Home page, click the Availability tab, and then click the Backup Settings link. You
can configure disk settings, such as the directory location of your disk backups and level of
O ly
parallelism. You can also choose the default backup type:
Backup set
Compressed backup set
l & On
Image copy
n a e
t e r U s
You can also specify important tape-related settings, such as the number of available tape
drives and vendor-specific media management parameters.

I n
c l e
r a
O
Oracle Database 11g: RAC Administration 13 - 11
Oracle Recovery Manager

RMAN provides the following


Recovery benefits for Real Application
Manager Clusters:
Recovery Can read cluster files or
catalog
Archived ASM files with no
log files Oracle configuration changes
Server
process Can access multiple
Stored
archive log destinations
scripts

Oracle
database
Backup
storage Snapshot
control file
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Oracle Recovery Manager
e A
l
Oracle Database provides RMAN for backing up and restoring the database. RMAN enables
c
r a
you to back up, restore, and recover data files, control files, SPFILEs, and archived redo logs.
You can run RMAN from the command line or you can use it from the Backup Manager in
O ly
Enterprise Manager. In addition, RMAN is the recommended backup and recovery tool if you

l & On
are using ASM. RMAN can use stored scripts, interactive scripts, or an interactive GUI front
end. When using RMAN with your RAC database, use stored scripts to initiate the backup and
a e
recovery processes from the most appropriate node.
n
t e r U s
If you use different Oracle Home locations for your RAC instances on each of your nodes,
create a snapshot control file in a location that exist on all your nodes. The snapshot control

I n
file is only needed on the nodes on which RMAN performs backups. The snapshot control file
does not need to be globally available to all instances in a RAC environment though.

c l e
You can use either a cluster file as well as a local directory that exists on each node in your

r a
cluster. Here is an example:
RMAN> CONFIGURE SNAPSHOT CONTROLFILE TO
O '/oracle/db_files/snaps/snap_prod1.cf';
For recovery, you must ensure that each recovery node can access the archive log files from all
instances by using one of the archive schemes discussed earlier, or make the archived logs
available to the recovering instance by copying them from another location.
Oracle Database 11g: RAC Administration 13 - 12
Configure RMAN Snapshot Control File Location

The snapshot control file path must be valid on every node


from which you might initiate an RMAN backup.
Configure the snapshot control file location in RMAN.
Determine the current location:
RMAN> SHOW SNAPSHOT CONTROLFILE NAME;
/u01/app/oracle/product/11.2.0/dbhome_1/dbs/snap_prod.f

You can use ASM or a shared file system location if you


prefer:
RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO
'+FRA/SNAP/snap_prod.cf';
RMAN> CONFIGURE SNAPSHOT CONTROLFILE NAME TO
'/ocfs2/oradata/dbs/scf/snap_prod.cf';

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Configure RMAN Snapshot Control File Location
e A
l
The snapshot control file is a temporary file that RMAN creates to resynchronize from a read-
c
r a
consistent version of the control file. RMAN needs a snapshot control file only when
resynchronizing with the recovery catalog or when making a backup of the current control file.
O ly
In a RAC database, the snapshot control file is created on the node that is making the backup.

l & On
You need to configure a default path and file name for these snapshot control files that are
valid on every node from which you might initiate an RMAN backup.

n a e
Run the following RMAN command to determine the configured location of the snapshot
control file:
t e r U s
I n
SHOW SNAPSHOT CONTROLFILE NAME
You can change the configured location of the snapshot control file. For example, on UNIX-

l e
based systems, you can specify the snapshot control file location as snap_prod.cf located in
c
the ASM disk group +FRA by entering the following at the RMAN prompt:

r a
CONFIGURE SNAPSHOT CONTROLFILE NAME TO '+FRA/SNAP/snap_prod.cf'

O
This command globally sets the configuration for the location of the snapshot control file
throughout your cluster database.
Note: The CONFIGURE command creates persistent settings across RMAN sessions.

Oracle Database 11g: RAC Administration 13 - 13


Configure Control File and SPFILE Autobackup

RMAN automatically creates a control file and SPFILE


backup after the BACKUP or COPY command:
RMAN> CONFIGURE CONTROLFILE AUTOBACKUP ON;

Change default location:


RMAN> CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR
DEVICE TYPE DISK TO '+DATA';

Location must be available to all nodes in your RAC


database.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Configure Control File and SPFILE Autobackup
e A
l
If you set CONFIGURE CONTROLFILE AUTOBACKUP to ON, RMAN automatically creates a
c
r a
control file and an SPFILE backup after you run the BACKUP or COPY command. RMAN can
also automatically restore an SPFILE if this is required to start an instance to perform
O ly
recovery. This means that the default location for the SPFILE must be available to all nodes in
your RAC database.

l & On
These features are important in disaster recovery because RMAN can restore the control file

n a e
even without a recovery catalog. RMAN can restore an autobackup of the control file even

t e r U s
after the loss of both the recovery catalog and the current control file.
You can change the default location that RMAN gives to this file with the CONFIGURE

I n
CONTROLFILE AUTOBACKUP FORMAT command. If you specify an absolute path name in this

c l e
command, this path must exist identically on all nodes that participate in backups.
Note: RMAN performs CONTROL FILE AUTOBACKUP on the first allocated channel.

r a
When you allocate multiple channels with different parameters (especially if you allocate a
channel with the CONNECT command), you must determine which channel will perform the
O
automatic backup. Always allocate the channel for the connected node first.

Oracle Database 11g: RAC Administration 13 - 14


Crosschecking on Multiple RAC Clusters Nodes

When crosschecking on multiple nodes make sure that all


backups can be accessed by every node in the cluster.
This allows you to allocate channels at any node in the
cluster during restore or crosscheck operations.
Otherwise you must allocate channels on multiple nodes
by providing the CONNECT option to the CONFIGURE
CHANNEL command.
If backups are not accessible because no channel was
configured on the node that can access those backups,
then those backups are marked EXPIRED.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Crosschecking on Multiple RAC Clusters Nodes
e A
l
When crosschecking on multiple RAC nodes, configure the cluster so that all backups can be
c
r a
accessed by every node, regardless of which node created the backup. When the cluster is
configured this way, you can allocate channels at any node in the cluster during restore or
crosscheck operations.
O ly
l & On
If you cannot configure the cluster so that each node can access all backups, then during
restore and crosscheck operations, you must allocate channels on multiple nodes by providing
a e
the CONNECT option to the CONFIGURE CHANNEL command, so that every backup can be
n
t e r U s
accessed by at least one node. If some backups are not accessible during crosscheck because
no channel was configured on the node that can access those backups, then those backups are

I n
marked EXPIRED in the RMAN repository after the crosscheck.

c l e
For example, you can use CONFIGURE CHANNEL ... CONNECT in an Oracle RAC
configuration in which tape backups are created on various nodes in the cluster and each

a
backup is only accessible on the node on which it is created.
r
O
Oracle Database 11g: RAC Administration 13 - 15
Channel Connections to Cluster Instances
When backing up, each allocated channel can connect to a
different instance in the cluster.
Instances to which the channels connect must be either all
mounted or all open.
When choosing a channel to use, RMAN gives preference
to the nodes with faster access to the data files that you
want to back up.
CONFIGURE DEFAULT DEVICE TYPE TO sbt;
CONFIGURE DEVICE TYPE sbt PARALLELISM 3;
CONFIGURE CHANNEL 1 DEVICE TYPE sbt CONNECT='sys/rac@orcl1';
CONFIGURE CHANNEL 2 DEVICE TYPE sbt CONNECT='sys/rac@orcl2';
CONFIGURE CHANNEL 3 DEVICE TYPE sbt CONNECT='sys/rac@orcl3';
OR
CONFIGURE DEFAULT DEVICE TYPE TO sbt;
CONFIGURE DEVICE TYPE sbt PARALLELISM 3;
CONFIGURE CHANNEL DEVICE TYPE sbt CONNECT='sys/rac@bkp_serv';
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Channel Connections to Cluster Instances
e A
l
When making backups in parallel, RMAN channels can connect to a different instance in the
c
r a
cluster. The examples in the slide illustrate two possible configurations:
If you want to dedicate channels to specific instances, you can control at which instance
O ly
the channels are allocated by using separate connect strings for each channel

l & On
configuration as shown by the first example.
If you define a special service for your backup and recovery jobs, you can use the second
a e
example shown in the slide. If you configure this service with load balancing turned on,
n
t e r U s
then the channels are allocated at a node as decided by the load balancing algorithm.
During a backup, the instances to which the channels connect must be either all mounted or all

I n
open. For example, if the orcl1 instance has the database mounted whereas the orcl2 and
orcl3 instances have the database open, then the backup fails.

c l e
In some cluster database configurations, some nodes of the cluster have faster access to certain

r a
data files than to other data files. RMAN automatically detects this, which is known as node
affinity awareness. When deciding which channel to use to back up a particular data file,
O
RMAN gives preference to the nodes with faster access to the data files that you want to back
up.
Note: You can protect the username and password in an Oracle wallet by using the mkstore
utility as outlined in MetaLink note 340559.1.
Oracle Database 11g: RAC Administration 13 - 16
RMAN Channel Support for the Grid

RAC allows the use of nondeterministic connect strings.


RMAN can use connect strings that are not bound to a
specific instance in the Grid environment.
It simplifies the use of parallelism with RMAN in a RAC
environment.
It uses the load-balancing characteristics of the Grid
environment.
Channels connect to RAC instances that are the least
loaded.

CONFIGURE DEFAULT DEVICE TYPE TO sbt;


CONFIGURE DEVICE TYPE sbt PARALLELISM 3;

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
RMAN Channel Support for the Grid
e A
c l
RAC allows the use of nondeterministic connect strings that can connect to different instances
based on RAC features, such as load balancing. Therefore, to support RAC, the RMAN polling
r a
mechanism no longer depends on deterministic connect strings, and makes it possible to use

O ly
RMAN with connect strings that are not bound to a specific instance in the Grid environment.
Previously, if you wanted to use RMAN parallelism and spread a job between many instances,
& On
you had to manually allocate an RMAN channel for each instance. To use dynamic channel
l
n a e
allocation, you do not need separate CONFIGURE CHANNEL CONNECT statements
anymore. You only need to define your degree of parallelism by using a command such as

t e r U s
CONFIGURE DEVICE TYPE disk PARALLELISM, and then run backup or restore
commands. RMAN then automatically connects to different instances and does the job in

I n
parallel. The Grid environment selects the instances that RMAN connects to, based on load
balancing. As a result of this, configuring RMAN parallelism in a RAC environment becomes
l e
as simple as setting it up in a non-RAC environment. By configuring parallelism when backing
c
up or recovering a RAC database, RMAN channels are dynamically allocated across all RAC

r a
instances.

O
Note: RMAN has no control over the selection of the instances. If you require a guaranteed
connection to an instance, you should provide a connect string that can connect only to the
required instance.

Oracle Database 11g: RAC Administration 13 - 17


RMAN Default Autolocation

Recovery Manager autolocates the following files:


Backup pieces
Archived redo logs during backup
Data file or control file copies
If local archiving is used, a node can read only those
archived logs that were generated on that node.
When restoring, a channel connected to a specific node
restores only those files that were backed up to the node.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
RMAN Default Autolocation
e A
l
Recovery Manager automatically discovers which nodes of a RAC configuration can access
c
files:
r a
the files that you want to back up or restore. Recovery Manager autolocates the following

O ly
Backup pieces during backup or restore

l & On
Archived redo logs during backup
Data file or control file copies during backup or restore

n a e
If you use a noncluster file system local archiving scheme, a node can read only those archived

t e r U s
redo logs that were generated by an instance on that node. RMAN never attempts to back up
archived redo logs on a channel that it cannot read.

I n
During a restore operation, RMAN automatically performs the autolocation of backups. A

c l e
channel connected to a specific node attempts to restore only those files that were backed up to
the node. For example, assume that log sequence 1001 is backed up to the drive attached to

r a
node 1, whereas log 1002 is backed up to the drive attached to node 2. If you then allocate
channels that connect to each node, the channel connected to node 1 can restore log 1001 (but
O
not 1002), and the channel connected to node 2 can restore log 1002 (but not 1001).

Oracle Database 11g: RAC Administration 13 - 18


Distribution of Backups

Several possible backup configurations for RAC:


A dedicated backup server performs and manages
backups for the cluster and the cluster database.
One node has access to a local backup appliance and
performs and manages backups for the cluster database.
Each node has access to a local backup appliance and
can write to its own local backup media.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Distribution of Backups
e A
l
When configuring the backup options for RAC, you have several possible configurations:
c
r a
Network backup server: A dedicated backup server performs and manages backups for
the cluster and the cluster database. None of the nodes have local backup appliances.
O ly
One local drive: One node has access to a local backup appliance and performs and

l & On
manages backups for the cluster database. All nodes of the cluster should be on a cluster
file system to be able to read all data files, archived redo logs, and SPFILEs. It is
a e
recommended that you do not use the noncluster file system archiving scheme if you
n
t e r U s
have backup media on only one local drive.
Multiple drives: Each node has access to a local backup appliance and can write to its

I n
own local backup media.
In the cluster file system scheme, any node can access all the data files, archived redo logs,
l e
and SPFILEs. In the noncluster file system scheme, you must write the backup script so that
c
r a
the backup is distributed to the correct drive and path for each node. For example, node 1 can
back up the archived redo logs whose path names begin with /arc_dest_1, node 2 can back
O
up the archived redo logs whose path names begin with /arc_dest_2, and node 3 can back
up the archived redo logs whose path names begin with /arc_dest_3.

Oracle Database 11g: RAC Administration 13 - 19


One Local Drive Shared Storage Backup Scheme

RMAN> CONFIGURE DEVICE TYPE sbt PARALLELISM 1;


RMAN> CONFIGURE DEFAULT DEVICE TYPE TO sbt;

RMAN> BACKUP DATABASE PLUS ARCHIVELOG DELETE INPUT;

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
One Local Drive CFS Backup Scheme
e A
l
In a shared storage backup scheme, each node in the cluster has read access to all the data
c
r a
files, archived redo logs, and SPFILEs. This includes Automated Storage Management (ASM),
cluster file systems, and Network Attached Storage (NAS).
O ly
When backing up to only one local drive in the cluster file system backup scheme, it is

l & On
assumed that only one node in the cluster has a local backup appliance such as a tape drive. In
this case, run the following one-time configuration commands:

n a e
CONFIGURE DEVICE TYPE sbt PARALLELISM 1;

t e r U s
CONFIGURE DEFAULT DEVICE TYPE TO sbt;
Because any node performing the backup has read/write access to the archived redo logs

I n
written by the other nodes, the backup script for any node is simple:

c l e
BACKUP DATABASE PLUS ARCHIVELOG DELETE INPUT;
In this case, the tape drive receives all data files, archived redo logs, and SPFILEs.

r a
O
Oracle Database 11g: RAC Administration 13 - 20
Multiple Drives Shared Storage Backup Scheme

CONFIGURE DEVICE TYPE sbt PARALLELISM 2;


CONFIGURE DEFAULT DEVICE TYPE TO sbt;
CONFIGURE CHANNEL 1 DEVICE TYPE sbt CONNECT 'usr1/pwd1@n1';
CONFIGURE CHANNEL 2 DEVICE TYPE sbt CONNECT 'usr2/pwd2@n2';

BACKUP DATABASE PLUS ARCHIVELOG DELETE INPUT;

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Multiple Drives CFS Backup Scheme
e A
l
When backing up to multiple drives in the shared storage backup scheme, it is assumed that
c
r a
each node in the cluster has its own local tape drive. Perform the following one-time
configuration so that one channel is configured for each node in the cluster. For example, enter
the following at the RMAN prompt:
O ly
CONFIGURE
CONFIGURE
l & On
DEVICE TYPE sbt PARALLELISM 2;
DEFAULT DEVICE TYPE TO sbt;
CONFIGURE
n a e
CHANNEL 1 DEVICE TYPE sbt CONNECT 'user1/passwd1@node1';
CONFIGURE

t e r s
CHANNEL 2 DEVICE TYPE sbt CONNECT 'user2/passwd2@node2';
Similarly, you can perform this configuration for a device type of DISK. The following
U
I n
backup script, which you can run from any node in the cluster, distributes the data files,
archived redo logs, and SPFILE backups among the backup drives:

l e
BACKUP DATABASE PLUS ARCHIVELOG DELETE INPUT;
c
r a
O
Oracle Database 11g: RAC Administration 13 - 21
Restoring and Recovering

Media recovery may require one or more archived log files


from each thread.
The RMAN RECOVER command automatically restores and
applies the required archived logs.
Archive logs may be restored to any node performing the
restore and recover operation.
Logs must be readable from the node performing the
restore and recovery activity.
Recovery processes request additional threads enabled
during the recovery period.
Recovery processes notify you of threads no longer
needed because they were disabled.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Restoring and Recovering
e A
l
Media recovery of a database that is accessed by RAC may require at least one archived log
c
r a
file for each thread. However, if a threads online redo log contains enough recovery
information, restoring archived log files for any thread is unnecessary.
O ly
If you use RMAN for media recovery and you share archive log directories, you can change

l & On
the destination of the automatic restoration of archive logs with the SET clause to restore the
files to a local directory of the node where you begin recovery. If you backed up the archive

n a e
logs from each node without using a central media management system, you must first restore

t e r U s
all the log files from the remote nodes and move them to the host from which you will start
recovery with RMAN. However, if you backed up each nodes log files using a central media

I n
management system, you can use RMANs AUTOLOCATE feature. This enables you to
recover a database using the local tape drive on the remote node.

c l e
If recovery reaches a time when an additional thread was enabled, the recovery process

r a
requests the archived log file for that thread. If you are using a backup control file, when all
archive log files are exhausted, you may need to redirect the recovery process to the online
O
redo log files to complete recovery. If recovery reaches a time when a thread was disabled, the
process informs you that the log file for that thread is no longer needed.

Oracle Database 11g: RAC Administration 13 - 22


Quiz

Which of the following statements regarding media recovery in


RAC is not true?
1. Media recovery must be user initiated through a client
application.
2. RMAN media recovery procedures for RAC are quite
different from those for single-instance environments.
3. The node that performs the recovery must be able to
restore all the required data files.
4. The recovering node must be able to either read all
required archived redo logs on disk or restore them from
backups.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 2
e A
Statement 2 is not correct.
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 13 - 23
Quiz

To use a fast recovery area in RAC, you must place it on an


ASM disk group, a cluster file system, or on a shared directory
that is configured through certified NFS for each RAC instance.
1. True
2. False

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 1
e A
The statement above is true.
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 13 - 24
Summary

In this lesson, you should have learned how to configure the


following:
The RAC database to use ARCHIVELOG mode and the fast
recovery area
RMAN for the RAC environment

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 13 - 25
Practice 13 Overview

This practice covers the following topics:


Configuring the archive log mode
Configuring specific instance connection strings
Configuring RMAN and performing parallel backups

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 13 - 26
RAC Database Monitoring and Tuning

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Objectives

After completing this lesson, you should be able to:


Determine RAC-specific tuning components
Tune instance recovery in RAC
Determine RAC-specific wait events, global enqueues, and
system statistics
Implement the most common RAC tuning tips
Use the Cluster Database Performance pages
Use the Automatic Workload Repository (AWR) in RAC
Use Automatic Database Diagnostic Monitor (ADDM) in
RAC

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 2
CPU and Wait Time Tuning Dimensions

CPU
time
Possibly Scalable
needs SQL application
tuning

Scalable Needs No gain achieved


instance/RAC by adding
application
tuning CPUs/nodes

Wait
time

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
CPU and Wait Time Tuning Dimensions
e A
l
When tuning your system, it is important that you compare the CPU time with the wait time of
c
r a
your system. Comparing CPU time with wait time helps to determine how much of the
response time is spent on useful work and how much on waiting for resources potentially held
by other processes.
O ly
l & On
As a general rule, the systems where CPU time is dominant usually need less tuning than the
ones where wait time is dominant. Alternatively, heavy CPU usage can be caused by badly
written SQL statements.
n a e
e r s
Although the proportion of CPU time to wait time always tends to decrease as load on the
t U
system increases, steep increases in wait time are a sign of contention and must be addressed

I
for good scalability.n
l e
Adding more CPUs to a node, or nodes to a cluster, would provide very limited benefit under

c
contention. Conversely, a system where the proportion of CPU time to wait time does not

r a
decrease significantly as load increases can scale better, and would most likely benefit from
adding CPUs or Real Application Clusters (RAC) instances if needed.
O
Note: Automatic Workload Repository (AWR) reports display CPU time together with wait
time in the Top 5 Timed Events section, if the CPU time portion is among the top five events.

Oracle Database 11g: RAC Administration 14 - 3


RAC-Specific Tuning

Tune for a single instance first.


Tune for RAC:
Instance recovery
Interconnect traffic
Point of serialization can be exacerbated
RAC-reactive tuning tools:
Certain combinations
Specific wait events are characteristic of
System and enqueue statistics well-known tuning cases.
Enterprise Manager performance pages
Statspack and AWR reports
RAC-proactive tuning tools:
AWR snapshots
ADDM reports m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
RAC-Specific Tuning
e A
l
Although there are specific tuning areas for RAC, such as instance recovery and interconnect
c
must be your starting point.
r a
traffic, you get most benefits by tuning your system like a single-instance system. At least, this

O ly
Obviously, if you have serialization issues in a single-instance environment, these may be
exacerbated with RAC.
l & On
n a e
As shown in the slide, you have basically the same tuning tools with RAC as with a single-
instance system. However, certain combinations of specific wait events and statistics are well-
known RAC tuning cases.
t e r U s
I n
In this lesson, you see some of those specific combinations, as well as the RAC-specific
information that you can get from the Enterprise Manager performance pages, and Statspack

l e
and AWR reports. Finally, you see the RAC-specific information that you can get from the

c
Automatic Database Diagnostic Monitor (ADDM).

r a
O
Oracle Database 11g: RAC Administration 14 - 4
RAC and Instance or Crash Recovery

Use information
Remaster for other caches
enqueue LMS
resources Remaster recovers
1 cache GRD
resources
2

Build re-
SMON covery set
recovers Resource
3 claim
the
database Merge failed 4 Roll forward
redo threads recovery set

Recovery time
5

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
RAC and Instance or Crash Recovery
e A
l
When an instance fails and the failure is detected by another instance, the second instance
c
performs the following recovery steps:
r a
1. During the first phase of recovery, Global Enqueue Services remasters the enqueues.

O ly
2. The Global Cache Services (GCS) remasters its resources. The GCS processes remaster
only those resources that lose their masters. During this time, all GCS resource requests

l & On
and write requests are temporarily suspended. However, transactions can continue to

resources.
n a e
modify data blocks as long as these transactions have already acquired the necessary

e r s
3. After enqueues are reconfigured, one of the surviving instances can grab the Instance
t U
Recovery enqueue. Therefore, at the same time as GCS resources are remastered, SMON

I n
determines the set of blocks that need recovery. This set is called the recovery set.

l e
Because, with Cache Fusion, an instance ships the contents of its blocks to the requesting
instance without writing the blocks to the disk, the on-disk version of the blocks may not
c
r acontain the changes that are made by either instance. This implies that SMON needs to
merge the content of all the online redo logs of each failed instance to determine the
O recovery set. This is because one failed thread might contain a hole in the redo that needs
to be applied to a particular block. So, redo threads of failed instances cannot be applied
serially. Also, redo threads of surviving instances are not needed for recovery because
SMON could use past or current images of their corresponding buffer caches.

Oracle Database 11g: RAC Administration 14 - 5


RAC and Instance or Crash Recovery (continued)
4. Buffer space for recovery is allocated and the resources that were identified in the
previous reading of the redo logs are claimed as recovery resources. This is done to avoid
other instances to access those resources.
5. All resources required for subsequent processing have been acquired and the Global
Resource Directory (GRD) is now unfrozen. Any data blocks that are not in recovery can
now be accessed. Note that the system is already partially available.
Then, assuming that there are past images or current images of blocks to be recovered in
other caches in the cluster database, the most recent image is the starting point of
recovery for these particular blocks. If neither the past image buffers nor the current
buffer for a data block is in any of the surviving instances caches, then SMON performs
a log merge of the failed instances. SMON recovers and writes each block identified in
step 3, releasing the recovery resources immediately after block recovery so that more
blocks become available as recovery proceeds. Refer to the section titled Global Cache
Coordination: Example in this lesson for more information about past images.
After all the blocks have been recovered and the recovery resources have been released, the
system is again fully available.
In summary, the recovered database or the recovered portions of the database becomes
available earlier, and before the completion of the entire recovery sequence. This makes the
system available sooner and it makes recovery more scalable.
Note: The performance overhead of a log merge is proportional to the number of failed
m y
e
instances and to the size of the amount of redo written in the redo logs for each instance.
d
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 6
Instance Recovery and Database Availability

Full A G H
Database availability

Partial B F

4
2
1 3
None C D E

Elapsed time

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Instance Recovery and Database Availability
e A
l
The graphic illustrates the degree of database availability during each step of Oracle instance
c
recovery:
r
A. Real Application Clusters is running on multiple nodes.a
B. Node failure is detected.
O ly
l & On
C. The enqueue part of the GRD is reconfigured; resource management is redistributed to
the surviving nodes. This operation occurs relatively quickly.
a e
D. The cache part of the GRD is reconfigured and SMON reads the redo log of the failed
n
t e r U s
instance to identify the database blocks that it needs to recover.
E. SMON issues the GRD requests to obtain all the database blocks it needs for recovery.

I n
After the requests are complete, all other blocks are accessible.
F. The Oracle server performs roll forward recovery. Redo logs of the failed threads are

l e
applied to the database, and blocks are available right after their recovery is completed.
c
G. The Oracle server performs rollback recovery. Undo blocks are applied to the database

r a for all uncommitted transactions.

O H. Instance recovery is complete and all data is accessible.


Note: The dashed line represents the blocks identified in step 2 in the previous slide. Also, the
dotted steps represent the ones identified in the previous slide.

Oracle Database 11g: RAC Administration 14 - 7


Instance Recovery and RAC

Instance startup
Instance + Instance
crashes crash recovery opens
FAST_START_MTTR_TARGET
Instance
starts

Rolling
Instance forward
recovery ends
Instance
crashes first pass + lock claim

FAST_START_MTTR_TARGET

Instance
recovery V$INSTANCE_RECOVERY.ESTD_CLUSTER_AVAILABLE_TIME

starts

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Instance Recovery and RAC
e A
l
In a single-instance environment, the instance startup combined with the crash recovery time is
c
r a
controlled by the setting of the FAST_START_MTTR_TARGET initialization parameter. You can
set its value if you want incremental checkpointing to be more aggressive than the autotuned
O ly
checkpointing. However, this is at the expense of a much higher I/O overhead.

l & On
In a RAC environment, including the startup time of the instance in this calculation is useless
because one of the surviving instances is doing the recovery.

n a e
In a RAC environment, it is possible to monitor the estimated target, in seconds, for the

e r s
duration from the start of instance recovery to the time when GCD is open for lock requests for
t U
blocks not needed for recovery. This estimation is published in the V$INSTANCE_RECOVERY

I n
view through the ESTD_CLUSTER_AVAILABLE_TIME column. Basically, you can monitor the

c l e
time your cluster is frozen during instance recovery situations.
In a RAC environment, the FAST_START_MTTR_TARGET initialization parameter is used to

r a
bound the entire instance recovery time, assuming it is instance recovery for single instance
death.
O
Note: If you really want to have small instance recovery time by setting
FAST_START_MTTR_TARGET, you can safely ignore the alert log messages indicating to raise its
value.

Oracle Database 11g: RAC Administration 14 - 8


Instance Recovery and RAC

Use parallel instance recovery.


Set PARALLEL_MIN_SERVERS.
Use asynchronous input/output (I/O).
Increase the size of the default buffer cache.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Instance Recovery and RAC (continued)
e A
l
Here are some guidelines you can use to make sure that instance recovery in your RAC
c
environment is faster:
r a
Use parallel instance recovery by setting RECOVERY_PARALLISM.
O ly
Set PARALLEL_MIN_SERVERS to CPU_COUNT-1. This will prespawn recovery
slaves at startup time.

l & On
If a system fails when there are uncommitted parallel DML or DDL transactions, you can
a e
speed up transaction recovery during startup by setting the
n
t e r U s
FAST_START_PARALLEL_ROLLBACK parameter.
Using asynchronous I/O is one of the most crucial factors in recovery time. The first-pass

I n
log read uses asynchronous I/O.
Instance recovery uses 50 percent of the default buffer cache for recovery buffers. If this

l e
is not enough, some of the steps of instance recovery will be done in several passes. You
c
should be able to identify such situations by looking at your alert.log file. In that

r a case, you should increase the size of your default buffer cache.

O
Oracle Database 11g: RAC Administration 14 - 9
Analyzing Cache Fusion Impact in RAC

The cost of block access and cache coherency is


represented by:
Global Cache Services statistics
Global Cache Services wait events
The response time for cache fusion transfers is determined
by:
Overhead of the physical interconnect components
IPC protocol
GCS protocol
The response time is not generally affected by disk I/O
factors.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Analyzing Cache Fusion Impact in RAC
e A
l
The effect of accessing blocks in the global cache and maintaining cache coherency is
c
represented by:
r a
The Global Cache Services statistics for current and cr blocksfor example, gc current
O ly
blocks received, gc cr blocks received, and so on

and so on
l & On
The Global Cache Services wait events for gc current block 3-way, gc cr grant 2-way,

n a e
The response time for cache fusion transfers is determined by the messaging time and

t e r U s
processing time imposed by the physical interconnect components, the IPC protocol, and the
GCS protocol. It is not affected by disk input/output (I/O) factors other than occasional log

I n
writes. The cache fusion protocol does not require I/O to data files in order to guarantee cache
coherency, and RAC inherently does not cause any more I/O to disk than a nonclustered
instance.
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 10
Typical Latencies for RAC Operations
AWR Report Latency Name Lower Typical Upper
Bound Bound
Average time to process cr block request 0.1 1 10
Avg global cache cr block receive time (ms) 0.3 4 12
Average time to process current block request 0.1 3 23
Avg global cache current block receive time(ms) 0.3 8 30

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Typical Latencies for RAC Operations
e A
l
In a RAC AWR report, there is a table in the RAC Statistics section containing average times
c
r a
(latencies) for some Global Cache Services and Global Enqueue Services operations. This
table is shown in the slide and is called Global Cache and Enqueue Services: Workload
O ly
Characteristics. Those latencies should be monitored over time, and significant increases in

l & On
their values should be investigated. The table presents some typical values, based on empirical
observations. Factors that may cause variations to those latencies include:
a e
Utilization of the IPC protocol. User-mode IPC protocols are faster, but only Tru64s
n
t e r
RDG is recommended for use.

U s
Scheduling delays, when the system is under high CPU utilization

I n
Log flushes for current blocks served
Other RAC latencies in AWR reports are mostly derived from V$GES_STATISTICS and
l e
may be useful for debugging purposes, but do not require frequent monitoring.
c
r a
Note: The time to process consistent read (CR) block request in the cache corresponds to
(build time + flush time + send time), and the time to process current block
O
request in the cache corresponds to (pin time + flush time + send time).

Oracle Database 11g: RAC Administration 14 - 11


Wait Events for RAC

Wait events help to analyze what sessions are waiting for.


Wait times are attributed to events that reflect the outcome
of a request:
Placeholders while waiting
Precise events after waiting
Global cache waits are summarized in a broader category
called Cluster Wait Class.
These wait events are used in ADDM to enable Cache
Fusion diagnostics.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Wait Events for RAC
e A
l
Analyzing what sessions are waiting for is an important method to determine where time is
c
r a
spent. In RAC, the wait time is attributed to an event that reflects the exact outcome of a
request. For example, when a session on an instance is looking for a block in the global cache,
O ly
it does not know whether it will receive the data cached by another instance or whether it will

l & On
receive a message to read from disk. The wait events for the global cache convey precise
information and wait for global cache blocks or messages. They are mainly categorized by the
following:
n a e
t e r U s
Summarized in a broader category called Cluster Wait Class
Temporarily represented by a placeholder event that is active while waiting for a block

I n
Attributed to precise events when the outcome of the request is known
The wait events for RAC convey information valuable for performance analysis. They are
l e
used in ADDM to enable precise diagnostics of the impact of cache fusion.
c
r a
O
Oracle Database 11g: RAC Administration 14 - 12
Wait Event Views

Total waits for an event V$SYSTEM_EVENT

Waits for a wait event class V$SESSION_WAIT_CLASS


by a session

Waits for an event by a session V$SESSION_EVENT

Activity of recent active sessions V$ACTIVE_SESSION_HISTORY

Last 10 wait events


V$SESSION_WAIT_HISTORY
for each active session
Events for which
active sessions are waiting V$SESSION_WAIT

Identify SQL statements impacted


by interconnect latencies
V$SQLSTATS
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Wait Event Views
e A
l
When it takes some time to acquire resources because of the total path length and latency for
c
r a
requests, processes sleep to avoid spinning for indeterminate periods of time. When the
process decides to wait, it wakes up either after a specified timer value expires (timeout) or
O ly
when the event it is waiting for occurs and the process is posted. The wait events are recorded

l & On
and aggregated in the views shown in the slide. The first three are aggregations of wait times,
timeouts, and the number of times waited for a particular event, whereas the rest enable the
a e
monitoring of waiting sessions in real time, including a history of recent events waited for.
n
t e r U s
The individual events distinguish themselves by their names and the parameters that they
assume. For most of the global cache wait events, the parameters include file number, block

I n
number, the block class, and access mode dispositions, such as mode held and requested. The
wait times for events presented and aggregated in these views are very useful when debugging
l e
response time performance issues. Note that the time waited is cumulative, and that the event
c
r a
with the highest score is not necessarily a problem. However, if the available CPU power
cannot be maximized, or response times for an application are too high, the top wait events
O
provide valuable performance diagnostics.
Note: Use the CLUSTER_WAIT_TIME column in V$SQLSTATS to identify SQL statements
impacted by interconnect latencies, or run an ADDM report on the corresponding AWR
snapshot.
Oracle Database 11g: RAC Administration 14 - 13
Global Cache Wait Events: Overview

Just requested gc [current/cr] [multiblock] request


(placeholder)

gc [current/cr] block [2/3]-way gc [current/cr] block busy


Received after two or three network Received but not sent immediately
hops, immediately after request

gc [current/cr] grant 2-way gc current grant busy


Not received and not mastered Not received and not mastered
locally. Grant received immediately. locally. Grant received with delay.

gc [current/cr] [block/grant] congested gc [current/cr] [failure/retry]


Block or grant received with delay Not received because of failure
because of CPU or memory lack

gc buffer busy
Block arrival time
m y
less than buffer pin time

d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Global Cache Wait Events: Overview
e A
l
The main global cache wait events are described briefly in the slide:
c

r a
gc current/cr request: These wait events are relevant only while a gc request
for a cr block or current buffer is in progress. They act as placeholders until the request
completes.
O ly

l & On
gc [current/cr] block [2/3]-way: A current or cr block is requested and
received after two or three network hops. The request is processed immediately; the
a e
block is not busy or congested.
n

t e r U s
gc [current/cr] block busy: A current or cr block is requested and received,
but is not sent immediately by LMS because some special condition that delayed the
sending was found.
I n
gc [current/cr] grant 2-way: A current or cr block is requested and a grant
e

c l
message received. The grant is given without any significant delays. If the block is not in
its local cache, a current or cr grant is followed by a disk read on the requesting instance.

r

agc current grant busy: A current block is requested and a grant message

O received. The busy hint implies that the request is blocked because others are ahead of it
or it cannot be handled immediately.
Note: For dynamic remastering, two events are of most importance: gc remaster and gc
quiesce. They can be symptoms of the impact of remastering on the running processes.

Oracle Database 11g: RAC Administration 14 - 14


Global Cache Wait Events: Overview (continued)
gc [current/cr] [block/grant] congested: A current or cr block is
requested and a block or grant message received. The congested hint implies that the
request spent more than 1 ms in internal queues.
gc [current/cr] [failure/retry]: A block is requested and a failure status
received or some other exceptional event has occurred.
gc buffer busy: If the time between buffer accesses becomes less than the time the
buffer is pinned in memory, the buffer containing a block is said to become busy and as a
result interested users may have to wait for it to be unpinned.
Note: For more information, refer to Oracle Database Reference.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 15
2-way Block Request: Example

Wait:
gc current block request
1
FGP
Direct send
Node1

SGA2 2

LGWR:
LGWR
Log sync

SGA1

Block transfer
Wait complete:
gc current block 2-way
3
LMS
Node2

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
2-way Block Request: Example
e A
l
This slide shows you what typically happens when the master instance requests a block that is
c
r a
not cached locally. Here it is supposed that the master instance is called SGA1, and SGA2
contains the requested block. The scenario is as follows:
O ly
1. SGA1 sends a direct request to SGA2. So SGA1 waits on the gc current block
request event.

l & On
2. When SGA2 receives the request, its local LGWR process may need to flush some
a e
recovery information to its local redo log files. For example, if the cached block is
n
t e r U s
frequently changed, and the changes have not been logged yet, LMS would have to ask
LGWR to flush the log before it can ship the block. This may add a delay to the serving

I n
of the block and may show up in the requesting node as a busy wait.
3. Then, SGA2 sends the requested block to SGA1. When the block arrives in SGA1, the

l e
wait event is complete, and is reflected as gc current block 2-way.
c
r a
Note: Using the notation R = time at requestor, W = wire time and transfer delay, and S = time
at server, the total time for a round-trip would be:
O
R(send) + W(small msg) + S(process msg,process block,send) + W(block) + R(receive block)

Oracle Database 11g: RAC Administration 14 - 16


3-way Block Request: Example

Wait:
gc current block request
1 LMS 2
FGP Resource
Direct Master
message
Node1
SGA2 3

LGWR

SGA1

LMS
Block transfer

Wait complete: 4
Node2

m y
gc current block 3-way

d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
3-way Block Request: Example
e A
l
This is a modified scenario for a cluster with more than two nodes. It is very similar to the
c
r a
previous one. However, the master for this block is on a node that is different from that of the
requestor, and where the block is cached. Thus, the request must be forwarded. There is an
O ly
additional delay for one message and the processing at the master node:

l & On
R(send) + W(small msg) + S(process msg,send) + W(small msg) + S(process msg,process
block,send) + W(block) + R(receive block)

n a e
While a remote read is pending, any process on the requesting instance that is trying to write

t e r U s
or read the data cached in the buffer has to wait for a gc buffer busy. The buffer remains
globally busy until the block arrives.

I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 17
2-way Grant: Example

Wait:
gc current block request
1 LMS
FGP Resource
Direct
Master
message

SGA2
Node1

2
SGA1
3
Node2

Wait complete:
Grant message

m y
gc current grant 2-way

d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
2-way Grant: Example
e A
l
In this scenario, a grant message is sent by the master because the requested block is not
c
cached in any instance.
r a
If the local instance is the resource master, the grant happens immediately. If not, the grant is
O ly
always 2-way, regardless of the number of instances in the cluster.

& On
The grant messages are small. For every block read from the disk, a grant has to be received
l
a e
before the I/O is initiated, which adds the latency of the grant round-trip to the disk latency:

n
t e r
R(send) + W(small msg) + S(process msg,send) + W(small msg) + R(receive block)

U s
The round-trip looks similar to a 2-way block round-trip, with the difference that the wire time

n
is determined by a small message, and the processing does not involve the buffer cache.
I
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 18
Global Enqueue Waits: Overview

Enqueues are synchronous.


Enqueues are global resources in RAC.
The most frequent waits are for:

TX TM

US HW

TA SQ

The waits may constitute serious serialization points.


m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Global Enqueue Waits: Overview
e A
l
An enqueue wait is not RAC specific, but involves a global lock operation when RAC is
c
r a
enabled. Most of the global requests for enqueues are synchronous, and foreground processes
wait for them. Therefore, contention on enqueues in RAC is more visible than in single-

O ly
instance environments. Most waits for enqueues occur for enqueues of the following types:
TX: Transaction enqueue; used for transaction demarcation and tracking

l & On
TM: Table or partition enqueue; used to protect table definitions during DML operations

a e
HW: High-water mark enqueue; acquired to synchronize a new block operation
n
t e r
SQ: Sequence enqueue; used to serialize incrementing of an Oracle sequence number
s
US: Undo segment enqueue; mainly used by the Automatic Undo Management (AUM)
U
feature
I n
TA: Enqueue used mainly for transaction recovery as part of instance recovery

l e
In all of the cases above, the waits are synchronous and may constitute serious serialization
c
points that can be exacerbated in a RAC environment.

r a
Note: The enqueue wait events specify the resource name and a reason for the waitfor

O
example, TX Enqueue index block split. This makes diagnostics of enqueue waits easier.

Oracle Database 11g: RAC Administration 14 - 19


Session and System Statistics

Use V$SYSSTAT to characterize the workload.


Use V$SESSTAT to monitor important sessions.
V$SEGMENT_STATISTICS includes RAC statistics.
RAC-relevant statistic groups are:
Global Cache Service statistics
Global Enqueue Service statistics
Statistics for messages sent
V$ENQUEUE_STATISTICS determines the enqueue with
the highest impact.
V$INSTANCE_CACHE_TRANSFER breaks down GCS
statistics into block classes.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Session and System Statistics
e A
l
Using system statistics based on V$SYSSTAT enables characterization of the database activity
c
r a
based on averages. It is the basis for many metrics and ratios used in various tools and
methods, such as AWR, Statspack, and Database Control.
O ly
In order to drill down to individual sessions or groups of sessions, V$SESSTAT is useful when

l & On
the important session identifiers to monitor are known. Its usefulness is enhanced if an
application fills in the MODULE and ACTION columns in V$SESSION.

n a e
V$SEGMENT_STATISTICS is useful for RAC because it also tracks the number of CR and

t e r U s
current blocks received by the object.
The RAC-relevant statistics can be grouped into:
I n
Global Cache Service statistics: gc cr blocks received, gc cr block receive time, and so
on
l e
Global Enqueue Service statistics: global enqueue gets, and so on
c
r a
Statistics for messages sent: gcs messages sent and ges messages sent
V$ENQUEUE_STATISTICS can be queried to determine which enqueue has the highest
O
impact on database service times and eventually, response times.
V$INSTANCE_CACHE_TRANSFER indicates how many current and CR blocks per block
class are received from each instance, including how many transfers incurred a delay.
Note: For more information about statistics, refer to Oracle Database Reference.
Oracle Database 11g: RAC Administration 14 - 20
Most Common RAC Tuning Tips

Application tuning is often the most beneficial!


Reduce long full-table scans in OLTP systems.
Use Automatic Segment Space Management (ASSM).
Increase sequence caches.
Use partitioning to reduce interinstance traffic.
Avoid unnecessary parsing.
Minimize locking usage.
Remove unselective indexes.
Configure interconnect properly.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Most Common RAC Tuning Tips
e A
l
In any database system, RAC or single instance, the most significant performance gains are
c
r a
usually obtained from traditional application-tuning techniques. The benefits of those
techniques are even more remarkable in a RAC database. In addition to traditional application
O ly
tuning, some of the techniques that are particularly important for RAC include the following:

l & On
Try to avoid long full-table scans to minimize GCS requests. The overhead caused by the
global CR requests in this scenario is because when queries result in local cache misses,
a e
an attempt is first made to find the data in another cache, based on the assumption that
n
t e r U s
the chance is high that another instance has cached the block.
Automatic Segment Space Management can provide instance affinity to table blocks.

I n
Increasing sequence caches improves instance affinity to index keys deriving their values
from sequences. That technique may result in significant performance gains for multi-

l e
instance insert-intensive applications.
c
Range or list partitioning may be very effective in conjunction with data-dependent

r arouting, if the workload can be directed to modify a particular range of values from a

O particular instance.
Hash partitioning may help to reduce buffer busy contention by making buffer access
distribution patterns sparser, enabling more buffers to be available for concurrent access.

Oracle Database 11g: RAC Administration 14 - 21


Most Common RAC Tuning Tips (continued)
In RAC, library cache and row cache operations are globally coordinated. So, excessive
parsing means additional interconnect traffic. Library cache locks are heavily used, in
particular by applications using PL/SQL or Advanced Queuing. Library cache locks are
acquired in exclusive mode whenever a package or procedure has to be recompiled.
Because transaction locks are globally coordinated, they also deserve special attention in
RAC. For example, using tables instead of Oracle sequences to generate unique numbers
is not recommended because it may cause severe contention even for a single instance
system.
Indexes that are not selective do not improve query performance, but can degrade DML
performance. In RAC, unselective index blocks may be subject to inter-instance
contention, increasing the frequency of cache transfers for indexes belonging to insert-
intensive tables.
Always verify that you use a private network for your interconnect, and that your private
network is configured properly. Ensure that a network link is operating in full duplex
mode. Ensure that your network interface and Ethernet switches support MTU size of 9
KB. Note that a single gigabit Ethernet interface can scale up to ten thousand 8-KB
blocks per second before saturation.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 22
Index Block Contention: Considerations

Wait events
Index
enq: TX - index
contention
block
Split in
gc buffer busy
progress
gc current block busy
gc current split

System statistics
Leaf node splits
Branch node splits
Exchange deadlocks
gcs refuse xid
gcs ast xid
Service ITL waits RAC01 RAC02
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Index Block Contention: Considerations
e A
l
In application systems where the loading or batch processing of data is a dominant business
c
r a
function, there may be performance issues affecting response times because of the high
volume of data inserted into indexes. Depending on the access frequency and the number of

O ly
processes concurrently inserting data, indexes can become hot spots and contention can be
exacerbated by:

l & On
Ordered, monotonically increasing key values in the index (right-growing trees)
Frequent leaf block splits
n a e
Low tree depth: All leaf block access go through the root block.

e r s
A leaf or branch block split can become an important serialization point if the particular leaf
t U
I n
block or branch of the tree is concurrently accessed. The tables in the slide sum up the most
common symptoms associated with the splitting of index blocks, listing wait events and

c l e
statistics that are commonly elevated when index block splits are prevalent. As a general
recommendation, to alleviate the performance impact of globally hot index blocks and leaf

r a
block splits, a more uniform, less skewed distribution of the concurrency in the index tree
should be the primary objective. This can be achieved by:
O Global index hash partitioning
Increasing the sequence cache, if the key value is derived from a sequence
Using natural keys as opposed to surrogate keys
Using reverse key indexes

Oracle Database 11g: RAC Administration 14 - 23


Oracle Sequences and Index Contention

Can contain 500 rows

150000 50001100000

CACHE 50000 NOORDER

RAC01 RAC02

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Oracle Sequences and Index Contention
e A
l
Indexes with key values generated by sequences tend to be subject to leaf block contention
c
r a
when the insert rate is high. That is because the index leaf block holding the highest key value
is changed for every row inserted, as the values are monotonically ascending. In RAC, this
O ly
may lead to a high rate of current and CR blocks transferred between nodes.

l & On
One of the simplest techniques that can be used to limit this overhead is to increase the
sequence cache, if you are using Oracle sequences. Because the difference between sequence

n a e
values generated by different instances increases, successive index block splits tend to create

t e r U s
instance affinity to index leaf blocks. For example, suppose that an index key value is
generated by a CACHE NOORDER sequence and each index leaf block can hold 500 rows. If

I n
the sequence cache is set to 50000, while instance 1 inserts values 1, 2, 3, and so on, instance 2
concurrently inserts 50001, 50002, and so on. After some block splits, each instance writes to a
l e
different part of the index tree.
c
r a
So, what is the ideal value for a sequence cache to avoid inter-instance leaf index block
contention, yet minimizing possible gaps? One of the main variables to consider is the insert
O
rate: the higher it is, the higher must be the sequence cache. However, creating a simulation to
evaluate the gains for a specific configuration is recommended.
Note: By default, the cache value is 20. Typically, 20 is too small for the preceding example.

Oracle Database 11g: RAC Administration 14 - 24


Undo Block Considerations

Index Changes
Reads

SGA1 SGA2

Undo Undo

Additional
interconnect traffic
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Undo Block Considerations
e A
l
Excessive undo block shipment and contention for undo buffers usually happens when index
c
r a
blocks containing active transactions from multiple instances are read frequently.
When a SELECT statement needs to read a block with active transactions, it has to undo the
O ly
changes to create a CR version. If the active transactions in the block belong to more than one

l & On
instance, there is a need to combine local and remote undo information for the consistent read.
Depending on the amount of index blocks changed by multiple instances and the duration of

n a e
the transactions, undo block shipment may become a bottleneck.

e r s
Usually this happens in applications that read recently inserted data very frequently, but
t U
commit infrequently. Techniques that can be used to reduce such situations include the
following:
I n
c l e
Shorter transactions reduce the likelihood that any given index block in the cache
contains uncommitted data, thereby reducing the need to access undo information for

r a consistent read.
As explained earlier, increasing sequence cache sizes can reduce inter-instance
O concurrent access to index leaf blocks. CR versions of index blocks modified by only one
instance can be fabricated without the need of remote undo information.
Note: In RAC, the problem is exacerbated by the fact that a subset of the undo information has
to be obtained from remote instances.
Oracle Database 11g: RAC Administration 14 - 25
High-Water Mark Considerations

Wait events
enq: HW -
contention
gc current grant Heavy
inserts

HWM

Heavy
inserts

New extent

RAC01 RAC02
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
High-Water Mark Considerations
e A
l
A certain combination of wait events and statistics presents itself in applications where the
c
r a
insertion of data is a dominant business function and new blocks have to be allocated
frequently to a segment. If data is inserted at a high rate, new blocks may have to be made
O ly
available after unfruitful searches for free space. This has to happen while holding the high-
water mark (HWM) enqueue.

l & On
Therefore, the most common symptoms for this scenario include:
a e
A high percentage of wait time for enq: HW contention
n
t e r U s
A high percentage of wait time for gc current grant events
The former is a consequence of the serialization on the HWM enqueue, and the latter is

I n
because of the fact that current access to the new data blocks that need formatting is required

c l e
for the new block operation. In a RAC environment, the length of this space management
operation is proportional to the time it takes to acquire the HWM enqueue and the time it takes

r a
to acquire global locks for all the new blocks that need formatting. This time is small under
normal circumstances because there is never any access conflict for the new blocks. Therefore,
O
this scenario may be observed in applications with business functions requiring a lot of data
loading, and the main recommendation to alleviate the symptoms is to define uniform and
large extent sizes for the locally managed and automatic space managed segments that are
subject to high-volume inserts.
Oracle Database 11g: RAC Administration 14 - 26
Concurrent Cross-Instance Calls: Considerations

Dirty
block

SGA1 SGA2
Table1 Table1

CKPT CKPT
Table2 Table2

2
3 4
Truncate Table1 Truncate Table2
Cross-instance call

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Concurrent Cross-Instance Calls: Considerations
e A
l
In data warehouse and data mart environments, it is not uncommon to see a lot of TRUNCATE
c
r a
operations. These essentially happen on tables containing temporary data.
In a RAC environment, truncating tables concurrently from different instances does not scale
O ly
well, especially if, in conjunction, you are also using direct read operations such as parallel
queries.
l & On
n a e
As shown in the slide, a truncate operation requires a cross-instance call to flush dirty blocks
of the table that may be spread across instances. This constitutes a point of serialization. So,

completes.
t e r U s
while the first TRUNCATE command is processing, the second has to wait until the first one

I n
There are different types of cross-instance calls. However, all use the same serialization
mechanism.

c l e
For example, the cache flush for a partitioned table with many partitions may add latency to a

r a
corresponding parallel query. This is because each cross-instance call is serialized at the

O
cluster level, and one cross-instance call is needed for each partition at the start of the parallel
query for direct read purposes.

Oracle Database 11g: RAC Administration 14 - 27


Monitoring RAC Database and Cluster
Performance
Database
Directly from Database Control and Grid Control:
View the status of each node in the cluster.
View the aggregated alert messages across
Instances
all the instances.
Review the issues that are affecting the entire cluster or
each instance.
Monitor the cluster cache coherency statistics.
Determine whether any of the services for the cluster
database are having availability problems.
Review any outstanding Clusterware interconnect alerts.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Monitoring RAC Database and Cluster Performance
e A
l
Both Oracle Enterprise Manager Database Control and Grid Control are cluster-aware and
c
page, you can do all of the following:
r a
provide a central console to manage your cluster database. From the Cluster Database Home

O ly
View the overall system status, such as the number of nodes in the cluster and their

l & On
current status, so you do not have to access each individual database instance for details
View the alert messages aggregated across all the instances with lists for the source of
each alert message.
n a e
t
individual instances.
e r U s
Review the issues that are affecting the entire cluster as well as those that are affecting

I n
Monitor cluster cache coherency statistics to help you identify processing trends and
optimize performance for your Oracle RAC environment. Cache coherency statistics

l e
measure how well the data in caches on multiple instances is synchronized.
c
Determine whether any of the services for the cluster database are having availability

r aproblems. A service is deemed to be a problem service if it is not running on all preferred

O instances, if its response time thresholds are not met, and so on.
Review any outstanding Clusterware interconnect alerts.

Oracle Database 11g: RAC Administration 14 - 28


Cluster Database Performance Page

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cluster Database Performance Page
e A
l
The Cluster Database Performance page provides a quick glimpse of the performance statistics
c
r a
for a database. Enterprise Manager accumulates data from each instance over specified periods
of time, called collection-based data. Enterprise Manager also provides current data from each
instance, known as real-time data.
O ly
l & On
Statistics are rolled up across all the instances in the cluster database. Using the links next to
the charts, you can get more specific information and perform any of the following tasks:

n a e
Identify the causes of performance issues.

t e r U s
Decide whether resources need to be added or redistributed.
Tune your SQL plan and schema for better optimization.

I n
Resolve performance issues.

c l e
The screenshot in the slide shows a partial view of the Cluster Database Performance page.
You access this page by clicking the Performance tab from the Cluster Database Home page.

r a
O
Oracle Database 11g: RAC Administration 14 - 29
Determining Cluster Host Load Average

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Determining Cluster Host Load Average
e A
l
The Cluster Host Load Average chart in the Cluster Database Performance page shows
c
r a
potential problems that are outside the database. The chart shows maximum, average, and
minimum load values for available nodes in the cluster for the previous hour.
O ly
If the load average is higher than the average of the total number of CPUs across all the hosts

l & On
in the cluster, then too many processes are waiting for CPU resources. SQL statements that are
not tuned often cause high CPU usage. Compare the load average values with the values

n a e
displayed for CPU Used in the Average Active Sessions chart. If the sessions value is low and

t e r
database, is consuming the CPU.
U s
the load average value is high, this indicates that something else on the host, other than your

I n
You can click any of the load value labels for the Cluster Host Load Average chart to view

c l e
more detailed information about that load value. For example, if you click the Average label,
the Hosts: Average Load page appears, displaying charts that depict the average host load for

r a
up to four nodes in the cluster.
You can select whether the data is displayed in a summary chart, combining the data for each
O
node in one display, or using tile charts, where the data for each node is displayed in its own
chart. You can click Customize to change the number of tile charts displayed in each row or
the method of ordering the tile charts.

Oracle Database 11g: RAC Administration 14 - 30


Determining Global Cache Block Access Latency

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Determining Global Cache Block Access Latency
e A
l
The Global Cache Block Access Latency chart shows the latency for each type of data block
c
r a
requests: current and consistent-read (CR) blocks. That is the elapsed time it takes to locate
and transfer consistent-read and current blocks between the buffer caches.
O ly
You can click either metric for the Global Cache Block Access Latency chart to view more

& On
detailed information about that type of cached block.
l
n a e
If the Global Cache Block Access Latency chart shows high latencies (high elapsed times),
this can be caused by any of the following:

e r s
A high number of requests caused by SQL statements that are not tuned
t U
A large number of processes in the queue waiting for the CPU, or scheduling delays

I n
Slow, busy, or faulty interconnects. In these cases, check your network connection for

c l e
dropped packets, retransmittals, or cyclic redundancy check (CRC) errors.
Concurrent read and write activity on shared data in a cluster is a frequently occurring activity.

r a
Depending on the service requirements, this activity does not usually cause performance
problems. However, when global cache requests cause a performance problem, optimizing
O
SQL plans and the schema to improve the rate at which data blocks are located in the local
buffer cache, and minimizing I/O is a successful strategy for performance tuning. If the latency
for consistent-read and current block requests reaches 10 milliseconds, then see the Cluster
Cache Coherency page for more detailed information.
Oracle Database 11g: RAC Administration 14 - 31
Determining Average Active Sessions

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Determining Average Active Sessions
e A
l
The Average Active Sessions chart on the Cluster Database Performance page shows potential
c
r a
problems inside the database. Categories, called wait classes, show how much of the database
is using a resource, such as CPU or disk I/O. Comparing CPU time with wait time helps to
O ly
determine how much of the response time is consumed with useful work rather than waiting

l & On
for resources that are potentially held by other processes.
At the cluster database level, this chart shows the aggregate wait class statistics across all the

n a e
instances. For a more detailed analysis, you can click the Clipboard icon at the bottom of the

t e r U s
chart to view the ADDM analysis for the database for that time period.
If you click the wait class legends beside the Average Active Sessions chart, you can view

I n
instance-level information stored in Active Sessions by Instance pages. You can use the

c l e
Wait Class action list on the Active Sessions by Instance page to view the different wait
classes. The Active Sessions by Instance pages show the service times for up to four

r a
instances. Using the Customize button, you can select the instances that are displayed. You can
view the data for the instances separately by using tile charts, or you can combine the data into
O
a single summary chart.

Oracle Database 11g: RAC Administration 14 - 32


Determining Database Throughput

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Determining Database Throughput
e A
l
The last chart on the Performance page monitors the usage of various database resources.
c
r a
Click the Throughput tab at the top of this chart to view the Database Throughput chart.
Compare the peaks on the Average Active Sessions chart with those on the Database
O ly
Throughput charts. If internal contention is high and throughput is low, consider tuning the
database.

l & On
The Database Throughput charts summarize any resource contention that appears in the
a e
Average Active Sessions chart, and also show how much work the database is performing on
n
t e r U s
behalf of the users or applications. The Per Second view shows the number of transactions
compared to the number of logons, and (not shown here) the number of physical reads

I n
compared to the redo size per second. The Per Transaction view shows the number of physical
reads compared to the redo size per transaction. Logons is the number of users that are logged

l e
on to the database.
c
To obtain information at the instance level, access the Database Throughput by Instance
r a
page by clicking one of the legends to the right of the charts. This page shows the breakdown

O
of the aggregated Database Throughput chart for up to four instances. You can select the
instances that are displayed. You can drill down further on the Database Throughput by
Instance page to see the sessions of an instance consuming the greatest resources. Click an
instance name legend under the chart to go to the Top Sessions sub-page of the Top
Consumers page for that instance.
Oracle Database 11g: RAC Administration 14 - 33
Determining Database Throughput

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Determining Database Throughput (continued)
e A
l
The last chart on the Performance page monitors the usage of various database resources. By
c
Instance chart.
r a
clicking the Instances tab at the top of this chart, you can view the Active Sessions by

O ly
The Active Sessions by Instance chart summarizes any resource contention that appears in

l & On
the Average Active Sessions chart. Using this chart, you can quickly determine how much of
the database work is being performed on each instance.

n a e
You can also obtain information at the instance level by clicking one of the legends to the right

e r s
of the chart to access the Top Sessions page. On the Top Session page, you can view real-time
t U
data showing the sessions that consume the greatest system resources. In the graph in the slide

I n
above, the orac2 instance after 8:20 PM is consistently showing more active sessions than the
orac1 instance.

c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 34
Accessing the Cluster Cache Coherency Page

Block
Class

Segment
name

Segment
name

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Accessing the Cluster Cache Coherency Page
e A
l
To access the Cluster Cache Coherency page, click the Performance tab on the Cluster
c
r a
Database Home page, and click Cluster Cache Coherency in the Additional Monitoring Links
section at the bottom of the page. Alternatively, click either of the legends to the right of the
O ly
Global Cache Block Access Latency chart.

the cluster:
l & On
The Cluster Cache Coherency page contains summary charts for cache coherency metrics for

n a e
Global Cache Block Access Latency: Shows the total elapsed time, or latency, for a

t e r U s
block request. Click one of the legends to the right of the chart to view the average time
it takes to receive data blocks for each block type (current or CR) by instance. On the

I n
Average Block Receive Time by Instance page, you can click an instance legend under
the chart to go to the Block Transfer for Local Instance page, where you can identify
l e
which block classes, such as undo blocks, data blocks, and so on, are subject to intense
c
r a global cache activity. This page displays the block classes that are being transferred, and
which instances are transferring most of the blocks. Cache transfer indicates how many
O current and CR blocks for each block class were received from remote instances,
including how many transfers incurred a delay (busy) or an unexpected longer delay
(congested).

Oracle Database 11g: RAC Administration 14 - 35


Accessing the Cluster Cache Coherency Page

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Accessing the Cluster Cache Coherency Page (continued)
e A
l
Global Cache Block Transfer Rate: Shows the total aggregated number of blocks
c
r a
received by all instances in the cluster by way of an interconnect. Click one of the
legends to the right of the chart to go to the Global Cache Blocks Received by Instance
O ly
page for that type of block. From there, you can click an instance legend under the chart

are causing cache contention.


l & On
to go to the Segment Statistics by Instance page, where you can see which segments

a e
Global Cache Block Transfers and Physical Reads: Shows the percentage of logical
n
t e r U s
read operations that retrieved data from the buffer cache of other instances by way of
Direct Memory Access and from disk. It is essentially a profile of how much work is

I n
performed in the local buffer cache, rather than the portion of remote references and
physical reads, which both have higher latencies. Click one of the legends to the right of

l e
the chart to go to the Global Cache Block Transfers vs. Logical Reads by Instance and
c
Physical Reads vs. Logical Reads by Instance pages. From there, you can click an

r a
instance legend under the chart to go to the Segment Statistics by Instance page, where

O you can see which segments are causing cache contention.

Oracle Database 11g: RAC Administration 14 - 36


Viewing the Cluster Interconnects Page

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Viewing the Cluster Interconnects Page
e A
l
The Cluster Interconnects page is useful for monitoring the interconnect interfaces,
c
r a
determining configuration issues, and identifying transfer raterelated issues including excess
traffic. This page helps to determine the load added by instances and databases on the
O ly
interconnect. Sometimes you can quickly identify interconnect delays that are due to
applications outside Oracle.

l & On
You can use this page to perform the following tasks:

n a e
View all interfaces that are configured across the cluster

t e r U s
View statistics for the interfaces, such as absolute transfer rates and errors
Determine the type of interfaces, such as private or public

I n
Determine whether the instance is using a public or private network
Determine which database instance is currently using which interface
l e
Determine how much the instance is contributing to the transfer rate on the interface
c
r a
The Private Interconnect Transfer Rate value shows a global view of the private interconnect
traffic, which is the estimated traffic on all the private networks in the cluster. The traffic is
O
calculated as the summary of the input rate of all private interfaces known to the cluster.
From the Cluster Interconnects page, you can access the Hardware Details page, on which you
can get more information about all the network interfaces defined on each node of your
cluster.
Oracle Database 11g: RAC Administration 14 - 37
Viewing the Cluster Interconnects Page (continued)
Similarly, you can access the Transfer Rate metric page, which collects the internode
communication traffic of a cluster database instance. The critical and warning thresholds of
this metric are not set by default. You can set them according to the speed of your cluster
interconnects.
Note: You can query the GV$CLUSTER_INTERCONNECTS view to see information about the
private interconnect:
SQL> select * from GV$CLUSTER_INTERCONNECTS;

INST_ID NAME IP_ADDRESS IS_PUBLIC SOURCE


-------- ----- --------------- --------- -------------------------
1 eth1 192.0.2.110 NO Oracle Cluster Repository
2 eth1 192.0.2.111 NO Oracle Cluster Repository
3 eth1 192.0.2.112 NO Oracle Cluster Repository

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 38
Viewing the Database Locks Page

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Viewing the Database Locks Page
e A
l
Use the Database Locks page to determine whether multiple instances are holding locks for the
c
r a
same object. The page shows user locks, all database locks, or locks that are blocking other
users or applications. You can use this information to stop a session that is unnecessarily
locking an object.
O ly
l & On
To access the Database Locks page, select Performance on the Cluster Database Home page,
and click Database Locks in the Additional Monitoring Links section at the bottom of the
Performance subpage.
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 39
AWR Snapshots in RAC

MMON Coordinator
In-memory
statistics
SYSAUX
SGA (Inst1)
AWR tables
6:00 a.m.
9:00 a.m. 7:00 a.m.
8:00 a.m.
MMON 9:00 a.m.
In-memory
statistics

SGA (Instn)

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
AWR Snapshots in RAC
e A
l
AWR automatically generates snapshots of the performance data once every hour and collects
c
r a
the statistics in the workload repository. In RAC environments, each AWR snapshot captures
data from all active instances within the cluster. The data for each snapshot set that is captured
O ly
for all active instances is from roughly the same point in time. In addition, the data for each

l & On
instance is stored separately and is identified with an instance identifier. For example, the
buffer_busy_wait statistic shows the number of buffer waits on each instance. The
a e
AWR does not store data that is aggregated from across the entire cluster. That is, the data is
n
t e r
stored for each individual instance.

U s
The statistics snapshots generated by the AWR can be evaluated by producing reports

I n
displaying summary data such as load and cluster profiles based on regular statistics and wait
events gathered on each instance.

c l e
The AWR functions in a similar way as Statspack. The difference is that the AWR

r a
automatically collects and maintains performance statistics for problem detection and self-
tuning purposes. Unlike in Statspack, in the AWR, there is only one snapshot_id per
O
snapshot across instances.

Oracle Database 11g: RAC Administration 14 - 40


AWR Reports and RAC: Overview

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
AWR Reports and RAC
e A
l
The RAC-related statistics in an AWR report are organized in different sections. A RAC
c
r a
statistics section appears after the Top 5 Timed Events. This section contains:
The number of instances open at the time of the begin snapshot and the end snapshot to
O ly
indicate whether instances joined or left between the two snapshots

l & On
The Global Cache Load Profile, which essentially lists the number of blocks and
messages that are sent and received, as well as the number of fusion writes
a e
The Global Cache Efficiency Percentages, which indicate the percentage of buffer gets
n
t e r U s
broken up into buffers received from the disk, local cache, and remote caches. Ideally,
the percentage of disk buffer access should be close to zero.

I n
GCS and GES Workload Characteristics, which gives you an overview of the more
important numbers first. Because the global enqueue convert statistics have been

l e
consolidated with the global enqueue get statistics, the report prints only the average
c
global enqueue get time. The round-trip times for CR and current block transfers follow,

r a as well as the individual sender-side statistics for CR and current blocks. The average log

O flush times are computed by dividing the total log flush time by the number of actual log
flushes. Also, the report prints the percentage of blocks served that actually incurred a
log flush.

Oracle Database 11g: RAC Administration 14 - 41


AWR Reports and RAC (continued)
GCS and GES Messaging Statistics. The most important statistic here is the average
message sent queue time on ksxp, which indicates how well the IPC works. Average
numbers should be less than 1 ms.
Additional RAC statistics are then organized in the following sections:
The Global Enqueue Statistics section contains data extracted from
V$GES_STATISTICS.
The Global CR Served Stats section contains data from V$CR_BLOCK_SERVER.
The Global CURRENT Served Stats section contains data from
V$CURRENT_BLOCK_SERVER.
The Global Cache Transfer Stats section contains data from
V$INSTANCE_CACHE_TRANSFER.
The Segment Statistics section also includes the GC Buffer Busy Waits, CR Blocks Received,
and CUR Blocks Received information for relevant segments.
Note: For more information about wait events and statistics, refer to Oracle Database
Reference.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 42
Active Session History Reports for RAC

Active Session History (ASH) report statistics provide


details about the RAC Database session activity.
The database records
information about active
sessions for all active RAC
instances.

Two ASH report sections


specific to Oracle RAC are
Top Cluster Events and
Top Remote Instance. m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Active Session History Reports for RAC
e A
l
Active Session History (ASH) is an integral part of the Oracle Database self-management
c
r a
framework and is useful for diagnosing performance problems in Oracle RAC environments.
ASH report statistics provide details about Oracle Database session activity. Oracle Database
O ly
records information about active sessions for all active Oracle RAC instances and stores this

l & On
data in the System Global Area (SGA). Any session that is connected to the database and using
CPU is considered an active session. The exception to this is sessions that are waiting for an
a e
event that belongs to the idle wait class.
n
t e r U s
ASH reports present a manageable set of data by capturing only information about active
sessions. The amount of the data is directly related to the work being performed, rather than

I n
the number of sessions allowed on the system. ASH statistics that are gathered over a specified
duration can be put into ASH reports.

c l e
Each ASH report is divided into multiple sections to help you identify short-lived performance

r a
problems that do not appear in the ADDM analysis. Two ASH report sections that are specific
to Oracle RAC are Top Cluster Events and Top Remote Instance.
O
Oracle Database 11g: RAC Administration 14 - 43
Active Session History Reports for RAC (continued)
Top Cluster Events
The ASH report Top Cluster Events section is part of the Top Events report that is specific to
Oracle RAC. The Top Cluster Events report lists events that account for the highest percentage
of session activity in the cluster wait class event along with the instance number of the affected
instances. You can use this information to identify which events and instances caused a high
percentage of cluster wait events.
Top Remote Instance
The ASH report Top Remote Instance section is part of the Top Load Profile report that is
specific to Oracle RAC. The Top Remote Instance report shows cluster wait events along with
the instance numbers of the instances that accounted for the highest percentages of session
activity. You can use this information to identify the instance that caused the extended cluster
wait period.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 44
Automatic Database Diagnostic Monitor for RAC
ADDM can perform analysis:
For the entire cluster
For a specific database instance
Database ADDM
For a subset of database instances

Self-diagnostic engine

Instance ADDM

AWR

Inst1 Instn
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Automatic Database Diagnostic Monitor for RAC
e A
l
Using the Automatic Database Diagnostic Monitor (ADDM), you can analyze the information
c
r a
collected by AWR for possible performance problems with your Oracle database. ADDM
presents performance data from a clusterwide perspective, thus enabling you to analyze
O ly
performance on a global basis. In an Oracle RAC environment, ADDM can analyze

granularity, including:
l & On
performance using data collected from all instances and present it at different levels of

a e
Analysis for the entire cluster
n
t e r U s
Analysis for a specific database instance
Analysis for a subset of database instances

I n
To perform these analyses, you can run the ADDM Advisor in Database ADDM for RAC
mode to perform an analysis of the entire cluster, in Local ADDM mode to analyze the
l e
performance of an individual instance, or in Partial ADDM mode to analyze a subset of
c
r a
instances. Database ADDM for RAC is not just a report of reports but has independent
analysis that is appropriate for RAC. You activate ADDM analysis using the advisor
O
framework through Advisor Central in Oracle Enterprise Manager, or through the
DBMS_ADVISOR and DBMS_ADDM PL/SQL packages.
Note: Database ADDM report is generated on AWR snapshot coordinator.

Oracle Database 11g: RAC Administration 14 - 45


Automatic Database Diagnostic Monitor for RAC

Identifies the most critical performance problems for the


entire RAC cluster database
Runs automatically when taking AWR snapshots
Performs database-wide analysis of:
Global resources (for example I/O and global locks)
High-load SQL and hot blocks
Global cache interconnect traffic
Network latency issues
Skew in instance response times
Is used by DBAs to analyze cluster performance
No need to investigate n reports to spot common problems

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Automatic Database Diagnostic Monitor for RAC (continued)
e A
l
In Oracle Database 11g, you can create a period analysis mode for ADDM that analyzes the
c
r a
throughput performance for an entire cluster. When the advisor runs in this mode, it is called
database ADDM. You can run the advisor for a single instance, which is called instance
ADDM.
O ly
l & On
Database ADDM has access to AWR data generated by all instances, thereby making the
analysis of global resources more accurate. Both database and instance ADDM run on
a e
continuous time periods that can contain instance startup and shutdown. In the case of database
n
t e r
ADDM, there may be several instances that are shut down or started during the analysis
s
period. However, you must maintain the same database version throughout the entire time
U
period.
I n
Database ADDM runs automatically after each snapshot is taken. You can also perform

l e
analysis on a subset of instances in the cluster. This is called partial analysis ADDM.
c
I/O capacity finding (the I/O system is overused) is a global finding because it concerns a
r a
global resource affecting multiple instances. A local finding concerns a local resource or issue

O
that affects a single instance. For example, a CPU-bound instance results in a local finding
about the CPU. Although ADDM can be used during application development to test changes
to either the application, the database system, or the hosting machines, database ADDM is
targeted at DBAs.

Oracle Database 11g: RAC Administration 14 - 46


What Does ADDM Diagnose for RAC?

Latency problems in interconnect


Congestion (identifying top instances affecting the entire
cluster)
Contention (buffer busy, top objects, etc.)
Top consumers of multiblock requests
Lost blocks
Reports information about interconnect devices; warns
about using PUBLIC interfaces
Reports throughput of devices, and how much of it is used
by Oracle and for what purpose (GC, locks, PQ)

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
What Does ADDM Diagnose for RAC?
e A
Data sources are:
c l
r a
Wait events (especially Cluster class and buffer busy)
Active Session History (ASH) reports
Instance cache transfer data
O ly
l & On
Interconnect statistics (throughput, usage by component, pings)
ADDM analyzes the effects of RAC for both the entire database (DATABASE analysis mode)

n a e
and for each instance (INSTANCE analysis mode).

t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 47
EM Support for ADDM for RAC

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
EM Support for ADDM for RAC
e A
l
Oracle Database 11g Enterprise Manager displays the ADDM analysis on the Cluster Database
c
Home page.
r a
On the Automatic Database Diagnostic Monitor (ADDM) page, the Database Activity chart
O ly
(not shown here) plots the database activity during the ADDM analysis period. Database

l & On
activity types are defined in the legend based on its corresponding color in the chart. Each icon
below the chart represents a different ADDM task, which in turn corresponds to a pair of

n a e
individual Oracle Database snapshots saved in the Workload Repository.

e r s
In the ADDM Performance Analysis section, the ADDM findings are listed in descending
t U
order, from highest impact to least impact. For each finding, the Affected Instances column

I n
displays the number (m of n) of instances affected. Drilling down further on the findings takes

c l e
you to the Performance Findings Detail page. The Informational Findings section lists the
areas that do not have a performance impact and are for informational purpose only.

r a
The Affected Instances chart shows how much each instance is impacted by these findings.
The display indicates the percentage impact for each instance.
O
Oracle Database 11g: RAC Administration 14 - 48
Quiz

Although there are specific tuning areas for RAC, such as


instance recovery and interconnect traffic, you get most
benefits by tuning your system like a single-instance system.
1. True
2. False

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 1
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 49
Quiz

Which of the following RAC tuning tips are correct?


1. Application tuning is often the most beneficial!
2. Reduce long full-table scans in OLTP systems
3. Eliminate sequence caches
4. Use partitioning to reduce inter-instance traffic
5. Configure the interconnects properly

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 1,2,4,5
e A
Statement 3 is incorrect.
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 50
Summary

In this lesson, you should have learned how to:


Determine RAC-specific tuning components
Tune instance recovery in RAC
Determine RAC-specific wait events, global enqueues, and
system statistics
Implement the most common RAC tuning tips
Use the Cluster Database Performance pages
Use the Automatic Workload Repository (AWR) in RAC
Use Automatic Database Diagnostic Monitor (ADDM) in
RAC

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 51
Practice 14 Overview

This practice covers the following topics:


This lab shows how to manually discover performance
issues using the EM performance pages as well as ADDM.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 14 - 52
Services

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Objectives

After completing this lesson, you should be able to:


Configure and manage services in a RAC environment
Use services with client applications
Use services with the Database Resource Manager
Use services with the Scheduler
Configure services aggregation and tracing

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 15 - 2
Oracle Services

To manage workloads or a group of applications, you can


define services for a particular application or a subset of an
applications operations.
You can also group work by type under services.
For example OLTP users can use one service while batch
processing can use another to connect to the database.
Users who share a service should have the same service-
level requirements.
Use srvctl or Enterprise Manager to manage services,
not DBMS_SERVICE.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Oracle Services
e A
l
To manage workloads or a group of applications, you can define services that you assign to a
c
r a
particular application or to a subset of an applications operations. You can also group work by
type under services. For example, online users can use one service, while batch processing can
O ly
use another, and reporting can use yet another service to connect to the database.

l & On
It is recommended that all users who share a service have the same service-level requirements.
You can define specific characteristics for services and each service can be a separate unit of

n a e
work. There are many options that you can take advantage of when using services. Although

t e r U s
you do not have to implement these options, using them helps optimize application
performance. You can define services for both policy-managed and administrator-managed
databases.
I n
c l e
Do not use DBMS_SERVICE with cluster-managed services. When Oracle Clusterware starts
a service, it updates the database with the attributes stored in the CRS resource. If you use

r a
DBMS_SERVICE to modify the service and do not update the CRS resource, the next time
CRS resource is started, it will override the database attributes set by DBMS_SERVICE.
O
Oracle Database 11g: RAC Administration 15 - 3
Services for Policy- and Administrator-Managed
Databases
You can define services for both policy-managed and
administrator-managed databases.
Services for a policy-managed database are defined to a
server pool where the database is running.
Services for policy-managed databases can be defined as:
UNIFORM (running on all instances in the server pool)
SINGLETON (running on only one instance in the pool)
For singleton services, RAC chooses on which instance in the
server pool the service is active.
Services for an administrator-managed database define
which instances normally support that service.
These are known as the PREFERRED instances.
Instances defined to support a service if the preferred
m y
instance fails are known as AVAILABLE instances.
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Services for Policy- and Administrator-Managed Databases
e A
l
It is recommended that all users who share a service have the same service-level requirements.
c
r a
You can define specific characteristics for services and each service can be a separate unit of
work. There are many options that you can take advantage of when using services. Although
O ly
you do not have to implement these options, they help optimize application performance. You

l & On
can define services for both policy-managed and administrator-managed databases.
Policy-managed database: When you define services for a policy-managed database,
a e
you define the service to a server pool where the database is running. You can define the
n
t e r U s
service as either uniform (running on all instances in the server pool) or singleton
(running on only one instance in the server pool). For singleton services, RAC chooses

I n
on which instance in the server pool the service is active. If that instance fails, then the
service fails over to another instance in the pool. A service can only run in one server
pool.
c l e
Administrator-managed database: When you define a service for an administrator-

r a managed database, you define which instances support that service. These are known as

O the PREFERRED instances. You can also define other instances to support a service if
the service's preferred instance fails. These are known as AVAILABLE instances.
Note: Fail back is not implemented by default because it is a disruptive operation. If you
require fail back, you can always try to implement it as a callout that relocates the service.

Oracle Database 11g: RAC Administration 15 - 4


Default Service Connections

Application services:
Limit of 115 services per database
Internal services:
SYS$BACKGROUND
SYS$USERS
Cannot be deleted or changed
A special Oracle database service is created by default for
the Oracle RAC database.
This default service is always available on all instances in
an Oracle RAC environment.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Default Service Connections
e A
l
There are two broad types of services: application services and internal services. Application
c
r a
services are mainly functional maps to workloads. Sessions doing work for a common business
function are grouped together. For Oracle E-Business Suite, AP, AR, GL, MFG, WIP, and so on
O ly
create a functional division of work within the database and can be categorized as services.

l & On
The RDBMS also supports two internal services. SYS$BACKGROUND is used by the
background processes only. SYS$USERS is the default service for user sessions that are not

n a e
associated with any application service. Both internal services support all the workload

t e r U s
management features and neither one can be stopped or disabled. A special Oracle database
service is created by default for your Oracle RAC database. This default service is always

I n
available on all instances in an Oracle RAC environment, unless an instance is in restricted
mode. You cannot alter this service or its properties. There is a limitation of 115 application
l e
services per database that you can create. Also, a service name is restricted to 64 characters.
c
r a
Note: Shadow services are also included in the application service category. For more
information about shadow services, see the lesson titled High Availability of Connections.
O
In addition, a service is also created for each Advanced Queue created. However, these types
of services are not managed by Oracle Clusterware. Using service names to access a queue
provides location transparency for the queue within a RAC database.

Oracle Database 11g: RAC Administration 15 - 5


Create Service with Enterprise Manager

Administration-Managed

Policy-Managed

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Create Services with Enterprise Manager
e A
l
From your Cluster Database home page, click the Availability tab, and then click Cluster
c
Service.
r a
Managed Database Services. On the Cluster Managed Database Services page, click Create

O ly
Use the Create Service page to configure a new service in which you do the following:

l & On
Select the desired service policy for each instance configured for the cluster database.
Select the desired service properties.
a e
If your database is administration managed, the High Availability Configuration section allows
n
t e r U s
you to configure preferred and available servers. If your database employs policy-managed
administration, you can configure the service cardinality to be UNIFORM or SINGLETON and

I n
assign the service to a server pool.
You can also define the management policy for a service. You can choose either an automatic
l e
or a manual management policy.
c
Automatic: The service always starts when the database starts.
r a
Manual: Requires that the service be started manually. Prior to Oracle RAC 11g Release

O 2, all services worked as though they were defined with a manual management policy.
Note: Enterprise Manager now generates the corresponding entries in your tnsnames.ora
files for your services. Just click the Update local naming parameter (tnsnames.ora) file
check box when creating the service.

Oracle Database 11g: RAC Administration 15 - 6


Create Services with SRVCTL

To create a service called GL with preferred instance


RAC02 and an available instance RAC01:
$ srvctl add service d PROD1 s GL -r RAC02 -a RAC01
To create a service called AP with preferred instance
RAC01 and an available instance RAC02:
$ srvctl add service d PROD1 s AP r RAC01 -a RAC02

To create a SINGLETON service called BATCH using server


pool SP1 and a UNIFORM service called ERP using server
pool SP2 :
$ srvctl add service -d PROD2 -s BATCH -g SP1 \
-c singleton -y manual

$ srvctl add service -d PROD2 -s ERP -g SP2 \


-c UNIFORM -y manual
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Create Services with SRVCTL
e A
l
For the example in the slide, assume a two-node, administration-managed database called
c
r a
PROD1 with an instance named RAC01 on one node and an instance called RAC02 on the
other. Two services are created, AP and GL, to be managed by Oracle Clusterware. The AP
O ly
service is defined with a preferred instance of RAC01 and an available instance of RAC02.

l & On
If RAC01 dies, the AP service member on RAC01 is restored automatically on RAC02. A
similar scenario holds true for the GL service.

n a e
Note that it is possible to assign more than one instance with both the -r and -a options.

e r s
However, -r is mandatory but -a is optional.
t U
I n
Next, assume a policy-managed cluster database called PROD2. Two services are created, a
SINGELTON service called BATCH and a UNIFORM service called ERP. SINGLETON

l e
services run on one of the active servers and UNIFORM services run on all active servers of
c
the server pool. The characteristics of the server pool determines how resources are allocated

r a
to the service.

O
Note: When services are created with srvctl, tnsnames.ora is not updated and the service
is not started.

Oracle Database 11g: RAC Administration 15 - 7


Manage Services with Enterprise Manager

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Manage Services with Enterprise Manager
e A
l
You can use Enterprise Manager to manage services within a GUI framework. The screenshot
c
r a
in the slide shows the main page for administering services within RAC. It shows you some
basic status information about a defined service.
O ly
To access this page, click the Cluster Managed Database Services link on the Cluster Database
Availability page.
l & On
n a e
You can perform simple service management such as enabling, disabling, starting, stopping,
and relocating services. All possible operations are shown in the slide.

e r s
If you choose to start a service on the Cluster Managed Database Services page, then EM
t U
I n
attempts to start the service on every preferred instance. Stopping the service stops it on all
instances that it is currently running.

l e
To relocate a service, select the service that you want to administer, select the Manage option
c
from the Actions drop-down list, and then click Go.

r a
Note: On the Cluster Managed Database Services page, you can test the connection for a
O
service.

Oracle Database 11g: RAC Administration 15 - 8


Managing Services with EM

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Manage Services with Enterprise Manager (continued)
e A
l
To access the Cluster Managed Database Service page for an individual service, you must
c
r a
select a service from the Cluster Managed Database Services page, select the Manage option
from the Actions drop-down list, and then click Go.
O ly
This is the Cluster Managed Database Service page for an individual service. It offers you the

instances of a service.
l & On
same functionality as the previous page, except that actions performed here apply to specific

n a e
This page also offers you the added functionality of relocating a service to an available

e r s
instance. Relocating a service from one instance to another stops the service on the first
t U
instance and then starts it on the second.

I n
c l e
r a
O
Oracle Database 11g: RAC Administration 15 - 9
Manage Services with srvctl

Start a named service on all configured instances:


$ srvctl start service d orcl s AP

Stop a service:
$ srvctl stop service d orcl s AP I orcl3,orcl4

Disable a service at a named instance:


$ srvctl disable service d orcl s AP i orcl4

Set an available instance as a preferred instance:


$ srvctl modify service d orcl s AP -i orcl5 r

Relocate a service from one instance to another:


$ srvctl relocate service d orcl s AP -i orcl5 t orcl4
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Manage Services with srvctl
e A
l
The slide demonstrates some management tasks with services by using SRVCTL.
c
r a
Assume that an AP service has been created with four preferred instances: orcl1, orcl2,
orcl3, and orcl4. An available instance, orcl5, has also been defined for AP.
O ly
In the first example, the AP service is started on all instances. If any of the preferred or
& On
available instances that support AP are not running but are enabled, then they are started.
l
a e
The stop command stops the AP service on instances orcl3 and orcl4. The instances
n
e r
themselves are not shut down, but remain running possibly supporting other services. The AP
s
service continues to run on orcl1 and orcl2. The intention might have been to perform
t U
maintenance on orcl4, and so the AP service was disabled on that instance to prevent
I n
automatic restart of the service on that instance. The OCR records the fact that AP is disabled

l e
for orcl4. Thus, Oracle Clusterware will not run AP on orcl4 until the service is enabled.

c
The next command in the slide changes orcl5 from being an available instance to a preferred

r a
one. This is beneficial if the intent is to always have four instances run the service because

O
orcl4 was previously disabled. The last example relocates the AP service from instance
orcl5 to orcl4. Do not perform other service operations while the online service
modification is in progress.
Note: For more information, refer to the Oracle Real Application Clusters Administrators
Guide.
Oracle Database 11g: RAC Administration 15 - 10
Use Services with Client Applications

ERP=(DESCRIPTION= ## Using the SCAN ##


(LOAD_BALANCE=on)
(ADDRESS=(PROTOCOL=TCP)(HOST=cluster01-scan)(PORT=1521))
(CONNECT_DATA=(SERVICE_NAME=ERP)))

ERP=(DESCRIPTION= ## Using VIPs ##


(LOAD_BALANCE=on)
(ADDRESS=(PROTOCOL=TCP)(HOST=node1-vip)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST=node2-vip)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST=node3-vip)(PORT=1521))
(CONNECT_DATA=(SERVICE_NAME=ERP)))

url="jdbc:oracle:oci:@ERP" ## Thick JDBC ##

url="jdbc:oracle:thin:@(DESCRIPTION= ## Thin JDBC ##


(LOAD_BALANCE=on)
(ADDRESS=(PROTOCOL=TCP)(HOST=cluster01-scan)(PORT=1521)))
(CONNECT_DATA=(SERVICE_NAME=ERP)))"
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Use Services with Client Applications
e A
l
The first example in the slide shows the TNS connect descriptor that can be used to access the
c
r a
ERP service. It uses the cluster's Single Client Access Name (SCAN). The SCAN provides a
single name to the clients connecting to Oracle RAC that does not change throughout the life
O ly
of the cluster, even if you add or remove nodes from the cluster. Clients connecting with

l & On
SCAN can use a simple connection string, such as a thin JDBC URL or EZConnect, and still
achieve the load balancing and client connection failover. The second example uses virtual IP
a e
addresses as in previous versions of the Oracle Database.
n
TNS connect descriptor.
t e r U s
The third example shows the thick JDBC connection description using the previously defined

I n
The third example shows the thin JDBC connection description using the same TNS connect

c l e
descriptor as the first example.
Note: The LOAD_BALANCE=ON clause is used by Oracle Net to randomize its progress

r a
through the protocol addresses of the connect descriptor. This feature is called client
connection load balancing.
O
Oracle Database 11g: RAC Administration 15 - 11
Services and Connection Load Balancing

The two load balancing methods that you can implement are:
Client-side load balancing: Balances the connection
requests across the listeners
Server-side load balancing: The listener directs a
connection request to the best instance currently providing
the service by using the load balancing advisory (LBA).
FAN, Fast Connection Failover, and LBA depend on a
connection load balancing configuration that includes
setting the connection load balancing goal for the service.
The load balancing goal for the service can be either:
LONG: For applications having long-lived connections. This is
typical for connection pools and SQL*Forms sessions.
SHORT: For applications that have short-lived connections
srvctl modify service -s service_name -j LONG|SHORT
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Services and Connection Load Balancing
e A
l
Oracle Net Services provides the ability to balance client connections across the instances in
c
r a
an Oracle RAC configuration. You can implement two types of load balancing: client-side and
server-side. Client-side load balancing balances the connection requests across the listeners.

O ly
With server-side load balancing, the listener directs a connection request to the best instance
currently providing the service by using the load balancing advisory. In a RAC database, client

l & On
connections should use both types of connection load balancing.

a e
FAN, Fast Connection Failover, and the load balancing advisory depend on an accurate
n
t e r
connection load balancing configuration that includes setting the connection load balancing
s
goal for the service. You can use a goal of either LONG or SHORT for connection load
U

I n
balancing. These goals have the following characteristics:
LONG: Use the LONG load balancing method for applications that have long-lived

c l e
connections. This is typical for connection pools and SQL*Forms sessions. LONG is the
default connection load balancing goal. The following is an example of modifying a

r a service, POSTMAN, with the srvctl utility to define the connection load balancing goal
for long-lived sessions:
O
srvctl modify service -s POSTMAN -j LONG
SHORT: Use the SHORT connection load balancing method for applications that have
short-lived connections. The following example modifies the ORDER service , using
srvctl to set the goal to SHORT:
srvctl modify service -s ORDER -j SHORT
Oracle Database 11g: RAC Administration 15 - 12
Services and Transparent Application Failover

Services simplify the deployment of Transparent


Application Failover (TAF).
You can define a TAF policy for a service and all
connections using this service will automatically have TAF
enabled.
The TAF setting on a service can be NONE, BASIC, or
PRECONNECT and overrides any TAF setting in the client
connection definition.
To define a TAF policy for a service, the srvctl utility can
be used as shown below:
srvctl modify service -s gl.example.com -q TRUE -P

y
BASIC -e SELECT -z 180 -w 5 -j LONG
Where -z is the number of retries, -w is the delay between retry attempts and -j is the
connection load balancing goal.
em
Copyright 2010, Oracle and/or its affiliates. All rights reserved.
a d
Services and Transparent Application Failover A c
l e
When Oracle Net Services establishes a connection to an instance, the connection remains
c
r a
open until the client closes the connection, the instance is shut down, or a failure occurs. If you
configure TAF for the connection, then Oracle Database moves the session to a surviving
instance when an outage occurs.
O ly
l & On
TAF can restart a query after failover has completed but for other types of transactions, such as
INSERT, UPDATE, or DELETE, the application must roll back the failed transaction and

n a e
resubmit the transaction. You must re-execute any session customizations, in other words,

t e r U s
ALTER SESSION statements, after failover has occurred. However, with TAF, a connection
is not moved during normal processing, even if the workload changes over time.

I n
Services simplify the deployment of TAF. You can define a TAF policy for a service, and all

c l e
connections using this service will automatically have TAF enabled. This does not require any
client-side changes. The TAF setting on a service overrides any TAF setting in the client

r a
connection definition. To define a TAF policy for a service, use the srvctl utility as in the
following example:
O srvctl modify service -s gl.example.com -q TRUE -P BASIC -e
SELECT -z 180 -w 5 -j LONG
Note: TAF applies only to an admin-managed database and not to policy-managed databases.

Oracle Database 11g: RAC Administration 15 - 13


Use Services with the Resource Manager

Consumer groups are automatically assigned to sessions


based on session services.
Work is prioritized by service inside one instance.

AP Instance resources

Connections AP 75%

BATCH BATCH 25%

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Use Services with the Resource Manager
e A
l
The Database Resource Manager (also called Resource Manager) enables you to identify work
c
r a
by using services. It manages the relative priority of services within an instance by binding
services directly to consumer groups. When a client connects by using a service, the consumer
O ly
group is assigned transparently at connect time. This enables the Resource Manager to manage

l & On
the work requests by service in the order of their importance.
For example, you define the AP and BATCH services to run on the same instance, and assign
a e
AP to a high-priority consumer group and BATCH to a low-priority consumer group. Sessions
n
t e r U s
that connect to the database with the AP service specified in their TNS connect descriptor get
priority over those that connect to the BATCH service.

I n
This offers benefits in managing workloads because priority is given to business functions

c l e
rather than the sessions that support those business functions.

r a
O
Oracle Database 11g: RAC Administration 15 - 14
Services and Resource Manager with EM

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Services and Resource Manager with EM
e A
l
Enterprise Manager (EM) presents a GUI through the Consumer Group Mapping page to
c
r a
automatically map sessions to consumer groups. You can access this page by clicking the
Consumer Group Mappings link on the Server page.
O ly
Using the General tabbed page of the Consumer Group Mapping page, you can set up a

right part of the slide.


l & On
mapping of sessions connecting with a service name to consumer groups as illustrated on the

n a e
With the ability to map sessions to consumer groups by service, module, and action, you have

e r s
greater flexibility when it comes to managing the performance of different application
workloads.
t U
I n
Using the Priorities tabbed page of the Consumer Group Mapping page, you can change

l e
priorities for the mappings that you set up on the General tabbed page. The mapping options

c
correspond to columns in V$SESSION. When multiple mapping columns have values, the

r a
priorities you set determine the precedence for assigning sessions to consumer groups.

O
Note: You can also map a service to a consumer group directly from the Create Service page
as shown on the left part of the slide.

Oracle Database 11g: RAC Administration 15 - 15


Use Services with the Scheduler

Services are associated with Scheduler classes.


Scheduler jobs have service affinity:
High Availability
Load balancing
HOT_BATCH_SERV HOT_BATCH_SERV LOW_BATCH_SERV

Job coordinator Job coordinator Job coordinator

Job slaves Job slaves Job slaves

Database

Job table
Job1 HOT_BATCH_CLASS HOT_BATCH_SERV
Job2 HOT_BATCH_CLASS HOT_BATCH_SERV
Job3 LOW_BATCH_CLASS LOW_BATCH_SERV

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Use Services with the Scheduler
e A
l
Just as in other environments, the Scheduler in a RAC environment uses one job table for each
c
r a
database and one job coordinator (CJQ0 process) for each instance. The job coordinators
communicate with each other to keep information current.
O ly
The Scheduler can use the services and the benefits they offer in a RAC environment. The

l & On
service that a specific job class uses is defined when the job class is created. During execution,
jobs are assigned to job classes and job classes run within services. Using services with job

n a e
classes ensures that the work of the Scheduler is identified for workload management and

t e r U s
performance tuning. For example, jobs inherit server-generated alerts and performance
thresholds for the service they run under.

I n
For High Availability, the Scheduler offers service affinity instead of instance affinity. Jobs

c l e
are not scheduled to run on any specific instance. They are scheduled to run under a service.
So, if an instance dies, the job can still run on any other instance in the cluster that offers the

r a
service.
Note: By specifying the service where you want the jobs to run, the job coordinators balance
O
the load on your system for better performance.

Oracle Database 11g: RAC Administration 15 - 16


Services and the Scheduler with EM

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Services and the Scheduler with EM
e A
l
To configure a job to run under a specific service, click the Job Classes link in the Database
c
r a
Scheduler section of the Server page. This opens the Scheduler Job Classes page. On the
Scheduler Job Classes page, you can see services assigned to job classes.
O ly
When you click the Create button on the Scheduler Job Classes page, the Create Job Class

service it must run under.


l & On
page is displayed. On this page, you can enter details of a new job class, including which

n a e
Note: Similarly, you can map a service to a job class on the Create Service page as shown at
the bottom of the slide.
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 15 - 17
Services and the Scheduler with EM

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Services and the Scheduler with EM (continued)
e A
l
After your job class is set up with the service that you want it to run under, you can create the
c
r a
job. To create the job, click the Jobs link on the Server page. The Scheduler Jobs page appears,
on which you can click the Create button to create a new job. When you click the Create
O ly
button, the Create Job page is displayed. This page has different tabs: General, Schedule, and

l & On
Options. Use the General tabbed page to assign your job to a job class.
Use the Options page (displayed in the slide) to set the Instance Stickiness attribute for your

n a e
job. Basically, this attribute causes the job to be load balanced across the instances for which

t e r U s
the service of the job is running. The job can run only on one instance. If the Instance
Stickiness value is set to TRUE, which is the default value, the Scheduler runs the job on the

I n
instance where the service is offered with the lightest load. If Instance Stickiness is set to
FALSE, then the job is run on the first available instance where the service is offered.

c l e
Note: It is possible to set job attributes, such as INSTANCE_STICKINESS, by using the

a
SET_ATTRIBUTE procedure of the DBMS_SCHEDULER PL/SQL package.
r
O
Oracle Database 11g: RAC Administration 15 - 18
Using Distributed Transactions with RAC

An XA transaction can span RAC instances, allowing any


application that uses XA to take full advantage of the
Oracle RAC environment.
Tightly coupled XA transactions no longer require the
special type of singleton services (DTP).
XA transactions are transparently supported on Oracle
RAC databases with any type of services configuration.
However, DTP services will improve performance for many
distributed transaction scenarios.
DTP services allow you to direct all branches of a
distributed transaction to a single instance in the cluster.
To load balance, it is better to have several groups of
smaller application servers with each group directing its
transactions to a single service. m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Using Distributed Transactions with RAC
e A
l
An XA transaction can span Oracle RAC instances by default, allowing any application that
c
r a
uses XA to take full advantage of the Oracle RAC environment to enhance the availability and
scalability of the application. This is controlled through the GLOBAL_TXN_PROCESSES
O ly
initialization parameter, which is set to 1 by default. This parameter specifies the initial

l & On
number of GTXn background processes for each Oracle RAC instance. Keep this parameter at
its default value clusterwide to allow distributed transactions to span multiple Oracle RAC
a e
instances. This allows the units of work performed across these Oracle RAC instances to share
n
t e r U s
resources and act as a single transaction (that is, the units of work are tightly coupled). It also
allows 2PC requests to be sent to any node in the cluster. Tightly coupled XA transactions no

I n
longer require the special type of singleton services (that is, Oracle Distributed Transaction
Processing [DTP] services) to be deployed on Oracle RAC database. XA transactions are

l e
transparently supported on Oracle RAC databases with any type of services configuration.
c
r a
To provide improved application performance with distributed transaction processing in
Oracle RAC, you may want to take advantage of the specialized service referred to as a DTP
O
Service. Using DTP services, you can direct all branches of a distributed transaction to a single
instance in the cluster. To load balance across the cluster, it is better to have several groups of
smaller application servers with each group directing its transactions to a single service, or set
of services, than to have one or two larger application servers.

Oracle Database 11g: RAC Administration 15 - 19


Distributed Transactions and Services
To leverage all instances, create one or more DTP services for
each instance that hosts distributed transactions.
Use EM or srvctl to create singleton services XA1, XA2
and XA3 for database CRM, enabling DTP for the service.
Below, EM is used to create service XA1:

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Distributed Transactions and Services
e A
l
To enhance the performance of distributed transactions, you can use services to manage DTP
c
r a
environments. By defining the DTP property of a service, the service is guaranteed to run on
one instance at a time in an Oracle RAC database. All global distributed transactions
O ly
performed through the DTP service are ensured to have their tightly coupled branches running

l & On
on a single Oracle RAC instance. This has the following benefits:
The changes are available locally within one Oracle RAC instance when tightly coupled
a e
branches need information about changes made by each other.
n
t e r U s
Relocation and failover of services are fully supported for DTP.
By using more DTP services than there are Oracle RAC instances, Oracle Database can

I n
balance the load by services across all the Oracle RAC database instances.
To leverage all the instances in a cluster, create one or more DTP services for each Oracle
l e
RAC instance that hosts distributed transactions. Choose one DTP service for one distributed
c
r a
transaction. Choose different DTP services for different distributed transactions to balance the
workload among the Oracle RAC database instances.
O
Because all the branches of a distributed transaction are on one instance, you can leverage all
the instances to balance the load of many DTP transactions through multiple singleton
services, thereby maximizing application throughput.

Oracle Database 11g: RAC Administration 15 - 20


Distributed Transactions and Services (continued)
An external transaction manager, such as OraMTS, coordinates DTP/XA transactions.
However, an internal Oracle transaction manager coordinates distributed SQL transactions.
Both DTP/XA and distributed SQL transactions must use the DTP service in Oracle RAC.
For services that you are going to use for distributed transaction processing, create the service
using Oracle Enterprise Manager or srvctl, and define only one instance as the preferred
instance. You can have as many AVAILABLE instances as you want. For example, the
following srvctl command creates a singleton service for database CRM,
xa1.example.com, whose preferred instance is orcl1:
srvctl add service -d CRM -s xa1.example.com -r orcl1 -a orcl2, orcl3
Then mark the service for distributed transaction processing by setting the DTP parameter to
TRUE; the default is FALSE.
srvctl modify service -d CRM -s xa1.example.com -x TRUE
Conversely, you can create the service and mark it for distributed transactions processing with
a single command if desired:
srvctl add service -d CRM -s xa1.example.com -r orcl1 -a orcl2,
orcl3 x TRUE
Oracle Enterprise Manager enables you to set this parameter on the Cluster Managed Database
Services: Create Service or Modify Service page.

m y
If, for example, orcl1 (that provides service XA1) fails, then the singleton service that it
e
provided fails over to another instance, such as orcl2 or orcl3. If services migrate to other
d
c a
instances after the cold-start of the Oracle RAC database, then you might have to force the
relocation of the service to evenly re-balance the load on all the available hardware. Use data
from the GV$ACTIVE_SERVICES view to determine whether to do this.

e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 15 - 21
Service Thresholds and Alerts

Service-level thresholds enable you to compare achieved


service levels against accepted minimum required levels.
You can explicitly specify two performance thresholds for
each service:
SERVICE_ELAPSED_TIME: The response time for calls
SERVICE_CPU_TIME: The CPU time for calls

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Service Thresholds and Alerts
e A
l
Service-level thresholds enable you to compare achieved service levels against accepted
c
r a
minimum required levels. This provides accountability for the delivery or the failure to deliver
an agreed service level. The end goal is a predictable system that achieves service levels.
O ly
There is no requirement to perform as fast as possible with minimum resource consumption;

l & On
the requirement is to meet the quality of service.
You can explicitly specify two performance thresholds for each service: the response time for

n a e
calls, or SERVICE_ELAPSED_TIME, and the CPU time for calls, or SERVICE_CPU_TIME. The

t e r U s
response time goal indicates that the elapsed time should not exceed a certain value, and the
response time represents wall clock time. Response time is a fundamental measure that reflects

I n
all delays and faults that might be blocking the call from running on behalf of the user.
Response time can also indicate differences in node power across the nodes of an Oracle RAC
database.
c l e
r a
The service time and CPU time are calculated as the moving average of the elapsed, server-
side call time. The AWR monitors the service time and CPU time and publishes AWR alerts
O
when the performance exceeds the thresholds. You can then respond to these alerts by
changing the priority of a job, stopping overloaded processes, or by relocating, expanding,
shrinking, starting, or stopping a service. This permits you to maintain service availability
despite changes in demand.
Oracle Database 11g: RAC Administration 15 - 22
Services and Thresholds Alerts: Example

EXECUTE DBMS_SERVER_ALERT.SET_THRESHOLD(
METRICS_ID => DBMS_SERVER_ALERT.ELAPSED_TIME_PER_CALL
, warning_operator => DBMS_SERVER_ALERT.OPERATOR_GE
, warning_value => '500000'
, critical_operator => DBMS_SERVER_ALERT.OPERATOR_GE
, critical_value => '750000'
, observation_period => 30
, consecutive_occurrences => 5
, instance_name => NULL
, object_type => DBMS_SERVER_ALERT.OBJECT_TYPE_SERVICE
, object_name => 'servall');

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Services and Thresholds Alerts: Example
e A
l
To check the thresholds for the servall service, use the AWR report. You should record
c
r a
output from the report over several successive intervals during which time the system is
running optimally. For example, assume that for an email server, the AWR report runs each
O ly
Monday during the peak usage times of 10:00 AM to 2:00 PM. The AWR report would

l & On
contain the response time, or DB time, and the CPU consumption time, or CPU time, for calls
for each service. The AWR report would also provide a breakdown of the work done and the
a e
wait times that are contributing to the response times.
n
t e r U s
Using DBMS_SERVER_ALERT, set a warning threshold for the servall service at 0.5
seconds and a critical threshold for the payroll service at 0.75 seconds. You must set these

I n
thresholds at all instances within an Oracle RAC database. The parameter instance_name
can be set to a NULL value to indicate database-wide alerts. You can schedule actions using
l e
Enterprise Manager jobs for alerts, or you can schedule actions to occur programmatically
c
r a
when the alert is received. In this example, thresholds are added for the servall service and
set as shown above.
O
Verify the threshold configuration using the following SELECT statement:
SELECT metrics_name, instance_name, warning_value,
critical_value, observation_period FROM dba_thresholds;

Oracle Database 11g: RAC Administration 15 - 23


Service Aggregation and Tracing

Statistics are always aggregated by service to measure


workloads for performance tuning.
Statistics can be aggregated at finer levels:
MODULE
ACTION
Combination of SERVICE_NAME, MODULE, ACTION
Tracing can be done at various levels:
SERVICE_NAME
MODULE
ACTION
Combination of SERVICE_NAME, MODULE, ACTION
This is useful for tuning systems that use shared sessions.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Service Aggregation and Tracing
e A
l
By default, important statistics and wait events are collected for the work attributed to every
c
r a
service. An application can further qualify a service by MODULE and ACTION names to
identify the important transactions within the service. This enables you to locate exactly the
O ly
poorly performing transactions for categorized workloads. This is especially important when

l & On
monitoring performance in systems by using connection pools or transaction processing
monitors. For these systems, the sessions are shared, which makes accountability difficult.
a e
SERVICE_NAME, MODULE, and ACTION are actual columns in V$SESSION.
n
t e r U s
SERVICE_NAME is set automatically at login time for the user. MODULE and ACTION names
are set by the application by using the DBMS_APPLICATION_INFO PL/SQL package or

I n
special OCI calls. MODULE should be set to a user-recognizable name for the program that is
currently executing. Likewise, ACTION should be set to a specific action or task that a user is
l e
performing within a module (for example, entering a new customer).
c
r a
Another aspect of this workload aggregation is tracing by service. The traditional method of
tracing each session produces trace files with SQL commands that can span workloads. This
O
results in a hit-or-miss approach to diagnose problematic SQL. With the criteria that you
provide (SERVICE_NAME, MODULE, or ACTION), specific trace information is captured in a
set of trace files and combined into a single output trace file. This enables you to produce trace
files that contain SQL that is relevant to a specific workload being done.
Oracle Database 11g: RAC Administration 15 - 24
Top Services Performance Page

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Top Services Performance Page
e A
l
From the Performance page, you can access the Top Consumers page by clicking the Top
c
Consumers link.
r a
The Top Consumers page has several tabs for displaying your database as a single-system
O ly
image. The Overview tabbed page contains four pie charts: Top Clients, Top Services, Top

l & On
Modules, and Top Actions. Each chart provides a different perspective regarding the top
resource consumers in your database.

n a e
The Top Services tabbed page displays performance-related information for the services that

e r s
are defined in your database. Using this page, you can enable or disable tracing at the service
t U
level, as well as view the resulting SQL trace file.

I n
c l e
r a
O
Oracle Database 11g: RAC Administration 15 - 25
Service Aggregation Configuration

Automatic service aggregation level of statistics


DBMS_MONITOR used for finer granularity of service
aggregations:
SERV_MOD_ACT_STAT_ENABLE
SERV_MOD_ACT_STAT_DISABLE
Possible additional aggregation levels:
SERVICE_NAME/MODULE
SERVICE_NAME/MODULE/ACTION
Tracing services, modules, and actions:
SERV_MOD_ACT_TRACE_ENABLE
SERV_MOD_ACT_TRACE_DISABLE
Database settings persist across instance restarts.
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Service Aggregation Configuration
e A
l
On each instance, important statistics and wait events are automatically aggregated and
c
r a
collected by service. You do not have to do anything to set this up, except connect with
different connect strings using the services you want to connect to. However, to achieve a finer
O ly
level of granularity of statistics collection for services, you must use the

l & On
SERV_MOD_ACT_STAT_ENABLE procedure in the DBMS_MONITOR package. This
procedure enables statistics gathering for additional hierarchical combinations of
a e
SERVICE_NAME/MODULE and SERVICE_NAME/MODULE/ACTION. The
n
t e r U s
SERV_MOD_ACT_STAT_DISABLE procedure stops the statistics gathering that was turned on.
The enabling and disabling of statistics aggregation within the service applies to every

I n
instance accessing the database. These settings are persistent across instance restarts.
The SERV_MOD_ACT_TRACE_ENABLE procedure enables tracing for services with three
l e
hierarchical possibilities: SERVICE_NAME, SERVICE_NAME/MODULE, and
c
SERVICE_NAME/MODULE/ACTION. The default is to trace for all instances that access the
r a
database. A parameter is provided that restricts tracing to specified instances where poor

O
performance is known to exist. This procedure also gives you the option of capturing relevant
waits and bind variable values in the generated trace files.
SERV_MOD_ACT_TRACE_DISABLE disables the tracing at all enabled instances for a given
combination of service, module, and action. Like the statistics gathering mentioned previously,
service tracing persists across instance restarts.
Oracle Database 11g: RAC Administration 15 - 26
Service, Module, and Action Monitoring

For the ERP service, enable monitoring for the exceptions


pay action in the PAYROLL module.
EXEC DBMS_MONITOR.SERV_MOD_ACT_STAT_ENABLE(
service_name => 'ERP', module_name=> 'PAYROLL',
action_name => 'EXCEPTIONS PAY')

For the ERP service, enable monitoring for all the actions in
the PAYROLL module:
EXEC DBMS_MONITOR.SERV_MOD_ACT_STAT_ENABLE(service_name =>
'ERP', module_name=> 'PAYROLL', action_name => NULL);

For the HOT_BATCH service, enable monitoring for all


actions in the posting module:
EXEC DBMS_MONITOR.SERV_MOD_ACT_STAT_ENABLE(service_name =>
'HOT_BATCH', module_name =>'POSTING', action_name => NULL);
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Service, Module, and Action Monitoring
e A
l
You can enable performance data tracing for important modules and actions within each
c
r a
service. The performance statistics are available in the V$SERV_MOD_ACT_STATS view.
Consider the following actions, as implemented in the slide above:
O ly
For the ERP service, enable monitoring for the exceptions pay action in the payroll
module.

l & On
Under the ERP service, enable monitoring for all the actions in the payroll module.
a e
Under the HOT_BATCH service, enable monitoring for all actions in the posting module.
n
COLUMN
t r U s
Verify the enabled service, module, action configuration with the SELECT statement below:
e
AGGREGATION_TYPE FORMAT A21 TRUNCATED HEADING 'AGGREGATION'
COLUMN
COLUMN I n
PRIMARY_ID FORMAT A20 TRUNCATED HEADING 'SERVICE'
QUALIFIER_ID1 FORMAT A20 TRUNCATED HEADING 'MODULE'
COLUMN
SELECT l e
QUALIFIER_ID2 FORMAT A20 TRUNCATED HEADING 'ACTION'

c
* FROM DBA_ENABLED_AGGREGATIONS ;

r a
The output might appear as follows:

O
AGGREGATION
---------------------
SERVICE_MODULE_ACTION
SERVICE
--------------------
ERP
MODULE ACTION
---------- -------------
PAYROLL EXCEPTIONS PAY
SERVICE_MODULE_ACTION ERP PAYROLL
SERVICE_MODULE_ACTION HOT_BATCH POSTING

Oracle Database 11g: RAC Administration 15 - 27


Service Performance Views

Service, module, and action information in:


V$SESSION
V$ACTIVE_SESSION_HISTORY
Service performance in:
V$SERVICE_STATS
V$SERVICE_EVENT
V$SERVICE_WAIT_CLASS
V$SERVICEMETRIC
V$SERVICEMETRIC_HISTORY
V$SERV_MOD_ACT_STATS
DBA_ENABLED_AGGREGATIONS
DBA_ENABLED_TRACES

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Service Performance Views
e A
l
The service, module, and action information are visible in V$SESSION and
c
V$ACTIVE_SESSION_HISTORY.
r a
The call times and performance statistics are visible in V$SERVICE_STATS,
O ly
V$SERVICE_EVENT, V$SERVICE_WAIT_CLASS, V$SERVICEMETRIC, and
V$SERVICEMETRIC_HISTORY.

l & On
When statistics collection for specific modules and actions is enabled, performance measures
a e
are visible at each instance in V$SERV_MOD_ACT_STATS.
n
t e r U s
There are more than 600 performance-related statistics that are tracked and visible in
V$SYSSTAT. Of these, 28 statistics are tracked for services. To see the statistics measured for

v$service_stats I n
services, run the following query: SELECT DISTINCT stat_name FROM

l e
Of the 28 statistics, DB time and DB CPU are worth mentioning. DB time is a statistic that
c
measures the average response time per call. It represents the actual wall clock time for a call

r a
to complete. DB CPU is an average of the actual CPU time spent per call. The difference

O
between response time and CPU time is the wait time for the service. After the wait time is
known, and if it consumes a large percentage of response time, then you can trace at the action
level to identify the waits.
Note: DBA_ENABLED_AGGREGATIONS displays information about enabled on-demand
statistic aggregation. DBA_ENABLED_TRACES displays information about enabled traces.
Oracle Database 11g: RAC Administration 15 - 28
Quiz

Which of the following statements regarding Oracle Services is


not correct?
1. You can group work by type under services.
2. Users who share a service should have the same service-
level requirements.
3. Use DBMS_SERVICE to manage services, not srvctl or
Enterprise Manager.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 3
e A
Statement 3 is not correct.
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 15 - 29
Quiz

Is the following statement regarding performance thresholds


true or false? The two performance thresholds that can be
explicitly set for each service are:
(a) SERVICE_ELAPSED_TIME: The response time for calls
(b) SERVICE_CPU_TIME: The CPU time for calls
1. True
2. False

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 1
e A
The statement is true.
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 15 - 30
Summary

In this lesson, you should have learned how to:


Configure and manage services in a RAC environment
Use services with client applications
Use services with the Database Resource Manager
Use services with the Scheduler
Set performance-metric thresholds on services
Configure services aggregation and tracing

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 15 - 31
Practice 15: Overview

This practice covers the following topics:


Creating and managing services using EM
Using server-generated alerts in combination with services

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 15 - 32
Design for High Availability

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Objectives

After completing this lesson, you should be able to:


Design a Maximum Availability Architecture in your
environment
Determine the best RAC and Data Guard topologies for
your environment
Configure the Data Guard Broker configuration files in a
RAC environment
Identify successful disk I/O strategies

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 16 - 2
Causes of Unplanned Down Time

Unplanned Down time

Software failures Hardware failures Human errors Disasters

Operating system CPU Operator error Fire

Database Memory User error Flood

Middleware Power supply DBA Earthquake

Application Bus System admin. Power failure

Network Disk Sabotage Bombing

Tape

Controllers
Network
Power
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Causes of Unplanned Down Time
e A
l
One of the true challenges in designing a highly available solution is examining and
c
r a
addressing all the possible causes of down time. It is important to consider causes of both
unplanned and planned down time. The diagram shown in the slide, which is a taxonomy of
O ly
unplanned failures, classifies failures as software failures, hardware failures, human error, and

category.
l & On
disasters. Under each category heading is a list of possible causes of failures related to that

n a e
Software failures include operating system, database, middleware, application, and network

t e r U s
failures. A failure of any one of these components can cause a system fault.
Hardware failures include system, peripheral, network, and power failures.

I n
Human error, which is a leading cause of failures, includes errors by an operator, user,

l e
database administrator, or system administrator. Another type of human error that can cause

c
unplanned down time is sabotage.

r a
The final category is disasters. Although infrequent, these can have extreme impacts on

O
enterprises, because of their prolonged effect on operations. Possible causes of disasters
include fires, floods, earthquakes, power failures, and bombings. A well-designed high-
availability solution accounts for all these factors in preventing unplanned down time.

Oracle Database 11g: RAC Administration 16 - 3


Causes of Planned Down Time

Planned down time

Routine operations Periodic maintenance New deployments

Backups Storage maintenance HW upgrade

Performance mgmt Initialization parameters OS upgrades

Security mgmt Software patches DB upgrades

Batches Schema management MidW upgrades

Operating system App upgrades

Middleware Net upgrades

Network

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Causes of Planned Down Time
e A
l
Planned down time can be just as disruptive to operations, especially in global enterprises that
c
r a
support users in multiple time zones, up to 24 hours per day. In these cases, it is important to
design a system to minimize planned interruptions. As shown by the diagram in the slide,
O ly
causes of planned down time include routine operations, periodic maintenance, and new

l & On
deployments. Routine operations are frequent maintenance tasks that include backups,
performance management, user and security management, and batch operations.

n a e
Periodic maintenance, such as installing a patch or reconfiguring the system, is occasionally

t e r U s
necessary to update the database, application, operating system middleware, or network.
New deployments describe major upgrades to the hardware, operating system, database,

I n
application, middleware, or network. It is important to consider not only the time to perform

c l e
the upgrade, but also the effect the changes may have on the overall application.

r a
O
Oracle Database 11g: RAC Administration 16 - 4
Oracles Solution to Down Time

RMAN backup/recovery
Fast-Start
RAC
Fault Recovery
Data Guard ASM
Streams
System
failures Flashback
Unplanned
down time
Data HARD
failures
Data Guard
&
Rolling upgrades/ Streams
System Online patching
changes
Planned Dynamic provisioning
down time
Data
changes
Online redefinition
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Oracles Solution to Down Time
e A
l
Unplanned down time is primarily the result of computer failures or data failures. Planned
c
r a
down time is primarily due to data changes or system changes:
RAC provides optimal performance, scalability, and availability gains.
O ly
Fast-Start Fault Recovery enables you to bound the crash/recovery time. The database

l & On
self-tunes checkpoint processing to safeguard the desired recovery time objective.
ASM provides a higher level of availability using online provisioning of storage.

n a e
Flashback provides a quick resolution to human errors.
Oracle Hardware Assisted Resilient Data (HARD) is a comprehensive program designed

e r s
to prevent data corruptions before they happen.
t U
Recovery Manager (RMAN) automates database backup and recovery.

I n
Data Guard must be the foundation of any Oracle database disaster-recovery plan.
The increased flexibility and capability of Streams over Data Guard with SQL Apply
l e
requires more expense and expertise to maintain an integrated high availability solution.
c
With online redefinition, the Oracle database supports many maintenance operations
r awithout disrupting database operations, or users updating or accessing data.

O Oracle Database continues to broaden support for dynamic reconfiguration, enabling it to


adapt to changes in demand and hardware with no disruption of service.
Oracle Database supports the application of patches to the nodes of a RAC system, as
well as database software upgrades, in a rolling fashion.

Oracle Database 11g: RAC Administration 16 - 5


RAC and Data Guard Complementarity

Resource Cause Protection

Nodes RAC

Component
Instances failure RAC
Software failure
Human
error Data Guard
Data &
Environment
Flashback

Data Guard
Site &
Streams
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
RAC and Data Guard Complementarity
e A
l
RAC and Data Guard together provide the benefits of system-level, site-level, and data-level
c
r a
protection, resulting in high levels of availability and disaster recovery without loss of data.
RAC addresses system failures by providing rapid and automatic recovery from failures,
O ly
such as node failures and instance crashes.

l & On
Data Guard addresses site failures and data protection through transactionally consistent
primary and standby databases that do not share disks, enabling recovery from site
a e
disasters and data corruption.
n
t e r U s
Note: Unlike Data Guard using SQL Apply, Oracle Streams enables updates on the replica and
provides support for heterogeneous platforms with different database releases. Therefore,

I n
Oracle Streams may provide the fastest approach for database upgrades and platform
migration.

c l e
r a
O
Oracle Database 11g: RAC Administration 16 - 6
Maximum Availability Architecture
Real-time query

Clients
Oracle Oracle
Application Application
Server Server
WAN Traffic
Manager

Real-time query

Primary Secondary
site Data Guard site

RAC
database
RAC databases:
Phys&log standby
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Maximum Availability Architecture (MAA)
e A
c l
RAC and Data Guard provide the basis of the database MAA solution. MAA provides the
most comprehensive architecture for reducing down time for scheduled outages and
r a
preventing, detecting, and recovering from unscheduled outages. The recommended MAA has

O ly
two identical sites. The primary site contains the RAC database, and the secondary site
contains both a physical standby database and a logical standby database on RAC. Identical
& On
site configuration is recommended to ensure that performance is not sacrificed after a failover
l
n a e
or switchover. Symmetric sites also enable processes and procedures to be kept the same
between sites, making operational tasks easier to maintain and execute.

t e r U s
The graphic illustrates identically configured sites. Each site consists of redundant components
and redundant routing mechanisms, so that requests are always serviceable even in the event of

I n
a failure. Most outages are resolved locally. Client requests are always routed to the site
playing the production role.
l e
After a failover or switchover operation occurs due to a serious outage, client requests are
c
r a
routed to another site that assumes the production role. Each site contains a set of application
servers or mid-tier servers. The site playing the production role contains a production database
O
using RAC to protect from host and instance failures. The site playing the standby role
contains one standby database, and one logical standby database managed by Data Guard.
Data Guard switchover and failover functions allow the roles to be traded between sites.
Note: For more information, see the following Web site:
http://otn.oracle.com/deploy/availability/htdocs/maa.htm
Oracle Database 11g: RAC Administration 16 - 7
RAC and Data Guard Topologies

Symmetric configuration with RAC at all sites:


Same number of instances
Same service preferences
Asymmetric configuration with RAC at all sites:
Different number of instances
Different service preferences
Asymmetric configuration with mixture of RAC and single
instance:
All sites running under Oracle Clusterware
Some single-instance sites not running under Oracle
Clusterware

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
RAC and Data Guard Topologies
e A
l
You can configure a standby database to protect a primary database in a RAC environment.
c
r a
Basically, all kinds of combinations are supported. For example, it is possible to have your
primary database running under RAC, and your standby database running as a single-instance
O ly
database. It is also possible to have both the primary and standby databases running under
RAC.

l & On
The slide explains the distinction between symmetric environments and asymmetric ones.

n a e
If you want to create a symmetric environment running RAC, then all databases need to have

e r s
the same number of instances and the same service preferences. As the DBA, you need to
t U
make sure that this is the case by manually configuring them in a symmetric way.

I n
However, if you want to benefit from the tight integration of Oracle Clusterware and Data

l e
Guard Broker, make sure that both the primary site and the secondary site are running under

c
Oracle Clusterware, and that both sites have the same services defined.

r a
Note: Beginning with Oracle Database 11g, the primary and standby systems in a Data Guard

O
configuration can have different CPU architectures, operating systems (for example, Windows
and Linux for physical standby database only with no EM support for this combination),
operating system binaries (32-bit and 64-bit), and Oracle database binaries (32-bit and 64-bit).

Oracle Database 11g: RAC Administration 16 - 8


RAC and Data Guard Architecture

Primary instance A Standby receiving instance C

ARCn
ARCn LGWR RFS
Flash
Primary recovery
database Standby area
Online redo
redo files
files Standby
Flash
recovery database
area LGWR RFS
Apply
ARCn
ARCn
Primary instance B Standby apply instance D
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
RAC and Data Guard Architecture
e A
l
Although it is perfectly possible to use a RAC to single-instance Data Guard (DG)
c
r a
configuration, you also have the possibility to use a RAC-to-RAC DG configuration. In this
mode, although multiple standby instances can receive redo from the primary database, only
O ly
one standby instance can apply the redo stream generated by the primary instances.

l & On
A RAC-to-RAC DG configuration can be set up in different ways, and the slide shows you one
possibility with a symmetric configuration where each primary instance sends its redo stream

n a e
to a corresponding standby instance using standby redo log files. It is also possible for each

t e r U s
primary instance to send its redo stream to only one standby instance that can also apply this
stream to the standby database. However, you can get performance benefits by using the

I n
configuration shown in the slide. For example, assume that the redo generation rate on the
primary is too great for a single receiving instance on the standby side to handle. Suppose
l e
further that the primary database is using the SYNC redo transport mode. If a single receiving
c
r a
instance on the standby cannot keep up with the primary, then the primarys progress is going
to be throttled by the standby. If the load is spread across multiple receiving instances on the
O
standby, then this is less likely to occur.
If the standby can keep up with the primary, another approach is to use only one standby
instance to receive and apply the complete redo stream. For example, you can set up the
primary instances to remotely archive to the same Oracle Net service name.
Oracle Database 11g: RAC Administration 16 - 9
RAC and Data Guard Architecture (continued)
You can then configure one of the standby nodes to handle that service. This instance then
both receives and applies redo from the primary. If you need to do maintenance on that node,
then you can stop the service on that node and start it on another node. This approach allows
for the primary instances to be more independent of the standby configuration because they are
not configured to send redo to a particular instance.
Note: For more information, refer to the Oracle Data Guard Concepts and Administration
guide.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 16 - 10
Data Guard Broker (DGB) and
Oracle Clusterware (OC) Integration
OC manages intrasite HA operations.
OC manages intrasite planned HA operations.
OC notifies when manual intervention is required.
DBA receives notification.
DBA decides to switch over or fail over using DGB.
DGB manages intersite planned HA operations.
DGB takes over from OC for intersite failover, switchover,
and protection mode changes:
DMON notifies OC to stop and disable the site, leaving all or
one instance.
DMON notifies OC to enable and start the site according to
the DG site role.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Data Guard Broker (DGB) and Oracle Clusterware Integration
e A
l
DGB is tightly integrated with Oracle Clusterware. Oracle Clusterware manages individual
c
r a
instances to provide unattended high availability of a given clustered database. DGB manages
individual databases (clustered or otherwise) in a Data Guard configuration to provide disaster
O ly
recovery in the event that Oracle Clusterware is unable to maintain availability of the primary
database.

l & On
For example, Oracle Clusterware posts NOT_RESTARTING events for the database group and

n a e
service groups that cannot be recovered. These events are available through Enterprise

t e r U s
Manager, ONS, and server-side callouts. As a DBA, when you receive those events, you might
decide to repair and restart the primary site, or to invoke DGB to fail over if not using Fast-
Start Failover.
I n
c l e
DGB and Oracle Clusterware work together to temporarily suspend service availability on the
primary database, accomplish the actual role change for both databases during which Oracle

r a
Clusterware works with the DGB to properly restart the instances as necessary, and then to
resume service availability on the new primary database. The broker manages the underlying
O
Data Guard configuration and its database roles, whereas Oracle Clusterware manages service
availability that depends upon those roles. Applications that rely upon Oracle Clusterware for
managing service availability will see only a temporary suspension of service as the role
change occurs within the Data Guard configuration.
Oracle Database 11g: RAC Administration 16 - 11
Fast-Start Failover: Overview

Fast-Start Failover implements automatic failover to a


standby database:
Triggered by failure of site, hosts, storage, data file offline
immediate, or network
Works with and supplements RAC server failover
Failover occurs in seconds (< 20 seconds).
Comparable to cluster failover
Original production site automatically rejoins the
configuration after recovery.
Automatically monitored by an Observer process:
Locate it on a distinct server on a distinct data center
Enterprise Manager can restart it on failure
Installed through Oracle Client Administrator
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Fast-Start Failover: Overview
e A
l
Fast-Start Failover is a feature that automatically, quickly, and reliably fails over to a
c
r a
designated, synchronized standby database in the event of loss of the primary database,
without requiring manual intervention to execute the failover. In addition, following a fast-
O ly
start failover, the original primary database is automatically reconfigured as a new standby

l & On
database upon reconnection to the configuration. This enables Data Guard to restore disaster
protection in the configuration as soon as possible.

n a e
Fast-Start Failover is used in a Data Guard configuration under the control of the Data Guard

t e r U s
Broker, and may be managed using either dgmgrl or Oracle Enterprise Manager Grid
Control. There are three essential participants in a Fast-Start Failover configuration:

I n
The primary database, which can be a RAC database
A target standby database, which becomes the new primary database following a fast-
l e
start failover
c
r a
The Fast-Start Failover Observer, which is a separate process incorporated into the
dgmgrl client that continuously monitors the primary database and the target standby
O database for possible failure conditions. The underlying rule is that out of these three
participants, whichever two can communicate with each other will determine the
outcome of the fast-start failover. In addition, a fast-start failover can occur only if there
is a guarantee that no data will be lost.

Oracle Database 11g: RAC Administration 16 - 12


Fast-Start Failover: Overview (continued)
For disaster recovery requirements, install the Observer in a location separate from the primary
and standby data centers. If the designated Observer fails, Enterprise Manager can detect the
failure and can be configured to automatically restart the Observer on the same host.
You can install the Observer by installing the Oracle Client Administrator (choose the
Administrator option from the Oracle Universal Installer). Installing the Oracle Client
Administrator results in a small footprint because an Oracle instance is not included on the
Observer system. If Enterprise Manager is used, also install the Enterprise Manager Agent on
the Observer system.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 16 - 13
Data Guard Broker Configuration Files

*.DG_BROKER_CONFIG_FILE1=+DG1/orcl/dr1config.dat
*.DG_BROKER_CONFIG_FILE2=+DG1/orcl/dr2config.dat

orcl1 orcl2

Shared storage

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Data Guard Broker Configuration Files
e A
l
Two copies of the Data Guard Broker (DGB) configuration files are maintained for each
c
r a
database so as to always have a record of the last-known valid state of the configuration. When
the broker is started for the first time, the configuration files are automatically created and
O ly
named using a default path name and file name that is operating system specific.

l & On
When using a RAC environment, the DGB configuration files must be shared by all instances
of the same database. You can override the default path name and file name by setting the
a e
following initialization parameters for that database: DG_BROKER_CONFIG_FILE1,
n
DG_BROKER_CONFIG_FILE2.

t e r U s
You have two possible options to share those files:

I n
Cluster file system
ASM

c l e
The example in the slide illustrates a case where those files are stored in an ASM disk group

r a
called DG1. It is assumed that you have already created a directory called orcl in DG1.

O
Oracle Database 11g: RAC Administration 16 - 14
Real-Time Query Physical Standby Database

Client Client Client Client


Read/Write Read only

Primary Standby
Redo apply
cluster cluster

Redo apply
instance

RAC RAC
database database

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Real-Time Query Physical Standby Database
e A
l
Data Guard Redo Apply (physical standby database) has proven to be a popular solution for
c
r a
disaster recovery due to its relative simplicity, high performance, and superior level of data
protection. Beginning with Oracle Database 11g, a physical standby database can be open
O ly
read-only while redo apply is active. This means that you can run queries and reports against

l & On
an up-to-date physical standby database without compromising data protection or extending
recovery time in the event a failover is required. This makes every physical standby database
a e
able to support productive uses even while in standby role. To enable real-time query, open the
n
t e r U s
database in read-only mode and then issue the ALTER DATABASE RECOVER MANAGED STANDBY
statement. Real-time query provides an ultimate high availability solution because it:

I n
Is totally transparent to applications
Supports Oracle RAC on the primary and standby databases: Although Redo Apply can

l e
be running on only one Oracle RAC instance, you can have all of the instances running
c
in read-only mode while Redo Apply is running on one instance.

r a
Returns transactionally consistent results that are very close to being up-to-date with the

O primary database.
Enables you to use fast-start failover to allow for automatic fast failover in the case the
primary database fails

Oracle Database 11g: RAC Administration 16 - 15


Hardware Assisted Resilient Data

Blocks validated and


Oracle database
protection information added to blocks
[DB_BLOCK_CHECKSUM=TRUE] Vol Man/ASM
Operating system
Prevents corruption introduced Device driver
in I/O path Host Bus Adapter
Is supported by major storage vendors:
EMC, Fujitsu, Hitachi, HP, NEC, SAN
Network Appliance &
Sun Microsystems Virtualization

All file types and block sizes checked


Protection information validated SAN interface
by storage device when enabled
symchksum type Oracle enable
Storage device
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Hardware Assisted Resilient Data
e A
l
One problem that can cause lengthy outages is data corruption. Today, the primary means for
c
r a
detecting corruptions caused by hardware or software outside of the Oracle database, such as
an I/O subsystem, is the Oracle database checksum. However, after a block is passed to the
O ly
operating system, through the volume manager and out to disk, the Oracle database itself

l & On
cannot validate whether the block being written is still correct.
With disk technologies expanding in complexity, and with configurations such as Storage Area

n a e
Networks (SANs) becoming more popular, the number of layers between the host processor

t e r U s
and the physical spindle continues to increase. With more layers, the chance of any problem
increases. With the HARD initiative, it is possible to enable the verification of database block

I n
checksum information by the storage device. Verifying that the block is still the same at the
end of the write as it was in the beginning gives you an additional level of security.

c l e
By default, the Oracle database automatically adds checksum information to its blocks. These

r a
checksums can be verified by the storage device if you enable this possibility. In case a block
is found to be corrupted by the storage device, the device logs an I/O corruption, or it cancels
O
the I/O and reports the error back to the instance.
Note: The way you enable the checksum validation at the storage device side is vendor
specific. The example given in the slide was used with EMC Symmetrix storage.

Oracle Database 11g: RAC Administration 16 - 16


Database High Availability: Best Practices

Use SPFILE. Create two or Set Multiplex


more control files. CONTROL_FILE_RECO production and
RD_KEEP_TIME long standby redo
enough. logs
Log checkpoints to Use auto-tune Enable ARCHIVELOG Enable
the alert log. checkpointing. mode and use a flash Flashback
recovery area. Database.

Enable block Use Automatic Use locally managed Use Automatic


checking. Undo tablespaces. Segment Space
Management. Management.

Use resumable Use Database Register all instances Use temporary


space allocation. Resource with remote listeners. tablespaces with
Manager. tempfiles.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Database High Availability: Best Practices
e A
l
The table in the slide gives you a short summary of the recommended practices that apply to
c
r a
single-instance databases, RAC databases, and Data Guard standby databases.
These practices affect the performance, availability, and mean time to recover (MTTR) of your
O ly
system. Some of these practices may reduce performance, but they are necessary to reduce or

l & On
avoid outages. The minimal performance impact is outweighed by the reduced risk of
corruption or the performance improvement for recovery.

n a e
Note: For more information about how to set up the features listed in the slide, refer to the
following documents:
Administrators Guide
t e r U s
I n
Data Guard Concepts and Administration

c l e
Net Services Administrators Guide

r a
O
Oracle Database 11g: RAC Administration 16 - 17
How Many ASM Disk Groups Per Database?

Two disk groups are recommended.


Leverage maximum of LUNs.
Data DG FRA DG
Backups can be stored on one
ERP DB
FRA disk group.
CRM DB
Lower performance may be
HR DB
used for FRA (or inner tracks).
Exceptions:
Additional disk groups for different capacity or performance
characteristics
Different ILM storage tiers

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
How Many ASM Disk Groups Per Database?
e A
l
Most of the time, only two disk groups are enough to share the storage between multiple
c
databases.
r a
That way you can maximize the number of Logical Unit Numbers (LUNs) used as ASM disks,
O ly
which gives you the best performance, especially if these LUNs are carved on the outer edge
of your disks.
l & On
n a e
Using a second disk group allows you to have a backup of your data by using it as your
common fast recovery area (FRA). You can put the corresponding LUNs on the inner edge of

e r s
your disks because less performance is necessary.
t U
I n
The two noticeable exceptions to this rule are whenever you are using disks with different
capacity or performance characteristics, or when you want to archive your data on lower-end

l e
disks for Information Lifecycle Management (ILM) purposes.

c
r a
O
Oracle Database 11g: RAC Administration 16 - 18
Which RAID Configuration for High Availability?

A. ASM mirroring
B. Hardware RAID 1 (mirroring)
C. Hardware RAID 5 (parity protection)
D. Both ASM mirroring and hardware RAID

Depends on business requirement and budget


Answer:
(cost, availability, performance, and utilization)

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Which RAID Configuration for Best Availability?
e A
l
To favor availability, you have multiple choices as shown in the slide.
c
r a
You could just use ASM mirroring capabilities, or hardware RAID 1 (Redundant Array of
Inexpensive Disks) which is a hardware mirroring technique, or hardware RAID 5. The last
O ly
possible answer, which is definitely not recommended, is to use both ASM mirroring and

l & On
hardware mirroring. Oracle recommends the use of external redundancy disk groups when
using hardware mirroring techniques to avoid an unnecessary overhead.

n a e
Therefore, between A, B, and C, the choice depends on your business requirements and
budget.
t e r U s
I n
RAID 1 has the best performance but requires twice the storage capacity. RAID 5 is a much
more economical solution but with a performance penalty essentially for write-intensive
workloads.

c l e
r a
O
Oracle Database 11g: RAC Administration 16 - 19
Should You Use ASM Mirroring Protection?

Best choice for low-cost storage


Enables extended clustering solutions
No hardware mirroring

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Should You Use ASM Mirroring Protection?
e A
l
Basically, leverage the storage array hardware RAID-1 mirroring protection when possible to
c
hardware RAID capability.
r a
offload the mirroring overhead from the server. Use ASM mirroring in the absence of a

O ly
However hardware RAID 1 in most Advanced Technology Attachment (ATA) storage

l & On
technologies is inefficient and degrades the performance of the array even more. Using ASM
redundancy has proven to deliver much better performance in ATA arrays.

n a e
Because the storage cost can grow very rapidly whenever you want to achieve extended

e r s
clustering solutions, ASM mirroring should be used as an alternative to hardware mirroring for
t
low-cost storage solutions.
U
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 16 - 20
What Type of Striping Works Best?

A. ASM only striping (no RAID 0)


B. RAID 0 and ASM striping
C. Use LVM
D. No striping

Answer: A and B

ASM and RAID striping are complementary.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
What Type of Striping Works Best?
e A
l
As shown in the slide, you can use ASM striping only, or you can use ASM striping in
c
combination with RAID 0.
r a
With RAID 0, multiple disks are configured together as a set, or a bank, and data from any one
O ly
data file is spread, or striped, across all the disks in the bank.

& On
Combining both ASM striping and RAID striping is called stripe-on-stripe. This combination
l
offers good performance too.

n a e
t e r
However, there is no longer a need to use a Logical Volume Manager (LVM) for your
s
database files, nor is it recommended to not use any striping at all.
U
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 16 - 21
ASM Striping Only

Pros: Cons:
Drives evenly distributed for Data & FRA Not well balanced across
Higher bandwidth ALL disks
Allows small incremental growth (73 GB) LUN size limited to disk size
No drive contention

Oracle DB size: 1 TB
Data DG FRA DG
Storage configuration:
1 TB 1673 GB 8arrays with 2 TB 3273 GB
LUNs 1273 GB disks per array LUNs
RAID 1

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
ASM Striping Only
e A
l
In the case shown in this slide, you want to store a one-terabyte database with a corresponding
c
r a
two-terabyte flash recovery area. You use RAID 1 to mirror each disk. In total, you have eight
arrays of twelve disks, with each disk being 73 GB. ASM mirroring and hardware RAID 0 are
not used.
O ly
l & On
In addition, each ASM disk is represented by one entire LUN of 73 GB. This means that the
Data disk group (DG) is allocated 16 LUNs of 73 GB each.

n a e
On the other side, the Fast Recovery Area disk group is assigned 32 LUNs of 73 GB each.

e r s
This configuration enables you to evenly distribute disks for your data and backups, achieving
t U
I n
good performance and allowing you to manage your storage in small incremental chunks.
However, using a restricted number of disks in your pool does not balance your data well

l e
across all your disks. In addition, you have many LUNs to manage at the storage level.
c
r a
O
Oracle Database 11g: RAC Administration 16 - 22
Hardware RAIDStriped LUNs

Pros: Cons:
Fastest region for Data DG Large incremental growth
Balanced data distribution Data & FRA contention
Fewer LUNs to manage while max
spindles

Oracle DB size: 1 TB
Data DG FRA DG
Storage configuration:
1 TB 4250 GB 8arrays with 2 TB 4500 GB
LUNs 1273 GB disks per array LUNs
RAID 0+1

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Hardware RAIDStriped LUNs
e A
l
In the case shown in this slide, you want to store a one-terabyte database with a corresponding
c
r a
two-terabyte flash recovery area. You use RAID 0+1, which is a combination of hardware
striping and mirroring to mirror and stripe each disk. In total, you have eight arrays of twelve
O ly
disks, with each disk being 73 GB. ASM mirroring is not used.

l & On
Here, you can define bigger LUNs not restricted to the size of one of your disks. This allows
you to put the Data LUNs on the fastest region of your disks, and the backup LUNs on slower

n a e
parts. By doing this, you achieve a better data distribution across all your disks, and you end

t e r U s
up managing a significantly less number of LUNs.
However, you must manipulate your storage in much larger chunks than in the previous
configuration.
I n
l e
Note: The hardware stripe size you choose is also very important because you want 1 MB

c
alignment as much as possible to keep in sync with ASM AUs. Therefore, selecting power-of-

r a
two stripe sizes (128 KB or 256 KB) is better than selecting odd numbers. Storage vendors
typically do not offer many flexible choices depending on their storage array RAID technology
O
and can create unnecessary I/O bottlenecks if not carefully considered.

Oracle Database 11g: RAC Administration 16 - 23


Hardware RAIDStriped LUNs HA

Pros: Cons:
Fastest region for Data DG Large incremental growth
Balanced data distribution Might waste space
Fewer LUNs to manage
More high available

Oracle DB size: 1 TB
Data DG FRA DG
Storage configuration:
1 TB 2500 GB 8arrays with 1.6 TB 2800 GB
LUNs 1273 GB disks per array LUNs
RAID 0+1

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Hardware RAIDStriped LUNs HA
e A
l
In the case shown in this slide, you want to store a one-terabyte database with a corresponding
c
r a
1.6-TB fast recovery area. You use RAID 0+1, which is a combination of hardware striping
and mirroring to mirror and stripe each disk. In total, you have eight arrays of twelve disks,
O ly
with each disk being 73 GB. ASM mirroring is not used.

l & On
Compared to the previous slide, you use bigger LUNs for both the Data disk group and the
Fast Recovery Area disk group. However, the presented solution is more highly available than

n a e
the previous architecture because you separate the data from the backups into different arrays

t e r U s
and controllers to reduce the risk of down time in case one array fails.
By doing this, you still have a good distribution of data across your disks, although not as

I n
much as in the previous configuration. You still end up managing a significantly less number

c l e
of LUNs than in the first case.
However, you might end up losing more space than in the previous configuration. Here, you

r a
are using the same size and number of arrays to be consistent with the previous example.

O
Oracle Database 11g: RAC Administration 16 - 24
Disk I/O Design Summary

Use external RAID protection when possible.


Create LUNs by using:
Outside half of disk drives for highest performance
Small disk, high rpm (that is, 73 GB/15k rpm)
Use LUNs with the same performance characteristics.
Use LUNs with the same capacity.
Maximize the number of spindles in your disk group.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Disk I/O Design Summary
e A
l
Use ASM for volume and file management to equalize the workload across disks and
c
ASM disk groups:
r a
eliminate hot spots. The following are simple guidelines and best practices when configuring

O ly
Use external RAID protection when possible.
Create LUNs using:

l & On
- Outside half of disk drives for highest performance

n a e
- Small disk with high rpm (for example, 73 GB with 15k rpm). The reason why
spindle (platter) speed is so important is that it directly impacts both positioning

e r s
time and data transfer. This means that faster spindle speed drives have improved
t U
performance regardless of whether they are used for many small, random accesses,

I n
or for streaming large contiguous blocks from the disk. The stack of platters in a

c l e
disk rotates at a constant speed. The drive head, while positioned close to the center
of the disk, reads from a surface that is passing by more slowly than the surface at

r a the outer edges.


Maximize the number of spindles in your disk group.
O LUNs provisioned to ASM disk groups should have the same storage performance and
availability characteristics. Configuring mixed speed drives will default to the lowest
common denominator.
ASM data distribution policy is capacity based. Therefore, LUNs provided to ASM
should have the same capacity for each disk group to avoid imbalance and hot spots.
Oracle Database 11g: RAC Administration 16 - 25
Extended RAC: Overview

Full utilization of resources, no matter where they are


located

Site A RAC Site B


database

Clients

Site A RAC Site B


database

Fast recovery from site failure

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Extended RAC: Overview
e A
l
Typically, RAC databases share a single set of storage and are located on servers in the same
c
data center.
r a
With extended RAC, you can use disk mirroring and Dense Wavelength Division Multiplexing
O ly
(DWDM) equipment to extend the reach of the cluster. This configuration allows two data

l & On
centers, separated by up to 100 kilometers, to share the same RAC database with multiple
RAC instances spread across the two sites.

n a e
As shown in the slide, this RAC topology is very interesting, because the clients work gets

e r s
distributed automatically across all nodes independently of their location, and in case one site
t U
goes down, the clients work continues to be executed on the remaining site. The types of

I n
failures that extended RAC can cover are mainly failures of an entire data center due to a

c l e
limited geographic disaster. Fire, flooding, and site power failure are just a few examples of
limited geographic disasters that can result in the failure of an entire data center.

r a
Note: Extended RAC does not use special software other than the normal RAC installation.

O
Oracle Database 11g: RAC Administration 16 - 26
Extended RAC Connectivity

Distances over ten kilometers require dark fiber.


Set up buffer credits for large distances.

Dark fiber
Site A Site B

DWDM DWDM
device device

DB DB
copy copy

Clients
Public network

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Extended RAC Connectivity
e A
l
In order to extend a RAC cluster to another site separated from your data center by more than
c
r a
ten kilometers, it is required to use DWDM over dark fiber to get good performance results.
DWDM is a technology that uses multiple lasers, and transmits several wavelengths of light
O ly
simultaneously over a single optical fiber. DWDM enables the existing infrastructure of a

l & On
single fiber cable to be dramatically increased. DWDM systems can support more than 150
wavelengths, each carrying up to 10 Gbps. Such systems provide more than a terabit per

n a e
second of data transmission on one optical strand that is thinner than a human hair.

e r s
As shown in the slide, each site should have its own DWDM device connected together by a
t U
dark fiber optical strand. All traffic between the two sites is sent through the DWDM and

I n
carried on dark fiber. This includes mirrored disk writes, network and heartbeat traffic, and

c l e
memory-to-memory data passage. Also shown on the graphic are the sets of disks at each site.
Each site maintains a copy of the RAC database.

r a
It is important to note that depending on the sites distance, you should tune and determine the
minimum value of buffer credits in order to maintain the maximum link bandwidth. Buffer
O
credit is a mechanism defined by the Fiber Channel standard that establishes the maximum
amount of data that can be sent at any one time.
Note: Dark fiber is a single fiber optic cable or strand mainly sold by telecom providers.

Oracle Database 11g: RAC Administration 16 - 27


Extended RAC Disk Mirroring

Need copy of data at each location


Two options:
Host-based mirroring
Remote array-based mirroring

Recommended solution with ASM

Site A Site B Primary Secondary

DB DB DB DB
copy copy copy copy

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Extended RAC Disk Mirroring
e A
l
Although there is only one RAC database, each data center has its own set of storage that is
c
r a
synchronously mirrored using either a cluster-aware, host-based Logical Volume Manager
(LVM) solution, such as SLVM with MirrorDiskUX, or an array-based mirroring solution,
such as EMC SRDF.
O ly
l & On
With host-based mirroring, shown on the left of the slide, the disks appear as one set, and all
I/Os get sent to both sets of disks. This solution requires closely integrated clusterware and

n a e
LVM, and ASM is the recommended solution.

e r s
With array-based mirroring, shown on the right, all I/Os are sent to one site, and are then
t U
mirrored to the other. In fact, this solution is like a primary/secondary site setup. If the primary

I n
site fails, all access to primary disks is lost. An outage may be incurred before you can switch

c l e
to the secondary site.
Note: With extended RAC, designing the cluster in a manner that ensures the cluster can

r a
achieve quorum after a site failure is a critical issue. For more information regarding this topic,
refer to the Oracle Technology Network site.
O
Oracle Database 11g: RAC Administration 16 - 28
Achieving Quorum with Extended RAC

Third
site
Voting disk (NFS or iSCSI)

Redundant public network

Redundant private network

Site A Site B

Redundant SAN with ASM mirroring


RAC RAC
database database

Database files (ASM failure group)


OCR
Database files (ASM failure group)
OCR
m y
Voting Disk Voting Disk

d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Achieving Quorum with Extended RAC
e A
l
As far as voting disks are concerned, a node must be able to access strictly more than half of
c
r a
the voting disks at any time, or that node will be evicted from the cluster. Extended clusters
are generally implemented with only two storage systems, one at each site. This means that the
O ly
site that houses the majority of the voting disks is a potential single point of failure for the

l & On
entire cluster. To prevent this potential outage, Oracle Clusterware supports a third voting disk
on an inexpensive, low-end, standard NFS-mounted device somewhere on the network. It is
a e
thus recommended to put this third NFS voting disk on a dedicated server visible from both
n
t e r U s
sites. This situation is illustrated in the slide. The goal is that each site can run independently
of the other when a site failure occurs.

I n
Note: For more information about NFS configuration of the third voting disk, refer to the
Oracle Technology Network site.

c l e
r a
O
Oracle Database 11g: RAC Administration 16 - 29
Additional Data Guard Benefits

Greater disaster protection


Greater distance
Additional protection against corruptions
Better for planned maintenance
Full rolling upgrades
More performance neutral at large distances
Option to do asynchronous
If you cannot handle the costs of a DWDM network, Data
Guard still works over cheap, standard networks.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Additional Data Guard Benefits
e A
l
Data Guard provides a greater disaster protection:
c
r a
Distance over 100 kilometers without performance hit
Additional protection against corruptions because it uses a separate database
O ly
Optional delay to protect against user errors

upgrades.
l & On
Data Guard also provides better planned maintenance capabilities by supporting full rolling

n a e
Also, if you cannot handle the costs of a DWDM network, then Data Guard still works over
cheap, standard networks.
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 16 - 30
Using a Test Environment

The most common cause of down time is change.


Test your changes on a separate test cluster before
changing your production environment.

Production Test
cluster cluster

RAC RAC
database database

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Using a Test Environment
e A
l
Change is the most likely cause of down time in a production environment. A proper test
c
r a
environment can catch more than 90 percent of the changes that could lead to a down time of
the production environment, and is invaluable for quick test and resolution of issues in
production.
O ly
l & On
When your production environment is RAC, your test environment should be a separate RAC
cluster with all the identical software components and versions.

n a e
Without a test cluster, your production environment will not be highly available.

e r s
Note: Not using a test environment is one of the most common errors seen by Oracle Support
t U
Services.
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 16 - 31
Quiz

Which of the following statements regarding Disk I/O design are


true?
1. Use external RAID protection when possible.
2. Use LUNs with the same performance characteristics.
3. Use LUNs with the same capacity.
4. Minimize the number of spindles in your disk group.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 1,2,3
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 16 - 32
Summary

In this lesson, you should have learned how to:


Design a Maximum Availability Architecture in your
environment
Determine the best RAC and Data Guard topologies for
your environment
Configure the Data Guard Broker configuration files in a
RAC environment
Identify successful disk I/O strategies

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration 16 - 33
m y
d e
ca
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Appendix A
Practices and Solutions

m y
d e
ca
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Table of Contents
Practices for Lesson 11 ....................................................................................................... 3
Practice 11-1: Installing the Oracle Database Software ................................................. 4
Practice 11-2: Creating a RAC Database ........................................................................ 6
Practices for Lesson 12 ....................................................................................................... 8
Practice 12-1: Operating System and Password File Authenticated Connections.......... 9
Practice 12-2: Oracle Database Authenticated Connections ........................................ 12
Practice 12-3: Stopping a Complete ORACLE_HOME Component Stack ................. 16
Practices for Lesson 13 ..................................................................................................... 19
Practice 13-1: Configuring the Archive Log Mode ...................................................... 20
Practice 13-2: Configuring Specific Instance Connection Strings ............................... 23
Practice 13-3: Configuring RMAN and Performing Parallel Backups ......................... 27
Practices for Lesson 14 ..................................................................................................... 33
Practice 14-1: ADDM and RAC Part I ......................................................................... 34
Practice 14-2: ADDM and RAC Part II ........................................................................ 41
Practice 14-3: ADDM and RAC Part III....................................................................... 45
Practices for Lesson 15 ..................................................................................................... 48
Practice 15-1: Working with Services .......................................................................... 49
Practice 15-2: Monitoring Services .............................................................................. 53
Practice 15-3: Services and Alert Thresholds ............................................................... 56

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 2
Practices for Lesson 11
In this practice, you will install the Oracle Database 11g Release 2 software and create a
three-node cluster database.
You must have Oracle Database 11g Release 2 Grid Infrastructure installed on all three
cluster nodes and the appropriate ASM disk groups available to support the cluster
database. Before beginning this practice, the
/home/oracle/solutions/catchup10/catchup.sh script must execute
successfully. To perform this step, change directory to
/home/oracle/solutions/catchup10 and execute as the root user. Answer
all password prompts with 0racle.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 3
Practice 11-1: Installing the Oracle Database Software
In this practice, you will install the Oracle Database 11g Release 2 software on three
nodes.
1) Use the VNC session on ST_NODE1 on display :1. This is the oracle users VNC
session. Click the VNC icon on your desktop, enter ST_NODE1:1, substituting the
name of your first node for ST_NODE1. Enter the password 0racle.
2) In the VNC session, confirm that you are connected as the oracle user with the
proper group memberships. Change directory to the staged software location provided
by your instructor and start the OUI by executing the runInstaller command
from the /staged_software_location/database/Disk1 directory.
$ id
uid=501(oracle) gid=502(oinstall)
groups=501(dba),502(oinstall),503(oper),505(asmdba)
$ cd /stage/database/Disk1

$ ./runInstaller

a) On the Configure Security Updates page, deselect the I wish to receive security
updates check box and click Next. Note: If this were a production machine or
part of an important test environment, you might consider this option. A dialog
box appears making sure that you want to remain uninformed about the updates.
m y
Click Yes to close the dialog box and continue.

d e
b) On the Select Installation Option page, select the Install database software only
option and click Next.
c a
A
c) On the Node Selection page, select Real Application Clusters database
installation. Select all three of your assigned hosts and click Next. If the ssh
e
c l
connectivity test fails, click the SSH Connectivity button. Enter the password
0racle for the oracle user. Select the Reuse public and private keys check box

r a
and click the Setup button. After the setup completes, click Next to continue.

O ly
d) On the Select Product Languages, promote all languages from the Available
Languages window to the Selected Languages window on the right-hand side.
Click Next to continue.
l & On
a e
e) On the Select Database Edition, select Enterprise Edition and click Next to
n
continue.

t e r U s
f) On the Specify Installation edition, the Oracle Base should be

I n
/u01/app/oracle and the Software Location should be
/u01/app/oracle/product/11.2.0/dbhome_1. Do not install the

c l e
database to a shared location. Click Next to continue.
g) On the Privileged Operating System Groups, select dba as the Database

r a Administrator Group and oper as the Database Operator Group. Click Next to
continue.
O
Oracle Database 11g: RAC Administration A - 4
Practice 11-1: Installing the Oracle Database Software
(continued)

h) When the prerequisites have successfully been checked on the Perform


Prerequisites Check page, click Next to continue. If any checks fail, click the Fix
and Check Again button. Run the scripts on the cluster nodes as root as directed,
and then click Next.
i) Check the information on the Summary page and click Finish.
j) The Install Progress screen allows you to monitor the progression of the
installation.
k) When the files have been copied to all nodes, the Execute Configuration Scripts
window is presented. Execute the
/u01/app/oracle/product/11.2.0/dbhome_1/root.sh script on
all three nodes.
# /u01/app/oracle/product/11.2.0/dbhome_1/root.sh

Running Oracle 11g root.sh script...

The following environment variables are set as:


ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/oracle/product/11.2.0/dbhome_1

Enter the full pathname of the local bin directory:


m y
[/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite
d e
it? (y/n)
[n]: n
c a
The file "oraenv" already exists in /usr/local/bin. Overwrite
it? (y/n)
e A
[n]: n

c l
The file "coraenv" already exists in /usr/local/bin.
Overwrite it? (y/n)
[n]: n
r a
O ly
Entries will be added to the /etc/oratab file as needed by

l & On
Database Configuration Assistant when a database is created
Finished running generic part of root.sh script.
a e
Now product-specific root actions will be performed.
n
#

t e r
Finished product-specific root actions.

U s
I n
Run this script on the remaining two nodes before continuing.

c l e
l) When you have run the root.sh scripts on all three nodes, click the OK button
to close the Execute Configuration Scripts window.

Ora m) Click the Close button on the Finish page to complete the installation and exit the
Installer.

Oracle Database 11g: RAC Administration A - 5


Practice 11-2: Creating a RAC Database
In this practice, you will create a three-node RAC database.
1) From the oracle VNC session that you used to install the database software in the
preceding practice, change directory to
/u01/app/oracle/product/11.2.0/dbhome_1/bin and launch the
Database Configuration Assistant by executing the dbca command.
$ id
uid=500(oracle) gid=501(oinstall)
groups=500(dba),501(oinstall),502(oper),505(asmdba)

$ cd /u01/app/oracle/product/11.2.0/dbhome_1/bin

$ ./dbca

a) On the Welcome page, select Oracle Real Application Clusters database and click
Next.
b) On the Operations page, select Create a Database and click Next.
c) On the Database Templates page, select the General Purpose or Transaction
Processing template. Click Next to continue.
d) On the Database Identification page, select Admin-Managed and enter orcl in
the Global Database Name field. Click the Select All button to install the database
to all three of your nodes and click Next.
m y
e) On the Management Options page, make sure that the Configure Enterprise
d e
management option is selected. Click Next to continue.
c a
Manager check box is selected and the Configure Database Control for local

e A
f) Select Use the Same Administrative Password for All Accounts on the
Database Credentials page. Enter oracle_4U in the Password and Confirm
Password fields and click Next.
c l
r a
g) On Database File locations, you can specify your storage type and location for
your database files. Select Automatic Storage Management (ASM) from the drop-
O ly
down list for Storage Type. Select Use Oracle-Managed Files and make sure that

l & On
the Database Area value is +DATA. Click Next to continue.
h) When the ASM Credentials box appears, enter the ASM password that you used
a e
during the clusterware installation practice 2.2, step k. The password should be
n
t e r U s
oracle_4U. Enter the correct password and click OK to close the box.
i) On the Recovery Configuration page, enter +FRA in the Recovery Area field and

I n
accept the default for the Recovery Area Size. Click Next to continue.

c l e
j) On the Database Content page, select the Sample Schemas check box and click
Next.

r a
O
Oracle Database 11g: RAC Administration A - 6
Practice 11-2: Creating a RAC Database (continued)
k) On the Memory tabbed page of the Initialization Parameters page, make sure that
you enter 550 in the Memory Size (SGA and PGA) field. Then click the
Character Sets tab, and select Use Unicode on the Database Character Set tabbed
page. Click Next.
l) Accept the default values on the Database Storage page and click Next to
continue.
m) Select Create Database on the Creation Options page and click Finish.
n) On the Database Configuration Assistant: Summary page, click OK.
o) You can monitor the database creation progress from the Database Configuration
Assistant window.
p) At the end of the installation, a dialog box with your database information
including the Database Control URL is displayed. Click Exit. This will close the
dialog box and end the Database Configuration Assistant.
q) Open a browser and enter the Database Control URL displayed in the previous
step. Verify that all instances are up. Verify that all cluster resources are up across
all three nodes.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 7
Practices for Lesson 12
In these practices, you will contrast operating system, password file authenticated
connections, and Oracle database authenticated connections. You will also learn to stop a
complete ORACLE_HOME component stack

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 8
Practice 12-1: Operating System and Password File
Authenticated Connections
In this practice, you will make both operating system authenticated connections and
password file authenticated connections to the database instance. You will also examine
problems with the oraenv script.
1) Connect to your first node as the oracle user and set up your environment variables
using the oraenv script.
$ . oraenv
ORACLE_SID = [oracle] ? orcl
The Oracle base for
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is
/u01/app/oracle

2) Identify all the database instance names that are currently executing on your machine
using the Linux ps command. Note: All database instances have a mandatory
background process named pmon, and the instance name will be part of the complete
process name.
$ ps -ef | grep -i pmon
grid 4111 1 0 10:53 ? 00:00:00 asm_pmon_+ASM1
oracle 4987 1 0 10:56 ? 00:00:00 ora_pmon_orcl1
oracle 5299 5257 0 10:58 pts/0 00:00:00 grep -i pmon

m y
e
3) Attempt to make a local connection to the orcl1 instance using SQL*Plus with the
d
c a
sysdba privilege. This is known as operating system authentication because a
password is not needed. What happens when trying to connect to the instance?
$ sqlplus / as sysdba

e A
2009 c l
SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 8 10:59:04

r a
Copyright (c) 1982, 2009, Oracle. All rights reserved.
O ly
& On
Connected to an idle instance.

l
SQL> exit
Disconnected.
n a e
$

t e r U s
I n
4) Attempt to connect to the instance using a network connection string @orcl with the

l e
sysdba privilege. This is known as password file authentication. Is the connection
successful this time?
c
Ora $ sqlplus sys@orcl as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 8 11:03:35


2009

Oracle Database 11g: RAC Administration A - 9


Practice 12-1: Operating System and Password File
Authenticated Connections (continued)

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Enter password: oracle_4U << Password is not displayed

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 -
Production
With the Partitioning, Real Application Clusters, Automatic
Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> exit

5) Display the values of the environment variables (ORACLE_BASE, ORACLE_HOME,


ORACLE_SID, PATH, and LD_LIBRARY_PATH) that were defined with the
oraenv script in step 1.
$ env | grep ORA
ORACLE_SID=orcl
ORACLE_BASE=/u01/app/oracle
m y
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1

d e
$ env | grep PATH
LD_LIBRARY_PATH=/u01/app/oracle/product/11.2.0/dbhome_1/lib
PATH=/usr/kerberos/bin:/usr/local/bin:/bin:/usr/bin:/home/orac c a
le/bin:/u01/app/oracle/product/11.2.0/dbhome_1/bin

e A
name for the orcl database. c l
6) Modify the ORACE_SID environment variable to match the actual database instance

$ export ORACLE_SID=orcl1 r a
O ly
7) Attempt the local connection with system authentication to the orcl1 instance using
& On
SQL*Plus with the sysdba privilege. This is the same command as in step 3.
l
$ sqlplus / as sysdba

n a e
2009
t e r U s
SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 8 11:01:48

I n
Copyright (c) 1982, 2009, Oracle. All rights reserved.

c l e
r aConnected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 -
Production
O
Oracle Database 11g: RAC Administration A - 10
Practice 12-1: Operating System and Password File
Authenticated Connections (continued)

With the Partitioning, Real Application Clusters, Automatic


Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL>

8) Query the instance_name column of the v$instance dynamic performance


view to validate the instance that you connected with. Exit SQL*Plus when finished.
SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
orcl1

SQL> exit

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 11
Practice 12-2: Oracle Database Authenticated Connections
In this practice, you will make multiple Oracle database authenticated connections to a
database instance and notice the effects of load balanced connections.
1) From your first node, connected as the oracle user, validate the instance names on
each host.
$ . /home/oracle/labs/st_env.sh
$ ssh $ST_NODE1 ps -ef | grep pmon
grid 4111 1 0 10:53 ? 00:00:00 asm_pmon_+ASM1
oracle 4987 1 0 10:56 ? 00:00:00 ora_pmon_orcl1
oracle 7932 7779 0 11:56 pts/0 00:00:00 grep pmon

$ ssh $ST_NODE2 ps -ef | grep pmon


grid 4104 1 0 10:53 ? 00:00:00 asm_pmon_+ASM2
oracle 4959 1 0 10:56 ? 00:00:00 ora_pmon_orcl2

$ ssh $ST_NODE3 ps -ef | grep pmon


grid 4096 1 0 10:53 ? 00:00:00 asm_pmon_+ASM3
oracle 4841 1 0 10:55 ? 00:00:00 ora_pmon_orcl3
2) Verify the current host name, and then set the environment variables using the
oraenv script.
$ hostname
host01.example.com
m y
$ . oraenv
d e
ORACLE_SID = [oracle] ? orcl
The Oracle base for
c a
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is
/u01/app/oracle

e A
c l
3) Connect to a database instance by using SQL*Plus with the system account. This is

r a
known as Oracle database authentication. After it is connected, query the
instance_name column from the v$instance dynamic performance view.

O ly
Note: Your instance names may vary from the ones displayed below.
$ sqlplus system@orcl

l & On
2009
n a e
SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 8 12:05:06

t e r U s
Copyright (c) 1982, 2009, Oracle. All rights reserved.

I n
Enter password: oracle_4U << Password is not displayed

l e
Connected to:
c
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 -

r aProduction
With the Partitioning, Real Application Clusters, Automatic
O Storage Management, OLAP,

Oracle Database 11g: RAC Administration A - 12


Practice 12-2: Oracle Database Authenticated Connections
(continued)

Data Mining and Real Application Testing options

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
orcl2

SQL>

4) Use the SQL*Plus host command to temporarily exit SQL*Plus and return to the
operating system prompt. Note: SQL*Plus is still running when this is performed.
Validate that you are still on your first node. Repeat step 3 from the operating system
prompt to establish a second SQL*Plus session and database instance connection.
What instance name did you connect to?
SQL> host
$ hostname
host01.example.com

$ sqlplus system@orcl

SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 8 12:13:02


m y
2009

d e
Copyright (c) 1982, 2009, Oracle. All rights reserved.

Enter password: oracle_4U << Password is not displayed c a


e A
Connected to:

c l
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 -
Production

r a
With the Partitioning, Real Application Clusters, Automatic
Storage Management, OLAP,
O ly
Data Mining and Real Application Testing options

l & On
SQL> select instance_name from v$instance;

INSTANCE_NAME
n a e
----------------
orcl1
t e r U s
SQL>
I n
l e
5) Use the SQL*Plus host command to temporarily exit SQL*Plus and return to the
c
operating system prompt. Note: SQL*Plus is still running when this is performed.

r aValidate that you are still on your first node. Repeat step 3 from the operating system
prompt to establish a third SQL*Plus session and database instance connection. What
O instance name did you connect to?

Oracle Database 11g: RAC Administration A - 13


Practice 12-2: Oracle Database Authenticated Connections
(continued)

SQL> HOST
$ hostname
host01.example.com
[oracle@host01 ~]$ sqlplus system@orcl

SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 8 12:15:52


2009

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Enter password: oracle_4U << Password is not displayed

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 -
Production
With the Partitioning, Real Application Clusters, Automatic
Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> select instance_name from v$instance;

INSTANCE_NAME
m y
----------------
orcl3
d e
SQL>
c a
A
6) Exit the three SQL*Plus sessions that are currently executing on the first node.

e
SQL> exit
c l
Disconnected from Oracle Database 11g Enterprise Edition

r
Release 11.2.0.1.0 - Production
a
With the Partitioning, Real Application Clusters, Automatic
Storage Management, OLAP,
O ly
Data Mining and Real Application Testing options

$ exit
l & On
exit
n a e
SQL> exit
t e r U s
Disconnected from Oracle Database 11g Enterprise Edition

I n
Release 11.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, Automatic

l e
Storage Management, OLAP,

c
Data Mining and Real Application Testing options

r a$ exit

O exit

Oracle Database 11g: RAC Administration A - 14


Practice 12-2: Oracle Database Authenticated Connections
(continued)

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition
Release 11.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, Automatic
Storage Management, OLAP,
Data Mining and Real Application Testing options

$ exit << Optional. This will exit your terminal session.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 15
Practice 12-3: Stopping a Complete ORACLE_HOME Component
Stack
In this practice, you will use the srvctl utility to stop all resource components
executing from a single home location.
1) From your first node, connect to the oracle account and set up the environment
variables for the database instance.
$ . oraenv
ORACLE_SID = [oracle] ? orcl
The Oracle base for
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is
/u01/app/oracle
$ . /home/oracle/labs/st_env.sh

2) Validate that the instances are running on each node of the cluster.
Note: There are several ways this step can be performed.
$ ssh $ST_NODE1 ps -ef | grep pmon
grid 4111 1 0 10:53 ? 00:00:00 asm_pmon_+ASM1
oracle 4987 1 0 10:56 ? 00:00:00 ora_pmon_orcl1
oracle 5321 5254 0 12:45 pts/0 00:00:00 grep pmon

$ ssh $ST_NODE2 ps -ef | grep pmon


grid
oracle
4104
4959
1 0 10:53 ?
1 0 10:56 ?
00:00:00 asm_pmon_+ASM2
00:00:00 ora_pmon_orcl2
m y
$ ssh $ST_NODE3 ps -ef | grep pmon
d e
grid
oracle
4096
4841
1 0 10:53 ?
1 0 10:55 ?
c a
00:00:00 asm_pmon_+ASM3
00:00:00 ora_pmon_orcl3

>>>>>>>>>>>>> or <<<<<<<<<<<<<<<<<
e A
$ srvctl
c l
status database -d orcl -v
Instance
Instance
r a
orcl1 is running on node host01
orcl2 is running on node host02
Instance
O ly
orcl3 is running on node host03

$ srvctl status asm -a


l & On
ASM is running on host01,host02,host03
ASM is enabled.
n a e
e r s
>>>>>>>>>>>>> or <<<<<<<<<<<<<<<<<
t U
I n
$ /u01/app/11.2.0/grid/bin/crsctl status resource ora.orcl.db
NAME=ora.orcl.db

c l e
TYPE=ora.database.type
TARGET=ONLINE , ONLINE , ONLINE

r aSTATE=ONLINE on host01, ONLINE on host02, ONLINE on host03

O $ /u01/app/11.2.0/grid/bin/crsctl status resource ora.asm


NAME=ora.asm

Oracle Database 11g: RAC Administration A - 16


Practice 12-3: Stopping a Complete ORACLE_HOME Component
Stack (continued)

TYPE=ora.asm.type
TARGET=ONLINE , ONLINE , ONLINE
STATE=ONLINE on host01, ONLINE on host02, ONLINE on host03

3) Display the syntax usage help for the srvctl status home command.
$ srvctl status home -help

Displays the current state of of all resources for the Oracle


home.

Usage: srvctl status home -o <oracle_home> -s <state_file> -n


<node_name>
-o <oracle_home> ORACLE_HOME path
-s <state_file> Specify a file path for the 'srvctl stop
home' command to store the state of the resources
-n <node_name> Node name
-h Print usage

4) Use the srvctl status home command to check the state of all resources
running from the /u01/app/oracle/product/11.2.0/dbhome_1 home
location. Create the required state file in the /tmp directory with the file name
host01_dbhome_state.dmp for the first node only.
m y
$ srvctl status home -o
/u01/app/oracle/product/11.2.0/dbhome_1 -s
d e
/tmp/host01_dbhome_state.dmp -n $ST_NODE1
c a
Database orcl is running on node host01

e A
l
5) Display the syntax usage help for the srvctl stop home command.
c
$ srvctl stop home -help

r a
O ly
Stops all Oracle clusterware resources that run from the
Oracle home.

l & On
Usage: srvctl stop home -o <oracle_home> -s <state_file> -n

-o <oracle_home>
n a e
<node_name> [-t <stop_options>] [-f]
ORACLE_HOME path
-s <state_file>

t e r U s
Specify a file path for the 'srvctl stop
home' command to store the state of the resources

I n
-n <node_name>
-t <stop_options>
Node name
Stop options for the database.

e
Examples of shutdown options are normal, transactional,
l
immediate, or abort.
c -f Force stop

r a -h Print usage

O
Oracle Database 11g: RAC Administration A - 17
Practice 12-3: Stopping a Complete ORACLE_HOME Component
Stack (continued)

6) Stop all resources executing in the


/u01/app/oracle/product/11.2.0/dbhome_1 home using the state file
created in step 4. Do not use the optional parameters identified by square brackets
[] displayed in the syntax usage help.
$ srvctl stop home -o /u01/app/oracle/product/11.2.0/dbhome_1
-s /tmp/host01_dbhome_state.dmp -n $ST_NODE1
$
7) Check the status of the database instances on each node.
Note: There are several ways this step can be performed. Do not use the srvctl
status home command with the same state file created above.
$ srvctl status database -d orcl -v
Instance orcl1 is not running on node host01
Instance orcl2 is running on node host02
Instance orcl3 is running on node host03

8) Start all resources for the /u01/app/oracle/product/11.2.0/dbhome_1


home using the state file created by the stop command.
$ srvctl start home -o /u01/app/oracle/product/11.2.0/dbhome_1
-s /tmp/host01_dbhome_state.dmp -n $ST_NODE1
$
m y
9) Check the status of the database instances on each node.
d e
Note: There are several ways this step can be performed.
$ srvctl c a
Instance
status database -d orcl -v
orcl1 is running on node host01

e A
Instance
Instance
c l
orcl2 is running on node host02
orcl3 is running on node host03

r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 18
Practices for Lesson 13
In this practice, you will configure ARCHIVELOG mode for your RAC database,
configure instance-specific connect strings for RMAN, and configure persistent RMAN
settings.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 19
Practice 13-1: Configuring the Archive Log Mode
In this practice, you will configure the archive log mode of a Real Applications Cluster
database.
1) From the first node of your cluster, open a terminal session as the oracle user and
set up the environment variables using the oraenv script for the database instance.
Change the value of the ORACLE_SID variable to allow local system authenticated
connections.
$ . oraenv
ORACLE_SID = [oracle] ? orcl
The Oracle base for
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is
/u01/app/oracle

$ export ORACLE_SID=orcl1

2) Make a local connection using operating system authentication to the database


instance and then use the archive log list SQL command to determine if the
database is in archive log mode. Exit SQL*Plus when done.
$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 8 15:48:58


2009
m y
Copyright (c) 1982, 2009, Oracle. All rights reserved.
d e
Connected to:
c a
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 -
Production
e A
Storage Management, OLAP,
c l
With the Partitioning, Real Application Clusters, Automatic

r a
Data Mining and Real Application Testing options

SQL> archive log list


O ly
Database log mode
Automatic archival
l & On No Archive Mode
Disabled
Archive destination

n a e
Oldest online log sequence
USE_DB_RECOVERY_FILE_DEST
11
Current log sequence
SQL> exit
$
t e r U s 12

I n
3) Stop the orcl database on each node of the cluster using the srvctl stop

l e
database command.

c
r a$ srvctl stop database -d orcl

4) Verify that the orcl database is not running on any node of the cluster using the
O srvctl status database command.

Oracle Database 11g: RAC Administration A - 20


Practice 13-1: Configuring the Archive Log Mode (continued)

$ srvctl status database -d orcl -v


Instance orcl1 is not running on node host01
Instance orcl2 is not running on node host02
Instance orcl3 is not running on node host03
5) Make a local connection using operating system authentication to the database
instance and then start up the database on the first node only with the mount option.
$ sqlplus / as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 8 16:27:17


2009

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to an idle instance.

SQL> startup mount


ORACLE instance started.

Total System Global Area 577511424 bytes


Fixed Size 1338000 bytes
Variable Size
Database Buffers
Redo Buffers
444597616
125829120
5746688
bytes
bytes
bytes
m y
Database mounted.
SQL>
d e
c
6) Issue the alter database archivelog SQL command to change the archive a
SQL command.
e A
mode of the database and then verify the results with the archive log list

SQL> alter database archivelog;


c l
Database altered.
r a
SQL> archive log list O ly
Database log mode
Automatic archival
l & On Archive Mode
Enabled
Archive destination

n a e
Oldest online log sequence
USE_DB_RECOVERY_FILE_DEST
11

t e
Current log sequence
r U s
Next log sequence to archive 12
12

I n
7) Shut down the database instance with the immediate option and exit SQL*Plus. Use

c l e
the srvctl utility to restart the database instances on all nodes of the cluster.
SQL> shutdown immediate

r aORA-01109: database not open

O Database dismounted.

Oracle Database 11g: RAC Administration A - 21


Practice 13-1: Configuring the Archive Log Mode (continued)

ORACLE instance shut down.

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition
Release 11.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, Automatic
Storage Management, OLAP,
Data Mining and Real Application Testing options

$ srvctl start database -d orcl


$

8) Verify that the orcl database is running on all the three nodes of your cluster by
using the srvctl status database command.
$ srvctl status database -d orcl -v
Instance orcl1 is running on node host01
Instance orcl2 is running on node host02
Instance orcl3 is running on node host03
$

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 22
Practice 13-2: Configuring Specific Instance Connection Strings
In this practice, you will modify the tnsnames.ora file to disable connection load
balancing and allow specific named instances to be used for connectivity.
1) Examine the $ORACLE_HOME/network/admin/tnsnames.ora file. There should
be only one entry. This entry allows load balancing of connections as you observed in
Practice 12-2.
$ cat
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames
.ora

# tnsnames.ora Network Configuration File:


/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames
.ora
# Generated by Oracle configuration tools.

ORCL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = cluster03-
scan.cluster03.example.com)(PORT = 1521))
(CONNECT_DATA =
(SERVER = DEDICATED)

)
)
(SERVICE_NAME = orcl.example.com)

m y
2) Execute the /home/oracle/labs/less_13/fix_tns.sh script. This script
d e
c a
will add three additional entries to the tnsnames.ora file that disable load
balancing of connections by requiring a specific INSTANCE_NAME when used.
Examine the changes made to the tnsnames.ora file.
e A
l
$ /home/oracle/labs/less_13/fix_tns.sh
c
$ cat
r a
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames
.ora
O ly
l & On
# tnsnames.ora Network Configuration File:
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames
.ora

n a e
# Generated by Oracle configuration tools.

ORCL =
t
(DESCRIPTION = e r U s
I n
(ADDRESS = (PROTOCOL = TCP)(HOST = cluster03-
scan.cluster03.example.com)(PORT = 1521))

c l e
(CONNECT_DATA =
(SERVER = DEDICATED)

r a )
(SERVICE_NAME = orcl)

O )

Oracle Database 11g: RAC Administration A - 23


Practice 13-2: Configuring Specific Instance Connection Strings
(continued)

# Added by lab 13
orcl1 =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = host01)(PORT=1521))
)
(CONNECT_DATA =
(SERVICE_NAME = orcl)
(INSTANCE_NAME = orcl1)
)
)
orcl2 =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = host02)(PORT=1521))
)
(CONNECT_DATA =
(SERVICE_NAME = orcl)
(INSTANCE_NAME = orcl2)
)
)
orcl3 =
(DESCRIPTION =
m y
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = host03)(PORT=1521))
d e
)
(CONNECT_DATA =
c a
(SERVICE_NAME = orcl)
(INSTANCE_NAME = orcl3)

e A
)
)

c l
r a
3) Using one of the three new entries in the tnsnames.ora file, connect to the system
database account using SQL*Plus and verify the instance name to see that it matches
the specific entry. O ly
$ sqlplus system@orcl2
l & On
n a e
SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 8 17:21:26
2009

t e r U s
Copyright (c) 1982, 2009, Oracle. All rights reserved.

I n
Enter password: oracle_4U << Password is not displayed

c l e
Connected to:

r aOracle Database 11g Enterprise Edition Release 11.2.0.1.0 -


Production

O With the Partitioning, Real Application Clusters, Automatic


Storage Management, OLAP,

Oracle Database 11g: RAC Administration A - 24


Practice 13-2: Configuring Specific Instance Connection Strings
(continued)

Data Mining and Real Application Testing options

SQL> select instance_name from v$instance;

INSTANCE_NAME
----------------
orcl2

SQL>

4) Use the SQL*Plus host command to temporarily exit SQL*Plus and return to the
operating system prompt. Note: SQL*Plus is still running when this is performed.
Repeat step 3 from the operating system prompt to establish a second SQL*Plus
session and database instance connection using the same connection string. Verify
that the INSTANCE_NAME stays the same.
SQL> host

$ sqlplus system@orcl2

SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 8 17:26:30


2009

m y
Copyright (c) 1982, 2009, Oracle. All rights reserved.

d e
Enter password: oracle_4U << Password is not displayed

c a
Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 -

e A
Production

c l
With the Partitioning, Real Application Clusters, Automatic
Storage Management, OLAP,

r a
Data Mining and Real Application Testing options

O ly
SQL> select instance_name from v$instance;

INSTANCE_NAME
l & On
----------------
orcl2
n a e
e r s
5) Exit both SQL*Plus sessions.
t U
SQL> exit
I n
Disconnected from Oracle Database 11g Enterprise Edition

l e
Release 11.2.0.1.0 - Production

c
With the Partitioning, Real Application Clusters, Automatic

r aStorage Management, OLAP,


Data Mining and Real Application Testing options

O $ exit

Oracle Database 11g: RAC Administration A - 25


Practice 13-2: Configuring Specific Instance Connection Strings
(continued)

exit

SQL> exit
Disconnected from Oracle Database 11g Enterprise Edition
Release 11.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, Automatic
Storage Management, OLAP,
Data Mining and Real Application Testing options
$

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 26
Practice 13-3: Configuring RMAN and Performing Parallel
Backups
In this practice, you will designate your first and second nodes of the cluster as nodes
responsible for performing parallel backups of the database. The database will be backed
up to the +FRA ASM disk group by default.
1) Using the recovery manager utility (RMAN), connect to the orcl database as the
target database.
$ rman target /

Recovery Manager: Release 11.2.0.1.0 - Production on Tue Sep 8


17:31:34 2009

Copyright (c) 1982, 2009, Oracle and/or its affiliates. All


rights reserved.

connected to target database: ORCL (DBID=1224399398)

RMAN>

2) Display all of the current RMAN settings.


RMAN> show all;

using target database control file instead of recovery catalog


m y
RMAN configuration parameters for database with db_unique_name
ORCL are:
d e
CONFIGURE RETENTION POLICY TO REDUNDANCY 1; # default
CONFIGURE BACKUP OPTIMIZATION OFF; # default
c a
CONFIGURE DEFAULT DEVICE TYPE TO DISK; # default
CONFIGURE CONTROLFILE AUTOBACKUP OFF; # default
e A
TO '%F'; # default
c l
CONFIGURE CONTROLFILE AUTOBACKUP FORMAT FOR DEVICE TYPE DISK

BACKUPSET; # default
r a
CONFIGURE DEVICE TYPE DISK PARALLELISM 1 BACKUP TYPE TO

O ly
CONFIGURE DATAFILE BACKUP COPIES FOR DEVICE TYPE DISK TO 1; #
default

# default
l & On
CONFIGURE ARCHIVELOG BACKUP COPIES FOR DEVICE TYPE DISK TO 1;

a e
CONFIGURE MAXSETSIZE TO UNLIMITED; # default
n
e r
CONFIGURE ENCRYPTION FOR DATABASE OFF; # default
s
CONFIGURE ENCRYPTION ALGORITHM 'AES128'; # default

t U
CONFIGURE COMPRESSION ALGORITHM 'BASIC' AS OF RELEASE

I n
'DEFAULT' OPTIMIZE FOR LOAD TRUE ; # default
CONFIGURE ARCHIVELOG DELETION POLICY TO NONE; # default

l e
CONFIGURE SNAPSHOT CONTROLFILE NAME TO

c
'/u01/app/oracle/product/11.2.0/dbhome_1/dbs/snapcf_orcl1.f';

r a# default

3) Configure RMAN to automatically back up the control file and server parameter file
O each time any backup operation is performed.

Oracle Database 11g: RAC Administration A - 27


Practice 13-3: Configuring RMAN and Performing Parallel
Backups (continued)

RMAN> configure controlfile autobackup on;

new RMAN configuration parameters:


CONFIGURE CONTROLFILE AUTOBACKUP ON;
new RMAN configuration parameters are successfully stored

4) Configure all backups done to disk to be done in parallel 2 degrees of parallelism.


RMAN> configure device type disk parallelism 2;

new RMAN configuration parameters:


CONFIGURE DEVICE TYPE DISK PARALLELISM 2 BACKUP TYPE TO
BACKUPSET;
new RMAN configuration parameters are successfully stored

5) Configure channel 1 and channel 2 to use the connect string


sys/oracle_4U@orcl# when performing a parallel backup to disk. Replace the
pound sign (#) with 1 for channel 1 and 2 for channel 2, respectively. This will
designate your first and second nodes as dedicated backup nodes for the cluster using
the node specific connection strings created earlier. Without node specific connection
strings, there would be no control over which nodes are being connected to in order to
perform the backups.
RMAN> configure channel 1 device type disk
m y
connect='sys/oracle_4U@orcl1';
d e
new RMAN configuration parameters:
CONFIGURE CHANNEL 1 DEVICE TYPE DISK CONNECT '*';
c a
new RMAN configuration parameters are successfully stored

e A
connect='sys/oracle_4U@orcl2';
c l
RMAN> configure channel 2 device type disk

r a
new RMAN configuration parameters:
O ly
CONFIGURE CHANNEL 2 DEVICE TYPE DISK CONNECT '*';

RMAN>
l & On
new RMAN configuration parameters are successfully stored

a e
6) Open a second terminal session as the oracle account and set up the environment
n
t e r s
variables for the orcl database. Navigate to the ~/labs/less_13 directory,
invoke SQL*plus as the system user, and run the monitor_rman.sql script. Do
U
prompt.
I n
not exit the first session with the RMAN prompt or this second session with the SQL

l e
$ . oraenv

c
ORACLE_SID = [oracle] ? orcl

r aThe Oracle base for


ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is

O /u01/app/oracle

Oracle Database 11g: RAC Administration A - 28


Practice 13-3: Configuring RMAN and Performing Parallel
Backups (continued)

$ cd /home/oracle/labs/less_13

$ sqlplus system@orcl

SQL*Plus: Release 11.2.0.1.0 Production on Tue Sep 8 20:37:36


2009

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Enter password: oracle_4U << Password is not displayed

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 -
Production
With the Partitioning, Real Application Clusters, Automatic
Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> @monitor_rman.sql

no rows selected
7) In the first session with the RMAN prompt, perform a full database backup with
m y
d e
archive logs. The backup should happen only on the designated nodes (your first and
second nodes) as the backup nodes. Do not wait for this step to finish before
proceeding to the next step!
RMAN> backup database plus archivelog; c a
e A
Starting backup at 08-SEP-09
current log archived
c l
allocated channel: ORA_DISK_1
r a
channel ORA_DISK_1: SID=31 instance=orcl1 device type=DISK
O ly
allocated channel: ORA_DISK_2

l & On
channel ORA_DISK_2: SID=55 instance=orcl2 device type=DISK
channel ORA_DISK_1: starting archived log backup set

n a e
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=2 sequence=4 RECID=1 STAMP=697048583

STAMP=697062223
t r U s
input archived log thread=3 sequence=10 RECID=4

e
channel ORA_DISK_1: starting piece 1 at 08-SEP-09

I n
channel ORA_DISK_2: starting archived log backup set
channel ORA_DISK_2: specifying archived log(s) in backup set

l e
input archived log thread=1 sequence=12 RECID=2

c
STAMP=697062220

r ainput archived log thread=2 sequence=5 RECID=3 STAMP=697062220


channel ORA_DISK_2: starting piece 1 at 08-SEP-09

O channel ORA_DISK_2: finished piece 1 at 08-SEP-09

Oracle Database 11g: RAC Administration A - 29


Practice 13-3: Configuring RMAN and Performing Parallel
Backups (continued)

piece
handle=+FRA/orcl/backupset/2009_09_08/annnf0_tag20090908t20234
7_0.268.697062231 tag=TAG20090908T202347 comment=NONE
channel ORA_DISK_2: backup set complete, elapsed time:
00:00:04
channel ORA_DISK_1: finished piece 1 at 08-SEP-09
piece
handle=+FRA/orcl/backupset/2009_09_08/annnf0_tag20090908t20234
7_0.267.697062229 tag=TAG20090908T202347 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time:
00:00:05
Finished backup at 08-SEP-09

Starting backup at 08-SEP-09


using channel ORA_DISK_1
using channel ORA_DISK_2
channel ORA_DISK_1: starting full datafile backup set
channel ORA_DISK_1: specifying datafile(s) in backup set
input datafile file number=00001
name=+DATA/orcl/datafile/system.260.696618497
input datafile file number=00005
name=+DATA/orcl/datafile/example.264.696618709
input datafile file number=00004
m y
name=+DATA/orcl/datafile/users.267.696618501
input datafile file number=00006
d e
name=+DATA/orcl/datafile/undotbs2.259.696619015
channel ORA_DISK_1: starting piece 1 at 08-SEP-09
c a
A
channel ORA_DISK_2: starting full datafile backup set
channel ORA_DISK_2: specifying datafile(s) in backup set

e
input datafile file number=00002

c l
name=+DATA/orcl/datafile/sysaux.268.696618499

r a
input datafile file number=00003
name=+DATA/orcl/datafile/undotbs1.263.696618499

O ly
input datafile file number=00007
name=+DATA/orcl/datafile/undotbs3.258.696619021

l & On
channel ORA_DISK_2: starting piece 1 at 08-SEP-09
channel ORA_DISK_2: finished piece 1 at 08-SEP-09
piece

n a e
handle=+FRA/orcl/backupset/2009_09_08/nnndf0_tag20090908t20235

t e r U s
4_0.270.697062239 tag=TAG20090908T202354 comment=NONE
channel ORA_DISK_2: backup set complete, elapsed time:
00:01:35
I n
channel ORA_DISK_1: finished piece 1 at 08-SEP-09
piece
l e
handle=+FRA/orcl/backupset/2009_09_08/nnndf0_tag20090908t20235
c
4_0.269.697062235 tag=TAG20090908T202354 comment=NONE

r a
channel ORA_DISK_1: backup set complete, elapsed time:
00:01:47
O Finished backup at 08-SEP-09

Oracle Database 11g: RAC Administration A - 30


Practice 13-3: Configuring RMAN and Performing Parallel
Backups (continued)

Starting backup at 08-SEP-09


current log archived
using channel ORA_DISK_1
using channel ORA_DISK_2
channel ORA_DISK_1: starting archived log backup set
channel ORA_DISK_1: specifying archived log(s) in backup set
input archived log thread=1 sequence=13 RECID=5
STAMP=697062345
input archived log thread=2 sequence=6 RECID=7 STAMP=697062345
channel ORA_DISK_1: starting piece 1 at 08-SEP-09
channel ORA_DISK_2: starting archived log backup set
channel ORA_DISK_2: specifying archived log(s) in backup set
input archived log thread=3 sequence=11 RECID=6
STAMP=697062345
channel ORA_DISK_2: starting piece 1 at 08-SEP-09
channel ORA_DISK_1: finished piece 1 at 08-SEP-09
piece
handle=+FRA/orcl/backupset/2009_09_08/annnf0_tag20090908t20254
6_0.274.697062347 tag=TAG20090908T202546 comment=NONE
channel ORA_DISK_1: backup set complete, elapsed time:
00:00:02
channel ORA_DISK_2: finished piece 1 at 08-SEP-09
m y
piece
handle=+FRA/orcl/backupset/2009_09_08/annnf0_tag20090908t20254
d e
6_0.275.697062347 tag=TAG20090908T202546 comment=NONE
channel ORA_DISK_2: backup set complete, elapsed time:
c a
00:00:01
Finished backup at 08-SEP-09

e A
c l
Starting Control File and SPFILE Autobackup at 08-SEP-09
piece

r a
handle=+FRA/orcl/autobackup/2009_09_08/s_697062349.276.6970623
53 comment=NONE
O ly
Finished Control File and SPFILE Autobackup at 08-SEP-09

RMAN>
l & On
n a e
8) While the backup is in progress, rerun the query on the second terminal window to

t e r U s
monitor the RMAN backup session progress within the cluster. The backup should be
done in parallel, with work distributed to both the backup nodes of the cluster. Enter

I n
the slash (/) symbol and press the Enter key to rerun the query. It may be necessary
to do this multiple times until the output appears.

l e
SQL> /
c
Ora no rows selected

SQL> /

Oracle Database 11g: RAC Administration A - 31


Practice 13-3: Configuring RMAN and Performing Parallel
Backups (continued)

no rows selected
SQL> /

INST_ID SID SERIAL# CONTEXT SOFAR TOTALWORK %_COMPLETE


------- ---- -------- ------- -------- ---------- ----------
1 31 1913 1 13308 104960 12.68
2 55 284 1 23934 101760 23.52

SQL> /

INST_ID SID SERIAL# CONTEXT SOFAR TOTALWORK %_COMPLETE


------- --- -------- -------- -------- ---------- ----------
1 31 1913 1 44798 104960 42.68
2 55 284 1 49278 101760 48.43

SQL> exit

9) Run the /home/oracle/labs/less_13/cleanup_13.sh script.


$ /home/oracle/labs/less_13/cleanup_13.sh
The Oracle base for
ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1 is
/u01/app/oracle
m y
10) Exit all windows when finished.
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 32
Practices for Lesson 14
This practice is designed to show you how to discover performance problems in your
RAC environment. In this practice, you identify performance issues by using Enterprise
Manager, and fix issues in three different steps. At each step, you will generate the same
workload to make sure that you are making progress in your resolution.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 33
Practice 14-1: ADDM and RAC Part I
The goal of this lab is to show you how to manually discover performance issues by
using the Enterprise Manager performance pages as well as ADDM. This first part
generates a workload that uses a bad RAC application design.
Note that all the necessary scripts for this lab are located in the
/home/oracle/labs/seq directory on your first cluster node.
1) Before you start this lab, make sure that you have the necessary TNS entries in the
tnsnames.ora file located in your ORACLE_HOME for the orcl database. You
can execute the following script to create those entries: add_tnsinstances.sh.
$ cd /home/oracle/labs/seq
$ ./add_tnsinstances.sh
# tnsnames.ora Network Configuration File:
/u01/app/oracle/product/11.2.0/dbhome_1/network/admin/tnsnames
.ora
# Generated by Oracle configuration tools.

ORCL =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = .example.com)(PORT =
1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
m y
)
(SERVICE_NAME = orcl)

d e
)

c a
orcl1 =

e A
(DESCRIPTION =
(ADDRESS_LIST =
c l
1521))
r a
(ADDRESS = (PROTOCOL = TCP)(HOST = <host01>)(PORT =

)
(CONNECT_DATA = O ly
l & On
(SERVICE_NAME = orcl)
(INSTANCE_NAME = orcl1)

)
)

n a e
orcl2 =
t e r U s
I n
(DESCRIPTION =
(ADDRESS_LIST =

l e
1521))
c )
(ADDRESS = (PROTOCOL = TCP)(HOST = <host02>)(PORT =

r a (CONNECT_DATA =
(SERVICE_NAME = orcl)
O (INSTANCE_NAME = orcl2)

Oracle Database 11g: RAC Administration A - 34


Practice 14-1: ADDM and RAC Part I (continued)

)
)

orcl3 =
(DESCRIPTION =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = <host03>)(PORT =
1521))
)
(CONNECT_DATA =
(SERVICE_NAME = orcl)
(INSTANCE_NAME = orcl3)
)
)

2) Execute the setupseq1.sh script to set up the necessary configuration for this lab.
$ ./setupseq1.sh

PL/SQL procedure successfully completed.

m y
PL/SQL procedure successfully completed.

d e
drop user jfv cascade
*
c a
ERROR at line 1:
ORA-01918: user 'JFV' does not exist
e A
c l
*
r a
drop tablespace seq including contents and datafiles

ERROR at line 1:
O ly
ORA-00959: tablespace 'SEQ' does not exist

l & On
Tablespace created.
n a e
t e r U s
User created.
I n
c l e
Grant succeeded.

Ora drop sequence s


*
ERROR at line 1:

Oracle Database 11g: RAC Administration A - 35


Practice 14-1: ADDM and RAC Part I (continued)

ORA-02289: sequence does not exist

drop table s purge


*
ERROR at line 1:
ORA-00942: table or view does not exist

drop table t purge


*
ERROR at line 1:
ORA-00942: table or view does not exist

Table created.

Table created.

Index created.

m y
1 row created.
d e
Commit complete. c a
e A
l
PL/SQL procedure successfully completed.
c
$
r a
O ly
l & On
3) Using Database Control, and connected as the SYS user, navigate to the Performance
page of your Cluster Database.

n a e
a) Click the Performance tab from the Cluster Database Home page.

e r s
b) On the Cluster Database Performance page, make sure that Real Time: 15
t U
Seconds Refresh is selected from the View Data drop-down list.

I n
4) Use PL/SQL to create a new AWR snapshot.

l e
$ ./create_snapshot.sh

c
Ora PL/SQL procedure successfully completed.

Oracle Database 11g: RAC Administration A - 36


Practice 14-1: ADDM and RAC Part I (continued)

5) Execute the startseq1.sh script to generate a workload on all instances of your


cluster. Do not wait; proceed with the next step.
$ ./startseq1.sh
$ old 7: insert into t values(v,'&1');
new 7: insert into t values(v,'orcl2');
old 7: insert into t values(v,'&1');
new 7: insert into t values(v,'orcl1');
old 7: insert into t values(v,'&1');
new 7: insert into t values(v,'orcl3');

Do not wait after this point and go to the next step.

PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

$
6) Using Database Control, determine the list of blocking locks in your database.
m y
d e
a) Still on the Performance page, click the Database Locks link in the Additional
Monitoring Links section of the page.

c a
b) On the Database Locks page, make sure that Blocking Locks is selected from the
View drop-down list.

e A
c l
c) If you do not see any locks, refresh the page by clicking Refresh. Perform this
until you see locks. When you see a session lock, you should also see that the

r a
other session is waiting for that same lock. By clicking Refresh several times, you
must see that all sessions are alternatively waiting for the other to release the
O ly
exclusive lock held on table S.

l & On
7) While the scripts are still executing, look at the Average Active Sessions graphic.
Then, drill down to the Cluster wait class for the first node. What are your
conclusions?
n a e
t e r U s
a) By using the drill-down method of Enterprise Manager, you can quickly identify
the top waiting SQL statements and the top waiting sessions on both instances.

I n
Here it appears that an UPDATE statement on table S is causing most of the waits
for the Cluster wait class.

l e
b) Click Cluster Database in the locator link at the top of the page to return to the
c Cluster Database Performance page.

r a
O
Oracle Database 11g: RAC Administration A - 37
Practice 14-1: ADDM and RAC Part I (continued)

c) From there you can now see the Average Active Sessions graph. Make sure that
the View Data field is set to Real Time:15 Seconds Refresh. After a few seconds,
the graphic must clearly show that the Cluster and Application wait classes are
causing most waits. Using the Throughput tabbed page graph underneath the
Average Active Sessions graph, you should also notice that the transaction rate is
about 250 per second.
d) In the Average Active Sessions graph, click the Cluster link on the right. This
takes you to the Active Sessions By Instance: Cluster page.
e) On the Active Sessions By Instance: Cluster page, you will see that the number of
active sessions is almost the same on all nodes. Click the first instances link
(instance number 1). This takes you to the Active Sessions Waiting: Cluster page
for the corresponding instance.
f) On the Active Sessions Waiting: Cluster page, you can see the most important
wait events causing most of the waits in the Cluster wait class on the first
instance. In the Top SQL: Cluster section, click the SQL identifier that uses most
of the resources. This takes you to the SQL Details page for the corresponding
statement. You will see that the script running on the first instance is executing a
SELECT/UPDATE statement on table S that causes most of the Cluster waits.
8) Using Database Control, look at the Cluster Cache Coherency page. What are your
conclusions?
m y
a) On the Cluster Database Home page, click the Performance tab.
b) On the Performance page, click the Cluster Cache Coherency link in the d e
Additional Monitoring Links section.
c a
e A
c) The Cluster Cache Coherency page clearly shows that there are lots of blocks
transferred per second on the system. This represents more than 17% of the total

c l
logical reads. This is reflected in both the Global Cache Block Transfer Rate and
the Global Cache Block Transfers and Physical Reads (vs. Logical Reads)
graphics.
r a
O ly
d) On the Cluster Cache Coherency page, you can also click Interconnects in the

interconnect.
l & On
Additional Links section of the page to get more information about your private

a e
9) While the scripts are still executing, look at the Average Active Sessions graph on the
n
t e r
Database Performance page. Then drill down to the Application wait class for the first
s
instance. What are your conclusions?
U
I n
a) By using the drill-down method of Enterprise Manager, you can quickly identify
the top waiting SQL statements and the top waiting sessions on both instances.

c l e
Here it appears that a LOCK statement on table S is causing most of the waits for
the Application wait class.

Ora b) Go back to the Cluster Database Home page by clicking the Database tab located
on the top right-end corner. On the Cluster Database Home page, click the
Performance tab.

Oracle Database 11g: RAC Administration A - 38


Practice 14-1: ADDM and RAC Part I (continued)

c) On the Performance page, make sure that the View Data field is set to Real Time:
15 Seconds Refresh. After a few seconds, the graphic should clearly show that the
Cluster and Application wait classes are causing most waits. You will also notice
that the transaction rate is about 100 per second.
d) In the Average Active Sessions graph, click the Application link on the right. This
takes you to the Active Sessions By Instance: Application page.
e) On the Active Sessions By Instance: Application page, you must see that the
number of active sessions is almost the same on all nodes. Click the link for the
first instance (number 1) on the Summary Chart graph. This takes you to the
Active Sessions Waiting: Application page of the first instance.
f) On the Active Sessions Waiting: Application page, you can see the most
important wait events causing most of the waits in the Application wait class on
the first instance. In the Top SQL: Application section, click the SQL identifier
that uses most of the resources. This takes you to the SQL Details page for the
corresponding statement. You must see that the script running on the first instance
is executing a LOCK statement on table S that causes most of the Application
waits.
g) After a while, you can see that all scripts are executed by looking at the Average
Active Sessions graph as well as the Database Throughput graphics again. You
should see the number of transactions per second going down.
m y
10) After the workload finishes, use PL/SQL to create a new AWR snapshot.

d e
$ ./create_snapshot.sh

c a
PL/SQL procedure successfully completed.

e A
$

c l
11) Using Database Control, review the latest ADDM run. What are your conclusions?

r a
a) On the Cluster Database Home page, click the Advisor Central link in the Related
Links section.
O ly
l & On
b) On the Advisor Central page, make sure that the Advisory Type field is set to All
Types, and that the Advisor Runs field is set to Last Run. Click Go.

a e
c) In the Results table, select the latest ADDM run corresponding to Instance All.
n
t e r
Monitor (ADDM) page.
U s
Then click View Result. This takes you to the Automatic Database Diagnostic

I n
d) On the Automatic Database Diagnostic Monitor (ADDM) page, the ADDM
Performance Analysis table shows you the consolidation of ADDM reports from

c l e
all instances running in your cluster. This is your first entry point before drilling
down to specific instances. From there, investigate the Top SQL Statements,

r a Table Locks, and Global Cache Messaging findings.

O
Oracle Database 11g: RAC Administration A - 39
Practice 14-1: ADDM and RAC Part I (continued)

e) Click the Top SQL Statements finding, which affects all instances, revealing
LOCK TABLE S and UPDATE S commands as a possible problem to
investigate. Click the Back button to return to the ADDM report.
f) Click the Table Locks finding, which affects all instances, revealing that you
should investigate your application logic regarding the JFV.S object.
g) Click the Global Cache Messaging finding revealing again the UPDATE S
command as responsible for approximately 30% of Cluster waits during the
analysis period.
h) Back to the Automatic Database Diagnostic Monitor (ADDM) page, you now
have the possibility to drill down to each instance using the links located in the
Affected Instances table. Click the link corresponding to the most affected
instance (although all should be equally affected).
i) On the corresponding ADDM Database Diagnostic Monitor (ADDM) instance
page, you should retrieve similar top findings you previously saw at the cluster
level.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 40
Practice 14-2: ADDM and RAC Part II
The goal of this lab is to show you how to manually discover performance issues by
using the Enterprise Manager performance pages as well as ADDM. In this second part
of the practice, you are going to correct the previously found issue by creating a sequence
number instead of by using a table.
Note that all the necessary scripts for this lab are located in the
/home/oracle/labs/seq directory on your first cluster node.
1) Execute the setupseq2.sh script to create the necessary objects used for the rest
of this practice.
$ ./setupseq2.sh

PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

User dropped.

Tablespace dropped.

m y
Tablespace created.

d e
User created.
c a
e A
Grant succeeded.

c l
drop table s purge
*
r a
ERROR at line 1:
O ly
ORA-00942: table or view does not exist

l & On
drop sequence s
*
n a e
ERROR at line 1:

t e r U s
ORA-02289: sequence does not exist

I n
c e
drop table t purge
l *
ERROR at line 1:

r aORA-00942: table or view does not exist

O
Oracle Database 11g: RAC Administration A - 41
Practice 14-2: ADDM and RAC Part II (continued)

Table created.

Index created.

Sequence created.

PL/SQL procedure successfully completed.

2) Using Database Control, and connected as the SYS user, navigate to the Performance
page of your Cluster Database.
a) Click the Performance tab from the Cluster Database Home page.
b) On the Cluster Database Performance page, make sure Real Time: 15 Seconds
Refresh is selected from the View Data drop-down list.
3) Use PL/SQL to create a new AWR snapshot.
$ ./create_snapshot.sh

m y
PL/SQL procedure successfully completed.

d e
$

c a
A
4) Execute the startseq2.sh script to generate a workload on all instances of your
e
$ ./startseq2.sh
c l
cluster. Do not wait; proceed with the next step.

$ old
new 3:
3: insert into
insert into t r a t values(s.nextval,'&1');
values(s.nextval,'orcl1');
old 3: insert into t
O ly values(s.nextval,'&1');
new
old
3:
3:
insert into t

l
insert into t& On values(s.nextval,'orcl3');
values(s.nextval,'&1');
new 3:
a e
insert into t

n
values(s.nextval,'orcl2');

t e r U s
Do not wait after this point and go to the next step.

PL/SQL procedure successfully completed.

I n
l e
PL/SQL procedure successfully completed.

c
r a PL/SQL procedure successfully completed.

O $

Oracle Database 11g: RAC Administration A - 42


Practice 14-2: ADDM and RAC Part II (continued)

5) While the scripts are still executing, look at the Average Active Sessions graphic.
Then drill down to the Cluster wait class for the first node. What are your
conclusions?
a) By using the drill-down method of Enterprise Manager, you can quickly identify
the top waiting SQL statements and the top waiting sessions on both instances.
Here it appears that an INSERT statement on table T is causing most of the waits
for the Cluster wait class.
b) Click Cluster Database in the locator link at the top of the page to return to the
Cluster Database Performance page.
c) From there you can now see the Average Active Sessions graph. Make sure that
the View Data field is set to Real Time:15 Seconds Refresh. After a few seconds,
the graphic will clearly show that the Cluster and Application wait classes are
causing most waits. Using the Throughput tabbed page graph underneath the
Average Active Sessions graph, you should also notice that the transaction rate is
about 320 per second (a better rate than in the previous practice).
d) In the Average Active Sessions graph, click the Cluster link on the right. This
takes you to the Active Sessions By Instance: Cluster page.
e) On the Active Sessions By Instance: Cluster page, you must see that the number
of active sessions is almost the same on all nodes. Click the first instances link
(instance number 1). This takes you to the Active Sessions Waiting: Cluster page
m y
for the corresponding instance.
d e
c a
f) On the Active Sessions Waiting: Cluster page, you can see the most important
wait events causing most of the waits in the Cluster wait class on the first

A
instance. In the Top SQL: Cluster section, click the SQL identifier that uses most
of the resources. This takes you to the SQL Details page for the corresponding
e
c l
statement. You will see that the script running on the first instance is executing an
INSERT statement on table T that causes most of the Cluster waits.

r a
g) After a while you can see that all are executed by looking at the Average Active

O ly
Sessions graphic again. The Database Throughput graphic tells you that this time,
the number of transactions per second was a bit higher than in the previous lab for
& On
the same workload. Using the sequence number was a bit better in this case.
l
a e
6) After the workload finishes, use PL/SQL to create a new AWR snapshot.

n
t e r
$ ./create_snapshot.sh

U s
PL/SQL procedure successfully completed.

$ I n
c l e
7) Using Database Control, review the latest ADDM run. What are your conclusions?

Ora a) On the Cluster Database Home page, click the Advisor Central link.
b) On the Advisor Central page, make sure that the Advisory Type field is set to All
Types, and that the Advisor Runs field is set to Last Run. Click Go.

Oracle Database 11g: RAC Administration A - 43


Practice 14-2: ADDM and RAC Part II (continued)

c) In the Results table, select the latest ADDM run corresponding to Instance All.
Then click View Result. This takes you to the Automatic Database Diagnostic
Monitor (ADDM) page.
d) On the Automatic Database Diagnostic Monitor (ADDM) page, the ADDM
Performance Analysis table shows you the consolidation of ADDM reports from
all instances running in your cluster. This is your first entry point before drilling
down to specific instances. From there, investigate the Top SQL Statements,
Sequence Usage, and Unusual Concurrency Wait Event findings.
e) The Top SQL Statements should reveal an INSERT INTO T command using
sequence S as a possible problem to investigate.
f) The Sequence Usage finding reveals that you should use larger cache size for
your hot sequences.
g) The Unusual Concurrency Wait Event finding asks you to investigate the cause
for high row cache lock waits. Refer to the Oracle Database Reference for the
description of this wait event.
h) Back to the Automatic Database Diagnostic Monitor (ADDM) page, you now
have the possibility to drill down to each instance using the links located in the
Affected Instances table. Click the link corresponding to the most affected
instance (although all should be equally affected).
8) On the corresponding ADDM Database Diagnostic Monitor (ADDM) instance page, m y
e
you should retrieve top findings similar to those you previously saw at the cluster
d
level.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 44
Practice 14-3: ADDM and RAC Part III
The goal of this lab is to show you how to manually discover performance issues by
using the Enterprise Manager performance pages as well as ADDM. This last part
generates the same workload as in the previous lab but uses more cache entries for
sequence number S.
Note that all the necessary scripts for this lab are located in the
/home/oracle/labs/seq directory on your first cluster node.
1) Execute the setupseq3.sh script to create the necessary objects used for the rest
of this practice.
$ ./setupseq3.sh

PL/SQL procedure successfully completed.

PL/SQL procedure successfully completed.

User dropped.

Tablespace dropped.

m y
Tablespace created.

d e
User created.
c a
e A
Grant succeeded.

c l
drop table s purge
*
r a
ERROR at line 1:
O ly
ORA-00942: table or view does not exist

l & On
drop sequence s
*
n a e
ERROR at line 1:

t e r U s
ORA-02289: sequence does not exist

I n
c e
drop table t purge
l *
ERROR at line 1:

r aORA-00942: table or view does not exist

O
Oracle Database 11g: RAC Administration A - 45
Practice 14-3: ADDM and RAC Part III (continued)

Table created.

Index created.

Sequence created.

PL/SQL procedure successfully completed.

2) Using Database Control, and connected as the SYS user, navigate to the Performance
page of your Cluster Database.
a) Click the Performance tab from the Cluster Database Home page.
b) On the Cluster Database Performance page, make sure that Real Time: 15
Seconds Refresh is selected from the View Data drop-down list.
3) Use PL/SQL to create a new AWR snapshot.
$ ./create_snapshot.sh

PL/SQL procedure successfully completed.


m y
$
d e
c a
4) Execute the startseq2.sh script to generate the same workload on both instances

$ ./startseq2.sh
e A
of your cluster as for the previous lab. Do not wait, and proceed with the next step.

$ old
new 3:
3: insert into
insert into t
c l
t values(s.nextval,'&1');
values(s.nextval,'orcl3');
old
new
3:
3:
insert into t
insert into t r a values(s.nextval,'&1');
values(s.nextval,'orcl2');
old 3: insert into t
O ly values(s.nextval,'&1');
new 3: insert into t

l & On values(s.nextval,'orcl1');

a e
Do not wait after this point and go to the next step.

n
t e r s
PL/SQL procedure successfully completed.
U
I n
PL/SQL procedure successfully completed.

c l e
Ora $
PL/SQL procedure successfully completed.

Oracle Database 11g: RAC Administration A - 46


Practice 14-3: ADDM and RAC Part III (continued)

5) Until the scripts are executed, look at the Average Active Sessions graphic. What are
your conclusions?
a) This time, looking at the Average Active Sessions graphic, it is clear that there are
no significant waits. The sequence has a big enough cache value to avoid the most
significant waits.
b) Click Cluster Database in the locator link at the top of the page to return to the
Cluster Database Performance page.
c) On the Performance page, make sure that the View Data field is set to Real
Time:15 Seconds Refresh. After all the scripts have finished their execution, the
Average Active Sessions graph will clearly show that there are no significant
waits on your cluster. You must also notice that the transaction rate is now around
2400 per second.
6) After the workload finishes, use PL/SQL to create a new AWR snapshot.
$ ./create_snapshot.sh

PL/SQL procedure successfully completed.

7) Using Database Control, review the latest ADDM run. What are your conclusions?
m y
a) On the Cluster Database Home page, click the Advisor Central link.

d e
b) On the Advisor Central page, make sure that the Advisory Type field is set to All
Types, and that the Advisor Runs field is set to Last Run. Click Go.
c a
A
c) In the Results table, select the latest ADDM run corresponding to Instance All.
Then click View Result. This takes you to the Automatic Database Diagnostic
e
Monitor (ADDM) page.
c l
r a
d) On the Automatic Database Diagnostic Monitor (ADDM) page, the ADDM
Performance Analysis table shows you the consolidation of ADDM reports from

O ly
all instances running in your cluster. This is your first entry point before drilling
down to specific instances. From there, investigate the Buffer Busy Hot Block,

l & On
Buffer Busy Hot Objects, and Global Cache Busy findings. You should no
longer see the Sequence Usage, nor specific instances impacted.

n a e
e) The Buffer Busy Hot Block finding should not reveal any particular object.

e r s
f) The Buffer Busy Hot Objects finding should not reveal any particular object.
t U
n
g) The Global Cache Busy finding should not reveal anything special.
I
c l e
r a
O
Oracle Database 11g: RAC Administration A - 47
Practices for Lesson 15
In these practices, you will create, manage, and monitor services.

m y
d e
ca
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 48
Practice 15-1: Working with Services
In this practice, you will use Enterprise Manager to create one service called PROD1.
You then observe what happens to your service when you terminate one of the instances
on which it is running.

1) Use Enterprise Manager to create the PROD1 service. Make sure that you define your
first and second instances (ORCL1 and ORCL2) as preferred, and the third instance
(ORCL3) as available.
a) Enter your EM address in a browser. It will look something like this:
https://your_host_name:1158/em
b) Log in using SYS credentials as SYSDBA.
c) Click the Availability folder tab.
d) Click the Cluster Managed Database Services link under the Services section.
e) On the Cluster Managed Database Services: Cluster and Database Login page,
provide the login credentials for the operating system user (oracle/0racle)
and the SYSDBA credentials for the database (sys/oracle_4U) and click
Continue.
f) Click the Create Service button on the Cluster Managed Database Services page.
g) On the Create Service page, enter PROD1 for the service name. Verify that the
m y
d e
Start service after creation check box is selected, and select the Update local
naming check box. Under the High Availability Configuration section, set the
service policy for orcl1 and orcl2 to Preferred and ORCL3 to Available.
c a
Leave the remaining fields with their default values and click the OK button.
A
h) After the service has been created, you will be returned to the Cluster Managed
e
c l
Database Services page. Check the Running Instances column for PROD1, it
should indicate the service running on orcl1 and orcl2. Select PROD1 from

r a
the Services list and click the Test Connection button. It should test successfully.

O ly
Click the Show All TNS Strings button and inspect the new entry to the
tnsnames.ora file. It should look like this:
PROD1 =
l & On
(DESCRIPTION = (ADDRESS = (PROTOCOL = TCP)
(HOST =
(PORT =
n a e
cluster01-scan.cluster01.example.com)
1521))(LOAD_BALANCE = YES)(CONNECT_DATA =
(SERVER

t e r U s
= DEDICATED)(SERVICE_NAME = PROD1)))
i) Click the Return button.

I n
2) Use the srvctl command to check the status of the new service.

l e
$ srvctl status service -d ORCL -s PROD1

c
Service PROD1 is running on instance(s) orcl1,orcl2

r a$

O 3) Use the crsctl command to view server pool relationships with the new service.

Oracle Database 11g: RAC Administration A - 49


Practice 15-1: Working with Services (continued)

$ /u01/app/11.2.0/grid/bin/crsctl status serverpool -p

NAME=Free
IMPORTANCE=0
MIN_SIZE=0
MAX_SIZE=-1
SERVER_NAMES=
PARENT_POOLS=
EXCLUSIVE_POOLS=
ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r-x

NAME=Generic
IMPORTANCE=0
MIN_SIZE=0
MAX_SIZE=-1
SERVER_NAMES=host01 host02 host03
PARENT_POOLS=
EXCLUSIVE_POOLS=
ACL=owner:grid:r-x,pgrp:oinstall:r-x,other::r-x

NAME=myApache_sp
IMPORTANCE=0
MIN_SIZE=0
MAX_SIZE=-1
SERVER_NAMES= m y
PARENT_POOLS=Generic
EXCLUSIVE_POOLS=
d e
ACL=owner:grid:rwx,pgrp:oinstall:rwx,other::r--
c a
NAME=ora.orcl
IMPORTANCE=1
e A
MIN_SIZE=0
MAX_SIZE=-1
c l
r a
SERVER_NAMES=host01 host02 host03
PARENT_POOLS=Generic
EXCLUSIVE_POOLS=
O ly
& On
ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r--

l
NAME=ora.orcl_PROD1
IMPORTANCE=0
n a e
MIN_SIZE=0
MAX_SIZE=-1
t e r U s
SERVER_NAMES=host01 host02 host03

I n
PARENT_POOLS=ora.orcl
EXCLUSIVE_POOLS=

l e
ACL=owner:oracle:rwx,pgrp:oinstall:rwx,other::r--

c
Ora $

Oracle Database 11g: RAC Administration A - 50


Practice 15-1: Working with Services (continued)

4) Connect to the service and look at the current value of the SERVICE_NAMES
initialization parameter, and verify that it is set correctly. Query V$INSTANCE and
determine what instance you are connected to.
$ /u01/app/oracle/product/11.2.0/dbhome_1/bin/sqlplus
sys/oracle_4U@PROD1 as sysdba

SQL*Plus: Release 11.2.0.1.0 Production on Fri Sep 4 11:10:29


2009

Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to:
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 -
Production
With the Partitioning, Real Application Clusters, Automatic
Storage Management, OLAP,
Data Mining and Real Application Testing options

SQL> show parameter service

NAME TYPE VALUE


---------------------- ----------- --------------
service_names string PROD1
m y
SQL> select instance_name from v$instance;
d e
INSTANCE_NAME c a
----------------
orcl1
e A
SQL> exit
c l
r a
O ly
5) From a terminal session as the oracle user, crash the instance on the first node.

l & On
Find and kill the ora_pmon_orcl process. Use the pkill -9 f pmon_orcl
command to crash the database instance. The orcl1 instance will crash and the

a e
clusterware services will restart it very quickly

n
t e r
$ pkill -9 -f pmon_orcl1

U s
I n
6) Use the srvctl command to check the status of the PROD1 service. (It may take a
few moments to show up on the orcl3)

c l e
$ srvctl status service -d ORCL -s PROD1

r aService PROD1 is running on instance(s) orcl2,orcl3

O $

Oracle Database 11g: RAC Administration A - 51


Practice 15-1: Working with Services (continued)

7) Return to Enterprise Manager. Click the Availability folder tab. In the instance list
under the Instances section, you should be able to verify that the first instance is
indeed down.
8) Click the Cluster Managed Database Services link. On the Cluster Managed Database
Service page, you can see orcl2 and orcl3 in the running instances column for
PROD1. Select Manage from the Actions drop-down list and click Go.
9) Under the instances section, find the host that orcl3 is running on and select the
option in the Select column for that host. Click the Relocate button.
10) On the Relocate Service from Instance: orcl03 page, select the host name that
orcl1 is running on and click OK.
11) You should see a message indicating that the service was relocated successfully.
Under the Instances section of the page, you should see the service running on
orcl1 and orcl2 and stopped on orcl3.
Note: The instance status shown in EM may still show that the instance is down.
Click the browser Refresh button to see the actual status of the instance.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 52
Practice 15-2: Monitoring Services
In this practice, you will use Database Control to determine the amount of resources used
by sessions executing under a particular service.

1) As the oracle user, open a terminal session to your first node. Execute the
/home/oracle/labs/less_15/createuser.sh script. This script creates a new
user called FOO identified by the password foo. The default tablespace of this user
is USERS, and its temporary tablespace is TEMP. This new user has the CONNECT,
RESOURCE, and DBA roles.
$ cat /home/oracle/labs/less_15/createuser.sh

export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export ORACLE_SID=orcl1
/u01/app/oracle/product/11.2.0/dbhome_1/bin/sqlplus -s /NOLOG
<<EOF

connect / as sysdba
drop user FOO cascade;
create user FOO identified by foo default tablespace users
temporary tablespace temp;
grant connect, resource, dba to FOO;

EOF
$ /home/oracle/labs/less_15/createuser.sh
m y
drop user FOO cascade
*
d e
ERROR at line 1:
ORA-01918: user 'FOO' does not exist
c a
e A
User created.
c l
r a
Grant succeeded.
O ly
$

l & On
a e
2) Using SQL*Plus, connect to PROD1 as FOO. When connected, determine the
n
query:
t r U s
instance on which your session is currently running. Then execute the following

e
select count(*) from dba_objects,dba_objects,dba_objects
I n
Do not wait; instead, proceed with the next step.

l e
$ sqlplus foo/foo@PROD1
SQL> select instance_name from v$instance;
c
Ora INSTANCE_NAME
----------------
orcl1

Oracle Database 11g: RAC Administration A - 53


Practice 15-2: Monitoring Services (continued)

SQL> select count(*) from dba_objects,dba_objects,dba_objects;

3) After a few moments, go to the Database Control Top Consumers page from the
Cluster Database page. Connect as user SYS. Then check that PROD1 is using more
and more resources.
a. From the Cluster Database Home page, click the Performance tab.
b. On the Performance page, click the Top Consumers link in the Additional
Monitoring Links section.
c. This takes you to the Top Consumers page with the Overview tab selected.
d. On the Overview page, you can see the Top Services pie chart.
e. Make sure that the View Data drop-down list is set to Real Time: 15 Second
Refresh. Wait for the page to be refreshed a couple of times. Little by little,
PROD1 is consuming almost all the resources (up to 100%).
f. To have more details, click Top Services tab on the Top Consumers page.
g. Make sure that the View Data drop-down list is set to Real Time: 15 Second
Refresh, and View drop-down list is set to Active Services. You can click the +
icon on the left of the PROD1 link to expand the service. This shows you the list
of instances currently running the service. You can also click the PROD1 link
itself to look at the detailed Statistics of the corresponding service.
m y
gv$service_stats from a SQL*Plus session connected as SYSDBA.
d e
4) In another terminal window as the oracle user, check statistics on your service with

$ /u01/app/oracle/product/11.2.0/dbhome_1/bin/sqlplus
sys/oracle_4U@orcl as sysdba
c a
e
SQL*Plus: Release 11.2.0.1.0 Production on Fri Sep 4 16:16:48 A
2009
c l
r a
Copyright (c) 1982, 2009, Oracle. All rights reserved.

Connected to: O ly
l & On
Oracle Database 11g Enterprise Edition Release 11.2.0.1.0 -
Production
a e
With the Partitioning, Real Application Clusters, Automatic
n
t e r
Storage Management, OLAP,
s
Data Mining and Real Application Testing options
U
I n
SQL> select stat_name, sum(value) from gv$service_stats where
service_name = 'PROD1' group by stat_name;

c l e
STAT_NAME SUM(VALUE)

r a--------------------------------------------------- ----------
user calls 43

O DB CPU
redo size
1368469523
1564

Oracle Database 11g: RAC Administration A - 54


Practice 15-2: Monitoring Services (continued)

db block changes 8
DB time 1473281835
user rollbacks 0
gc cr blocks received 2
gc cr block receive time 0
gc current blocks received 2
opened cursors cumulative 99
workarea executions - multipass 0

STAT_NAME SUM(VALUE)
-------------------------------------------------- -----------
session cursor cache hits 45
user I/O wait time 3540
parse count (total) 71
physical reads 4
gc current block receive time 0
workarea executions - optimal 22
concurrency wait time 17361
parse time elapsed 110704
physical writes 0
workarea executions - onepass 0
execute count 96

STAT_NAME SUM(VALUE)
--------------------------------------------------- ---------- m y
session logical reads
cluster wait time
3825
d
2161
e
application wait time
logons cumulative c a
20622
2
sql execute elapsed time
user commits
e A
1473090354
0

28 rows selected.
c l
SQL> r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 55
Practice 15-3: Services and Alert Thresholds
In this practice, you will set thresholds for service PROD1, and use Database Control to
monitor the response time metric for this service. In this practice, you will set the Elapsed
Time in seconds warning threshold at 4 and the critical threshold at 1. Preferred instances
should be orcl1 and orcl2, and orcl3 should be available.

1) Set alert thresholds for your service PROD1 using Database Control.
a) Log in as sys with SYSDBA privileges.
b) On the Database Home page, click the Availability folder tab. Then click the
Cluster Managed Database Services link.
c) Select PROD1 from the Services list, select Edit Properties from the Actions
drop-down list and click Go.
d) Under the High Availability Configuration section, set the Service Policy for
orcl1 to Preferred and Available for orcl2 and orcl3. Then click OK.
e) Return to the Cluster Database home page, click the link corresponding to your
first instance in the Instances table. This is the instance currently running PROD1.
f) On the Database Instance page, click Metric and Policy settings in the Related
Links section at the bottom of the page.
g) On the Metric and Policy Settings page, select All metrics from the View
m y
drop-down list.
h) Scroll down the Metric and Policy Settings page until you find the Service
d e
Response Time (per user call) (microseconds) metric.
c a
column).
e A
i) On the same line, click the corresponding multi-pens icon in the last column (Edit

l
j) On the Edit Advanced Settings: Service Response Time (per user call)
c
r a
(microseconds) page, click Add.
k) The Monitored Objects table should now show two entries.
O ly
l) Enter PROD1 in the Service Name field, 40000000 in the Warning Threshold

l & On
field, and 100000000 in the Critical Threshold field. Make sure that the
corresponding line is selected, and click Continue.

n a e
m) On the Metric and Policy Settings page, you should see an Information warning

t e
the new settings.
r U s
explaining that your settings have been modified but not saved. Click OK to save

I n
n) On the Confirmation page, you can see an Update succeeded message. Click OK.

l e
o) This takes you back to the Database Instance page.

c
2) Use Database Control to view the Service Response Time Metric Value graphic for

r aPROD1.

O
Oracle Database 11g: RAC Administration A - 56
Practice 15-3: Services and Alert Thresholds (continued)

a) From the Database Instance page, click All Metrics in the Related Links section at
the bottom of the page.
b) On the All Metrics page, expand the Database Services link. On the All Metrics
page, click the Service Response Time (per user call) (microseconds) link.
c) On the Service Response Time (per user call) (microseconds) page, click the
PROD1 link in the Service Name column.
d) On the Service Response Time (per user call) (microseconds): Service Name
PROD1: Last 24 hours page, select Real Time: 30 Second Refresh from the View
Data drop-down list.
e) You should now see the Service Response Time (per user call) (microseconds):
Service Name PROD1 page with your warning and critical thresholds set
correctly.
3) Execute the serv_wkload.sh script to generate workload on your database.
Looking at the Service Response time graphic for PROD1, what do you observe?
$ cd /home/oracle/labs/less_15
$ ./serv_wkload.sh

a) Still looking at the Service Response Time (per user call) (microseconds): Service
Name PROD1 page on your first session, you should see the graphic crossing the
warning threshold after few minutes. This will trigger a warning alert soon after
m y
the warning threshold is crossed.
d e
Cluster Database Home page.
c a
b) You can see this alert propagated to your Database Instance Home page, and

e
locator link on the Service Response Time page. A
c) To go back to your Database Instance Home page, click the Database Instance

c l
d) You should see the warning raised in the Alerts section of the Database Instance
page.
r a
O ly
e) On the Database Instance page, click the Cluster Database locator link of the
page.

l & On
f) You should see the warning alert in the Problem Services line in the High

n a e
Availability section of the page. Clicking this link takes you to the Cluster Home
page. From there you can click the PROD1 link to directly go to the Cluster

t e r U s
Managed Database Services: PROD1 page after you clicked Continue on the
Login page. The PROD1 page shows you the alert with its details.

I n
g) Soon after the script finishes its execution, you should not see the corresponding
alert on your Cluster Database Home page anymore. You can go to the Alert

c l e
History page on the first instance to look at the alert history for your services. You
can go to the Database Instance Home page using the locator links at the top of

r a any pages. From the Database Instance Home page, scroll down to the bottom of
the page, and click Alert History in the Related Links section.
O
Oracle Database 11g: RAC Administration A - 57
Practice 15-3: Services and Alert Thresholds (continued)

4) Use Database Control to remove the thresholds that you specified during this practice.
a) From the Cluster Database Home page, click the link corresponding to the first
instance of your cluster in the Instances section at the bottom of the page.
b) On the Database Instance page, scroll down to the bottom of the page. Click
Metric and Policy Settings in the Related Links section.
c) On the Metric and Policy Settings page, scroll down the page until you see
PROD1 in the Metric Thresholds table.
d) On the line corresponding to the PROD1 entry, remove both the Warning
Threshold and Critical Threshold values.
e) Click OK.
f) On the Confirmation page, you should see an Update succeeded message. Click
OK.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration A - 58
DHCP and DNS Configuration For GNS

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Objectives

After completing this lesson, you should be able to:


Configure or communicate DHCP configuration needs in
support of GNS.
Configure or communicate DNS configuration needs in
support of GNS.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration B - 2
GNS Overview

In a static configuration all the addresses are assigned by


administrative action.
DHCP provides dynamic configuration of host IP
addresses.
DHCP does not provide a good way to produce good names
that are useful to external clients
Because of this, it is rarely used in server complexes.
To configure GNS:
It is necessary configure the higher level DNS to forward or
delegate a subdomain to the cluster.
The cluster must run GNS on an address known to the DNS.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
GNS Overview
e A
l
In a static configuration, all the addresses are assigned by administrative action and given names
c
r a
that resolve with whatever name service is provided for the environment. This is universal
historic practice, as there has been no realistic alternative. One result is significant turn around
O ly
time to obtain the address, and to make the name resolvable. This is undesirable for dynamic

l & On
reassignment of nodes from cluster to cluster and function to function
DHCP provides for dynamic configuration of the hosts IP address but doesnt provide a good
a e
way to produce good names that are useful to external clients. As a result, it is rarely used in
n
t e r U s
server complexes because the point of a server is to provide service, and the clients need to be
able to find the server. This is solved in the current release by providing a service (GNS) for

I n
resolving names in the cluster, and defining this to the DNS service used by the clients.
To properly configure GNS to work for clients, it is necessary to configure the higher level DNS
l e
to forward or delegate a subdomain to the cluster and the cluster must run GNS on an address
c
known to the DNS, by number. This GNS address is maintained as a VIP in the cluster, run on a

r a
single node, and a GNSD process that follows that VIP around the cluster and service names in

O
the subdomain. To fully implement GNS, you need four things.
1. DHCP service for the public network in question;
2. A single assigned address in the public network for the cluster to use as the GNS VIP.
3. A forward from the higher level DNS for the cluster to the GNS VIP.
4. A running cluster with properly configured GNS
Oracle Database 11g: RAC Administration B - 3
DHCP Service

With DHCP, a host needing an address sends a broadcast


message to the network.
A DHCP server on the network responds to the request, and
assigns an address, along with other information like:
What gateway to use
What DNS servers to use
What domain should be used
In the request, a host typically sends a client identifier,
usually the MAC address of the interface in question.
The identifier sent by Clusterware is not a MAC address, but
a VIP resource name like ora.hostname.vip
Because the IP address is not bound to a fixed MAC
address, Clusterware can move it between hosts as needed.
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
DHCP SERVICE
e A
c l
With DHCP, a host needing an address sends a broadcast message to the network. A DHCP

r a
server on the network can respond to the request, and give back an address, along with other
information such as what gateway to use, what DNS server(s) to use, what domain should be
O ly
used, what NTP server should be used, and so on.

l & On
In the request, a host typically identifies itself to the server by sending the MAC address of the

n a e
interface in question. It can also send other values. Our uses for VIPs send a client identifier
that is the name of the CRS resource associated. Therefore, instead of sending MAC address

e r s
00:04:23:A5:B2:C0, well send something like ora.hostname.vip. This lets Clusterware
t U
easily move the address from one physical host to another, because it is not bound to a particular
I n
hardware MAC address. There are three ways to get DHCP service:

c l e
It can be there already, provided by the network administrator.
You can provide it yourself from a host on the network.

a
It can be provided with an appliance.
r
O
In production environments, the DHCP service would most likely already be configured. In test
environments, you either setup your own server, or use a one that comes in a box. For the sake of
example, well concentrate setting up our own DHCP server.

Oracle Database 11g: RAC Administration B - 4


DHCP Configuration Example

Assumptions about the environment:


The hosts have known addresses on the public network.
DHCP provides one address per node plus three for the SCAN.
The subnet and netmask on the interface to be serviced is
10.228.212.0/255.255.252.0.
The address range the DHCP server will serve is
10.228.212.10 through 10.228.215.254.
The gateway address is 10.228.212.1.
The name server is known to the cluster nodes.
The domain your hosts are in is us.example.com.
You have root access.
You have the RPM for the DHCP server if it is not already
installed.
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
DHCP Configuration Example
e A
l
When we use DHCP for the public network, we will need two addresses per host (host address
c
r a
and VIP), plus three for cluster wide SCAN. The GNS VIP can not be obtained from DHCP,
because it must be known in advance, so must be statically assigned.
O ly
For this example, lets make the following assumptions:

l & On
The hosts have known addresses on the public network accessible by their hostname so
they may be reached when Clusterware is not running.
a e
DHCP must provide four addresses, one per node plus three for the SCAN.
n
always be up.
t e r U s
You have a machine to use as your DHCP server. It is a single-point of failure, so it must

I n
The subnet and netmask on the interface to be serviced is
10.228.212.0/255.255.252.0.

l e
The address range the DHCP server will serve are 10.228.212.10 through
c
10.228.215.254.

r a
The gateway address is 10.228.212.1.

O We have two name servers; M.N.P.Q and W.X.Y.Z.


The domain your hosts are in for DNS search path purposes is us.example.com.
You have root access.
You have the RPM for the DHCP server if it is not already installed.

Oracle Database 11g: RAC Administration B - 5


DHCP Configuration Example

The /etc/dhcp.conf file:

subnet 10.228.212.0 netmask 255.255.252.0


{
default-lease-time 43200;
max-lease-time 86400;
option subnet-mask 255.255.252.0;
option broadcast-address 10.228.215.255;
option routers 10.228.212.1;
option domain-name-servers M.N.P.Q, W.X.Y.Z;
option domain-name "us.example.com";
Pool {
range 10.228.212.10 10.228.215.254;
}
}

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
DHCP Configuration Example (continued)
e A
l
To install and configure DHCP, follow the steps below:
c
r a
1. As the root user, install the DHCP rpm: # rpm Ivh dhcp-3.0.1-62.EL4.rpm
2. The DHCP configuration file is /etc/dhcp.conf. Given our example, the minimal
O ly
configuration for the public network will look something like this:

l & On
subnet 10.228.212.0 netmask 255.255.252.0
{
default-lease-time 43200;
n a e
max-lease-time 86400;

t e r U
option subnet-mask 255.255.252.0; s
I n
option broadcast-address 10.228.215.255;
option routers 10.228.212.1;

l e
option domain-name-servers M.N.P.Q, W.X.Y.Z;

c
option domain-name "us.example.com";

r a
Pool {
range 10.228.212.10 10.228.215.254;

O
} }
3. Start the DHCP service: # /etc/init.d/dhcp start
If you encounter any issues, check /var/log/messages for errors. You can adjust the lease
time to suit your needs within the subnet.

Oracle Database 11g: RAC Administration B - 6


DNS Concepts

Host name resolution uses the gethostbyname family of


library calls.
These calls do a configurable search of name space
providers, typically including:
Local /etc/hosts file
DNS service
Other directory services like NIS or LDAP
DNS traffic is sent as UDP packets, usually to port 53.
A query may ask for particular types of record, or all
records for a name.
Name resolution for Ipv4 uses A records for addresses, and
CNAME records for aliases that must be re-resolved.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
DNS Concepts
e A
l
Host name resolution uses the gethostbyname family of library calls. These calls go through a
c
r a
configurable search path of name space providers, typically including a local /etc/hosts file,
DNS service, and possibly other directories, such as NIS and/or LDAP. On Linux, these are
O ly
usually defined in /etc/nsswitch.conf. For example, the line: hosts: files dns nis

l & On
says to look in /etc/hosts, then consult DNS, then NIS.
When doing a lookup for a non-qualified name, the library will look for the unadorned name in
a e
all the name space providers above. If no answer is found, it will then successively apply
n
t e r U s
domains from the search entry in the file /etc/resolv.conf. This also defines the DNS
servers consulted when there is a dns entry in /etc/nsswitch.conf , for example:

I n
search us.example.com example.com

c l e
nameserver M.N.P.Q
nameserver W.X.Y.Z

r a
Usually the machines domain is the first entry in the search path, followed by others of common
use. DNS messages are sent as UDP packets, usually to the reserved port 53. A query may ask
O
for particular types of record, or all records for a name. Name resolution for Ipv4 uses A records
for addresses, and CNAME records for aliases (canonical names) that must be re-resolved. For
IPv6, addresses are four times as large, and an AAAA record is used.

Oracle Database 11g: RAC Administration B - 7


DNS Concepts (continued)
PTR records are used for reverse lookups (ask for the name given an address).
SRV records are used to define service offerings, providing both address and port, and are used
in the DNS-SD protocol we use for multicast discovery. TXT records contain arbitrary
information, and are used in DNS-SD to carry attributes of a service.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration B - 8
DNS Forwarding For GNS

To work properly, GNS needs to receive queries for all


names in a sub-domain, from anywhere in the corporation.
This is done by having the corporate DNS delegate the
sub-domain to the cluster.
DNS must know the full name of the sub-domain and the
address where GNS will listen for queries.
This clause from the /etc/named.conf file configures
zone delegation for cluster01.us.example.com.
/etc/named.conf:
zone cluster01.us.example.com{
type forward;
forward only;
forwarders {
10.228.212.2 port 53;

};
};
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
DNS Forwarding For GNS
e A
l
For GNS to function properly, it needs to receive queries for all names in a sub-domain (zone),
c
r a
from anywhere in the corporation. This is done by having the corporate DNS delegate the sub-
domain to the cluster. The two key pieces of data needed are the full name of the sub-domain
O ly
being created and the address where GNS will listen for queries.

l & On
We typically use the clustername as the sub-domain. The GNS address must be statically
assigned and on the public subnet of the cluster. It is virtual, and will be moved from host to host
a e
within the cluster to ensure the presence of the GNS service.
n
t e r U s
When using the common ISC BIND V8 name server, the network administrator sets up the zone
delegation with an entry in the server configuration that looks like this:

I n
zone cluster01.us.example.com{

l e
type forward;

c
forward only;

r aforwarders {

O };
10.228.212.2 port 53;

};

Oracle Database 11g: RAC Administration B - 9


DNS Forwarding For GNS (continued)
Here, the sub-domain is cluster01.us.example.com. When the us.example.com DNS gets any
query for anything under cluster01.us.example.com, it will send it to the server at that address
and port, and return any results received. It is also likely to cache the returned result for the time-
to-live (TTL) in the returned answer.
This does not establish any address for the name cluster01.us.example.com. Rather, it creates a
way of resolving anything underneath, such as prod.cluster01.us.example.com.
In production sites, it is necessary to work with the network administrators to change the
configuration of the DNS system, wherever it may reside.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration B - 10
DNS Configuration: Example

We will make the following assumptions about our


environment:
The cluster sub-domain is cluster01.us.example.com.
The address GNS will listen on is 10.228.212.2 port 53
The address for our new DNS server is 10.228.212.3.
The parent name servers are M.N.P.Q and W.X.Y.Z
A summary of the steps is shown below:
1. As root install the BIND (DNS) rpm: # rpm Ivh bind-9.2.4-30.rpm
2. Configure delegation in /etc/named.conf
3. Populate the cache file: $ dig . ns > /var/named/db.cache
4. Populate /var/named/db.127.0.0 to handle reverse lookups.
5.
6.
Start the name service (named): # /etc/init.d/named start
Modify /etc/resolv.conf on all nodes.
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
DNS Configuration: Example
e A
l
If you would like to set up something for testing, you can configure your own DNS service.
c
r a
Lets go through the exercise of configuring a local nameserver that all the other machines use
for all addresses. Queries for the GNS domain will be sent to the GNS service, and all others will
O ly
be sent up to the corporate DNS. This server will cache all intermediate results, so it is

l & On
configured almost identically to a caching-only DNS server.
We will require the full name of the sub-domain being created and the address where GNS will
a e
listen for queries. Our GNS address will be 10.228.212.2. Other requirements are listed below.
n
t e r U s
You must have a machine to use as your DHCP server. Remember, it is a single-point of
failure so it must always be up. The address for our new DNS server is 10.228.212.3. Do

I n
not confuse this with our GNS address.
The parent name servers are M.N.P.Q and W.X.Y.Z
l e
You have access to the root account.
c
You have the RPM for the DNS server.

r a
A summary of the steps required to install and configure the name server are listed below.
O
1.As root install the BIND (DNS) rpm:
# rpm Ivh bind-9.2.4-30.rpm
2.Configure delegation in /etc/named.conf

Oracle Database 11g: RAC Administration B - 11


DNS Configuration: Example (continued)
3. Populate the cache file:
$ dig . ns > /var/named/db.cache
4. Populate /var/named/db.127.0.0 to handle reverse lookups.
5. Start the name service (named):
# /etc/init.d/named start
6. Modify /etc/resolv.conf to correctly configure the name space search order on all
nodes.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration B - 12
DNS Configuration: Detail

1. As the root user, install the BIND DNS RPM.

# rpm Ivh bind-9.2.4-30.rpm ## Install the current RPM ##

2. Configure the behavior of the DNS server, named by editing


the /etc/named.conf file. Define the:
Working directory
The nameserver cache file
Reverse lookup configuration
Delegation for your cluster sub-domain

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
DNS Configuration: Detail
e A
l
1. The first action you will take is to install the most current BIND DNS rpm for your kernel
c
# rpm Ivh bind-9.2.4-30.rpm
r a
version. This should be done as the root user. For example:

O ly
2. Next, configure the behavior of the DNS server, named by editing the /etc/named.conf

l & On
file. You will define the working directory, the root nameserver cache file, reverse lookup
configuration, and delegation for your cluster sub-domain.
# vi /etc/named.conf
n a e
t e r U s
n
options {

e I
directory "/var/named"; # The named working directory (default value) #
forwarders { M.N.P.Q; W.X.Y.Z; }; ## Where to resolve unknown addresses.

c l
forward only; These are the same corporate nameservers used in the DHCP example ##
};

r a
O zone "." in {
type hint;
file "db.cache"; ## Defines the cache for root nameservers as /var/named/db.cache ##
};

Oracle Database 11g: RAC Administration B - 13


DNS Configuration: Detail (continued)
zone "0.0.127.in-addr.arpa" in {
type master;
file "db.127.0.0"; ## localhost reverse lookup file ##
};

zone cluster01.us.example.com{
type forward;
forward only; ## This section defines the cluster GNS IP address to which requests for address
forwarders { resolution for the cluster sub-domain cluster01.us.example.com are sent ##
10.228.212.2 port 53;
};
};

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration B - 14
DNS Configuration: Detail

3. Populate the cache file /var/named/db.cache.


# dig . ns > /var/named/db.cache

4. Populate /var/named/db.127.0.0.
$TTL 345600
@IN SOA localhost. root.localhost. (
00 ; Serial
86400 ; Refresh
7200 ; Retry
2592000 ; Expire
345600 ) ; Minimum
IN NS localhost.
1 IN PTR localhost.

3. Start the named server


# /etc/init.d/named start
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
DNS Configuration: Detail (continued)
e A
c l
3. Populate the cache file /var/named/db.cache
The zone . section of named.conf establishes root name servers. We need to populate

r a
this with data, and can do so with a simple lookup:
# dig . ns > /var/named/db.cache
O ly
The output of this command looks something like this:
[root@host01 ~]# dig ns .
l & On
a e
...

n
;. IN NS
;; ANSWER SECTION:
.

t e r288

U s IN NS a.root-servers.net.

n
. 288 IN NS b.root-servers.net.
.
.
e I 288
288
IN
IN
NS
NS
c.root-servers.net.
d.root-servers.net.
.
...
c l 288 IN NS e.root-servers.net.

r a
4. ;; ADDITIONAL SECTION:

O a.root-servers.net.
a.root-servers.net.
b.root-servers.net.
459
459
459
IN
IN
IN
A
AAAA
A
198.41.0.4
2001:503:ba3e::2:30
192.228.79.201
c.root-servers.net. 459 IN A 192.33.4.12
...

Oracle Database 11g: RAC Administration B - 15


DNS Configuration: Detail (continued)
4. Populate /var/named/db.127.0.0
The zone 0.0.127.in-addr.arpa section of named.conf handles reverse lookups
for the localhost. We need to create the /var/named/db.127.0.0 data file for the
127.0.0.0 domain, which should contain:
$TTL 345600
@ IN SOA localhost. root.localhost. (
00 ; Serial
86400 ; Refresh
7200 ; Retry
2592000 ; Expire
345600 ) ; Minimum
IN NS localhost.
1 IN PTR localhost.
5. Start the Name server named.
# /etc/init.d/named start

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration B - 16
Summary

In this lesson, you should have learned how to:


Configure or communicate DHCP configuration needs in
support of GNS.
Configure or communicate DNS configuration needs in
support of GNS.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration B - 17
m y
d e
ca
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
High Availability of Connections

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Objectives

After completing this lesson, you should be able to:


Configure client-side, connect-time load balancing
Configure client-side, connect-time failover
Configure server-side, connect-time load balancing
Use the Load Balancing Advisory (LBA)
Describe the benefits of Fast Application Notification (FAN)
Configure server-side callouts
Configure Transparent Application Failover (TAF)

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Objectives
e A
For more information see
c l
r a
http://www.oracle.com/technology/products/database/clustering/pdf/awmrac11g.pdf
Note: Much of this appendix is geared toward Oracle Database 11g Release 1 connections.
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration C - 2
Types of Workload Distribution

Connection balancing is rendered possible by configuring


multiple listeners on multiple nodes:
Client-side, connect-time load balancing
Client-side, connect-time failover
Server-side, connect-time load balancing
Run-time connection load balancing is rendered possible
by using connection pools:
Work requests automatically balanced across the pool of
connections
Native feature of the JDBC implicit connection cache and
ODP.NET connection pool

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Types of Workload Distribution
e A
l
With RAC, multiple listeners on multiple nodes can be configured to handle client connection
c
requests for the same database service.
r a
A multiple-listener configuration enables you to leverage the following failover and load-
balancing features: O ly
l & On
Client-side, connect-time load balancing
Client-side, connect-time failover
a e
Server-side, connect-time load balancing
n
t e r U s
These features can be implemented either one by one, or in combination with each other.
Moreover, if you are using connection pools, you can benefit from readily available run-time
I n
connection load balancing to distribute the client work requests across the pool of connections

c l e
established by the middle tier. This possibility is offered by the Oracle JDBC implicit
connection cache feature as well as Oracle Data Provider for .NET (ODP.NET) connection pool.

r a
O
Oracle Database 11g: RAC Administration C - 3
Client-Side Load Balancing

Oracle RAC
Database

SCAN
Local
Listeners
Listeners

Clients
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Client-Side, Connect-Time Load Balancing
e A
l
Client-side load balancing is defined in your client connection definition by setting the
c
r a
parameter LOAD_BALANCE=ON. When you set this parameter to ON, Oracle Database
randomly selects an address in the address list, and connects to that node's listener. This balances
O ly
client connections across the available SCAN listeners in the cluster.

l & On
The SCAN listener redirects the connection request to the local listener of the instance that is
least loaded and provides the requested service. When the listener receives the connection
a e
request, the listener connects the user to an instance that the listener knows provides the
n
t e r U s
requested service. When using SCAN, Oracle Net automatically load balances client connection
requests across the three IP addresses you defined for the SCAN, except when using EZConnect.

n
To see what services a listener supports, run the lsnrctl services command.
I
c l e
Other Client-Side Connection Features
With the current release, we now have the ability to add the connect_timeout and

a
retry_count parameters to individual tnsnames.ora connection strings.
r
O
(CONNECT_TIMEOUT=10)(RETRY_COUNT=3)
The granularity is seconds. Oracle Net waits for 10 seconds to receive a response, after which it
assumes a failure. Oracle Net goes through the address list three times before it returns a failure
to the client.

Oracle Database 11g: RAC Administration C - 4


Client-Side, Connect-Time Failover

ERP =
(DESCRIPTION =
(ADDRESS_LIST =
(LOAD_BALANCE=ON)
(FAILOVER=ON) 3
(ADDRESS=(PROTOCOL=TCP)(HOST=node1vip)(PORT=1521))
(ADDRESS=(PROTOCOL=TCP)(HOST=node2vip)(PORT=1521))
) 4
(CONNECT_DATA=(SERVICE_NAME=ERP)))

node2vip
2 node1vip
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Client-Side, Connect-Time Failover
e A
c l
This feature enables clients to connect to another listener if the initial connection to the first
listener fails. If an error is returned from the chosen address in the list, Oracle Net Services tries

r a
the next address in the list until it is either successful or it has exhausted all addresses in its list.
For SCAN, Oracle Net Services tries all three addresses before returning a failure to the client.
O ly
EZConnect with SCAN includes this connection failover feature.
& On
To increase availability, you can specify a timeout that specifies how long Oracle Net waits for a
l
n a e
response from the listener before returning an error. The method of setting this timeout
parameter depends on the type of client access. As shown above, client-side, connect-time

t e r U s
failover is enabled by setting FAILOVER=ON in the corresponding client-side TNS entry.
In the example, the client will randomly attempt connections to either NODE1VIP or NODE2VIP,

I n
because LOAD_BALANCE is set to ON. In the case where one of the nodes is down, the client
cannot know this. If a connection attempt is made to a down node, the client must wait until

l e
notification is received that the node is not accessible before an alternate address in the
c
ADDRESS_LIST is tried.

r a
Using virtual host names in the ADDRESS_LIST of your connect descriptors is recommended. If

O
a failure of a node occurs (1), the virtual IP address assigned to that node is failed over and
brought online on another node in the cluster (2). Thus, all client connection attempts are still
able to get a response from the IP address, without the need to wait for the operating system
TCP/IP timeout (3). Therefore, clients get an immediate acknowledgement from the IP address,
and are notified that the service on that node is not available.

Oracle Database 11g: RAC Administration C - 5


Client-Side, Connect-Time Failover (continued)
The next address in the ADDRESS_LIST can then be tried immediately with no delay (4).
Note: If you use connect-time failover, do not set GLOBAL_DBNAME in your listener.ora
file.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration C - 6
Server-Side Load Balancing

Server-side load balancing:


Causes the listener to direct connection requests to the best
instance currently providing the service
Uses connection information from the LBA
When DBCA is used to create a RAC database:
Server-side load balancing is configured and enabled.
The REMOTE_LISTENER parameter is set to the SCAN
listener.
If DBCA is not used or listener ports other than 1521 are
used, LOCAL_LISTENER and REMOTE_LISTENER
parameters should point to scan_name:scan_port
The LBA depends on an accurate configuration that
includes setting the CLB_GOAL for the service
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Server-Side Load Balancing
e A
l
With server-side load balancing, the listener directs a connection request to the best instance
c
r a
currently providing the service by using information from the Load Balancing Advisory. When
you create a RAC database with DBCA, it automatically configures and enables server-side load
O ly
balancing. It also sets the remote listener parameter to the SCAN listener and creates a sample

l & On
client-side load balancing connection definition in the tnsnames.ora file on the server. If you
did not use DBCA, or if you are using listener ports other than the default of 1521, then you
a e
must configure the LOCAL_LISTENER and REMOTE_LISTENER database initialization
n
t e r
parameters for your cluster database to point to scan_name:scan_port.

U s
FAN, Fast Connection Failover, and the load balancing advisory depend on an accurate

I n
connection load balancing configuration that includes setting the connection load balancing goal
for the service. You can use a goal of either LONG or SHORT for connection load balancing.
l e
Use the LONG connection load balancing method for applications that have long-lived
c
r a
connections. This is typical for connection pools and SQL*Forms sessions. LONG is the default
CLB_GOAL value. The following is an example of modifying a service, BATCH, with the srvctl

O
utility to define the connection load balancing goal for long-lived sessions:
srvctl modify service -d db_unique_name -s BATCH -j LONG

Oracle Database 11g: RAC Administration C - 7


Server-Side Load Balancing (continued)
Use the SHORT connection load balancing method for applications that have short-lived
connections. When using connection pools that are integrated with FAN, set the CLB_GOAL to
SHORT. The following example modifies the service known as OLTP, using SRVTCL to set the
connection load balancing goal to SHORT:
srvctl modify service -d db_unique_name -s OLTP -j SHORT

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration C - 8
Runtime Connection Load Balancing and
Connection Pools

CRM Client
Connection Requests

Connection
Cache

LBA indicates Instance1 and


60% 10% 30% Instance3 have the best
performance
CRM
CRM CRM
is
is is very
busy
bored busy
Database

CPU CPU CPU

Instance1 Instance2 Instance3

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Runtime Connection Load Balancing and Connection Pools
e A
l
Runtime Connection Load Balancing is a feature of Oracle connection pools that can distribute
c
r a
client work requests across the instances in an Oracle RAC database based on the Load
Balancing Advisory information. The connection allocation is based on the current performance
O ly
level provided by the database instances as indicated by the Load Balancing Advisory FAN

l & On
events. This provides load balancing at the transaction level, instead of load balancing at the
time of the initial database connection.
a e
With Runtime Connection Load Balancing, applications use LBA information to provide better
n
t e r U s
performance to users. OCI Session pools and ODP.NET connection pools support Runtime
Connection Load Balancing. For Java applications, Oracle recommends the Universal

n
Connection Pool (UCP). The UCP is integrated to take advantage of LBA information.
I
l e
You must enable the client data source for Runtime Connection Load Balancing with a service
that has the following configuration:
c
r a
The Load Balancing Advisory is enabled and the service-level goal is set to either Service
Time or Throughput.
O The service connection load balancing goal is set to Short.
The figure above illustrates Runtime Connection Load Balancing. In this illustration, the Oracle
RAC database has three instances.

Oracle Database 11g: RAC Administration C - 9


Runtime Connection Load Balancing and Connection Pools (continued)
Suppose that the Load Balancing Advisory indicates that Instance1 and Instance3 have the best
performance, while Instance2 currently has less than optimal performance. When Runtime
Connection Load Balancing is enabled on the implicit connection cache, the following process
occurs:
1. A client requests a connection from the connection pool.
2.Runtime Connection Load Balancing selects the connection that belongs to the most efficient
(best) instance from the connection pool. In the example, there are three possible nodes to which
the connection can be routed. Instance1, which has the least amount of CPU workload, is
currently being assigned about 60 percent of the incoming connections. Instance2, which is
currently overloaded, is only being assigned around 10 percent of the incoming connections.
Instance3, which has a high workload, is being assigned around 30 percent of the incoming
connections. The best instance to handle the connection request in this case would be Instance1.
3. The client receives the connection that would process the work request with the best response
time.
Oracle Database 11g introduces an additional flag in the load balancing advisory event called
affinity hint. The affinity hint is automatic when load balancing advisory is turned on when
setting the goal on the service. This flag is for temporary affinity that lasts for the duration of a
web session. Web conversations often connect and disconnect many times during the entire

shopping cart, Siebel, and so on. Affinity can improve buffer cache efficiency, which lowers
m y
session. During each of these connects, it may access the same or similar data, for example, a

CPU usage and transaction latency. The Affinity hint is a flag that indicates if Affinity is active

d e
or inactive for a particular instance and service combination. Different instances offering the
same service can have different settings for the Affinity hint.
c a
Applications using Oracle Database 11g and UCP can take advantage of this new affinity
A
feature. If the affinity flag is turned on in the Load Balancing Advisory event, then UCP creates
e
c l
an affinity context for the web session such that when that session does a get connection from
the pool, the pool always tries to give it a connection to the instance it connected to the first time

r a
it acquired a session. The choice of instance for the first connection is based on the current load
balancing advisory information.
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration C - 10
Fast Application Notification: Overview

.NET app C app C app Java app Java app

ODP.NET OCI API ONS OCI ONS Java JDBC


API API

ONS ONS

AQ
HA
Proxy Events
ONS
app

Callout
HA HA DB
CRS EMD
script Events Events Control

Callout
exec HA
Events Node1
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Fast Application Notification: Overview
e A
l
Fast Application Notification (FAN) enables end-to-end, lights-out recovery of applications and
c
r a
load balancing based on real transaction performance in a RAC environment. With FAN, the
continuous service built into Oracle Real Application Clusters is extended to applications and
O ly
mid-tier servers. When the state of a database service changes, (for example, up, down, or not

l & On
restarting), the new status is posted to interested subscribers through FAN events. Applications
use these events to achieve very fast detection of failures, and rebalancing of connection pools
a e
following failures and recovery. The easiest way to receive all the benefits of FAN, with no
n
Oracle Database JDBC
t e r
effort, is to use a client that is integrated with FAN:

U s
I n
Server-side callouts
Oracle Notification Service (ONS) API

l e
Oracle Universal Connection Pool (UCP) for Java
c
OCI Connection Pool or Session Pool

r a
Transparent Application Failover (TAF)
ODP.NET Connection Pool
O
Note: The integrated Oracle clients must be Oracle Database 10g Release 2 or later to take
advantage of the load balancing advisory FAN events.

Oracle Database 11g: RAC Administration C - 11


Fast Application Notification: Benefits

No need for connections to rely on connection timeouts


Used by Load Balancing Advisory to propagate load information
Designed for enterprise application and management console
integration
Reliable distributed system that:
Detects high-availability event occurrences in a timely
manner
Pushes notification directly to your applications
Tightly integrated with:
Oracle JDBC applications using connection pools
Enterprise Manager
Data Guard Broker

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Fast Application Notification: Benefits
e A
l
Traditionally, client or mid-tier applications connected to the database have relied on connection
c
r a
timeouts, out-of-band polling mechanisms, or other custom solutions to realize that a system
component has failed. This approach has huge implications in application availability, because
O ly
down times are extended and more noticeable.

l & On
With FAN, important high-availability events are pushed as soon as they are detected, which
results in a more efficient use of existing computing resources, and a better integration with your
a e
enterprise applications, including mid-tier connection managers, or IT management consoles,
n
t e r U s
including trouble ticket loggers and email/paging servers.
FAN is a distributed system that is enabled on each participating node. This makes it very

I n
reliable and fault tolerant because the failure of one component is detected by another.
Therefore, event notification can be detected and pushed by any of the participating nodes.

l e
FAN events are tightly integrated with Oracle Data Guard Broker, Oracle UCP, ODP.NET, TAF,
c
and Enterprise Manager. For example, Oracle Database JDBC applications managing

r a
connection pools do not need custom code development. They are automatically integrated with

O
the ONS if implicit connection cache and fast connection failover are enabled.

Oracle Database 11g: RAC Administration C - 12


FAN Events

Event type Description


SERVICE Primary application service

SRV_PRECONNECT Shadow application service event


(mid-tiers and TAF using primary and
secondary instances)
SERVICEMEMBER Application service on a specific instance

DATABASE Oracle database

INSTANCE Oracle instance

ASM Oracle ASM instance

NODE Oracle cluster node

SERVICE_METRICS Load Balancing Advisory


m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
FAN Events
e A
l
Before using some of the FAN features such as Server Side Callouts, you should understand
c
r a
FAN events. Oracle RAC FAN events consist of header and detail information delivered as a set
of name-value pairs accurately describing the name, type and nature of the event. Based on this
O ly
information, the event recipient can take concrete management, notification, or synchronization

l & On
steps, such as shutting down the application connection manager, rerouting existing database
connection requests, refreshing stale connection references, logging a trouble ticket, or sending a
a e
page to the database administrator.
n
t e r U s
FAN events are system events, sent during periods when cluster nodes may become unreachable
and network interfaces slow or non-functional. There is an inherent reliance on minimal

I n
communication channel overhead to send, queue and receive notifications quickly. The objective
is to deliver FAN events so that they precede regular connection timeouts or typical polling
l e
intervals. Three categories of events are supported in Oracle RAC FAN:
c
Service events, which includes both application services and database services

r a
Node events, which includes cluster membership states and native join/leave operations

O Load Balancing Events sent from the RAC Load Balancing Advisory
RAC standardizes the generation, presentation, and delivery of events pertaining to managed
cluster resources..

Oracle Database 11g: RAC Administration C - 13


FAN Event Status

Event status Description


up Managed resource comes up.
down Managed resource goes down.
preconn_up Shadow application service comes up.
preconn_down Shadow application service goes down.
nodedown Managed node goes down.

not_restarting Managed resource cannot fail over to a


remote node.
unknown Status of managed resource is unknown

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
FAN Event Status
e A
l
This table describes the event status for each of the managed cluster resources seen previously.
c
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration C - 14
FAN Event Reasons

Event Reason Description

user User-initiated commands, such as srvctl and


sqlplus
failure Managed resource polling checks for and detects a
failure.
dependency Dependency of another managed resource that
triggered a failure condition
autostart Initial cluster boot: Managed resource has profile
attribute AUTO_START=1, and was offline before the
last Oracle Clusterware shutdown.
public_nw_down Public network is down

member_leave Member node has left the cluster.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
FAN Event Reasons
e A
l
The event status for each managed resource is associated with an event reason. The reason
c
r
reasons with a corresponding description.a
further describes what triggered the event. The table in the slide gives you the list of possible

O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration C - 15
FAN Event Format

<Event_Type>
VERSION=<n.n>
[service=<serviceName.dbDomainName>]
[database=<dbName>] [instance=<sid>]
[host=<hostname>]
status=<Event_Status>
reason=<Event_Reason>
[card=<n>]
timestamp=<eventDate> <eventTime>

SERVICE VERSION=1.0 service=ERP.oracle.com


database=ORCL status=up reason=user card=4
timestamp=21-Jul-2009 15:24:19

NODE VERSION=1.0 host=host01


status=nodedown timestamp=21-Jul-2009 12:41:02
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
FAN Event Format
e A
l
In addition to its type, status, and reason, a FAN event has other payload fields to further
c
r a
describe the unique cluster resource whose status is being monitored and published:
The event payload version, which is currently 1.0
O ly
The name of the primary or shadow application service. This name is excluded from NODE
events.

l & On
The name of the RAC database, which is also excluded from NODE events
a e
The name of the RAC instance, which is excluded from SERVICE, DATABASE, and
n
NODE events

t e r s
The name of the cluster host machine, which is excluded from SERVICE and DATABASE
U
events
I n
The service cardinality, which is excluded from all events except for SERVICE

l e
status=up events

c
The server-side date and time when the event is detected

r a
The general FAN event format is described in the slide along with possible FAN event examples.

O
Note the differences in event payload for each FAN event type.

Oracle Database 11g: RAC Administration C - 16


Load Balancing Advisory: FAN Event

Parameter Description
Version Version of the event record
Event type SERVICE, SERVICE_MEMBER, DATABASE,
INSTANCE, NODE, ASM, SRV_PRECONNECT
Service Matches the service in DBA_SERVICES
Database unique name Unique DB name supporting the service
Time stamp Date and time stamp (local time zone)
Instance Instance name supporting the service
Percent Percentage of work to send to this database
and instance
Flag GOOD, VIOLATING, NO DATA, UNKNOWN

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Load Balancing Advisory: FAN Event
e A
l
The Load Balancing Advisory FAN event is described in the slide. Basically, it contains a
c
r a
calculated percentage of work requests that should be sent to each instance. The flag indicates
the behavior of the service on the corresponding instance relating to the thresholds set on that
instance for the service.
O ly
l & On
Use the following example to monitor load balancing advisory events:
SET PAGES 60 COLSEP '|' LINES 132 NUM 8 VERIFY OFF FEEDBACK OFF
a e
COLUMN user_data HEADING "AQ Service Metrics" FORMAT A60 WRAP
n
BREAK ON service_name SKIP 1

t e r U s
I n
SELECT TO_CHAR(enq_time, 'HH:MI:SS') Enq_time, user_data
FROM sys.sys$service_metrics_tab
ORDER BY 1 ;

c l e
r a
O
Oracle Database 11g: RAC Administration C - 17
Implementation of Server-Side Callouts

The callout directory:


<GRID_Home>/racg/usrco
Can store more than one callout
Grants execution on callouts and the callout directory to the
Oracle Clusterware user
The order in which callouts are executed is
nondeterministic.
Writing callouts involves:
1. Parsing callout arguments: The event payload
2. Filtering incoming FAN events
3. Executing event-handling programs

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Implementation of Server-Side Callouts
e A
l
Each database event detected by the RAC High Availability (HA) framework results in the
c
r a
execution of each executable script or program deployed in the standard Oracle Clusterware
callout directory. On UNIX, it is GRID_Home/racg/usrco. You must deploy each new callout
on each RAC node.
O ly
l & On
The order in which these callouts are executed is nondeterministic. However, RAC guarantees
that all callouts are invoked once for each recognized event in an asynchronous fashion. Thus,
a e
merging callouts whose executions need to be in a particular order is recommended.
n
t e r U s
You can install as many callout scripts or programs as your business requires, provided each
callout does not incur expensive operations that delay the propagation of HA events. If many

I n
callouts are going to be written to perform different operations based on the event received, it
might be more efficient to write a single callout program that merges each single callout.

c l e
Writing server-side callouts involves the steps shown in the slide. In order for your callout to

r a
identify an event, it must parse the event payload sent by the RAC HA framework to your
callout. After the sent event is identified, your callout can filter it to avoid execution on each
Oevent notification. Then, your callout needs to implement a corresponding event handler that
depends on the event itself and the recovery process required by your business.
Note: As a security measure, make sure that the callout directory and its contained callouts have
write permissions only to the system user who installed Oracle Clusterware.

Oracle Database 11g: RAC Administration C - 18


Server-Side Callout Parse: Example
#!/bin/sh
NOTIFY_EVENTTYPE=$1
for ARGS in $*; do
PROPERTY=`echo $ARGS | $AWK -F"=" '{print $1}'`
VALUE=`echo $ARGS | $AWK -F"=" '{print $2}'`
case $PROPERTY in
VERSION|version) NOTIFY_VERSION=$VALUE ;;
SERVICE|service) NOTIFY_SERVICE=$VALUE ;;
DATABASE|database) NOTIFY_DATABASE=$VALUE ;;
INSTANCE|instance) NOTIFY_INSTANCE=$VALUE ;;
HOST|host) NOTIFY_HOST=$VALUE ;;
STATUS|status) NOTIFY_STATUS=$VALUE ;;
REASON|reason) NOTIFY_REASON=$VALUE ;;
CARD|card) NOTIFY_CARDINALITY=$VALUE ;;
TIMESTAMP|timestamp) NOTIFY_LOGDATE=$VALUE ;;
??:??:??) NOTIFY_LOGTIME=$PROPERTY ;;
esac
done
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Server-Side Callout Parse: Example
e A
l
Unless you want your callouts to be executed on each event notification, you must first identify
c
r a
the event parameters that are passed automatically to your callout during its execution. The
example in the slide shows you how to parse these arguments by using a sample Bourne shell
script.
O ly
l & On
The first argument that is passed to your callout is the type of event that is detected. Then,
depending on the event type, a set of PROPERTY=VALUE strings are passed to identify exactly
the event itself.
n a e
t e r U s
The script given in the slide identifies the event type and each pair of PROPERTY=VALUE
string. The data is then dispatched into a set of variables that can be used later in the callout for
filtering purposes.
I n
c l e
As mentioned in the previous slide, it might be better to have a single callout that parses the
event payload, and then executes a function or another program on the basis of information in

r a
the event, as opposed to having to filter information in each callout. This becomes necessary
only if many callouts are required.
O
Note: Make sure that executable permissions are set correctly on the callout script.

Oracle Database 11g: RAC Administration C - 19


Server-Side Callout Filter: Example

if ((( [ $NOTIFY_EVENTTYPE = "SERVICE" ] ||


[ $NOTIFY_EVENTTYPE = "DATABASE" ] || \
[ $NOTIFY_EVENTTYPE = "NODE" ] \
) && \
( [ $NOTIFY_STATUS = "not_restarting" ] \
)) && \
( [ $NOTIFY_DATABASE = "PROD" ] || \
[ $NOTIFY_SERVICE = "ERP" ] \
))
then
/usr/local/bin/logTicket $NOTIFY_LOGDATE \
$NOTIFY_LOGTIME \
$NOTIFY_SERVICE \
$NOTIFY_DBNAME \

y
$NOTIFY_HOST
fi

em
Copyright 2010, Oracle and/or its affiliates. All rights reserved.
a d
Server-Side Callout Filter: Example A c
l e
The example in the slide shows you a way to filter FAN events from a callout script. This
c
r a
example is based on the example in the previous slide.
Now that the event characteristics are identified, this script triggers the execution of the trouble-
O ly
logging program /usr/local/bin/logTicket only when the RAC HA framework posts

l & On
a SERVICE, DATABASE, or NODE event type, with a status set to not_restarting, and
only for the production PROD RAC database or the ERP service.

n a e
It is assumed that the logTicket program is already created and that it takes the arguments
shown in the slide.
t e r U s
It is also assumed that a ticket is logged only for not_restarting events, because they are
I n
the ones that exceeded internally monitored timeouts and seriously need human intervention for

c l e
full resolution.

r a
O
Oracle Database 11g: RAC Administration C - 20
Configuring the Server-Side ONS

Mid-tier1
localport=6100
remoteport=6200 ONS

$ srvctl add ons -t 6200 node1:6200 node2:6200


2
$ srvctl add ons -t midtier1:6200

$ onsctl reload 3 3 $ onsctl reload

ONS ONS

OCR
ons.config 1 1 ons.config
Node1 Node2
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Configuring the Server-Side ONS
e A
l
The ONS configuration is controlled by the <GRID Home>/opmn/conf/ons.config
c
r a
configuration file. This file is automatically created during installation. There are three
important parameters that should always be configured for each ONS:
O ly
The first is localport, the port that ONS uses to talk to local clients.

l & On
The second is remoteport, the port that ONS uses to talk to other ONS daemons.
The third parameter is called nodes. It specifies the list of other ONS daemons to talk to.
a e
This list should include all RAC ONS daemons, and all mid-tier ONS daemons. Node
n
t e r
values are given as either host names or IP addresses followed by its remoteport.

U s
In the slide, it is assumed that ONS daemons are already started on each cluster node. This

I n
should be the default situation after a correct RAC installation. However, if you want to use
OCR, you should edit the ons.config file on each node, and then add the configuration to OCR

l e
before reloading it on each cluster node. This is illustrated in the slide.
c
r a
O
Oracle Database 11g: RAC Administration C - 21
Optionally Configure the Client-Side ONS

Mid-tier1

ons.config ONS $ onsctl start 2

localport=6100
remoteport=6200 1
nodes=node1:6200,node2:6200

ONS ONS

OCR
ons.config ons.config
Node1 Node2
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Optionally Configure the Client-Side ONS
e A
l
Oracle Database FAN uses Oracle Notification Service (ONS) on the mid-tier to receive FAN
c
r a
events when you are using the Java Database Connectivity (JDBC) connection cache. To use
ONS on the mid-tier, you need to install ONS on each host where you have client applications
O ly
that need to be integrated with FAN. Most of the time, these hosts play the role of a mid-tier

l & On
application server. Therefore, on the client side, you must configure all the RAC nodes in the
ONS configuration file. A sample configuration file might look like the one shown in the slide.
a e
After configuring ONS, you start the ONS daemon with the onsctl start command. It is
n
t e r U s
your responsibility to make sure that an ONS daemon is running at all times. You can check that
the ONS daemon is active by executing the onsctl ping command.

I n
c l e
r a
O
Oracle Database 11g: RAC Administration C - 22
JDBC Fast Connection Failover: Overview

Mid-tier1
Service Service or node
ONS
UP event DOWN event
JDBC ICC
Event
handler

Connections Connections
reconnected Connection Cache marked down &
cleaned up
Connections Connections
using Listeners using
service names service names
Connections
load balancing
ONS ONS
y

Node1 Noden
e m
Copyright 2010, Oracle and/or its affiliates. All rights reserved.
a d
JDBC Fast Connection Failover: Overview A c
l e
Oracle Application Server integrates Implicit Connection Cache (ICC) with the ONS API by
c
r a
having application developers enable Fast Connection Failover (FCF). FCF works in
conjunction with the ICC to quickly and automatically recover lost or damaged connections.
O ly
This automatic connection management results from FAN events received by the local ONS

l & On
daemon, or by a remote ONS if a local one is not used, and handled by a special event handler
thread. Both JDBC thin and JDBC OCI drivers are supported.
a e
Therefore, if ICC and FCF are enabled, your Java program automatically becomes an ONS
n
t e r U s
subscriber without having to manage FAN events directly.
Whenever a service or node down event is received by the mid-tier ONS, the event handler

I n
automatically marks the corresponding connections as down and cleans them up. This prevents
applications that request connections from the cache from receiving invalid or bad connections.
l e
Whenever a service up event is received by the mid-tier ONS, the event handler recycles some
c
r a
unused connections, and reconnects them using the event service name. The number of recycled
connections is automatically determined by the connection cache. Because the listeners perform

O
connection load balancing, this automatically rebalances the connections across the preferred
instances of the service without waiting for application connection requests or retries.
For more information, refer to the Oracle Database JDBC Developers Guide and Reference.
Note: Similarly, ODP.NET also allows you to use FCF using AQ for FAN notifications.

Oracle Database 11g: RAC Administration C - 23


Using Oracle Streams Advanced Queuing for FAN

Use AQ to publish FAN to ODP.NET and OCI.


Turn on FAN notification to alert queue.

$ srvctl modify service -d crm -s gl.us.oracle.com q TRUE

View published FAN events using the


DBA_OUTSTANDING_ALERTS or DBA_ALERT_HISTORY
views

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Using Oracle Streams Advanced Queuing for FAN
e A
l
RAC publishes FAN events to a system alert queue in the database by using Oracle Streams
c
FAN events.
r a
Advanced Queuing (AQ). ODP.NET and OCI client integration uses this method to subscribe to

O ly
To have FAN events for a service posted to that alert queue, the notification must be turned on

l & On
for the service. You can do this using the Enterprise Manager interface.
To view FAN events that are published, you can use the DBA_OUTSTANDING_ALERTS or
DBA_ALERT_HISTORY views.
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration C - 24
JDBC/ODP.NET FCF Benefits

Database connections are balanced across preferred


instances according to LBA.
Database work requests are balanced across preferred
instances according to LBA.
Database connections are anticipated.
Database connection failures are immediately detected
and stopped.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
JDBC/ODP.NET FCF Benefits
e A
l
By enabling FCF, your existing Java applications connecting through Oracle JDBC Universal
c
r a
Connection Pool (UCP) and application services, or your .NET applications using ODP.NET
connection pools and application services benefit from the following:
O ly
All database connections are balanced across all RAC instances that support the new

l & On
service name, instead of having the first batch of sessions routed to the first RAC instance.
This is done according to the Load Balancing Advisory algorithm you use (see the next
a e
slide). Connection pools are rebalanced upon service, instance, or node up events.
n
t e r
The connection cache immediately starts placing connections to a particular RAC instance
s
when a new service is started on that instance.
U
I n
The connection cache immediately shuts down stale connections to RAC instances where
the service is stopped, or whose node goes down.

l e
Your application automatically becomes a FAN subscriber without having to manage FAN
c
events directly by just setting up flags in your connection descriptors..

r a
Note: For more information about how to subscribe to FAN events, refer to the Oracle Database

O
JDBC Developers Guide.

Oracle Database 11g: RAC Administration C - 25


Load Balancing Advisory

The Load Balancing Advisory (LBA) is an advisory for


sending work across RAC instances.
LBA advice is available to all applications that send work:
JDBC and ODP connection pools
Connection load balancing
There are two types of service-level goals:
SERVICE_TIME: Directs work requests to instances
according to response time.
$ srvctl modify service -d PROD -s OE -B SERVICE_TIME -j SHORT

THROUGHPUT: Directs requests based on the rate that work is


completed in the service plus available bandwidth.
$ srvctl modify service -d PROD -s BATCH -B THROUGHPUT -j LONG

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Load Balancing Advisory
e A
l
Load balancing distributes work across all available database instances. The LBA provides
c
r a
advice about how to direct incoming work to the instances that provide the optimal quality of
service for that work, minimizing the need to relocate the work later. By using the
O ly
SERVICE_TIME or THROUGHPUT goals, feedback is built into the system.

l & On
SERVICE_TIME: Attempts to direct work requests to instances according to response
time. Load balancing advisory data is based on elapsed time for work done in the service
a e
plus available bandwidth to the service. An example for the use of SERVICE_TIME is for
n

e r
workloads such as Internet shopping where the rate of demand changes
s
THROUGHPUT: Attempts to direct work requests according to throughput. The load
t U
I n
balancing advisory is based on the rate that work is completed in the service plus available
bandwidth to the service. An example for the use of THROUGHPUT is for workloads such

l e
as batch processes, where the next job starts when the last job completes:
c
Work is routed to provide the best service times globally, and routing responds gracefully to

r a
changing system conditions. In a steady state, the system approaches equilibrium with improved

O
throughput across all of the Oracle RAC instances. The load balancing advisory is deployed with
key Oracle clients, such as a listener, the JDBC universal connection pool, and the ODP.NET
Connection Pool. The load balancing advisory is also open for third-party subscription by way of
the JDBC and Oracle RAC FAN API or through callbacks with the Oracle Call Interface.

Oracle Database 11g: RAC Administration C - 26


Runtime Connection Load Balancing and
Connection Pools

CRM Client
Connection Requests

Connection
Cache

LBA indicates Instance1 and


60% 10% 30% Instance3 have the best
performance
CRM
CRM CRM
is
is is very
busy
bored busy

CPU CPU CPU

Instance1 Instance2 Instance3

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Runtime Connection Load Balancing and Connection Pools
e A
l
Runtime Connection Load Balancing is a feature of Oracle connection pools that can distribute
c
r a
client work requests across the instances in an Oracle RAC database based on the Load
Balancing Advisory information. The connection allocation is based on the current performance
O ly
level provided by the database instances as indicated by the Load Balancing Advisory FAN

l & On
events. This provides load balancing at the transaction level, instead of load balancing at the
time of the initial database connection.
a e
With Runtime Connection Load Balancing, applications use LBA information to provide better
n
t e r U s
performance to users. OCI Session pools and ODP.NET connection pools support Runtime
Connection Load Balancing. For Java applications, Oracle recommends the Universal

n
Connection Pool (UCP). The UCP is integrated to take advantage of LBA information.
I
l e
You must enable the client data source for Runtime Connection Load Balancing with a service
that has the following configuration:
c
r a
The Load Balancing Advisory is enabled and the service-level goal is set to either Service
Time or Throughput.
O The service connection load balancing goal is set to Short.
The figure above illustrates Runtime Connection Load Balancing. In this illustration, the Oracle
RAC database has three instances.

Oracle Database 11g: RAC Administration C - 27


Monitor LBA FAN Events

SQL> SELECT TO_CHAR(enq_time, 'HH:MI:SS') Enq_time, user_data


2 FROM sys.sys$service_metrics_tab
3 ORDER BY 1 ;

ENQ_TIME USER_DATA
-------- -----------------------------------------------------

04:19:46 SYS$RLBTYP('JFSERV', 'VERSION=1.0 database=xwkE
service=JFSERV { {instance=xwkE2 percent=50
flag=UNKNOWN}{instance=xwkE1 percent=50 flag=UNKNOWN}
} timestamp=2009-01-02 06:19:46')
04:20:16 SYS$RLBTYP('JFSERV', 'VERSION=1.0 database=xwkE
service=JFSERV { {instance=xwkE2 percent=80
flag=UNKNOWN}{instance=xwkE1 percent=20 flag=UNKNOWN}
} timestamp=2009-01-02 06:20:16')
SQL>

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Monitor LBA FAN Events
e A
l
You can use the SQL query shown in the slide to monitor the Load Balancing Advisory FAN
c
events for each of your services.
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration C - 28
Transparent Application Failover: Overview

TAF Basic TAF Preconnect

2 Application 2 Application
OCI Library 6 OCI Library 6

4 Net Services 4 Net Services

5 3 7 5 3

8 7
AP 3 ERP 3

7 3
1 1

AP ERP ERP_PRECONNECT

1
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Transparent Application Failover (TAF): Overview
e A
l
TAF is a run-time feature of the OCI driver. It enables your application to automatically
c
r a
reconnect to the service if the initial connection fails. During the reconnection, although your
active transactions are rolled back, TAF can optionally resume the execution of a SELECT

O ly
statement that was in progress. TAF supports two failover methods:

l & On
With the BASIC method, the reconnection is established at failover time. After the service
has been started on the nodes (1), the initial connection (2) is made. The listener establishes

n a e
the connection (3), and your application accesses the database (4) until the connection fails
(5) for any reason. Your application then receives an error the next time it tries to access

e r s
the database (6). Then, the OCI driver reconnects to the same service (7), and the next time
t U
your application tries to access the database, it transparently uses the newly created

I n
connection (8). TAF can be enabled to receive FAN events for faster down events detection

l e
and failover.
The PRECONNECT method is similar to the BASIC method except that it is during the
c
r a initial connection that a shadow connection is also created to anticipate the failover. TAF
guarantees that the shadow connection is always created on the available instances of your

O service by using an automatically created and maintained shadow service.


Note: Optionally, you can register TAF callbacks with the OCI layer. These callback functions
are automatically invoked at failover detection and allow you to have some control of the
failover process. For more information, refer to the Oracle Call Interface Programmers Guide.

Oracle Database 11g: RAC Administration C - 29


TAF Basic Configuration Without FAN: Example

$ srvctl add service -d RACDB -s AP -r I1,I2 \


> -P BASIC
$ srvctl start service -d RACDB -s AP

AP =
(DESCRIPTION =(FAILOVER=ON)(LOAD_BALANCE=ON)
(ADDRESS=(PROTOCOL=TCP)(HOST=cluster01-scan)(PORT=1521))
(CONNECT_DATA =
(SERVICE_NAME = AP)
(FAILOVER_MODE =
(TYPE=SELECT)
(METHOD=BASIC)
(RETRIES=180)
(DELAY=5))))
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
TAF Basic Configuration Without FAN: Example
e A
l
Before using TAF, it is recommended that you create and start a service that is used during
c
r a
connections. By doing so, you benefit from the integration of TAF and services. When you want
to use BASIC TAF with a service, you should have the -P BASIC option when creating the

O ly
service. After the service is created, you simply start it on your database.

l & On
Then, your application needs to connect to the service by using a connection descriptor similar
to the one shown in the slide. The FAILOVER_MODE parameter must be included in the
a e
CONNECT_DATA section of your connection descriptor:
n

t e r
TYPE specifies the type of failover. The SELECT value means that not only the user
s
session is reauthenticated on the server side, but also the open cursors in the OCI can
U
continue fetching. This implies that the client-side logic maintains the fetch-state of each
I n
open cursor. A SELECT statement is reexecuted by using the same snapshot, discarding

c l e
those rows already fetched, and retrieving those rows that were not fetched initially. TAF
verifies that the discarded rows are those that were returned initially, or it returns an error

r
a message.
METHOD=BASIC is used to reconnect at failover time.
O

RETRIES specifies the number of times to attempt to connect after a failover.
DELAY specifies the amount of time in seconds to wait between connect attempts.

Oracle Database 11g: RAC Administration C - 30


TAF Basic Configuration with FAN: Example

$ srvctl add service -d RACDB -s AP -r I1,I2

$ srvctl start service -d RACDB -s AP

srvctl modify service -d RACDB -s AP -q TRUE -P BASIC \


-e SELECT -z 180 -w 5 -j LONG

AP =
(DESCRIPTION =
(ADDRESS=(PROTOCOL=TCP)(HOST=cluster01-scan)(PORT=1521))
(LOAD_BALANCE = YES)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = TEST)
(FAILOVER_MODE =
(TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5))))
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
TAF Basic Configuration with FAN: Example
e A
l
Oracle Database 11g Release 2 supports server-side TAF with FAN. To use server-side TAF,
c
r a
create and start your service using SRVCTL, and then configure TAF in the RDBMS by using
the srvctl command as shown in the slide. When done, make sure that you define a TNS

O ly
entry for it in your tnsnames.ora file. Note that this TNS name does not need to specify TAF

& On
parameters as in the previous slide.

l
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration C - 31
TAF Preconnect Configuration: Example

ERP =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = cluster01-scan)(PORT = 1521))
(LOAD_BALANCE = YES)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ERP)
(FAILOVER_MODE =
(BACKUP = ERP_PRECONNECT)
(TYPE = SELECT)(METHOD = PRECONNECT)(RETRIES = 180)(DELAY = 5))))

ERP_PRECONNECT =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = cluster01-scan)(PORT = 1521))
(LOAD_BALANCE = YES)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = ERP_PRECONNECT)
(FAILOVER_MODE =
(TYPE = SELECT)(METHOD = BASIC)(RETRIES = 180)(DELAY = 5)))
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
TAF Preconnect Configuration: Example
e A
l
In order for the shadow service to be created and managed automatically by Oracle Clusterware,
c
r a
you must define the service with the P PRECONNECT option. The shadow service is always
named using the format <service_name>_PRECONNECT.

O ly
The main differences with the previous example are that METHOD is set to PRECONNECT and

l & On
an additional parameter is added. This parameter is called BACKUP and must be set to another
entry in your tnsnames.ora file that points to the shadow service.

n a e
Note: In all cases where TAF cannot use the PRECONNECT method, TAF falls back to the
BASIC method automatically.

t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration C - 32
TAF Verification
SELECT machine, failover_method, failover_type,
failed_over, service_name, COUNT(*)
FROM v$session
GROUP BY machine, failover_method, failover_type,
failed_over, service_name;

MACHINE FAILOVER_M FAILOVER_T FAI SERVICE_N COUNT(*)


First ------- ---------- ---------- --- -------- --------
node node1 BASIC SESSION NO AP 1
node1 PRECONNECT SESSION NO ERP 1

MACHINE FAILOVER_M FAILOVER_T FAI SERVICE_N COUNT(*)


Second
------- ---------- ---------- --- --------- --------
node
node2 NONE NONE NO ERP_PRECO 1

MACHINE FAILOVER_M FAILOVER_T FAI SERVICE_N COUNT(*)


Second
------- ---------- ---------- --- -------- --------
node
after
node2
node2
BASIC
PRECONNECT
SESSION
SESSION
YES
YES
AP
ERP_PRECO
1
1
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
TAF Verification
e A
l
To determine whether TAF is correctly configured and that connections are associated with a
c
r a
failover option, you can examine the V$SESSION view. To obtain information about the
connected clients and their TAF status, examine the FAILOVER_TYPE, FAILOVER_METHOD,

O ly
FAILED_OVER, and SERVICE_NAME columns. The example includes one query that you

l & On
could execute to verify that you have correctly configured TAF.
This example is based on the previously configured AP and ERP services, and their
a e
corresponding connection descriptors.
n
t e r U s
The first output in the slide is the result of the execution of the query on the first node after two
SQL*Plus sessions from the first node have connected to the AP and ERP services, respectively.

I n
The output shows that the AP connection ended up on the first instance. Because of the load-
balancing algorithm, it can end up on the second instance. Alternatively, the ERP connection
l e
must end up on the first instance because it is the only preferred one.
c
r a
The second output is the result of the execution of the query on the second node before any
connection failure. Note that there is currently one unused connection established under the
O
ERP_PROCONNECT service that is automatically started on the available ERP instance.
The third output is the one corresponding to the execution of the query on the second node after
the failure of the first instance. A second connection has been created automatically for the AP
service connection, and the original ERP connection now uses the preconnected connection.
Oracle Database 11g: RAC Administration C - 33
FAN Connection Pools and TAF Considerations

Both techniques are integrated with services and provide


service connection load balancing.
Do not use FCF when working with TAF, and vice versa.
Connection pools that use FAN are always preconnected.
TAF may rely on operating system (OS) timeouts to detect
failures.
FAN never relies on OS timeouts to detect failures.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
FAN Connection Pools and TAF Considerations
e A
l
Because the connection load balancing is a listener functionality, both FCF and TAF
c
r a
automatically benefit from connection load balancing for services.
When you use FCF, there is no need to use TAF. Moreover, FCF and TAF cannot work together.
O ly
For example, you do not need to preconnect if you use FAN in conjunction with connection
& On
pools. The connection pool is always preconnected.
l
n a e
With both techniques, you automatically benefit from VIPs at connection time. This means that
your application does not rely on lengthy operating system connection timeouts at connect time,

e r s
or when issuing a SQL statement. However, when in the SQL stack, and the application is
t U
blocked on a read/write call, the application needs to be integrated with FAN in order to receive
I n
an interrupt if a node goes down. In a similar case, TAF may rely on OS timeouts to detect the

c l e
failure. This takes much more time to fail over the connection than when using FAN.

r a
O
Oracle Database 11g: RAC Administration C - 34
Summary

In this lesson, you should have learned how to:


Configure client-side, connect-time load balancing
Configure client-side, connect-time failover
Configure server-side, connect-time load balancing
Use the Load Balancing Advisory
Describe the benefits of Fast Application Notification
Configure server-side callouts
Configure Transparent Application Failover

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration C - 35
m y
d e
ca
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle RAC One Node

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle RAC One Node

Allows you to standardize Oracle database


App
deployments across the enterprise
Servers Allows you to consolidate many databases
into a single cluster with minimal overhead
Supports live migration of instances across
Front Back servers
Office Office
Simplifies rolling patches for single instance
databases
Employs built-in cluster failover for high
availability
DW Free
Is supported on all platforms where Oracle
Real Application Clusters is certified
Complements virtual and physical servers
RAC One
Node

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Oracle RAC One Node
e A
l
Oracle RAC One Node allows you to standardize all Oracle Database deployments across the
c
r a
enterprise. Oracle RAC One Node is supported on all platforms where Oracle Real Application
Clusters (RAC) is certified. Many databases can be consolidated into a single cluster with
O ly
minimal overhead while providing the high availability benefits of failover protection, online

Clusterware.
l & On
rolling patch application, as well as rolling upgrades for the operating system and Oracle

a e
Oracle Clusterware provides failover protection to Oracle RAC One Node. If the node fails,
n
server in the cluster.
t e r U s
Oracle Clusterware will automatically restart the Oracle RAC One Node instance on another

I n
c l e
r a
O
Oracle Database 11g: RAC Administration D - 2
The Omotion Utility

Omotion migrates an Oracle RAC One Node instance to


another cluster node without application down time.
Once the instance has been migrated, the node can be
patched or upgraded.
When the maintenance is complete, the Oracle RAC One
Node instance can be migrated back to its original home.
Omotion provides the same load balancing benefits of virtual
machines (VMs) by allowing a migration of a database from
a busy server to a server with spare capacity.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
The Omotion Utility
e A
l
The Omotion utility is provided to migrate an Oracle RAC One Node instance to another node in
c
r a
the cluster without down time to the application. Once the instance has been migrated, the node
can be patched or upgraded. Once the maintenance is complete, the Oracle RAC One Node can
be migrated back to its original home.
O ly
l & On
The Omotion feature provides the same load balancing benefits of VMs by allowing a migration
of a database from a busy server to a server with spare capacity. Omotion leverages the ability of
a e
Oracle Real Application Clusters to simultaneously run multiple instances servicing a single
n
t e r U s
database. In the figure above, the DB2 RAC One Node database on Server A is migrated to
Server B. Oracle RAC One Node starts up a second DB2 instance on server B, and for a short

I n
period of time runs in an active-active configuration. As connections complete their transactions
on server A, they are migrated to the instance on server B. Once all the connections have
l e
migrated, the instance on server A is shut down and the migration is complete.
c
r a
Omotion does not require quiescing the environment even when the system is running at peak
capacity. VMs generally require the environment to be quiesced in order for medium to heavy
O
database workloads to be moved from one server to another. This requirement does not apply for
light workloads or demos.

Oracle Database 11g: RAC Administration D - 3


Oracle RAC One Node and OVM

Using Oracle RAC One Node with Oracle VM increases


the scalability and high availability benefits of Oracle VM.
If a VM is sized too small, you can migrate the Oracle RAC
One instance to another Oracle VM node in your cluster
and resize the Oracle VM.
Usually, migrating VMs requires quiescing the source VM,
mirroring the memory, and then switching to the migrated
VM.
Using Omotion, highly loaded instances can be migrated
as the work is split between the two servers during the
migration.
Omotion provides the ability to move between servers of
different processor generations.
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Oracle RAC One Node and OVM
e A
l
Oracle VM is a free server virtualization and management solution that makes enterprise
c
r a
applications easier to deploy, manage, and support. Using Oracle RAC One Node with Oracle
VM increases the benefit of Oracle VM with the high availability and scalability of Oracle RAC.

O ly
If your VM is sized too small, then you can migrate the Oracle RAC One instance to another
Oracle VM node in your cluster using Omotion, and then resize the Oracle VM. When you move

l & On
the instance back to the newly resized Oracle VM node, you can dynamically increase any limits

a e
programmed with Resource Manger Instance Caging.
n
t e r s
When migrating a VM, the VM must mirror its complete memory state across a network to the
target host, recreating the state of that machine. If the database in question is highly loaded and
U
I n
is actively changing blocks in its database cache, it is very difficult for the memory mirroring
function to keep up with the rate of changes. It becomes likely that the only way to successfully

l e
mirror the memory is to quiesce the source VM, mirror the memory, and then switch to the

c
migrated VM. With Omotion, highly loaded instances pose no problem, as the work is actually

r a
split between two servers during the migration. Oracle RAC One Node can, therefore, easily
migrate even heavy workloads to another server. VM migration normally requires that the
O
processors be identical. Both processors must have exactly the same instruction set. Omotion
provides the ability to move between servers of different processor generations. Omotion
supports migration to new processors, or even between different vendors like Intel and AMD.

Oracle Database 11g: RAC Administration D - 4


Creating and Managing RAC One Node
1. Create a RAC database on one node only.

2. Initialize the database with the raconeinit command.


3. Execute the raconestatus command to confirm that the
initialization was successful.
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Creating and Managing RAC One Node
e A
l
Oracle RAC One Node is a single instance of Oracle RAC that runs on one node in a cluster.
c
r a
You can create a RAC One Node database on an existing cluster using DBCA. On the Database
Identification page, you simply select one node only on which to create the database as shown
O ly
above. After the database has been created, it must be initialized as a RAC One Node database

l & On
using the raconeinit command. It will display a list of databases running in the cluster and
allow you to choose which one to initialize:
[oracle@host01 ~]$ raconeinit
n a e
# Database r U s
Candidate Databases on this cluster:

t e
RACOne Node Fix Required
=== ========
I n ============= ============

e
[1] ORCL NO N

l
Enter the database to initialize [1]: 1

c
r a
Database orcl is now running on server host01.
Candidate servers that may be used for this DB: host02

O
Enter the names of additional candidate servers where this DB may run (space
delimited): host02
Database configuration modified.
Execute the raconestatus command to confirm that the initialization was successful.

Oracle Database 11g: RAC Administration D - 5


Creating and Managing RAC One Node (continued)
[oracle@host01 ~]$ raconestatus
RAC One Node databases on this cluster:
Database UP Fix Required Current Server Candidate Server Names
======== == ============ ============== ======================
orcl Y y host01 host02 host01
One of the unique features of RAC One Node is the ability to online migrate a running database from one
node in the cluster to another. This migration, called Omotion, is controlled by the Omotion command.
To continue with our example, move the database now running on host01 to host02 using the Omotion
command. Choose 1 to select the orcl database. Hit return to accept the default maximum migration time
of 30 minutes.
[oracle@host01 ~]$ Omotion
RAC One Node databases on this cluster:
# Database Server Fix Required
=== ======== =========== ===================
[1] orcl rac1 N
Enter number of the database to migrate [1]: 1
Specify maximum time in minutes for migration to complete (max 30) [30]:
Available Target Server(s) :
# Server Available
===
[1]
=========== =========
host02 Y
m y
Enter number of the target node [1]: 1
Omotion Started...
d e
Starting target instance on host02...
c a
Migrating sessions...
Stopping source instance on host01...
e A
Omotion Completed...

c l
=== Current Status ===

r
Database racone is running on node host02a
O ly
Assume host01 has crashed. Running raconestatus shows the following:
[oracle@host01 ~]$ raconestatus

l & On
RAC One Node databases on this cluster:
Database UP Fix Required
n a e
Current Server Candidate Server Names
======== == ============
orcl Y y
t e r U s
==============
host02
======================
host02 host01

I n
After a failover, you must run the raconefix command to reset the RAC One metadata.

c l e
[oracle@host01 ~]$ raconefix
RAC One Node databases on this cluster:
#

r
=== a Database
========
Server
==============================
Fix Required
============

O
[1] orcl
Enter number of the database to fix [1]: 1
host01 Y

If database is up, it will be checked and cleaned after a previous fail over.

Oracle Database 11g: RAC Administration D - 6


Cloning Oracle Clusterware

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Objectives

After completing this lesson, you should be able to:


Describe the cloning procedure
Prepare the software for cloning
Describe the cloning script variables
Clone Oracle Clusterware to create a new cluster
Clone Oracle Clusterware to extend an existing cluster
Examine cloning log files

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 2
What Is Cloning?

Cloning is the process of copying an existing Oracle


Clusterware installation to a different location. It:
Requires a successful installation as a baseline
Can be used to create new clusters
Cannot be used to remove nodes from the cluster
Does not perform the operating system prerequisites to an
installation
Is useful to build many clusters in an organization

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
What Is Cloning?
e A
l
Cloning is a process that allows the copying of an existing Oracle Clusterware installation to a
c
r a
different location and then updating the copied installation to work in the new environment. The
cloned copy can be used to create a new cluster from a successfully installed cluster. To add or
O ly
delete Oracle Clusterware from nodes in the cluster, use the addNode.sh and rootcrs.pl

l & On
scripts. The cloning procedure cannot be used to remove a node from an existing cluster.
The cloning procedure is responsible for the work that would have been done by the Oracle
a e
Universal Installer (OUI) utility. It does not automate the prerequisite work that must be done on
n
t e r U s
each node before installing the Oracle software.
This technique is very useful if a large number of clusters need to be deployed in an

I n
organization. If only one or two clusters are being deployed, you should probably use the

c l e
traditional installation program to perform the installations.
Note: The Oracle Enterprise Manager administrative tool with the Provisioning Pack feature

a
installed has automated wizards to assist with cloning exercises.
r
O
Oracle Database 11g: RAC Administration E - 3
Benefits of Cloning Oracle Clusterware

The following are some of the benefits of cloning Oracle


Clusterware:
Can be completed in silent mode from a Secure Shell
(SSH) terminal session
Contains all patches applied to the original installation
Can be done very quickly
Is a guaranteed method of repeating the same installation
on multiple clusters

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Benefits of Cloning Oracle Clusterware
e A
l
Using the cloning procedure presented in this lesson has several benefits compared to the
c
r a
traditional Oracle Universal Installer (OUI) installation method. The OUI utility is a graphical
program and must be executed from a graphical session. Cloning can be completed in silent
O ly
mode from a command-line Secure Shell (SSH) terminal session without the need to load a

l & On
graphical windows system. If the OUI program were to be used to install the software from the
original installation media, all patches that have been applied since the first installation would
a e
have to be reapplied. The clone technique presented in this lesson includes all successfully
n
t e r
applied patches, and can be performed very quickly to a large number of nodes. When the OUI
s
utility performs the copying of files to remote servers, the job is executed serially to one node at
U
I n
a time. With cloning, simultaneous transfers to multiple nodes can be achieved. Finally, the
cloning method is a guaranteed way of repeating the same installation on multiple clusters to

l e
help avoid human error.
c
The cloned installation acts the same as the source installation. It can be patched in the future

r a
and removed from a cluster if needed using ordinary tools and utilities. Cloning is an excellent

O
way to instantiate test clusters or a development cluster from a successful base installation.

Oracle Database 11g: RAC Administration E - 4


Creating a Cluster by Cloning
Oracle Clusterware

Cluster A Cluster B

Node 1 Node 1

Clusterware installed

Node 2

Clusterware cloned

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Creating a Cluster by Cloning Oracle Clusterware
e A
l
This example shows the result when you successfully clone an installed Oracle Clusterware
c
r a
environment to create a cluster. The environment on Node 1 in Cluster A is used as the source,
and Cluster B, Nodes 1 and 2 are the destination. The clusterware home is copied from Cluster
O ly
A, Node 1, to Cluster B, Nodes 1 and 2. When completed, there will be two separate clusters.

a cluster from a clone.


l & On
The OCR and Voting disks are not shared between the two clusters after you successfully create

n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 5
Preparing the Oracle Clusterware
Home for Cloning
The following procedure is used to prepare the Oracle
Clusterware home for cloning:
1. Install Oracle Clusterware on the first machine.
A. Use the Oracle Universal Installer (OUI) GUI interactively.
B. Install patches that are required (for example, 11.1.0.n).
C. Apply one-off patches, if necessary.
2. Shut down Oracle Clusterware.
# crsctl stop crs -wait

3. Make a copy of the Oracle Clusterware home.


# mkdir /stagecrs
# cp prf /u01/app/11.2.0/grid /stagecrs
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Preparing the Oracle Clusterware Home for Cloning
e A
l
The cloning method requires that an existing, successful installation of Oracle Clusterware be
c
r a
already performed in your organization. Verify that all patch sets and one-off patches have been
applied before starting the clone procedure to minimize the amount of work that will have to be
performed for the cloning exercise.
O ly
l & On
To begin the cloning process, start by performing a shutdown of Oracle Clusterware on one of
the nodes in the existing cluster with the crsctl stop crs wait command. The other
a e
nodes in the existing cluster can remain active. After the prompt is returned from the shutdown
n
t e r U s
command, make a copy of the existing Oracle Clusterware installation into a temporary staging
area of your choosing. The disk space requirements in the temporary staging area will be equal

n
to the current size of the existing Oracle Clusterware installation.
I
l e
The copy of the Oracle Clusterware files will not include files that are outside the main
installation directory such as /etc/oraInst.loc and the /etc/oracle directory. These
c
a
files will be created later by running various root scripts.
r
O
Oracle Database 11g: RAC Administration E - 6
Preparing the Oracle Clusterware
Home for Cloning
4. Remove files that pertain only to the source node.
# cd /stagecrs/grid
# rm -rf /stagecrs/grid/log/<hostname>
# rm -rf root.sh*
# rm -rf gpnp/*
# find . -name '*.ouibak' -exec rm {} \;
# find . -name '*.ouibak.1' -exec rm {} \;
# rm rf \
inventory/ContentsXML/oraclehomeproperties.xml
# cd ./cfgtoollogs
# find . -type f -exec rm -f {} \;

5. Create an archive of the source.


# cd /stagecrs/grid
# tar zcvf /tmp/crs111060.tgz .
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Preparing the Oracle Clusterware Home for Cloning (continued)
e A
l
Each installation of Oracle Clusterware on local storage devices contains files and directories
c
r a
that are applicable to only that node. These files and directories should be removed before
making an archive of the software for cloning. Perform the commands in the slide to remove
O ly
node-specific information from the copy that was made in step 3. Then create an archive of the

l & On
Oracle Clusterware copy. An example of using the Linux tar command followed by the
compress command to reduce the file size is shown in the slide. On Windows systems, use the
a e
WinZip utility to create a zip file for the archive.
n
home.
t e r U s
Note: Do not use the Java Archive (JAR) utility to copy and compress the Oracle Clusterware

I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 7
Preparing the Oracle Clusterware
Home for Cloning
6. Restart Oracle Clusterware.
# crsctl start crs

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 8
Cloning to Create a New Oracle
Clusterware Environment
The following procedure uses cloning to create a new cluster:
1. Prepare the new cluster nodes. (See the lesson titled Grid
Infrastructure Installation for details.)
A. Check system requirements.
B. Check network requirements.
C. Install the required operating system packages.
D. Set kernel parameters.
E. Create groups and users.
F. Create the required directories.
G. Configure installation owner shell limits.
H. Configure SSH and enable user equivalency.
I. Use the Cluster Verify Utility (cluvfy) to check
prerequisites.
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cloning to Create a New Oracle Clusterware Environment
e A
l
The archive of the source files that was made when preparing the Oracle Clusterware home for
c
r a
cloning will become the source files for an installation on the new cluster system and the
clone.pl script will be used to make it a working environment. These actions do not perform

O ly
the varied prerequisite tasks that must be performed on the operating system before an Oracle

l & On
Clusterware installation. The prerequisite tasks are the same as the ones presented in the lesson
titled Grid Infrastructure Installation in the topic about installing Oracle Clusterware.
a e
One of the main benefits of cloning is rapid instantiation of a new cluster. The prerequisite tasks
n
t e r U s
are manual in nature, but it is suggested that a shell script be developed to automate these tasks.
One advantage of the Oracle Enterprise Linux operating system is that an oracle-

I n
validated-1.x.x.rpm file can be obtained that will perform almost all the prerequisite
checks including the installation of missing packages with a single command.
l e
This cloning procedure is used to build a new distinct cluster from an existing cluster. In step 1,
c
r a
prepare the new cluster nodes by performing prerequisite setup and checks.
If cluvfy fails to execute because of user equivalence errors, the passphrase needs to be loaded
O
with the following commands before executing cluvfy:
exec /usr/bin/ssh-agent $SHELL
ssh-add

Oracle Database 11g: RAC Administration E - 9


Cloning to Create a New Oracle
Clusterware Environment
2. Deploy Oracle Clusterware on each of the destination
nodes.
A. Extract the tar file created earlier.
# mkdir p /u01/app/11.20/grid
# cd /u01/app/11.2.0
# tar zxvf /tmp/crs111060.tgz

B. Change the ownership of files, and create Oracle Inventory.


# chown R crs:oinstall /u01/app/11.2.0/grid
# mkdir p /u01/app/oraInventory
# chown grid:oinstall /u01/app/oraInventory

C. Remove any network files from


/u01/app/11.2.0/grid/network/admin
$ rm /u01/app/11.2.0/grid/network/admin/*
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cloning to Create a New Oracle Clusterware Environment (continued)
e A
l
Step 2 is the deployment and extraction of the source archive to the new cluster nodes. If a
c
r a
shared Oracle Cluster home on a Cluster File System (CFS) is not being utilized, extract the
source archive to each nodes local file system. It is possible to change the operating system
O ly
owner to a different one from that of the source with recursive chown commands as illustrated

l & On
in the slide. If other Oracle products have been previously installed on the new nodes, the
Central Oracle Inventory directory may already exist. It is possible for this directory to be owned
a e
by a different user than the Oracle Grid Infrastructure user; however, both should belong to the
n
t e r
same primary group oinstall. Execute the preupdate.sh script on each target node,
s
logged in as root. This script will change the ownership of the CRS home and root-owned
U
n
files to the Oracle CRS user.
I
c l e
r a
O
Oracle Database 11g: RAC Administration E - 10
The clone.pl Script

Cloning to a new cluster and cloning to extend an existing


cluster both use a PERL script. The clone.pl script is used,
which:
Can be used on the command line
Can be contained in a shell script
Accepts many parameters as input
Is invoked by the PERL interpreter

# perl <CRS_home>/clone/bin/clone.pl $E01 $E02 $E03


$E04 $C01 $CO2

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
The clone.pl Script
e A
l
The PERL-based clone.pl script is used in place of the graphical OUI utility to perform the
c
r a
installation on the new nodes so that they may participate in the existing cluster or become valid
nodes in a new cluster. The script can be executed directly on the command line or at a DOS
O ly
command prompt for Windows platforms. The clone.pl script accepts several parameters as

l & On
input, typed directly on the command line. Because the clone.pl script is sensitive to the
parameters being passed to it, including the use of braces, single quotation marks, and double
a e
quotation marks, it is recommended that a shell script be created to execute the PERL-based
n
t e r
clone.pl script to input the arguments. This will be easier to rework if there is a syntax error
s
generated. There are a total of seven arguments that can be passed as parameters to the
U
I n
clone.pl script. Four of them define environment variables, and the remaining three supply
processing options. If your platform does not include a PERL interpreter, you can download one
at:
c l e
http://www.perl.org

r a
O
Oracle Database 11g: RAC Administration E - 11
The clone.pl Environment Variables

The clone.pl script accepts four environment variables as


input. They are as follows:

Symbol Variable Description


E01 ORACLE_BASE The location of the Oracle base directory
E02 ORACLE_HOME The location of the Oracle Grid Infrastructure
home. This directory location must exist and
must be owned by the Oracle operating
system group: oinstall.
E03 ORACLE_HOME_NAME The name of the Oracle Grid Infrastructure
home
E04 INVENTORY_LOCATION The location of the Oracle Inventory

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
The clone.pl Environment Variables
e A
l
The clone.pl script accepts four environment variables as command-line arguments
c
r a
providing input. These variables would correlate to some of the questions presented by the OUI
utility during an installation. Each variable is associated with a symbol that is used for reference
O ly
purposes for developing a shell script. The choice of symbol is arbitrary. Each variable is case-


l & On
sensitive. The variables are as follows:
ORACLE_BASE: The location of the Oracle Base directory. A suggested value is
a e
/u01/app/grid. The ORACLE_BASE value should be unique for each software owner.
n

t e r
ORACLE_HOME: The location of the Oracle Grid Infrastructure home. This directory
s
location must exist and be owned by the Oracle Grid Infrastructure software owner and the
U
I n
Oracle Inventory group, typically grid:oinstall. A suggested value is
/u01/app/<version>/grid.

c l e
ORACLE_HOME_NAME: The name of the Oracle Grid Infrastructure home. This is stored
in the Oracle inventory and defaults to the name Ora11g_gridinfrahome1 when

r a performing an installation with OUI. Any name can be selected, but it should be unique in
the organization.
O INVENTORY_LOCATION: The location of the Oracle Inventory. This directory location
must exist and must initially be owned by the Oracle operating system group: oinstall. A
typical location is /u01/app/oraInventory.

Oracle Database 11g: RAC Administration E - 12


The clone.pl Command Options

The clone.pl script accepts two required command options


as input. They are as follows:
# Variable Data Type Description
C01 CLUSTER_NODES String This list of short node names for the
nodes in the cluster
C02 LOCAL_NODE String The short name of the local node

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
The clone.pl Command Options
e A
l
The clone.pl script accepts two command options as input. The command options are case-
c
sensitive and are as follows:

r a
CLUSTER_NODES: This list of short node names for the nodes in the cluster

O ly
LOCAL_NODES: The short name of the local node

l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 13
Cloning to Create a New Oracle
Clusterware Environment
3. Create a shell script to invoke clone.pl supplying input.
#!/bin/sh
ORACLE_BASE=/u01/app/oracle
GRID_HOME=/u01/app/11.2.0/grid
THIS_NODE=`hostname -s`

E01=ORACLE_BASE=${ORACLE_BASE}
E02=ORACLE_HOME=${ORACLE_HOME}
E03=ORACLE_HOME_NAME=OraGridHome1
E04=INVENTORY_LOCATION=${ORACLE_BASE}/oraInventory

#C00="-O'-debug'"
C01="-O'\"CLUSTER_NODES={node1,node2}\"'" Syntax per bug
C02="-O'\"LOCAL_NODE=${THIS_NODE}\"'" 8718041

perl ${GRID_HOME}/clone/bin/clone.pl -silent $E01 $E02


$E03 $E04 $C01 $C02
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cloning to Create a New Oracle Clusterware Environment
e A
l
The clone.pl script requires you to provide many setup values to the script when it is
c
r a
executed. You may enter the values interactively on the command line or create a script to
supply the input values. By creating a script, you will have the ability to modify it and execute
O ly
the script a second time if errors exist. The setup values to the clone.pl script are case-

l & On
sensitive and sensitive to the use of braces, single quotation marks, and double quotation marks.
For step 3, create a shell script that invokes clone.pl supplying command-line input
a e
variables. All the variables should appear on a single line.
n
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 14
Cloning to Create a New Oracle
Clusterware Environment
4. Run the script created in Step 3 on each node.
$ /tmp/my-clone-script.sh

5. Prepare the crsconfig_params file.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cloning to Create a New Oracle Clusterware Environment (continued)
e A
l
Step 4: Run the script you created in Step 3 as the operating system user that installed Oracle
c
r a
Clusterware. If you do not have a shared Oracle grid infrastructure home run this script on each
node. The clone.pl command instantiates the crsconfig_params file in the next step.

O ly
Step 5: Prepare the /u01/app/11.2.0/grid/install/crsconfig_params file on all

l & On
of the nodes in the cluster. You can copy the file from one node to all of the other nodes. More
than 50 parameters are named in this file.

n a e
Notes: In 11.2.0.1 the clone.pl script does not propagate the crsconfig_params file.

t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 15
Cloning to Create a New Oracle
Clusterware Environment
6. Run the orainstRoot.sh script on each node.
# /u01/app/oraInventory/orainstRoot.sh

7. Run the <CRS_home>/root.sh and rootcrs.pl scripts


on each node.
# /u01/app/11.2.0/grid/root.sh
# /u01/app/11.2.0/grid/perl/bin/perl \
-I/u01/app/11.2.0/grid/perl/lib \
-I/u01/app/11.2.0/grid/crs/install \
/u01/app/11.2.0/grid/crs/install/rootcrs.pl

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cloning to Create a New Oracle Clusterware Environment (continued)
e A
l
When the shell script containing clone.pl successfully completes, it is necessary to execute
c
as the root user.
r a
the orainstRoot.sh script for step 6 and the root.sh script for step 7 on each node, both

Note O ly
& On
Step 4 must be run to completion before you start Step 5. Similarly, Step 5 must be run to
l
n a e
completion before you start Step 6.
You can perform Step 4, Step 5, and Step 6 simultaneously on different nodes. Step 6 must be

e r s
complete on all nodes before you can run Step 7.
t U
I n
Step 7: Ensure that the root.sh and rootcrs.pl scripts have completed on the first node
before running them on the second node and subsequent nodes.

c l e
r a
O
Oracle Database 11g: RAC Administration E - 16
Cloning to Create a New Oracle
Clusterware Environment
8. Run the configuration assistants on each new node.
$ /u01/app/11.2.0/grid/bin/netca \
/orahome /u01/app/11.2.0/grid \
/orahnam OraGridHome1 /instype typical \
/inscomp client,oraclenet,javavm,server\
/insprtcl tcp /cfg local \
/authadp NO_VALUE \
/responseFile \
/u01/app/11.2.0/grid/network/install/netca_typ.rsp \
/silent

9. Run the cluvfy utility to validate the installation.

$ cluvfy stage post crsinst n all -verbose

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cloning to Create a New Oracle Clusterware Environment (continued)
e A
l
Step 8: You should execute the command shown in the slide, on the first node.
c
a
If you are using ASM, then run the following command:
r
O ly
$ /u01/app/11.2.0/grid/bin/asmca -silent postConfigureASM \

& On
-sysAsmPassword oracle -asmsnmpPassword oracle

l
If you a plan to run a pre-11g Release 2 (11.2) database on this cluster, then you should run
a e
oifcfg as described in the Oracle Database 11g Release 2 (11.2) documentation.

r n s
To use IPMI, use the crsctl command to configure IPMI on each node:

t e U
# crsctl set css ipmiaddr ip_address
n
e I
At this point, Oracle Clusterware is fully installed and configured to operate in the new cluster.
The cluvfy utility can be invoked in the post-CRS installation stage to verify the installation

c l
with the following syntax:

r a cluvfy stage post crsinst -n all verbose

O
If cluvfy fails to execute because of user equivalence errors, the passphrase needs to be loaded
with the following commands before executing cluvfy again:
exec /usr/bin/ssh-agent $SHELL
ssh-add

Oracle Database 11g: RAC Administration E - 17


Log Files Generated During Cloning

The following log files are generated during cloning to assist


with troubleshooting failures.
Detailed log of the actions that occur during the OUI part:
/u01/app/oraInventory/logs/cloneActions<timestamp>.log

Information about errors that occur when OUI is running:


/u01/app/oraInventory/logs/oraInstall<timestamp>.err

Other miscellaneous messages generated by OUI:


/u01/app/oraInventory/logs/oraInstall<timestamp>.out

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Log Files Generated During Cloning
e A
l
Several log files are generated when the clone.pl script is executed and are useful in
c
r a
diagnosing any errors that may occur. For a detailed log of the actions that occur during the OUI
part of the cloning, examine the log file:
O ly
<Central_Inventory>logs/cloneActions/<timestamp>.log

l & On
If errors occurred during the OUI portion of the cloning process, examine the log file:
<Central_Inventory>logs/oraInstall/<timestamp>.err

n a e
Other miscellaneous messages generated by OUI can be found in the output file:

t e r U s
<Central_Inventory>logs/oraInstall/<timestamp>.out

I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 18
Log Files Generated During Cloning

The following log files are generated during cloning to assist


with troubleshooting failures:
Detailed log of the actions that occur before cloning as well
as during cloning operations:
/u01/app/grid/clone/logs/clone-<timestamp>.log

Information about errors that occur before cloning as well


as during cloning operations:
/u01/app/grid/clone/logs/error-<timestamp>.log

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Log Files Generated During Cloning (continued)
e A
l
Several log files are generated when the clone.pl script is executed and are useful in
c
r a
diagnosing errors that may occur. For a detailed log of the actions that occur before cloning as
well as during the cloning process, examine the log file:
O ly
<CRS home>/clone/logs/clone-<timestamp>.log

examine the log file:


l & On
For a detailed log of the errors that occur before cloning as well as during the cloning process,

a e
<CRS home>/clone/logs/error-<timestamp>.log
n
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 19
Cloning to Extend Oracle Clusterware
to More Nodes
The procedure is very similar to cloning to create new clusters.
1. Prepare the new cluster nodes. (See the lesson titled
Oracle Clusterware Installation for details. Suggest that
this be made into a shell script for reuse.)
A. Check system requirements.
B. Check network requirements.
C. Install the required operating system packages.
D. Set kernel parameters.
E. Create groups and users.
F. Create the required directories.
G. Configure installation owner shell limits.
H.
I.
Configure SSH and enable user equivalency.
Use the cluvfy utility to check prerequisites.
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cloning to Extend Oracle Clusterware to More Nodes
e A
l
The cloning procedure using the clone.pl script can also be used to extend Oracle
c
r a
Clusterware to more nodes within the same cluster with steps similar to the procedure for
creating a new cluster. The first step shown here is identical to the first step performed when
O ly
cloning to create a new cluster. These steps differ depending on the operating system used.

l & On
When you configure Secure Shell (SSH) and enable user equivalency, remember that the
authorized_keys and known_hosts files exist on each node in the cluster. Therefore, it
a e
will be necessary to update these files on the existing nodes with information about the new
n
t e r
Note: Not supported in 11.2.0.1
U s
nodes that Oracle Clusterware will be extended to.

I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 20
Cloning to Extend Oracle Clusterware
to More Nodes
2. Deploy Oracle Clusterware on the destination nodes.
A. Extract the tar file created earlier.
# mkdir p /u01/app/crs
# cd /u01/app/crs
# tar zxvf /tmp/crs111060.tgz

B. Change the ownership of files and create Oracle Inventory.


# chown R crs:oinstall /u01/app/crs
# mkdir p /u01/app/oraInventory
# chown crs:oinstall /u01/app/oraInventory

C. Run the preupdate.sh script on each target node.


# /u01/app/crs/install/preupdate.sh \
crshome /u01/app/crs crsuser crs -noshutdown

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cloning to Extend Oracle Clusterware to More Nodes (continued)
e A
l
Step 2 is the deployment and extraction of the source archive to the new cluster nodes. If a
c
r a
shared Oracle Cluster home on a CFS is not being used, extract the source archive to each
nodes local file system. Because this node will participate with the existing nodes in a cluster, it
O ly
is not possible to use different account names for the Oracle Clusterware software owner. If

l & On
other Oracle products have been previously installed on the new nodes, the Central Oracle
Inventory directory may already exist. It is possible for this directory to be owned by a different
a e
user than the Oracle Clusterware user; however, both should belong to the same primary group
n
t e r
oinstall. Execute the preupdate.sh script on each target node, logged in as root. This
s
script will change the ownership of the CRS home and root-owned files to the Oracle CRS
U
user if needed.
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 21
Cloning to Extend Oracle Clusterware
to More Nodes
3. Create a shell script to invoke clone.pl supplying input.
#!/bin/sh
# /tmp/my-clone-script.sh
E01=ORACLE_BASE=/u01/app
E02=ORACLE_HOME=/u01/app/crs
E03=ORACLE_HOME_NAME=OraCrs11g
C01="-O'sl_tableList={node3:node3-priv:node3-vip:N:Y}'"
C02="-O'INVENTORY_LOCATION=/u01/app/oraInventory'"
C03="-O'-noConfig'"

perl /u01/app/crs/clone/bin/clone.pl $E01 $E02 $E03 $C01 $C02 $C03

4. Run the shell script created in step 3 on each new node.


$ /tmp/my-clone-script.sh

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cloning to Extend Oracle Clusterware to More Nodes (continued)
e A
l
For step 3, the quantity of input variables to the clone.pl procedure is greatly reduced
c
r a
because the existing cluster has already defined many of the settings that are needed. Creating a
shell script to provide these input values is still recommended. For step 4, run the shell script
O ly
that you developed to invoke the clone.pl script.

l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 22
Cloning to Extend Oracle Clusterware
to More Nodes
5. Run the orainstRoot.sh script on each new node.
# /u01/app/oraInventory/orainstRoot.sh

6. Run the addNode script on the source node.


$ /u01/app/crs/oui/bin/addNode.sh silent \
"CLUSTER_NEW_NODES={new_nodes}" \
"CLUSTER_NEW_PRIVATE_NODE_NAMES={new_node-priv}" \
"CLUSTER_NEW_VIRTUAL_HOSTNAMES={new_node-vip}" \
noCopy

7. Run the rootaddnode.sh script on the source node.


# /u01/app/crs/install/rootaddnode.sh

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cloning to Extend Oracle Clusterware to More Nodes (continued)
e A
l
For step 5, run the orainstRoot.sh script on each new node as the root user. For step 6,
c
r a
run the addNode.sh script on the source node, not on the destination node. Because the
clone.pl scripts have already been run on the new nodes, this script only updates the

O ly
inventories of the existing nodes. For step 7, again on the source node, run the

& On
rootaddnode.sh script as root to instantiate the node.

l
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 23
Cloning to Extend Oracle Clusterware
to More Nodes
8. Run the <CRS_home>/root.sh script on the new node.
# /u01/app/crs/root.sh

9. Run the configuration assistants on each node that is listed


in the configToolAllCommands file.
$ cat /u01/app/crs/cfgtoollogs/configToolAllCommands

$ onsconfig add_config node3:6501

10. Run cluvfy to validate the installation.


$ cluvfy stage post crsinst n all -verbose

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Cloning to Extend Oracle Clusterware to More Nodes (continued)
e A
l
Next, in step 8, run the root.sh script on each new node joining the cluster as the root user.
c
r a
For step 9, additional commands for the configuration assistants need to be run on the new nodes
joining the cluster as the Oracle Clusterware software owner. The list of commands can be found
O ly
in the <CRS home>/cfgtoollogs/configToolAllCommands file. Finally, for step

installation.
l & On
10, which is the last step, run cluvfy to verify the success of the Oracle Clusterware

n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 24
Quiz

An Oracle Clusterware home that was created with cloning


techniques can be used as the source for additional cloning
exercises.
1. True
2. False

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 1
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 25
Quiz

Which scripting language is used for the cloning script?


1. Java
2. PERL
3. Shell
4. Python

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Answer: 2
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 26
Summary

In this lesson, you should have learned how to:


Describe the cloning process
Describe the clone.pl script and its variables
Perform a clone of Oracle Clusterware to a new cluster
Extend an existing cluster by cloning

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration E - 27
m y
d e
ca
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Clusterware Concepts

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Objectives

After completing this lesson, you should be able to:


Explain the principles and purposes of clusters
Describe the Oracle Clusterware architecture
Describe how Grid Plug and Play affects Clusterware

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration F - 2
Oracle Grid Infrastructure
ASM and Oracle Clusterware are installed into a single
home directory called Oracle Grid Infrastructure 11g
Release 2.
This directory is referred to as the Grid Infrastructure
home.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Oracle Grid Infrastructure
e A
l
With Oracle Database 11g Release 2, Automatic Storage Management (ASM) and Oracle
c
r a
Clusterware are installed into a single home directory, collectively called Oracle Grid
Infrastructure. This directory is referred to as the Grid Infrastructure home. Configuration
O ly
assistants start after the Oracle Universal Installer interview process and binary installation that

l & On
configure ASM and Oracle Clusterware. Although the installation is called Oracle Grid
Infrastructure, Oracle Clusterware and Automatic Storage Manager remain separate
components.
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration F - 3
What Is a Cluster?

A group of independent, but interconnected, computers


that act as a single system
Usually deployed to
Network
increase availability and Interconnect
performance or Users
to balance a dynamically
changing workload

Storage Network

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
What Is a Cluster?
e A
c l
A cluster consists of a group of independent but interconnected computers whose combined
resources can be applied to a processing task. A common cluster feature is that it should appear
r a
to an application as though it were a single server. Most cluster architectures use a dedicated

O ly
network (cluster interconnect) for communication and coordination between cluster nodes.

l & On
A common cluster architecture for data-intensive transactions and computations is built around
shared disk storage. Shared-nothing clusters use an alternative architecture where storage is not
a e
shared and data must be either replicated or segmented across the cluster. Shared-nothing
n
e r
clusters are commonly used for workloads that can be easily and predictably divided into small
s
units that can be spread across the cluster in parallel. Shared disk clusters can perform these
t U
tasks but also offer increased flexibility for varying workloads. Load balancing clusters allow a
I n
single application to balance its workload across the cluster. Alternatively, in a failover cluster,

c l e
some nodes can be designated as the primary host for an application, whereas others act as the
primary host for different applications. In a failover cluster, the failure of a node requires that the

r a
applications it supports be moved to a surviving node. Load balancing clusters can provide
failover capabilities but they can also run a single application across multiple nodes providing
O
greater flexibility for different workload requirements. Oracle supports a shared disk cluster
architecture providing load balancing and failover capabilities. In an Oracle cluster, all nodes
must share the same processor architecture and run the same operating system.

Oracle Database 11g: RAC Administration F - 4


What Is Clusterware?

Software that provides various interfaces and services for a


cluster. Typically, this includes capabilities that:
Allow the cluster to be managed as a whole
Protect the integrity of the cluster
Maintain a registry of resources across the cluster
Deal with changes to the cluster
Provide a common view of resources

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
What Is Clusterware?
e A
l
Clusterware is a term used to describe software that provides interfaces and services that enable
c
and support a cluster.
r a
Different cluster architectures require clusterware that delivers different services. For example,
O ly
in a simple failover cluster the clusterware may monitor the availability of applications and

l & On
perform a failover operation if a cluster node becomes unavailable. In a load balancing cluster,
different services are required to support workload concurrency and coordination.

n a e
Typically, clusterware includes capabilities that:

desired
t e r U s
Allow the cluster to be managed as a single entity (not including OS requirements), if

I n
Protect the integrity of the cluster so that data is protected and the cluster continues to

l e
function even if communication with a cluster node is severed
Maintain a registry of resources so that their location is known across the cluster and so
c
r athat dependencies between resources is maintained
Deal with changes to the cluster such as node additions, removals, or failures
O Provide a common view of resources such as network addresses and files in a file system

Oracle Database 11g: RAC Administration F - 5


Oracle Clusterware

Oracle Clusterware is:


A key part of Oracle Grid Infrastructure
Integrated with Oracle
Automatic Storage
Management (ASM)
The basis for ASM Cluster
File System (ACFS)
A foundation for Oracle
Real Application Clusters
(RAC)
A generalized cluster
infrastructure for all kinds
of applications
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Oracle Clusterware
e A
l
Oracle Clusterware is a key part of Oracle Grid Infrastructure, which also includes Automatic
c
r a
Storage Management (ASM) and the ASM Cluster File System (ACFS).
In Release 11.2, Oracle Clusterware can use ASM for all the shared files required by the cluster.
O ly
Oracle Clusterware is also an enabler for the ASM Cluster File System, a generalized cluster file

& On
system that can be used for most file-based data such as documents, spreadsheets, and reports.
l
n a e
The combination of Oracle Clusterware, ASM, and ACFS provides administrators with a unified
cluster solution that is not only the foundation for the Oracle Real Application Clusters (RAC)

t e r U s
database, but can also be applied to all kinds of other applications.
Note: Grid Infrastructure is the collective term that encompasses Oracle Clusterware, ASM, and
I n
ACFS. These components are so tightly integrated that they are often collectively referred to as

c l e
Oracle Grid Infrastructure.

r a
O
Oracle Database 11g: RAC Administration F - 6
Oracle Clusterware Architecture and Services

Shared disk cluster architecture supporting application


load balancing and failover
Services include:
Cluster management
Node monitoring
Event services
Time synchronization
Network management
High availability

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Oracle Clusterware Architecture and Services
e A
l
Oracle Clusterware provides a complete set of cluster services to support the shared disk, load
c
r a
balancing cluster architecture of the Oracle Real Application Cluster (RAC) database. Oracle
Clusterware can also be used to provide failover clustering services for single instance Oracle
databases and other applications.
O ly
l & On
The services provided by Oracle Clusterware include:
Cluster Management, which allows cluster services and application resources to be
a e
monitored and managed from any node in the cluster
n
t e r U s
Node Monitoring, which provides real-time information regarding which nodes are
currently available and the resources they support. Cluster integrity is also protected by

I n
evicting or fencing unresponsive nodes.
Event Services, which publishes cluster events so that applications are aware of changes in
l e
the cluster
c
Time Synchronization, which synchronizes the time on all nodes of the cluster

r a
Network Management, which provisions and manages Virtual IP (VIP) addresses that are

O associated with cluster nodes or application resources to provide a consistent network


identity regardless of which nodes are available. In addition, Grid Naming Service (GNS)
manages network naming within the cluster.
High Availability, which services, monitors, and restarts all other resources as required

Oracle Database 11g: RAC Administration F - 7


Oracle Clusterware Networking
Each node must have at least two network adapters.
Each public network adapter must support TCP/IP.
The interconnect adapter must support:
User Datagram Protocol (UDP) or Reliable Data Socket (RDS)
for UNIX and Linux for database communication
TCP for Windows platforms for database communication
All platforms use Grid Interprocess Communication (GIPc)
Public network

NIC1 NIC1
NIC2 NIC2

NIC1
Interconnect: Private network
m y
NIC2

d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Oracle Clusterware Networking
e A
l
Each node must have at least two network adapters: one for the public network interface and the
c
r a
other for the private network interface or interconnect. In addition, the interface names
associated with the network adapters for each network must be the same on all nodes. For
O ly
example, in a two-node cluster, you cannot configure network adapters on node1 with eth0 as

l & On
the public interface, but on node2 have eth1 as the public interface. Public interface names must
be the same, so you must configure eth0 as public on both nodes. You should configure the
a e
private interfaces on the same network adapters as well. If eth1 is the private interface for node1,
n
t e r
then eth1 should be the private interface for node2.

U s
Before starting the installation, on each node, you must have at least two interfaces to configure

following options: I n
for the public and private IP addresses. You can configure IP addresses with one of the

l e
Oracle Grid Naming Service (GNS) using one static address defined during installation,
c
which dynamically allocates VIP addresses using Dynamic Host Configuration Protocol

r a (DHCP), which must be running on the network. You must select the Advanced Oracle

O Clusterware installation option to use GNS.


Static addresses that network administrators assign on a network domain name server
(DNS) or each node. To use the Typical Oracle Clusterware installation option, you must
use static addresses.

Oracle Database 11g: RAC Administration F - 8


Oracle Clusterware Networking (continued)
For the public network, each network adapter must support TCP/IP.
For the private network, the interconnect must support UDP or RDS (TCP for Windows) for
communications to the database. Grid Interprocess Communication (GIPc) is used for Grid
(Clusterware) interprocess communication. GIPC is a new common communications
infrastructure to replace CLSC/NS. It provides a full control of the communications stack from
the operating system up to whatever client library uses it. The dependency on network services
(NS) prior to 11.2 is removed, but there is still backwards compatibility with existing CLSC
clients (primarily from 11.1). GIPC can support multiple communications types: CLSC, TCP,
UDP, IPC and of course the communication type GIPC.
Use high-speed network adapters for the interconnects and switches that support TCP/IP.
Gigabit Ethernet or an equivalent is recommended.
Each node in a cluster requires a supported interconnect protocol to support Cache Fusion and
TCP/IP to support Clusterware polling. Token Ring is not supported for cluster interconnects on
IBM AIX. Your interconnect protocol must be certified by Oracle for your platform.
Note: Cross-over cables are not supported for use with Oracle Clusterware interconnects.

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration F - 9
Oracle Clusterware Startup
Oracle Clusterware is started by the OS init daemon.
Operating system Clusterware Oracle Clusterware processes
init daemon startup script
ohasd.bin oclskd.bin
init octssd.bin crsd.bin
/etc/init.d/init.ohasd
oraagent.bin gipcd.bin
diskmon.bin mdnsd.bin
ocssd.bin gpnpd.bin
evmd.bin scriptagent.bin
cssdagent oraagent.bin
orarootagent.bin

Oracle Clusterware installation modifies the


/etc/inittab file to restart ohasd in the event of a
crash.
# cat /etc/inittab
..
h1:35:respawn:/etc/init.d/init.ohasd run >/dev/null 2>&1 </dev/null
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Oracle Clusterware Startup
e A
l
During the installation of Oracle Clusterware, the init.ohasd startup script is copied to
c
r a
/etc/init.d. The wrapper script is responsible for setting up environment variables and then
starting the Oracle Clusterware daemons and processes.
O ly
The Oracle High Availability Services daemon (ohasd) is responsible for starting in proper

l & On
order, monitoring, and restarting other local Oracle daemons, up through the crsd daemon,
which manages clusterwide resources. When init starts ohasd on Clusterware startup,
a e
ohasd starts orarootagent, cssdagent, and oraagent. These processes then carry out
n
the following tasks:

t e r
orarootagent starts crsd.
U s
I n
- crsd starts another orarootagent process responsible for root-owned CRS
resources including the SCAN VIPS.

c l e
cssdagent starts cssd (ocssd).
oraagent starts mdnsd, evmd, ASM, ctssd, and gpnpd. oraagent also starts gsd,

r a Oracle Notification Service (ONS), and the listeners.

O
Some of the high availability daemons will be running under the root user with real-time
priority, and others will be running under the Clusterware owner with user-mode priorities after
they are started. When a command is used to stop Oracle Clusterware, the daemons will be
stopped, but the ohasd process will remain running.

Oracle Database 11g: RAC Administration F - 10


Oracle Clusterware Process Architecture
Clusterware processes are organized into several component groups.
They include:
Component Processes Owner
Cluster Ready Service (CRS) crsd root
Cluster Synchronization Service (CSS) ocssd,cssdmonitor, grid owner,
cssdagent root, root
Event Manager (EVM) evmd, evmlogger grid owner
Cluster Time Synchronization Service octssd root
(CTSS)
Oracle Notification Service (ONS) ons, eons grid owner
Oracle Agent oraagent grid owner
Oracle Root Agent orarootagent root
Grid Naming Service (GNS) gnsd root
Grid Plug and Play (GPnP)
Multicast domain name service (mDNS)
gpnpd
mdnsd
grid owner
grid owner
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Oracle Clusterware Process Architecture
e A
c l
Oracle Clusterware comprises several processes that facilitate cluster operations. The Cluster
Ready Service (CRS), Cluster Synchronization Service (CSS), Event Management (EVM), and
r a
Oracle Notification Service (ONS) components communicate with other cluster component

O ly
layers in the same cluster database environment. These components are also the main
communication links between Oracle Database, applications, and the Oracle Clusterware high
& On
availability components. In addition, these background processes monitor and manage database
l
n a e
operations. The following list describes some major Oracle Clusterware background processes.
The list includes components that are processes on Linux and UNIX, or services on Windows.

t e r U s
Cluster Ready Service (CRS): Is the primary program for managing high availability
operations in a cluster. The CRS process manages two types of CRS resources:

I n
- Cluster resources: A cluster resource is an Oracle Clusterware resource. Cluster
resources are viewed, added, modified, or removed using the crsctl command.
l e
- Local resources: A local resource runs on every node in the cluster (no failover) and
c can be, for example, a listener, ASM, a disk group, or Oracle Notification Service

r a (ONS).

O The CRS daemon (crsd) manages cluster resources based on configuration information
that is stored in Oracle Cluster Registry (OCR) for each resource. This includes start, stop,
monitor, and failover operations. The crsd process generates events when the status of a
resource changes.

Oracle Database 11g: RAC Administration F - 11


Oracle Clusterware Process Architecture (continued)
When you have Oracle RAC installed, the crsd process monitors the Oracle database
instance, listener, and so on, and automatically restarts these components when a failure
occurs. When a CRS resource fails, the CRS daemon attempts to restart it, if the resource is
so configured. CRS fails the resource over to another node (again, if it is configured to do
so) after exhausting restart attempts.
Cluster Synchronization Service (CSS): Manages the cluster configuration by
controlling which nodes are members of the cluster and by notifying members when a
node joins or leaves the cluster. If you are using certified third-party clusterware, then CSS
processes interfaces with your clusterware to manage node membership information. CSS
has three separate processes: the CSS daemon (ocssd), the CSS Agent (cssdagent),
and the CSS Monitor (cssdmonitor). The cssdagent process monitors the cluster
and provides input/output fencing. This service formerly was provided by Oracle Process
Monitor daemon (oprocd), also known as OraFenceService on Windows. A
cssdagent failure results in Oracle Clusterware restarting the node.
Disk Monitor daemon (diskmon): Monitors and performs input/output fencing for
Oracle Exadata Storage Server. As Exadata storage can be added to any Oracle RAC node
at any point in time, the diskmon daemon is always started when ocssd is started.
Event Manager (EVM): Is a background process that publishes Oracle Clusterware
events
Multicast domain name service (mDNS): Allows DNS requests. The mDNS process is a
background process on Linux and UNIX, and a service on Windows.
m y
Oracle Grid Naming Service (GNS): Is a gateway between the cluster mDNS and
d e
external DNS servers. The GNS process performs name resolution within the cluster.

c a
Oracle Notification Service (ONS): Is a publish-and-subscribe service for communicating
Fast Application Notification (FAN) events
A
oraagent: Extends clusterware to support Oracle-specific requirements and complex
e
c l
resources. It runs server callout scripts when FAN events occur. This process was known
as RACG in Oracle Clusterware 11g Release 1 (11.1).
r a
Oracle root agent (orarootagent): Is a specialized oraagent process that helps

O ly
CRSD manage resources owned by root, such as the network, and the Grid virtual IP
address

l & On
Cluster kill daemon (oclskd): Handles instance/node evictions requests that have been
escalated to CSS
n a e
t e r
Grid IPC daemon (gipcd): Is a helper daemon for the communications infrastructure

U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration F - 12
Grid Plug and Play
In previous releases, adding or removing servers in a
cluster required extensive manual preparation.
In Oracle Database 11g Release 2, GPnP allows each
node to perform the following tasks dynamically:
Negotiating appropriate network identities for itself
Acquiring additional information from a configuration profile
Configuring or reconfiguring itself using profile data, making
host names and addresses resolvable on the network
To add a node, simply connect the server to the cluster
and allow the cluster to configure the node.

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Grid Plug and Play
e A
c l
With past releases, adding or removing servers in a cluster required extensive manual
preparation. With the release of Oracle Database 11g Release 2, Grid Plug and Play (GPnP)

r a
reduces the costs of installing, configuring, and managing server nodes by using a Grid Naming
Service within the cluster to allow each node to perform the following tasks dynamically:
O ly
Negotiating appropriate network identities for itself

l & On
Acquiring additional information it needs to operate from a configuration profile
Configuring or reconfiguring itself using profile data, making host names and addresses
resolvable on the network
n a e
t e r U s
As servers perform these tasks dynamically, adding and removing nodes simply requires an
administrator to connect the server to the cluster and to allow the cluster to configure the node.

I n
Using Grid Plug and Play, and using best practices recommendations, adding a node to the
database cluster is part of the normal server restart, and removing a node happens when a server

l e
is turned off. This removes many manual operations, reduces opportunity for error, and

c
encourages configurations that can be changed more easily than those requiring fixed per-node

r a
configuration.
The best case uses ASM and Automatic Undo Management so there is no particular policy
O
decision to make if an undo tablespace needs to be allocated for a newly identified database
instance.

Oracle Database 11g: RAC Administration F - 13


GPnP Domain

The GPnP domain is a collection of nodes belonging to a


single cluster served by the GPnP service:
Cluster name: cluster01
Network domain: example.com
GPnP domain: cluster01.example.com
Each node participating in a GPnP domain has the
following characteristics:
Must have at least one routable interface with connectivity
outside of the GPnP domain for the public interface
A unique identifier that is unique within the GPnP domain
A personality affected by the GPnP profile, physical
characteristics, and software image of the node

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
GPnP Domain
e A
c l
The GPnP domain is a collection of nodes served by the GPnP service. In most cases, the size of
the domain is limited to the domain served by the multicast domain. The nodes in a multicast
r a
domain implicitly know which GPnP domain they are in by the public key of their provisioning

O ly
authority. A GPnP domain is defined by the nodes in a multicast domain that recognize a
particular provisioning authority, as determined by the certificate in their software images. A

l & On
GPnP node refers to a computer participating in a GPnP domain. It must have the following:

n a e
IP Connectivity: The node must have at least one routable interface with connectivity outside of
the GPnP domain. If the node has varying connectivity (multiple interfaces, subnets, and so on),

t e r U s
the required binding (public, private, storage) needs to be identified in the GPnP profile. The
physical connectivity is controlled by some outside entity and is not modified by GPnP.

I n
Unique identifier: Associated with a node is a unique identifier, established through OSD. The
ID is required to be unique within the GPnP domain, but is usefully unique globally within the
l e
largest plausible domain.
c
Personality: Characteristics of the node personality include:

r a
Cluster name

O Network classifications (public/private)


Storage to be used for ASM and CSS
Digital signatures
The software image for the node, including the applications that may be 14

Oracle Database 11g: RAC Administration F - 14


GPnP Components

Software image
A software image is a read-only collection of software to be
run on nodes of the same type.
At a minimum, the image must contain:
An operating system
The GPnP software
A security certificate from the provisioning authority
Other software required to configure the node when it starts
up

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
GPnP Components
e A
l
A software image is a read-only collection of software to be run on nodes of the same type. At a
c
r a
minimum, it must contain an operating system, the GPnP software, the security certificate of the
provisioning authority, and other software required to configure the node when it starts up. In the
O ly
fullest form, the image may contain all application software to be run on the node, including the

l & On
Oracle Database, iAS, and customer applications. Under GPnP, an image need not be specific to
a particular node and may be deployed to as many nodes as needed.
a e
An image is created by a provisioning authority through means not defined by GPnP. It may
n
node-dependent data.
t e r U s
involve doing an installation on an example machine, then scraping the storage to remove

I n
An image is distributed to a node by the provisioning authority in ways outside the scope of

mechanism.
c l e
GPnPit may be a system administrator carrying CDs around, a network boot, or any other

r a
O
Oracle Database 11g: RAC Administration F - 15
GPnP Profile
The profile.xml file:
$ cat GRID_HOME/gpnp/profiles/peer/profiles.xml
<?xml version="1.0" encoding="UTF-8"?><gpnp:GPnP-Profile Version="1.0"
xmlns="http://www.grid-pnp.org/2005/11/gpnp-profile" ...
xsi:schemaLocation="http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd"
ProfileSequence="4" ClusterUId="2deb88730e0b5f1bffc9682556bd548e" ClusterName="cluster01"
PALocation=""><gpnp:Network-Profile><gpnp:HostNetwork id="gen" HostName="*"><gpnp:Network
id="net1" IP="192.0.2.0" Adapter="eth0" Use="public"/><gpnp:Network id="net2"
IP="192.168.1.0" Adapter="eth1"
Use="cluster_interconnect"/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile
id="css" DiscoveryString="+asm" LeaseDuration="400"/><orcl:ASM-Profile id="asm"
DiscoveryString="/dev/sd*" SPFile="+data/spfile.ora"/><ds:Signature
<ds:SignedInfo>
<ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#"/>
<ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
<ds:Reference URI="">
<ds:Transforms>
<ds:Transform Algorithm="http://www.w3.org/2000/09/xmldsig#enveloped-signature"/>
<ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
<InclusiveNamespaces xmlns="http://www.w3.org/2001/10/xml-exc-c14n#"
PrefixList="gpnp orcl xsi"/>
...
<ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
<ds:DigestValue>gIBakmtUNi9EVW/XQoE1mym3Bnw=</ds:DigestValue>
...

m y
<ds:SignatureValue>cgw3yhP/2oEm5DJzdachtfDMbEr2RSfFFUlZujLemnOgsM...=</ds:SignatureValue>

d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
GPnP Profile
e A
l
A GPnP profile is a small XML file that is used to establish the correct global personality for
c
r a
otherwise identical images. A profile may be obtained at boot time from a remote location, or it
may be burned permanently in an image. It contains generic GPnP configuration information
O ly
and may also contain application-specific configuration data. The generic GPnP configuration

l & On
data is used to configure the network environment and the storage configuration during bootup.
Application parts of the profile may be used to establish application-specific personalities. The
a e
profile is security sensitive. It might identify the storage to be used as the root partition of a
n
t e r
machine. A profile must be digitally signed by the provisioning authority, and must be validated
s
on each node before it is used. Therefore, the image must contain the public certificate of the
U
I n
authority to be used for this validation. Profiles are intended to be difficult to change. They may
be burned into read-only images or replicated to machines that are down when a change is made.

l e
Profile attributes defining node personality include:
c
Cluster name

r a
Network classifications (public/private)
Storage to be used for ASM and CSS
O Digital signature information
Changes made to a cluster and therefore the profile are replicated by the gpnpd daemon during
installation, system boot, or when updated. These updates can be triggered by making changes
with configuration tools such as oifcfg, crsctl, asmca, and so on.
Oracle Database 11g: RAC Administration F - 16
Grid Naming Service

GNS is an integral component of Grid Plug and Play.


The only static IP address required for the cluster is the GNS
virtual IP address.
The cluster subdomain is defined as a delegated domain.
[root@my-dns-server ~]# cat /etc/named.conf
// Default initial "Caching Only" name server configuration
...
# Delegate to gns on cluster01
cluster01.example.com #cluster sub-domain# NS cluster01-
gns.example.com
# Let the world know to go to the GNS vip
cluster01-gns.example.com 192.0.2.155 # cluster GNS Address
A request to resolve cluster01-scan.cluster01.example.com
would be forwarded to the GNS on 192.0.2.155.
Each node in the cluster runs a multicast DNS (mDNS)
process. m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Grid Naming Service
e A
l
Employing Grid Naming Service (GNS) assumes that there is a DHCP server running on the
c
r a
public network with enough addresses to assign to the VIPs and single client access name
(SCAN) VIPs. With GNS, only one static IP address is required for the cluster, the GNS virtual
O ly
IP address. This address should be defined in the DNS domain. GNS sets up a multicast DNS

l & On
(mDNS) server within the cluster, which resolves names in the cluster without static
configuration of the DNS server for other node IP addresses.
a e
The mDNS server works as follows: Within GNS, node names are resolved using link-local
n
t e r U s
multicast name resolution (LLMNR). It does this by translating the LLMNR .local domain
used by the multicast resolution to the subdomain specified in the DNS query. When you select

I n
GNS, an mDNS server is configured on each host in the cluster. LLMNR relies on the mDNS
that Oracle Clusterware manages to resolve names that are being served by that host.
l e
To use GNS, before installation, the DNS administrator must establish domain delegation to the
c
r a
subdomain for the cluster. Queries to the cluster are sent to the GNS listener on the GNS virtual
IP address. When a request comes to the domain, GNS resolves it using its internal mDNS and
O
responds to the query.

Oracle Database 11g: RAC Administration F - 17


Single Client Access Name
The single client access name (SCAN) is the address used
by clients connecting to the cluster.
The SCAN is a fully qualified host name located in the GNS
subdomain registered to three IP addresses.
# dig @192.0.2.155 cluster01-scan.cluster01.example.com
...
;; QUESTION SECTION:
;cluster01-scan.cluster01.example.com. IN A
;; ANSWER SECTION:
cluster01-scan.cluster01.example.com. 120 IN A 192.0.2.244
cluster01-scan.cluster01.example.com. 120 IN A 192.0.2.246
cluster01-scan.cluster01.example.com. 120 IN A 192.0.2.245
;; AUTHORITY SECTION:
cluster01.example.com. 10800 IN A 192.0.2.155
;; SERVER: 192.0.2.155#53(192.0.2.155)

The SCAN provides a stable, highly available name for


clients to use, independent of the nodes that make up the
cluster. m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Single Client Access Name
e A
l
The single client access name (SCAN) is the address used by clients connecting to the cluster.
c
r a
The SCAN is a fully qualified host name (host name + domain) registered to three IP addresses.
If you use GNS, and you have DHCP support, then the GNS will assign addresses dynamically
to the SCAN.
O ly
l & On
If you do not use GNS, the SCAN should be defined in the DNS to resolve to the three addresses
assigned to that name. This should be done before you install Oracle Grid Infrastructure. The
a e
SCAN and its associated IP addresses provide a stable name for clients to use for connections,
n
t e r U s
independent of the nodes that make up the cluster.
SCANs function like a cluster alias. However, SCANs are resolved on any node in the cluster, so

I n
unlike a VIP address for a node, clients connecting to the SCAN no longer require updated VIP

l e
addresses as nodes are added to or removed from the cluster. Because the SCAN addresses
resolve to the cluster, rather than to a node address in the cluster, nodes can be added to or
c
r a
removed from the cluster without affecting the SCAN address configuration.
During installation, listeners are created on each node for the SCAN IP addresses. Oracle
O
Clusterware routes application requests to the cluster SCAN to the least loaded instance
providing the service.

Oracle Database 11g: RAC Administration F - 18


Single Client Access Name (continued)
SCAN listeners can run on any node in the cluster. SCANs provide location independence for
the databases so that client configuration does not have to depend on which nodes run a
particular database.
Oracle Database 11g Release 2 and later instances register with SCAN listeners only as remote
listeners. Upgraded databases register with SCAN listeners as remote listeners, and also continue
to register with all other listeners.
If you specify a GNS domain during installation, the SCAN defaults to clustername-
scan.GNS_domain. If a GNS domain is not specified at installation, the SCAN defaults to
clustername-scan.current_domain.
Note: dig: Domain Information Groper

m y
d e
c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration F - 19
GPnP Architecture Overview
VIP resolution Host name to address resolution

VIP name and hostname resolution


DNS GNS
Gets the three SCAN VIPS

Register node VIP & SCAN VIP


GNS Static VIP
One name that
Use 1 SCAN VIP

resolves to three VIPs


Load balacing

SL SL SL
remote_listener

LL LL LL LL LL
Client local_listener
Scan+port+service Node1 Node2 Node3 Node4 Noden
Get Node VIP
Least Node &
profile.xml

& SCAN VIP



obtained by orarootagent
Dynamic VIP addresses

loaded SCAN
node VIP
for agents
service orarootagent
DHC
P Profile
GPnP GPnP GPnP GPnP GPnP replication
GPnPd discovery

mDNS mDNS mDNS mDNS mDNS


m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
GPnP Architecture Overview
e A
GPnP Service
c
The GPnP service is collectively provided by all the GPnP agents. It is a distributed method ofl
r a
replicating profiles. The service is instantiated on each node in the domain as a GPnP agent. The

O ly
service is peer-to-peer, there is no master process. This allows high availability because any
GPnP agent can crash and new nodes will still be serviced. GPnP requires standard IP multicast

l & On
protocol (provided by mDNS), to locate peer services. Using multicast discovery, GPnP locates

a e
peers without configuration. This is how a GPnP agent on a new node locates another agent that
n
may have a profile it should use.
Name Resolution
t e r U s
A name defined within a GPnP domain is resolvable in the following cases:
I n
Hosts inside the GPnP domain use normal DNS to resolve the names of hosts outside of the

l e
GPnP domain. They contact their regular DNS service and proceed. They may get the

c
address of their DNS server by global configuration, or by having been told by DHCP.

r a
Within the GPnP domain, host names are resolved using mDNS. This requires an mDNS
responder on each node that knows the names and addresses used by this node, and
O operating system client library support for name resolution using this multicast protocol.
Given a name, a client executes gethostbyname, resulting in an mDNS query. If the
name exists, the responder on the node that owns the name will respond with the IP
address. The client software may cache the resolution for the given time-to-live value.

Oracle Database 11g: RAC Administration F - 20


GPnP Architecture Overview (continued)
Machines outside the GPnP domain cannot resolve names in the GPnP domain by using
multicast. To resolve these names, they use their regular DNS. The provisioning authority
arranges the global DNS to delegate a sub-domain (zone) to a known address that is in the
GPnP domain. GPnP creates a service called called GNS to resolve the GPnP names on
that fixed address.
The node on which the GNS server is running listens for DNS requests. On receipt, they
translate and forward to mDNS, collect responses, translate, and send back to the outside
client. GNS is virtual because it is stateless. Any node in the multicast domain may host
the server. The only GNS configuration is global:
- The address on which to listen on standard DNS port 53;
- The name(s) of the domains to serviced.
There may be as many GNS entities as needed for availability reasons. Oracle-provided
GNS may use CRS to ensure availability of a single GNS provider.
SCAN and Local Listeners
When a client submits a connection request, the SCAN listener listening on a SCAN IP address
and the SCAN port is contracted on the clients behalf. Because all services on the cluster are
registered with the SCAN listener, the SCAN listener replies with the address of the local
listener on the least-loaded node where the service is currently being offered. Finally, the client
establishes connection to the service through the listener on the node where service is offered.

required in the client. m y


All of these actions take place transparently to the client without any explicit configuration

d e
During installation, listeners are created on nodes for the SCAN IP addresses. Oracle Net

c a
Services routes application requests to the least loaded instance providing the service. Because
the SCAN addresses resolve to the cluster, rather than to a node address in the cluster, nodes can
A
be added to or removed from the cluster without affecting the SCAN address configuration.
e
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration F - 21
How GPnP Works: Cluster Node Startup

1. IP addresses are negotiated for public interfaces using


DHCP:
VIPs
SCAN VIPs
2. A GPnP agent is started from the nodes Clusterware
home.
3. The GPnP agent either gets its profile locally or from one
of the peer GPnP agents that responds.
4. Shared storage is configured to match profile
requirements.
5. Service startup is specified in the profile, which includes:
Grid Naming Service for external names resolution
Single client access name (SCAN) listener
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
How GPnP Works: Cluster Node Startup
e A
l
When a node is started in a GPnP environment:
c
r a
Network addresses are negotiated for all interfaces using DHCP
The Clusterware software on the starting node starts a GPnP agent
O ly
The GPnP agent on the starting node gets its profile locally or uses resource discovery

l & On
(RD) to discover the peer GPnP agents in the grid. If RD is used, it gets the profile from
one of the GPnP peers that responds.
a e
The GPnP agent acquires the desired network configuration from the profile. This includes
n
t e r
creation of reasonable host names. If there are static configurations, they are used in
s
preference to the dynamic mechanisms. Network interfaces may be reconfigured to match
U
I n
the profile requirements.
Shared storage is configured to match the profile requirements

l e
System and service startup is done as configured in the image. In the case of RAC, the CSS
c
and CRS system will then be started, which will form the cluster and bring up appropriate

r a
database instances. The startup of services may run down their own placeholder values, or
may dynamically negotiate values rather than rely on fixed-up configurations. One of the
O services likely to be started somewhere is the GNS system for external name resolution.
Another of the services likely to be started is an Oracle SCAN listener.

Oracle Database 11g: RAC Administration F - 22


How GPnP Works: Client Database Connections

GNS
DNS

Database
client
SCAN Listener1
listener

Listener2
SCAN
listener

Listener3
SCAN
listener
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
How GPnP Works: Client Database Connections
e A
l
In a GPnP environment, the database client no longer has to use the TNS address to contact the
c
r a
listener on a target node. Instead, it can use the EZConnect method to connect to the database.
When resolving the address listed in the connect string, the DNS will forward the resolution
O ly
request to the GNS with the SCAN VIP address for the chosen SCAN listener and the name of

l & On
the database service that is desired. In EZconnect syntax, this would look like:
scan-name.cluster-name.company.com/ServiceName, where the service name might
a e
be the database name. The GNS will respond to the DNS server with the IP address matching the
n
t e r U s
name given; this address is then used by the client to contact the SCAN listener. The SCAN
listener uses its connection load balancing system to pick an appropriate listener, whose name it

I n
returns to the client in an OracleNet Redirect message. The client reconnects to the selected
listener, resolving the name through a call to the GNS.
l e
The SCAN listeners must be known to all the database listener nodes and clients. The database
c
r a
instance nodes cross-register only with the known SCAN listeners, also sending them per-
service connection metrics. The SCAN known to the database servers may be profile data or
O
stored in OCR.

Oracle Database 11g: RAC Administration F - 23


Summary

In this lesson, you should have learned how to:


Explain the principles and purposes of clusters
Describe the Oracle Clusterware architecture
Describe how Grid Plug and Play affects Clusterware

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration F - 24
RAC Concepts

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Objectives

After completing this lesson, you should be able to:


Explain the necessity of global resources
Describe global cache coordination
Explain object affinity and dynamic remastering

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration G - 2
Benefits of Using RAC

High availability: Surviving node and instance failures


Scalability: Adding more nodes as you need them in the
future
Pay as you grow: Paying for only what you need today
Key grid computing features:
Growth and shrinkage on demand
Single-button addition of servers
Automatic workload management for services

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Benefits of Using RAC
e A
l
Oracle Real Application Clusters (RAC) enables high utilization of a cluster of standard, low-
c
cost modular servers such as blades.
r a
RAC offers automatic workload management for services. Services are groups or classifications
O ly
of applications that comprise business components corresponding to application workloads.

l & On
Services in RAC enable continuous, uninterrupted database operations and provide support for
multiple services on multiple instances. You assign services to run on one or more instances, and
a e
alternate instances can serve as backup instances. If a primary instance fails, the Oracle server
n
t e r U s
moves the services from the failed instance to a surviving alternate instance. The Oracle server
also automatically load-balances connections across instances hosting a service.

I n
RAC harnesses the power of multiple low-cost computers to serve as a single large computer for

l e
database processing, and provides the only viable alternative to large-scale symmetric
multiprocessing (SMP) for all types of applications.
c
r a
RAC, which is based on a shared-disk architecture, can grow and shrink on demand without the
need to artificially partition data among the servers of your cluster. RAC also offers a single-
O
button addition of servers to a cluster. Thus, you can easily provide or remove a server to or
from the database.

Oracle Database 11g: RAC Administration G - 3


Clusters and Scalability

SMP model RAC model

Shared
Memory
storage

Cache Cache SGA SGA

CPU CPU CPU CPU BGP BGP BGP BGP

Cache coherency Cache fusion

BGP (background process)


m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Clusters and Scalability
e A
l
If your application scales transparently on SMP machines, then it is realistic to expect it to scale
c
r a
well on RAC, without having to make any changes to the application code.
RAC eliminates the database instance, and the node itself, as a single point of failure, and
O ly
ensures database integrity in the case of such failures.

& On
Following are some scalability examples:
l
n a e
Allow more simultaneous batch processes.
Allow larger degrees of parallelism and more parallel executions to occur.

(OLTP) systems.
t e r U s
Allow large increases in the number of connected users in online transaction processing

I n
c l e
r a
O
Oracle Database 11g: RAC Administration G - 4
Levels of Scalability

Hardware: Disk input/output (I/O)


Internode communication: High bandwidth and low latency
Operating system: Number of CPUs
Database management system: Synchronization
Application: Design

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Levels of Scalability
e A
l
Successful implementation of cluster databases requires optimal scalability on four levels:
c
r a
Hardware scalability: Interconnectivity is the key to hardware scalability, which greatly
depends on high bandwidth and low latency.
O ly
Operating system scalability: Methods of synchronization in the operating system can

l & On
determine the scalability of the system. In some cases, potential scalability of the hardware
is lost because of the operating systems inability to handle multiple resource requests
simultaneously.
n a e
t e r
Database management system scalability: A key factor in parallel architectures is
s
whether the parallelism is affected internally or by external processes. The answer to this
U
I n
question affects the synchronization mechanism.
Application scalability: Applications must be specifically designed to be scalable. A

l e
bottleneck occurs in systems in which every session is updating the same data most of the
c
time. Note that this is not RAC specific and is true on single-instance system too.

r a
It is important to remember that if any of the areas above are not scalable (no matter how

O
scalable the other areas are), then parallel cluster processing may not be successful. A typical
cause for the lack of scalability is one common shared resource that must be accessed often. This
causes the otherwise parallel operations to serialize on this bottleneck. A high latency in the
synchronization increases the cost of synchronization, thereby counteracting the benefits of
parallelization. This is a general limitation and not a RAC-specific limitation.
Oracle Database 11g: RAC Administration G - 5
Scaleup and Speedup

Original system

Hardware Time 100% of task

Cluster system scaleup Cluster system speedup

Hardware Time Up to
200% Hardware
of Up to
100%
task 300%
Hardware of task
Time of
Hardware
task Time/2
Hardware
Time
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Scaleup and Speedup
e A
l
Scaleup is the ability to sustain the same performance levels (response time) when both
c
r a
workload and resources increase proportionally:
Scaleup = (volume parallel) / (volume original)
O ly
For example, if 30 users consume close to 100 percent of the CPU during normal processing,
& On
then adding more users would cause the system to slow down due to contention for limited CPU
l
n a e
cycles. However, by adding CPUs, you can support extra users without degrading performance.
Speedup is the effect of applying an increasing number of resources to a fixed amount of work to

e r s
achieve a proportional reduction in execution times:
t U
I n
Speedup = (time original) / (time parallel)
Speedup results in resource availability for other tasks. For example, if queries usually take ten
l e
minutes to process and running in parallel reduces the time to five minutes, then additional
c
queries can run without introducing the contention that might occur were they to run

r a
concurrently.

O
Oracle Database 11g: RAC Administration G - 6
Speedup/Scaleup and Workloads

Workload Speedup Scaleup

OLTP and Internet No Yes

DSS with parallel query Yes Yes

Batch (mixed) Possible Yes

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Speedup/Scaleup and Workloads
e A
l
The type of workload determines whether scaleup or speedup capabilities can be achieved using
c
parallel processing.
r a
Online transaction processing (OLTP) and Internet application environments are characterized
O ly
by short transactions that cannot be further broken down and, therefore, no speedup can be

l & On
achieved. However, by deploying greater amounts of resources, a larger volume of transactions
can be supported without compromising the response.

n a e
Decision support systems (DSS) and parallel query options can attain speedup, as well as

t e r U s
scaleup, because they essentially support large tasks without conflicting demands on resources.
The parallel query capability within the Oracle database can also be leveraged to decrease

I n
overall processing time of long-running queries and to increase the number of such queries that

c l e
can be run concurrently.
In an environment with a mixed workload of DSS, OLTP, and reporting applications, scaleup

r a
can be achieved by running different programs on different hardware. Speedup is possible in a
batch environment, but may involve rewriting programs to use the parallel processing
O
capabilities.

Oracle Database 11g: RAC Administration G - 7


I/O Throughput Balanced: Example

Each machine has 2 CPUs:


2 200 MB/s 4 = 1600 MB/s

Each machine has 2 HBAs:


HBA1
HBA2

HBA1
HBA2

HBA1
HBA2

HBA1
HBA2
8 200 MB/s = 1600 MB/s

Each switch needs to support 800 MB/s


FC-switch to guarantee a total system throughput
of 1600 MB/s.

Each disk array


Disk Disk Disk Disk Disk Disk Disk Disk has one 2-Gbit
array 1 array 2 array 3 array 4 array 5 array 6 array 7 array 8 controller:
8 200 MB/s =
1600 MB/s

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
I/O Throughput Balanced: Example
e A
l
To make sure that a system delivers the IO demand that is required, all system components on
c
r
The weakest link determines the IO throughput. a
the IO path need to be orchestrated to work together.

O ly
On the left, you see a high-level picture of a system. This is a system with four nodes, two Host
& On
Bus Adapters (HBAs) per node, two fibre channel switches, which are attached to four disk
l
n a e
arrays each. The components on the IO path are the HBAs, cables, switches, and disk arrays.
Performance depends on the number and speed of the HBAs, switch speed, controller quantity,

t e r U s
and speed of disks. If any one of these components is undersized, the system throughput is
determined by this component. Assuming you have a 2-GBit HBA, the nodes can read about 8

I n
200 MB/s = 1.6 GBytes/s. However, assuming that each disk array has one controller, all 8

l e
arrays can also do 8 200 MB/s = 1.6 GBytes/s. Therefore, each of the fibre channel switches
also need to deliver at least 2 GBit/s per port, to a total of 800 MB/s throughput. The two
c
r a
switches will then deliver the needed 1.6 GBytes/s.
Note: When sizing a system, also take the system limits into consideration. For instance, the
O
number of bus slots per node is limited and may need to be shared between HBAs and network
cards. In some cases, dual port cards exist if the number of slots is exhausted. The number of
HBAs per node determines the maximal number of fibre channel switches. And the total number
of ports on a switch limits the number of HBAs and disk controllers.

Oracle Database 11g: RAC Administration G - 8


Performance of Typical Components

Throughput Performance
Component Theory (Bit/s) Maximal Byte/s
HBA Gbit/s 100/200 Mbytes/s
16 Port Switch 8 2 Gbit/s 1600 Mbytes/s
Fibre Channel 2 Gbit/s 200 Mbytes/s
Disk Controller 2 Gbit/s 200 Mbytes/s
GigE NIC 1 Gbit/s 80 Mbytes/s
Infiniband 10 Gbit/s 890 Mbytes/s
CPU 200250 MB/s

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Performance of Typical Components
e A
l
While discussing, people often confuse bits with bytes. This confusion originates mainly from
c
r a
the fact that hardware vendors tend to describe components performance in bits/s whereas
database vendors and customers describe their performance requirements in bytes/s.
O ly
The following is a list of common hardware components with their theoretical performance in

l & On
bits/second and typical performance in bytes/second:
HBAs come in 1 or 2 GBit per second with a typical throughput of 100 or 200 MB/s.
a e
A 16 Port Switch comes with sixteen 2-GBit ports. However, the total throughput is 8
n
t e r U s
times 2 Gbit, which results in 1600 Mbytes/s.
Fibre Channel cables have a 2-GBit/s throughput, which translates into 200 MB/s.

I n
Disk Controllers come in 2-GBit/s throughput, which translates into about 200 MB/s.
GigE has a typical performance of about 80 MB/s whereas Infiniband delivers about
l
160 MB/s.
c e
r a
O
Oracle Database 11g: RAC Administration G - 9
Necessity of Global Resources

SGA1 SGA2 SGA1 SGA2


1008

1008 1008
1 2

SGA1 SGA2 SGA1 SGA2


1009 1008 1009

Lost
updates!
1008
4
1008
3
m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Necessity of Global Resources
e A
l
In single-instance environments, locking coordinates access to a common resource such as a row
c
time.
r a
in a table. Locking prevents two processes from changing the same resource (or row) at the same

O ly
In RAC environments, internode synchronization is critical because it maintains proper

l & On
coordination between processes on different nodes, preventing them from changing the same
resource at the same time. Internode synchronization guarantees that each instance sees the most
a e
recent version of a block in its buffer cache.
n
prohibits this problem.
t e r U s
Note: The slide shows you what would happen in the absence of cache coordination. RAC

I n
c l e
r a
O
Oracle Database 11g: RAC Administration G - 10
Global Resources Coordination

Cluster
Node1 Noden
Instance1 Instancen
GRD Master Cache GRD Master Cache
Global
LMON GES resources GES LMON
LMD0 LMD0
LMSx GCS GCS LMSx
LCK0 Interconnect LCK0
DIAG DIAG

Global Resource Directory (GRD)

Global Cache Services (GCS) Global Enqueue Services (GES)

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Global Resources Coordination
e A
l
Cluster operations require synchronization among all instances to control shared access to
c
r a
resources. RAC uses the Global Resource Directory (GRD) to record information about how
resources are used within a cluster database. The Global Cache Services (GCS) and Global
O ly
Enqueue Services (GES) manage the information in the GRD.

l & On
Each instance maintains a part of the GRD in its System Global Area (SGA). The GCS and GES
nominate one instance to manage all information about a particular resource. This instance is
a e
called the resource master. Also, each instance knows which instance masters which resource.
n
t e r U s
Maintaining cache coherency is an important part of a RAC activity. Cache coherency is the
technique of keeping multiple copies of a block consistent between different Oracle instances.

I n
GCS implements cache coherency by using what is called the Cache Fusion algorithm.

c l e
The GES manages all nonCache Fusion interinstance resource operations and tracks the status
of all Oracle enqueuing mechanisms. The primary resources of the GES controls are dictionary

r a
cache locks and library cache locks. The GES also performs deadlock detection to all deadlock-
sensitive enqueues and resources.
O
Oracle Database 11g: RAC Administration G - 11
Global Cache Coordination: Example

Cluster
Node1 Node2
Instance1 Instance2
Cache 1009 1009 Cache
3
LMON LMON
LMD0 LMD0
LMSx 4 LMSx
LCK0 LCK0
DIAG DIAG
2 1
Instance 2 has
Block mastered the current version of the block. Which instance
by instance 1 GCS masters the block?

1008 No disk I/O


m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Global Cache Coordination: Example
e A
l
The scenario described in the slide assumes that the data block has been changed, or dirtied, by
c
the block is represented by its SCN.
r a
the first instance. Furthermore, only one copy of the block exists clusterwide, and the content of

O ly
1. The second instance attempting to modify the block submits a request to the GCS.

l & On
2. The GCS transmits the request to the holder. In this case, the first instance is the holder.
3. The first instance receives the message and sends the block to the second instance. The
a e
first instance retains the dirty buffer for recovery purposes. This dirty image of the block is
n
t e r
also called a past image of the block. A past image block cannot be modified further.
s
4. On receipt of the block, the second instance informs the GCS that it holds the block.
U
n
Note: The data block is not written to disk before the resource is granted to the second instance.
I
c l e
r a
O
Oracle Database 11g: RAC Administration G - 12
Write to Disk Coordination: Example

Cluster
Node1 Node2
Instance1 Instance2
Cache 1009 1010 Cache
3
LMON LMON
LMD0 LMD0
LMSx LMSx
LCK0 5 4 LCK0
DIAG DIAG
1 2
Block flushed, make room
Need to make room
Instance 2 owns it.
in my cache.
GCS Instance 2, flush the block
Who has the current version
to disk.
of that block?

Only one
1010
disk I/O

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Write to Disk Coordination: Example
e A
l
The scenario described in the slide illustrates how an instance can perform a checkpoint at any
c
r a
time or replace buffers in the cache as a response to free buffer requests. Because multiple
versions of the same data block with different changes can exist in the caches of instances in the
O ly
cluster, a write protocol managed by the GCS ensures that only the most current version of the

l & On
data is written to disk. It must also ensure that all previous versions are purged from the other
caches. A write request for a data block can originate in any instance that has the current or past
a e
image of the block. In this scenario, assume that the first instance holding a past image buffer
n
t e r
requests that the Oracle server writes the buffer to disk:
s
1. The first instance sends a write request to the GCS.
U
I n
2. The GCS forwards the request to the second instance, which is the holder of the current
version of the block.

l e
3. The second instance receives the write request and writes the block to disk.
c
4. The second instance records the completion of the write operation with the GCS.

r a
5. After receipt of the notification, the GCS orders all past image holders to discard their past
images. These past images are no longer needed for recovery.
O
Note: In this case, only one I/O is performed to write the most current version of the block to
disk.

Oracle Database 11g: RAC Administration G - 13


Dynamic Reconfiguration

Reconfiguration remastering

Node1 Node2 Node3


Instance1 Instance2 Instance3

masters granted masters granted masters granted


R1 1, 2, 3 R3 2, 3 R5 2
R2 1, 3 R4 1, 2 R6 1, 2, 3

Node1 Node2 Node3


Instance1 Instance2 Instance3

masters granted masters granted masters granted


R1 1, 3 R3 2, 3 R5
R2 1, 3 R4 1, 2 R6 1, 3
R3 3 R4 1

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Dynamic Reconfiguration
e A
l
When one instance departs the cluster, the GRD portion of that instance needs to be redistributed
c
r a
to the surviving nodes. Similarly, when a new instance enters the cluster, the GRD portions of
the existing instances must be redistributed to create the GRD portion of the new instance.
O ly
Instead of remastering all resources across all nodes, RAC uses an algorithm called lazy

l & On
remastering to remaster only a minimal number of resources during a reconfiguration. This is
illustrated on the slide. For each instance, a subset of the GRD being mastered is shown along
a e
with the names of the instances to which the resources are currently granted. When the second
n
t e r U s
instance fails, its resources are remastered on the surviving instances. As the resources are
remastered, they are cleared of any reference to the failed instance.

I n
c l e
r a
O
Oracle Database 11g: RAC Administration G - 14
Object Affinity and Dynamic Remastering
Messages are sent to remote node when reading into cache.
Node1 Node2
Instance1 Before

GCS message to master dynamic

remastering

Instance2

Object
Read from
disk

Node2


After
dynamic
remastering

y
Instance1
Instance2
Node1
No messages are sent to remote node when reading into cache.

e m
Copyright 2010, Oracle and/or its affiliates. All rights reserved.
a d
Object Affinity and Dynamic Remastering A c
l e
In addition to dynamic resource reconfiguration, the GCS, which is tightly integrated with the
c
r a
buffer cache, enables the database to automatically adapt and migrate resources in the GRD.
This is called dynamic remastering. The basic idea is to master a buffer cache resource on the
O ly
instance where it is mostly accessed. In order to determine whether dynamic remastering is

l & On
necessary, the GCS essentially keeps track of the number of GCS requests on a per-instance and
per-object basis. This means that if an instance, compared to another, is heavily accessing blocks
a e
from the same object, the GCS can take the decision to dynamically migrate all of that objects
n
t e r
resources to the instance that is accessing the object most.

U s
The upper part of the graphic shows you the situation where the same object has master

I n
resources spread over different instances. In that case, each time an instance needs to read a
block from that object whose master is on the other instance, the reading instance must send a
l e
message to the resources master to ask permission to use the block.
c
r a
The lower part of the graphic shows you the situation after dynamic remastering occurred. In
this case, blocks from the object have affinity to the reading instance which no longer needs to
O
send GCS messages across the interconnect to ask for access permissions.
Note: The system automatically moves mastership of undo segment objects to the instance that
owns the undo segments.

Oracle Database 11g: RAC Administration G - 15


Global Dynamic Performance Views

Retrieve information about all started instances


Have one global view for each local view
Use one parallel slave on each instance

Cluster
Node1 Noden
GV$INSTANCE
Instance1 Instancen

V$INSTANCE V$INSTANCE

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Global Dynamic Performance Views
e A
l
Global dynamic performance views retrieve information about all started instances accessing
c
the local instance only.
r a
one RAC database. In contrast, standard dynamic performance views retrieve information about

O ly
For each of the V$ views available, there is a corresponding GV$ view except for a few

l & On
exceptions. In addition to the V$ information, each GV$ view possesses an additional column
named INST_ID. The INST_ID column displays the instance number from which the
a e
associated V$ view information is obtained. You can query GV$ views from any started
n
instance.

t e r U s
GV$ views use a special form of parallel execution. The parallel execution coordinator runs on

I n
the instance that the client connects to, and one slave is allocated in each instance to query the
underlying V$ view for that instance.

c l e
r a
O
Oracle Database 11g: RAC Administration G - 16
Efficient Internode Row-Level Locking

UPDATE UPDATE
1 2

Node1 Node2 Node1 Node2


Instance1 Instance2 Instance1 Instance2

No block-level
COMMIT lock UPDATE

Node1 Node2 Node1 Node2


Instance1 Instance2 Instance1 Instance2
3

y
4

e m
Copyright 2010, Oracle and/or its affiliates. All rights reserved.
a d
Efficient Internode Row-Level Locking A c
l e
Oracle supports efficient row-level locks. These row-level locks are created when data
c
r a
manipulation language (DML) operations, such as UPDATE, are executed by an application.
These locks are held until the application commits or rolls back the transaction. Any other
O ly
application process will be blocked if it requests a lock on the same row.

l & On
Cache Fusion block transfers operate independently of these user-visible row-level locks. The
transfer of data blocks by the GCS is a low-level process that can occur without waiting for row-
a e
level locks to be released. Blocks may be transferred from one instance to another while row-
n
level locks are held.

t e r U s
GCS provides access to data blocks allowing multiple transactions to proceed in parallel.

I n
c l e
r a
O
Oracle Database 11g: RAC Administration G - 17
Parallel Execution with RAC

Execution slaves have node affinity with the execution


coordinator but will expand if needed.

Node 1 Node 2 Node 3 Node 4

Execution
coordinator

Shared disks Parallel


execution
server

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
Parallel Execution with RAC
e A
l
Oracles cost-based optimizer incorporates parallel execution considerations as a fundamental
c
r a
component in arriving at optimal execution plans.
In a RAC environment, intelligent decisions are made with regard to intranode and internode
O ly
parallelism. For example, if a particular query requires six query processes to complete the work

l & On
and six parallel execution slaves are idle on the local node (the node that the user connected to),
then the query is processed by using only local resources. This demonstrates efficient intranode
a e
parallelism and eliminates the query coordination overhead across multiple nodes. However, if
n
t e r U s
there are only two parallel execution servers available on the local node, then those two and four
of another node are used to process the query. In this manner, both internode and intranode

n
parallelism are used to speed up query operations.
I
l e
In real-world decision support applications, queries are not perfectly partitioned across the
various query servers. Therefore, some parallel execution servers complete their processing and
c
r a
become idle sooner than others. The Oracle parallel execution technology dynamically detects
idle processes and assigns work to these idle processes from the queue tables of the overloaded
O
processes. In this way, the Oracle server efficiently redistributes the query workload across all
processes. Real Application Clusters further extends these efficiencies to clusters by enabling
the redistribution of work across all the parallel execution slaves of a cluster.

Oracle Database 11g: RAC Administration G - 18


Summary

In this lesson, you should have learned how to:


Explain the necessity of global resources
Describe global cache coordination
Explain object affinity and dynamic remastering

m y
d e
Copyright 2010, Oracle and/or its affiliates. All rights reserved.

c a
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O
Oracle Database 11g: RAC Administration G - 19
m y
d e
ca
e A
c l
r a
O ly
l & On
n a e
t e r U s
I n
c l e
r a
O

S-ar putea să vă placă și