5/13/2016

RACCheatsheet

RACCheatsheet
ThisisaquickanddirtycheatsheetonOracleRAC10g,asmyexperiencewithRACgrowsIwillupdatethissection,belowisabeginners
guideonthecommandsandinformationthatyouwillrequiretoadministerOracleRAC.
Acronyms
Acronyms
GCS

Global Cache Services

in memory database containing current locks and awaiting locks, also known as PCM

GES

Global Enqueue Services

coordinates the requests of all global enqueues uses the GCS, also known as nonPCM

GRD

Global Resource Directory

all resources available to the cluster (formed and managed by GCS and GES), see GRD
for more details

GRM

Global Resource Manager

helps to coordinate and communicate the locks requests between Oracle processes

GSD

Global Services Daemon

runs on each node with one GSD process per node. The GSD coordinates with the
cluster manager to receive requests from clients such as the DBCA, EM, and the
SRVCTL utility to execute administrative job tasks such as instance startup or
shutdown. The GSD is not an Oracle instance background process and is therefore not
started with the Oracle instance

PCM (IDLM)

Parallel Cache Management

formly know as (integrated) Distributed Lock Manager, its another name for GCS

Resource

n/a

it is a identifiable entity it basically has a name or a reference, it can be a area in

memory, a disk file or an abstract entity
a resource that can be accessed by all the nodes within the cluster examples would
be the following

Resource (Global)

n/a

LVB

Lock Value Block

contains a small amount of data regarding the lock

TRFC

Traffic Controller

controls the DLM traffic between instances (messaging tickets)

Data Buffer Cache Block

Transaction Enqueue
Database Data Structures

FilesandDirectories
Files and Directories
$ORA_CRS_HOME/cdata/<cluster_name>

OCR backups (default location)

$ORA_HOME/log/<hostname>/client/ocrconfig_<pid>.log

OCR command log file

$ORA_CRS_HOME/crs/log

contains trace files for the CRS resources

$ORA_CRS_HOME/crs/init

contains trace files for the CRS daemon during startup, a good place to start

$ORA_CRS_HOME/css/log

contains cluster reconfigurations, missed checkins, connects and disconnects from the
client CSS listener. Look here to obtain when reboots occur

$ORA_CRS_HOME/css/init

contains core dumps from the cluster synchronization service daemon (OCSd)

$ORA_CRS_HOME/evm/log

logfiles for the event volume manager and eventlogger daemon

$ORA_CRS_HOME/evm/init

pid and lock files for EVM

$ORA_CRS_HOME/srvm/log

logfiles for Oracle Cluster Registry (OCR)

$ORA_CRS_HOME/log

log fles for Oracle clusterware which contains diagnostic messages at the Oracle cluster
level

UsefulViews/Tables
GCS and Cache Fusion Diagnostics
v$cache

contains information about every cached block in the buffer cache

v$cache_transfer

contains information from the block headers in SGA that have been pinged at least once

v$instance_cache_transfer

contains information about the transfer of cache blocks through the interconnect

v$cr_block_server

contains statistics about CR block transfer across the instances

v$current_block_server

contains statistics about current block transfer across the instances

v$gc_element

contains onetoone information for each global cache resource used by the buffer cache

v$lock

contains information about locks held within a database and outstanding requests for locks and latches

v$ges_blocking_enqueue

contains information about locks that are being blocked or blocking others and locks that are known to the lock manager

v$enqueue_statistics

contains details about enqueue statistics in the instance

v$resource_limits

display enqueue statistics

v$locked_object

contains information about DML locks acquired by different transactions in databases with their mode held

v$ges_statistics

contains miscellaneous statistics for GES

v$ges_enqueue

contains information about all locks known to the lock manager

v$ges_convert_local

contains information about all local GES operations

v$ges_convert_remote

contains information about all remote GES operations

v$ges_resource

contains information about all resources known to the lock manager

v$ges_misc

contains information about messaging traffic information

v$ges_traffic_controller

contains information about the message ticket usage

GES diagnostics

http://www.datadisk.co.uk/html_docs/rac/rac_cs.htm

1/5

5/13/2016

RACCheatsheet
Dynamic Resource Remastering

v$hvmaster_info

contains information about current and previous master instances of GES resources in relation to hash value ID of resource

v$gcshvmaster_info

the same as above but globally

v$gcspfmaster_info

conatins information about current and previous masters about GCS resources belonging to files mapped to a particular
master, including the number of times the resource has remastered

v$cluster_interconnects

contains information about interconnects that are being used for cluster communication

v$configured_interconnects

same as above but also contains interconnects that AC is aware off that are not being used

v$service

services running on an instance

x$kjmsdp

display LMS daemon statistics

x$kjmddp

display LMD daemon statistics

Cluster Interconnect

Miscellanous

UsefulParameters
Parameters
cluster_interconnects

specify a specific IP address to use for the inetrconnect

_gcs_fast_config

enables fast reconfiguration for gcs locks (true|false)

_lm_master_weight

controls which instance will hold or (re)master more resources than others

_gcs_resources

controls the number of resources an instance will master at a time

_lm_tickets

controls the number of message tickets

_lm_ticket_active_sendback

controls the number of message tickets (aggressive messaging)

_db_block_max_cr_dba

limits the number of CR copies per DBA on the buffer cache (see grd)

_fairness_threshold

used when too many CR requested arrive for a particular buffer and the block becomes disowned (see grd)

_gc_affinity_time

specifies interval minutes for reamstering

_gc_affinity_limit

defines the number of times a instance access the resource before remastering

_gc_affinity_minimum

defines the minimum number of times a instance access the resource before remastering

_lm_file_affinity

disables dynamic remastering for the objects belonging to those files

_lm_dynamic_remastering

enable or disable remastering

_gc_defer_time

define the time by which an instance deferred downgrading a lock (see Cache Fusion)

_lgwr_async_broadcast

change the SCN boardcast method (see troubleshooting)

Processes

Oracle RAC Daemons and Processes

OPROCd

Process Monitor

provides basic cluster integrity services

EVMd

Event Management

spawns a child process event logger and generates callouts

OCSSd

Cluster Synchronization Services

basic node membership, group services, basic locking

CRSd

Cluster Ready Services

resource monitoring, failover and node recovery

Lock Manager Server process GCS

this is the cache fusion part, it handles the consistent copies of blocks that are tranferred
between instances. It receives requests from LMD to perform lock requests. I rools back
any uncommitted transactions. There can be upto ten LMS processes running and can be
started dynamically if demand requires it.

LMSn

they manage lock manager service requests for GCS resources and send them to a service
queue to be handled by the LMSn process. It also handles global deadlock detection and
monitors for lock conversion timeouts.

this process manages the GES, it maintains consistency of GCS memory in case of process
death. It is also responsible for cluster reconfiguration and locks reconfiguration (node

http://www.datadisk.co.uk/html_docs/rac/rac_cs.htm

2/5

5/13/2016
LMON

RACCheatsheet
Lock Monitor Process GES

joining or leaving), it checks for instance deaths and listens for local messaging.
A detailed log file is created that tracks any reconfigurations that have happened.

LMD

Lock Manager Daemon GES

this manages the enqueue manager service requests for the GCS. It also handles deadlock
detention and remote resource requests from other instances.

LCK0

Lock Process GES

manages instance resource requests and crossinstance call operations for shared
resources. It builds a list of invalid lock elements and validates lock elements during
recovery.

DIAG

Diagnostic Daemon

This is a lightweight process, it uses the DIAG framework to monitor the healt of the
cluster. It captures information for later diagnosis in the event of failures. It will perform
any neccessary recovery if an operational hang is detected.

GeneralAdministration
Managing the Cluster
starting

/etc/init.d/init.crsstart
crsctlstartcrs

stopping

/etc/init.d/init.crsstop
crsctlstopcrs

enable/disable at boot time

/etc/init.d/init.crsenable
/etc/init.d/init.crsdisable
crsctlenablecrs
crsctldisablecrs

Managing the database configuration with SRVCTL

srvctlstartdatabased<database>o<option>

start all instances

Note:startslistenersifnotalreadyrunning,youcanusetheooptiontospecify
startup/shutdownoptions
force
open
mount
nomount
srvctlstopdatabased<database>o<option>

stop all instances

Note:thelistenersarenotstopped,youcanusetheooptiontospecifystartup/shutdown
options
immediate
abort
normal
transactional

start/stop particular instance

srvctl[start|stop]databased<database>i<instance>,<instance>

display the registered databases

srvctlconfigdatabase

status

srvctlstatusdatabased<database>
srvctlstatusinstanced<database>i<instance>,<instance>
srvctlstatusserviced<database>
srvctlstatusnodeappsn<node>
srvctlstatusasmn<node>

stopping/starting

srvctlstopdatabased<database>
srvctlstopinstanced<database>i<instance>,<instance>
srvctlstopserviced<database>s<service>,<service>i<instance>,<instance>
srvctlstopnodeappsn<node>
srvctlstopasmn<node>
srvctlstartdatabased<database>
srvctlstartinstanced<database>i<instance>,<instance>
srvctlstartserviced<database>s<service>,<service>i<instance>,<instance>
srvctlstartnodeappsn<node>
srvctlstartasmn<node>

adding/removing

srvctladddatabased<database>o<oracle_home>
srvctladdinstanced<database>i<instance>n<node>
srvctladdserviced<database>s<service>r<preferred_list>
srvctladdnodeappsn<node>o<oracle_home>A<name|ip>/network
srvctladdasmn<node>i<asm_instance>o<oracle_home>
srvctlremovedatabased<database>o<oracle_home>
srvctlremoveinstanced<database>i<instance>n<node>
srvctlremoveserviced<database>s<service>r<preferred_list>
srvctlremovenodeappsn<node>o<oracle_home>A<name|ip>/network
srvctlasmremoven<node>

OCR utilities
log file

$ORA_HOME/log/<hostname>/client/ocrconfig_<pid>.log
ocrcheck

checking

Note:willreturntheOCRversion,totalspaceallocated,spaceused,freespace,locationof
eachdeviceandtheresultoftheintegritycheck
ocrdumpbackupfile<file>

http://www.datadisk.co.uk/html_docs/rac/rac_cs.htm

3/5

5/13/2016

RACCheatsheet

dump contents
Note:bydefaultitdumpsthecontentsintoafilenamedOCRDUMPinthecurrentdirectory

export/import

ocrconfigexport<file>
ocrconfigrestore<file>
#showbackups
ocrconfigshowbackup
#tochangethelocationofthebackup,youcanevenspecifyaASMdisk
ocrconfigbackuploc<path|+asm>

backup/restore

#performabackup,willusethelocationspecifiedbythebackuploclocation
ocrconfigmanualbackup
#performarestore
ocrconfigrestore<file>
#deleteabackup
orcconfigdelete<file>
Note:therearemanymoreoptionsoseetheocrconfigmanpage
##add/relocatetheocrmirrorfiletothespecifiedlocation
ocrconfigreplaceocrmirror'/ocfs2/ocr2.dbf'

add/remove/replace

##relocateanexistingOCRfile
ocrconfigreplaceocr'/ocfs1/ocr_new.dbf'
##removetheOCRorOCRMirrorfile
ocrconfigreplaceocr
ocrconfigreplaceocrmirror

CRSAdministration
CRS Administration

starting

##StartingCRSusingOracle10gR1
notpossible
##StartingCRSusingOracle10gR2
$ORA_CRS_HOME/bin/crsctlstartcrs

stopping

##StoppingCRSusingOracle10gR1
srvctlstopdatabased<database>
srvctlstopasmn<node>
srvctlstopnodeappsn<node>
/etc/init.d/init.crsstop
##StoppingCRSusingOracle10gR2
$ORA_CRS_HOME/bin/crsctlstopcrs
##usetostopCRSrestartingafterareboot

disabling/enabling

##Oracle10gR1
/etc/init.d/init.crs[disable|enable]
##Oracle10gR2
$ORA_CRS_HOME/bin/crsctl[disable|enable]crs

checking

$ORA_CRS_HOME/bin/crsctlcheckcrs
$ORA_CRS_HOME/bin/crsctlcheckevmd
$ORA_CRS_HOME/bin/crsctlcheckcssd
$ORA_CRS_HOME/bin/crsctlcheckcrsd
$ORA_CRS_HOME/bin/crsctlcheckinstallwait600

status

$ORA_CRS_HOME/bin/crs_stat

create profile

$ORA_CRS_HOME/bin/crs_profile

register/unregister application

$ORA_CRS_HOME/bin/crs_register
$ORA_CRS_HOME/bin/crs_unregister

Start/Stop an application

$ORA_CRS_HOME/bin/crs_start
$ORA_CRS_HOME/bin/crs_stop

Resource permissions

$ORA_CRS_HOME/bin/crs_getparam
$ORA_CRS_HOME/bin/crs_setparam

Relocate a resource

$ORA_CRS_HOME/bin/crs_relocate

member number/name

olsnodesn

local node name

olsnodesl

activates logging

olsnodesg

display

oifcfggetif

delete

oicfgdeligglobal

set

oicfgsetifglobal<interfacename>/<subnet>:public
oicfgsetifglobal<interfacename>/<subnet>:cluster_interconnect

starting

gsdctlstart

Resource Applications (CRS Utilities)

Nodes

Oracle Interfaces

Global Services Daemon Control

http://www.datadisk.co.uk/html_docs/rac/rac_cs.htm

4/5

5/13/2016

RACCheatsheet

stopping

gsdctlstop

status

gsdctlstatus

create a new configuration

clscfginstall

Cluster Configuration (clscfg is used during installation)

upgrade or downgrade and existing clscfgupgrade
configuration
clscfgdowngrade
add or delete a node from the
configuration

clscfgadd
clscfgdelete

create a special singlenode

configuration for ASM

clscfglocal

brief listing of terminology used in

the other nodes

clscfgconcepts

used for tracing

clscfgtrace

help

clscfgh

Cluster Name Check

print cluster name

cemulton
Note:inOracle9itheulitywascalled"cemutls"

print the clusterware version

cemultow
Note:inOracle9itheulitywascalled"cemutls"

Node Scripts
Add Node

addnode.sh
Note:seeaddinganddeletingnodes

Delete Node

deletenode.sh
Note:seeaddinganddeletingnodes

Enqueues

displaying statistics

SQL>columncurrent_utilizationheadingcurrent
SQL>columnmax_utilizationheadingmax_usage
SQL>columninitial_allocationheadinginitial
SQL>columnresource_limitformata23
SQL>select*fromv$resource_limit

Messaging (tickets)
ticket usage

selectlocal_nidlocal,remote_nidremote,tckt_availavail,tckt_limitlimit,snd_q_len
send_queue,tckt_waitwaitingfromv$ges_traffic_controller

dump ticket information

SQL>oradebugsetmypid
SQL>oradebugunlimit
SQL>oradebuglkdebugt

Lighwork Rule and Fairness Threshold

selectcr_requests,light_works,data_requests,fairness_down_convertsfromv$cr_block_server

downconvert

Note:lowerthe_fairness_thresholdiftheratiogoesabove40%,setto0iftheinstanceisa
queryonlyinstance.

Remastering
##ObtaintheOBJECT_IDformthebelowtable
SQL>select*fromv$gcspfmaster_info

force dynamic remastering (DRM)

##Determinewhomastersit
SQL>oradebugsetmypid
SQL>oradebuglkdebuga<OBJECT_ID>
##Nowremastertheresource
SQL>oradebugsetmypid
SQL>oradebuglkdebugmpkey<OBJECT_ID>

GRD, SRVCTL, GSD and SRVCONFIG Tracing

Enable tracing

$exportSRVM_TRACE=true

Disable tracing

$exportSRVM_TRACE=""

VotingDisk
adding

crsctladdcssvotedisk<file>

deleting

crsctldeletecssvotedisk<file>

querying

crsctlquerycssvotedisk

http://www.datadisk.co.uk/html_docs/rac/rac_cs.htm

5/5