Documente Academic
Documente Profesional
Documente Cultură
Voting Disk is a file that sits in the shared storage area and must be
accessible by all nodes in the cluster. All nodes in the cluster
registers their heart-beat information in the voting disk, so as to
confirm that they are all operational. If heart-beat information of
any node in the voting disk is not available that node will be evicted
from the cluster. The CSS (Cluster Synchronization Service) daemon
in the clusterware maintains the heart beat of all nodes to the
voting disk. When any node is not able to send heartbeat to voting
disk, then it will reboot itself, thus help avoiding the split-brain
syndrome.
For high availability, Oracle recommends that you have a minimum
of three or odd number (3 or greater) of votingdisks.
Voting Disk is file that resides on shared storage and Manages
cluster members. Voting disk reassigns cluster ownership between
the nodes in case of failure.
The Voting Disk Files are used by Oracle Clusterware to determine
which nodes are currently members of the cluster. The voting disk
files are also used in concert with other Cluster components such as
CRS to maintain the clusters integrity.
Oracle Database 11g Release 2 provides the ability to store the
voting disks in ASM along with the OCR. Oracle Clusterware can
access the OCR and the voting disks present in ASM even if the ASM
instance is down. As a result CSS can continue to maintain the
Oracle cluster even if the ASM instance has failed.
How many voting disks are you maintaining ?
By default Oracle will create 3 voting disk files in ASM.
Oracle expects that you will configure at least 3 voting disks for
redundancy purposes. You should always configure an odd number
of voting disks >= 3. This is because loss of more than half your
voting disks will cause the entire cluster to fail.
You should plan on allocating 280MB for each voting disk file. For
example, if you are using ASM and external redundancy then you
will need to allocate 280MB of disk for the voting disk. If you are
using ASM and normal redundancy you will need 560MB.
Why we need to keep odd number of voting disks ?
Oracle expects that you will configure at least 3 voting disks for
redundancy purposes. You should always configure an odd number
of voting disks >= 3. This is because loss of more than half your
voting disks will cause the entire cluster to fail.
What are Oracle RAC software components?
Oracle RAC is composed of two or more database instances. They
are composed of Memory structures and background processes
same as the single instance database.Oracle RAC instances use two
processes GES(Global Enqueue Service), GCS(Global Cache Service)
that enable cache fusion.Oracle RAC instances are composed of
following background processes:
ACMSAtomic Controlfile to Memory Service (ACMS)
GTX0-jGlobal Transaction Process
LMONGlobal Enqueue Service Monitor
LMDGlobal Enqueue Service Daemon
LMSGlobal Cache Service Process
LCK0Instance Enqueue Process
RMSnOracle RAC Management Processes (RMSn)
RSMNRemote Slave Monitor
What are Oracle Clusterware processes for 10g ?
Cluster Synchronization Services (ocssd) Manages cluster node
membership and runs as the oracle user; failure of this process
results in cluster restart.
Cluster Ready Services (crsd) The crs process manages cluster
resources (which could be a database, an instance, a service, a
Listener, a virtual IP (VIP) address, an application process, and so
on) based on the resources configuration information that is stored
in the OCR. This includes start, stop, monitor and failover
operations. This process runs as the root user
Event manager daemon (evmd) A background process that
publishes events that crs creates.
Process Monitor Daemon (OPROCD) This process monitor the
cluster and provide I/O fencing. OPROCD performs its check, stops
running, and if the wake up is beyond the expected time, then
OPROCD resets the processor and reboots the node. An OPROCD
failure results in Oracle Clusterware restarting the node. OPROCD
uses the hangcheck timer on Linux platforms.
RACG (racgmain, racgimon) Extends clusterware to support
Oracle-specific requirements and complex resources. Runs server
callout scripts when FAN events occur.
What are Oracle database background processes specific to RAC?
LMSGlobal Cache Service Process
LMDGlobal Enqueue Service Daemon
LMONGlobal Enqueue Service Monitor
LCK0Instance Enqueue Process
Oracle RAC instances use two processes, the Global Cache Service
(GCS) and the Global Enqueue Service (GES). The GCS and GES
maintain records of the statuses of each data file and each cached
block using a Global Resource Directory (GRD). The GRD contents
are distributed across all of the active instances.
What is Cache Fusion?
Transfor of data across instances through private interconnect is
called cachefusion.Oracle RAC is composed of two or more
instances. When a block of data is read from datafile by an instance
within the cluster and another instance is in need of the same
block,it is easy to get the block image from the insatnce which has
the block in its SGA rather than reading from the disk. To enable
inter instance communication Oracle RAC makes use of
interconnects. The Global Enqueue Service(GES) monitors and
Instance enqueue process manages the cahce fusion
What is SCAN? (11gR2 feature)
Single Client Access Name (SCAN) is s a new Oracle Real Application
Clusters (RAC) 11g Release 2 feature that provides a single name
for clients to access an Oracle Database running in a cluster. The
benefit is clients using SCAN do not need to change if you add or
remove nodes in the cluster.
SCAN provides a single domain name via (DNS), allowing and-users
to address a RAC cluster as-if it were a single IP address. SCAN
works by replacing a hostname or IP list with virtual IP addresses
(VIP).
Single client access name (SCAN) is meant to facilitate single name
for all Oracle clients to connect to the cluster database, irrespective
of number of nodes and node location. Until now, we have to keep
adding multiple address records in all clients tnsnames.ora, when a
new node gets added to or deleted from the cluster.
Single Client Access Name (SCAN) eliminates the need to change
TNSNAMES entry when nodes are added to or removed from the
Cluster. RAC instances register to SCAN listeners as remote
listeners. Oracle recommends assigning 3 addresses to SCAN, which
will create 3 SCAN listeners, though the cluster has got dozens of
nodes.. SCAN is a domain name registered to at least one and up to
three IP addresses, either in DNS (Domain Name Service) or GNS
(Grid Naming Service). The SCAN must resolve to at least one
address on the public network. For high availability and scalability,
Oracle recommends configuring the SCAN to resolve to three
addresses.
Network Card 2 (with IP address set 2) for private network (for inter
node communication between rac nodes used by clusterware and
rac database)
IP address set 3 for Virtual IP (VIP) (used as Virtual IP address for
client connection and for connection failover)
3. Storage Option for OCR and Voting Disk RAW, OCFS2 (Oracle
Cluster File System), NFS, ..
Which enable the load balancing of applications in RAC?
Oracle Net Services enable the load balancing of application
connections across all of the instances in an Oracle RAC database.
How to find location of OCR file when CRS is down?
If you need to find the location of OCR (Oracle Cluster Registry) but
your CRS is down.
When the CRS is down:
Look into ocr.loc file, location of this file changes depending on
the OS:
On Linux: /etc/oracle/ocr.loc
On Solaris: /var/opt/oracle/ocr.loc
When CRS is UP:
Set ASM environment or CRS environment then run the below
command:
ocrcheck
In 2 node RAC, how many NICs are r using ?
2 network cards on each clusterware node
Network Card 1 (with IP address set 1) for public network
Network Card 2 (with IP address set 2) for private network (for inter node
communication between rac nodes used by clusterware and rac database)
In 2 node RAC, how many IPs are r using ?
6 3 set of IP address
## eth1-Public: 2
## eth0-Private: 2
## VIP: 2
How to find IPs information in RAC ?
Edit the /etc/hosts file as shown below:
# Do not remove the following line, or various programs
# that requires network functionality will fail.
127.0.0.1 localhost.localdomain localhost
## Public Node names
192.168.10.11 node1-pub.hingu.net node1-pub
192.168.10.22 node2-pub.hingu.net node2-pub
## Private Network (Interconnect)
192.168.0.11 node1-prv node1-prv
RAC system must equipped with low-latency and high speed inter-connect to
make it happen.
What are the different network components are in 10g RAC?
Public Ip, Private Ip, and vip components in 10g
Scan and Scan Lsnr in 11g along with 10g components.
Mention the Oracle RAC background process :
ACMSAtomic Controlfile to Memory Service (ACMS)
GTX0-jGlobal Transaction Process
LMONGlobal Enqueue Service Monitor
LMDGlobal Enqueue Service Daemon
LMSGlobal Cache Service Process
LCK0Instance Enqueue Process
RMSnOracle RAC Management Processes (RMSn)
RSMNRemote Slave Monitor
Daig:
ACMS:
Atomic Controlfile Memory Service is an agent that ensures SGA memory
update.(ie) SGA updates are globally committed if success or globally
aborted if failed.
GTX0-j :
The process provides transparent support for XA global transactions in a RAC
environment.The database auto tunes the number of these processes based
on the workload of XA global transactions.
LMON:
Global Enqueue Service Monitor .This process monitors global enques and
resources across the cluster and performs global enqueue recovery
operations.
LMD:
Global enqueue service daemon. This process manages incoming remote
RSMN:
This process is called as Remote Slave Monitor. This process manages
background slave process creation and communication on remote instances.
Give Details on Cache Fusion:
Oracle RAC is composed of two or more instances. When a block of data is
read from datafile by an instance within the cluster and another instance is
in need of the same block,it is easy to get the block image from the insatnce
which has the block in its SGA rather than reading from the disk. To enable
inter instance communication Oracle RAC makes use of interconnects. The
Global Enqueue Service(GES) monitors and Instance enqueue process
manages the cahce fusion.
No,it can be used only for Oracle database 11g releases(from 11.1).
State the initialization parameters that must have same value for
every instance in an Oracle RAC database:
ACTIVE_INSTANCE_COUNT
ARCHIVE_LAG_TARGET
COMPATIBLE
CLUSTER_DATABASE
CLUSTER_DATABASE_INSTANCE
CONTROL_FILES
DB_BLOCK_SIZE
DB_DOMAIN
DB_FILES
DB_NAME
DB_RECOVERY_FILE_DEST
DB_RECOVERY_FILE_DEST_SIZE
DB_UNIQUE_NAME
INSTANCE_TYPE (RDBMS or ASM)
PARALLEL_MAX_SERVERS
REMOTE_LOGIN_PASSWORD_FILE
UNDO_MANAGEMENT
What is ORA-00603: ORACLE server session terminated by fatal error or ORA29702: error occurred in Cluster Group Service operation?
RAC node name was listed in the loopback address...
Can the DML_LOCKS and RESULT_CACHE_MAX_SIZE be identical on all
instances?
These parameters can be identical on all instances only if these parameter
values are set to zero.
What two parameters must be set at the time of starting up an ASM instance
in a RAC environment?The parameters CLUSTER_DATABASE and
INSTANCE_TYPE must be set.
Name some Oracle clusterware tools and their uses?
OIFCFG - allocating and deallocating network interfaces
OCRCONFIG - Command-line tool for managing Oracle Cluster Registry
OCRDUMP - Identify the interconnect being used
CVU - Cluster verification utility to get status of CRS resources
Oracle Database 11g Release 2 provides the ability to store the voting disks
in ASM along with the OCR
Oracle Cluster Registry (OCR) It contains vital information cluster and
configuration information
The OCR must reside on shared disk that is accessible by all of the nodes in
your cluster. The daemon OCSSd manages.
Virtual IP (VIP) - A virtual IP is an alternate IP address used for a node for
the client connections instead of the standard public IP address. To configure
VIP address, we need to reserve a spare IP address for each node.
If a node fails, then the node's VIP fails over to another surviving node and
session will be establishing in another surviving node automatically.
Past Image:
Earlier version of the block while updating.
12. How to take backup of OCR file?
#ocrconfig -manualbackup
#ocrconfig -export file_name.dmp
#ocrdump -backupfile my_file
$cp -p -R /u01/app/crs/cdata /u02/crs_backup/ocrbackup/RAC1
13. How to recover OCR file?
Ans:
#ocrconfig -restore backup_file.ocr
#ocrconfig -import file_name.dmp
14. What is local OCR?
Ans:
/etc/oracle/local.ocr
/var/opt/oracle/local.ocr
15. How to check backup of OCR files?
Ans:
#ocrconfig showbackup
16. How to take backup of voting file?
Ans:
dd if=/u02/ocfs2/vote/VDFile_0 of=$ORACLE_BASE/bkp/vd/VDFile_0
crsctl backup css votedisk
-- from 11g R2
17. How do I identify the voting disk location?
Ans:
# crsctl query css votedisk
18. How do I identify the OCR file location?
check /var/opt/oracle/ocr.loc or /etc/ocr.loc
# ocrcheck
19. If voting disk/OCR file got corrupted and dont have backups,
how to get them?
We have to install Clusterware.
21. What is Oracle RAC Node Eviction?
Oracle Clusterware is designed to perform a node eviction by removing one
or more nodes from the cluster if some critical problem is detected.
A critical problem could be a node not responding via a network heartbeat, a
node not responding via a disk heartbeat, a hung or severely degraded
machine, or a hung ocssd.bin process.
During failures, to avoid data corruption, the failing instance evicts itself from
the cluster group
22. What is the major difference between 10g and 11g RAC?
What is TAF?
TAF (Transparent Application Failover) is a configuration that allows session
fail-over between nodes.
TAF offers the ability to restart transactions at SELECT or session level.
Failover_mode = Type= select /session
= Method= Basic/ Preconnect
What is cluvfy ?
It checks the cluster configuration.
Components installed, node connectivity, shared storage access, user
equivalence
It is a private network which is used to ship data blocks from one instance to
another for cache fusion. The physical data blocks as well as data dictionary
blocks are shared across this interconnect.
How do you determine what protocol is being used for Interconnect traffic?
One of the ways is to look at the database alert log for the time period when
the database was started up.
What methods are available to keep the time synchronized on all nodes in
the cluster?
Where does the Clusterware write when there is a network or Storage missed
heartbeat?
The ocrconfig -showbackup can be run to find out the automatic and
manually run backups.
You can use either the logical or the physical OCR backup copy to restore the
Repository.
How do you find out what object has its blocks being shipped across the
instance the most?
The Cluster Health Monitor (CHM) stores operating system metrics in the
CHM repository for all nodes in a RAC cluster. It stores information on CPU,
memory, process, network and other OS data, This information can later be
retrieved and used to troubleshoot and identify any cluster related issues. It
is a default component of the 11gr2 grid install. The data is stored in the
master repository and replicated to a standby repository on a different node.
All processing will show down to the CPU speed of the slowest server.
Oracle Local repository contains information that allows the cluster processes
to be started up with the OCR being in the ASM storage ssytem. Since the
ASM file system is unavailable until the Grid processes are started up a local
copy of the contents of the OCR is required which is stored in the OLR.
In 10g the default SGA size is 1G in 11g it is set to 256M and in 12c ASM it is
set back to 1G.
You can use md_backup to restore the ASM diskgroup configuration in-case of
ASM diskgroup storage loss.
Datafiles
Redo logfiles
Spfiles
In 12c the files below can also new be stored in the ASM Diskgroup
Password file
What it the ASM POWER_LIMIT?
This is the parameter which controls the number of Allocation units the ASM
instance will try to rebalance at any given time. In ASM versions less than
11.2.0.3 the default value is 11 however it has been changed to unlimited in
later versions.
CLUSTER_DATABASE
CLUSTER_DATABASE_INSTANCE
INSTANCE_TYPE (RDBMS or ASM)
ACTIVE_INSTANCE_COUNT
UNDO_MANAGEMENT
What is the future of the Oracle Grid?
The Grid software is becoming more and more capable of not just supporting
HA for Oracle Databases but also other applications including Oracles
applications. With 12c there are more features and functionality built-in and
it is easier to deploy these pre-built solutions, available for common Oracle
applications.
Is there an easy way to verify the inventory for all remote nodes
You can run the opatch lsinventory -all_nodes command from a single node
to look at the inventory details for all nodes in the cluster.
Q What is SCAN?
Single Client Access Name (SCAN) is s a new Oracle Real Application Clusters
(RAC) 11g Release 2 feature that provides a single name for clients to access
an Oracle Database running in a cluster. The benefit is clients using SCAN do
not need to change if you add or remove nodes in the cluster.
Odd number of disk are to avoid split brain, When Nodes in cluster can't talk
to each other they run to lock the Voting disk and whoever lock the more
disk will survive, if disk number are even there are chances that node might
lock 50% of disk (2 out of 4) then how to decide which node to evict.
whereas when number is odd, one will be higher than other and each for
cluster to evict the node with less number
Q If there is some issue with virtual IP how will you troubleshoot it?
How will you change virtual ip?
To change the VIP (virtual IP) on a RAC node, use the command
Q What is RAC?
RAC stands for Real Application cluster. It is a clustering solution from Oracle
Corporation that ensures high availability of databases by providing instance
failover, media failover features.
Q What is GRD?
GRD stands for Global Resource Directory. The GES and GCS maintains
records of the statuses of each datafile and each cahed block using global
Q How to export and import crs resources while migrating Oracle RAC to new
server.
Below script generate svrctl add script for database, instance, service and
11G listeners from OCR from current RAC.
Save the result of the script and run it at new RAC.
# Generate DB resource
END { if (avail == "-a ") {avail = ""}; printf "%s %s %s %s %s\n", "srvctl add
database -d ", dbname, dbhome, spfile, dg }'
If a node fails, then the node's VIP address fails over to another node on
which the VIP address can accept TCP connections but it cannot accept
Oracle connections.
Q What are the administrative tools used for Oracle RAC environments?
Oracle RAC cluster can be administered as a single image using
OEM(Enterprise
Manager),SQL*PLUS,Servercontrol(SRVCTL),clusterverificationutility(cvu),DBC
A,NETCA
Q What is FAN?
Fast application Notification as it abbreviates to FAN relates to the events
related to instances,services and nodes.This is a notification mechanism that
Oracle RAc uses to notify other processes about the configuration and
service level information that includes service status changes such as,UP or
DOWN events.Applications can respond to FAN events and take immediate
action.
Q State the initialization parameters that must have same value for every
instance in an Oracle RAC database
Some initialization parameters are critical at the database creation time and
must have same values.Their value must be specified in SPFILE or PFILE for
every instance.The list of parameters that must be identical on every
instance are given below:
ACTIVE_INSTANCE_COUNT
ARCHIVE_LAG_TARGET
COMPATIBLE
CLUSTER_DATABASE
CLUSTER_DATABASE_INSTANCE
CONTROL_FILES
DB_BLOCK_SIZE
DB_DOMAIN
DB_FILES
DB_NAME
DB_RECOVERY_FILE_DEST
DB_RECOVERY_FILE_DEST_SIZE
DB_UNIQUE_NAME
INSTANCE_TYPE (RDBMS or ASM)
PARALLEL_MAX_SERVERS
REMOTE_LOGIN_passWORD_FILE
UNDO_MANAGEMENT
What two parameters must be set at the time of starting up an ASM instance
in a RAC environment?The parameters CLUSTER_DATABASE and
INSTANCE_TYPE must be set.
Q What are the modes of deleting instances from ORacle Real Application
cluster Databases?
We can delete instances using silent mode or interactive mode using
DBCA(Database Configuration Assistant).
Q How do we verify that an instance has been removed from OCR after
deleting an instance?
Issue the following srvctl command:
srvctl config database -d database_name
cd CRS_HOME/bin
./crs_stat
GC CR request: the time it takes to retrieve the data from the remote cache
We have public, private, and VIP components. Private interfaces is for intra
node communication. VIP is all about availability of application. When a node
fails then the VIP component will fail over to some other node, this is the
reason that all applications should be based on VIP components. This means
that tns entries should have VIP entry in the host list.
PROD1
PROD2
PROD3
CPU 8
CPU 15
CPU 8
32 GB RAM
12 GB RAM
16 GB RAM
What are you looking for here? What tuning information do you expect?
It is a 3 node cluster with different hardware configuration running RAC.
I would put 20% of the memory for Oracle in each node. So that would mean
that the SGA is different in each of the nodes.
Also since the CPU's are different PROD2 can have more number of max
number of processes as compared to the rest of them.
But as I said this is just configuration, this is not tuning. Question is not clear.
Q Write a sample script for RMAN for the recovery if all the instance are
down.(First explain the procedure how you will restore)
Bring all nodes down.
Start one Node
Restore all datafiles and archive logs.
Recover 1 Node.
Open the database.
bring other nodes up.
Confirm that all nodes are operational.
Q. Clients are performing some operation and suddenly one of the datafile is
experiencing problem what do you do? The cluster is a two node one.
A. Bring the datafile offline recover the datafile.
Make sure there is a recent backup of the OCR file before making any
changes:
ocrconfig showbackup
If there is not a recent backup copy of the OCR file, an export can be taken
for the current OCR file. Use the following command to generate an export of
the online OCR file:
In 10.2
In 11g
# ocrconfig -manualbackup
The new OCR disk must be owned by root, must be in the oinstall group, and
must have permissions set to 640. Provide at least 100 MB disk space for the
OCR.
Now run ocrcheck to verify if the OCR is pointing to the new file
Shutdown the Oracle Clusterware (crsctl stop crs as root) on all nodes before
making any modification to the voting disk. Determine the current voting
disk location using:
dd if=voting_disk_name of=backup_file_name
To move a Voting Disk, provide the full path including file name:
After modifying the voting disk, start the Oracle Clusterware stack on all
nodes
including:
Shared memory parameters.
Open file descriptor and UDP send/receive parameters.
Sets permissions on the Oracle Inventory (central inventory) directory.
Reconfigures primary and secondary group memberships for the
installation
owner, if necessary, for the Oracle Inventory directory and the operating
system
privileges groups.
Sets shell limits if necessary to required values.
3. After the Summary screen, OUI will start copying under the $CRS_HOME
(this is the $ORACLE_HOME for Oracle Clusterware) in the local node the
libraries and executables.
- here we will have the daemons and scripts init.* created and configured
properly.
- note that for CRS only some client libraries are recreated, but not all the
executables (as for the RDBMS).
4. Later the software is propagated to the rest of the nodes in the cluster and
the oraInventory is updated.
5. The installer will ask to execute root.sh on each node. Until this step the
software for Oracle Clusterware is inside the $CRS_HOME.
- control files (or SCLS_SRC files ) will be created with the correct contents to
start Oracle Clusterware.
These files are used to control some aspects of Oracle Clusterware like:
- enable/disable processes from the CSSD family (Eg. oprocd, oslsvmon)
In order to start the Oracle Clusterware daemons, the init.* scripts first need
to be run. These scripts are executed by the daemon init. To accomplish this
some entries must be created in the file /etc/inittab.
- the different processes init.* (init.cssd, init.crsd, etc) will start the daemons
(ocssd.bin, crsd.bin, etc). When all the daemons are running then we can say
that the installation was successful
- On 10.2 and later, running root.sh on the last node in the cluster also will
create the nodeapps (VIP, GSD and ONS). On 10.1, VIPCA is executed as part
of the RAC installation.
6. After running root.sh on each node, we need to continue with the OUI
session. After pressing the 'OK' button OUI will include the information for the
public and cluster_interconnect interfaces. Also CVU (Cluster Verification
Utility) will be executed.
Q What are Oracle Clusterware processes for 10g on Unix and Linux
Cluster Ready Services (crsd) The crs process manages cluster resources
(which could be a database, an instance, a service, a Listener, a virtual IP
(VIP) address, an application process, and so on) based on the resource's
configuration information that is stored in the OCR. This includes start, stop,
monitor and failover operations. This process runs as the root user
Process Monitor Daemon (OPROCD) This process monitor the cluster and
provide I/O fencing. OPROCD performs its check, stops running, and if the
wake up is beyond the expected time, then OPROCD resets the processor
and reboots the node. An OPROCD failure results in Oracle Clusterware
restarting the node. OPROCD uses the hangcheck timer on Linux platforms.
To ensure that each Oracle RAC database instance obtains the block that it
needs to satisfy a query or transaction, Oracle RAC instances use two
processes, the Global Cache Service (GCS) and the Global Enqueue Service
(GES). The GCS and GES maintain records of the statuses of each data file
and each cached block using a Global Resource Directory (GRD). The GRD
contents are distributed across all of the active instances.
Voting Disk Oracle RAC uses the voting disk to manage cluster
membership by way of a health check and arbitrates cluster ownership
among the instances in case of network failures. The voting disk must reside
on shared disk.
To display backups :
#ocrconfig -showbackup
To restore a backup :
#ocrconfig -restore
With Oracle RAC 10g Release 2 or later, you can also use the export
command:
#ocrconfig -export -s online, and use -import option to restore the contents
back.
With Oracle RAC 11g Release 1, you can do a manaual backup of the OCR
with the command:
# ocrconfig -manualbackup
"ssh" are not required for normal Oracle RAC operation. However "ssh"
should be enabled for Oracle RAC and patchset installation.
Q What is SCAN?
Single Client Access Name (SCAN) is s a new Oracle Real Application Clusters
(RAC) 11g Release 2 feature that provides a single name for clients to access
an Oracle Database running in a cluster. The benefit is clients using SCAN do
not need to change if you add or remove nodes in the cluster.
Without using VIPs or FAN, clients connected to a node that died will often
wait for a TCP timeout period (which can be up to 10 min) before getting an
error. As a result, you don't really have a good HA solution without using
VIPs.
When a node fails, the VIP associated with it is automatically failed over to
some other node and new node re-arps the world indicating a new MAC
address for the IP. Subsequent packets sent to the VIP go to the new node,
which will send error RST packets back to the clients. This results in the
clients getting errors immediately
10g Release 2, support 100 nodes in a cluster using Oracle Clusterware, and
100 instances in a RAC database.
Q Srvctl cannot start instance, I get the following error PRKP-1001 CRS-0215,
however sqlplus can start it on both nodes? How do you identify the
problem?
Set the environmental variable SRVM_TRACE to true.. And start the instance
with srvctl. Now you will get detailed error stack.
1) Oracle recommends that you back up your voting disk after the initial
cluster creation and after we complete any node addition or deletion
procedures.
2) First, as root user, stop Oracle Clusterware (with the crsctl stop crs
command) on all nodes. Then, determine the current voting disk by issuing
the following command:
crsctl query votedisk css
3) Then, issue the dd or ocopy command to back up a voting disk, as
appropriate.
Give the syntax of backing up voting disks:On Linux or UNIX systems:
dd if=voting_disk_name of=backup_file_name
where,
voting_disk_name is the name of the active voting disk
backup_file_name is the name of the file to which we want to back up the
voting disk contents
On Windows systems, use the ocopy command:
ocopy voting_disk_name backup_file_name
To move a voting disk, issue the following commands as the root user,
replacing the path variable with the fully qualified path name for the voting
disk we want to move:
crsctl delete css votedisk path -force
crsctl add css votedisk path -force
What is RAC? What is the benefit of RAC over single instance database?
Benefits:
Improve response time
Improve throughput
High availability
Transparency
Oracle RAC one Node is a single instance running on one node of the cluster
while the 2nd node is in cold standby mode. If the instance fails for some
reason then RAC one node detect it and restart the instance on the same
node or the instance is relocate to the 2nd node incase there is failure or
fault in 1st node.
The benefit of this feature is that it provides a cold failover solution and it
automates the instance relocation without any downtime and does not need
a manual intervention. Oracle introduced this feature with the release of
11gR2 (available with Enterprise Edition).
The Clusterware is installed on each node (on an Oracle Home) and on the
shared disks (the voting disks and the CSR file)
The base software is installed on each node of the cluster and the database
storage on the shared disks.
What kind of storage we can use for the shared Clusterware files?
OCFS (Release 1 or 2)
Raw devices
Third party cluster file system such as GPFS or Veritas
When a VIP address failover happens, Clients that attempt to connect to the
VIP address receive a rapid connection refused error .They dont have to wait
for TCP connection timeout messages.
Voting Disk is a file that sits in the shared storage area and must be
accessible by all nodes in the cluster. All nodes in the cluster registers their
heart-beat information in the voting disk, so as to confirm that they are all
operational. If heart-beat information of any node in the voting disk is not
available that node will be evicted from the cluster.
For high availability, Oracle recommends that you have a minimum of three
or odd number (3 or greater) of votingdisks.
Voting Disk is file that resides on shared storage and Manages cluster
members. Voting disk reassigns cluster ownership between the nodes in case
of failure.
The Voting Disk Files are used by Oracle Clusterware to determine which
nodes are currently members of the cluster. The voting disk files are also
used in concert with other Cluster components such as CRS to maintain the
clusters integrity.
Oracle Database 11g Release 2 provides the ability to store the voting disks
in ASM along with the OCR. Oracle Clusterware can access the OCR and the
voting disks present in ASM even if the ASM instance is down. As a result CSS
can continue to maintain the Oracle cluster even if the ASM instance has
failed.
What kind of storage we can use for the RAC database storage?
OCFS (Release 1 or 2)
ASM
raw devices
third party cluster file system such as GPFS or Veritas
What is a CFS?
A cluster File System (CFS) is a file system that may be accessed (read and
write) by all members in a cluster at the same time. This implies that all
members of a cluster have the same view.
What is an OCFS2?
The OCFS2 is the Oracle (version 2) Cluster File System which can be used
for the Oracle Real Application Cluster.
A raw device is a disk drive that does not yet have a file system set up. Raw
devices are used for Real Application Clusters since they enable the sharing
of disks.
Oracle expects that you will configure at least 3 voting disks for redundancy
purposes. You should always configure an odd number of voting disks >= 3.
This is because loss of more than half your voting disks will cause the entire
cluster to fail.
created and logical partitions are assigned to it without any formatting. Once
formatting is complete, it is called cooked partition.
A CFS offers:
Simpler management
Use of Oracle Managed Files with RAC
Single Oracle Software installation
Autoextend enabled on Oracle datafiles
Uniform accessibility to archive logs in case of physical node failure
With Oracle_Home on CFS, when you apply Oracle patches CFS guarantees
that the updated Oracle_Home is visible to all nodes in the cluster.
Oracle RAC 10g Release 1 introduced Oracle Cluster Ready Services (CRS), a
platform-independent set of system services for cluster environments. In
Release 2, Oracle has renamed this product to Oracle Clusterware.
SSH (Secure Shell,10g+) or RSH (Remote Shell, 9i+) allows oracle UNIX
account connecting to another RAC node and copy/ run commands as the
local oracle UNIX account.
No. SSH or RSH are needed only for RAC, patch set installation and clustered
database creation.
Each node of a cluster that is being used for a clustered database will
typically have the RDBMS and RAC software loaded on it, but not actual data
files (these need to be available via shared disk).
What are the restrictions on the SID with a RAC database? Is it limited to 5
characters?
The SID prefix in 10g Release 1 and prior versions was restricted to five
characters by install/ config tools so that an ORACLE_SID of up to max of
5+3=8 characters can be supported in a RAC environment. The SID prefix is
relaxed up to 8 characters in 10g Release 2, see bug 4024251 for more
information.
Are there any issues for the interconnect when sharing the same switch as
the public network by using VLAN to separate the network?
The Cluster Verification Utility (CVU) is a validation tool that you can use to
check all the important components that need to be verified at different
stages of deployment in a RAC environment.
No, the OCR and voting disk must be on raw or CFS (cluster file system).
The voting disk is nothing but a file that contains and manages information
of all the node memberships.
RMAN to make backups of the database, dd to backup your voting disk and
hard copies of the OCR file.
What command would you use to check the availability of the RAC system?
What is SCAN?
Single Client Access Name (SCAN) is s a new Oracle Real Application Clusters
(RAC) 11g Release 2 feature that provides a single name for clients to access
an Oracle Database running in a cluster. The benefit is clients using SCAN do
not need to change if you add or remove nodes in the cluster.
It is a private network which is used to ship data blocks from one instance to
another for cache fusion. The physical data blocks as well as data dictionary
blocks are shared across this interconnect.
How do you determine what protocol is being used for Interconnect traffic?
One of the ways is to look at the database alert log for the time period when
the database was started up.
What methods are available to keep the time synchronized on all nodes in
the cluster?
Where does the clusterware write when there is a network or Storage missed
heartbeat?
The ocrconfig -showbackup can be run to find out the automatic and
manually run backups.
You can use either the logical or the physical OCR backup copy to restore the
Repository.
How do you find out what object has its blocks being shipped across the
instance the most?
The Cluster Health Monitor (CHM) stores operating system metrics in the
CHM repository for all nodes in a RAC cluster. It stores information on CPU,
memory, process, network and other OS data, This information can later be
retrieved and used to troubleshoot and identify any cluster related issues.
It is a default component of the 11gr2 grid install. The data is stored in the
master repository and replicated to a standby repository on a different node.
All processing will show down to the CPU speed of the slowest server.
CLUSTER_DATABASE
CLUSTER_DATABASE_INSTANCE
INSTANCE_TYPE (RDBMS or ASM)
ACTIVE_INSTANCE_COUNT
UNDO_MANAGEMENT
The Grid software is becoming more and more capable of not just supporting
HA for Oracle Databases but also other applications including Oracles
applications. With 12c there are more features and functionality built-in and
it is easier to deploy these pre-built solutions, available for common Oracle
applications.
Is there an easy way to verify the inventory for all remote nodes
You can run the opatch lsinventory -all_nodes command from a single node
to look at the inventory details for all nodes in the cluster.
Oracle RAC instances use two processes, the Global Cache Service (GCS) and
the Global Enqueue Service (GES). The GCS and GES maintain records of the
statuses of each data file and each cached block using a Global Resource
Directory (GRD). The GRD contents are distributed across all of the active
instances.
Single Client Access Name (SCAN) is s a new Oracle Real Application Clusters
(RAC) 11g Release 2 feature that provides a single name for clients to access
an Oracle Database running in a cluster. The benefit is clients using SCAN do
not need to change if you add or remove nodes in the cluster.
SCAN Name
SCAN IPs (3)
SCAN Listeners (3)
What is FAN?
If you need to find the location of OCR (Oracle Cluster Registry) but your CRS
is down.
ocrcheck
Voting Disk > Oracle RAC uses the voting disk to manage cluster
membership by way of a health check and arbitrates cluster ownership
among the instances in case of network
failures. The voting disk must reside on shared disk.
Without using VIPs or FAN, clients connected to a node that died will often
wait for a TCP timeout period (which can be up to 10 min) before getting an
error. As a result, you
dont really have a good HA solution without using VIPs.
When a node fails, the VIP associated with it is automatically failed over to
some other node and new node re-arps the world indicating a new MAC
address for the IP. Subsequent
packets sent to the VIP go to the new node, which will send error RST
packets back to the clients. This results in the clients getting errors
immediately.
What is GRD?
In a RAC environment the buffer cache is global across all instances in the
cluster and hence the processing differs.The most common wait events
related to this are gc cr request and gc buffer busy
GC CR request :the time it takes to retrieve the data from the remote cache
GC BUFFER BUSY: It is the time the remote instance locally spends accessing
the requested data block.
Issue the following query from any one node connecting through SQL*PLUS.
$connect sys/sys as sysdba
SQL>select * from V$ACTIVE_INSTANCES;
The query gives the instance number under INST_NUMBER
column,host_:instancename under INST_NAME column.
We need to stop and delete the instance in the node first in interactive or
silent mode.After that asm can be removed using srvctl tool as follows:
srvctl stop asm -n node_name
srvctl remove asm -n node_name
We can verify if ASM has been removed by issuing the following command:
srvctl config asm -n node_name
To restore the backup of your voting disk, issue the dd or ocopy command for
Linux and UNIX systems or ocopy for Windows systems respectively.
On Linux or UNIX systems:
dd if=backup_file_name of=voting_disk_name
On Windows systems, use the ocopy command:
ocopy backup_file_name voting_disk_name
where,
backup_file_name is the name of the voting disk backup file
voting_disk_name is the name of the active voting disk