Documente Academic
Documente Profesional
Documente Cultură
This article describes the installation of Oracle 10g release 2 (10.2.0.1) RAC on Linux (CentOS 4)
using VMware Server with no additional shared disk devices.
Introduction
Download Software
VMware Server Installation
Virtual Machine Setup
Guest Operating System Installation
Oracle Installation Prerequisites
Install VMware Client Tools
Create Shared Disks
Clone the Virtual Machine
Install the Clusterware Software
Install the Database Software and Create an ASM Instance
Create a Database using the DBCA
TNS Configuration
Check the Status of the RAC
Introduction
One of the biggest obstacles preventing people from setting up test RAC environments is the
requirement for shared storage. In a production environment, shared storage is often provided by
a SAN or high-end NAS device, but both of these options are very expensive when all you want
to do is get some experience installing and using RAC. A cheaper alternative is to use a FireWire
disk enclosure to allow two machines to access the same disk(s), but that still costs money and
requires two servers. A third option is to use VMware Server to fake the shared storage.
Using VMware Server you can run multiple Virtual Machines (VMs) on a single server, allowing
you to run both RAC nodes on a single machine. In additon, it allows you to set up shared virtual
disks, overcoming the obstacle of expensive shared storage.
Before you launch into this installation, here are a few things to consider.
The finished system includes the host operating system, two guest operating systems,
two sets of Oracle Clusterware, two ASM instances and two Database instances all on a
single server. As you can imagine, this requires a significant amount of disk space, CPU
and memory. I tried this installation on a 3.4G Pentium 4 with 2G of memory and it failed
abysmally. When I used a dual 3.0G Xeon server with 4G of memory it worked fine, but it
wasn't exactly fast.
This procedure provides a bare bones installation to get the RAC working. There is no
redundancy in the Clusterware installation or the ASM installation. To add this, simply
create double the amount of shared disks and select the "Normal" redundancy option
when it is offered. Of course, this will take more disk space.
During the virtual disk creation, I always choose not to preallocate the disk space. This
makes virtual disk access slower during the installation, but saves on wasted disk space.
This is not, and should not be considered, a production-ready system. It's simply to allow
you to get used to installing and using RAC.
Download Software
Download the following software.
CentOS 4.x
VMware Server
Oracle 10g (10.2.0.1) CRS and DB software
OK
You must read and accept the End User License Agreement to continue.
Press enter to display it.
VMWARE, INC.
SOFTWARE BETA TEST AGREEMENT
*** Editied out license agreement ***
Do you accept? (yes/no) yes
Thank you.
Configuring fallback GTK+ 2.4 libraries.
In which directory do you want to install the mime type icons?
[/usr/share/icons]
What directory contains your desktop menu entry files? These files have
a
.desktop file extension. [/usr/share/applications]
In which directory do you want to install the application's icon?
[/usr/share/pixmaps]
Trying to find a suitable vmmon module for your running kernel.
The module bld-2.6.9-5.EL-i686smp-RHEL4 loads perfectly in the running
kernel.
[
[
OK
OK
]
]
ENTER-YOUR-SERIAL-
[
[
[
[
[
[
OK
OK
OK
OK
OK
OK
]
]
]
]
]
]
The configuration of VMware Server e.x.p build-22874 for Linux for this
running
kernel completed successfully.
#
The VMware Server Console is started by issuing the command "vmware" at the command
prompt, or by selecting it from the "System Tools" menu.
On the "Connect to Host" dialog, accept the "Local host" option by clicking the "Connect" button.
You are then presented with the main VMware Server Console screen.
Select the "Custom" virtual machine configuration and click the "Next" button.
Select the "Linux" guest operating system option, and set the version to "Red Hat Enterprise
Enter the name "RAC1" and the location should default to "/u01/VM/RAC1", then click the "Next"
button.
Select the required number of processors and click the "Next" button.
Uncheck the "Make this virtual machine private" checkbox and click the "Next" button.
Select the amount of memory to associate with the virtual machine. Remember, you are going to
need two instances, so don't associate too much, but you are going to need approximately 1 Gig
(1024 Meg) to compete the installation successfully.
Accept the "Use bridged networking" option by clicking the "Next" button.
Select the "Create a new virtual disk" option and click the "Next" button.
Accept the "SCSI" option by clicking the "Next" button. It's a virtual disk, so you can still use this
option even if your physical disk is IDE or SATA.
Set the disk size to "10.0" GB and uncheck the "Allocate all disk space now" option. The latter will
make disk access slower, but will save you wasting disk space.
Accept "RAC1.vmdk" as the disk file name and complete the VM creation by clicking the "Finish"
button.
On the "VMware Server Console" screen, click the "Edit virtual machine settings" button.
On the "Virtual Machine Settings" screen, highlight the "Floppy 1" drive and click the "- Remove"
button.
Click the "+ Add" button and select a hardware type of "Ethernet Adapter", then click the "Next"
button.
Click on the "Options" tab, highlight the "Startup/Shutdown" setting and select the "Don't power
on virtual machine" in the "On host startup" option. Finish by clicking the "OK" button.
The virtual machine is now configured so we can start the guest operating system installation.
Continue through the CentOS 4 installation as you would for a normal server. A general pictorial
guide to the installation can be found here. More specifically, it should be a server installation with
a minimum of 2G swap, secure Linux disabled and the following package groups installed:
X Window System
GNOME Desktop Environment
Editors
Graphical Internet
Server Configuration Tools
FTP Server
Development Tools
Legacy Software Development
Administration Tools
System Tools
To be consistent with the rest of the article, the following information should be set during the
installation:
hostname: rac1.localdomain
IP Address eth0: 192.168.2.101 (public address)
Default Gateway eth0: 192.168.2.1 (public address)
IP Address eth1: 192.168.0.101 (private address)
Default Gateway eth1: none
You are free to change the IP addresses to suit your network, but remember to stay consistent
with those adjustments throughout the rest of the article.
Once the basic installation is complete, install the following packages whilst logged in as the root
user.
# From CentOS4 Disk 1
cd /media/cdrecorder/CentOS/RPMS
rpm -Uvh setarch-1*
rpm -Uvh compat-libstdc++-33-3*
rpm -Uvh make-3*
rpm -Uvh glibc-2*
cd /
eject
# From CentOS4 Disk 2
cd /media/cdrecorder/CentOS/RPMS
rpm -Uvh openmotif-2*
rpm -Uvh compat-db-4*
rpm -Uvh gcc-3*
cd /
eject
# From CentOS4 Disk 3
cd /media/cdrecorder/CentOS/RPMS
rpm -Uvh libaio-0*
rpm -Uvh rsh-*
rpm -Uvh compat-gcc-32-3*
rpm -Uvh compat-gcc-32-c++-3*
rpm -Uvh openmotif21*
cd /
eject
This should mount a virtual CD containing the VMware Tools software. Double-click on the CD
icon labelled "VMware Tools" to open the CD. Right-click on the ".rpm" package and select the
"Open with 'Install Packages'" menu option.
Click the "Continue" button on the "Completed System Preparation" screen and wait for the
installation to complete.
Once the package is loaded, the CD should unmount automatically. You must then run the
"vmware-config-tools.pl" script as the root user. The following listing is an example of the output
you should expect.
# vmware-config-tools.pl
Stopping VMware Tools services in the virtual machine:
Guest operating system daemon:
[ OK
Trying to find a suitable vmhgfs module for your running kernel.
[
[
[
[
[
OK
OK
OK
OK
OK
]
]
]
]
]
The configuration of VMware Tools e.x.p build-22874 for Linux for this
running
kernel completed successfully.
You must restart your X session before any mouse or graphics changes
take
effect.
You can now run VMware Tools by invoking the following command:
"/usr/bin/vmware-toolbox" during an XFree86 session.
To use the vmxnet driver, restart networking using the following
commands:
/etc/init.d/network stop
rmmod pcnet32
rmmod vmxnet
depmod -a
modprobe vmxnet
/etc/init.d/network start
Enjoy,
--the VMware team
#
The VMware client tools are now installed.
Select the hardware type of "Hard Disk" and click the "Next" button.
Accept the "Create a new virtual disk" option by clicking the "Next" button.
Set the disk size to "10.0" GB and uncheck the "Allocate all disk space now" option, then click the
"Next" button.
Set the disk name to "/u01/VM/shared/ocr.vmdk" and click the "Advanced" button.
Set the virtual device node to "SCSI 1:1" and the mode to "Independent" and "Persistent", then
click the "Finish" button.
Repeat the previous hard disk creation steps 4 more times, using the following values:
At the end of this process, the virtual machine should look something like the picture below.
Edit the contents of the "/u01/VM/RAC1/RAC1.vmx" file using a text editor, making sure the
following entries are present. Some of the tries will already be present, some will not.
disk.locking = "FALSE"
diskLib.dataCacheMaxSize = "0"
diskLib.dataCacheMaxReadAheadSize = "0"
diskLib.dataCacheMinReadAheadSize = "0"
diskLib.dataCachePageSize = "4096"
diskLib.maxUnsyncedWrites = "0"
scsi1.present = "TRUE"
scsi1.virtualDev = "lsilogic"
scsi1.sharedBus = "VIRTUAL"
scsi1:1.present = "TRUE"
scsi1:1.mode = "independent-persistent"
scsi1:1.fileName = "/u01/VM/shared/ocr.vmdk"
scsi1:1.deviceType = "plainDisk"
scsi1:1.redo = ""
scsi1:2.present = "TRUE"
scsi1:2.mode = "independent-persistent"
scsi1:2.fileName = "/u01/VM/shared/votingdisk.vmdk"
scsi1:2.deviceType = "plainDisk"
scsi1:2.redo = ""
scsi1:3.present = "TRUE"
scsi1:3.mode = "independent-persistent"
scsi1:3.fileName = "/u01/VM/shared/asm1.vmdk"
scsi1:3.deviceType = "plainDisk"
scsi1:3.redo = ""
scsi1:4.present = "TRUE"
scsi1:4.mode = "independent-persistent"
scsi1:4.fileName = "/u01/VM/shared/asm2.vmdk"
scsi1:4.deviceType = "plainDisk"
scsi1:4.redo = ""
scsi1:5.present = "TRUE"
scsi1:5.mode = "independent-persistent"
scsi1:5.fileName = "/u01/VM/shared/asm3.vmdk"
scsi1:5.deviceType = "plainDisk"
scsi1:5.redo = ""
Start the RAC1 virtual machine by clicking the "Power on this virtual machine" button on the
VMware Server Console. When the server has started, log in as the root user so you can partition
the disks. The current disks can be seen by issueing the following commands.
# cd /dev
# ls sd*
sda sda1 sda2 sdb sdc sdd sde sdf
#
Use the "fdisk" command to partition the disks sdb to sdf. The following output shows the
expected fdisk output for the sdb disk.
# fdisk /dev/sdb
Device contains neither a valid DOS partition table, nor Sun, SGI or OSF
disklabel
Building a new DOS disklabel. Changes will remain in memory only,
until you decide to write them. After that, of course, the previous
content won't be recoverable.
The number of cylinders for this disk is set to 1305.
There is nothing wrong with that, but this is larger than 1024,
and could in certain setups cause problems with:
1) software that runs at boot time (e.g., old versions of LILO)
2) booting and partitioning software from other OSs
Start
1
End
1305
Blocks
10482381
Id
83
System
Linux
Ignore any errors during the server startup. We are expecting the networking components to fail
at this point.
Log in to the RAC2 virtual machine as the root user and start the "Network Configuration" tool
(Applications > System Settings > Network).
Highlight the "eth0" interface and click the "Edit" button on the toolbar and alter the IP address to
"192.168.2.102" in the resulting screen.
Click on the "Hardware Device" tab and click the "Probe" button. Then accept the changes by
clicking the "OK" button.
Repeat the process for the "eth1" interface, this time setting the IP Address to "192.168.0.102".
Click on the "DNS" tab and change the host name to "rac2.localdomain", then click on the
"Devices" tab.
Once you are finished, save the changes (File > Save) and activate the network interfaces by
highlighting them and clicking the "Activate" button. Once activated, the screen should look like
the following image.
Edit the /home/oracle/.bash_profile file on the RAC2 node to correct the ORACLE_SID value.
ORACLE_SID=RAC2; export ORACLE_SID
Start the RAC1 virtual machine and restart the RAC2 virtual machine. Whilst starting up, the
"Kudzu" detection screen may be displayed.
Press a key and accept the configuration change on the following screen.
When both nodes have started, check they can both ping all the public and private IP addresses
using the following commands.
ping -c 3 rac1
ping -c 3 rac1-priv
ping -c 3 rac2
ping -c 3 rac2-priv
At this point the virtual IP addresses defined in the /etc/hosts file will not work, so don't bother
testing them.
It's a good idea to take a snapshot of the virtual machines, so you can repeat the following stages
if you run into any problems. To do this, shutdown both virtual machines and issue the following
commands.
# cd /u01/VM
# tar -cvf RAC-PreClusterware.tar RAC1 RAC2 shared
# gzip RAC-PreClusterware.tar
The virtual machine setup is now complete.
Enter the appropriate name and path for the Oracle Home and click the "Next" button.
Wait while the prerequisite checks are done. If you have any failures correct them and retry the
tests before clicking the "Next" button.
You can choose to ignore the warnings from the prerequisite checks and click the "Next" button. If
you do, you will also need to ignore the subsequent warning message by clicking the "Yes"
button.
The "Specify Cluster Configuration" screen shows only the RAC1 node in the cluster. Click the
"Add" button to continue.
Enter the details for the RAC2 node and click the "OK" button.
The "Specific Network Interface Usage" screen defines how each network interface will be used.
Highlight the "eth0" interface and click the "Edit" button.
Set the "eht0" interface type to "Public" and click the "OK" button.
Leave the "eth1" interface as private and click the "Next" button.
Click the "External Redundancy" option, enter "/dev/raw/raw1" as the OCR Location and click the
"Next" button. To have greater redundancy we would need to define another shared disk for an
alternate location.
Click the "External Redundancy" option, enter "/dev/raw/raw2" as the Voting Disk Location and
click the "Next" button. To have greater redundancy we would need to define another shared disk
for an alternate location.
Once the install is complete, run the orainstRoot.sh and root.sh scripts on both nodes as directed
on the following screen.
The output from the orainstRoot.sh file should look something like that listed below.
# cd /u01/app/oracle/oraInventory
# ./orainstRoot.sh
Changing permissions of /u01/app/oracle/oraInventory to 770.
Changing groupname of /u01/app/oracle/oraInventory to oinstall.
The execution of the script is complete
#
The output of the root.sh will vary a little depending on the node it is run on. The following text is
the output from the RAC1 node.
# cd /u01/crs/oracle/product/10.2.0/crs
# ./root.sh
WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/crs/oracle/product' is not owned by root
WARNING: directory '/u01/crs/oracle' is not owned by root
WARNING: directory '/u01/crs' is not owned by root
WARNING: directory '/u01' is not owned by root
Checking to see if Oracle CRS stack is already configured
/etc/oracle does not exist. Creating it now.
Setting the permissions on OCR backup directory
Setting up NS directories
Oracle Cluster Registry configuration upgraded successfully
WARNING: directory '/u01/crs/oracle/product/10.2.0' is not owned by root
WARNING: directory '/u01/crs/oracle/product' is not owned by root
WARNING: directory '/u01/crs/oracle' is not owned by root
WARNING: directory '/u01/crs' is not owned by root
WARNING: directory '/u01' is not owned by root
assigning default hostname rac1 for node 1.
assigning default hostname rac2 for node 2.
Successfully accumulated necessary OCR keys.
Using ports: CSS=49895 CRS=49896 EVMC=49898 and EVMR=49897.
node <nodenumber>: <nodename> <private interconnect name> <hostname>
node 1: rac1 rac1-priv rac1
node 2: rac2 rac2-priv rac2
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
Enter the vitual IP alias and address for each node. Once you enter the first alias, the remaining
values should default automatically. Click the "Next" button to continue.
Wait until the configuration is complete, then click the "OK" button.
You should now return to the "Execute Configuration Scripts" screen on RAC1 and click the "OK"
button.
When the installation is complete, click the "Exit" button to leave the installer.
It's a good idea to take a snapshot of the virtual machines, so you can repeat the following stages
if you run into any problems. To do this, shutdown both virtual machines and issue the following
commands.
# cd /u01/VM
# tar -cvf RAC-PostClusterware.tar RAC1 RAC2 shared
# gzip RAC-PostClusterware.tar
The clusterware installation is now complete.
Select the "Enterprise Edition" option and click the "Next" button.
Enter the name and path for the Oracle Home and click the "Next" button.
Select the "Cluster Install" option and make sure both RAC nodes are selected, the click the
"Next" button.
Wait while the prerequisite checks are done. If you have any failures correct them and retry the
tests before clicking the "Next" button.
You can choose to ignore the warnings from the prerequisite checks and click the "Next" button. If
you do, you will also need to ignore the subsequent warning message by clicking the "Yes"
button.
Select the "Configure Automatic Storage Management (ASM)" option, enter the SYS password
for the ASM instance, then click the "Next" button.
Select the "External" redundancy option (no mirroring), select all three raw disks (raw3, raw4 and
raw5), then click the "Next" button.
Once the installation is complete, wait while the configuration assistants run.
Execute the "root.sh" scripts on both nodes, as instructed on the "Execute Configuration scripts"
screen, then click the "OK" button.
When the installation is complete, click the "Exit" button to leave the installer.
It's a good idea to take a snapshot of the virtual machines, so you can repeat the following stages
if you run into any problems. To do this, shutdown both virtual machines and issue the following
commands.
# cd /u01/VM
# tar -cvf RAC-PostASM.tar RAC1 RAC2 shared
# gzip RAC-PostASM.tar
The database software installation and ASM creation step is now complete.
Start the RAC1 and RAC2 virtual machines, login to RAC1 as the oracle user and start the
Database Configuration Assistant.
dbca
On the "Welcome" screen, select the "Oracle Real Application Clusters database" option and click
the "Next" button.
Select the "Create a Database" option and click the "Next" button.
Select the "Custom Database" option and click the "Next" button.
Enter the values "RAC.WORLD" and "RAC" for the Global Database Name and SID Prefix
respectively, then click the "Next" button.
Accept the management options by clicking the "Next" button. If you are attempting the
installation on a server with limited memory, you may prefer not to configure Enterprise Manager
at this time.
Select the "Automatic Storage Management (ASM)" option, then click the "Next" button.
Select the "DATA" disk group, then click the "Next" button.
Accept the "Use Oracle-Managed Files" database location by the "Next" button.
Check both the "Specify Flash Recovery Area" and "Enable Archiving" options. Enter "+DATA" as
the Flash Recovery Area, then click the "Next" button.
Uncheck all but the "Enterprise Manager Repository" option, then click the "Standard Database
Components..." button.
Uncheck all but the "Oracle JVM" option, then click the "OK" button, followed by the "Next" button
on the previous screen. If you are attempting the installation on a server with limited memory, you
may prefer not to install the JVM at this time.
Accept the current database services configuration by clicking the "Next" button.
Select the "Custom" memory management option and accept the default settings by clicking the
"Next" button.
Once the database creation is complete you are presented with the following screen. Make a note
of the information on the screen and click the "Exit" button.
TNS Configuration
Once the installation is complete, the "$ORACLE_HOME/network/admin/listener.ora" file on each
RAC node will contain entries similar to the following.
LISTENER_RAC1 =
(DESCRIPTION_LIST =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = IPC)(KEY = EXTPROC1))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip.localdomain)(PORT =
1521)(IP = FIRST))
(ADDRESS = (PROTOCOL = TCP)(HOST = 192.168.2.101)(PORT = 1521)(IP
= FIRST))
)
)
SID_LIST_LISTENER_RAC1 =
(SID_LIST =
(SID_DESC =
(SID_NAME = PLSExtProc)
(ORACLE_HOME = /u01/app/oracle/product/10.2.0/db_1)
(PROGRAM = extproc)
)
)
The "$ORACLE_HOME/network/admin/tnsnames.ora" file on each RAC node will contain entries
similar to the following.
RAC =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip.localdomain)(PORT =
1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip.localdomain)(PORT =
1521))
(LOAD_BALANCE = yes)
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = RAC.WORLD)
)
)
LISTENERS_RAC =
(ADDRESS_LIST =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip.localdomain)(PORT =
1521))
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip.localdomain)(PORT =
1521))
)
RAC2 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac2-vip.localdomain)(PORT =
1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = RAC.WORLD)
(INSTANCE_NAME = RAC2)
)
)
RAC1 =
(DESCRIPTION =
(ADDRESS = (PROTOCOL = TCP)(HOST = rac1-vip.localdomain)(PORT =
1521))
(CONNECT_DATA =
(SERVER = DEDICATED)
(SERVICE_NAME = RAC.WORLD)
(INSTANCE_NAME = RAC1)
)
)
This configuration allows direct connections to specific instance, or using a load balanced
connection to the main service.
$ sqlplus / as sysdba
SQL*Plus: Release 10.2.0.1.0 - Production on Tue Apr 18 12:27:11 2006
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining
options
SQL> CONN sys/password@rac1 AS SYSDBA
Connected.
SQL> SELECT instance_name, host_name FROM v$instance;
INSTANCE_NAME
HOST_NAME
------------------------------------------------------------------------------RAC1
rac1.localdomain
SQL> CONN sys/password@rac2 AS SYSDBA
Connected.
SQL> SELECT instance_name, host_name FROM v$instance;
INSTANCE_NAME
HOST_NAME
------------------------------------------------------------------------------RAC2
rac2.localdomain
SQL> CONN sys/password@rac AS SYSDBA
Connected.
SQL> SELECT instance_name, host_name FROM v$instance;
INSTANCE_NAME
HOST_NAME
------------------------------------------------------------------------------RAC1
rac1.localdomain
SQL>
Connected to:
Oracle Database 10g Enterprise Edition Release 10.2.0.1.0 - Production
With the Partitioning, Real Application Clusters, OLAP and Data Mining
options
SQL> SELECT * FROM v$active_instances;
INST_NUMBER
----------1
2
INST_NAME
-----------------------------------------------------------rac1.localdomain:RAC1
rac2.localdomain:RAC2
SQL>
Finally, the GV$ allow you to display global information for the whole RAC.
SQL> SELECT inst_id, username, sid, serial# FROM gv$session WHERE
username IS NOT NULL;
INST_ID
---------1
1
1
1
1
1
1
1
2
2
2
USERNAME
SID
SERIAL#
------------------------------ ---------- ---------SYS
127
2
SYS
128
28
SYS
130
10
SYS
131
4
SYS
133
9
DBSNMP
134
27
DBSNMP
135
1
SYS
153
122
SYSMAN
120
243
DBSNMP
122
37
DBSNMP
124
93
INST_ID
---------2
2
2
2
2
2
2
2
USERNAME
SID
SERIAL#
------------------------------ ---------- ---------SYSMAN
125
2
SYSMAN
127
6
SYS
128
26
SYS
129
30
SYS
130
3
SYS
133
149
SYSMAN
134
58
SYS
136
32
19 rows selected.
SQL>
If you have configured Enterprise Manager, it can be used to view the configuration and current
status of the database.