REAL APPLICATION CLUSTERS SETUP

Operating System Red Hat Enterprise Linux 5.5

Public Network Interface eth0
Private Network Interface eth1
Number of Nodes 2
1st Node lab1
2nd Node lab2
iSCSI Target (server) lab6

1. Install operating system.

2. Install the required RPMs.
3. Configure the network and configure the /etc/hosts file:

127.0.0.1 localhost

200.200.200.71 lab1 Public IP addresses and public node names for

200.200.200.72 lab2 the public network

10.0.0.71 lab1-priv Private IP addresses and private node names

10.0.0.72 lab2-priv for the private network

10.0.0.76 lab6 – Private IP address and node name for the iSCSI target

200.200.200.81 lab1-vip
200.200.200.82 lab2-vip Virtual IP addresses and virtual hostnames

200.200.200.90 crs-scan – SCAN VIP address and SCAN

4. Configure the kernel parameters in the /etc/sysctl.conf file.

5. Configure the Network Time protocol (NTP):

Edit the NTP configuration in the /etc/sysconfig/ntpd file to add the -x flag, which
prevents the time from being adjusted backward. Means if the local clock time is in the
future by 600 seconds relative to the time server than the local clock time is not
adjusted. Restart the network time protocol daemon after you complete this task:

#service ntpd restart

1
6. Size the Shared Memory File System /dev/shm in the /etc/fstab file to support
Automatic Memory Management. By default, it is configured to be half the size of the
system memory. Add the size=2048m option to the entry for /dev/shm in the /etc/fstab
file.

To check the size - #df –m /dev/shm

7. Create the operating system groups:

#groupadd –g 500 oinstall

#groupadd –g 501 asmadmin
#groupadd –g 502 asmdba
#groupadd –g 503 asmoper
#groupadd –g 504 dba
#groupadd –g 505 oper

Oracle ASM Groups for Job Role Separation

1. asmadmin

OSASM group
The Oracle ASM group
This is a required group
Use SQL to connect to the ASM instance as SYSASM
Provides storage administration privilege
Permits mounting and dismounting disk groups, and other storage administration tasks
Does not provide access privileges on the RDBMS instance

2. asmdba

OSDBA group
The ASM DBA group
Use SQL to connect to the ASM instance as SYSDBA
Granted read and write access to files managed by the ASM

3. asmoper

OSOPER group
The ASM operator group
Use SQL to connect to the ASM instance as SYSOPER
Provides the ASM instance administrative privileges, including starting up and stopping the ASM
instance

2
8. Create the operating system users:

#useradd –g oinstall –G asmadmin, asmdba, asmoper grid (software owner for the
Oracle Grid Infrastructure / Cluster Owner)

#useradd –g oinstall –G asmdba, dba, oper oracle (software owner for the Oracle
Database / Database Owner)

9. Specify the password for the grid user and for the oracle user.
10. Edit the /etc/security/limits.conf file to set the resource limits for the grid user and for
the oracle user.
11. Create the Oracle base directory path for the Oracle Grid Infrastructure:

#mkdir –p /u01/app/grid

12. Create the Oracle base directory path for the Oracle database software:

#mkdir –p /u01/app/oracle

13. Assign the ownership of the /u01/app directory to the grid user and the oinstall group:

#chown –R grid:oinstall /u01/app

14. Assign the ownership of the /u01/app/oracle directory to the oracle user and the
oinstall group:

#chown –R oracle:oinstall /u01/app/oracle

15. Set the permission to 775 for the /u01/app directory:

#chmod –R 775 /u01/app

16. Assign the ownership of the /softwares directory to the grid user and the oinstall group:

#chown –R grid:oinstall /softwares

17. Set the permission to 775 for the /softwares directory:

#chmod –R 775 /softwares

3
18.Edit the /home/grid/.bashrc file to set environment variables for the grid user:

export ORACLE_BASE=/u01/app/grid
export ORACLE_HOME=/u01/app/11.2.0/grid
export ORACLE_SID=+ASM1
export PATH=$PATH:$HOME/bin:$ORACLE_HOME/bin

19.Edit the /home/oracle/.bashrc file to set environment variables for the oracle user:

export ORACLE_BASE=/u01/app/oracle
export GRID_HOME=/u01/app/11.2.0/grid
export ORACLE_HOME=/u01/app/oracle/product/11.2.0/dbhome_1
export ORACLE_UNQNAME=orcl
export ORACLE_SID=orcl1
export PATH=$PATH:$HOME/bin:$GRID_HOME/bin:$ORACLE_HOME/bin

20.Configure the iSCSI target using the Openfiler (lab6).

INTERNET SMALL COMPUTER SYSTEM INTERFACE (iSCSI)

 iSCSI (eye-scuzzy), is an abbreviation of Internet Small Computer System Interface, an

Internet Protocol (IP)-based storage networking standard for linking data storage
facilities.

 It enables computers on a network to share storage.

 Primarily used in Storage Area Networks (SANs).

 It is a storage transport protocol that moves I/O traffic over a TCP/IP connection.

 It can be used to transmit data over Local Area Networks (LANs), Wide Area
Networks (WANs), or the Internet.

 It enables location-independent data storage and retrieval.

 The node on which iSCSI storage device is located is called iSCSI target (server) and
other nodes that are allowed to access this storage device are called iSCSI initiators
(clients).

21.Start the iscsi service:

#service iscsi start

4
22.Discover the iSCSI target:

#iscsiadm –m discovery –t st –p lab6

It returns the name of the iSCSI target.

23.Log in to the iSCSI target:

#iscsiadm –m node –T targetname -l

24.Display the iSCSI storage device:

#fdisk –l

25.Create the following disk partitions in the /dev/sdb iSCSI storage device using the fdisk
command:

PARTITION PARTITION PARTITION PURPOSE

NUMBER TYPE SIZE
1 Extended Entire Space (16G)
5 to 6 Logical 500M Primary OCR & Voting Disk
7 to 8 Logical 500M Mirror OCR
9 to 10 Logical 3G Database Files
11 to 12 Logical 3G FRA
13 to 14 Logical 1G ACFS Volume

26.Inform the operating system of partition table changes:

#partprobe

27. Install the required RPMs for ASM.

28. Configure the Oracle ASMLIB driver:

#/usr/sbin/oracleasm configure –i

29. Load the oracleasm kernel module:

#/usr/sbin/oracleasm init

5
30. Mark a disk as an ASM disk:

#/usr/sbin/oracleasm createdisk diskname /dev/sdbn

DISK NAME DISK

GI_D1 /dev/sdb5
GI_D2 /dev/sdb6
GI_D3 /dev/sdb7
GI_D4 /dev/sdb8
DB_D1 /dev/sdb9
DB_D2 /dev/sdb10
FRA_D1 /dev/sdb11
FRA_D2 /dev/sdb12
ACFS_D1 /dev/sdb13
ACFS_D2 /dev/sdb14

31. List ASM disks:

#/usr/sbin/oracleasm listdisks

32. Enter the following command on each node of the cluster to make ASM disks available:

# usr/sbin/oracleasm scandisks

#/usr/sbin/oracleasm listdisks

33. Install the cvqdisk RPM.

Before the actual installation starts, the Cluster Verification Utility (CVU) automatically
checks that all components required for a successful installation of the Oracle
Clusterware and RAC database are installed and configured correctly. By default, the
CVU performs shared storage accessibility checks for the iSCSI storage device. For that,
it requires root privilege to gather information about the iSCSI storage device. The
cvqdisk package contains the mechanism to gather information about the iSCSI storage
device as the root user. This package is applicable on Linux platform only.