Yahoo!

White Paper

Oracle Cluster Node Addition

By, Vishal Thammali Yahoo! DBA Team

Yahoo! White Paper

Introduction In Yahoo! , we run more than 1200+ database servers of which 95% are multinode Real Application Clusters enabling us to deliver horizontal scalability as well as high availability on our database systems running on commodity servers to meet the increasing business demands of the customers we serve. The aforementioned features of Real Application Clusters also allow us to add additional nodes to the existing clusters that use Real Application Clusters. One major feature is the ability to add nodes online without any pre-specified downtime. Nodes can be added to an existing pool of RAC nodes for load balancing resources and to achieve Service Level Agreements (SLAs). This paper explains the Oracle 11gR2 Cluster Node Addition automated script usage which reduces the manual intervention with reduction in overall time by ~40%. The step by step process of the script usage is being described below in two different cases: Case 1: Adding a new node to the existing cluster. This case is applicable for adding additional capacity for either Load Balancing or High Availability. Case 2: Adding a node back to the existing cluster. This case is applicable the nodes that crashed with some Hardware issues and need to be re-added to the same cluster to bring it back into rotation. Lets discuss the two cases and the scripts usage in detail with example below: Note: This is for Oracle Grid 11.2.0.2 version only.

Yahoo! White Paper

Case 1: New Node Addition Adding a new node to the cluster for increasing system availability/loadbalancing: /*In the below example we would be adding node ne1-int1addb-003 to the existing cluster nodes "ne1-int1addb-00[1-2] */ a. Verify OS level configs are accurate and IPs are configured correctly. b. Prepare the config file: vi config.env export CLUSTER_CURRENT_NODES=2 #Current number of nodes existing in the cluster export CLUSTER_EXISTING_NODES=ne1-int1addb-001,ne1-int1addb-002 # Current Node names in the cluster, if more than one node put as comma (,) separated nodenames. O_HOME=/home/oracle/product/11.2 # Oracle Home Location G_HOME=/home/oragrid/product/11.2 # Grid Home Location export CLUSTER_NEW_NODES=ne1-int1addb-003 # New Node that would be added/re-added to the cluster export CLUSTER_NEW_VIRTUAL_HOSTNAMES=ne1-int1addb-003-v # New Node with VIP that would be added/re-added to the cluster export TIME_ZONE=PDT # Time Zone c. Setup ssh oracle user equivalence between all nodes. d. Login to all the cluster nodes and run the pre_step.sh ssh to all nodes including the node that would be added newly and run the pre_step.sh script as Root user ex.: ssh to ne1-int1addb-00[1-3].adx.ne1.yahoo.com -bash-3.2$ sudo su - root Password: [root@ne1-int1addb-001 ~]# cd <script_location> [root@ne1-int1addb-001 ~]# sh pre_step.sh

Yahoo! White Paper

-bash-3.2$ sudo su - root Password: [root@ne1-int1addb-002 ~]# cd <script_location> [root@ne1-int1addb-002 ~]# sh pre_step.sh -bash-3.2$ sudo su - root Password: [root@ne1-int1addb-003 ~]# cd <script_location> [root@ne1-int1addb-003 ~]# sh pre_step.sh e. Switch user as Oracle and execute the actual script "new_node_addition.sh" passing the config file as an argument. This adds the new node ne1-int1addb-003 to the existing cluster nodes : ne1int1addb-00[1-2] [root@ne1-int1addb-001 ~]# su oracle [oracle@ne1-int1addb-001]~% sh new_node_addition.sh config.env This completes the new node addition to the 2 node cluster. f. Post Node addition proceed with the instance addition/startup and services configuration/relocation as appropriate. Scripts Download Path: pre_step.sh http://svn.corp.yahoo.com/view/yahoo/ysm/AdSys/DBs/trunk/dba/Oracle_add_ node/pre_step.sh config.env ! this script needs to be modified as per the requirement with the hostnames, oracle home and grid home. http://svn.corp.yahoo.com/view/yahoo/ysm/AdSys/DBs/trunk/dba/Oracle_add_ node/config.env new_node_addition.sh http://svn.corp.yahoo.com/view/yahoo/ysm/AdSys/DBs/trunk/dba/Oracle_add_ node/new_node_addition.sh README.txt http://svn.corp.yahoo.com/view/yahoo/ysm/AdSys/DBs/trunk/dba/Oracle_add_ node/README.txt

Yahoo! White Paper

Case 2: Adding back the node: /*In the below example have taken the host "ne1-int1addb-002" which was the re-imaged host due to corrupt hard disk adding this back to the existing cluster node "ne1-int1addb-001"*/ a. Verify OS level configs are accurate and IPs are configured correctly. b. Prepare the config file: vi config.env export CLUSTER_CURRENT_NODES=1 #Current number of nodes existing in the cluster export CLUSTER_EXISTING_NODES=ne1-int1addb-001 # Current Node names in the cluster, if more than one node put as comma (,) separated nodenames. O_HOME=/home/oracle/product/11.2 # Oracle Home Location G_HOME=/home/oragrid/product/11.2 # Grid Home Location export CLUSTER_NEW_NODES=ne1-int1addb-002 # New Node that would be added/re-added to the cluster export CLUSTER_NEW_VIRTUAL_HOSTNAMES=ne1-int1addb-002-v # New Node with VIP that would be added/re-added to the cluster export TIME_ZONE=PDT # Time Zone c. Setup ssh oracle user equivalence between all nodes. d. Login to all the cluster nodes and run the pre_step.sh ssh to all nodes including the re-imaged node and run the pre_step.sh script as root user ex.: ssh to ne1-int1addb-00[1-2].adx.ne1.yahoo.com -bash-3.2$ sudo su - root Password: [root@ne1-int1addb-001 ~]# cd <script_location> [root@ne1-int1addb-001 ~]# sh pre_step.sh -bash-3.2$ sudo su - root Password: [root@ne1-int1addb-002 ~]# cd <script_location>

Yahoo! White Paper

[root@ne1-int1addb-002 ~]# sh pre_step.sh e. Switch user as oracle and execute the actual script "node_re_add.sh" passing the config.env file as an argument. This adds the node: ne1-int1addb-002 back to the cluster. [root@ne1-int1addb-001 ~]# su oracle [oracle@ne1-int1addb-001]~% sh node_re_add.sh config.env This completes the node addition to the cluster. f. Post Node addition proceed with the instance startup if instance is already showing up in the srvctl config if not add instance and startup the instance and services startup/ relocations. Scripts Download Paths: Scripts have been checked into SVN, can be downloaded by using the below URLs: Pre_step.sh http://svn.corp.yahoo.com/view/yahoo/ysm/AdSys/DBs/trunk/dba/Oracle_add_ node/pre_step.sh config.env ! this script needs to be modified as per the requirement with the hostnames, oracle home and grid home. http://svn.corp.yahoo.com/view/yahoo/ysm/AdSys/DBs/trunk/dba/Oracle_add_ node/config.env node_re_add.sh http://svn.corp.yahoo.com/view/yahoo/ysm/AdSys/DBs/trunk/dba/Oracle_add_ node/node_re_add.sh README.txt http://svn.corp.yahoo.com/view/yahoo/ysm/AdSys/DBs/trunk/dba/Oracle_add_ node/README.txt

Yahoo! White Paper

Conclusion: The above procedures have been time tested and setup in order to reduce the human effort to perform cluster node operations in terms of adding a new node or re-adding an existing node as when the requirement comes up. Weve seen significant improvement in alignment of the standards followed by the team by simplifying these complicated operations of performing such tasks. Reference: Scripts Location: http://svn.corp.yahoo.com/view/yahoo/ysm/AdSys/DBs/trunk/dba/Oracle_add_ node