White Paper

Configuring Oracle RAC over InfiniBand

Introduction
This document is intended to provide the reader with general instructions on how to do the following:

Install Oracle Cluster Ready Services (CRS) over InfiniBand Install Real Application Clusters (RAC) 10g R2 over InfiniBand Configure a cluster database on RedHat Enterprise Linux (RHEL) 3.0 or 4.0 over InfiniBand

Supporting Documentation
This document is not intended to replace any Oracle RAC installation or deployment guides, but should be used as a supplement to configure Oracle RAC to work with InfiniBand. Cisco InfiniBand documentation can be found at http://www.cisco.com/en/US/products/ps6418/tsd_products_support_category_home.html.

Certifications
Oracle 10g CRS and Oracle 10g RAC Database are certified to use IP over InfiniBand (IPoIB) Linux drivers for cluster interconnection over InfiniBand. Cisco does not provide direct attached InfiniBand storage, but provides a topology-transparent view of the existing Fibre Channel storage solution over InfiniBand. Oracle RAC is certified to work over existing Fibre Channel solutions from major storage vendors; therefore, the Cisco InfiniBandbased Fibre Channel solutions and Oracle RAC can be run without requiring separate certification.

Hardware Configuration Options

The user can decide to use the storage and cluster interconnects to communicate over InfiniBand or to use only one of these. The solutions are independent, and each does not affect the use of other. Cisco recommends the use of one unified fabric for both storage and cluster interconnect traffic. Two configuration types are available for Oracle 10g CRS and Oracle 10g RAC Database with InfiniBand. Non-High-Availability Configuration One port of the servers is connected to an InfiniBand switch that supports the Fibre Channel gateway. Both the storage traffic and interconnect traffic flow over a single Oracle 10g pipe (Figure 1). The ib0 interface and the subpartition of ib0 (ib0:) with the same IP partition key (p_key) and subnet can be used for the public interface and virtual IP respectively. An additional subinterface (ib0):priv can be in a different subnet, which is used for the private interface and Cache Fusion. Alternatively, both ports of the host channel adapter (HCA) can be connected to the InfiniBand switch, ib0 and the subpartition of ib0 (ib0:1) with the same p_key, and the subnet can be used for the public interface and virtual IP respectively. Then ib1 can be used for the private interface.

All contents are Copyright 19922006 Cisco Systems, Inc. All rights reserved. This document is Cisco Public Information.

Page 1 of 8

White Paper

Figure 1.

Non-High-Availability InfiniBand Configuration to Ethernet and Fibre Channel

High-Availability Configuration In the high-availability configuration, both ports of one HCA are logically merged. Both HCA ports are physically connected to the same InfiniBand switch, which supports a Fibre Channel gateway. This hardware configuration provides a high-availability solution for cluster-interconnect traffic as well as for storage traffic (Figure 2). Use the ib0 interface and subinterface ib0:1 in the same subnet with the same p_key for the public interface and virtual IP. Use ib0:priv with a different subnet and different p_key for the private interface or Cache Fusion.
Figure 2. High-Availability InfiniBand Configuration (InfiniBand Ports Dual Connected and Merged)

Compatibility with Blade Servers All of the configurations described here can be implemented in blade server technology. Each configuration has its own pros and cons, and the organization should pick the solution that best serves its requirements and budget.

All contents are Copyright 19922006 Cisco Systems, Inc. All rights reserved. This document is Cisco Public Information.

Page 2 of 8

White Paper

Oracle InfiniBand Implementation

Oracle software does not need any separate binary or patch to run over InfiniBand technology. Oracle components use InfiniBand drivers in the following way:

IPoIB driver

Cluster membership voting for Oracle CRS Membership voting for Automatic Storage Management (ASM) Membership voting for Oracle Cluster File System (OCFS) Cache Fusion and Internet Protocol Control (IPC) for Oracle RAC Database

Small Computer System Interface (SCSI) Remote Direct Memory Access (RDMA) Protocol (SRP) driver

Storage for ASM Storage for raw devices Storage for OCFS

Socket Direct Protocol (SDP) driver

Oracle Net Clients

Figure 3 shows the protocols in an Oracle environment.

Figure 3. Protocols in the Oracle Environment

System Requirements
Please check the Oracle RAC Linux certification matrix for information about currently supported hardware and software. The hardware and software requirements listed in this section are mandatory, as are the hardware requirements mentioned in the Oracle deployment guide, for configuring Oracle RAC over InfiniBand. Hardware Requirements

All contents are Copyright 19922006 Cisco Systems, Inc. All rights reserved. This document is Cisco Public Information.

Page 3 of 8

White Paper

An InfiniBand switch is required for network connectivity. Back-to-back connection between two hosts over HCA is not supported. Depending on the configuration you pick from the supported solutions, you may require an InfiniBand switch with Fibre Channel gateway support.

At least one HCA is required for each host. Depending on the configuration you pick from the supported solutions, you may need two HCAs per host. Blade servers tend to support only a single HCA, but often two host switches.

Fibre Channel gateways are required if you intend to use Fibre Channel storage over InfiniBand. Depending on your I/O requirements, more than one Fibre Channel gateway may be required. You can also use direct attached InfiniBand storage when it is available.

(Optional) An Ethernet gateway can be used in your configuration for connecting outside the fabric for backup.

The host memory environment must meet the minimum requirements, as defined in the Oracle RAC Installation Guide.

Software Requirements

InfiniBand switch software depending on the switch type you useThe latest version can be downloaded from the Cisco support site.

Linux host-side drivers for HCA firmware and InfiniBand driversThe latest version can be downloaded from the Cisco support site.

Configuring Oracle CRS and RAC over InfiniBand

Set Up Fabric 1. 2. Plug in the switch, server, and storage. Connect the hardware, as described in the configuration setup diagram.

Configure InfiniBand Switch 1. Configure the InfiniBand switch, as described in the appropriate switch hardware user guide.

Configure Fibre Channel Storage 1. 2. 3. Configure the Fibre Channel storage, as described in the storage vendors instructions. Configure the Fibre Channel switch, as described in the storage vendors instructions. Configure the Fibre Channel gateway, as described in the Fibre Channel Gateway User Guide. Configure Servers Configure Host Channel Adapters 1. 2. Install and configure the HCAs. Refer to the Host Channel Adapter Guide to install the HCA on each host. After the HCA is installed and configured, all the InfiniBand drivers will be installed as well. 3. Use the tsinstall utility to install host drivers. Do not perform an RPM installation, as RPM will not flash the HCA firmware. Incorrect firmware on the HCA may cause unexpected behavior in your cluster environment. Configure Private and Public Interconnects for Non-High-Availability Environment Each host requires the following IP addresses:

All contents are Copyright 19922006 Cisco Systems, Inc. All rights reserved. This document is Cisco Public Information.

Page 4 of 8

White Paper

One address for the public network One address for the private cluster interconnect One address for the virtual IP

For non-high-availability environments, InfiniBand ports do not need to be merged. The ib0 interface and subpartition of ib0 (ib0:1) with the same p_key and subnet can be used for the public interface and virtual IP respectively. An additional subinterface (ib0):priv can be in a different subnet, which is used for the private interface and Cache Fusion. Alternatively, both ports of the HCA can be connected to the InfiniBand switch, ib0 and subpartition of ib0 (ib0:1) with the same p_key, and the subnet can be used for the public interface and virtual IP respectively. Then ib1 can be used for the private interface. These are the same IP requirements that are described in the Oracle RAC Installation Guide. Additional IP addresses are not required. 1. 2. For the public network, get the address from your network manager. Configure the private interconnect over InfiniBand. All host drivers require at least one interface to be configured for use except SRP. SRP does not need any InfiniBand interfaces to be available. SRP will work as long as InfiniBand ports are connected. 3. Configure the InfiniBand (ib) interface by creating a ifcfg-ib<interface number> file in /etc/sysconfig/network-scripts. 4. 5. Add all the addresses to the file /etc/hosts. (Optional) Make sure that your public network is accessible through the Ethernet gateway, if an Ethernet gateway is used. Refer to the Ethernet Gateway User Guide to configure the Ethernet gateway. 6. 7. Make sure that all the nodes can ping each others InfiniBand interface. Make sure that you can use Remote Copy Protocol (RCP) or Secure Copy Protocol (SCP) to all nodes from the master node over the InfiniBand interface. Configure Public and Private Interconnects for High-Availability Environment For a high-availability configuration, merge both InfiniBand ports of the HCA. Define ib0 and ib0:1 with the same subnet and same p_key for the public interface and ib0:priv with a different subnet and different p_key for the private interface. These are the same IP requirements that are described in the Oracle RAC Installation Guide. Additional IP addresses are not required. 1. 2. For the public network, get the address from your network manager. Configure the private interconnect over InfiniBand. All host drivers require at least one interface to be configured for use. 3. Configure the InfiniBand (ib) interface by creating a ifcfg-ib<interface number> file in /etc/sysconfig/network-scripts. 4. 5. Add all the addresses to the file /etc/hosts. (Optional) Make sure that your public network is accessible through the Ethernet gateway, if an Ethernet gateway is used. Refer to the Ethernet Gateway User Guide to configure the Ethernet gateway.

All contents are Copyright 19922006 Cisco Systems, Inc. All rights reserved. This document is Cisco Public Information.

Page 5 of 8

White Paper

6.

Make sure that all the nodes can ping each others InfiniBand interfaces.

All contents are Copyright 19922006 Cisco Systems, Inc. All rights reserved. This document is Cisco Public Information.

Page 6 of 8

White Paper

7.

Make sure you can use RCP or SCP to all nodes from the master node over the InfiniBand interface.

Install the Shared Disk Subsystem Installation of the disk subsystem is highly dependent on the subsystem you choose. If you choose not to use InfiniBand for your shared subsystem configuration, please refer to your hardware documentation for installation and configuration. Additional drivers and patches may be required. You will also need a host bus adapter and relevant drivers. Oracle RAC over InfiniBand without storage using InfiniBand assumes that the shared subsystem is configured and that all the shared disks are visible to all nodes in the cluster. To use InfiniBand to access the storage subsystem, follow these steps: 1. 2. 3. 4. 5. Configure the Fibre Channel gateway. Follow the usual steps to create a disk partition. Map the raw partition to a raw device to use the raw device for data files. If you are using ASM, no additional work is required. The disk partition can be used in ASM. If you are using OCFS, replace your load_ocfs script under /sbin with the patched load_ocfs script from Cisco. OCFS depends on the MAC address of the network interface card (NIC) or HCA. However, because the MAC address of the HCA can be different across reboots, the load_ocfs script has been changed to incorporate the MAC address dependency. OCFS will not function properly in a Cisco VFrame Server Fabric Virtualization Software environment. Configure Kernel Parameters Configure kernel parameters as required by Oracle CRS and RAC installation. You do not need any additional kernel parameters for configuration over InfiniBand. Install Oracle CRS 1. 2. Install Oracle CRS, as described in the Oracle CRS and RAC Installation Guide. Implement the following exception to the Oracle CRS installation instructions:

a. Select the InfiniBand interface for the private and public interconnect. Also select an interface for virtual IP if InfiniBand is used for the public interface. Install Oracle RAC Database 1. Install Oracle RAC software and create the database cluster, as described in the Oracle CRS and RAC Installation Guide. 2. 3. 4. 5. Create the database cluster, as described in the Oracle CRS and RAC Installation Guide. Use the ib0 and ib0:1 public and virtual interface. Use ib1/ib0:priv for the private interface. For Oracle 10g R1, implement the following exception to the Oracle RAC instructions:

a. Define the CLUSTER_INTERCONNECT parameter. Use the interface IP address of the InfiniBand interface as the CLUSTER_INTERCONNECT value.

All contents are Copyright 19922006 Cisco Systems, Inc. All rights reserved. This document is Cisco Public Information.

Page 7 of 8

White Paper

Configure Oracle Net over IPoIB No additional steps are required to configure Oracle Net to use IPoIB. Traffic will go over IPoIB by default as long as the host in listener.ora and tnsnames.ora are pointing to the hostname of the InfiniBand interface. Configure High Availability 1. 2. No action is required to configure high availability for InfiniBand storage. Configure high availability for IPoIB, as described in the Cisco Linux Host Driver Guide.

Troubleshoot InfiniBand Setup

1. Check the InfiniBand network interface.

a. Use the ifconfig a command. You should see interfaces that begin with ib (ib0, ib1, etc.). b. If there are no ib interfaces, create the config scripts under /etc/sysconfig/network-scripts. The file name will be ifcfg-ib0, ifcfg-ib1, etc. The scripts should bring up the interfaces at boot time automatically. Note: InfiniBand interfaces must be up at boot time or servers will not be able to join the cluster

and will reboot each other to evict the node from the cluster. 2. Check the HCA port status.

Use the Run HCA self-test to verify that HCA is installed and configured properly. $ /usr/local/topspin/sbin/hca_self_test ---- Performing InfiniBand HCA Self Test ---Number of HCAs Detected ............... 1 PCI Device Check ....................... PASS Host Driver Version .................... rhel3-2.4.21-27.ELsmp-3.1.0111 Host Driver RPM Check .................. PASS HCA Type of HCA #0 ..................... Cougar HCA Firmware on HCA #0 ................. v3.2.0 build 3.1.0.111 HCA.Cougar.A1 HCA Firmware Check on HCA #0 ........... PASS Host Driver Initialization ............. PASS Number of HCA Ports Active ............. 2 Port State of Port #0 on HCA #0 ........ UP Port State of Port #1 on HCA #0 ........ UP Error Counter Check .................... PASS Kernel Syslog Check .................... FAIL REASON: Kernel syslog reported: Driver messages [KERNEL_IB][ib_mad_static_compute_base][mad_static.c:126]Couldn't find a suitable network device; setting lid_base to 1 ------------------ DONE --------------------3. Verify that Oracle CRS is communicating over InfiniBand.

a. Reboot the servers and verify that CRS started the init process. b. Check $ORACLE_CRS_HOME/css/log for ocssd<node number>.log for the following: >TRACE: clsc_listen: (0x824cb80) Listening on (ADDRESS=(PROTOCOL=tcp)(HOST=qa-oce4)(PORT=49895))

All contents are Copyright 19922006 Cisco Systems, Inc. All rights reserved. This document is Cisco Public Information.

Page 8 of 8

White Paper

The qa-oce4 equivalent in your system should be the InfiniBand interface. 4. Verify that the cluster interconnect is communicating over InfiniBand.

a. Log in to the sqlplus prompt and run the following: Oradebug setmypid Oradebug ipc Oradebug tracefile_name The last command will show the name of the created trace file. b. Open the newly created trace file in the udump directory and look for the string SKGXPCTX. Sample output: SKGXPCTX: 0xcd1e730 ctx admno 0x7a0be402 admport: SSKGXPT 0xcd1e884 flags SSKGXPT_READPENDING socket no 8 IP 140.87.79.67 sflags SSKGXPT_WRITESSKGXPT_UP info for network 1 c. Alternatively, you can run the following command: oifcfg getif Sample output: eth0 ib0 142.2.166.0 192.169.1.0 global global public cluster_interconnect info for network 0 UDP 9152

Cisco Tested and Recommended Solution

Use InfiniBand for both cluster interconnect and storage. Use high-availability configuration. Use raw devices for OCR and voting. Use Oracle 10g ASM for data files. Do not use Oracle OCFS if Cisco VFrame is used. Use a separate Oracle home for each node.

Printed in USA

C11-378083-00 11/06

All contents are Copyright 19922006 Cisco Systems, Inc. All rights reserved. This document is Cisco Public Information.

Page 9 of 8