Sun Cluster 3.2 Student Guide

Unauthorized reproduction or distribution prohibited Copyright 2010, Oracle and/or its affiliates
s
) ha
y
m
om
c
csm Guide
e
t
@
Sun Cluster
3.2 Administration
n
b
e
u
ak Stud
y
(
in this
d
u
al use
m
o Guide
ka e tStudent
g
ban licens
a
b
ES-345 REV C
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
D61742GC20
Edition 2.0
Copyright 2009, Oracle and/or its affiliates. All rights reserved.
Disclaimer
This document contains proprietary information, is provided under a license agreement containing restrictions on use and
disclosure, and is protected by copyright and other intellectual property laws. You may copy and print this document solely for
your own use in an Oracle training course. The document may not be modified or altered in any way. Except as expressly
permitted in your license agreement or allowed by law, you may not use, share, download, upload, copy, print, display,
perform, reproduce, publish, license, post, transmit, or distribute this document in whole or in part without the express
authorization of Oracle.
The information contained in this document is subject to change without notice. If you find any problems in the document,
please report them in writing to: Oracle University, 500 Oracle Parkway, Redwood Shores, California 94065 USA. This
document is not warranted to be error-free.
Sun Microsystems, Inc. Disclaimer
This training manual may include references to materials, offerings, or products that were previously offered by Sun
Microsystems, Inc. Certain materials, offerings, services, or products may no longer be offered or provided.Oracle and its
affiliates cannot be held responsible for any such references should they appear in the text provided.
Restricted Rights Notice
If this documentation is delivered to the U.S. Government or anyone using the documentation on behalf of the U.S.
Government, the following notice is applicable:
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
U.S. GOVERNMENT RIGHTS

The U.S. Governments rights to use, modify, reproduce, release, perform, display, or disclose these training materials are
restricted by the terms of the applicable Oracle license agreement and/or the applicable U.S. Government contract.
Trademark Notice
Oracle and Java are registered trademarks of Oracle and/or its affiliates. Other names may be trademarks of their respective
owners.
AMD, Opteron, the AMD logo, and the AMD Opteron logo are trademarks or registered trademarks of Advanced Micro
Devices. Intel and Intel Xeon are trademarks or registered trademarks of Intel Corporation. All SPARC trademarks are used
under license and are trademarks or registered trademarks of SPARC International, Inc. UNIX is a registered trademark
licensed through X/Open Company, Ltd.
This page intentionally left blank.
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
This page intentionally left blank.
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
Table of Contents
s
) ha
About This Course ...........................................................Preface-xxiii

Course Goals...................................................................... Preface-xxiii
Course Map.........................................................................Preface-xxiv
Topics Not Covered............................................................ Preface-xxv
How Prepared Are You?...................................................Preface-xxvi
Introductions .....................................................................Preface-xxvii
How to Use Course Materials ....................................... Preface-xxviii
Conventions ........................................................................ Preface-xxix
Icons ............................................................................ Preface-xxix
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
a Cluster
Introducing k
Sun
o Hardware and SoftwareSTUFF ABOUT
t
g
e
NAS PG
an14FUZZY
nsPIC ZONE CLUSTER .........................................1-1
bObjectives
e
...........................................................................................
1-1
c
a
li
b
e
u
l
Relevance.............................................................................................
1-2
k
b
a
a
l y sfer Additional Resources ........................................................................ 1-3
u
d
Defining Clustering ........................................................................... 1-4
ab -tran
High-Availability (HA) Platforms.......................................... 1-4
g
n
n
o
a
n
Platforms for Scalable Applications ....................................... 1-5
ab
Sun Cluster 3.2 Hardware and Software Environment............... 1-6
Sun Cluster 3.2 Hardware Environment ........................................ 1-8
Cluster Host Systems................................................................ 1-9
Using Logical Domains (LDoms) as Cluster Host Systems 1-9
Cluster Transport Interface...................................................... 1-9
Public Network Interfaces ..................................................... 1-10
Boot Disks ................................................................................ 1-11
Administrative Workstation.................................................. 1-12
Cluster in a Box ....................................................................... 1-18
Sun Cluster 3.2 Software Support.................................................. 1-19
Software Revisions.................................................................. 1-20
Types of Applications in the Sun Cluster Software Environment....
.................................................................................................... 1-21
Cluster-Unaware (Off-the-Shelf) Applications................... 1-21
Sun Cluster 3.2 Software Data Service Support .......................... 1-26
HA and Scalable Data Service Support ............................... 1-26
vii
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Learning Services, Revision C
Exploring the Sun Cluster Software HA Framework................. 1-28

Node Fault Monitoring and Cluster Membership ............. 1-28
Network Fault Monitoring .................................................... 1-28
Application Traffic Striping................................................... 1-29
Global Storage Services ................................................................... 1-31
Global Naming (DID devices)............................................... 1-31
Global Devices......................................................................... 1-32
Device Files for Global Devices ............................................ 1-34
Global File Systems................................................................. 1-35
Shared QFS Shared-Access File System............................... 1-36
Failover (Non-Global) File Systems in the Cluster ............ 1-37
Solaris 10 Zones................................................................................ 1-38
LDoms and Zones ................................................................... 1-38
Sun Cluster Distinct Zone Features...................................... 1-38
Exercise: Guided Tour of the Training Lab.................................. 1-42
Preparation............................................................................... 1-42
Task ........................................................................................... 1-42
Exercise Summary............................................................................ 1-43
s
) ha
y
m
om
c
csm Guide
e
t Console
@the eCluster
n
b
Exploring Node Console Connectivity
and
u
ak Stud
y
Software............................................................................................
2-1
(
n
s
i
i
Objectives ...........................................................................................
2-1
lud se th
a
Relevance.............................................................................................
2-2
m
u
a
o
Additional k
Resourcest ........................................................................ 2-3
g
n
se Node Consoles ............................................ 2-4
Accessing
the
Cluster
a
n
b
e
a Accessing
lic Serial Port Consoles on Traditional Nodes......... 2-4
b
e
u
l
k rabSun Terminal Concentrator (Sun NTS).................................. 2-7
a
y
Terminal Concentrators ................................................ 2-7
ul nsfe Other
d
Alternatives
to a Terminal Concentrator (TC) for Nodes with
b
a
a
r
t
g
a Serial Port Console.............................................................. 2-8
an non
ab
viii
Accessing the Node Console on Servers with Virtual Consoles

............................................................................................... 2-8
Describing Sun Cluster Console Software for an Administration
Workstation ................................................................................... 2-10
Console Software Installation ............................................... 2-10
Cluster Console Window Variations ................................... 2-10
Cluster Console Tools Look and Feel................................... 2-12
Cluster Console Common Window ..................................... 2-13
Cluster Control Panel ............................................................. 2-14
Configuring Cluster Console Tools............................................... 2-15
Configuring the /etc/clusters File .................................. 2-15
Exercise: Configuring the Administrative Console .................... 2-17
Preparation............................................................................... 2-17
Task 1 Updating Host Name Resolution.......................... 2-18
Task 2 Installing the Cluster Console Software............... 2-18
Task 3 Verifying the Administrative Console Environment .
Sun Cluster 3.2 Administration

2-19
Task 4 Configuring the /etc/clusters File................... 2-19
Task 5 Configuring the /etc/serialports File ............ 2-20
Task 6 Starting the cconsole Tool .................................... 2-21
Task 7 Using the ccp Control Panel .................................. 2-21
Preparing for Installation and Understanding Quorum Devices..3-1
Objectives ........................................................................................... 3-1
Relevance............................................................................................. 3-2
Additional Resources ........................................................................ 3-3
Configuring Cluster Servers............................................................. 3-4
Boot Device Restrictions .......................................................... 3-4
Boot Device Partitioning (ZFS-root)....................................... 3-6
Configuring Cluster Storage Connections ..................................... 3-8
Cluster Topologies .................................................................... 3-8
Clustered Pairs Topology ........................................................ 3-9
Single-Node Cluster Topology ............................................. 3-15
Describing Quorum Votes and Quorum Devices ....................... 3-19
Why Have Quorum Voting at All?....................................... 3-19
Failure Fencing ........................................................................ 3-20
Amnesia Prevention ............................................................... 3-20
Quorum Mathematics and Consequences .......................... 3-21
Two-Node Cluster Quorum Devices ................................... 3-22
Clustered-Pair Quorum Disk Devices ................................. 3-23
Pair+N Quorum Disks ........................................................... 3-24
N+1 Quorum Disks................................................................. 3-25
Quorum Devices in the Scalable Storage Topology........... 3-26
Quorum Server Quorum Devices.................................................. 3-29
Preventing Cluster Amnesia With Persistent Reservations....... 3-31
SCSI-2 and SCSI-3 Reservations............................................ 3-34
SCSI-3 Persistent Group Reservation (PGR) ....................... 3-35
SCSI-3 PGR Scenario With More Than Two Nodes........... 3-35
NAS Quorum and Quorum Server Persistent Reservations ....
............................................................................................. 3-37
Intentional Reservation Delays for Partitions With Fewer
Than Half of the Nodes....................................................... 3-37
Data Fencing ..................................................................................... 3-38
Configuring a Cluster Interconnect............................................... 3-40
Point-to-Point Cluster Interconnect...................................... 3-40
Switch-based Cluster Interconnect....................................... 3-41
Cluster Transport Interface Addresses and Netmask ....... 3-41
Choosing the Cluster Transport Netmask Based on
Anticipated Nodes and Private Subnets........................... 3-42
Identifying Public Network Adapters .......................................... 3-45
Configuring Shared Physical Adapters ........................................ 3-46
Configuring the Public Network .......................................... 3-46
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
ix
Allocating a Different VLAN ID for the Private Network 3-46

Exercise: Preparing for Installation ............................................... 3-47
Preparation............................................................................... 3-47
Task 1 Verifying the Solaris OS.......................................... 3-47
Task 3 Selecting Quorum Devices ..................................... 3-48
Installing and Configuring the Sun Cluster Software Framework ...
.................................................................................................. 4-1
Objectives ........................................................................................... 4-1
Relevance............................................................................................. 4-2
Sun Cluster Software Installation and Configuration .................. 4-4
Introduction to Sun Cluster Package Installation ................ 4-4
Sun Cluster Packaging ............................................................. 4-4
Patches for the OS and for the Sun Cluster Software .......... 4-5
Installing the Sun Cluster Packages With the Java ES Installer4-6
Prerequisites for Installing Sun Cluster Software ................ 4-6
Configuring the User root Environment............................ 4-13
Sun Cluster Framework Configuration ........................................ 4-14
Understanding the installmode Flag ................................ 4-14
Automatic Quorum Configuration (Two-Node Cluster Only)
........................................................................................... 4-15
Automatic Reset of installmode Without Quorum Devices
(Clusters With More Than Two Nodes Only) ................. 4-16
Configuration Information Required to Run scinstall. 4-16
Variations in Interactive scinstall.................................... 4-20
Configuring the Entire Cluster at Once ............................... 4-20
Typical Installation Compared to Custom Installation ..... 4-21
Configuring Using All-at-Once and Typical Modes: Example . 4-22
Configuring Using One-at-a-Time and Custom Modes: Example
(First Node) .................................................................................... 4-30
Configuring Additional Nodes for One-at-a-Time Method:
Example .......................................................................................... 4-43
Solaris OS Files and Settings Automatically Configured by
scinstall...................................................................................... 4-49
Changes to the /etc/hosts File........................................... 4-49
Changes to the /etc/nsswitch.conf File ......................... 4-49
Modifying /etc/hostname.xxx Files to Include IPMP .. 4-50
Modifying the /etc/vfstab File (dedicated partition for /
global/.devices/node@#)............................................... 4-51
Creating the /etc/notrouter File ...................................... 4-51
Modifying the local-mac-address? EEPROM variable 4-51
Automatic Quorum Configuration and installmode Resetting.....
...................................................................................................... 4-52
Manual Quorum Selection.............................................................. 4-53
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

Verifying DID Devices ........................................................... 4-53

Choosing Quorum and Resetting the installmode Attribute
(Two-Node Cluster)............................................................. 4-54
Performing Post-Installation Verification .................................... 4-59
Verifying General Cluster Status .......................................... 4-59
Exercise: Installing the Sun Cluster Server Software.................. 4-64
Task 1 Verifying the Environment .................................... 4-64
Task 2 Updating Local Name Resolution......................... 4-65
Task 3 Installing the Sun Cluster Packages...................... 4-65
Task 4 Configuring a New Cluster The All-Nodes-at-Once
Method .................................................................................. 4-66
Task 5 Configuring a New Cluster The
One-Node-at-a-Time Method............................................. 4-67
Task 6 Verifying an Automatically Selected Quorum Device
(Two-Node Cluster)............................................................. 4-69
Task 7 Configuring a Quorum Device (Three-Node Cluster
or Two-Node Cluster With No Automatic Selection) .... 4-70
s
) ha
ab
y
m
om
c
csm Guide
e
t
@ ......................................5-1
n
b
Performing Basic Cluster Administration
e
u
ak Stud
y
Objectives ...........................................................................................
5-1
(
n
s
i
i
Relevance.............................................................................................
5-2
th
lud ........................................................................
e
a
Additional Resources
5-3
us
am
o
Identifying
Cluster
Daemons ........................................................... 5-4
k
t
e
ngCluster
sCommands
Using
................................................................ 5-8
a
n
b
e
c
a
Commands
Relating
to
Basic Cluster Administration ........ 5-8
li
b
e
u
l
k rab Additional Commands............................................................. 5-9
a
y
Cluster Command Self-Documentation ................................ 5-9
ul nsfe Viewing
d
and Administering Cluster Global Properties............. 5-11
b
a
a
r
t
g
Renaming the Cluster............................................................. 5-11
an non
Setting Other Cluster Properties........................................... 5-12

Viewing and Administering Nodes .............................................. 5-13
Viewing Node Status and Configuration ............................ 5-13
Modifying Node Information ............................................... 5-14
Viewing Software Release Information on a Node............ 5-15
Viewing and Administering Quorum .......................................... 5-17
Viewing Quorum Status and Configuration....................... 5-17
Adding and Removing (and Replacing) Quorum Devices.......
........................................................................................ 5-18
Installing a Quorum Server (Outside the Cluster) ............. 5-18
Adding a Quorum Server Device to a Cluster.................... 5-19
Registering NAS Devices....................................................... 5-20
Adding NetApp NAS iSCSI LUN as a Quorum Device ... 5-20
Adding Sun NAS iSCSI LUN as a Quorum Device........... 5-21
Registering NAS Mounted Directories (for Data Fencing: both
NetApp and Sun NAS)........................................................ 5-22
xi
Viewing and Administering Disk Paths and Settings ................ 5-23

Displaying Disk Paths ............................................................ 5-23
Displaying Disk Path Status .................................................. 5-24
Changing Disk Path Monitoring Settings............................ 5-25
Unmonitoring All Non-Shared Devices and Enabling
reboot_on_path_failure................................................ 5-26
Viewing Settings Related to SCSI-2 and SCSI-3 Disk
Reservations.......................................................................... 5-27
Modifying Properties to use SCSI-3 Reservations for Disks
With Two Paths .................................................................... 5-27
Eliminating SCSI Fencing for Particular Disk Devices...... 5-30
Viewing and Administering Interconnect Components ............ 5-33
Viewing Interconnect Status.................................................. 5-33
Adding New Private Networks ............................................ 5-33
Using the clsetup Command ....................................................... 5-35
Sun Cluster Manager ....................................................................... 5-37
Logging Into the Sun Java Web Console ............................. 5-38
Accessing Sun Cluster Manager ........................................... 5-39
Running Cluster Commands as Non-root User or Role Using Role
Based Access Control (RBAC)..................................................... 5-41
Controlling Clusters ........................................................................ 5-42
Starting and Stopping Cluster Nodes .................................. 5-42
Booting a SPARC Platform Machine With the -x Command
.................................................................................... 5-44
Maintenance Mode Example................................................. 5-48
Modifying Private Network Address and Netmask .................. 5-49
Exercise: Performing Basic Cluster Administration ................... 5-51
Preparation............................................................................... 5-51
Task 1 Verifying Basic Cluster Configuration and Status......
...................................................................................... 5-51
Task 2 Reassigning a Quorum Device .............................. 5-52
Task 3 Adding a Quorum Server Quorum Device ......... 5-52
Task 5 Changing the Cluster Private IP Address Range 5-54
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Using VERITAS Volume Manager With Sun Cluster Software .... 6-1
Objectives ........................................................................................... 6-1
Relevance............................................................................................. 6-2
Introducing VxVM in the Sun Cluster Software Environment... 6-4
Exploring VxVM Disk Groups......................................................... 6-5
Shared Storage Disk Groups ................................................... 6-5
VERITAS Management on Local Disks (Optional in VxVM 4.x
and Above).............................................................................. 6-5
Sun Cluster Management of Disk Groups ............................ 6-7
Sun Cluster Global Devices Within a Disk Group ............... 6-7
xii

VxVM Cluster Feature Used Only for Oracle RAC ............. 6-8
Initializing a VERITAS Volume Manager Disk ............................. 6-9
Traditional Solaris OS Disks and Cross-Platform Data Sharing
(CDS) Disks ............................................................................. 6-9
Reviewing the Basic Objects in a Disk Group.............................. 6-11
Disk Names or Media Names ............................................... 6-11
Subdisk ..................................................................................... 6-11
Plex ............................................................................................ 6-11
Volume ..................................................................................... 6-12
Layered Volume...................................................................... 6-12
Exploring Volume Requirements in the Sun Cluster Environment .
................................................................................................. 6-13
Simple Mirrors......................................................................... 6-13
Mirrored Stripe (Mirror-Stripe) ............................................ 6-14
Striped Mirrors (Stripe-Mirror)............................................. 6-15
Dirty Region Logs for Volumes in the Cluster ................... 6-16
Viewing the Installation and bootdg/rootdg Requirements in the
Sun Cluster Environment ............................................................ 6-17
Requirements for bootdg/rootdg....................................... 6-17
DMP Restrictions in Sun Cluster 3.2 .................................... 6-18
Installing Supported Multipathing Software...................... 6-19
Installing VxVM in the Sun Cluster 3.2 Software Environment 6-20
Using the installer or installvm Utility........................ 6-20
Manually Using vxdiskadm to Encapsulate the OS Disk . 6-24
Configuring a Pre-Existing VxVM for Sun Cluster 3.2
Software................................................................................. 6-24
Creating Shared Disk Groups and Volumes................................ 6-26
Listing Available Disks .......................................................... 6-26
Initializing Disks and Putting Them into a New Disk Group..
......................................................................................... 6-27
Verifying Disk Groups Imported on a Node ...................... 6-27
Building a Mirrored Striped Volume (RAID 0+1).............. 6-29
Building a Striped Mirrored Volume (RAID 1+0).............. 6-30
Examining Hot Relocation.............................................................. 6-31
Registering VxVM Disk Groups .................................................... 6-32
Using the clsetup Command to Register Disk Groups . 6-33
Viewing and Controlling Registered Device Groups ........ 6-34
Managing VxVM Device Groups .................................................. 6-35
Resynchronizing Device Groups .......................................... 6-35
Making Other Changes to Device Groups .......................... 6-35
Putting a Device Group Offline and Back Online .............. 6-35
Using Global and Failover File Systems on VxVM Volumes .... 6-37
Creating File Systems ............................................................. 6-37
Mounting File Systems........................................................... 6-37
Mirroring the Boot Disk With VxVM............................................ 6-39
Exercise: Configuring Volume Management............................... 6-40
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
xiii
Task 1 Selecting Disk Drives .............................................. 6-42

Task 4 Adding vxio on Any Non-Storage Node on Which
You Have Not Installed VxVM.......................................... 6-44
Task 5 Rebooting All Nodes............................................... 6-44
Task 7 Configuring webdg ................................................... 6-46
Task 9 Creating a Global nfs File System ......................... 6-48
Task 10 Creating a Global web File System ...................... 6-48
Task 11 Testing Global File Systems ................................. 6-49
Task 12 Managing Disk Device Groups ........................... 6-49
Task 13 (Optional) Viewing and Managing VxVM Device
Groups Using Sun Cluster Manager ................................. 6-52
s
) ha
Using Solaris Volume Manager With Sun Cluster Software ........ 7-1
Objectives ........................................................................................... 7-1
Relevance............................................................................................. 7-2
Exploring Solaris Volume Manager Disk Space Management ... 7-4
Solaris Volume Manager Partition-Based Disk Space
Management ........................................................................... 7-4
Solaris Volume Manager Disk Space Management With Soft
Partitions ................................................................................. 7-5
Exploring Solaris Volume Manager Disksets ................................ 7-8
Solaris Volume Manager Multi-Owner Disksets (for Oracle RAC) .
.................................................................................................... 7-9
Using Solaris Volume Manager Database Replicas (metadb
replicas) .......................................................................................... 7-10
Local Replica Management ................................................... 7-10
Shared Diskset Replica Management................................... 7-11
Shared Diskset Replica Quorum Mathematics................... 7-11
Shared Diskset Mediators ...................................................... 7-12
Installing Solaris Volume Manager and Tuning the md.conf File...
.................................................................................................. 7-13
Modifying the md.conf File (Solaris 9 OS Only)................ 7-13
Initializing the Local metadb Replicas on Local Disks ............... 7-14
Using DIDs Compared to Using Traditional c#t#d#........ 7-14
Leaving Dedicated Partitions Even When Using ZFS-root ......
......................................................................................... 7-14
Adding the Local metadb Replicas to the Local Disks ...... 7-15
Repartitioning a Mirror Boot Disk and Adding metadb
Replicas.................................................................................. 7-15
Using the metadb or metadb -i Command to Verify metadb
Replicas.................................................................................. 7-15
Creating Shared Disksets and Mediators ..................................... 7-17
Automatic Repartitioning and metadb Placement on Shared
Disksets.................................................................................. 7-18
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
xiv

Using Shared Diskset Disk Space .................................................. 7-20

Building Volumes in Shared Disksets With Soft Partitions of
Mirrors ............................................................................................ 7-21
Using Solaris Volume Manager Status Commands.................... 7-22
Checking Volume Status........................................................ 7-22
Managing Solaris Volume Manager Disksets and Sun Cluster
Device Groups ............................................................................... 7-24
Managing Solaris Volume Manager Device Groups .................. 7-26
Device Group Resynchronization......................................... 7-26
Other Changes to Device Groups ......................................... 7-26
Putting a Device Group Offline ............................................ 7-26
Using Global and Failover File Systems on Shared Diskset Volumes
................................................................................................... 7-27
Creating File Systems ............................................................. 7-27
Mounting File Systems........................................................... 7-27
Using Solaris Volume Manager to Mirror the Boot Disk ........... 7-28
Verifying Partitioning and Local metadbs.......................... 7-28
Building Volumes for Each Partition Except for Root ....... 7-29
Building Volumes for Root Partition ................................... 7-30
Running the Commands........................................................ 7-30
Rebooting and Attaching the Second Submirror ............... 7-31
Exercise: Configuring Solaris Volume Manager ......................... 7-32
Preparation............................................................................... 7-32
Task 1 Initializing the Solaris Volume Manager Local
metadb Replicas ................................................................... 7-34
Task 2 Selecting the Solaris Volume Manager Demo Volume
Disk Drives............................................................................ 7-34
Task 3 Configuring Solaris Volume Manager Diskset and
Volumes for NFS .................................................................. 7-36
Task 3 Configuring Solaris Volume Manager Diskset and
Volumes for Web ................................................................. 7-37
Task 5 Creating a Global nfs File System ........................ 7-38
Task 6 Creating a Global web File System ........................ 7-39
Task 8 Managing Disk Device Groups ............................. 7-40
Task 9 Viewing and Managing Solaris Volume Manager
Device Groups Using Sun Cluster Manager .................... 7-41
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Using ZFS With Sun Cluster Software............................................8-1

Objectives ........................................................................................... 8-1
Relevance............................................................................................. 8-2
ZFS for Sun Cluster Failover Data................................................... 8-4
Using ZFS with Sun Cluster: Volume Management............ 8-4
Using ZFS with Sun Cluster: Eliminating /etc/vfstab
Entries ...................................................................................... 8-4
xv
Building ZFS Pools and File Systems.............................................. 8-5

Growing a ZFS Storage Pool ................................................... 8-6
Quotas and Reservations ......................................................... 8-6
ZFS Snapshots ........................................................................... 8-7
Zpool Ownership: In General and in Sun Cluster ........................ 8-8
ZFS Pool Automatic Failover in the Cluster ......................... 8-8
ZFS Root in Sun Cluster. ................................................................... 8-9
ZFS Root Pool and Sun Cluster............................................... 8-9
Typical ZFS Root Configuration........................................... 8-10
Exercise: Configuring Volume Management............................... 8-13
Task 1 Create a ZFS Pool and File System for Cluster Data ..
............................................................................................ 8-13
Task 2 Create a Snapshot and then Modify your Data... 8-14
Task 3 Manually Migrate your ZFS Pool to Another Node....
......................................................................................... 8-14
Task 4 Examine ZFS Root and Mirror Your Root Pool... 8-15
s
) ha
y
m
om
c
Managing the Public Network With IPMP ......................................

sm uide 9-1
c
e
G
Objectives ...........................................................................................
9-1
t
@
n
b
Relevance.............................................................................................
9-2
u ude
k
a
t
Additional Resources ........................................................................
9-3
(y is S
n
i
Introducing IPMP ..............................................................................
9-4
th ................................................ 9-5
ud Concepts
lIPMP
e
a
Describing General
m Group
usRequirements..................................... 9-5
aIPMP
o
Defining
k
t
e
ng ensStandby
Configuring
Adapters in a Group .......................... 9-7
a
b
c
a
Examining
IPMP
Group
Examples
................................................. 9-8
li
b
e
u
l
k rabSingle IPMP Group With Two Members and No Standby. 9-8
a
y
IPMP Group With Three Members Including a Standby
ul nsfe Single
d
9-9
b
a
tra
g
Two IPMP Groups on Different Subnets............................... 9-9
n
n
a
no
ab
xvi
Two IPMP Groups on the Same Subnet .............................. 9-10

Describing IPMP .............................................................................. 9-11
Network Path Failure Detection ........................................... 9-11
Network Path Failover ........................................................... 9-12
Network Path Failback........................................................... 9-12
Configuring IPMP............................................................................ 9-13
Examining ifconfig Options for IPMP ............................. 9-13
Putting Test Addresses on Physical or Virtual Interfaces. 9-14
Using ifconfig Commands to Configure IPMP .............. 9-14
Configuring the /etc/hostname.xxx Files for IPMP....... 9-15
Using IPV6 Test Address Only.............................................. 9-16
Performing Failover and Failback Manually ............................... 9-18
Configuring IPMP in the Sun Cluster 3.2 Environment............. 9-19
Configuring IPMP Before or After Cluster Installation .... 9-19
Using Same Group Names on Different Nodes ................. 9-19
Understanding Standby and Failback ................................. 9-20

Integrating IPMP Into the Sun Cluster 3.2 Software Environment ..

.................................................................................................... 9-21
Capabilities of the pnmd Daemon in Sun Cluster 3.2 Software.
9-21
Summary of IPMP Cluster Integration ................................ 9-22
Exercise: Configuring and Testing IPMP ..................................... 9-24
Preparation............................................................................... 9-24
Task 1 Verifying the local-mac-address? Variable .... 9-24
Task 2 Verifying the Adapters for the IPMP Group ....... 9-25
Task 3 Verifying or Entering Test Addresses in the /etc/
hosts File .............................................................................. 9-25
Task 4 Creating /etc/hostname.xxx Files ..................... 9-26
Task 6 Verifying IPMP Failover and Failback ................. 9-27
s
Introducing Data Services, Resource Groups, and HA-NFSy......10-1
) ha
Objectives .........................................................................................
m 10-1
m
Relevance...........................................................................................
co e 10-2
m
Additional Resources ......................................................................
10-3
cs Guid
e
Introducing Data Services in the Cluster......................................
10-4
t
@
n
b
Solaris 10 OS Non-GlobalkZones
Act
as
Virtual
Nodes
for
Data
u ude
a
t
y
Services ..................................................................................
10-4
(
S
n
s
i
i
Off-the-Shelf Application.......................................................
10-4
thData Service Agents ................... 10-6
lu3.2d Software
e
a
Sun Cluster
usof a Data Service Agent ....................... 10-7
am
o
Reviewing
Components
k
t
ngFaulteMonitor
se Components ................................................... 10-7
a
n
b
a Introducing
lic Data Service Packaging, Installation, and Registration
b
e
u
l
10-8
k rab
a
y
Data Service Packages and Resource Types ....................... 10-8
ul nsfe Introducing
d
Resources, Resource Groups, and the Resource Group
b
a
a
r
t
g
Manager.......................................................................................... 10-9
n non
a
b
Resource Groups ................................................................... 10-10
a
Resource Group Manager.................................................... 10-12
Describing Failover Resource Groups ........................................ 10-13
Resources and Resource Types ........................................... 10-14
Resource Type Versioning................................................... 10-14
Using Special Resource Types...................................................... 10-15
The SUNW.LogicalHostname Resource Type................... 10-15
The SUNW.SharedAddress Resource Type ....................... 10-15
Guidelines for Using Global and Failover File Systems.. 10-17
HAStoragePlus and ZFS ...................................................... 10-18
Generic Data Service............................................................. 10-18
Understanding Resource Dependencies and Resource Group
Dependencies............................................................................... 10-20
Resource Group Dependencies........................................... 10-21
Configuring Resource and Resource Groups Through Properties...
10-22
xvii
Standard Resource Properties ............................................. 10-22

Some Significant Standard Resource Properties .............. 10-23
Resource Group Properties.................................................. 10-25
Some Significant Resource Group Properties ................... 10-26
Specifying Non-Global Zone Names in Place of Node Names.........
................................................................................................ 10-28
Using the clresourcetype (clrt) Command ......................... 10-29
Registering Resource Types ................................................ 10-29
Unregistering Types ............................................................. 10-30
Configuring Resource Groups Using the clresourcegroup
(clrg) Command ........................................................................ 10-31
Displaying Group Configuration Information ................. 10-31
Configuring a LogicalHostname or a SharedAddress Resource ...
10-33
Examples of Using clrslh to Add a LogicalHostname 10-33
Configuring Other Resources Using the clresource (clrs)
Command..................................................................................... 10-34
Displaying Resource Configuration Information............. 10-34
Complete Resource Group Example for NFS ............................ 10-36
Controlling the State of Resources and Resource Groups ....... 10-38
Introduction to Resource Group State ............................... 10-38
Introduction to Resource State............................................ 10-38
Suspended Resource Groups ....................................................... 10-42
Displaying Resource and Resource Group Status Using the clrg
status and clrs status Commands.................................... 10-43
Example of Status Commands for a Single Failover
Application ......................................................................... 10-43
Using the clsetup Utility for Resource and Resource Group
Operations.................................................................................... 10-44
Using the Data Service Wizards in clsetup and Sun Cluster
Manager........................................................................................ 10-45
Exercise: Installing and Configuring HA-NFS .......................... 10-46
Preparation............................................................................. 10-46
Task 1 Installing and Configuring the HA-NFS Agent and
Server ................................................................................... 10-47
Task 2 Registering and Configuring the Sun Cluster
HA-NFS Data Services ...................................................... 10-49
Task 4 Observing Sun Cluster HA-NFS Failover Behavior ...
................................................................................... 10-51
Task 5 Generating Cluster Failures and Observing Behavior
of the NFS Failover ............................................................ 10-52
Task 9 Viewing and Managing Resources and Resource
Groups Using Sun Cluster Manager ............................... 10-57
Exercise Summary.......................................................................... 10-58
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Configuring Scalable Services and Advanced Resource Group
xviii

Relationships ..................................................................................11-1
Objectives ......................................................................................... 11-1
Relevance........................................................................................... 11-2
Additional Resources ...................................................................... 11-3
Using Scalable Services and Shared Addresses........................... 11-4
Exploring Characteristics of Scalable Services............................. 11-5
File and Data Access............................................................... 11-5
File Locking for Writing Data ............................................... 11-5
Using the SharedAddress Resource............................................. 11-6
Client Affinity.......................................................................... 11-6
Load-Balancing Weights ........................................................ 11-6
Exploring Resource Groups for Scalable Services....................... 11-7
Understanding Properties for Scalable Groups and Services ... 11-9
The Desired_primaries and Maximum_primaries
Properties .............................................................................. 11-9
The Load_balancing_policy Property ............................. 11-9
The Load_balancing_weights Property......................... 11-10
The Resource_dependencies Property ........................... 11-10
Adding Auxiliary Nodes for a SharedAddress Property ....... 11-11
Reviewing Command Examples for a Scalable Service ........... 11-12
Controlling Scalable Resources and Resource Groups............. 11-13
Resource Group Operations ................................................ 11-13
Fault Monitor Operations .................................................... 11-14
Using the clrg status and clrs status Commands for a
Scalable Application ................................................................... 11-15
Advanced Resource Group Relationships ................................. 11-16
Weak Positive Affinities and Weak Negative Affinities . 11-16
Strong Positive Affinities ..................................................... 11-17
Strong Positive Affinity With Failover Delegation .......... 11-18
Exercise: Installing and Configuring Sun Cluster Scalable Service
for Apache.................................................................................... 11-20
Preparation............................................................................. 11-20
Task 1 Preparing for Apache Data Service Configuration.....
.............................................................................. 11-21
Task 2 Configuring the Apache Environment............... 11-21
Task 3 Testing the Server on Each Node Before Configuring
the Data Service Resources ............................................... 11-23
Task 4 Registering and Configuring the Sun Cluster Apache
Data Service ........................................................................ 11-25
Task 5 Verifying Apache Web Server Access ................ 11-26
Task 6 Observing Cluster Failures .................................. 11-26
Task 7 Configuring Advanced Resource Group
Relationships ...................................................................... 11-27
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Performing Supplemental Exercises for
xix
Sun Cluster 3.2 Software............................................................... 12-1

Objectives ......................................................................................... 12-1
Relevance........................................................................................... 12-2
Additional Resources ...................................................................... 12-3
Exercise 1: Running a Clustered Scalable Application in
Non-global Zones.......................................................................... 12-4
Preparation............................................................................... 12-4
Task 1 Configuring and Installing the Zones .................. 12-5
Task 2 Booting the Zones .................................................... 12-6
Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2 Software as
a Failover Application .................................................................. 12-9
Preparation............................................................................... 12-9
Task 2 Creating oracle and dba Accounts.................... 12-11
Task 3A Creating a Shared Storage File System for Oracle
Software (VxVM) ............................................................... 12-12
Task 3B Creating a Shared Storage File System for Oracle
Software (SVM) .................................................................. 12-13
Task 3C: Create a Shared Storage File System for Oracle
Software (ZFS) .................................................................... 12-14
Task 4 Preparing the oracle User Environment .......... 12-14
Task 5 Disabling Access Control of X Server on the Admin
Workstation ........................................................................ 12-15
Task 6 Running the runInstaller Installation Script 12-15
Task 8 Registering the SUNW.HAStoragePlus Type..... 12-20
Task 9 Installing and Registering Oracle Data Service. 12-20
Task 10 Creating Resources and Resource Groups for Oracle
.................................................................................... 12-21
Task 11 Verifying That Oracle Runs Properly in the Cluster
..................................................................................... 12-22
Exercise 3: Running Oracle 11g RAC in Sun Cluster 3.2 Software ...
12-23
Preparation............................................................................. 12-24
Task 1 Selecting the Nodes That Will Run Oracle RAC12-26
Task 3A (If Using VxVM) Installing RAC Framework
Packages for Oracle RAC With VxVM Cluster Volume
Manager............................................................................... 12-28
Task 3B (If Using Solaris Volume Manager) Installing RAC
Framework Packages for Oracle RAC With Solaris Volume
Manager Multi-Owner Disksets ...................................... 12-29
Task 4 Installing Oracle Distributed Lock Manager ..... 12-29
Task 5.5A (If using VxVM): Creating Shared Volume Manager
Framework Resource Group ............................................ 12-30
Task 5.5B (If Using Solaris Volume Manager) Creating
Shared Volume Manager Framework Resource Group12-31
Task 6B Creating Raw Volumes (Solaris Volume Manager) .
...................................................................................... 12-33
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
xx

Task 7 Configuring Oracle Virtual IPs............................ 12-36

Task 9A Creating the dbca_raw_config File (VxVM) 12-38
Task 9B Creating the dbca_raw_config File (Solaris
Volume Manager) .............................................................. 12-39
Task 10 Disabling Access Control on X Server of the Admin
Workstation ........................................................................ 12-39
Task 13 Create Sun Cluster Resources to Control Oracle
RAC Through CRS............................................................. 12-46
Task 14 Verifying That Oracle RAC Works Properly in a
Cluster.................................................................................. 12-49
Terminal Concentrator .................................................................... A-1
Viewing the Terminal Concentrator .............................................. A-2
Setup Port.................................................................................. A-3
Terminal Concentrator Setup Programs............................... A-3
Setting Up the Terminal Concentrator........................................... A-4
Connecting to Port 1 ................................................................ A-4
Enabling Setup Mode .............................................................. A-4
Setting the Terminal Concentrator Load Source ................. A-5
Specifying the Operating System Image .............................. A-6
Setting the Serial Port Variables............................................. A-6
Setting the Port Password....................................................... A-7
Setting a Terminal Concentrator Default Route ........................... A-9
Using Multiple Terminal Concentrators ..................................... A-11
Troubleshooting Terminal Concentrators ................................... A-12
Using the telnet Command to Manually Connect to a Node
................................................................................. A-12
Using the telnet Command to Abort a Node.................. A-12
Using the Terminal Concentrator help Command .......... A-13
Identifying and Resetting a Locked Port ............................ A-13
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Configuring Multi-Initiator SCSI ..................................................... B-1

Multi-Initiator Overview ..................................................................B-2
Installing a Sun StorEdge Multipack Device .................................B-3
Installing a Sun StorEdge D1000 Array ..........................................B-7
The nvramrc Editor and nvedit Keystroke Commands...........B-11
Quorum Server ................................................................................ C-1
Quorum Server Software Installation ............................................ C-2
Configuring the Quorum Server..................................................... C-3
Starting and Stopping a Quorum Server Daemon....................... C-4
Clearing a Cluster From the Quorum Server................................ C-6
Role-Based Access Control ........................................................... D-1
Brief Review of RBAC Terminology .............................................. D-2
Role............................................................................................. D-2
Authorization ........................................................................... D-2
xxi
Assigning Authorizations....................................................... D-3

Command Privileges ............................................................... D-3
Profiles ....................................................................................... D-3
The Basic Solaris User Profile............................................. D-4
RBAC Relationships ................................................................ D-4
Simplified RBAC Authorizations in the Sun Cluster 3.2
Environment ................................................................................... D-5
RBAC in the Sun Cluster 3.1 Environment (for Backward
Compatibility) ....................................................................... D-5
Assigning Sun Cluster Command Privileges to a User...... D-6
Assigning Sun Privileges to a Role........................................ D-6
Logical Domains (LDoms) Guidelines ........................................... E-1
LDom Support....................................................................................E-2
LDom Configuration for Sun Cluster .............................................E-3
Guidelines for LDom Configuration......................................E-3
Guidelines for Multiple LDoms on Same Machine (in Different
Clusters)............................................................................................E-7
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
xxii

Preface
About This Course

Course Goals
s
) ha
y
m
omand
Describe the major Sun Cluster software components
c
functions
csm Guide
e
t console software
@the cluster
Configure access to node consoles and
n
b
e
u
akinstalling
tudSun Cluster software,
Understand the prerequisites
for
y
(
S
into understanding
with particular attention
quorum devices
d
his
t
u
l
e
a theuSun
s Cluster 3.2 software
Install and configure
m
a
o
k
t
e framework
Manage
ngthe Sun
sCluster
a
n
b
e
c
a
liVERITAS
Configure
Volume Manager for Sun Cluster data
b
e
u
l
k
b
ra
ya feConfigure
Solaris Volume Manager for Sun Cluster data
l
u
s
d
ab -tran Configure ZFS for Sun Cluster data
g
n non
a
Manage public network adapters using IPMP in the Sun Cluster 3.2
b
a
Upon completion of this course, you should be able to:
environment
Use resources and resource groups to manage clustered applications
Use storage resources to manage both global and failover file

systems, including ZFS
Configure a failover application (Network File System (NFS))
Configure a load-balanced application (Apache)
Configure applications in non-global zones
Configure HA-Oracle and Oracle Real Application Clusters (RAC)
Preface-xxiii
Course Map
Course Map
The following course map enables you to see what you have
accomplished and where you are going in reference to the course goals.
Product Introduction
Introducing
Sun Cluster
Hardware and Software
s
) ha
Installation
ab
y
m
om
Exploring Node Console
Installing and Configuring
Preparing for Installation
c
mCluster
Connectivity and the
the s
and Understanding
ide
c Sun
u
Cluster Console Software
e
Software
Framework
Quorum Devices
G
t
@
n
b
ku tude
a
y
Operation
n ( this S
i
d
lu se
a
m toBasic
u
aPerforming
k
e
ng eAdministration
sCluster
a
n
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
Customization
d
b
a
a
r
-t
Using
ang non
Using VERITAS
Volume Manager for
Volume Management
Managing the
Public Network
With IPMP
Using ZFS
Solaris Volume Manager

For Volume Mangement
Introducing Data
Services, Resource
Groups, and HA-NFS
Configuring Scalable
Services and Advanced
Resource Group Relationships
Supplemental Exercises
Performing Supplemental
Exercises for Sun
Cluster 3.2 Software
Preface-xxiv

Topics Not Covered
Topics Not Covered

This course does not cover the following topics. Many of the topics listed
on the overhead are described in other courses offered by Sun Services:
Solaris 10 Operating System (OS) Advanced Features or

Differences.
Network administration.
Solaris OS administration
Refer to the Sun Services catalog for specific information and registration.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
About This Course

Preface-xxv
How Prepared Are You?
How Prepared Are You?

To be sure you are prepared to take this course, can you answer yes to the
following questions?
Can you explain virtual volume management terminology, such as

mirroring, striping, concatenation, volumes, and mirror
synchronization?
Can you perform basic Solaris OS administration tasks, such as using

tar and ufsdump commands, creating user accounts, formatting disk
drives, using vi, installing the Solaris OS, installing patches, and
adding packages?
s
) ha
Do you have prior experience with Sun hardware and the

OpenBoot programmable read-only memory (PROM) technology?
y
m
om
c
Are you familiar with general computer hardware, electrostatic

m uide
precautions, and safe handling practices? ecs
G
t
@
n
b
ku tude
a
y
n ( this S
i
d
lu se
a
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba
Preface-xxvi

Introductions
Introductions
Now that you have been introduced to the course, introduce yourself to
each other and the instructor, addressing the items shown in the following
bullets.
Name
Company affiliation
Title, function, and job responsibility
Experience related to topics presented in this course
Reasons for enrolling in this course
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Expectations for this course
About This Course

Preface-xxvii
How to Use Course Materials
How to Use Course Materials

To enable you to succeed in this course, these course materials use a
learning model that is composed of the following components:
Goals You should be able to accomplish the goals after finishing

this course and meeting all of its objectives.
Objectives You should be able to accomplish the objectives after

completing a portion of instructional content. Objectives support
goals and can support other higher-level objectives.
Lecture The instructor will present information specific to the

objective of the module. This information should help you learn the
knowledge and skills necessary to succeed with the activities.
s
) ha
y
m
Activities The activities take on various forms, such as m

exercise,
o an
c
self-check, discussion, and demonstration. Activities help
to e
facilitate
m
d
s
i
mastery of an objective.
ec t Gu
@
n aids to convey a
b dvisual
Visual aids The instructor might useuseveral
e
k
a form.
tuVisual aids commonly
concept, such as a process, in a(y
visual
S
n
contain graphics, animation,
di ande video.
his
t
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab
Preface-xxviii

Conventions
Conventions
The following conventions are used in this course to represent various
training elements and alternative learning resources.
Icons
Additional resources Indicates other references that provide additional
information on the topics described in the module.
s
) ha
y
m
oonmthe current
Discussion Indicates a small-group or class discussion
c
topic is recommended at this time.

csm Guide
!
e
?
@ ent
b
u
ak Stud
y
(
in this
d
u
Note Indicates additional
e that can help students but is not
al information
s
m
u
crucial to their understanding
of the concept being described. Students
kaunderstand
to the
g
e
should be able
to
concept or complete the task without
n ens
a
b
this information.
a
icExamples of notational information include keyword
lminor
b
e
u
shortcuts
and
system adjustments.
l
k
b
a
a
l y sfer
u
d
ab -tranCaution Indicates that there is a risk of personal injury from a
g
n non
nonelectrical hazard, or risk of irreversible damage to data, software, or
a
b
a
the operating system. A caution indicates that the possibility of a hazard

(as opposed to certainty) might happen, depending on the action of the
user.
About This Course

Preface-xxix
Conventions
Typographical Conventions
Courier is used for the names of commands, files, directories,

programming code, and on-screen computer output; for example:
Use ls -al to list all files.
system% You have mail.
Courier is also used to indicate programming constructs, such as class
names, methods, and keywords; for example:
Use the getServletInfo method to get author information.
The java.awt.Dialog class contains Dialog constructor.
s
) ha
y
m
om
c
To list the files in this directory, type:

csm Guide
e
# ls
@ ent
b
u
akof programming
tud
y
(
S
Courier bold is also used for each
line
code that is
n
s
i
i
d
h
t
referenced in a textual description;
example:
lu sfor
e
a
am to u
1 import java.io.*;
k
g nse
njavax.servlet.*;
2 import
a
b
ce
a
3 import
javax.servlet.http.*;
i
l
b
uNotice bthelejavax.servlet interface is imported to allow access to its
k
a
ra methods (Line 2).
y life
cycle
l
e
f
u
s
bd tran
a
g on- Courier italic is used for variables and command-line placeholders
n
a
n
that are replaced with a real name or value; for example:
ab
Courier bold is used for characters and numbers that you type; for
example:
To delete a file, use the rm filename command.

Courier italic bold is used to represent variables whose values are to
be entered by the student as part of an activity; for example:
Type chmod a+rwx filename to grant read, write, and execute
rights for filename to world, group, and users.
Palatino italic is used for book titles, new words or terms, or words that
you want to emphasize; for example:
Read Chapter 6 in the Users Guide.
These are called class options.
Preface-xxx

Module 1
Introducing Sun Cluster Hardware and

Software
s
) ha
Objectives
y
m
om
Upon completion of this module, you should be able to:
c
csm Guide
Define the concept of clustering
e
t
@ andesoftware
n
b
u
Describe the Sun Cluster 3.2 hardware
environment
k tud
a
y
Senvironment
Explain the Sun Clusteri3.2
n (hardware
s
i
d
h
t support
lu3.2 software
e
a
View the Sun Cluster
s
m to u
katypes
Describegthe
eof applications in the Sun Cluster 3.2 software
n
s
a
n
environment
ab lice
b
u Identify
le the Sun Cluster 3.2 software data service support
k
b
a
a
r
l y sfeExplore
u
the Sun Cluster 3.2 software high-availability (HA)
d
b tran framework
a
g onn
a
n
Define global storage services differences
ab
Describe the different Sun Cluster 3.2 zone features
1-1
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved. Sun Services, Revision C
Relevance
Relevance
Discussion The following questions are relevant to understanding the

content of this module:
Are there similarities between the redundant hardware components

in a standalone server and the redundant hardware components in a
cluster?
Why is it impossible to achieve industry standards of high

availability on a standalone server?
Why are a variety of applications that run in the Sun Cluster

software environment said to be cluster-unaware?
s
) ha
y
m
om
c
ide for
csm Guprovide
What services does the Sun Cluster softwareeframework
@ ent
all the applications running in the cluster?
b
u
k tud
asystems
y
Do global devices and global file
(
Shave uses other than for
n
s
i
i
scalable applications in u
th
l thed cluster?
e
a
am to us
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
a
no
How do cluster-unaware applications differ from cluster-aware

applications?
ab
1-2

Additional resources The following references provide additional
information on the topics described in this module:
Sun Cluster System Administration Guide for Solaris OS, part number
820-7458 (for Sun Cluster 3.2 11/09).
Sun Cluster Software Installation Guide for Solaris OS, part number 8207356 (for Sun Cluster 3.2 11/09).
Sun Cluster Concepts Guide for Solaris OS, part number 821-0259 (for
Sun Cluster 3.2 11/09)
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Introducing Sun Cluster Hardware and Software

1-3
Defining Clustering
Defining Clustering
Clustering is a general term that describes a group of two or more separate

servers operating as a harmonious unit. Clusters generally have the
following characteristics:
Separate server nodes, each booting from its own, non-shared, copy of
the OS
Dedicated hardware interconnect, providing a private transport only

between the nodes of the same cluster
Multiported storage, providing paths from at least two nodes in the

cluster to each physical storage device that stores data for the
applications running in the cluster
s
) ha
y
m
om
c
csm Guide
e
t
@
n
b
General goal of providing a platformu
for
HA and
scalability for the
e
ak Stud
applications running in the cluster
y
(
din e thisapplications and cluster-aware
Support for a variety oflu
cluster-unaware
a us
applications
m
a
k e to
g
ban licens
a
High-Availability
b
e (HA) Platforms
u
l
k
b
ya fera
l
u
ns are generally marketed as the only way to provide highbd trClusters
a
a
g on- availability (HA) for the applications that run on them.
n
a
n
ab
Cluster software framework, providing cluster-specific knowledge to

the nodes in the cluster about the health of the hardware and the
health and state of their peer nodes
HA can be defined as a minimization of downtime rather than the

complete elimination of downtime. Many standalone servers themselves
are marketed as providing higher levels of availability than our competitors (or
predecessors). Most true standards of HA cannot be achieved in a
standalone server environment.
HA Standards
HA standards are usually phrased with wording such as provides
5 nines availability. This means 99.999 percent uptime for the application
or about five minutes of downtime per year. One clean server reboot often
already exceeds that amount of downtime.
1-4

Defining Clustering
How Clusters Provide HA (Inter-Node Failover)

Clusters provide an environment where, in the case of any single
hardware or software failure in the cluster, application services and data
are recovered automatically (without human intervention) and quickly
(faster than a server reboot). The existence of the redundant servers in the
cluster and redundant server-storage paths makes this possible.
HA Benefits for Unplanned and Planned Outages

The HA benefit that cluster environments provide involves not only
hardware and software failures, but also planned outages. While a cluster
can automatically relocate applications inside the cluster in the case of
failures, it can also manually relocate services for planned outages. As
such, normal reboots for hardware maintenance in the cluster only affects
the uptime of the applications for as much time as it takes to manually
relocate the applications to different servers in the cluster.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
Fault-Tolerant Servers Are Notya
ankAlternative
tud to HA Clusters
(
S
in this
d
u
l
Many vendors provideaservers
that
e are marketed as fault-tolerant. These
s
m
u
servers are designed
to be able
tolerate any single hardware failure, for
kafailure,
to toprocessing
g
e
example memory
central
unit (CPU) failure, and so on,
n ens
a
b
without
a any downtime.
lic
b
e
u
l
k rab
a
y
l There
e is a common misconception that fault-tolerant servers are an
f
u
s
d
to HA clusters, or that a fault-tolerant server supersedes HA in
ab -tranalternative
some
way.
In
fact, while fault-tolerant servers can hide any hardware
g
n
n
o
a
n
failure, they are not designed to provide especially fast recovery in the
ab
case of a software failure, such as a Solaris OS kernel panic or an
application failure. Recovery in these circumstances on a single
fault-tolerant server might still require a full OS reboot which, as
previously stated, might already exceed maximum downtime permitted
by the HA standards to which you aspire.
Platforms for Scalable Applications

Clusters also provide an integrated hardware and software environment
for scalable applications. Scalability is defined as the ability to increase
application performance by supporting multiple instances of applications
on different nodes in the cluster. These instances are generally accessing
the same data as each other.

1-5
Sun Cluster 3.2 Hardware and Software Environment

Clusters generally do not require a choice between availability and
performance. HA is generally built-in to scalable applications as well as
non-scalable ones. In scalable applications, you might not need to relocate
failed applications because other instances are already running on other
nodes. You might still need to perform recovery on behalf of failed
instances.

The Sun Cluster 3.2 hardware and software environment is Suns latest
generation clustering product. The following features distinguish the Sun
Cluster product from competitors in the field:
s
) ha
y
m
om
c
csm Guide
e
t be physically
@
Global device implementation While data
storage must
n
b
e
u
connected on paths from at least two
ak different
tudnodes in the Sun
y
(
S
Cluster 3.2 hardware and software
all the storage in
in thenvironment,
is
d
u
the cluster is logicallyaavailable
l
sefrom every node in the cluster using
m
u
standard devicea
semantics.
k e to
g
s
This provides
to run applications on nodes that use
an ctheenflexibility
b
a
data
that arelinot even physically connected to the nodes. More
b
u
le about global device file naming and location can be
k
information
b
a
a
l y sfound
er in Global Storage Services on page 1-31.
f
u
d
ab -tran Global file system implementation The Sun Cluster software
g
n non
a
framework provides a global file service independent of any
b
a
Support for 1 to 16 nodes Nodes can be added, removed, and

replaced from the cluster without any interruption of application
service.
particular application running in the cluster, so that the same files

can be accessed on every node of the cluster, regardless of the
storage topology.
1-6
Cluster framework services implemented in the kernel The Sun Cluster

software is tightly integrated with the Solaris 9 OS and Solaris 10 OS
kernels. Node monitoring capability, transport monitoring capability,
and the global device and file system implementation are
implemented in the kernel to provide higher reliability and
performance.
Off-the-shelf application support The Sun Cluster product includes

data service agents for a large variety of cluster-unaware
applications. These are tested programs and fault monitors that

make applications run properly in the cluster environment. A full list

of data service agents is in Sun Cluster 3.1 Software Data Service
Support on page 1-26.
Support for some off-the-shelf applications as scalable applications with

built-in load balancing (global interfaces) The scalable application
feature provides a single Internet Protocol (IP) address and loadbalancing service for some applications, such as Apache Web Server
and Sun Java System Web Server. Clients outside the cluster see
the multiple node instances of the service as a single service with a
single IP address.
Tight integration with Solaris 10 zones The Sun Cluster 3.2 framework
is aware of Solaris 10 zones and can manage failover and scalable
applications running in non-global zones.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

1-7
Sun Cluster 3.2 Hardware Environment

The Sun Cluster 3.2 hardware environment supports a maximum of
sixteen nodes. Figure 1-1 demonstrates the hardware components of a
typical two-node cluster:
Cluster nodes that are running Solaris 9 OS, or Solaris 10 OS (specific

Solaris updates listed later in this module)
Each node must run the same revision and same update of the OS
Separate boot disks on each node (with a preference for mirrored

boot disks)
One or more public network interfaces per system per subnet (with a
preferred minimum of at least two)
s
) ha
y
m
A redundant private cluster transport interface

om
c
Dual-hosted, mirrored disk storage

csm Guide
e
t method)
@
One terminal concentrator (or any other
consoleeaccess
n
b
u
k tud
Administrative workstation (ya
n this S
i
d
lu se
a
Administration
am to u
Workstation
k
Network
ng ense
a
b
a
lic
b
e
u
l
Public net
b
Serialak
a
r
y
port
IPMP
ul A nsfe
Terminal
d
group
b
a
Concentrator
a
r
t
g
an non
ab
Node 1
Redundant
Transport
Node 2
Boot Disks
Boot Disks
Multihost Storage
Figure 1-1
1-8
Multihost Storage
Minimal Sun Cluster 3.2 Hardware Environment

Cluster Host Systems

A wide range of Sun server platforms are supported for use in the
clustered environment. These range from small rackmounted servers
(SPARC and x86 servers) up to Suns largest enterprise-level servers,
including Sun Fire 25K and M-series servers
Heterogeneous environments are supported (that is, different types of
servers used as nodes in the same cluster), based on the network and
storage host adapters used, not on the servers themselves.
You cannot mix SPARC and x86 servers in the same cluster.
s
) ha
ab
y
m
Using Logical Domains (LDoms) as ClusteroHost

m
c
e
Systems
m
d
s
i
ec t Gu
@
n with Sun Cluster
b beginning
e
u
LDoms are fully supported as clusterknodes,
d
tu domains are supported.
ya andSguest
3.2 2/08 (Update 1). Both I/O domains
(
n
s
A discussion of requirements
for configuring guest
di andebest
hipractices
t
u
l
a
LDoms for use as Sun
s is included in an appendix.
unodes
am Cluster
o
k
t
e and match one or more LDoms and one or more
nglegaletonsmix
a
It is perfectly
b
c using LDoms in the same cluster.
a nodeslinot
physical
b
e
u
l
k rab
a
y
ul nsfe
d
b
a Cluster
tra Transport Interface
g
n
n
a
no
All nodes in a cluster are linked by a private cluster transport. The
transport can be used for the following purposes:
Cluster-wide monitoring and recovery
Global data access (transparent to applications)
Application-specific transport for cluster-aware applications (such as

Oracle RAC)
It is highly recommended to use two separate private networks that form

the cluster transport. You can have more than two private networks (and
you can add more later). More private networks can provide a
performance benefit in certain circumstances because global data access
traffic is striped across all of the transports.

1-9
Sun Cluster 3.2 now allows you to build configurations with a single
private network forming the cluster transport. This would be
recommended in production only if the single private network is already
redundant (using a lower level device aggregation).
Crossover cables are often used in a two-node cluster. Switches are
optional when you have two nodes and are required for more than two
nodes.
The following types of cluster transport hardware are supported:
Ethernet (100mb, 1gb, and 10gb adapters)
vnet virtual ethernet adapters in Logical Domains (LDoms)
s
) ha
y
m
om
c
csm Guide
Infiniband (Solaris 10 OS only)
e
@ ent
b
u
This is a relatively new industry standard
used outside
d
k interconnect
u
a
t
y
of the Sun Cluster environment( for interconnecting
a variety of hosts
S
n Cluster
s
i
diSun
h
and storage devices. Inlu
the
environment
it is supported
t
e
a
only as an interconnect
us hosts, not between hosts and
am between
o
k
t
storage devices.
ng ense
a
b
a
lic
b
e
u
l
kNetwork
Public
a
ab Interfaces
r
y
l
e
f
ns
bdu trEach
a
a
node must have public network interfaces under control of the
ang non Solaris OS IP Multipathing (IPMP) software. It is recommended to have at
Scalable coherent interface (SCI) intended for remote shared memory

(RSM) applications. The SCI interconnect is supported only in
clusters with a maximum of four nodes.
ab
least two interfaces in each IPMP group. You may configure as many
IPMP groups as required.
A Sun Cluster server is not allowed to act as a router.
Not all available network interface cards (NICs) are supported in the Sun
Cluster 3.2 hardware environment.
1-10

Cluster Disk Storage

The Sun Cluster hardware environment can use several Sun storage
models. They must all accept multihost connections. The Sun StorEdge
6120 array has a single connection and must be used with a hub or a
switch.
Some data storage arrays support only two physically connected nodes.
Many other storage configurations support more than two nodes
connected to the storage.
ZFS, VERITAS Volume Manager or Solaris Volume Manager software is
typically used to mirror the storage across controllers. You can choose not
to use any Volume Manager if each node has multipathed access to HA
hardware redundant array of independent disks (RAID) storage.
s
) ha
y
m
om
c
csm Guide
e
@ ent
Boot Disks
b
u
ak Stud
y
(
The Sun Cluster 3.2 environment
disks for each node be
in requires
d
thisarethatnotboot
u
local to the node. That a
is,l the boot
disks
connected
or not visible to
e
s
m
u
any other node. k
For
boot device was connected through a
a example,
to ifit the
g
e
Storage Area
Network
(SAN),
would
still be supported if the LUN is
n ens
a
b
not visible
a to any
lic other nodes.
b
e
u
l
k rab
a
y
l Itsisferecommended to mirror boot disks using any of the following
u
d
ab -tranmethods; the first two listed are recommended for their simplicity:
g
n non
ZFS (supported for boot disks starting in Solaris 10 10/08)
a
b
a
Built-in RAID controller for local disks, on systems that have it
Veritas Volume Manager

1-11
Terminal Concentrator (or Other Console Access)
Servers supported in the Sun Cluster environment have a variety of

console access mechanisms.
If you are using a serial port console access mechanism (ttya), then you
might likely have a terminal concentrator in order to provide the
convenience of remote access to your node consoles.A terminal
concentrator (TC) is a device that provides data translation from the
network to serial port interfaces. Each of the serial port outputs connects
to a separate node in the cluster through serial port A.
s
) ha
y
m
There is always a trade-off between convenience and security.m

You might
o nodes,
c
prefer to have only dumb-terminal console access to the cluster
e and
m usecurity
d
s
i
keep these terminals behind locked doors requiringcstringent
eless convenient
G
checks to open them. This is acceptable (although
to
t
@
n
b
e
u
administer) for Sun Cluster 3.2 hardware
akas well.
tud
y
(
S
in this
d
u
al use
Administrative Workstation
m
ka e to
g
s
bantheliSun
enCluster
Included a
with
software is the administration console
c
b
e
software,
which
can
be
installed
on any SPARC or x86 Solaris OS
u
l
k
b
a
a
workstation.
The
software
can
be
a convenience in managing the multiple
y fer
l
u
s
n of the cluster from a centralized location. It does not affect the
bd trnodes
a
a
- cluster in any other way.
ang non
ab
1-12

Clusters With More Than Two Nodes
In clusters with more than two nodes:
Switches are required for the transport. In the typical case of a

redundant transport, there must be separate, isolated switches for
each private network. Each node is connected to each switch to form
the redundant transport.
A variety of storage topologies are supported. Figure 1-2 on

page 1-13 shows an example of a Pair + N topology. All of the nodes
in the cluster have access to the data in the shared storage through
the global devices and global file system features of the Sun Cluster
environment.
s
) ha
y
m
om
c
m uide
csPublic
e
Gnet
t
@
Redundant
n
b
Transport
ku tude
a
y
n ( this S
i
d
lu se
a
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
Node
Node
Node
ul nsfeNode
d
b
a
tra
g
n
n
no
aba
See Module 3, Preparing for Installation and Understanding
Quorum Devices for more information about these subjects.
Figure 1-2
Pair + N Topology

1-13
Cluster With Network Attached Storage (NAS) Devices
Sun Cluster 3.2 provides a framework for supporting network-attached

storage (NAS) devices as the shared data storage available to the Sun
Cluster nodes without requiring any other form of shared storage devices
(Figure 1-3).
A NAS device provides file services to all of the cluster nodes through the
Network File System (NFS), another network file system, or a network
virtual disk protocol (iSCSI). File and sisk services can run between the
cluster nodes and the network storage server on a dedicated subnet or,
less likely, on the same public subnet providing access to the clients of the
cluster services. In other words, file traffic is supported on any network
except those that make up the cluster interconnect.
s
) ha
y
m
omNAS
c
In the current release, Sun Cluster nodes can use the following
devices:
csm Guide
e
@ ent
b
iSCSI only: ST2510
u
k tud
yaStorageTek
(
S 5200 and 5300 Series
NFS only: NetApp Filer andnSun
s
i
i
d
h
t RAC only): Sun StorEdge 7000
lu for Oracle
iSCSI only (NFS allowed
e
a
s
am to u
Series (Amber k
Road)
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba
Switch
Switch
Node 1
Node 2
Node 3
Node 4
Network (not the cluster tranport)
NAS Device (Sun or Netapp)
Figure 1-3
1-14
Cluster With NAS Device as Only Shared Storage

Cluster With Controller-Based Data Replication

Sun Cluster 3.2 supports a variation on the traditional hardware
environment. In this variation data is replicated between nodes, rather than
each physical storage array being dual hosted. When Node 1 takes
ownership of the data it is reading only its own copy, and similarly for
Node 2 (see Figure 1-4).
Switch
Switch
Node 1
Node 2
s
) ha
y
m
om
Data Replication
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
Storage
inReplication
d
hisor Medium
t
u
l
(Private fiber,
public
network,
some combination)
e
a us
m
ka of Controller-Based
to
Figure 1-4 ng
Diagramse
Data Replication
a
n
b
e
a
lic
b
e
u
l
kThe original
b release of Sun Cluster 3.2 supported only Hitachi Truecopy.
a
a
r
y
feCluster 3.2 2/08 (Update 1) added support for EMC Symmetrix
ul nSun
s
d
b
Remote
Data Facility (SRDF). Sun Cluster 3.2 11/09 (Update 3) adds
a
a
r
t
g
support for Hitachi Universal Replicator.
n non
a
b
a
The data replication topology is ideal for wider area clusters where the data
replication solution is preferred to the extra connectivity that would be
involved to actually connect the storage to nodes that are far apart. The
reasons for this preference can be:
There might be an existing data replication framework in place, and

using it within the cluster might be cheaper than physically crosscabling all nodes to all storage.
You might benefit from each node having a physically local copy of
the data because it can guarantee optimal data read behavior.

1-15
Cluster With Shared Physical Interconnects
In Sun Cluster 3.2 certain types of transport adapters can be used as both
public network adapters and private network transport adapters
simultaneously. The purpose of this feature is to allow certain types of
servers that can never have more than two physical network adapters,
such as servers in a blade architecture, to be used as Sun Cluster nodes
with a fully redundant transport.
Using this feature, such servers could use each physical adapter as both a
single transport adapter and a single public network adapter. Both public
and private networks can still be fully redundant.
s
) ha
Tagged Virtual Local Area Networks (VLANs) and Adaptermy

Restrictions
com
e
m
d
s
i
c support
ua
This feature makes use of network device driversethat
G
t
@
n how packets
b specifies
specification called 802.1q tagged VLANs. 802.1q
e
u
d
k
a trunked)
from multiple VLANs can be passed(y
(or
tuover the same physical
S
n
link by including a VLAN identifier
(VLAN-ID)
is in network packets. This
di ethe
thheader
VLAN-ID is encapsulateda
aslupart of
information at the media
s
m
u
access control (MAC)
by the switches and host
alevel and
kdrivers).
tointerpreted
g
e
adapters (or device
On
the
host
end,
multiple virtual adapters
n ens
a
b
can be configured
c top of a physical adapter. Each virtual adapter
a
lion
b
e
accepts
and
sends
packets
only for a particular VLAN.
u
l
k
b
a
a
l y sfer
u
d
n of the supported Sun Cluster interconnect devices support
asubset
ab -trAtagged
g
VLAN. The list, at the time of writing this course, is:
n
n
o
a
n
ab
1-16
ce
bge
ixge
ixgb
e1000g
ipge
vnet (in LDoms)

Illustration of Shared Private and Public Networks
Figure 1-5 is a diagram of a two-node cluster where each node has only
two physical network adapters that are capable of tagged VLANs.
To public net
V3
V3
Adapter 1
V1
Adapter 1
Switch 0
V1
as
Node 1
Node y
2) h
m
m
co e
m
cs GuV3id
V3
e
@ ent
b
Adapter 2
u
Adapter 2
k tud
a
y
1 S
V2
V2
n (Switch
s
i
i
d
h
lu se t To public net
a
aV1m to uPublic VLAN V3
k
Private VLAN V2
Private VLAN
g nse
n
a
ab lice
b
u
le
k
b
a
a
l y Figure
er 1-5 Adapters Being Shared Between Public and Private Nets
f
u
s
d
ab -tran
g
n
As shown in Figure 1-5, the switches are interconnected to support the
ban no
two adapters on each node existing in the same IPMP group for public
network address failover. In addition, each switch is being used as a
private network interconnect. The isolation of traffic between the private
network and public network adapters is controlled by the VLAN-ID that
is assigned to the adapters themselves, which pass the information to the
network fabric. This requires switches that are capable of tagged VLANs.

1-17
Sun Cluster Hardware Redundancy Features
The following items summarize the generally required and optional

hardware redundancy features in the Sun Cluster hardware environment:
Redundant server nodes are required.
Redundant transport is highly recommended.
HA access to data storage is required. That is, at least one of the

following is required.
Mirroring across controllers for Just a Bunch Of Disks (JBOD) or

for hardware RAID devices without multipathing
Multipathing from each connected node to hardware RAID

devices
s
) ha
y
m
om
Redundant public network interfaces per subnet are recommended.
c
csm Guide
Mirrored boot disks are recommended.
e
@ ent
b
u
d as possible. For
You should locate redundant components
ak as far
tuapart
y
(
S
example, on a system with multiple
(I/O) boards, you
in input/output
is
d
h
t
u
l
should put the redundanta
transport
interfaces,
the redundant public nets,
e
s
m
u
and the redundant storage
array
controllers
on
two different I/O boards.
a
o
k
t
ng ense
a
b
a
lic
Clusterkuinba Box
e
l
a erab
y
l
f
u ThenSun
s
d
b
Cluster 3.2 hardware environment supports clusters with both
a
a
r
t
g
an non (or all) nodes in the same physical frame. This is true for:
ab
Servers supporting hardware domains
Servers supporting LDoms
You should take as many redundancy precautions as possible, for

example, running multiple domains in different segments on
Sun Fire 3x00 through 6x00 servers and segmenting across the power
plane on Sun Fire 6x00 servers.
A cluster formed of multiple LDoms in a single box is particularly
susceptible to a single point of failure in the control LDom of the physical
machine. Such a cluster is supported but does not provide true high
availability.
1-18

Sun Cluster 3.2 Software Support

To function as a cluster member, the following types of software must be
installed on every Sun Cluster node:
Solaris OS software
Sun Cluster software
Data service application software
Logical volume management
An exception is a configuration that uses hardware RAID. This

configuration might not require a software volume manager.
s
) ha
y
m
Figure 1-6 provides a high-level overview of the softwareocomponents

m
c
that work together to create the Sun Cluster software

environment.
e
m
d
s
i
ec t Gu
@
b den
u
k
ya Stu
(
Cluster-Unaware
n
di e this
Applications
u
l
a us
userland
m
a
o
k
t
gService nse
Data
nAgents
a
b
ce
a
i
l
b
u
le
k
b
a
a
r Cluster Framework (userland daemons)
l y sfeSun
u
d
ab -tran Sun Cluster Framework (kernel Portions)
g
n non
a
b
a
Solaris OS and
Logical Volume Managment
Figure 1-6
kernel
Sun Cluster Software Layered Model

1-19
Software Revisions
The following software revisions are supported by Sun Cluster 3.2

software:
Solaris OS versions for Sun Cluster 3.2 updates earlier than Update 2
(1/09) (each node must run the same version):
Solaris 9 OS 9/05 or 9/05 HW (Update 8 or 9) (SPARC only)
Solaris 10 OS 11/06 through 10/08 (Updates 3 through 6)
Solaris OS versions for Sun Cluster 3.2 Update 2 (1/09) (each node
must run the same version):
s
) ha
Solaris 9 OS 9/05 or 9/05 HW (Update 8 or 9) (SPARC only)
Solaris 10 OS 5/08 through 5/09 (Updates 5 through 7)
y
m
m
o
c
node
Solaris OS versions for Sun Cluster 3.2 Update 3 (11/09)
(each
e
m
d
s
i
must run the same version):
ec t Gu
@
b 8doren9) (SPARC only)
Solaris 9 OS 9/05 or 9/05 HWk(Update
u
a (Updates
tu 7 and 8)
y10/09
(
S
Solaris 10 OS 5/09 through
n
i
dversions
hisSun Cluster 3.2 updates earlier
t
u
l
Veritas Volume Manager
for
e
a
am to us
than Update 2k(1/09)
e 9 OS/Solaris 10 OS (SPARC)
ng4.1 e
sSolaris
a
n
VxVM
MP1
b
a
licMP2 for Solaris 10 OS (x86)
b
e
VxVM
4.1
u
l
k rab
a
y
ul nsfe VxVM 5.0 Solaris 9 OS/Solaris 10 OS (SPARC)
d
b
a
tra Veritas Volume Manager Versions for Sun Cluster 3.2 Update 2
g
n
n
a
(1/09)
no
ab
Veritas Volume Manager Versions for Sun Cluster 3.2 Update 3

(11/09)
VxVM 5.0 MP3 RP1 Solaris 9 OS/Solaris 10 OS (SPARC and

x86)
VxVM 5.0 MP3 RP2 and RP3 (SPARC and x86)
Solaris Volume Manager revisions part of the base OS in the

Solaris 9 OS and Solaris 10 OS
Note New Solaris 10 OS updates might have some lag before being
officially supported in the cluster. If there is any question, consult Sun
support personnel who have access to the latest support matrices.
1-20

Types of Applications in the Sun Cluster Software Environment
Types of Applications in the Sun Cluster Software

Environment
The Sun Cluster software environment supports both cluster-unaware and
cluster-aware applications.
Cluster-Unaware (Off-the-Shelf) Applications

A cluster-unaware application is an application that typically runs on a
single server.
s
) ha
ab
y
m
om
c
e the Sun
idin
Two main categories of cluster-unaware applications
run
csmcan
u
e
G
t
Cluster 3.2 environment. These are described
on the
following
pages of
@
n
b
e
u
d 11. The applications
this module and also in detail in modules
ak S10tuand
y
(
include the following:
in this
d
u
Failover applications
al use
m
ka e to
Scalableg
applications
ban licens
a
bcommon
The
e elements of all of these applications are the following:
u
l
k
b
ra clusters resource group manager (RGM) coordinates all stopping
ya feThe
l
u
s
bd tran and starting of the applications, which are never started and stopped
a
by traditional Solaris OS boot methods.
ang non
The majority of applications that run in a cluster are in the clusterunaware category and are part of the main focus of this course.
A data service agent for the application provides the glue to make it
work properly in the Sun Cluster software environment. This
includes methods to start and stop the application appropriately in
the cluster, as well as fault monitors specific to that application.

1-21
Failover Applications
The failover model is the easiest to support in the cluster. Failover
applications run on only one node of the cluster at a time. The cluster
provides HA by providing automatic restart on the same node or a
different node of the cluster.
Failover services are usually paired with an application IP address. This is
an IP address that always fails over from node to node along with the
application. In this way, clients outside the cluster see a logical host name
with no knowledge of which node a service is running on. The client
should not even be able to tell that the service is running in a cluster.
s
) ha
y
m
om
c
idean IP
Multiple failover applications in the same resource group
share
csmcan
u
e
G
address, with the restriction that they must all@
fail overnto
the same node
t
b
e
u
together as shown in Figure 1-7.
ak Stud
y
(
in this
d
u
al use
Application IPs
m
a
to
Applications g k
e
an cMounts)
ns
(Local File
bSystem
e
a
li
Group
bResource
e
u
l
k
a erab
y
l
u nsf Node 1
Node 2
d
b
a
a
r
-t
ang non Figure 1-7 Multiple Failover Applications in the Same Resource Group
Note Both IPV4 addresses and IPV6 addresses are supported.
ab
1-22

Scalable Applications
Scalable applications involve running multiple instances of an application
in the same cluster and making it look like a single service by means of a
global interface that provides a single IP address and load balancing.
While scalable applications (Figure 1-8) are still off-the-shelf, not every
application can be made to run as a scalable application in the Sun Cluster
3.2 software environment. Applications that write data without any type
of locking mechanism might work as failover applications but do not
work as scalable applications.
s
) ha
y
m
om
c
Client 1
Client 2
Client
3
sm uide
Requests
Requestsec
G
t
Network
@
n
b
ku tude
a
y
Node 1
n ( this S
i
d
lu se
Global Interface
a
Web
xyz.com
am to u
k
Page
ng enseTransport
a
Request
b
aDistributionlic
b
Node 2
Node 3
u
le
k
b
a
a
l y sfer
u
d
HTTP
ab -tran
HTTP
HTTP
g
Application
n
n
o
a
Application
Application
n
ab
<HTML>
<HTML>
<HTML>
</HTML>
</HTML>
</HTML>
Globally Available HTML Documents
Figure 1-8
Scalable Application Work Flow

1-23
Cluster-Aware Applications
Cluster-aware applications are applications where knowledge of the cluster

is built-in to the software. They differ from cluster-unaware applications in
the following ways:
Multiple instances of the application running on different nodes are

aware of each other and communicate across the private transport.
It is not required that the Sun Cluster software framework RGM start
and stop these applications. Because these applications are
cluster-aware, they can be started in their own independent scripts,
or by hand.
s
) ha
Applications are not necessarily logically grouped with external

application IP addresses. If they are, the network connections can be
monitored by cluster commands. It is also possible to monitor these
cluster-aware applications with Sun Cluster 3.2 software framework
resource types.
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
Parallel Database Applications
in this
d
u
al type
seof cluster application. Multiple
Parallel databases areaam
special
u
k server
to cooperate in the cluster, handling
instances of the database
g
e
s database and even providing parallel query
nsame
different queries
the
ban on
e
c
a
li queries. The following are supported in the Sun
capability
b on large
e
u
l
k
b
Cluster
environment:
a e3.2rasoftware
y
l
f
bdu transOracle 9i Real Application Clusters (RAC)
a
g onn
Oracle 10g RAC
a
n
ab
Oragle 11g RAC
Remote Shared Memory (RSM) Applications

Applications that run on Sun Cluster 3.2 hardware, can make use of an
application programming interface (API) called Remote Shared Memory
(RSM). This API maps data from an application instance running on one
node into the address space of an instance running on another node. This
can be a highly efficient way for cluster-aware applications to share large
amounts of data across the transport. This requires the SCI interconnect.
Oracle RAC is the only application that is supported in the Sun Cluster 3.2
software environment that can make use of RSM. This is also limited to
Oracle 9i and 10gR1 configurations running on Solaris 9 only.
1-24

Note This course focuses on the cluster-unaware applications rather

than the types presented on this page. Module 12 contains procedures and
an optional exercise for running Oracle 10g RAC (without RSM).
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

1-25
Sun Cluster 3.2 Software Data Service Support

This section contains a list of Sun-supported data service agents that make
cluster-unaware applications HA, either in failover or scalable
configurations.
HA and Scalable Data Service Support

The Sun Cluster software provides preconfigured components that
support the following HA data services. These are the components
available at the time of the writing this course:
s
) ha
y
m
Sun Cluster HA for Oracle E-Business Suite (failover) om

c e
m
s
Sun Cluster HA for Oracle 9i Application Server
(failover)
c Guid
e
t
@Serveren(failover
Sun Cluster HA for Sun Java System Web
or
b
u
d
k
scalable)
ya Stu
(
n
s
Sun Cluster HA for SunuJava
Server (failover)
di System
hiApplication
t
l
e
a
mSuntJava
usSystem Application Server EE HADB
Sun Cluster HAa
for
o
k
(failover) ng
se
a
n
b
e
Sun a
Cluster HA
lic for Sun Java System Directory Server (failover)
b
e
u
l
k Sun rCluster
a
ab HA for Sun Java System Messaging Server (failover)
y
l
e
f
bdu transSun Cluster HA for Sun Java System Calendar Server (failover)
a
Sun Cluster HA for Sun Instant Messaging (failover)
ang non
Sun Cluster HA for Oracle Server (failover)
ab
1-26
Sun Cluster HA for Sun Java System Message Queue (failover)
Sun Cluster HA for Apache Web Server (failover or scalable)
Sun Cluster HA for Apache Tomcat (failover or scalable)
Sun Cluster HA for Domain Name Service (DNS) (failover)
Sun Cluster HA for Network File System (NFS) (failover)
Sun Cluster HA for Dynamic Host Configuration Protocol (DHCP)

(failover)
Sun Cluster HA for SAP (failover or scalable)
Sun Cluster HA for SAP LiveCache (failover)
Sun Cluster HA for SAP Web Application Server (failover or

scalable)

Sun Cluster HA for SAP DB (failover)
Sun Cluster HA for Sybase ASE (failover)
Sun Cluster HA for Informix (failover)
Sun Cluster HA for Siebel (failover)
Sun Cluster HA for Samba (failover)
Sun Cluster HA for BEA WebLogic Application Server (failover)
Sun Cluster HA for IBM Websphere MQ (failover)
Sun Cluster HA for IBM Websphere MQ Message Broker (failover)
Sun Cluster HA for MySQL (failover)
Sun Cluster HA for PostgreSQL (failover)
Sun Cluster HA for SWIFTalliance Access (failover)
s
) ha
y
m
m
o
c
Sun Cluster HA for SWIFTalliance Gateway (failover)

e
m
d
s
i
c Gu
Sun Cluster HA for Agfa Impax (failover)e
t
@
n
b
e
u
d
Sun Cluster HA for Sun N1 Grid
akEngine
tu(failover)
y
(
S
inN1 Service
Sun Cluster HA for Sun
d
his Provisioning System (failover)
t
u
l
e
a Kerberos
Sun Cluster HA
for
m
us (failover)
a
o
k
t
Sun Cluster
ng eHA
sfore Solaris Containers (Solaris 10 OS zones)
a
n
b
a Cluster
Sun
lic HA for xVM Server Sparc Guest Domains (failover x86
b
e
u
l
k virtual
a
ab machines and Sparc LDOMs)
r
y
l
e
f
bdu trans
a
ang non
ab

1-27
Exploring the Sun Cluster Software HA Framework

The Sun Cluster software framework is the software layer that provides
generic cluster services to the nodes in the cluster, regardless of which
applications are running in the cluster. The Sun Cluster software
framework is implemented as a series of daemons and kernel modules.
One advantage of the Sun Cluster software environment is that much of
the framework resides in the kernel, where it is fast, reliable, and always
memory-resident. Some of the services provided by the framework are
described in the following sections.
Node Fault Monitoring and Cluster Membership
s
) ha
y
m
The cluster membership monitor (CMM) is kernel-resident on

each node
om
c
and detects major cluster status changes, such as loss of

communication
e
m
d
s
i
u kernel
between one or more nodes. The CMM relies on the
ectransport
G
t
@
module to generate heartbeats across the transport
n to other
b dmedium
e
u
k
nodes in the cluster. If the heartbeat from
node
tu is not detected within
ya any
(
S
a defined time-out period, it is considered
as
having
failed, and a cluster
n
i
is
d
h
t
u
l renegotiate
reconfiguration is initiatedato
e cluster membership.
s
m
u
ka e to
g
n ens
baMonitoring
Network Fault
a
lic
b
e
u
l
k rab
a
y
the
fepublic network interfaces and the cluster transport interfaces are
ul Both
s
d
n
b
monitored
for potential failures.
a
a
r
t
g
an non
ab
Public Network Management
The Sun Cluster 3.2 software environment requires the use of IPMP, a
standard Solaris OS feature, to control interface failures on a node. The
Sun Cluster software adds a layer of monitoring to detect total network
failure on one node and to drive the possible failover of applications to
another node.
Cluster Transport Monitoring

The cluster transport interfaces are monitored on each node through
heartbeats. If an active cluster transport interface on any node is
determined to be inoperative, all nodes route interconnect traffic to
functional transport interfaces. The failure is transparent to Sun Cluster
applications.
1-28

Application Traffic Striping

Applications written correctly can use the transport for data transfer. This
feature stripes IP traffic sent to the per-node logical IP addresses across all
private interconnects. Transmission Control Protocol (TCP) traffic is
striped on a per connection granularity. User Datagram Protocol (UDP)
traffic is striped on a per-packet basis. The cluster framework uses the
virtual network device clprivnet0 for these transactions. This network
interface is visible with ifconfig. No manual configuration is required.
Figure 1-9 shows the benefit of application traffic striping to a clusteraware application. The application receives the benefit of striping across
all of the physical private interconnects, but only needs to be aware of a
single IP address on each node configured on that nodes clprivnet0
adapter.
s
) ha
y
m
om
c
csm Guide
e
t
@ enCluster-aware
Cluster-aware application
application
b
u
ak Stud
y
(
in this
d
u
clprivnet0
clprivnet0
al use
172.16.193.1
172.16.193.2
m
ka e to
g
ban licens
a
b 1-9 le Application Traffic Striping
u
Figure
k
a erab
y
l
u nsf
d
b
a
tra Note Non-global zones can optionally be given their own IP addresses on
g
n
n
no
the clprivnet0 subnet. You would only do this if you wanted to support
aba
cluster-aware applications inside non-global zones. At this time Oracle
RAC is not supported in non-global zones.

1-29
Cluster Configuration Repository

General cluster configuration information is stored in global configuration
files collectively referred to as the cluster configuration repository (CCR).
The CCR must be kept consistent between all nodes and is a critical
element that enables each node to be aware of its potential role as a
designated backup system.
Caution Never attempt to modify any of the CCR-related files. The files
contain generation number information that is critical to the operation of
the cluster software. The CCR is automatically modified as the result of
administrative command execution.
y
m
m
o
c
Cluster and node names

e
m
d
s
i
ec t Gu
Cluster transport configuration
@
b den
u
k
Disk ID (DID) device configuration
ya Stu
(
n
isregistered VERITAS disk
Device group configuration
di (including
h
t
u
l
e
groups or Solaris Volume
a Manager
s disksets)
m
u
a
o
k NAS
tdevices
Informationgabout
e
n
s
ba type,
en group, and resource configuration
Resource
resource
c
a
i
l
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
s
) ha
The CCR structures contain the following types of information:
ab
1-30

Global Storage Services

The Sun Cluster software framework provides global storage services, a
feature which greatly distinguishes the Sun Cluster software product. Not
only do these features enable scalable applications to run in the cluster,
they also provide a much more flexible environment for failover services
by freeing applications to run on nodes that are not physically connected
to the data.
It is important to understand the differences and relationships between
the following services:
Global naming (DID devices)
s
) ha
y
m
Global file system

om
c
csm Guide
e
Global Naming (DID devices) kub@ dent
ya Stu
(
n
i
his name for every disk drive,
The DID feature provides
adunique device
t
u
l
e
adriveuinsthe cluster. Shared disks that might have
CD-ROM drive, or tape
m
a
knameseontodifferent nodes (different controller numbers)
different logical
g
aa ncluster-wide
are given
ns unique DID instance number. Different local
b
e
c
a
liuse the same logical name (for example, c0t0d0 for each
disks
b that may
e
u
l
k
b disk) are each given different unique DID instance numbers.
a nodes
aroot
r
y
l
e
f
bdu trans
a
ang non
Global devices
ab

1-31

Figure 1-10 demonstrates the relationship between typical Solaris OS
logical path names and DID instances.
Junction
Node 1
Node 2
d1=c0t0d0
d2=c2t18d0
c0
d2=c5t18d0
d3=c0t0d0
c2
c5
c0
Node 3
d4=c0t0d0
d5=c1t6d0
d8191=/dev/rmt/0
c0
c1
rmt/0
s
) ha
y
m
om
c
sm uide
c
e
G
Figure 1-10 DID devices
t
@
n
b
ku tude
a
y
( standard
S eight Solaris OS disk
Device files are created for each iof
n the
s
i
d
h
t /dev/did/rdsk directories; for
partitions in both the /dev/did/dsk
lu seand
a
example: /dev/did/dsk/d2s3
and
/dev/did/rdsk/d2s3.
am to u
k
ngare ejust
saeglobal naming scheme and not a global access
a
n
DIDs themselves
b
a
lic
b
scheme.
e
u
l
k rab
a
y
fe
ul DIDs
s
are used as components of Solaris Volume Manager volumes and in
d
n
b
a
a
trchoosing
cluster quorum devices, as described in Module 3, Preparing
g
n
n
o
a
for Installation and Understanding Quorum Devices
n
t0d0
Boot Disk
t18d0
Array Disk
t0d0
Boot Disk
t0d0
Boot Disk
t6d0
CD-ROM
ab
DIDs are not used as components of VERITAS Volume Manager volumes.
Global Devices
The global devices feature of Sun Cluster 3.2 software provides
simultaneous access to the raw (character) device associated with storage
devices from all nodes, regardless of where the storage is physically
attached.
This includes individual DID disk devices, CD-ROMs and tapes, as well
as VERITAS Volume Manager volumes and Solaris Volume Manager
volumes.
1-32


The Sun Cluster 3.2 software framework manages automatic failover of
the primary node for global device groups. All nodes use the same device
path, but only the primary node for a particular device actually talks
through the storage medium to the disk device. All other nodes access the
device by communicating with the primary node through the cluster
transport. In Figure 1-11, all nodes have simultaneous access to the device
/dev/vx/rdsk/nfsdg/nfsvol. Node 2 becomes the primary node if
Node 1 fails.
s
) ha
y
m
om
c
Node 1
Primary
Node 2
Node 3
Nodec4sm
ide
u
for nfsdg
e
G
t
@
n
b
ku tude
a
y
n ( this S
i
d
lu se
a
am to u
k
ng ense
a
b
nfsdg/nfsvol
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
Figure 1-11 Node Access Diagram
no
aba

1-33
Device Files for Global Devices

The Sun Cluster 3.2 software maintains a special file system on each node,
completely dedicated to storing the device files for global devices. This
file system has the mount point /global/.devices/node@nodeID, where
nodeID is an integer representing a node in the cluster. The file system is
stored on a dedicated partition on the boot disk, or, beginning in Sun
Cluster 3.2 1/09 (update 2), on a virtual partition (lofi device).
All of the /global/.devices file systemsone for each nodeare
visible from each node. In other words, they are examples of global file
systems. The global file system feature is described more on the following
pages.
s
) ha
y
m
om
c
81M
5%
/global/.devices/node@1
csm Guide
e
@ ent
b
81M
5%
u
ak Stud
y
(
in this
The device names under the /global/.devices/node@nodeID
arena can
d
u
l
e
be used directly, but, because
a theyusare unwieldy, the Sun Cluster 3.2
m
a
environment provides
symbolic
k e to links into this namespace.
g
an cens
b
a
For VERITAS
Volume
li Manager and Solaris Volume Manager, the Sun
bsoftware
e
u
l
Cluster
links
the standard device access directories into the
k
b
a
a
r
y
e
fnamespace:
ul global
s
d
n
b
a
tra ls -l /dev/vx/rdsk/nfsdg
proto192:/dev/vx#
g
n
n
a
root
40 Nov 25 03:57
no 1 root
ablrwxrwxrwx
# df -h
/dev/lofi/127
94M
3.6M
/dev/lofi/126
94M
3.6M
/dev/vx/rdsk/nfsdg ->/global/.devices/node@1/dev/vx/rdsk/nfsdg/
For individual DID devices, the standard directories /dev/did/dsk,

/dev/did/rdsk, and /dev/did/rmt are not global access paths. Instead,
Sun Cluster software creates alternate path names under the /dev/global
directory that link into the global device space:
proto192:/dev/md/nfsds# ls -l /dev/global
lrwxrwxrwx
1 root
other
34 Nov 6 13:05
/dev/global -> /global/.devices/node@1/dev/global/
proto192:/dev/md/nfsds# ls -l /dev/global/rdsk/d3s0
lrwxrwxrwx
1 root
root
39 Nov 4 17:43
/dev/global/rdsk/d3s0 -> ../../../devices/pseudo/did@0:3,3s0,raw
proto192:/dev/md/nfsds# ls -l /dev/global/rmt/1
lrwxrwxrwx
1 root
root
39 Nov 4 17:43
/dev/global/rmt/1 -> ../../../devices/pseudo/did@8191,1,tp
1-34

Note You do have raw (character) device access from one node, through
the /dev/global device paths, to the boot disks of other nodes. In other
words, while you cannot mount one nodes root disk from another node,
you can overwrite it with newfs or dd. It is not necessarily advisable to
take advantage of this feature.
Global File Systems

The global file system feature makes file systems simultaneously available
on all nodes, regardless of their physical location.
s
) ha
y
m
om
c
csm Guide
e
@ ent
The UNIX File System (ufs), kub
a Stud
y
(
The VERITAS File System
(VxFS)
in this
d
u
e(hsfs)
The High Sierra File
al System
s
m
u
ka e to
g
The Sun Cluster
an csoftware
ns makes a file system global with a global mount
b
e
a
option.
islitypically in the /etc/vfstab file, but can be put on the
b Thislline
e
u
command
k
b of a standard mount command:
a
a
r
y
fe/dev/vx/dsk/nfs-dg/vol-01 /global/nfs
u-ol global
s
# mount
d
n
b
a
tra
g
n
n
The equivalent mount entry in the /etc/vfstab file is:
no
aba
The global file system capability is independent of the structure of the
actual file system layout on disk. However, only certain file system types
are supported by Sun Cluster3.2 to be file systems underlying the global
file system. These are:
/dev/vx/dsk/nfs-dg/vol-01 /dev/vx/rdsk/nfs-dg/vol-01 \
/global/nfs ufs 2 yes global
The global file system works on the same principle as the global device
feature. That is, only one node at a time is the primary and actually
communicates with the underlying file system. All other nodes use
normal file semantics but actually communicate with the primary over the
cluster transport. The primary node for the file system is always the same
as the primary node for the device on which it is built.
The global file system is also known by the following names:
Cluster file system
Proxy file system

1-35
Shared QFS Shared-Access File System
Shared QFS is a different solution to the Sun Cluster global file system
that also provides simultaneous access from multiple nodes to the same
file system data.
Unlike the Sun Cluster global file system, Shared QFS provides access to
the file system data only from nodes directly connected to the storage. File
data access is provided directly over the storage connections, rather than
by proxy. File system metadata (inodes, directories and the like) are still
provided by proxy from a single node at a time for a particular filesystem,
although the metadata server functionality can fail over. The Shared QFS
functionality is presented in the following diagram:
s
) ha
y
m
om
c
Node 1
csm Guide
e
t2
@ Node
Metadata
n
b
e
u
Server
ak Stud
y
(
in this
d
u
l
e
File a
Data
File Data
s
m
u
a
o
k et
Metadata g
n
ba licens
Shared QFS
a
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba
Shared QFS has been supported in Sun Cluster 3.1 and Sun Cluster 3.2
only as a filesystem for storing Oracle RAC data and binaries. In the near
future it will be supported as a general-purpose shared file system, for all
applications that require simultaneous access to the same data from
multiple nodes.
1-36

Failover (Non-Global) File Systems in the Cluster

Sun Cluster 3.2 software also has support in the cluster for failover file
system access. Failover file systems are available only on one node at a
time, on a node which is running a service and has a physical connection
to the storage in question.
In Sun Cluster 3.2 more file system types are supported as a failover file
system than are supported as underlying file systems for a global file
system. These are the supported types for failover file systems:
UFS
VxFS
Sun StorageTek QFS software (regular, non-shared filesystem)
s
) ha
y
m
om
c
Solaris ZFS (Solaris 10 OS only)
csm Guide
e
t Services
@
n
Failover File System Access Is Notufor
Scalable
b
e
ak Stud
y
(
n this for failover services that run
Failover file system access d
is iappropriate
u
only on the nodes physically
e to storage devices.
al uconnected
s
m
ka e to
g
Failover file
system access is not suitable for scalable services.
ban licens
a
b filelesystem access, when used appropriately, can have a
u
Failover
k
a performance
ab benefit over global file system access. There is overhead in
r
y
l
e
f
s global file system infrastructure of maintaining replicated state
bdu tranthe
a
g on- information on multiple nodes simultaneously.
n
a
n
ab

1-37
Solaris 10 Zones
Solaris 10 Zones
The Solaris 10 OS zones feature is a powerful facility that allows you to

create multiple lightweight virtual instances of the Solaris 10 OS, each
running inside a physical instance of the Solaris OS. Each configured nonglobal zone (the main Solaris 10 OS instance is known as the global zone)
appears to be a complete OS instance isolated from every other zone and
from the global zone. Each zone has its own zonepath, that is, its own
completely separate directory structure for all of its files starting with the
root directory. When configuring a non-global zone, its zonepath is
expressed as a subdirectory somewhere inside a global zone mounted file
system. When you log in to a non-global zone you see the file systems
belonging only to that non-global zone, users and processes belonging
only to that non-global zone, and so on. Root users logged into one nonglobal zone have superuser privileges only in that zone.
s
) ha
y
m
om
c
csm Guide
e
LDoms and Zones
@ ent
b
u
ak Stud
y
(
A zone is a lightweight virtual machine,
in thinisthat it only appears to be an
d
u
independent instance of Solaris.
e does not have a separate kernel or
al Auzone
s
m
address space.
ka e to
g
baa nfulllivirtual
ensmachine with its own kernel and complete
An LDomais
c
b of theleSolaris OS. An LDom can do anything that a Solaris
instance
u
k
a emachine
ab can, including containing its own zones.
r
y
physical
l
f
bdu trans
a
g onn
a
nSun Cluster Distinct Zone Features
ab
Sun Cluster has three distinct ways of dealing with zones, as outlined in
the subsections below. You can mix and match zone features in the same
cluster. A particular zone can be used with only one of the features.
1-38

Solaris 10 Zones
Failover Zones (HA-Containers feature)

This is the oldest of the zone features, dating from Sun Cluster 3.1. Here, a
zone is treated as a black box that fails over from node to node. The
zonepath for the zone lives in shared storage, and the zone is booted
under control of Sun Cluster, on only one node at a time. Figure 1-12
illustrates this feature:
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le HA-Containers feature
k
b
Figure
1-12
a
a
l y sfer
u
d
ab -tranAn advantage of this feature is that the zone can be treated as a
g
n non
a
completely cluster-unaware entity. Whichever applications happen to run
b
a
when the zone boots will fail over as the zone itself fails or switches from
node to node.
Zones as Virtual Nodes in a Single Global Cluster

This second model was introduced in the original Sun Cluster 3.2. Zones
run on different cluster nodes and are booted outside the control of Sun
Cluster. The cluster framework is completely aware of these zones, and
can launch applications to run in the zones.
In this model, you do not think of zones as having their own cluster.
Instead, there is a single global cluster, as usual; zones are simply a
location where applications can be launched. They act as virtual nodes for
that part of the cluster control that launches applications.

1-39
Solaris 10 Zones
Figure 1-8 illustrates this feature:
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
l
e zones
afailover
Figure 1-13 Application
between
s
m
u
ka e to
g
n ehas
nsseveral advantages over the HA-containers
Note that this
bamodel
c
a
li
model:
b
e
u
l
k
a Applications
ab fail from zone to zone, rather than the whole zone
r
y
l
e
f
bdu transfailing over. This will result in faster failover.
a
g onn
The same application-specific code (Sun Cluster agent) can be used,
a
n
ab
usually without any modifications, to launch the applications in the

global zone and within non-global zones.
1-40

Solaris 10 Zones
Zone Cluster
This is the newest model for zones, introduced in Sun Cluster 3.2 1/09
(Update 2). Here, we think of a set of zones (across different nodes) as
forming their own cluster, a zone cluster within the normal global
cluster. Several zone clusters can exist within the global cluster, as
illustrated by Figure 1-14:
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g

ban licens
a
b 1-14le Zone clusters within a global cluster
u
Figure
k
a erab
y
l
u nsf
d
b
a
tra You can accomplish many of the same tasks using a zone cluster and the
g
n
n
previous feature, with a slightly different point of view. Here, the
no
aba
configuration of the cluster applications in the zones is isolated from the
the rest of the configuration, much like the processes in a zone are isolated
from other zones.
Zone cluster is addressed in detail in the Sun Cluster 3.2 Advanced
Administration course (ES-445), rather than in this course.

1-41
Exercise: Guided Tour of the Training Lab
Exercise: Guided Tour of the Training Lab

By the end of this exercise, you should be able to identify Sun Cluster
hardware components located in the training lab.
Preparation
No special preparation is required for this lab.
Task
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
k tud
aaccess
y
(
If this course is being run without local
the equipment, take this
n of tthe
sSunStoCluster
i
i
d
h
opportunity to review the essentials
hardware and
u
l
e
a
software, or to familiarize
uswith the remote lab environment.
am yourself
o
k
t
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba
You will participate in a guided tour of the training lab. While
participating in the guided tour, you will identify the Sun Cluster
hardware components including the cluster nodes, terminal concentrator,
and administrative workstation.
1-42

Exercise Summary
Exercise Summary
Discussion Take a few minutes to discuss what experiences, issues, or

discoveries you had during the lab exercises.
Experiences
Interpretations
Conclusions
Applications
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

1-43
Exercise Summary
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
1-44

Module 2
Exploring Node Console Connectivity and

the Cluster Console Software
s
) ha
Objectives
y
m
om
c
ide
csa m
u
Describe the different methods for accessing
console
e
G
t
@
n
b
Configure the Sun Cluster console
deon the administration
kusoftware
u
a
t
y
workstation
n ( this S
i
d
Use the Cluster console
lu tools
e
a
s
m
u
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab
2-1
Relevance
Relevance

What is the trade-off between convenience and security in different

console access methods?
How do you reach the node console on domain-based clusters?
What benefits does using a terminal concentrator give you?
Is installation of the administration workstation software essential

for proper cluster operation?
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
2-2

820-4679 (for Sun Cluster 3.2 1/09).
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Exploring Node Console Connectivity and the Cluster Console Software

2-3
Accessing the Cluster Node Consoles

This section describes different methods for achieving access to the Sun
Cluster 3.2 node consoles. It is expected that a Sun Cluster 3.2
environment administrator:
Does not require node console access for most operations described
during the duration of the course. Most cluster operations require
only that you be logged in on a cluster node as root or as a user
with cluster authorizations in the Role-Based Access Control (RBAC)
subsystem. It is acceptable to have direct telnet, rlogin, or ssh
access to the node.
Must have console node access for certain emergency and

informational purposes. If a node is failing to boot, the cluster
administrator will have to access the node console to figure out why.
The cluster administrator might like to observe boot messages even
in normal, functioning clusters.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
Accessing Serial Port Consoles
in thon
is Traditional Nodes
d
u
al use
m
Traditional Sun Cluster
ka 3.2enodes
to usually use serial port ttyA as the
g
console.
ban licens
a
byou have
u
le a graphics monitor and system keyboard you are
k
Even
if
b
a
a
l ysupposed
erto redirect console access to the serial port or emulation thereof.
f
u
s
d
ab -tran
g
n non
a
b
a
The rule for console connectivity is simple. You can connect to the node
ttyA interfaces any way you prefer, if whatever device you have
connected directly to the interfaces does not spuriously issue BREAK
signals on the serial line.
BREAK signals on the serial port bring a cluster node to the OK prompt,
killing all cluster operations on that node.
You can disable node recognition of a BREAK signal by a hardware
keyswitch position (on some nodes), a software keyswitch position (on
midrange and high-end sersvers), or a file setting (on all nodes).
For those servers with a hardware keyswitch, turn the key to the third
position to power the server on and disable the BREAK signal.
2-4

For those servers with a software keyswitch, issue the setkeyswitch

command with the secure option to power the server on and disable the
BREAK signal.
For all servers, while running Solaris OS, uncomment the line
KEYBOARD_ABORT=alternate in /etc/default/kbd to disable
receipt of the normal BREAK signal through the serial port. This setting
takes effect on boot, or by running the kbd -i command as root.
The Alternate Break signal is defined by the particular serial port
driver you happen to have on your system. You can use the prtconf
command to figure out the name of your serial port driver, and then use
man serial-driver to figure out the sequence. For example, for the zs
driver, the sequence is carriage return, tilde (~), and control-B: CR ~
CTRL-B. When the Alternate Break sequence is in effect, only serial
console devices are affected.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

2-5
Accessing Serial Port Node Consoles Using a Terminal

Concentrator
One of the popular ways of accessing traditional node consoles is through

a TC, a device which listens for connections on the network and passes
through traffic (unencapsulating and reencapsulating all the TCP/IP
headers) to the various serial ports.
A TC is also known as a Network Terminal Server (NTS).
Figure 2-1 shows a terminal concentrator network and serial port
interfaces. The node public network interfaces are not shown. While you
can attach the TC to the public net, most security-conscious
administrators would attach it to a private management network.
s
) ha
y
m
om
c
sm uide
c
Administrative
e
G
Cluster Console
t
@
Console
n
b
Software
ku tude
a
y
n ( this S
i
Network
d
lu se
a
Terminal
am to u
k
Concentrator
ng ense
a
Network
b
a Interface
lic
b
e
Serial Ports
u
l
k
b
a
a
Serial port A
y fer
l
7
4
8
2
3
5
6
1
u
s
bd tran
a
g onSetup port
n
a
n
b
Node 1
Node 2
a
Setup Device
Figure 2-1
TC Network and Serial Port Interfaces
Most TCs enable you to administer TCP pass-through ports on the TC.
When you connect with telnet to the TCs IP address and pass through
port, the TC transfers traffic directly to the appropriate serial port
(perhaps with an additional password challenge).
2-6

Sun Terminal Concentrator (Sun NTS)

Sun rebrands a terminal concentrator, the Xylogics NTS. Sun guarantees
that this TC does not spuriously issue BREAK signals when it is powered
on and off, for example.
The Sun TC supports telnet and rlogin access. The TCP ports it uses as
pass-through ports for telnet access are 5002 for serial port 2, 5003 for
serial port 3, and so forth.
When you use telnet to connect to the Sun TC from an administrative
workstation using, for example, the following command, the Sun TC
passes you directly to serial port 2, connected to your first cluster node.
s
) ha
ab
y
m
om
c
e
The Sun TC supports an extra level of passwordcchallenge,
sm uiadper-port
e
G
password that you have to enter before going
through
to the serial port.
t
@
n
b
ku tude
a
y
Full instructions about installing(and configuring
the Sun TC are in
n this S
i
d
Appendix A, Terminal Concentrator.
lu se
a
am to u
k
ng Concentrators
se
Other Terminal
a
n
b
e
a
lic
b
e
u
l
kYou can
b any type of TC as long as it does not issue BREAK signals
a
achoose
r
y
l
e
f
s the serial ports when it is powered on, powered off, reset, or any other
bdu tranon
a
- time that might be considered spurious.
ang non
# telnet tc-ipname 5002
If your TC cannot meet that requirement, you can still disable recognition
of the BREAK signal or enable an alternate abort signal for your node.
Some terminal concentrators support Secure Shell (the Sun NTS does not
support Secure Shell). This might influence your choice, if you are
concerned about passing TC traffic in the clear on the network.

2-7
Alternatives to a Terminal Concentrator (TC) for Nodes

with a Serial Port Console
Some possible alternatives to the terminal concentrator (TC) include the
following:
Use dumb terminals for each node console. If these are in a secure
physical environment, this is certainly the most secure, but least
convenient, method.
Use a workstation that has two serial ports as a tip launchpad,

especially for a cluster with only two nodes.
s
) ha
You can attach a workstation on the network exactly as you would

place a TC, and attach its serial ports to the node consoles. You then
add lines to the /etc/remote file of the Solaris OS workstation as
follows:
y
m
om
c
node1:\
csm Guide
e
@ ent
:dv=/dev/term/a:br#9600:el=^C^S^QÛ^D:ie=%$:oe=^D:
b
u
node2:\
ak Stud
y
(
:dv=/dev/term/b:br#9600:el=^C^S^QÛ^D:ie=%$:oe=^D:
in this
d
u
This allows you to access
econsoles by accessing the launchpad
al node
s
m
u
workstation and
kamanually
to typing tip node1 or tip node2.
g
e
s a Solaris OS workstation instead of a TC is
One advantage
ban liceofnusing
a
that
tighten the security in the Solaris OS workstation.
b it is easier
e youto could
u
l
k
b
For
example,
easily disable telnet and rlogin access and
a era
y
l
f
that administrators access the tip launchpad through Secure
bdu transrequire
a
Shell.
ang non
ab
Accessing the Node Console on Servers with Virtual

Consoles
Many of the servers supported with Sun Cluster have console access
through a network connection to a virtual console device.
These include:
2-8
Hardware domain-based systems: the console access device is the

system controller (SC) or system service processor (SSP)
Servers such as Sun Fire V890: you can choose to have console access
through the Remote System Control (RSC) device and software

Modern rack-based servers: the console access device is a small

onboard system controller running Advanced Lights-out
Management (ALOM)
LDoms: console access to an LDom is through a network connection

to the service domain, which provides the virtual console service for
the LDom.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

2-9
Describing Sun Cluster Console Software for an Administration Workstation
Describing Sun Cluster Console Software for an

Administration Workstation
The Sun Cluster 3.2 software includes a small amount of software to be
installed on an administrative workstation. This software can be installed
on any Solaris OS workstation running Solaris OS version 2.6 and above.
The console software is a convenience that displays new windows for you
to the cluster nodes and issues telnet, rlogin, or ssh commands to
connect you either directly to the nodes or to the node console access
point.
as
h
The console software includes a common window feature that lets you issue
)
y
keystrokes simultaneously to all the nodes in the cluster (or even tom
all the
om
nodes in multiple clusters).
c
sm uide
c
e
G
The Sun Cluster 3.2 software environment itself
has no dependency
on the
t
@
n
b
e an administrative
administration workstations console software.
ku Ittuisdjust
a
y
convenience.
n ( this S
i
d
lu se
a
am to u
Console Software
Installation
k
ng ense
a
b
a
licconsole software is contained in a single package,
b
The u
administrative
e
l
k raThe
b SUNWccon package is installed manually from the Sun
a
SUNWccon.
y
l
e
f
s 3.2. There are internationalized packages as well (SUNWjccon,
n
bdu trCluster
a
a
- SUNWkccon, and so on) but you always have to install SUNWccon first.
ang non
ab
Cluster Console Window Variations

The three variations of the cluster console tool each use a different method
to access the cluster hosts. They all look and behave the same way. The
cluster console window variations are the following:
Cluster console (cconsole)

The cconsole program accesses the node consoles through the TC
or other remote console access method. Depending on your access
method, you might be directly connected to the console node, you
might have to respond to additional password challenges, or you
might have to login to an SSP or SC and manually connect to the
2-10


node console. cconsole by default uses telnet to reach the console
device, but you can request that it use ssh by adding the -s option to
cconsole.
Cluster console (crlogin)

The crlogin program accesses the nodes directly using the rlogin
command.
Cluster console (ctelnet)

The ctelnet program accesses the nodes directly using the telnet
command.
Cluster console (cssh)

The cssh program accesses the nodes directly using the ssh
command.
s
) ha
y
m
m
o
c
want to
In certain scenarios, some tools might be unusable, but
you might
e
m
d
s
i
c toGyou.
install this software to use the tools that makeesense
u
t
@
b den
u
k
For example, if you are using dumb
tuto attach to the node
(yaterminals
S
consoles, you will not be ableinto use the
cconsole
variation, but you can
is
d
h
t
u
use the other three after
e are booted.
althe nodes
s
m
u
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab

2-11
Cluster Console Tools Look and Feel
All the tools have the same general look and feel. The tool automatically
shows one new window for each node, and a small common keystroke
window (Figure 2-2). You can type in each individual window as desired.
Input directed to the common window is automatically replicated to all
the other windows.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Figure 2-2
2-12
Cluster Console Windows

Starting the Tools Manually
As shown, you can use the tools manually to connect to a single cluster
node or to the entire cluster.
#
#
#
#
#
/opt/SUNWcluster/bin/cconsole node1 &

/opt/SUNWcluster/bin/cconsole -s my-cluster &
/opt/SUNWcluster/bin/ctelnet my-cluster &
/opt/SUNWcluster/bin/crlogin node3 &
/opt/SUNWcluster/bin/cssh node3 &
Cluster Console Host Windows
s
) ha
y
m
om
c
ideoperation.
csm Gforubest
Set the TERM environment variable to vt100 oredtterm
@ ent
b
u
ak Stud
y
(
Cluster Console Common
in Window
d
his
t
u
l
e
a us
m
a
The common window,
k eshown
to in Figure 2-3, allows you to enter
g
commands
all host
at the same time. All of the
nssystemso windows
baarentotied
e
windows
together,
when
you
move the common window, the
c
a
li
b
e
u
l
khost windows
b follow. The Options menu allows you to ungroup the
a
a
r
y
windows,
l
e move them into a new arrangement, and group them again.
f
u
s
d
ab -tran
g
an non
Each node in the cluster has a host window. You can enter commands in
each host window separately.
ab
Figure 2-3
Cluster Console Common Window

2-13
Cluster Control Panel
As shown in Figure 2-4, the Cluster Control Panel provides centralized

access to three variations of the cluster console tool.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
k tud
Figure 2-4 Cluster Control Panel (ya
n this S
i
d
lu Panel
e
a
s
Starting the Cluster Control
m
u
ka e to
g
an cControl
To start thebCluster
ns Panel, type the following command:
e
a
li
e
u
l
#b
/opt/SUNWcluster/bin/ccp
[clustername] &
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
2-14

Configuring Cluster Console Tools

All the information needed for the cluster administration tools that run on
the administrative console is configured in two files. The files are:
The /etc/clusters file
The /etc/serialports file (for cconsole variation only)
When you install the Sun Cluster console software on the administrative
console, you must manually create the clusters and serialports files
and populate them with the necessary information.
s
) ha
y
m
om
c
The /etc/clusters file contains the name of a cluster

sm followed
ide by the
u
names of the nodes that are part of the cluster.ec
G
t
@
n
b
u ude
k/etc/clusters
a
t
The following is a typical entry in
the
file:
y
(
S
n
s
i
i
th
sc-cluster sc-node1
lud sc-node2
e
a
am to us
k
e a cluster named sc-cluster, which has two
The single-line
ng entry
sdefines
a
n
b
e
nodesanamed sc-node1
and sc-node2.
lic
b
e
u
l
k rab
a
y
fe The cluster name is purely arbitrary. The Sun Cluster 3.2 software
ul nNote
s
d
b
a
tra itself requires you to define a cluster name, which will likely agree with
g
n
n
this one, although nothing will break if it does not agree.
no
aba
Configuring the /etc/clusters File
You can define many different clusters in a single /etc/clusters file, so

you can administer several clusters from a single administrative console.

2-15
Configuring the /etc/serialports File

The /etc/serialports file defines the connection to the console for each
node defined in the /etc/clusters file. You enter a separate line for each
node in this file. This file uses three columns: the node name listed in
/etc/hosts, the host name or IP address of the device providing the
connection as listed in /etc/hosts, and the port for telnet or ssh to
use for the connection to or through this device.
The following are typical entries in the /etc/serialports file when
using a terminal concentrator to pass through to the physical serial
connection (ttyA) on each cluster node:
sc-node1 sc-tc 5002
sc-node2 sc-tc 5003
s
) ha
y
m
om
c
The following is a typical example for nodes with network

sm access
ideto a
c
u
e
virtual console access device. When the cconsole command
G is executed,
t
@
n
b
you are connected to the console device,kbut
the
command
u ude has no idea
a
t console. You must log
what you need to type to actually access
y the node
(
S
n
i
is type your console access
into the console access device d
manually,
hand
t
u
l
e
command (typically console)
a manually.
m
us
a
o
k
t
sc-node1 node1-sc
g nse22
n
a
sc-node2
ce 22
ab node2-sc
i
l
b
u
le
k
b
a
a
l yNotesfeInrorder to use ssh to the console device, you must put the correct
u
d
an number in /etc/serialports (typically 22) and use cconsole with
ab -trport
g
an non the -s option.
ab
2-16

Exercise: Configuring the Administrative Console

In this exercise, you complete the following tasks:
Task 1 Updating Host Name Resolution
Task 2 Installing the Cluster Console Software
Task 3 Verifying the Administrative Console Environment
Task 4 Configuring the /etc/clusters File
Task 5 Configuring the /etc/serialports File
Task 6 Starting the cconsole Tool
Task 7 Using the ccp Control Panel
s
) ha
y
m
om
c
Preparation
csm Guide
e
t is already installed
n
bOS@software
This exercise assumes that the Solarisk10
e
u
a the following
tud steps to prepare for
on all of the cluster systems. Perform
y
(
S
in this
the lab:
d
u
e assigned to your cluster.
al for the
s
m
u
1. Ask your instructor
name
ka e to
g
s
Cluster
n_______________________________
banname:
e
c
a
b Recordletheli information in Table 2-1 about your assigned cluster
2.
u
k
a ebefore
ab proceeding with this exercise.
r
y
l
f
s
bdu tranTable
a
2-1 Cluster Names and Addresses
g onn
a
n
ab
System
Name
IP Address
Administrative
console
TC
Node 1
Node 2
Node 3 (if any)

2-17

3.
Ask your instructor for the location of the Sun Cluster software.
Software location: _______________________________
Task 1 Updating Host Name Resolution

Even though your site might use Network Information Service (NIS) or
DNS to resolve host names, it can be beneficial to resolve the names
locally on the administrative console and cluster hosts. This can be
valuable in the case of naming service failures. The cconsole program
does not start unless it can first resolve the host names in the
/etc/clusters file.
s
) ha
y
m
om
1. If necessary, edit the /etc/hosts file on your administrative
c
mTC uand
console, and add the IP addresses and names of
sthe
idethe host
c
e
G
systems in your cluster.
t
@
n
b
e hosts has files
dfor
kufile entry
2. Verify that the /etc/nsswitch.conf
u
a
t
y
( service.
SConsult your instructor
listed first before any otherin
name
s
i
d
h
t such as dns should be used.
about whether any other
lu nameseservice,
a
m to u
hosts: fileska
dns
ng ense
a
b
a
lic
b
e
u
l
Task a
2k Installing
the Cluster Console Software
b
a
r
y
ul nsfe
d
b
a
the following steps to install the cluster console software:
a
trPerform
g
n
n
a
no
Perform the following steps:
ab
1.
Log in to your administrative console as user root.
Note If your administrative console is in a remote lab, your instructor

will assign you a user login. In this case, the cluster console software is
already installed.
2.
Check to see if the cluster console software is already installed:

(# or $) pkginfo SUNWccon
If it is already installed, you can skip the rest of the steps of this task.
3.
Move to the Sun Cluster 3.2 packages directory:
(# or $) cd sc32_location/Solaris_{sparc_or_x86}/Products
(# or $) cd sun_cluster/Solaris_10/Packages
2-18


4.
Verify that you are in the correct location:
# ls -d SUNWccon
SUNWccon
5.
Install the cluster console software package:

# pkgadd -d . SUNWccon
Task 3 Verifying the Administrative Console

Environment
Perform the following steps to verify the administrative console:
as
h
1. Verify that the following search paths and variables are present
in
)
y
m
the .profile file in your home directory:
om
c
PATH=$PATH:/opt/SUNWcluster/bin
MANPATH=$MANPATH:/opt/SUNWcluster/man
csm Guide
e
export PATH MANPATH
@ ent
b
u
ak Stud
y
(
n in your
ifile
Note Create the .profile
d
hishome directory if necessary, and
t
u
l
e
add the changes. ma
us
a
o
k
t
ngthe .profile
se file to verify changes:
2. Execute
a
n
b
e
a
ic
b
(# orle$)l . $HOME/.profile
u
k rab
a
y
ul nsfe
d
b
a
tra Note You can also log out and log in again to set the new variables.
g
n
n
a
no
ab
Task 4 Configuring the /etc/clusters File

The /etc/clusters file has a single line entry for each cluster you intend
to monitor. The entries are in the form:
clustername host1name host2name host3name host4name
Sample /etc/clusters File
sc-cluster pnode1 pnode2 pnode3

2-19

Perform the following to configure the /etc/clusters file:
Edit the /etc/clusters file, and add a line using the cluster and
node names assigned to your system.
Note If you are using a remote lab environment, the /etc/clusters file
might already be set up for you. Examine the file.
Task 5 Configuring the /etc/serialports File
s
) ha
y
m
om
c
csm Guide
hostname tcname tcport
e
@ ent
b
u
k tud
Sample /etc/serialports File (ya
S
n
s
i
i
d
h
t
pnode1
cluster-tc
lu s5002
e
a
pnode2
cluster-tc
5003
am to u 5004
k
pnode3
cluster-tc
ng ense
a
b
a the following
lic to configure the /etc/serialports file:
b
Perform
e
u
l
k rab
a
y
ul nsfeEdit the /etc/serialports file and add lines using the node and
d
b
TC names assigned to your system.
a
tra
g
n
n
a
no
The /etc/serialports file has an entry for each cluster host describing
the connection path. This example is for a three-node cluster using a Sun
NTS.
ab
2-20
Note In the remote lab environment, this file might already be set up for
you. Examine the file.

Task 6 Starting the cconsole Tool
This section provides a good functional verification of the TC in addition

to the environment configuration.
Perform the following steps to start the cconsole tool:
1.
Make sure power is on for the TC and all of the cluster hosts.
2.
Start the cconsole tool on the administrative console:

(# or $) cconsole clustername &
ab
as
Note Substitute the name of your cluster for clustername. Consult
your
h
)
y so you
instructor about whether the console access device supports ssh;
if
m
can add a -s option above.

om
c
sm uand
idepress the
cwindow,
e
3. Place the cursor in the cconsole Common
G
t
@
n
b
Return key several times. You should
see
a
response
on all of the
u ude
k
a
cluster host windows. If not,
ask
your
instructor
for
assistance.
t
(y is S
n
i
4. If your console device
th dedicated interface, you might
udis anseALOM
l
a
need to enter an
command
to access the actual node consoles.
mALOM
u
a
o
k
t
Consult g
your instructor.
n
se
a
n
b
e
5. Ifathe cluster
lic host systems are not booted, boot them now.
b
e
u
l
k ok
bboot
a
a
r
y
ul n6.sfeAfter all cluster host systems have completed their boot, log in as
d
b
a
tra
user root.
g
n
n
o
a
n
7.
Practice using the Common window Group Term Windows feature

under the Options menu. You can ungroup the cconsole windows,
rearrange them, and then group them together again.
Task 7 Using the ccp Control Panel

The ccp control panel can be useful if you must use the console tool
variations.
Perform the following steps to use the ccp control panel:
1.
Start the ccp tool (ccp clustername &).
2.
Practice using the cssh, ctelnet console tool variations.
3.
Quit the cssh, ctelnet, and ccp tools.

2-21
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
2-22

Module 3
Preparing for Installation and Understanding

Quorum Devices
s
) ha
Objectives
ab
y
m
om
c
idehardware
csm Guand
List the Sun Cluster software boot disk requirements
e
@ ent
restrictions
b
u
k tud
atopologies
y
Identify typical cluster storage
(
n this S
i
d
Describe quorum a
votes
lu andsequorum devices
u reservations and cluster amnesia
am quorum
o
Describe persistent
k
t
ngdataefencing
se
a
n
Describe
b
a
lic
b
e
u
l
k Configure
b a supported cluster interconnect system
a
a
r
y
ul nsfeIdentify public network adapters
d
b
a
tra
Configure shared physical adapters
g
n
n
a
no
3-1
Relevance
Relevance

Is cluster planning required even before installation of the Solaris OS

on a node?
Do certain cluster topologies enforce which applications are going to

run on which nodes, or do they just suggest this relationship?
Why is a quorum device absolutely required in a two-node cluster?
What is meant by a cluster amnesia problem?
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
3-2

820-7458 (for Sun Cluster 3.2 11/09).
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Preparing for Installation and Understanding Quorum Devices

3-3
Configuring Cluster Servers

You must plan your cluster installation before you install the Solaris OS on
your cluster nodes. If you install Solaris blindly on cluster nodes you
are likely to miss several cluster-specific considerations and requirements,
outlined in the following subsections.
Boot Device Restrictions

With the Sun Cluster 3.2 software release, there are several restrictions on
boot devices including the following:
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
d 10 OS (revisions
ak 9), S
The Solaris 9 OS 9/05 (Update (8yand
and
tuSolaris
vary, listed in Module 1) d
are
insupported:
his
t
u
l
e
a of theusSolaris OS, including the update
The same version
m
a
k example,
to Solaris 10 11/09), must be installed on all
revision
(for
g
e
s
nodes
ban inlictheencluster.
a
b VERITAS
u
le Dynamic Multipathing (DMP) is not supported.
k
b
a
a
l y sferSee Module 6, Using VERITAS Volume Manager With Sun
u
d
ab -tran Cluster Software for more details about this restriction.
g
n non
a
The supported versions of VERITAS Volume Manager must
b
a
You cannot use a shared storage device as a boot device. If a storage

device is visible to more than one host, it is shared. For example, a
SAN LUN that is masked so that it is visible to only one node can be
used as a boot device, regardless of the number of physical
connections to the device.
have the DMP device drivers enabled, but Sun Cluster 3.2
software still does not support an actual multipathing topology
under control of Volume Manager DMP.
3-4

Boot Device Partitioning (without ZFS root)
If you use a traditional partitioned root disk with UFS-root:
Sun Cluster 3.2 software recommends a minimum swap

partition of 750 megabytes (Mbytes).
Solaris Volume Manager software requires a 32 Mbyte partition

for its meta-state databases.
VERITAS Volume Manager software requires two unused

partitions for encapsulation of the boot disk.
Prior to Sun Cluster 3.2 1/09 (Update 2) you must have a

dedicated partition on the boot disk on each node for the
/global/.devices/node@nodeid file system. If you choose
this option, have the Solaris installer create an empty filesystem
mounted on /globaldevices as a placeholder.
s
) ha
ab
y
m
om
c
csm Guide
e
t
n
b@
e
u
Note The /globaldevices mount k
point
is modified
during the Sun
d
u
a
t
y
Cluster 3.2 software installation.(It is automatically
to
n twhere
s Snodeid renamed
i
i
d
h
/global/.devices/node@nodeid,
represents
the
u se
l
a
number that is assigned
m totoa node
u when it becomes a cluster member.
a
k
The original /globaldevices
mount point is removed. The
g nse
n
a
/globaldevices
file
system
must
have ample space and ample inode
b lice
a
capacity
b forlecreating both block-special devices and character-special
u
k
b is especially important if a large number of disks are in the
devices.
a eraThis
y
l
f
s A file system size of 512 Mbytes should suffice for even the largest
bdu trancluster.
a
- cluster configurations.
ang non
Starting in Sun Cluster 3.2 1/09 you can choose to use a lofi
device rather than a dedicated partition for the
/global/.devices/node@nodeid file system. A lofi device is
a virtual disk device that uses a file as the backing store for the
entire contents of a file system. If you choose this option, do not
create a /globaldevices placeholder. You will then be able to
choose the lofi option as you build the cluster using scinstall,
as discussed in Module 4.
Note scinstall will automatically provision the backing store for the
lofi device in a file named /.globaldevices. While 512MB is still
recommended when you use a dedicated partition, scinstall uses
100MB as the size of the backing store when you use a lofi device.

3-5
Typical Boot Disk JumpStart Software Profile Examples

(Without ZFS root)
If you decide to configure your cluster servers using the JumpStart
software, the following JumpStart software profiles represents a starting
point for boot disk configuration.
# example Jumpstart profile -- with dedicated /globaldevices
install_type initial_install
system_type standalone
partitioning explicit
cluster SUNWCXall
filesys c0t0d0s0 free /
filesys c0t0d0s1 1024 swap
filesys c0t0d0s3 512 /globaldevices
filesys c0t0d0s7 32
s
) ha
y
m
om
c
sm uide
c
e
# example Jumpstart profile no ZFS root
but not G
@
n
b
e
u
# dedicated /globaldevices --- will
choose
lofi in scinstall
d
k
a Stu
install_type initial_install (y
n
system_type standalone udi
his
t
l
e
a us
partitioning explicit
m
a
k e to
cluster SUNWCXall
g
filesys c0t0d0s0
an cefree
ns /
b
a
filesys
c0t0d0s1
li 1024 swap
b
e
u
l
k
filesys
32
a erc0t0d0s7
ab
y
l
f
bdu trans
a
ang non Note The logging option is the default starting in Solaris 9 9/04
ab
(Update 7), and including all updates of Solaris 10 OS.
Boot Device Partitioning (ZFS-root)

Solaris OS supports ZFS-root beginning in Solaris 10 10/08 (Update 6).
This greatly simplifies root disk provisioning and high availability, and
gives you the ability for checkpointing and recovery using snapshots. ZFS
for both data and root are discussed more in Module 8.
3-6


With ZFS root you will choose to use the lofi device option for the
/global/.devices/node@nodeid file system; the two choices go
together naturally. It is not possible that
/global/.devices/node@nodeid itself be a ZFS file system, because it
is mounted globally and that is not supported by ZFS. But the lofi
solution works nicely; a file in a ZFS file system can be used as the
backing store for a lofi device that contains the UFS file system required
for /global/.devices/node@nodeid .
Typical Boot Disk JumpStart Software Profile Example (With

ZFS root)
s
) ha
If you decide to configure your cluster servers using the JumpStart

software, the following JumpStart software profile represents a starting
point for boot disk configuration.
ab
y
m
m
o
c
# example Jumpstart profile -- ZFS with m

e
d
s
i
u
# space on s7 left out of the zpool for
ecSVMt metadb
G
@
b den
u
k
cluster SUNWCXall
ya Stu
(
filesys c0t0d0s7 32
n
di2G e this2G c0t0d0s0
u
pool
rpool
auto
l
a us
m
a
k ZFS root
tois configured on a disk partition, rather than on
g
e
Since the pool
for
n ens
badisk,
an entire
reserve a boot disk partition for Solaris Volume
a
ic can
lwe
b
e
Manager
metadbs
outside
the scope of the ZFS root. You might need to
u
l
k
b
a
a
do
this
to
support
scalable
applications
that require global file systems or
y fer
l
u
s
bd tranif you need to use SVM with Oracle RAC.
a
ang non
.

3-7
Configuring Cluster Storage Connections

While previous versions of the Sun Cluster software had strict rules
regarding how many nodes were supported in various disk topologies,
the only rules in the Sun Cluster 3.2 software regarding the data storage
for the cluster are the following:
Sun Cluster software never supports more than sixteen nodes. Some
storage configurations have restrictions on the total number of nodes
supported.
A shared storage device can connect to as many nodes as the storage

device supports.
as
Shared storage devices do not need to connect to all nodes of the) h
ytwo
cluster. However, these storage devices must connect to at least
m
nodes.
om
c
csm Guide
e
@ ent
b
Cluster Topologies
u
ak Stud
y
(
din ways
hiniswhich cluster nodes can be
Cluster topologies describe ltypical
t
u
e
adevices.uWhile
s Sun Cluster does not require you
connected to data storage
m
a
o
k
t
to configure a cluster
using
specific topologies, the following
g bynsto
eprovide
ndescribed
a
topologies b
are
the vocabulary to discuss a clusters
e
c
a
i
l
connection
b scheme.
e The following are some typical topologies:
u
l
k
b
ra pairs topology
ya Clustered
l
e
f
u
s
bd tran Pair+N topology
a
g onn
a
n
N+1 topology
ab
3-8
Multiported (more than two node) N*N scalable topology
NAS device-only topology
Data replication topology
Non-storage topology
Single-node cluster topology

Clustered Pairs Topology

As shown in Figure 3-1, a clustered pairs topology is two or more pairs of
nodes with each pair physically connected to some storage. Because of the
global device and global file system infrastructure, this does not restrict
where applications can fail over to and run. Still, it is likely you will
configure applications to fail over in pairs of nodes attached to the same
storage.
Switch
Switch
s
) ha
y
m
om
c
Node 3
Node 4 ide
Node 2
Node 1
csm
u
e
G
t
@
b den
u
k
ya Stu
(
n
di e this
u
l
a us
m
a
k e to
g
Storage an Storage
Storage
Storage
b licens
a
b 3-1 le Clustered Pairs Topology Configuration
u
Figure
k
a erab
y
l
u nsf
d
b
a
tra The features of clustered pairs configurations are as follows:
g
n
n
no
Nodes are configured in pairs. You can have any even number of
aba
nodes from 2 to 16.
Each pair of nodes shares storage. Storage is connected to both nodes

in the pair.
All nodes are part of the same cluster. You are likely to design
applications that run on the pair of nodes physically connected to the
data storage for that application, but you are not restricted to this
design.
Because each pair has its own storage, no one node must have a
significantly higher storage capacity than the others.
The cost of the cluster interconnect is spread across all the nodes.
This configuration is well suited for failover data services.
This configuration is well suited if you have legacy SCSI-array, or

any disk array that can only be attached to two nodes.

3-9
Pair+N Topology
As shown in Figure 3-2, the Pair+N topology includes a pair of nodes

directly connected to shared storage and nodes that must use the cluster
interconnect to access shared storage because they have no direct
connection themselves.
Switch
Switch
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
Storage
al use
Storage
m
ka e to
g
n eTopology
Figure 3-2 ba
Pair+N
ns
c
a
li
b
e
u
l
k
a features
abof the Pair+N configurations are as follows:
The
r
y
l
e
f
bdu transAll shared storage is connected to a single pair.
a
g onn
a
Additional cluster nodes support scalable data services or failover
n
ab
Node 1
Node 2
Node 3
Node 4
data services with the global device and file system infrastructure.
A maximum of sixteen nodes are supported.
There are common redundant interconnects between all nodes.
The Pair+N configuration is well suited for scalable data services

The limitations of a Pair+N configuration is that there can be heavy data

traffic on the cluster interconnects. You can increase bandwidth by adding
more cluster transports.
3-10

N+1 Topology
The N+1 topology, shown in Figure 3-3, enables one system to act as the
storage backup for every other system in the cluster. All of the secondary
paths to the storage devices are connected to the redundant or secondary
system, which can be running a normal workload of its own.
Switch
Switch
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
Storage
Storage
Storage
g
n
s
ba licen
a
b 3-3 le N+1 Topology Configuration
Figure
u
k
a erab
y
l
f
u nThe
s
d
b
a features of the N+1 configurations are as follows:
a
r
t
g
n non
a
The secondary node is the only node in the configuration that is
b
a
Node 1
Primary
Node 2
Primary
Node 3
Primary
Node 4
Secondary
physically connected to all the multihost storage.
The backup node can take over without any performance

degradation.
The backup node is more cost effective because it does not require
additional data storage.
This configuration is best suited for failover data services.

A limitation of the N+1 configuration is that if there is more than one

primary node failure, you can overload the secondary node.

3-11
Scalable Storage Topology
In a scalable, or N*N topology, shown in Figure 3-4, more than two nodes
can be physically connected to the same storage.
Switch
Switch
as
Node 3
Node 4y) h
Node 2
Node 1
m
m
co e
m
cs Guid
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
Storage
Storage
g
n
s
ba licen
a
b
u
le
k
b
a
a
Figure
3-4
Scalable
Storage Topology
y fer
l
u
s
bd tran
a
g on- This configuration is required for running the ORACLE Real Application
n
a
n
Clusters (RAC) across more than two nodes.
ab
For ordinary, cluster-unaware applications, each particular disk group or
diskset in the shared storage still only supports physical traffic from one
node at a time. However, having more than two nodes physically
connected to the storage adds flexibility and reliability to the cluster.
3-12

NAS Device-only Topology

Sun Cluster supports the Network Appliance (NetApp) Filer NAS
product, Sun ST2510, Sun StorageTek 5200 and 5300, and Sun Storage 7000
series.
Switch
Switch
s
) ha
y
m
om
c
e
cs3 m GuidNode
Node 1
Node 2
Ne
ode
4
t
@
b den
u
k
ya Stu
(
Network (not the cluster tranport)
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
NAS Device (Sun or Netapp)
b
e
u
l
k
b
ya fera
l
u
s 3-5 NAS Device-Only Topology
bd tranFigure
a
g onn
a
n
ab
Note There is support for using a NAS device as a quorum device, and
support for NAS data fencing. These concepts are described in more detail
later in this module.
To support the specific NetApp Filer implementation you must install a
package called NTAPclnas on the cluster nodes. This package is available
for NetApp customers with a support contract from
http://now.netapp.com.

3-13
Data Replication Topology
Starting in Sun Cluster 3.2, Sun Cluster supports a data replication

topology. In this topology, shown in Figure 3-6 data storage is not
physically multi-ported between nodes but is rather replicated between
storage attached to the individual nodes using controller based
replication.
Switch
s
) ha
Switch
y
m
Node 1
Node
om2
c
csm Guide
e
Data Replication
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
kaStorage
toReplication Medium
g
e
an cfiber,
nspublic network, or some combination)
b(Private
e
a
li
b
e
u
l
k
a erab
y
l
u nsf
d
b
a 3-6 Data Replication Topology
a
trFigure
g
n
n
no
aba
At the time of the writing this course, the supported mechanisms for
performing the data replication are Hitachi TrueCopy, Hitachi Universal
Replicator and EMC SRDF,
The data replication topology is ideal for wider area clusters where the
data replication solution is preferred to the extra connectivity that would
be involved to actually connect the storage to nodes that are far apart.
This topology would be ideal with the quorum server feature introduced
in Sun Cluster 3.2.
3-14

Non-Storage Topology
This can be suitable for an application that is purely compute-based and
requires no data storage. Prior to Sun Cluster 3.2, two-node clusters
require shared storage because they require a quorum device. Starting with
Sun Cluster 3.2, you could choose to use a quorum server device instead,
described later in this module.
Single-Node Cluster Topology

In this configuration, one node or domain forms the entire cluster. This
configuration allows for a single node to run as a functioning cluster. It
offers users the benefits of having application management functionality
and application restart functionality. The cluster starts and is fully
functional with just one node.
s
) ha
ab
y
m
om
c
csm Guide
e
tto manage a cluster,
@ how
Single node clusters are ideal for usersu
learning
n
b
e
d
k agenttudevelopment
to observe cluster behavior (possibly
purposes), or
afor
y
(
S
to begin a cluster with the intention
of
adding
nodes
as
time
goes on.
n
i
is
d
h
t
u
al use
m
Sun Cluster 3.2 provides,
o the first time, the ability to experience
ka even
tfor
g
e
application n
failovers,
s on a single-node cluster; you could have an
a
n
b
e
application
a thatlicfails over between different non-global zones on the node.
b
u
le
k
b
a
a
l y Single
ernode clusters can also be useful in the Sun Cluster Geographic
f
u
s
d
Edition product, which manages a partnership of two clusters with data
ab -tranreplication
g
across a wide area. Each member of such a partnership must
an non
be a full Sun Cluster installation, and a one-node cluster on either or both
ends is acceptable.

3-15
The Sun Cluster Geographic Edition Software: A

Cluster of Clusters
Sun Cluster Geographic Edition allows you to implement a disasterrecovery scenario by forming a conceptual cluster of clusters across a
wide area. Application data is then replicated using data replication.
A typical scenario using Sun Cluster Geographic Edition is illustrated in
Figure 3-7:
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in tPrimary
Figure 3-7 Data Replication d
Between
his and Secondary Clusters
u
l
e
a us
m
a
o software has the following
k e tEdition
The Sun Cluster g
Geographic
properties: an
b licens
a
b Cluster
Sun
u
le Geographic Edition software is configured on top of
k
b
a
a
l y sstandard
er Sun Cluster software on the participating clusters.
f
u
d
ab -tran Exactly two clusters are involved in the relationship shown in the
g
n non
diagram, and are said to form a partnership.
a
b
a
3-16
There is no conceptual limit to the distance between the two clusters.
Sun Cluster Geographic Edition does not currently provide an

automatic failover of an application across the two clusters. Instead it
provides very simple commands to migrate an application (either
nicely or forcefully) across a wide area, while simultaneously
performing the correct operations on the data replication framework.
Sun Cluster provides support for hardware (RAID-controller) based

data replication (currently Hitachi TrueCopy, Hitachi Universal
Replicator, and EMC SRDF) as well as software based data
replication (Available Suite 4.0 remote mirroring (formerly SNDR))

More Complex Partnerships Using Sun Cluster Geographic

Edition
It is possible to set up a Sun Cluster Geographic Edition partnership as a
more symmetric entity as shown in Figure 3-8.
s
) ha
y
m
m
co e
Figure 3-8 Each Cluster is Primary for a Differentm
Application
cs Guid
e
t
@ it isepossible
n
b
While each partnership has only two clusters,
for a single
u
d
k
u
a
cluster to be a member of different
partnerships,
as
shown
in
figure
t
y
(
S
n
Figure 3-9
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Figure 3-9
Cluster Participating in Multiple Partnerships

3-17
Differences in Single Cluster and Sun Cluster Geographic Edition

(two cluster partnership) Configurations
The following points reinforce the main concepts of Sun Cluster

Geographic Edition by comparing its elements to a single cluster
configuration:
Sun Cluster Resource Groups control manual and automatic

migration/failover of applications within a single cluster. Sun
Cluster Geographic Edition Protection Groups provide a framework
for control of application migration and data replication between
remote clusters, but the actual migration/takeover is manual (an
easy three-word command)
s
) ha
Single cluster configurations do support data replication as an

alternative to full storage multi-porting between the nodes. This
allows single clusters to run in a wider area (campus or metro
clusters) without having to connect nodes to storage that is far
away). Sun Cluster Geographic Edition depends on data replication to
provide a disaster recovery scenario for data and applications that
can be an arbitrarily wide distance apart.
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
3-18

Describing Quorum Votes and Quorum Devices

The cluster membership subsystem of the Sun Cluster 3.2 software
framework operates on a voting system. Following is an explanation of
how this voting system operates:
Each node is assigned exactly one vote.
Certain devices can be identified as quorum devices and are assigned

votes. The following are types of quorum devices:
Directly attached, multiported disks.

Disks are the traditional type of quorum device and have been
supported in all versions of Sun Cluster 3.x.
s
) ha
ab
y
m
om
Quorum server quorum devices.
c
ide votes
csmofGallupossible
There must be a majority (more than 50 percent
e
t
present) to form a cluster or remainbin@
a cluster.
n
e
u
ak Stud
y
(
in this
d
u
Why Have Quorum Voting
e All?
al usat
m
ka e to
g
n presented
Given thearules
ns in the bulleted items listed in the previous
b
e
c
a
section,
it is clear
li by looking at a simple two-node cluster why you need
b quorum
e
u
l
extra
k
b device votes. If a two-node cluster had only node votes, then
a
a
r
y
feneed to have both nodes booted to run the cluster. This defeats one of
ul nyou
s
d
b
the
a major goals of the cluster, which is to be able to survive node failure.
a
r
t
g
an non
NAS quorum devices.
But why have quorum voting at all? If there were no quorum rules, you
could run as many nodes in the cluster as were able to boot at any point in
time. However, the quorum vote and quorum devices solve the following
two major problems:
Failure fencing
Amnesia prevention
These are two distinct problems and it is actually quite clever that they are
solved by the same quorum mechanism in the Sun Cluster 3.x software.
Other vendors cluster implementations have two distinct mechanisms for
solving these problems, making cluster management more complicated.

3-19
Failure Fencing
As shown in Figure 3-10, if interconnect communication between nodes

ceases, either because of a complete interconnect failure or a node
crashing, each node must assume the other is still functional. This is called
split-brain operation. Two separate clusters cannot be allowed to exist
because of the potential for data corruption. Each node tries to establish a
cluster by gaining another quorum vote. Both nodes attempt to reserve
the designated quorum device. The first node to reserve the quorum
device establishes a majority and remains as a cluster member. The node
that fails the race to reserve the quorum device aborts the Sun Cluster
software because it does not have a majority of votes.
Node 2 (1)
Node 1 (1)
s
) ha
y
m
CCR Database
CCR Database
om
c
Quorum
Quorum
cs=m Guide
device
device =
e
@ ent
b
u
ak Stud
y
(
in this
QD(1)
d
u
al use
m
ka eStorage
to array
g
n ens
baFailure
Figure 3-10
a
lic Fencing
b
e
u
l
k rab
a
y
e
ul nsfPrevention
d
Amnesia
b
a
tra
g
n
n
no If it were allowed to happen, a cluster amnesia scenario would involve one
aba
Interconnect
or more nodes being able to form the cluster (boot first in the cluster) with
a stale copy of the cluster configuration. Imagine the following scenario:
1.
In a two-node cluster (Node 1 and Node 2), Node 2 is halted for

maintenance or crashes.
2.
Cluster configuration changes are made on Node 1.
3.
Node 1 is shut down.
4.
You try to boot Node 2 to form a new cluster. If allowed, the cluster
would have lost configuration changes.
There is more information about how quorum votes and quorum devices
prevent amnesia in Preventing Cluster Amnesia With Persistent
Reservations on page 3-31.
3-20

Quorum Device Rules
The general rules for quorum devices are as follows:
A quorum device must be available to both nodes in a two-node

cluster.
Quorum device information is maintained locally in the Cluster

Configuration Repository (CCR) database.
A traditional disk quorum device can also contain user data.
The maximum and optimal number of votes contributed by quorum

devices should be the number of node votes minus one (N-1).
s
) ha
If the number of quorum devices equals or exceeds the number of

nodes, the cluster cannot come up if too many quorum devices fail,
even if all nodes are available. Clearly, this is unacceptable.
y
m
om
c
m greater
Quorum devices are not required in clusters
swith
idethan two
c
u
e
G availability.
nodes, but they are recommended for@
higher cluster
t
n
b
ku tude configured by
A single quorum device can be
automatically
a
y
S using a shared disk quorum
scinstall, for a two-node
only,
n (cluster
s
i
i
d
h
device only.
lu se t
a
u are manually configured after the Sun
am devices
o
All other quorum
k
t
g nsinstallation
e
nsoftware
Cluster
is complete.
a
b
e
c
a
li devices are configured (specified) using DID device
b Disk quorum
e
u
l
k
a enames.
ab
r
y
l
f
bdu trans
a
g onn
a
n Quorum Mathematics and Consequences
ab
When the cluster is running, it is always aware of the following:
The total possible quorum votes (number of nodes plus the number of
disk quorum votes defined in the cluster)
The total present quorum votes (number of nodes booted in the

cluster plus the number of disk quorum votes physically accessible
by those nodes)
The total needed quorum votes, which is greater than 50 percent of the
possible votes

3-21
The consequences are quite simple:
A node that cannot find the needed number of votes at boot time
freezes waiting for other nodes to join to obtain the needed vote
count.
A node that is booted in the cluster but can no longer find the needed
number of votes kernel panics.
Two-Node Cluster Quorum Devices

As shown in Figure 3-11, a two-node cluster requires a single quorum
device; this is typically a quorum disk. The total votes are three. With the
quorum disk, a single node can start clustered operation with a majority
of votes (two votes, in this example).
s
) ha
y
m
om
c
Interconnect
csm Guide
e
@ ent
b
u
Node 1 (1)
ak Stud Node 2 (1)
y
(
in this
d
u
al use
m
ka e to
g
QD(1)
ban licens
a
b
u
le
k
b
a
a
Storage Array
y fer
l
u
s
bd trFigure
an 3-11 Two-Node Cluster Quorum Devices
a
g
n non
a
b
a
3-22

Clustered-Pair Quorum Disk Devices

A clustered-pair configuration, shown in Figure 3-12, always has an even
number of cluster nodes. The nodes in each pair usually provide data
service failover backup for one another.
Switch
Switch
Node 1
(1)
Node 2
(1)
Node 3
(1)
Node 4
(1)
s
) ha
y
m
m
o
c
QD(1)
e
m
d
s
i
ec t Gu
@
QD(1)
QD(1)
b den
u
k
Sun StorEdgeya
( is Stu
MultiPackin
System
lud se th
a
u
Figure 3-12 Clustered-Pair
Devices
am toQuorum
k
ng ense
a
b
aare manylicpossible split-brain scenarios. Not all of the possible
There
b
e
u
lcombinations
split-brain
allow the continuation of clustered operation.
k
b
a
a
r
y
Thefe
following is true for a clustered pair configuration without the extra
ul nquorum
s
d
b
device shown in Figure 3-12.
a
a
r
t
g
n non
a
There are six possible votes.
b
a
One quorum device from each pair was configured to allow one
node from each pair together to form the cluster and to keep the
cluster up.
A quorum is four votes.
If both quorum devices fail, the cluster can still come up.
The nodes wait until all are present (booted).
If Nodes 1 and 2 fail, there are not enough votes for Nodes 3 and 4 to
continue running.
A token quorum device between Nodes 2 and 3 can eliminate this

problem. An entire pair of nodes can fail, and there are still four votes out
of seven. Another possibility is to use a quorum server quorum device.

3-23
Pair+N Quorum Disks
Figure 3-13 shows a typical quorum disk configuration in a Pair+2

configuration. Three quorum disks are used.
Switch
Switch
Node 1
(1)
Node 2
(1)
Node 3
(1)
Node 4
(1)
s
) ha
y
m
QD(1)
om
c
sm uide
c
e
G
QD(1)
t
@
n
b
ku tude
a
QD(1)
y
n ( this S
i
d
lu se
a
m toDevices
Figure 3-13 Pair+N a
Quorum
u
k
ng ense
a
b
The following
a is true
lic for the Pair+N configuration shown in Figure 3-13:
b
e
u
l
k Thereraare
b three quorum disks.
a
y
l
e
f
bdu transThere are seven possible votes.
a
g onn
A quorum is four votes.
a
n
ab
3-24
Nodes 3 and 4 do not have access to any quorum devices.
Nodes 1 or 2 can start clustered operation by themselves.
Up to three nodes can fail (Nodes 1, 3, and 4 or Nodes 2, 3, and 4),

and clustered operation can continue.

N+1 Quorum Disks
The N+1 configuration shown in Figure 3-14 requires a different

approach. Node 3 is the failover backup for both Node 1 and Node 2.
Switch
Switch
Node 1
(1)
Node 2
(1)
Node 3
(1)
s
) ha
y
m
om
c
csm Guide
e
QD(1)
QD(1)
@ ent
b
u
ak Stud
y
(
in this
d
u
e
al Devices
Figure 3-14 N+1 Quorum
s
m
u
ka e to
g
The following
ban lisictrue
ensfor the N+1 configuration shown in Figure 3-14:
a
b Thereleare five possible votes.
u
k
a erab
y
l
u nsf A quorum is three votes.
d
b
a
tra
g
If Nodes 1 and 2 fail, Node 3 can continue.
n
n
o
a
n
b
a

3-25
Quorum Devices in the Scalable Storage Topology
Quorum Devices in the scalable storage topology, as shown in Figure 3-15,

differ significantly from any other topology.
Switch
Switch
Node 1
(1)
Node 2
(1)
Node 3
(1)
s
) ha
y
m
om
c
sm uide
c
e
QD(2)
G
t
@
n
b
ku tude
a
y
( is SStorage Topology
Figure 3-15 Quorum Devices ininthe Scalable
d
lu se th
a
u devices shown in Figure 3-15:
aformthe quorum
The following is true
o
k
t
g nse
The a
single
banquorum
e device has a vote count equal to the votes of the
c
i
l
b
unodes directly
le attached to it minus one.
k
b
a
a
l y sfer
u
d
an This rule is universal. In all the previous examples, there were two
ab -trNote
g
n non nodes (with one vote each) directly connected to the quorum device, so
a
b
a
that the quorum device had one vote.
The mathematics and consequences still apply.
The reservation is performed using a SCSI-3 Persistent Group

Reservation, discussed in more detail later in this module.
If, for example, Nodes 1 and 3 can intercommunicate but Node 2 is
isolated, Node 1 or 3 can reserve the quorum device on behalf of
both of them.
Note It would seem that in the same race, Node 2 could win and
eliminate both Nodes 2 and 3. The section entitled Intentional
Reservation Delays for Partitions With Fewer Than Half of the Nodes on
page 3-37 shows why this is unlikely.
3-26

Using a NAS Device as a Quorum Device

You can use a NAS device as a quorum device. In a two-node cluster, it
can be your single-vote quorum device. In a cluster with more than two
nodes, it is configured as a quorum device accessible to all nodes and
given a number of votes that is one fewer than the number of nodes in the
cluster. This architecture is illustrated in Figure 3-16.
Switch
Switch
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tranFigure 3-16 NAS Device Used as a Quorum Device
g
n non
a
b
a
Node 1 (1)
Node 2 (1)
Node 3 (1)
Node 4 (1)
Network (not the cluster transport)
NAS D evice Q D (3 votes)

iSCSI LUN
NAS Quorum Device Implementation
The quorum implementations for NAS use iSCSI:
NetApp uses a vendor-specific client implementation. This requires

that the NTAPclnas package be installed on each node
Sun NAS uses the native iSCSI built into Solaris 10. You can not use
Sun NAS as quorum device in Solaris 9.
Note At the time of writing this course, you can only use Sun NAS 5200
and 5300 as a 2-node quorum device. Its iSCSI LUNs do not fully support
the SCSI-3 PGR. protocol required for a quorum device attached to more
than two nodes. No other supported NAS devices have this restriction.

3-27
On the NAS device side, the requirements for operation as a Sun Cluster
quorum device are as follows:
You must install the iSCSI license from your NAS device vendor.
You must configure an iSCSI Logical Unit (LUN) for use as the
quorum device.
When booting the cluster, you must always boot the NAS device
before you boot the cluster nodes.
On the cluster side, the requirements and restrictions are as follows:
A cluster can use a NAS device for only a single quorum device.
There should be no need for other quorum devices. This is true
because, in a cluster of more than two nodes, the quorum acts like a
SCSI-3 quorum device attached to all the nodes.
s
) ha
y
m
om
c
Multiple clusters using the same NAS device can

use separate
sm
ide iSCSI
c
u
e
G
LUNs on that device as their quorum devices.
t
@
n
b
ku tude
a
y
n ( this S
i
d
lu se
a
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba
3-28

Quorum Server Quorum Devices

Sun Cluster 3.2 introduced a new kind of quorum device called a quorum
server quorum device.
The quorum server software is installed on some machine external to the
cluster. A quorum server daemon (scqsd) runs on this external machine.
The daemon essentially takes the place of a directly connected quorum
disk. Figure 3-17 shows a cluster using the quorum server as the quorum
device.
Switch
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
Node 1 (1) am
Node 2 (1)
Node 3 (1)
Node 4 (1)
o
k
t
ng ense
a
b
c transport)
a thelicluster
b
Network (not
e
u
l
k rab
a
y
ul nsfe
d
b
scqsd daemon (3 votes for this cluster)
a
tra
g
[ can be quorum for other clusters too ]
n
n
o
a
n
b
a
Switch
External machine running Quorum

Server Software
Figure 3-17 Quorum Server Quorum Devices

The characteristics of the quorum server quorum device are:
The same quorum server daemon can be used as a quorum device

for an unlimited number of clusters.
The quorum server software must be installed separately on the

server (external side).
No additional software is necessary on the cluster side.

3-29
A quorum server is especially useful when there is a great physical

distance between quorum nodes. It would be an ideal solution for a
cluster using the data replication topology.
A quorum server can be used on any cluster where you prefer the
logic of having a single cluster quorum device with quorum device
votes automatically assigned to be one fewer than the node votes.
For example, with a clustered pairs topology, you might prefer the
simplicity of a quorum server quorum device. In that example, any
single node could boot into the cluster by itself, if it could access the
quorum server. Of course, you might not be able to run clustered
applications unless the storage for a particular application is also
available, but those relationships can be controlled properly by the
application resource dependencies that we will learn about in
Module 10.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
3-30

Preventing Cluster Amnesia With Persistent Reservations

Quorum devices in the Sun Cluster software environment are used not
only as a means of failure fencing but also as a means to prevent cluster
amnesia.
Earlier you reviewed the following scenario:
1.
In a two-node cluster (Node 1 and Node 2), Node 2 is halted for

maintenance.
2.
Meanwhile Node 1, running fine in the cluster, makes all sorts of

cluster configuration changes (new device groups, resource groups).
3.
Now Node 1 is shut down.
s
) ha
y
m
4. You try to boot Node 2 to form a new cluster.

om
c
m allowed
ide to boot
cswere
In this simple scenario, the problem is that if you
u
e
G
t
@
Node 2 at the end, it would not have thebcorrect
copy
of the CCR. Node 2
n
e is no other copy
dthere
ku(because
would have to use the copy that it a
has
u
t
y
( changes
Sto the cluster configuration
available) and you would loseinthe
s
i
d
h
made in Step 2.
lu se t
a
am to u
k
e quorum involves persistent reservations that
The Sun Cluster
ng software
sbooting
a
n
b
e
prevent
Node
2
from
into the cluster. It is not able to count the
c
a
i
l
b
quorum
device
u
le as a vote. Therefore, Node 2 waits until the other node
k
b
a
a
boots
to
achieve
the correct number of quorum votes.
y fer
l
u
s
bd tran
a
ang non
ab

3-31
Persistent Reservations and Reservation Keys
A persistent reservation means that reservation information on a quorum

device:
Survives even if all nodes connected to the device are reset
Survives even after the quorum device itself is powered on and off
Clearly, this involves writing some type of information on the disk itself.
The information is called a reservation key and is as follows:
Each node is assigned a unique 64-bit reservation key value.
Every node that is physically connected to a quorum device has its

reservation key physically written onto the device. This set of keys is
called the keys registered on the device.
s
) ha
y
m
om
c
e
Exactly one nodes key is recorded on the devicesm
reservation
idany
c as the
u
holder, but this node has no special priviligesegreater
than
other
G
t
@
n
b
registrant. You can think of the reservation
e as the last node to
u holder
d
k
u
a
t
ever manipulate the keys, but the
(y reservation
S holder can later be
n
s
i
i
fenced out by another registrant.
lud se th
a
u
am
Figure 3-18 and Figure
3-19 onto
page 3-33 consider two nodes booted into a
k
ngto aequorum
se device.
cluster connected
a
n
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
Node 2
tra Node 1
g
n
n
o
a
n
ab
Quorum Device
Node 1 key
Node 2 key
Figure 3-18 Two Nodes Booted and Connected to a Quorum Device
3-32

If Node 1 needs to fence out Node 2 for any reason it will preempt Node
2s registered key off of the device. If a nodes key is preempted from the
device, it is fenced from the device. If there is a split brain, each node is
racing to preempt the others key.
Node 1
Node 2
Quorum Device
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
n Cluster
ithe
Figure 3-19 Node 2 Leaves
d
his
t
u
l
e
a us
m
a
to is clear. The reservation is persistent, so if a
Now the rest g
of k
the equation
e
s cluster, a node cannot count the quorum device
node is b
booting
an into
nthe
e
c
a
i
vote
key is already registered on the quorum device.
b unlessleitslreservation
u
k
a Therefore,
ab in the scenario illustrated in the previous paragraph, if Node 1
r
y
l
e
f
subsequently
goes down so there are no remaining cluster nodes, only
u ns
d
b
1s key remains registered on the device. If Node 2 tries to boot first
a
tra Node
g
n
n
into
the
cluster, it will not be able to count the quorum vote, and must
o
a
n
b
a
wait for Node 1 to boot.
Node 1 key
Node 2 key
After Node 1 joins the cluster, it can detect Node 2 across the transport
and add Node 2s reservation key back to the quorum device so that
everything is equal again.
A reservation key only gets added back to a quorum device by another
node in the cluster whose key is already there.

3-33
SCSI-2 and SCSI-3 Reservations
Sun Cluster 3.2 supports both SCSI-2 and SCSI-3 disk reservations. The
default policy is called pathcount:
Disks to which there are exactly two paths use SCSI-2.
Disks to which there are more than two paths (for example, any disk
with physical connections from more than two nodes) must use
SCSI-3.
Module 5 describes how you can change the first of these policies so that
you could use SCSI-3 even for disks with only two paths. The next several
subsections outline the differences between SCSI-2 and SCSI-3
reservations
s
) ha
y
m
om
c
e
SCSI-2 Reservations and SCSI-2 Persistent Group
Reservation
csm G
uid
e
Emulation (PGRE)
@ ent
b
u
d
aaksimple
tureservation
SCSI-2 reservations themselves provide
mechanism
y
(
S
n out
s
i
i
(first one to reserve the deviced
fences
the
other
one)
but
it
is not
h
t
u
l
e
persistent and does not involve
a registered
s keys. In other words, SCSI-2 is
u
amfencing
sufficient to supportkthe
goals
in Sun Cluster but does not include
o
t
g
e
the persistence
required to implement amnesia prevention.
ban licens
a
b
e
u
lamnesia
To
implement
prevention using SCSI-2 quorum devices,
k
b
a
a
r
y
Cluster
fe must make use of Persistent Group Reservation Emulation
ul Sun
s
d
n
(PGRE)
b
a to implement the reservation keys. PGRE has the following
a
r
t
g
an non characteristics:
ab
The persistent reservations are not supported directly by the SCSI-2

command set. Instead, they are emulated by the Sun Cluster
software.
Reservation keys are written (by the Sun Cluster software, not
directly by the SCSI reservation mechanism) on private cylinders of
the disk (cylinders that are not visible in the format command, but
are still directly writable by the Solaris OS).
The reservation keys have no impact on using the disk as a regular
data disk, where you will not see the private cylinders.
3-34
The race (for example, in a split brain scenario) is still decided by a

normal SCSI-2 disk reservation. It is not really a race to eliminate the
others key, it is a race to do a simple SCSI-2 reservation. The winner
will then use PGRE to eliminate the others reservation key.

SCSI-3 Persistent Group Reservation (PGR)
SCSI-3 reservations have the Persistent Group Reservations (PGR)

mechanism built in. They have the following characteristics:
The persistent reservations are implemented directly by the SCSI-3

command set. Disk firmware itself must be fully SCSI-3 compliant.
Removing another nodes reservation key is not a separate step from

physical reservation of the disk, as it is in SCSI-2. With SCSI-3, the
removal of the other nodes key is both the fencing and the amnesia
prevention.
SCSI-3 reservations are generally simpler in the cluster because

everything that the cluster needs to do (both fencing and persistent
reservations to prevent amnesia) is done directly and simultaneously
with the SCSI-3 reservation mechanism.
s
) ha
y
m
om
c
e two
m more
idthan
With more than two disk paths (that is, anycs
time
u
e
Guse SCSI-3.
nodes are connected to a device), Sun@
Cluster n
must
t
b
ku tude
a
y
( is S
n
i
d
SCSI-3 PGR Scenario
More
th Than Two Nodes
luWith
e
a
am to us
k
In Figure 3-20,
ngfourennodes
se are all physically connected to a quorum drive.
a
b
Remember
the
single
c quorum drive has three quorum votes.
a
i
l
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran Node 1
Node 3
Node 4
g
Node 2
an non
ab
Node 1 key
Node 2 key
Node 3 key
Node 4 key
Quorum Device
Figure 3-20 Four Nodes Physically Connected to a Quorum Drive

3-35

Now imagine that because of multiple transport failures there is a
partitioning where Nodes 1 and 3 can see each other over the transport
and Nodes 2 and 4 can see each other over the transport.
In each pair, the node with the lower reservation key tries to eliminate the
registered reservation key of the other pair. The SCSI-3 protocol assures
that only one pair will remain registered (the operation is atomic). This is
shown in Figure 3-21.
Node 1
Node 2
Node 3
Node 4
s
) ha
y
m
om
c
sm uide
c
e
G
Quorum Device
t
@
n
b
Node 1 key
ku tud(3evotes)
a
Node 2 key
y
n ( this S
i
Node 3 key
d
u se
Nodea
4lkey
am to u
k
ng ense
a
b
a OnelPair
Figureb3-21
ic of Nodes Eliminating the Other
e
u
l
k rab
a
y
e
fdiagram,
ul In n
the
Node 1 has successfully won the race to eliminate the
s
d
b
a
a
keys
for
Nodes
2
and 4. Because Nodes 2 and 4 have their reservation key
r
t
g
n
n
they cannot count the three votes of the quorum device.
no eliminated,
aba
Because they fall below the needed quorum, they will kernel panic.
Cluster amnesia is avoided in the same way as in a two-node quorum
device. If you now shut down the whole cluster, Node 2 and Node 4
cannot count the quorum device because their reservation key is
eliminated. They would have to wait for either Node 1 or Node 3 to join.
One of those nodes can then add back reservation keys for Node 2 and
Node 4.
3-36

NAS Quorum and Quorum Server Persistent

Reservations
Both NAS quorum and quorum server provide reservation key-based
persistent emulations. Fencing and amnesia prevention is provided in an
analogous way to how they are provided with a SCSI-3 quorum device. In
both implementations, the keys are maintained in a persistent fashion on
the server side; that is, the state of the registration keys recorded with the
quorum device survives rebooting of both the cluster nodes and the
quorum server device.
Intentional Reservation Delays for Partitions With ) has

Fewer Than Half of the Nodes
my
ab
om
c
idetalk to each
Imagine the same scenario just presented, but three
nodesu
can
csm
e
Gas shown in
other while the fourth is isolated on the cluster
transport,
t
@
n
b
Figure 3-22.
ku tude
a
y
n ( this S
i
d
lu seNode 3
a
m 2to u (isolated)
aNode
Node 1
Node 4
k
g
e
(delay)
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
Quorum Device
an non
Node 1 key
Node 2 key
Node 3 key
Node 4 key
Figure 3-22 Node 3 Is Isolated on the Cluster Transport

Is there anything to prevent the lone node from eliminating the cluster
keys of the other three and making them all kernel panic?
In this configuration, the lone node intentionally delays before racing for
the quorum device. The only way it can win is if the other three nodes are
really dead, or if each is isolated and delaying the same amount. The
delay is implemented when the number of nodes that a node can see on
the transport (including itself) is fewer than half the total nodes.

3-37
Data Fencing
Data Fencing
As an extra precaution, nodes that are eliminated from the cluster because
of quorum problems also lose access to all shared data devices.
The reason for this is to eliminate a potential timing problem. The node or
nodes that remain in the cluster have no idea whether the nodes being
eliminated from the cluster are actually still running. If they are running,
they will have a kernel panic (after they recognize that they have fallen
beneath the required quorum votes). However, the surviving node or
nodes cannot wait for the other nodes to kernel panic before taking over
the data. The reason nodes are being eliminated is that there has been a
communication failure with them.
s
) ha
y
m
To eliminate this potential timing problem, which could otherwise

m lead to
o
c
data
data corruption, before a surviving node or nodes reconfigures
the
e
m
d
s
i
c from
and applications, it fences the eliminated node or e
nodes
uall shared
G
t
@
data devices, in the following manner:
b den
u
k
u
treservation
ya SCSI-2
With the default pathcount policy,
is used for
(
S
n
s
i
i
two-path devices and SCSI-3
devices
th with more than two paths.
lud for
e
a
s SCSI-3 even if there are only two
mpolicy
uuse
You can change a
the
to
o
k
t
paths (seenModule
g n5).
se
a
b
e
a do uselthe
Ifbyou
ic default SCSI-2 for a two-path device, data fencing is
e
u
l
k just rthe
breservation and does not include any PGRE.
a
a
y
feNetApp Filer NAS devices, a surviving node informs the NAS
ul nsFor
d
b
a
tra device to eliminate the NFS share from the eliminated node or nodes.
g
n
n
a
no
ab
3-38
Data fencing is released when a fenced node is able to boot successfully

into the cluster again.

Data Fencing
Optional Data Fencing

Beginning with Sun Cluster 3.2 1/09 (Update 2) you have the option to
disable fencing, either on a disk-by-disk basis or globally in the cluster.
We will see how this can be done in Modules 4 and 5.
It is highly recommend to keep the fencing on normal SCSI-capable
shared disks.
With the new option to disable fencing, Sun Cluster can support SATA
disks that are incapable of either SCSI-2 or SCSI-3 fencing in any cluster,
or disks incapable of SCSI-3 fencing in a cluster where more than two
nodes are connected to the storage. Sun Cluster can also support access to
a storage device from servers outside of the cluster, if fencing is disabled
on the device.
s
) ha
y
m
om
c
sm uide
c
e
Quroum Device on a Disk with No Fencing
G
t
@
n
b
e
ku ontua ddisk
a
Sun Cluster can support a quorum
device
on which it is doing
y
( is S
n
i
neither SCSI-2 nor SCSI-3 fencing.Sun
Cluster
will
implement
a
d e th
u
l
software reservationa
process, s
whereby races for the quorum devices can
m
u
a
be decided atomically
and
reliably
without use of any SCSI-2 or SCSI-3
o
k et
g
protocols.aThe
persistent
for a disk with no fencing works
nsas areservation
bthe nsame
e
exactly
way
disk
on
which
you are doing SCSI-2 fencing
c
a
li
b
e
u
l
k(the persistent
b reservation is emulated using PGRE).
a
a
r
y
ul nsfe
d
b
a
tra
g
n
n
no
aba

3-39
Configuring a Cluster Interconnect

There are two variations of cluster interconnects: point-to-point and switchbased.
Point-to-Point Cluster Interconnect

In a two-node cluster, you can directly connect interconnect interfaces
using crossover cables. Figure 3-23 shows a point-to-point interconnect
configuration using 100BASE-T interfaces.
Node 2
Node 1
s
) ha
y
m
om
c
csm Guide
e
t board
@ eSystem
hme2
System board
hme2
n
b
u
ak Stud
y
(
in Interconnect
Figure 3-23 Point-to-Point Cluster
d
his
t
u
l
e
a us
m
a
k 3.2esoftware
to installation, you must provide the
During the Sun Cluster
g
n eninterfaces
s
names of the
for each cable.
baend-point
c
a
i
l
b
u
le provide the wrong interconnect interface names during
k
b
Caution
If
you
a
a
l ythe initial
erSun Cluster software installation, the first node is installed
f
u
s
d
an errors, but when you try to manually install the second node, the
ab -trwithout
g
n non installation hangs. You have to correct the cluster configuration error on
a
b
a
the first node and then restart the installation on the second node.
System board
3-40
hme1
hme1
System board

Switch-based Cluster Interconnect

In cluster configurations with more than two nodes, you must join the
interconnect interfaces using switches. You can also use switches to join
two-node cluster interconnects to prepare for the expansion of the number
of nodes at a later time. A typical switch-based interconnect is shown in
Figure 3-24.
During the Sun Cluster 3.2 software installation, you are asked whether
the interconnect system uses switches. If you answer yes, you must
provide names for each of the switches.
Switch
s
) ha
y
m
om
c
Switch
csm Guide
e
@ ent
b
u
ak Stud
y
(
Node 4
Node 2
Node 1
in thisNode 3
d
u
l
e
a us
m
a
k e to
g
Figure 3-24
Switch-based Cluster Interconnect
ban licens
a
b
u
le
k
b
a
a
l y Note
er If you specify more than two nodes during the initial portion of
f
u
s
d
ab -tranthe Sun Cluster software installation, the use of switches is assumed.
g
n non
a
b
a
Cluster Transport Interface Addresses and Netmask
During the Sun Cluster software installation, the cluster interconnects are
assigned IP addresses based on a base address of 172.16.0.0. If
necessary, you can override the default address, but this is not
recommended. Uniform addresses can be a benefit during problem
isolation.

3-41

The netmask property associated with the entire cluster transport
describes, together with the base address, the entire range of addresses
associated with the transport. For example, if you used the default base
address of 172.16.0.0 and the default netmask of 255.255.240.0, you
would be dedicating an 12-bit range (255.255.240.0 has 12 0s at the
end) to the cluster transport. This range is 172.16.0.0 172.16.15.255.
Note When you set 255.255.240.0 as the cluster transport netmask
you will not see this netmask actually applied to any of the private
network adapters. Once again, the cluster uses this netmask to define the
entire range it has access to, and then subdivides the range even further to
cover the multiple separate networks that make up the cluster transport.
s
) ha
y
m
omon
Choosing the Cluster Transport Netmask Based
c
e
Anticipated Nodes and Private Subnetsecsm uid
G
t
@
n
b
deby hand, the cluster
kunetmask
u
a
While you can choose the cluster transport
t
y
n ( this S
prefers instead that you specify:
i
d
lu number
e of private networks
a
s
The maximum anticipated
m
u
a
kanticipated
to number of nodes
g
e
The maximum
an cens
b
a
li anticipated number of virtual clusters
The
b maximum
e
u
l
k
a Virtual
abclusters are another term for zone clusters. This feature is
r
y
l
e
f
in detail in the Advanced Sun Cluster Administration
bdu transdiscussed
a
course.
ang non
ab
Note In Sun Cluster 3.2, if you want to restrict private network

addresses with a class C-like space, similar to 192.168.5.0, you can do it
easily even with relatively large numbers of nodes and subnets, and zone
clusters.
3-42

Identifying Cluster Transport Interfaces
Identifying network interfaces is not an easy task. To accurately determine

the logical name of each interface on a system, use the following steps:
1.
Look for network interfaces. Solaris 10 has a convenient command

that lists all of your adapters.
host01:/#
ce0 link:
ce1 link:
ce2 link:
ce3 link:
2.
dladm show-dev
unknown
speed:
unknown
speed:
unknown
speed:
unknown
speed:
100
100
100
100
Mbps
Mbps
Mbps
Mbps
duplex:
duplex:
duplex:
duplex:
full
full
full
full
s
) ha
With a public network address, verify which interfaces are already

up. These are not candidates for a cluster transport interface.
y
m
# ifconfig -a
om
c
lo0: flags=1000849<UP,LOOPBACK,RUNNING,MULTICAST,IPv4>
csm Guide mtu
8232 index 1 inet 127.0.0.1 netmask ff000000
e
@ ent
b
ce0: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4>
mtu
u
d
k
u
a
1500 index 2 inet 129.200.9.2
netmask
ffffff00
broadcast
t
(y is S
n
129.200.9.255
i
lud se th
ether 8:0:20:96:3:86
a
kam e interface
to u (at this point, it might be an actual
3. Choose g
a candidate
s ready to be set up as a secondary public net, or
private
annet,cor
none
b
e
a
notliconnected to anything at all).
b whenle
u
k
a erab
y
l
u nsf
d
b
a
tra
g
n
n
a
no
ab

3-43

4.
Make sure that the interface you are looking at is not actually on the
public net:
a.
In one window run the snoop command for the interface in

question:
# ifconfig ce1 plumb
# snoop -d ce1
hope to see no output here
b.
In another window, ping the public broadcast address, and

make sure no traffic is seen by the candidate interface.
# ping public_net_broadcast_address
If you do see snoop output, you are looking at a public network
adapter. Do not continue. Just return to step 3 and pick a new
candidate.
s
) ha
y
m
omcheck
5. Now that you know your adapter is not on the public
net,
to
c
e
m
see if it is connected on a private net. Make up some
unused
id subnet
casprivate
unetwork.
e
G
address just to test out interconnectivity across
Do
t
@
n
b
e
not use addresses in the existing public
dspace.
ku subnet
u
a
t
y
# ifconfig ce1 192.168.1.1
n ( thupis S
i
d
u try toseguess the matching candidate on
6. Perform Steps 4 and a
5 lto
u
am atocorresponding
the other node.kChoose
IP address, for example
g
e
192.168.1.2.
an cens
b
a
i can ping across each private network, for
lnodes
7. Test
b that lthe
e
u
k
a example:
ab
r
y
l
e
f
ping 192.168.1.2
bdu trans#192.168.1.2
a
is alive
ang non
ab
8.
After you have identified the new network interfaces, bring them
down again. Cluster installation fails if your transport network
interfaces are still up from testing.
# ifconfig ce1 down unplumb
9.
3-44
Repeat Steps 3 through 8 with transport adapter candidates for the

second cluster transport. Repeat again if you are configuring more
than two cluster transports.

Identifying Public Network Adapters
Identifying Public Network Adapters

You will not be asked about public network configuration at the time you
are installing the cluster.
The public network interfaces must be managed by IPMP, which can be
administered either before or after cluster installation. In this course, you
will do it after cluster installation in Module 9.
Because you are identifying your private transport interfaces before
cluster installation, it can be useful to identify your public network
interfaces at the same time, so as to avoid confusion.
s
) ha
y
m
om
c
csm Guide
e
# ls -l /etc/hostname.*
@ ent
b
u
# ifconfig -a
ak Stud
y
(
inpublicthnetwork
is
d
You can verify your secondary
adapter, if applicable, by:
u
l
e
a
s
u
aitm
Making sure
is notto
one of those you identified to be used as the
k
g nse
ntransport
private
a
b
ceit can snoop public network broadcast traffic
a
i
l
Making
sure
b
u
le
k
b
a
a
ifname plumb
l y sfe##r ifconfig
u
d
snoop
-d
ifname
b tran (other window
a
or node)# ping -s pubnet_broadcast_addr
g onn
a
n
ab
Your primary public network adapter should be the only one currently
configured on the public network. You can verify this with the following
command:

3-45
Configuring Shared Physical Adapters
Configuring Shared Physical Adapters

Recall that certain adapters are capable of participating in 802.1q tagged

VLANs, and can be used as both private and public network adapters
assuming that the switches are also capable of tagged VLANs. This allows
blade architecture servers that have only two physical network adapters
to be clustered and to still have redundant public and private networks.
An adapter that is participating in a tagged VLAN configuration is
assigned an instance number 1000*(Vlan_identifer) +
physical_instance_number.
s
) ha
For example, if you have a physical adapter ce1, and it is participating in

a tagged VLAN with ID 3 as its public network personality, and a tagged
VLAN with ID 5 as its private network personality, then it will appear as if it
were two separate adapters ce3001 and ce5001.
y
m
om
c
csm Guide
e
@ ent
b
u
Configuring the Public Network
ak Stud
y
(
in this
d
u
e network personality, all you have
To configure a shared adapters
al public
s
m
u
to do is configure the
kaadapter
toinstance according to the mathematical
g
e
formula above.
s is going to be used for the public network
anVLAN
nID3
bwhat
e
identity of
would
otherwise
be ce1, therefore you configure the
c
a
li
b
e
u
l
adapter
k instance
b ce3001 by creating a file /etc/hostname.ce3001. When
a
a
r
y
instance
ce3001
is plumbed, the adapter driver understands that it is
l
e
f
u
s
d
an tagged VLAN ID 3 on physical instance number 1.
ab -trusing
g
n non
a
b
a
Allocating a Different VLAN ID for the Private Network
You should never configure the private network ID manually. Instead,

you should perform the initial configuration using the scinstall utility.
This procedure is documented in Module 4. All you need to do is ensure
that you have a different VLAN ID for the public and private networks.
The scinstall utility automatically detects a tagged VLAN-capable
adapter and queries for the private VLAN ID.
3-46

Exercise: Preparing for Installation

Task 1 Verifying the Solaris OS
Task 2 Identifying a Cluster Topology
Task 3 Selecting Quorum Devices
Task 4 Verifying the Cluster Interconnect Configuration
Task 5 Selecting Public Network Interfaces
Preparation
s
) ha
y
m
omhosts through
To begin this exercise, you must be connected to the cluster
c
m asuiuser
de root.
the cconsole tool, and you should be logged into
csthem
e
G
t
@
n
b
de variable names, such
kusee italicized
u
a
t
y
Note During this exercise, when
you
S
n (node1,
s
i
i
as IPaddress, enclosure_name,
or
clustername embedded in a
d
h
t
lu thesnames
e
a
command string, substitute
appropriate
for your cluster.
m
u
a
o
k et
g
n
ba licens
a
Taskku
1b Verifying
the Solaris OS
le
b
a
a
l y sfer
u
d
ab -tranIn this section, you verify that the boot disk is correctly partitioned on all
g
nodes.
an non
ab

1.
Type the /usr/sbin/prtconf command on each node and record

the size of physical memory (/usr/sbin/prtconf | grep
Memory).
Node 1 memory: _______________
Node 2 memory: _______________
2.
Type the df -h command on each node. You should be running a

ZFS root. Verify the root pool using zpool status.
3.
Type swap -l on each node and verify that it has at least 750 Mbytes
of swap space. When you have ZFS-root, swap space is on a zvol.

3-47
Task 2 Identifying a Cluster Topology

1.
Record the desired topology configuration of your cluster in

Table 3-1.
Table 3-1 Topology Configuration

Number of nodes
Number of storage arrays
Types of storage arrays
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
Task 3 Selecting Quorum Devices
ak Stud
y
(
in this
d
u
al use
m
ka ofequorum
to devices you must configure after the
1. Record the g
number
an software
ns installation.
clusterbhost
e
c
a
li
b
e
u
Number
of
quorum
devices: __________
l
k
b
a
a
l y sfer
u
d
an Consult with your instructor if you are not sure about your
ab -trNote
g
an non
2.
ab
Verify that the storage arrays in your cluster are properly connected
for your target topology. Recable the storage arrays if necessary.
quorum device configuration.

2.
Decide whether the scinstall utility will automatically be able to

choose a quorum device (it can for a single quorum device for a twonode cluster).
Automatic Quorum Configuration: (yes/no)?
3.
If there can be no automatic quorum configuration (for a three-node

cluster, for example), type the format command and record the
logical path to the disks that you want to use as quorum disk drives
in your storage arrays.
Quorum disks: ________________________________
Type Control-D to cleanly exit the format utility.
3-48

Task 4 Verifying the Cluster Interconnect

Configuration
This task describes how to verify the cluster interconnect configuration.
Skip this section if your cluster interconnect is not point-to-point.
Perform the following steps to configure a point-to-point Ethernet
interconnect:
1.
Determine the names of your cluster interconnect adapters.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
d uses an
2. Complete the form in Figure 3-25
ak if your
tucluster
y
(
S
Ethernet-based point-to-point
interconnect
configuration.
n this
i
d
lu se
a
Node 2
Node 1
am to u
k
Primary ng
Primary
se
a
n
b
Interconnect
Interconnect
e
c
a
i
l
b
Interface
Interface
e
u
l
k
b
ra
ya fSecondary
l
e
Secondary
u
s
bd tran Interconnect
Interconnect
a
g
n
Interface
Interface
n
o
a
n
b
a
Note You can use the strategy presented on page 3-43, if you are remote
from the cluster equipment, you want to pretend you are remote, or your
instructor does not want to tell you so that you gain experience doing it
yourself. Do not use IP addresses in the existing public network space.
Figure 3-25 Ethernet Interconnect Point-to-Point Form

Perform the following steps to configure a switch-based Ethernet
interconnect:
3.
Complete the form in Figure 3-26 on page 3-50 if your cluster uses an
Ethernet-based cluster interconnect with switches.
a.
Record the logical names of the cluster interconnect interfaces

(bge2, qfe1, and so forth).

3-49
yourself.
b.
Add or delete nodes to the diagram as appropriate.
Node 1
Switch 1
Node 2
Primary
interconnect
interface
Primary
interconnect
interface
Secondary
interconnect
interface
Secondary
interconnect
interface
s
) ha
y
m
om
Node 3
Node 4
c
m
Primary
Primary
s interconnect
ide
c
u
interconnect
e
G interface
interface
t
@
n
b
Secondary
Secondary
ku tude interconnect
a
interconnect
y
interface
interface
n ( this S
i
d
lu se
a
am to uSwitch 2
k
ng ense
a
b
a
lic Interconnect With Switches Form
b
Figure
3-26 Ethernet
e
u
l
k rab
a
y
fe
ul 4. nsVerify
d
that each Ethernet interconnect interface is connected to the
b
a
a
r
t
g
correct switch. If you are remote, and you do not want to take your
n non
a
b
instructors word for it, you can verify that all nodes can ping each
a
other across the private switches by using the strategy presented on

page 3-43.
Note If you have any doubt about the interconnect cabling, consult with
your instructor now. Do not continue this exercise until you are confident
that your cluster interconnect system is cabled correctly, and that you
know the names of the cluster transport adapters.
3-50

Task 5 Selecting Public Network Interfaces
Ask for help from your instructor in identifying public network interfaces
on each node that can be used in IPMP groups.
Perform the following steps to select public network interfaces:
1.
Record the logical names of potential IPMP Ethernet interfaces on

each node in Table 3-2.
yourself. Do not use IP addresses in the existing public network range.
s
) ha
y
m
om
c
ide
Table 3-2 Logical Names of Potential IPMP Ethernet
csm Interfaces
u
e
G
t
@
n
b
Primary IPMP ku Backup
deIPMP
System
u
a
t
y
interface (
S
n this interface
i
d
lu se
Node 1
a
am to u
k
Node 2
ng ense
a
b
a3 (if any)lic
Node
b
u
le
k
b
a
a
Node
4
(if
y fer any)
l
u
bd trans
a
g onn
a
n
ab
Note It is important that you are sure about the logical name of each
public network interface (ce1, qfe3, and so on).

3-51
Exercise Summary
Exercise Summary

Experiences
Interpretations
Conclusions
Applications
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
3-52

Module 4
Installing and Configuring the Sun Cluster

Software Framework
s
) ha
Objectives
ab
y
m
om
c
m uide
csconfiguration
Understand the Sun Cluster installation and
steps
e
G
t
@
n
b
e Java Enterprise
Install the Sun Cluster packageskusing
Sun
a u Sthe
tud
System (Java ES) installer (y
in this
d
u
Describe the Sun a
Cluster
configuration
l framework
e
s
m
u
a installation
Configure akcluster
using all-at-once and typical modes
to
g
e
n
s
n installation using one-at-a-time and custom
Configure
ba liacecluster
a
b modesle
u
k
a eConfigure
ab additional nodes for the one-at-a-time method
r
y
l
f
bdu trans Describe the Solaris OS files and settings that are automatically
a
configured by scinstall
ang non
Perform automatic quorum configuration
Describe the manual quorum selection
Perform post-installation configuration
4-1
Relevance
Relevance

What advantages are we given by standardizing on the Java

Enterprise System (Java ES) installer as the utility to lay down the
cluster packages?
Why might you not want the configuration utilities to automatically

choose a quorum device for you for a two-node cluster?
Are there pieces of the cluster configuration you might need to

configure manually after running the configuration utility
scinstall?
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
4-2

820-7458 (for Sun Cluster 3.2 11/09).
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Installing and Configuring the Sun Cluster Software Framework

4-3
Sun Cluster Software Installation and Configuration

You must complete two distinct steps to successfully initialize the
Sun Cluster 3.2 framework and boot into the cluster:
1.
Install the Sun Cluster 3.2 framework packages.
2.
Configure the Sun Cluster 3.2 software with scinstall.
It is required that these be done as separate steps, although both steps can
be automated as part of Solaris jumpstart.
Introduction to Sun Cluster Package Installation
s
) ha
y
m
An initial (non-upgrade) installation of the Sun Cluster packages

om is
c
performed one of the following ways

csm Guide
e
t
@ epackaged
Using the Java ES installer utility that isbalways
with the
n
u
Sun Cluster framework (this can also
ud as part of a
ak be Sautomated
t
y
(
jumpstart install).
in this
d
u
e on a system where the Sun
Using a flash image that
al wasuscreated
m
Cluster packages
kahaveebeen
to installed using the Java ES installer. The
g
s been created before the cluster was actually
flash image
have
an must
n
b
e
c
a
configured with
li scinstall.
b
e
u
l
k
a erab
y
l
f
u Note
s
d
n
b
a Flash images are supported with a ZFS-root environment starting
a
r
t
g
an non with Solaris 10 10/09 (Update 8).
ab
Sun Cluster Packaging

You always install the Sun Cluster packages the same way despite the fact
that Sun Cluster 3.2 may be bundled with either:
The full collection of Java ES software (Sun Cluster, Java System Web
Server, Messaging Server, Directory Server, and so on).
A partial set of Java ES software, containing only availability-related

products. This is called the Java ES Availability Suite.
The download currently comes as a single zip file (for either SPARC or
x86 version) or a single DVD image containing both.
4-4

Sun Cluster 3.2 Software Licensing
No license keys are required for the Sun Cluster software. You must,
however, furnish paper license numbers to obtain service.
Patches for the OS and for the Sun Cluster Software

You must consistently manage the following types of patches:
OS and hardware patches

These must be installed before the Sun Cluster framework packages
are installed. OS patches should be consistent between cluster nodes,
although when you add new ones after cluster configuration it is
almost always possible to do so in a rolling fashion, patching and
rebooting one node at a time.
s
) ha
ab
y
m
om
c
sm uide
c
e
G
Sun Cluster framework patches
t
@
n
b
deSun Cluster framework
ku aftertuthe
Available patches should be added
a
y
( ithe
SSun Cluster framework is
packages are installed but
n before
s
i
d
h
configured.
lu se t
a
u
am
o
You add these
patches
manually before you configure the Sun
k
t
g nsewith scinstall.
nframework
Cluster
a
b
ctoe add new patches after the cluster is already installed
ayou need
i
l
b
If
u
le
k
b
a
a
and
your
applications are already running, you will usually be able
y fer
l
u
s
bd tran to do so in a rolling fashion.
a
ang non
Sun Cluster Data Service Agent patches
You cannot add these until you install the particular agent being
patched.

4-5
Installing the Sun Cluster Packages With the Java ES Installer
Installing the Sun Cluster Packages With the Java ES

Installer
The Java ES installer provides both graphical and terminal-based

interfaces for installing the Sun Cluster software. Other Java System
applications can be installed along with Sun Cluster when Sun Cluster is
included in a full Java ES release.
Prerequisites for Installing Sun Cluster Software
s
) ha
Before you can use the Java ES installer to install the Sun Cluster software
framework on a cluster node, you must meet the following prerequisites:
y
m
m
1. Boot disks must be configured according to Sun Clusterostandards,
c
as defined in Module 3 of this course.

e
m
d
s
i
ec t Gu
2. The Solaris OS and OS patches must be installed.
@
b den
u
k
ya Stu
(
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
4-6

Auxiliary Software Automatically Installed by Java ES

Installer
Sun Cluster 3.2 has several required auxiliary components. All of these
components are automatically installed or upgraded as needed when you
use the Java ES installer to install the Sun Cluster software.
Auxiliary Components Listed as Shared Components

The JES installer groups a large handful of auxiliary components as shared
components. These are some examples; there are several other Java libraries
and similar components..
s
) ha
y
m
oma Sun Cluster

This is a Java application that serves as a container
for
c
e of Java
remote management interface that is implemented
sm asuaidseries
c
e
G this interface is
objects. In the standard Sun Cluster implementation,
t
@
n
b
e
used only by the Sun Cluster Manager
and by the data
dinterface
ku web
u
a
t
y
service wizards that are part
n ( ofthclsetup.
sS
i
i
d
Java Runtime Environment
lu se1.5
a
u
am toDevelopment
The Java Management
Kit (JMDK) runtime libraries
k
g
e
n
s
These
ba Javaliclibraries
en are required by the management objects.
a
b The Sun
u
le Java Web Console Application 3.0.2
k
b
a
a
r Java Web Console serves as a single point of entry and single
l y sfeSun
u
d
ab -tran sign-on (SSO) for a variety of Web-based applications, including the
g
Sun Cluster Manager used in the Sun Cluster software.
an non
The Common Agent Container (CACAO)
ab

4-7
Auxiliary Components Listed as Separate Applications
For the initial release of Sun Cluster 3.2, there is one component listed on
the Java ES install screens as a separate application.
This is Java DB, Suns implementation of the Apache pure-Java database
standard known as Derby.
Like the shared components, it is automatically selected when installing
the Sun Cluster software.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
4-8

Running the Java ES Installer

If installing directly from CDROM or DVD medium, the Java ES installer
is run from CDROM 1 of 2 or from the DVD. If installing from spooled
software, recall that CDROM 1 of 2 and CDROM 2 of 2 must have
previously been spooled together into the same arena.
The installer runs as a graphical utility if you have a correct X-Windows
DISPLAY variable set in your environment. It can run as a terminal-based
utility if you explicitly run the installer with the -nodisplay option.
Launching the Java ES Installer
s
) ha
y
m
om
c
csm Guide
e
# DISPLAY=display-name-or-IP:display-#;
@ ent export DISPLAY
b
u
# cd cdrom-or-dvd-spooled-arena-path/Solaris_sparc
ak Stud
y
(
# ./installer
in this
d
u
e the initial screen for the graphical
al usshows
The screen shownainmFigure 4-1
k ES installer
to (the splash screen is not shown, and some
version of thegJava
e
nasslicense acceptance, are also omitted on following
other screens,
ban lsuch
e
c
a
i
b
pages).
e
u
l
k
a erab
y
l
u nsf
d
b
a
tra
g
n
n
a
no
You run the installer from the Solaris_sparc or Solaris_x86
subdirectory of CD-ROM 1 of 2 (or of the DVD or combined spooled
arena).
ab
Figure 4-1
Java ES Installer Initial Screen

4-9
Component Selection Screen

You can choose the Java ES components that you want. The screen shown
in Figure 4-2 is stretched to show the entire component selection screen
from the Java ES Availability Suite distribution. Recall that this
distribution includes only the availability products.
In the screenshot, Sun Cluster 3.2 is selected. This includes the Sun Cluster
core product and the Sun Cluster Manager.
A check box near the bottom of the screen allows you to install
multilingual support. By default this is checked. Adding the multilingual
packages for Sun Cluster can significantly increase the time required for
your installation.
s
) ha
y
m
om
Note that you do not need to explicitly select the All Shared
c
Components. A later screen will inform you about which

smof these
ideneed to
c
u
e
G
be installed or upgraded, Note that the Java @
DB is listedt separately,
and
n
b
noted as required since we have chosen tokinstall
Sun
Cluster.
u ude
a
t
y
(
S
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab
Figure 4-2
4-10
Java ES Installer Component Selection

Shared Components Screen
The screen shown in Figure 4-3 shows only the shared components which
the Java ES installer has determined must be installed or upgraded.
Note in this example some shared components listed on a previous page
of this module, such as JRE 1.5 and Web Console are not shown. This is
because the base OS we are running (Solaris 10 Update 8) already contains
the correct versions of these components; this determination is made
automatically, correctly, and silently by the Java ES installer.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Figure 4-3
Java ES Installer shared components

4-11
Java ES Configuration: Configure Later
For some of the other Java System applications, the Java ES installer has
the ability to actually configure the applications. It does not have this
ability for the Sun Cluster framework, and you must choose the Configure
Later option, as shown in Figure 4-4.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l yFigure
er Choose Configure Later for Sun Cluster
f
4-4
u
s
d
ab -tran
g
n non
a
b
a
Note If you choose Configure Now you will get a screen saying that
this option is not applicable for Sun Cluster. However, you will also get an
extra screen asking if you want to enable the remote configuration utility
(discussed later). If you choose Configure Later as is shown here, the
installer assumes you do want to enable remote configuration.
Software Installation
After confirmation, the installer proceeds to install the shared components
(the auxiliary software identified on page 4-7), and then the actual
Sun Cluster framework packages.
You do not need to reboot your node before proceeding to the
configuration of the Sun Cluster framework. You will be allowing the
configuration utility, scinstall, to reboot all of your nodes.
4-12

Configuring the User root Environment
The root login environment should include the following search path and
man page information:
PATH=$PATH:/usr/cluster/bin
MANPATH=$MANPATH:/usr/cluster/man:/usr/share/man
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

4-13
Sun Cluster Framework Configuration

The Sun Cluster configuration is performed using one of the following
two methods:
Using the scinstall utility interactively This is the most common

method of configuring Sun Cluster, and the only one that is
described in detail in this module.
Using JumpStart software off a jumpstart server The scinstall

can be run on the jumpstart server to provision the jumpstart server
so that the client (node) performs cluster configuration as part of
jumpstart finish scripts.
If you used this method, you would manually provision the
jumpstart server one of two ways:
s
) ha
y
m
om the
c
Provide clients a flash image from a cluster node

on which
de
isoftware
Java ES installer had been used to install
cluster
cthesm
u
e
G scinstall
t
@
packages. The jumpstart finish scripts
then run
the
n
b
de
configuration functionality. aku
u
t
y
n ( thifresh
s S OS (or a flash image from
i
Provide the clients ad
completely
lu packages.).
e
a Solaris without
cluster
The jumpstart finish scripts
a
s
m
u
a Java EStoinstaller to provide the cluster packages,
then run k
the
ng run
and
then
the
sescinstall configuration functionality.
a
n
b
e
a
lic
b
e
u
l
k rab
a
y
Understanding
the installmode Flag
ul nsfe
d
b
a
tra
g
n
n
a
no As you configure Sun Cluster software on cluster nodes and reboot the
ab
nodes into the cluster, a special flag called the installmode flag is set in
the cluster CCR. When this flag is set, the following happens:
The first node installed (node ID 1) has a quorum vote of one.
All other nodes have a quorum vote of zero.
This allows you to complete the rebooting of the second node into the
cluster while maintaining the quorum mathematics rules. If the second
node had a vote (making a total of two in the cluster), the first node
would kernel panic when the second node was rebooted after the cluster
software was installed because the first node would lose operational
quorum.
4-14


One important side effect of the installmode flag is that you must be
careful not to reboot the first node (node ID 1) until you can choose
quorum devices and eliminate (reset) the installmode flag. If you
accidentally reboot the first node, all the other nodes will kernel panic
because they have zero votes out of a possible total of one.
If the installation is a single-node cluster, the installmode flag is not set.
Post-installation steps to choose a quorum device and reset the
installmode flag are unnecessary.
Automatic Quorum Configuration (Two-Node Cluster

a
Only)
as
)h
y
m
On a two-node cluster only, you have the option of having m
the scinstall
co edevice
utility insert a script that automatically chooses your quorum
as
m
d
s
i
the second node boots into the cluster. The defaults
always
u be to
ec will
G
t
accept the option.
@
b den
u
k
tu
yathe firstSdual-ported
(
The quorum device chosen will
be
disk or LUN (the
n
di e this
one with the lowest DIDlu
number).
a us
m
a
kallow automatic
to quorum configuration, the installmode
g
If you choose
to
e
s after the quorum device is automatically
ban licenreset
flag isaautomatically
b
configured.
u
le
k
b
a
a
l y sfer
u
d
b tranYou can disable the two-node cluster automatic quorum configuration for
a
g on- one of the following reasons:
n
a
n
ab
You want to choose the quorum device yourself.
You want to use a NAS device as a quorum device.
You want to use the quorum server as a quorum device.

4-15
Automatic Reset of installmode Without Quorum

Devices (Clusters With More Than Two Nodes Only)
In clusters with more than two nodes, the scinstall inserts a script to
automatically reset the installmode flag. It will not automatically
configure a quorum device. If you want a quorum device, you still have to
do that manually after the installation. By resetting installmode, each
node is assigned its proper single quorum vote.
Configuration Information Required to Run scinstall

The following information is required by scinstall and should be
prepared in advance.
s
) ha
y
m
om
c
Name of the Cluster and Names of All the Nodes

csm Guide
e
@ ent
b
u
k by tthe
The cluster name is just a name agreed a
upon
udnodes, it is not a name
y
(
S
that resolves to an IP address. in
d e this
u
l
a us
m
a
k e to
g
n enmust
s be able to resolve each others host name. If
The nodes b
ina
the cluster
c
a
li is true but the names are not in each nodes
b reason
for some
this
e
u
l
k
a erabfile (the names could have been resolved through a name
/etc/hosts
y
l
f the /etc/hosts file is automatically modified to include these
u server),
s
d
n
b
a
a
trnames.
g
n
n
no
aba
4-16

Cluster Transport IP Network Number and Netmask

As described in module 3, the default cluster transport IP address range
begins with 172.16.0.0 and the netmask is 255.255.240.0. You should keep
the default if it causes no conflict with anything else visible on any other
network. It is perfectly fine for multiple clusters to use the same addresses
on their cluster transports, as these addresses are not visible anywhere
outside the cluster.
Note The netmask refers to the range of IP addresses that are reserved
for all possible cluster transport addresses. This will not match the actual
netmask that you will see configured on the transport adapters if you
check using ifconfig -a.
s
) ha
ab
y
m
m
otransport,
c
If you need to specify a different IP address range for the
you
e
m
d
s
i
can do so. rather than being asked initially for a specific
netmask,
you will
c Gu
e
be asked for the anticipated maximum number
t and physical
@ ofenodes
n
b
u
private networks. A suggested netmask
is
then
calculated
for you.
k tud
a
y
n ( this S
i
d
lu sand
Cluster Transport Adapters
e Switches
a
m
u
ka toe identify
to transport adapters on at least the first
g
You must benprepared
s
a you
nrun
bwhich
e
node on
scinstall. On other nodes you will normally let
c
a
li its auto-discovery
b
e
u
scinstall
use
feature to automatically determine the
l
k
b
a
a
l y transport
er adapters.
f
u
s
d
ab -tranYou can define a two-node cluster topology as using switches or just
g
an non
using point-to-point cables. This does not even need to match the actual
topology; the software really has no way of telling. It is just the definitions
in the Cluster Configuration Repository (CCR), and whichever way you
define it will be the way it is presented when you view the cluster
configuration using command-line commands or the graphical web-based
administration tool.
Names that you provide for switches are arbitrary, and are just used to
match up transport connections between the various nodes.
Port names for specific switch connections are arbitrary except for SCI
switches. For SCI, the port name must match the switch port number to
which an actual cable is connected.

4-17
Partition or Placeholder File System for the

/global/.devices/node@# File System
You can choose one of two options for this file system:
Have a dedicated partition on the boot disk (recommended size

512MB). The standard convention is that you make an empty file
system mounted on /globaldevices as a placeholder.
Use a loop-back (lofi) device. In this case do not have any

placeholder file system. In the absence of any placeholder, the
scinstall utility will ask if you want to use the lofi method. It
will automatically create a 100MB UFS file system using a file that it
creates named /.globaldevices for the backing store.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
4-18

Using DES Authentication for Nodes to Authenticate With Each

Other as They Join the Cluster
By default, nodes are authenticated as they join the cluster using standard
UNIX authentication. A reverse IP address lookup is done for a node
trying to join the cluster, and if the resulting name is in the list of nodes
that you typed in as nodes for this cluster, it is allowed to add itself to the
cluster.
The reasons for considering more stringent authentication are the
following:
Nodes that are adding themselves to the cluster communicate across

the public network.
s
) ha
y
m
om
c
sm uide
c
e
DES authentication, also known as secure remote
procedure
G call (secure
t
@
n
b
RPC) authentication, is a much stronger
authentication
u ude that cannot be
k
a
spoofed by something simple like
yspoofingSIPt addresses.
(
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
A bogus node adding itself to the cluster can add bogus information
to the CCR or copy out the CCR.

4-19
Variations in Interactive scinstall
There are the four different ways of using the interactive scinstall
utility to configure a new cluster:
Configure the entire cluster at once:
Typical installation
Custom installation
Configure cluster nodes one at a time:
Typical installation
Custom installation
s
) ha
y
m
om
Configuring the Entire Cluster at Once
c
csm Guide
e
t run
@cluster,
If you choose the option to configure the entire
you
n
b
e
u
scinstall on only one node. You should
aware
ak be S
tudof the following
y
(
behavior:
in this
d
u
e be the last node to join the cluster
The node you are driving
will
al from
s
m
u
because it needs
kato configure
to and reboot all the other nodes first.
g
e
s node IDs are assigned to the nodes, you
anabout
nwhich
If youbcare
e
c
a
lifrom the node that you want to be the highest node ID,
b drive
should
e
u
l
k
a anderlist
abthe other nodes in reverse order.
y
l
f
bdu transThe Sun Cluster software packages must already be installed on all
a
g onnodes (by the Java ES installer). Therefore you do not need remote
n
a
n
shell access (neither rsh nor ssh) between the nodes. The remote
ab
configuration is performed using RPC installed by the Sun Cluster

packages. If you are concerned about authentication, you can use
DES authentication as described on page 4-19.
4-20

Configuring Cluster Nodes One at a Time
If you choose this method, you run scinstall separately on each node.
You must complete scinstall and reboot into the cluster on the first
node. This becomes the sponsoring node for the remaining nodes.
If you have more than two nodes you can run scinstall simultaneously
on all but the first node, but it might be hard to predict which node gets
assigned which node ID. If you care, you should just run scinstall on
the remaining nodes one at a time, and wait for each node to boot into the
cluster before starting the next one.
s
) ha
ab
y
m
Typical Installation Compared to Custom Installation

om
c
e
idand
csmTypical
u
Both the all-at-once and one-at-a-time methodsehave
Custom
G
t
@
n
b
configuration options (to make a total of
u fouruvariations).
de
k
a
t
y
Sthe following responses:
n ( assumes
s
i
i
The Typical configurationdmode
h
lu se t
a
u172.16.0.0 with netmask 255.255.240.0
It uses network
amaddress
o
k
t
for thencluster
interconnect.
g nse
a
b
e you want to perform autodiscovery of cluster
cthat
Itaassumes
i
l
b
u transport
le adapters on the other nodes with the all-at-once method.
k
b
a
a
r the one-node-at-a-time method, it asks you if you want to use
l y sfe(On
u
d
ab -tran autodiscovery in both Typical and Custom modes.)
g
It uses the names switch1 and switch2 for the two transport
an non
switches and assumes the use of switches even for a two-node

cluster.
It assumes that you either have the dedicated partition identified by

the placeholder /globaldevices file system, or if it doesnt detect
that, it will ask if you want to use the lofi virtual partition.
It assumes that you want to use standard system authentication

(not DES authentication) for new nodes configuring themselves into
the cluster.

4-21
Configuring Using All-at-Once and Typical Modes: Example
Configuring Using All-at-Once and Typical Modes:

Example
The following example shows the full dialog for the cluster installation
that requires the least information, the all-at-once, Typical mode
installation. The example is from a two-node cluster, where the default is
to let scinstall set up a script that automates configuration of the
quorum device. In the example, scinstall is running on the node named
theo, which becomes the second node (node ID 2).
# /usr/cluster/bin/scinstall
*** Main Menu ***
s
) ha
y
m
om
c
e
* 1) Create a new cluster or add a cluster node csm uid
e
G server
2) Configure a cluster to be JumpStarted from
this install
t
@
n
b
3) Manage a dual-partition upgrade
ku tude
a
y
( is S
4) Upgrade this cluster node
dinthis
h
5) Print release informationlu
for
node
tcluster
e
a
am to us
k
* ?) Help with menu options
ng ense
* q) Quit
a
b
a
lic
b
e
u
l
Option: 1 ak
ab
r
y
l
e
f
s Cluster Node Menu ***
du ranand
bCluster
*** New
a
g on-t
n
a
n select from any one of the following options:
ab Please
Please select from one of the following (*) options:
1) Create a new cluster

2) Create just the first node of a new cluster on this machine
3) Add this machine as a node in an existing cluster
?) Help with menu options
q) Return to the Main Menu
Option: 1
4-22


Selecting Option 1 from the scinstall installation menu presents a
description of the requirements necessary for installation of the entire
cluster from a single node as follows:
*** Create a New Cluster ***

This option creates and configures a new cluster.
You must use the Java Enterprise System (JES) installer to install the
Sun Cluster framework software on each machine in the new cluster
before you select this option.
If the "remote configuration" option is unselected from the JES
installer when you install the Sun Cluster framework on any of the new
nodes, then you must configure either the remote shell (see rsh(1)) or
he secure shell (see ssh(1)) before you select this option. If rsh or
ssh is used, you must enable root access to all of the new member
nodes from this node.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
Press Control-d at any time to return to the Main
Menu.
u
ak Stud
y
(
in this
d
u
e
al uyes
Do you want to continue (yes/no) [yes]?
s
m
ka e to
g
an ceMode
ns
Typical or
bCustom
a
li
b
e
u
l
k
b <<<
aCustom
aMode
>>> Typical or
r
y
l
e
f
du rans
b
a
tool
-t supports two modes of operation, Typical mode and Custom.
gThis
n
n
o
a
nmost clusters, you can use Typical mode. However, you might need
ab For
to select the Custom mode option if not all of the Typical defaults
can be applied to your cluster.
For more information about the differences between Typical and Custom
modes, select the Help option from the menu.
Please select from one of the following options:
1) Typical
2) Custom
?) Help
Option [1]: 1

4-23
Cluster Name, Cluster Nodes, and Remote Installation

Confirmation
>>> Cluster Name <<<
Each cluster has a name assigned to it. The name can be made up of
any characters other than whitespace. Each cluster name should be
unique within the namespace of your enterprise.
What is the name of the cluster you want to establish? orangecat
>>> Cluster Nodes <<<
y
m
This Sun Cluster release supports a total of up to 16 nodes.

om
c
m uide
csthe
Please list the names of the other nodes planned for
initial
e
G
t
@
cluster configuration. List one node name per bline.
When
finished,
n
e
u
d
k
type Control-D:
ya Stu
(
n
di e this
u
Node name (Control-D to finish):
vincent
l
a ^D us
m
a
k e to
g
ban licens
a
b
This is the complete
list
u
le of nodes:
k
b
a
a
l y sfer
u
vincent
d
abtheo-tran
g
n non
a
b
a
s
) ha
Is it correct (yes/no) [yes]? yes
Attempting to contact "vincent" ... done

Searching for a remote configuration method ... done
The Sun Cluster framework is able to complete the configuration
process without remote shell access.
Press Enter to continue:
4-24

Cluster Transport Adapters
>>> Cluster Transport Adapters and Cables <<<

You must identify the cluster transport adapters which attach this
node to the private cluster interconnect.
Select the first cluster transport adapter for "vincent":
1)
2)
3)
4)
ce1
ce2
ce3
Other
s
) ha
y
m
m
o
c
yes
Will this be a dedicated cluster transport adapter (yes/no)
[yes]?
e
m
d
s
i
ec t Gu
@
b den
u
k
a Stu
ytraffic
(
Searching for any unexpected network
n
i
is on "ce2" ... done
d
h
t
u
Verification completed. No traffic
was
detected
over a 10 second
l
e
a
s
sample period.
am to u
k
ng transport
se adapter for "vincent":
a
n
Select the second cluster
b
e
a
lic
b
e
u
l
k rab
a
1)yce1
u2)l ce2nsfe
d
b
a
a 3)-trce3
g
n
n
no 4) Other
aba
Option:2
Option: 3
Will this be a dedicated cluster transport adapter (yes/no) [yes]? yes
Plumbing network address 172.16.0.0 on adapter ce2>> NOT DUPLICATE ...

done
done

4-25
Cluster Transport Adapters (Actually Using Shared Adapters)

In the following variation, you actually choose an adapter for the private
network that is already configured (already using tagged VLAN) for the
public network.

You must identify the two cluster transport adapters which attach
this node to the private cluster interconnect.
Select the first cluster transport adapter for "dani":
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
Option: 2
(
in this
d
u
e also, you will need to
al network
This adapter is used on the public
s
m
u
ka adapter
configure it as a tagged VLAN
to for cluster transport.
g
e
an cens
b
a
What is the cluster
VLAN ID for this adapter? 5
li
b transport
e
u
l
k
b
a any
aunexpected
r
y
l
e
Searching
for
network traffic on "ce5000" ... done
f
u ns
d
b
Verification
a
tra completed. No traffic was detected over a 10 second
g
n
sample
period.
n
no
aba
s
) ha
1)
2)
3)
4)
4-26
bge1
ce0
ce1
Other

Automatic Quorum Configuration
>>> Quorum Configuration <<<

Every two-node cluster requires at least one quorum device. By
default, scinstall selects and configures a shared disk quorum device
for you.
This screen allows you to disable the automatic selection and
configuration of a quorum device.
You have chosen to turn on the global fencing. If your shared storage
devices do not support SCSI, such as Serial Advanced Technology
Attachment (SATA) disks, or if your shared disks do not support
SCSI-2, you must disable this feature.
s
) ha
y
m
om
c
sm oruifideyou
cnow,
If you disable automatic quorum device selection
e
G you must
t
@
intend to use a quorum device that is not abshared
disk,
n
de once both nodes
ku tquorum
instead use clsetup(1M) to manually configure
u
a
y
S
have joined the cluster for the first
n ( time.
s
i
i
d
h
lu se t
a
u
Do you want to disable automatic
device selection (yes/no)
am to quorum
k
[no]? no
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
Verification and cluster check
fe
ul nInstallation
s
d
b
a
ra
tokay
g
n
Is
it
to create the new cluster (yes/no) [yes]? yes
n
a
no
ab
During the cluster creation process, cluster check is run on each of

the new cluster nodes. If cluster check detects problems, you can
either interrupt the process or check the log files after the cluster
has been established.
Interrupt cluster creation for cluster check errors (yes/no) [no]?
no

4-27
Letting You Use lofi Device for /global/.devices/node@#
Log file - /var/cluster/logs/install/scinstall.log.1251

Testing for "/globaldevices" on "theo" ... failed
/globaldevices is not a directory or file system mount point.
Cannot use "/globaldevices" on "theo".
Do you want to use a lofi device instead and continue the
installation (yes/no) [yes]? yes
Testing for "/globaldevices" on "vincent" ... failed
y
m
om
c
de
csm
Do you want to use a lofi device instead and continue
theGui
e
@ ent
b
u
ak Stud
y
(
in this
d
u
e
al uMessages
Installation and Configuration
s
m
ka e to
g
Installation and Configuration
ban licens
a
bof thelecluster transport configuration.
u
k
Starting discovery
a erab
y
l
u nsf
d
b
The
a following
tra connections were discovered:
g
n
n
o
ntheo:ce2
switch1 vincent:ce2
aba
s
) ha

Cannot use "/globaldevices" on "vincent".
theo:ce3
switch2
vincent:ce3
Completed discovery of the cluster transport configuration.

Started cluster check on "theo".
Started cluster check on "vincent".
cluster check completed with no errors or warnings for "theo".
cluster check completed with no errors or warnings for "vincent".
Configuring "vincent" ... done
Rebooting "vincent" ...
Configuring "theo" ... done

Rebooting "theo" ...
4-28

Rebooting ...
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

4-29
Configuring Using One-at-a-Time and Custom Modes: Example (First Node)
Configuring Using One-at-a-Time and Custom Modes:

Example (First Node)
The following is an example of using the one-at-a-time node
configuration. The dialog is shown for vincent, the first node in the
cluster. You cannot install other cluster nodes until this node is rebooted
into the cluster and can then be the sponsor node for the other nodes.
vincent:/# /usr/cluster/bin/scinstall
*** Main Menu ***
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
* ?) Help with menu options
in this
d
* q) Quit
u
al use
m
ka e to
Option: 1
g
an cens
b
a
i Menu ***
*** New Cluster andbCluster lNode
e
u
l
k
b
raany
ya from
l
e
Pleaseuselect
one of the following options:
f
s
d
n
b
a
tra a new cluster
g
n
1)
Create
n
n2)o Create just the first node of a new cluster on this machine
aba
s
) ha
* 1)
2)
3)
4)
5)
Create a new cluster or add a cluster node

Configure a cluster to be JumpStarted from this install server
Manage a dual-partition upgrade
Upgrade this cluster node
Print release information for this cluster node

Option: 2
4-30

First Node Introduction
*** Establish Just the First Node of a New Cluster ***
This option is used to establish a new cluster using this machine as

the first node in that cluster.
Before you select this option,the Sun Cluster framework software must
already be installed. Use the Java Enterprise System (JES) installer
to install Sun Cluster software.
Press Control-d at any time to return to the Main Menu.
s
) ha
y
m
m
o
c
Do you want to continue (yes/no) [yes]? yes

e
m
d
s
i
ec t Gu
@
b den
u
k
ya Stu
(
n
Typical or Custom Mode
di e this
u
l
a us
m
a
>>> Typical or Custom Mode k
<<<
to
g
e
an cens
b
a
This tool supports
two
li modes of operation, Typical mode and Custom.
b
e
u
l
k
For mostaclusters,
can use Typical mode. However, you might need
ab youmode
r
y
l
e
to select
the
Custom
option if not all of the Typical defaults
f
u ns
d
b
to your cluster.
a
acan be-trapplied
g
n
n
no more information about the differences between Typical and Custom
For
aba

1) Typical
2) Custom
?) Help
Option [1]:2

4-31
Cluster Name

Each cluster has a name assigned to it.The name can be made up of any
characters other than whitespace. Each cluster name should be unique
within the namespace of your enterprise.
What is the name of the cluster you want to establish?
Option for cluster check
orangecat
s
) ha
y
m
om
This step allows you to run cluster check to verify that c
certain
sm uhave
basic hardware and software pre-configuration requirements
idebeen
c
e
G
met. If cluster check detects potential problems
with configuring
t
@
n
b
this machine as a cluster node, a report ofku
violated
dechecks is
u
a
t
y
prepared and available for display on the
n ( tscreen.
sS
i
i
d
h
lu(yes/no)
e [yes]? no
a
Do you want to run cluster check
s
m
u
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
>>> Check <<<
ab
4-32

Cluster Nodes
>>> Cluster Nodes <<<

This Sun Cluster release supports a total of up to 16 nodes.
Please list the names of the other nodes planned for the initial
cluster configuration. List one node name per line. When finished,
type Control-D:
theo
^D
s
) ha
y
m
m
o
c
vincent
e
m
d
s
i
theo
ec t Gu
@
b den
u
k
Is it correct (yes/no) [yes]? yes
ya Stu
(
n
di e this
u
l
a DES
Authenticating Nodes
With
m
us
a
o
k
t
ngto eAdd
seNodes <<<
>>> Authenticating Requests
a
n
b
a
lic
b
e
u
l
k node
Once the first
itself as a single node cluster,other
a
rabestablishes
y
l
e
nodes u
attempting
to
add
themselves
to the cluster configuration must
f
s
d
n
b
bea found ton
ra the list of nodes you just provided. You can modify this
g
n
list
by
using
claccess(1CL) or other tools once the cluster has been
n
o
a
n
b
established.
a
This is the complete list of nodes:
By default, nodes are not securely authenticated as they attempt to

add themselves to the cluster configuration. This is generally
considered adequate,since nodes which are not physically connected to
the private cluster interconnect will never be able to actually join
the cluster. However, DES authentication is available. If DES
authentication is selected, you must configure all necessary
encryption keys before any node will be allowed to join the cluster
(see keyserv(1M), publickey(4)).
Do you need to use DES authentication (yes/no) [no]? no

4-33
Asking If you Want Two Private Networks
>>> Minimum Number of Private Networks <<<

Each cluster is typically configured with at least two private
networks. Configuring a cluster with just one private interconnect
provides less availability and will require the cluster to spend more
time in automatic recovery if that private interconnect fails.
Should this cluster use at least two private networks (yes/no) [yes]?
yes
s
) ha
Choosing Whether to Define Switches and Switch Names

mm
co e
m
>>> Point-to-Point Cables <<<
cs Guid
e
b@ dent
The two nodes of a two-node cluster may useku
a directly-connected
tu
ya are Sconfigured.
interconnect. That is, no cluster switches
However,
(
n
s
i
i
when there are greater than two nodes,
this
interactive
form
of
d
h
t
u
l
e
a
scinstall assumes that there m
one switch for each
s
a will tobe uexactly
k
private network.
ng ense
a
b
a cluster
Does this two-node
lic use switches (yes/no) [yes]? yes
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
>>>
Cluster
Switches
<<<
n
o
a
n
b
a
All cluster transport adapters in this
"switch". And, each adapter on a given
different switch.Interactive scinstall
switch for each private network in the
cluster must be cabled to a

node must be cabled to a
requires that you identify one
cluster.
What is the name of the first switch in the cluster [switch1]? <CR>
What is the name of the second switch in the cluster [switch2]? <CR>
4-34

Transport Adapters and Connections to Switches

You must configure the cluster transport adapters for each node in
the
cluster. These are the adapters which attach to the private cluster
interconnect.
Select the first cluster transport adapter:
1)
2)
3)
4)
ce1
ce2
ce3
Other
s
) ha
y
m
m
o
c
Option: 2
e
m
d
s
i
ec t Gu
@
Will this be a dedicated cluster transport adapter
n [yes]? yes
b (yes/no)
e
u
d
k
ya Stu
(
n
di e this
u
l
aadapter.
m
us
Adapter "ce2" is an Ethernet
a
o
k
t
ng ensenetwork traffic on "ce2" ... done
a
Searching for any
unexpected
b
a
lic No traffic was detected over a 10 second
b
Verification
completed.
e
u
l
k rab
a
sampleyperiod.
ul nsfe
d
b
aThe "dlpi"
tra transport type will be set for this cluster.
g
n
n
no
aba
Name of the switch to which "ce2" is connected [switch1]? <CR>
Each adapter is cabled to a particular port on a switch. And, each

port is assigned a name. You can explicitly assign a name to each
port. Or, for Ethernet and Infiniband switches, you can choose to
allow scinstall to assign a default name for you. The default port
name assignment sets the name to the node number of the node hosting
the transport adapter at the other end of the cable.
Use the default port name for the "ce2" connection (yes/no) [yes]?
yes

4-35
Second Transport Adapter

Select the second cluster transport adapter:
1)
2)
3)
4)
ce1
ce2
ce3
Other
Option: 3
Will this be a dedicated cluster transport adapter (yes/no) [yes]?yes
Adapter "ce3" is an Ethernet adapter.
y
m
m
Searching for any unexpected network traffic on "ce3" ... done
o
c
Verification completed. No traffic was detected over am

10 second
e
d
s
i
sample period.
ec t Gu
@
b den
u
k
Name of the switch to which "ce3" is connected
<CR>
tu
ya S[switch2]?
(
n
di econnection
his
t
u
Use the default port name for the
"ce3"
(yes/no) [yes]?
l
a
s
yes
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba
s
) ha
4-36

Transport IP Address Range and Netmask
>>> Network Address for the Cluster Transport <<<

The cluster transport uses a default network address of 172.16.0.0.If
this IP address is already in use elsewhere within your enterprise,
specify another address from the range of recommended private
addresses (see RFC 1918 for details).
The default netmask is 255.255.240.0. You can select another netmask,
as long as it minimally masks all bits that are given in the network
address.
ab
s
The default private netmask and network address result in an IP ha
)
y 10
address range that supports a cluster with a maximum of 64 nodes,
m
private networks and 0 virtual clusters.

om
c
de
sm ui[yes]?yes
c(yes/no)
Is it okay to accept the default network address
e
G
t
@
n
b
u ude[yes]? no
k(yes/no)
Is it okay to accept the default netmask
a
t
y
(
S
n
di and
his address will dictate
t
u
The combination of private netmask
network
l
e
a us private networks, and virtual
m
the maximum number of both
nodeso and
a
k e tby a cluster. Given your private
clusters that can be
supported
g
n
a program
ns will generate a range of recommended
network address,
bthis
e
c
a
liare based on the maximum number of nodes and
b that
private netmasks
e
u
l
k
anetworks
aband the virtual clusters that you anticipate for
private
r
y
l
e
f
s
this
n
bdu cluster.
a
a
r
-t
n
o
ang In
n specifying the anticipated maximum number of nodes and private
networks and virtual clusters for this cluster, it is important that
you give serious consideration to future growth potential. While both
the private netmask and network address can be changed later, the
tools for making such changes require that all nodes in the cluster
be booted in noncluster mode.
Maximum number of nodes anticipated for future growth [64]?
Maximum number of private networks anticipated for future growth

[10]? 4
Maximum number of virtual clusters expected [12]?
Specify a netmask of 255.255.255.192 to meet anticipated future

requirements of 4 cluster nodes and 4 private networks.

4-37

To accommodate more growth, specify a netmask of 255.255.255.0 to
support up to 8 cluster nodes and 8 private networks.
What netmask do you want to use [255.255.255.192]? 255.255.255.192

done
done
s
) ha
y
m
om
c
sm uide
c
e
G
>>> Global Devices File System <<<
t
@
n
b
u ude
kfile
a
t
y
Each node in the cluster must have a local
system
mounted on
(
S
n
s
i
i
/global/.devices/node@<nodeID> before
participate
thnotsuccessfully
lud siteiscan
as a cluster member. Since the a
"nodeID"
assigned until
m set
uthis up for you.
awill
o
scinstall is run, scinstall
k
t
ng ense
a
b
c either an already-mounted file system or
a nameliof
You must supply the
b
e
u
l
raw disk partition
k rawhich
b scinstall can use to create the global
a
y
devices
lfile system.
fe This file system or partition should be at least
uin
s
d
n
512
MB
size.
b
a
tra
g
n
n
no
you can use a loopback file (lofi), with a new file
aba Alternatively,
Global Devices File System
system, and mount it on /global/.devices/node@<nodeid>.
If an already-mounted file system is used, the file system must be

empty. If a raw disk partition is used, a new file system will be
created for you.
If the lofi method is used, scinstall creates a new 100 MB filesystem
from a lofi device by using the file /.globaldevices. The lofi method
is typically preferred, since it does not require the allocation of a
dedicated disk slice.
The default is to use /globaldevices.
Is it okay to use this default (yes/no) [yes]? no
Is it okay to use the lofi method (yes/no) [yes]? yes
4-38

Global Fencing
>>> Set Global Fencing <<<

Fencing is a mechanism that a cluster uses to protect data integrity
when the cluster interconnect between nodes is lost. By default,
fencing is turned on for global fencing,and each disk uses the global
fencing setting. This screen allows you to turn off the global
fencing.
Most of the time, leave fencing turned on. However, turn off fencing
when at least one of the following conditions is true: 1) Your shared
storage devices, such as Serial Advanced Technology Attachment (SATA)
disks, do not support SCSI; 2) You want to allow systems outside your
cluster to access storage devices attached to your cluster; 3) Sun
Microsystems has not qualified the SCSI persistent group reservation
(PGR) support for your shared storage devices.
s
) ha
y
m
om
c
csm Guide
e
t cluster
@after
If you choose to turn off global fencing now,
your
n
b
e
u
starts you can still use the cluster(1CL)
ak command
tud to turn on global
y
(
S
fencing.
in this
d
u
al fencing
se (yes/no) [no]? no
m
u
Do you want to turn off a
global
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab

4-39
Automatic Quorum Configuration (Two-Node Cluster)

>>> Quorum Configuration <<<
Every two-node cluster requires at least one quorum device. By

default, scinstall selects and configures a shared disk quorum device
for you.
This screen allows you to disable the automatic selection and
configuration of a quorum device.
You have chosen to turn on the global fencing. If your shared storage
devices do not support SCSI, such as Serial Advanced Technology
Attachment (SATA) disks, or if your shared disks do not support
SCSI-2, you must disable this feature.
s
) ha
y
m
m
o
c
If you disable automatic quorum device selection now,or

if you e
intend
m
d
s
i
u
to use a quorum device that is not a shared disk,you
instead
use
ec must
G
t
@
clsetup(1M) to manually configure quorum once bboth nodes
have joined
n
ku tude
the cluster for the first time.
a
y
n ( this S
i
d
Do you want to disable automatic
luquorumsedevice selection (yes/no)
a
[no]? no
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba
4-40

Automatic Reboot
>>> Automatic Reboot <<<

Once scinstall has successfully initialized the Sun Cluster software
for this machine, the machine must be rebooted. After the reboot,
this machine will be established as the first node in the new
cluster.
Do you want scinstall to reboot for you (yes/no) [yes]? yes
Option Confirmation
s
) ha
y
m
om
Your responses indicate the following options to scinstall:
c
sm uide
c
e
G
Your responses indicate the following options to@
scinstall:
t
n
b
ku tude
a
y
scinstall -i \
n ( this S
i
-C orangecat \
d
lu se
a
-F \
am to u
k
-G lofi \
ng ense
-T node=vincent,node=theo,authtype=sys
\
a
b
c
a
-w
li
b
e
u
l
k
netaddr=172.16.0.0,netmask=255.255.255.192,maxnodes=4,maxprivatenets=4,nu
a er\ab
y
l
mvirtualclusters=2
u nsf
d
b
a trtype=dlpi,name=ce2 -A trtype=dlpi,name=ce3 \
a
tr-A
g
n
-B
-B type=switch,name=switch2 \
n
o -m type=switch,name=switch1
a
n
b
endpoint=:ce2,endpoint=switch1
\
a
>>> Confirmation <<<
-m endpoint=:ce3,endpoint=switch2 \
-P task=quorum,state=INIT
Are these the options you want to use (yes/no) [yes]? yes
Do you want to continue with the this configuration step (yes/no)
[yes]? yes

4-41
Configuration Messages
Some of the post-installation steps for which scinstall is printing out
messages here (NTP, hosts file, nsswitch.conf file, and IPMP) are
described later in this module.
Initializing
Initializing
Initializing
Initializing
Initializing
Initializing
Initializing
Initializing
Initializing
cluster name to "orangecat" ... done

authentication options ... done
configuration for adapter "ce2" ... done
configuration for adapter "ce3" ... done
configuration for switch "switch1" ... done
configuration for switch "switch2" ... done
configuration for cable ... done
configuration for cable ... done
private network address options ... done
y
m
om
c
Setting the node ID for "vincent" ... done (id=1)

csm Guide
e
@ ent
b
u
Verifying that NTP is configured ... done
ak Stud
y
Initializing NTP configuration ... done
(
in this
d
u
l
e
Updating nsswitch.conf ... done ma
s
u
ka e to
g
Adding cluster node entries
/etc/inet/hosts
... done
ns
ban lto
e
c
a
i
b
e
u
l
k
Configuring IP amultipathing
ab groups ...done
r
y
l
e
f
ns management is NOT configured ... done
bduthattrapower
Verifying
a
g on- power management ... done
Unconfiguring
n
a
n
has been renamed to /etc/power.conf.041309025345
ab/etc/power.conf
s
) ha
Power management is incompatible with the HA goals of the cluster.

Please do not attempt to re-configure power management.
Ensure that the EEPROM parameter "local-mac-address?" is set to "true"

... done
Ensure network routing is disabled ... done
Rebooting ...
4-42

Configuring Additional Nodes for One-at-a-Time Method: Example
Configuring Additional Nodes for One-at-a-Time Method:

Example
In the one-at-a-time method, after the first node has rebooted into the
cluster, you can configure the remaining node or nodes. Here, there is
hardly any difference between the Typical and Custom modes. The only
difference is that the Typical mode does not ask about the global devices
file system (it assumes the placeholder is /globaldevices). Here you
have no choice about the automatic quorum selection or the
authentication mechanism because it was already chosen on the first
node.
theo:/# scinstall
s
) ha
y
m
om
c

csm Guide
e
@ ent
b
u
* 1) Create a new cluster or add a cluster
k node
a
tud this install server
y
(
2) Configure a cluster to be JumpStarted
from
S
din e this
3) Manage a dual-partitionluupgrade
anode us
4) Upgrade this cluster
m
a
k e to for this cluster node
5) Print release ginformation
an cens
b
a
* ?) Help b
with menulioptions
u
le
k
b
* q) Quit
a
a
l y sfer
u
d
b tra1n
aOption:
g
an non
*** Main Menu ***
ab
*** New Cluster and Cluster Node Menu ***

Please select from any one of the following options:
1) Create a new cluster
2) Create just the first node of a new cluster on this machine
Option: 3

4-43
Additional Node Configuration and Choosing Typical Instead of

Custom
*** Add a Node to an Existing Cluster ***
This option is used to add this machine as a node in an already

established cluster. If this is a new cluster, there may only be a
single node which has established itself in the new cluster.
Before you select this option,the Sun Cluster framework software must
already be installed. Use the Java Enterprise System (JES) installer
to install Sun Cluster software.
s
) ha
y
m
m
o
c
Press Control-d at any time to return to the Main Menu.

e
m
d
s
i
ec t Gu
@
b den
u
k
Do you want to continue (yes/no) [yes]? y
yes
a Stu
(
n
di e this
u
l
a us
m
a
Typical or CustomkMode to
ng ense
a
b
a
lic
b
e
u
l
k raMode
>>> Typical or aCustom
b <<<
y
l
e
f
u ns
bdtool
This
two modes of operation, Typical mode and Custom.
a
a
rsupports
tclusters,
g
n
For
most
can use Typical mode. However, you might need
n
o the Customyoumode
a
n
b
to
select
option if not all of the Typical defaults
a
can be applied to your cluster.
For more information about the differences between Typical and Custom
1) Typical
2) Custom
?) Help
Option [1]: 1
4-44

Sponsoring Node
After entering the name of the sponsoring node (first node already booted
into the cluster and the cluster name), the authentication and the cluster
name are checked with the sponsoring node.
>>> Sponsoring Node <<<
For any machine to join a cluster, it must identify a node in that
cluster willing to "sponsor" its membership in the cluster. When
configuring a new cluster, this "sponsor" node is typically the first
node used to build the new cluster. However,if the cluster is already
established, the "sponsoring" node can be any node in that cluster.
s
) ha
ab
y
m
om
c
csm Guide
e
@ ent
b
u
k tud
aauthenticate
y
And, if the target cluster uses DES (to
new machines
S
nas new
s
i
i
attempting to configure themselves
cluster
members,
the
d
h
t
u
l
e
a
necessary encryption keys m
before any attempt to
s
uconfigured
a must tbe
o
k
join.
ng ense
a
b
a of the
What is the b
name
lic sponsoring node? vincent
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra Cluster Name
g
n
n
a
no
Already established clusters can keep a list of hosts which are able
to configure themselves as new cluster members.This machine should be
in the join list of any cluster which it tries to join. If the list
does not include this machine, you may need to add it by using
claccess(1CL) or other tools.

Each cluster has a name assigned to it. When adding a node to the
cluster, you must identify the name of the cluster you are attempting
to join. A sanity check is performed to verify that the "sponsoring"
node is a member of that cluster.
What is the name of the cluster you want to join? orangecat
Attempting to contact "vincent" ... done
Cluster name "orangecat" is correct.

4-45
Option for cluster check
>>> Check <<<

This step allows you to run cluster check to verify that certain
basic hardware and software pre-configuration requirements have been
met. If cluster check detects potential problems with configuring
his machine as a cluster node, a report of violated checks is
prepared and available for display on the screen.
Do you want to run cluster check (yes/no) [yes]? no
Cluster Transport Autodiscovery Option
s
) ha
y
m
om
>>> Autodiscovery of Cluster Transport <<<
c
sm uide
c
e
G
If you are using Ethernet or Infiniband adapters
as the tcluster
@
n
b
transport adapters, autodiscovery is the best
defor configuring
ku method
u
a
t
y
the cluster transport.
n ( this S
i
d
lu(yes/no)
e [yes]? yes
a
Do you want to use autodiscovery
s
m
u
ka e to
g
an cens
b
a
Probing ........
li
b
e
u
l
k
a connections
ab
r
y
l
e
The following
were discovered:
f
u ns
d
b
a
tra
g
n
vincent:ce2
switch1 theo:ce2
n
o
a
n
b
vincent:ce3
switch2 theo:ce3
a
Is it okay to add these connections to the configuration (yes/no)
[yes]? yes
Letting You Use lofi Device for for /global/.devices/node@#

Cannot use "/globaldevices".
Do you want to use a lofi device instead and continue the
4-46

Automatic Reboot and Option Confirmation
>>> Automatic Reboot <<<

Once scinstall has successfully initialized the Sun Cluster software
for this machine, the machine must be rebooted. The reboot will cause
this machine to join the cluster for the first time.
Do you want scinstall to reboot for you (yes/no) [yes]? yes
>>> Confirmation <<<
Your responses indicate the following options to scinstall:
s
) ha
y
m
m
scinstall -i \
o
c
-C orangecat \
e
m
d
s
i
-N vincent \
ec t Gu
@
-G lofi \
b den
u
k
-A trtype=dlpi,name=ce2 -A trtype=dlpi,name=ce3
\
ya \ Stu
(
-m endpoint=:ce2,endpoint=switch1
n
di e this
u
-m endpoint=:ce3,endpoint=switch2
l
a us
m
a
o use (yes/no) [yes]? yes
k want
tto
Are these the options
you
g
e
ban licens
a
Do you wantub
to continue
le with this configuration step (yes/no) [yes]?
k
b
a
a
yes
l y sfer
u
d
ab -tran
g
an non
ab

4-47
Configuration Message
Adding
Adding
Adding
Adding
Adding
node "theo" to the cluster configuration ... done

adapter "ce2" to the cluster configuration ... done
adapter "ce3" to the cluster configuration ... done
cable to the cluster configuration ... done
cable to the cluster configuration ... done
Copying the config from "vincent" ... done

Copying the postconfig file from "vincent" if it exists ... done
Setting the node ID for "theo" ... done (id=2)
y
m
m
o
c
Verifying that NTP is configured ... done

e
m
d
s
i
Initializing NTP configuration ... done
ec t Gu
@
b den
u
k
Updating nsswitch.conf ... done
ya Stu
(
n
is
di e th...
u
Adding cluster node entries to /etc/inet/hosts
done
l
a
s
am to u
k
nggroups
se...done
a
n
Configuring IP multipathing
b
e
a
lic
b
e
u
l
k rab
a
y
l
femanagement is NOT configured ... done
s
Verifyingdu
that power
n
b
a management ... done
a
trpower
Unconfiguring
g
n
n
a
has been renamed to /etc/power.conf.041309030943
no
ab/etc/power.conf
Power management is incompatible with the HA goals of the cluster.
s
) ha
Verifying the major number for the "did" driver with "vincent" ... done
Please do not attempt to re-configure power management.

Ensure that the EEPROM parameter "local-mac-address?" is set to "true"
... done
Ensure network routing is disabled ... done
Updating file ("ntp.conf.cluster") on node vincent ... done
Updating file ("hosts") on node vincent ... done
Rebooting ...
4-48

Solaris OS Files and Settings Automatically Configured by scinstall
Solaris OS Files and Settings Automatically Configured by

scinstall
No matter which of the four scinstall variations that you use,
scinstall automatically configures the following files and settings on
each cluster node:
/etc/hosts
/etc/nsswitch.conf
/etc/inet/ntp.conf.cluster
/etc/hostname.xxx
s
) ha
y
m
/etc/notrouter
om
c
local-mac-address? Setting in electrically

erasableide
sm
c
u only)
e
programmable read-only memory (EEPROM)
(SPARC
G
t
@
b den
u
k
ya Stu
(
n
Changes to the /etc/hosts
di eFile
his
t
u
l
a us
m
a
kutility eautomatically
to
The scinstall
adds all the cluster names and IP
g
n
s
a
n
addresses
hosts file if it was not there already. (All the
b to each
cenodes
aalready
i
l
names
had
to
be
resolvable,
through some name service, for
b
e
u
l
k
b
ra to work at all.)
ya scinstall
l
e
f
u
s
bd tran
a
ang nonChanges to the /etc/nsswitch.conf File
/etc/vfstab
ab
The scinstall utility makes the following changes:
It makes sure the files keyword precedes every other name service
for every entry in the file.
It adds the cluster keyword for the hosts and netmasks

keywords. This keyword modifies the standard Solaris OS resolution
libraries so that they can resolve the cluster transport host names and
netmasks directly from the CCR. The default transport host names
(associated with IP addresses on the clprivnet0 adapter) are
clusternode1-priv, clusternode2-priv, and so on. These names
can be used by any utility or application as normal resolvable names
without having to be entered in any other name service.

4-49
Creating the /etc/inet/ntp.conf.cluster File

This file contains a Network Time Protocol Configuration which, if used,
has all nodes synchronize their time clocks against each other (and only
against each other). This file is automatically modified to contain only
lines for cluster nodes defined during scinstall, and therefore should
not need to be modified manually. For a two-node cluster it includes the
lines:
peer clusternode1-priv prefer
peer clusternode2-priv
s
) ha
The entire file is ignored if there is a standard /etc/inet/ntp.conf file

at boot time.
y
m
om
c
Modifying /etc/hostname.xxx Files to Include

de
csm GuiIPMP
e
@ ent
b
u
Any existing /etc/hostname.xxx files that
do
dyet indicate IPMP
k not
u
a
t
y
group membership are rewritten so (that the adapter
S in question is placed
n thwas
s already
i
i
d
in an IPMP group. Whatever
IP
address
in the file (for
u
l
e
a
s given the -failover flag and used
example, the node IP address)
is u
just
amfollowing
o
k
t
as the test address.
The
shows
an example file, as modified by
g
e
n
s
scinstall:ba
en
c
a
i
l
b cat
vincent:/#
u
le /etc/hostname.qfe1
k
b
a
a
l yvincent
ergroup sc_ipmp0 -failover
f
u
s
d
n
ab -trIfayou
g
happen to already for some reason have multiple
an non
ab
/etc/hostname.xxx files with IP addresses on the same subnet, but not

yet in IPMP groups, scinstall will rewrite both files to place them in
the same IPMP group. You are still likely to need to modify these by hand,
for example, to give separate node and test addresses on your primary
adapter so that the node address itself can fail over. We will cover this in
Module 9.
Of course it is perfectly legal to fully configure IPMP with as much
customization as you like before scinstall is run. scinstall detects that
IPMP is already configured, and leaves it alone.
4-50

Modifying the /etc/vfstab File (dedicated partition

for /global/.devices/node@#)
The vfstab file is modified so that the /global/.devices/node@#
mount point replaces any previous placeholder, such as /globaldevices.
In addition, the DID device name is used for this file system rather than
the traditional /dev/rdsk/c#t#d#. This ensures that each of these devices
has a unique name cluster-wide. The following shows the modification to
the vfstab file:
vincent:/etc/inet# grep global /etc/vfstab
#/dev/dsk/c0t0d0s3 /dev/rdsk/c0t0d0s3 /globaldevices ufs 2 yes /dev/did/dsk/d1s3 /dev/did/rdsk/d1s3 /global/.devices/node@1 ufs 2 no
global
s
) ha
y
m
om
c
Note If you use a lofi device it is configured and

smmounted
idevia explicit
c
u
e
SMF actions automatically inserted by scinstall.
There
G is no entry in
t
@
n
b
/etc/vfstab.
ku tude
a
y
n ( this S
i
d
lu se
a
Creating the /etc/notrouter
File
am to u
k
ng ense
a
b
a filelicassures that cluster nodes do not accidentally turn
This
empty
b
u
leinto routers. It is not supported to have a cluster node
themselves
k
b
a
a
l y function
er as a router.
f
u
s
d
ab -tran
g
n non
a
b
Modifying the local-mac-address? EEPROM variable
a
On SPARC systems, this EEPROM variable is set to true so that each

network adapter is given a unique Media Access Control (MAC) address
(that is, Ethernet address for Ethernet adapters) in order to support IPMP.
This is described in more detail in Module 9.

4-51
Automatic Quorum Configuration and installmode Resetting
Automatic Quorum Configuration and installmode

Resetting
On a two-node cluster on which you chose to allow automatic quorum
configuration, the quorum device is chosen (the lowest possible DID
device number) as the second node boots into the cluster for the first time.
If your cluster has more than two nodes, no quorum device is selected
automatically, but the installmode flag is automatically reset as the last
node boots into the cluster.
s
) ha
On the Solaris 9 OS, the auto-configuration of the quorum happens as part

of a boot script on the last node booting into the cluster. It will be
complete by the time you get the login prompt on this last node.
y
m
om
c
In the Solaris 10 OS, as the last node boots into the c

cluster,
sm you
idgete the
u
e
Gthe quorum
login prompt on the last node booting into the@
cluster before
t
n
b
u bootuenvironment
auto-configuration runs. This is because k
the
is
de
aOS,
t
y
controlled by the SMF of the Solaris(10
which
runs
boot
services
in
S
n
s
i
i
parallel and gives you the login
thbefore many of the services are
lud prompt
e
a
complete. The auto-configuration
of
the
s quorum device does not complete
m
u
a
o
until a minute or soklater. Dotnot attempt to configure the quorum device
ngthe eauto-configuration
se
by hand because
eventually runs to completion.
a
n
b
c
a
li
b
e
u
l
k
The
a following
ab console messages indicate the automatic quorum selection:
r
y
l
e
f
s cl_runtime: NOTICE: CMM: Cluster members: vincent
du rantheo
Apr 13 b
03:17:39
a
theo.
g on-t
n
a
n
theo cl_runtime: NOTICE: CMM: node reconfiguration #4
abApr 13 03:17:39
completed.
Apr 13 03:17:39 theo cl_runtime: NOTICE: CMM: Votecount changed from 0 to
1 for node theo.
Apr 13 03:17:39 theo cl_runtime: NOTICE: CMM: Cluster members: vincent
theo.
Apr 13 03:17:39 theo cl_runtime: NOTICE: CMM: node reconfiguration #5
complete
4-52

Manual Quorum Selection

You need to choose a quorum device or quorum devices manually in the
following circumstances:
Two-node cluster where you disabled automatic quorum selection
Any cluster of more than two nodes where a quorum device is

desired
Verifying DID Devices
ab
s
If you are going to be manually choosing quorum devices that are ) ha
yquorum
physically attached disks or LUNs (as opposed to a NAS device
m
or quorum server), you need to know the DID device number

the
om for
c
quorum device or devices that you want to choose.

csm Guide
e
t assigned to the
@DID enumbers
n
b
The cldevice (cldev) command shows
the
u
d shows the mapping
akoption
tuthat
disks in the cluster. The most succinct
y
(
S
inthe corresponding
is
between DID numbers anddall
disk paths is cldev
h
t
u
l
e
list -v.
a us
m
a
k e to
g
sDID device number for the quorum device you
You needa
tonknow the
n
b
e
c
a
choose in the next
li step. You can choose any multiported disk.
b
e
u
l
k
a erab
y
l
f The local disks (single-ported) appear at the beginning and end of
u nNote
s
d
b
a
tra the output and cannot be chosen as quorum devices.
g
n
n
a
no
vincent:/# cldevice list -v
DID Device
Full Device Path
------------------------d1
vincent:/dev/rdsk/c0t0d0
d2
d3
d4
d4
theo:/dev/rdsk/c1t0d0
d5
d5
d6
d6
d7
d7
d8
d8

4-53

d9
d9
d10
d10
d11
d11
d12
d12
d13
d13
d14
d14
d15
d15
d16
d16
d17
d17
d18
d18
d19
d19
d20
d21
d22
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
r
l y sfeQuorum
u
Choosing
and Resetting the installmode
d
b tran
a
(Two-Node Cluster)
g Attribute
nn
o
a
n
ab
Before a new cluster can operate normally, you must reset the
installmode attribute on all nodes. On a two-node cluster where
automatic quorum selection was disabled, the installmode will still be
set on the cluster. You must choose a quorum device as a prerequisite to
resetting installmode.
4-54

Choosing Quorum Using the clsetup Utility

The clsetup utility is a menu-driven interface which, when the
installmode flag is reset, turns into a general menu-driven alternative to
low-level cluster commands. This is described in more detail in Module 5.
The clsetup utility recognizes if the installmode flag is still set, and will
not present any of its normal menus until you reset it. For a two-node
cluster, this involves choosing a single quorum device first.
# /usr/cluster/bin/clsetup
>>> Initial Cluster Setup <<<
s
) ha
This program has detected that the cluster "installmode" attribute is

still enabled. As such, certain initial cluster setup steps will be
performed at this time. This includes adding any necessary quorum
devices, then resetting both the quorum vote counts and the
"installmode" property.
y
m
om
c
csm Guide
e
@ ent
b
u
Please do not proceed if any additional
ud yet to join the
aknodesSthave
y
(
cluster.
in this
d
u
e yes
al [yes]?
s
Is it okay to continue (yes/no)
m
u
ka e to
g
Do you want to add
bananylicquorum
ens devices (yes/no) [yes]? yes
a
b
u
le Quorum Devices types in Sun Cluster. Please
k
b
Following
are
supported
a
a
r
y SunfeCluster
l to
refer
documentation for detailed information on these
u
s
d
b tranquorum device topologies.
supported
a
g onn
a
n
ab
What is the type of device you want to use?
1) Directly attached shared disk
2) Network Attached Storage (NAS) from Network Appliance
3) Quorum Server
q)
Option: 1
>>> Add a SCSI Quorum Disk <<<
A SCSI quorum device is considered to be any Sun Cluster supported

attached storage which connected to two or more nodes of the cluster.
Dual-ported SCSI-2 disks may be used as quorum devices in two-node
clusters. However, clusters with more than two nodes require that

4-55
SCSI-3 PGR disks be used for all disks with more than two
node-to-disk paths.
You can use a disk containing user data or one that is a member of a
device group as a quorum device.
For more information on supported quorum device topologies, see the
Sun Cluster documentation.
Is it okay to continue (yes/no) [yes]? yes
Which global device do you want to use (d<N>)?
d4
Is it okay to proceed with the update (yes/no) [yes]?
yes
s
) ha
y
m
clquorum add d4
om
c
Command completed successfully.

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al device
se (yes/no) [yes]? no
Do you want to add another a
quorum
m
u
k e to
g
ns has been reset, this program will
Once the "installmode"
banproperty
e
c
a
li each time it is run again in the future.
skip "Initial uCluster
Setup"
b
e
l
k
However, y
quorum
a edevices
ab can always be added to the cluster using the
r
l
f
regular
options.
Resetting this property fully activates quorum
s
n
bdu menu
a
a
r
settings
and
is
necessary
for the normal and safe operation of the
t
g
n
n
cluster.
no
aba
Is it okay to reset "installmode" (yes/no) [yes]?
yes
clquorum reset
claccess deny-all
Cluster initialization is complete.
Type ENTER to proceed to the main menu:
4-56


*** Main Menu ***

1)
2)
3)
4)
5)
6)
7)
8)
Quorum
Resource groups
Data Services
Cluster interconnect
Device groups and volumes
Private hostnames
New nodes
Other cluster tasks

q) Quit
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

4-57
Choosing a Quorum Device (Clusters With More Than

Two Nodes)
In a cluster of more than two nodes the installmode flag is always
automatically reset, but the quorum device or devices are never
automatically selected.
You should use clsetup to choose quorum devices, but the initial screens
look a little different because the installmode flag is already reset.
# /usr/cluster/bin/clsetup
*** Main Menu ***
s
) ha
y
m
om
c
sm uide
1) Quorum
c
e
G
2) Resource groups
t
@
n
b
3) Data Services
ku tude
a
y
4) Cluster interconnect
n ( this S
5) Device groups and volumes udi
al use
6) Private hostnames
m
ka e to
7) New nodes
g
8) Other clusteraproperties
b n licens
a
ub menu
leoptions
?) Helpak
with
b
a
l y sfer
q)uQuit
d
ab -tran
g
n Option:
on 1
a
n
b
a
*** Quorum Menu ***

1) Add a quorum device
2) Remove a quorum device
?) Help
Option: 1
From here, the dialog looks similar to the previous example, except that
the installmode is already reset so after adding your quorum devices
you just return to the main menu.
4-58

Performing Post-Installation Verification

When you have completed the Sun Cluster software installation on all
nodes, verify the following information:
General cluster status
Cluster configuration information
Verifying General Cluster Status
s
) ha
The status sub-command of the cluster utilities shows the current status
of various cluster components, such as:
y
m
om
c
Quorum votes (including node and device quorum

ide
csm Gvotes)
u
e
Device groups
@ ent
b
u
Resource groups and relatedyresources
ak Stud
(
in this
Cluster interconnectustatus
d
al use
m
ka e to
g
Note The
command-line
interface (CLI) commands are described
ancluster
ns
b
e
c
a
in detail
starting
li in the next module and continuing on a per-subject basis
b configure
e
u
l
k
as
you
storage and applications into the cluster in Modules 6
b
a
a
r
y
fe 11.
ul nthrough
s
d
b
a
tra
g
n
n
no
aba
Nodes

4-59

The following two commands give identical output, and show the cluster
membership and quorum vote information:
vincent:/# cluster status -t quorum

vincent:/# clquorum status
Cluster Quorum ===
--- Quorum Votes Summary --Needed
-----2
Present
------3
Possible
-------3
s
) ha
y
m
om
c
Node Name
Present
Possible
Status
sm uide
c
e
G
---------------------------@
t
n
b
vincent
1
1
Online
ku tude
a
y
theo
1
1
Online
n ( this S
i
d
lu se
a
m to u
--- Quorum Votes by Device ---ka
ng ense
a
b
a
Device Name
Present
Status
lic Possible
b
e
u
l
----------------------k ------b
a
a
r
y
d4
1
Online
ul ns1fe
d
b
a
tra
g
n
n
no The following two commands are identical, and show status of the private
aba
--- Quorum Votes by Node ---
networks that make up the cluster interconnect (cluster transport):
vincent:/# cluster status -t interconnect

vincent:/# clinterconnect status
Cluster Transport Paths ===
Endpoint1
--------vincent:ce3
vincent:ce2
4-60
Endpoint2
--------theo:ce3
theo:ce2
Status
-----Path online
Path online

Verifying Cluster Configuration Information
Cluster configuration is displayed in general using the list, list -v,

show, and show -v sub-commands of the various cluster utilities.
The following command shows the configuration of everything. If you
added a -t global at the end of the command it would list only the
cluster global properties which appear in the first section of output.
vincent:/# cluster show
s
) ha
Cluster ===
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
anControl ===
ab Access
r
t
Host
g
n non
a
b
a
Cluster Name:
clusterid:
installmode:
heartbeat_timeout:
heartbeat_quantum:
private_netaddr:
private_netmask:
max_nodes:
max_privatenets:
num_zoneclusters:
udp_session_timeout:
global_fencing:
Node List:
Cluster name:
Allowed hosts:
Authentication Protocol:
orangecat
0x49E30B79
disabled
10000
1000
172.16.0.0
255.255.255.192
6
4
2
480
pathcount
vincent, theo
orangecat
None
sys
Cluster Nodes ===

Node Name:
Node ID:
Enabled:
privatehostname:
reboot_on_path_failure:
globalzoneshares:
defaultpsetmin:
quorum_vote:
quorum_defaultvote:
quorum_resv_key:
vincent
1
yes
clusternode1-priv
disabled
1
1
1
1
0x49E30B7900000001

4-61
Transport Adapter List:

Node Name:
Node ID:
Enabled:
privatehostname:
globalzoneshares:
defaultpsetmin:
quorum_vote:
quorum_defaultvote:
quorum_resv_key:
ce2, ce3
theo
2
yes
clusternode2-priv
disabled
1
1
1
1
0x49E30B7900000002
ce2, ce3
y
m
om
c
Transport Cable:
vincent:ce2,switch1@1
Endpoint1:
vincent:ce2
csm Guide
e
t
@
Endpoint2:
switch1@1
n
b
e
u
State:
ak SEnabled
tud
y
(
s
in thivincent:ce3,switch2@1
d
Transport Cable:
u
l
e
a us
Endpoint1:
vincent:ce3
m
a
o
k
t
Endpoint2:
switch2@1
g nse
n
a
State:
Enabled
ab lice
b
u
le
k
b
Transport Cable:
theo:ce2,switch1@2
a
a
y fer
l
u
Endpoint1:
theo:ce2
s
bd tran
a
Endpoint2:
switch1@2
g
n
n
State:
Enabled
no
aba
s
) ha
Transport Cables ===
Transport Cable:
Endpoint1:
Endpoint2:
State:
theo:ce3,switch2@2
theo:ce3
switch2@2
Enabled
Transport Switches ===

Transport Switch:
State:
Type:
Port Names:
Port State(1):
Port State(2):
switch1
Enabled
switch
1 2
Enabled
Enabled
Transport Switch:
switch2
4-62


State:
Type:
Port Names:
Port State(1):
Port State(2):
Enabled
switch
1 2
Enabled
Enabled
Quorum Devices ===

Quorum Device Name:
Enabled:
Votes:
Global Name:
Type:
Access Mode:
Hosts (enabled):
d2
yes
1
/dev/did/rdsk/d2s2
shared_disk
scsi2
vincent, theo
s
) ha
y
m
om
c
Device Groups ===

csm Guide
e
@ ent
.
b
u
.
ak Stud
y
(
.
in thisof resource types,
d
// This continues on to show configuration
u
l
e will cover these in later
adisks.
sWe
resource groups resources a
and
m
u
k e to
chapters
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

4-63
Exercise: Installing the Sun Cluster Server Software

Task 1 Verifying the Environment
Task 2 Updating Local Name Resolution
Task 3 Installing the Sun Cluster Packages
Task 4 Configuring a New Cluster The All-Nodes-at-Once

Method
Task 5 Configuring a New Cluster The One-Node-at-a-Time

Method
s
) ha
y
m
om
c
Task 7 Configuring a Quorum Device (Three-Node

eor Twom Cluster
d
s
i
c
u
Node Cluster With No Automatic Selection)e
G
t
@
n
b
Task 8 Verifying the Cluster Configuration
de Status
ku tuand
a
y
n ( this S
i
d
lu se
a
am to u
k
Task 1 Verifying
e
ng the
sEnvironment
a
n
b
e
a
lic
b
e
u
l
Perform
k the
bfollowing steps on all nodes:
a
a
r
y
fe that the /.profile file on each cluster node contains the
ul 1. nsVerify
d
b
a
tra following environment variables:
g
n
n
no
aba
Task 6 Verifying an Automatically Selected Quorum Device (TwoNode Cluster)
Note The PATH and MANPATH need not be identical to those below; the
important part is that you include the path to the cluster utilities and man
pages, as highlighed. You may have a better, more explicit path already.
Consult your instructor if you are unsure.
PATH=$PATH:/usr/cluster/bin
MANPATH=$MANPATH:/usr/cluster/man:/usr/share/man
export PATH MANPATH
4-64
2.
If you edit the file, verify that the changes are correct by logging out
and in again as user root.
3.
On all cluster nodes, edit the /etc/default/login file and

comment out the CONSOLE=/dev/console line.

Task 2 Updating Local Name Resolution
Perform the following steps to update local name resolution.

1.
Edit the /etc/hosts file on the administrative workstation and all

cluster nodes. Make sure the IP addresses of your administrative
workstation and of all your nodes are in each /etc/hosts file.
Note In the Remote Lab Environment, the machine named vnchost

may be serving as the shared administrative station for all the students.
s
) ha
y
m
Task 3 Installing the Sun Cluster Packages

om
c
m uide
cssave
Perform the following steps on all nodes. Youe
will
Gtime if you
t
@
n
b
perform the installation simultaneouslyuon all nodes.
k tude
a
y
n ( this S
i
d
luhave you
Note These instructions
e do a graphical installation. Feel free to
a
s
m
u
try the non-graphical
o (./installer -nodisplay), or do
ka installation
tsome
g
e
graphical installations
on
nodes and non-graphical installations on
n ens
a
b
other a
nodes tolicompare
the two.
c
b
e
u
l
k rab
a
y
1. feOn your administrative workstation or display, figure out the
l
u
d
b trans number of your display. In the RLDC environment, each student or
a
group might have a different display number. The output might look
ang non
ab
similar to unix:6.0 or :9.0. In either case, the display number is the

number between the colon and the .0:
(# or $) echo $DISPLAY
2.
On the administrative workstation or display, enable remote Xwindows display from the nodes:
(# or $) /usr/openwin/bin/xhost +nodename1
3.
On your cluster node, set and export the DISPLAY, if it is not already
set:
# env|grep DISPLAY
# DISPLAY=display-name-or-IP:display-#
# export DISPLAY

4-65

4.
Run the installer:
# cd sc32_software_location/Solaris_sparc
# ./installer
a.
Accept the license agreement.
b.
On the non-graphical installer only, say no when asked if you

want to install the full set of products.
c.
Choose Sun Cluster 3.2 (not Sun Cluster Agents or Sun Cluster
Geographic Edition). This is #5 only on the non-graphical
installer. You will notice that Java DB is also automatically
selected, and you cannot deselect it as long as Sun Cluster is
selected.
s
d. Feel free to skip multilingual support if English-only is fine )forha
y
you, or feel free to select it (it adds significant time to the
m
installation.) Be persistent if you want to skip it;cthe

installer is
om
e
pushy.
m
d
s
i
ec t Gu
e. Confirm upgrade of the shared components.
@
b den
u
k
f.
Verify that system requirements
tu
ya areSmet.
(
n
s
g. Choose Configure u
Later
choose Configure
di (ifeyou
hiaccidentally
t
l
a
Now, you will
be asked
if you want to enable remote
usYes).
amplease
o
k
t
configuration;
say
g nse
ban licthat
e you want to continue with the installation.
h. aConfirm
b
u
le
k
b
a
a
l y 4 sConfiguring
er
f
u
d
Task
a New Cluster The All-Nodesb tran
a
Method
onang nat-Once
ab
If you prefer to use the one-node-at-a-time method, skip this task and
complete Task 5 instead.
Perform the following steps to configure the entire cluster using the allnodes-at-once method:
4-66
1.
Log in as user root on the node that you want to have assigned the
highest node ID in the cluster. This is the node you will drive from.
2.
Start the scinstall utility (/usr/cluster/bin/scinstall).
3.
As the installation proceeds, make the following choices:

a.
From the Main Menu choose Option 1, Create a new cluster or

add a cluster node.
b.
From the Install Menu choose Option 1, Create a new cluster.


c.
From the Typical or Custom Menu choose Option 1, Typical.
d.
Furnish your assigned cluster name.
e.
Enter the names of the other nodes in your cluster. They will be
rebooted and assigned Node IDs in reverse order to what you
type.
f.
Select the adapters that will be used for the cluster transport. If
you are asked (with a tagged VLAN-capable adapter) if it is a
dedicated cluster transport adapter, answer Yes.
g.
For a two-node cluster, do not disable the automatic quorum

selection (answer No).
h.
Accept the offer to use the lofi method for the global devices
file system for each node as scinstall detects that there is no
dedicated file system.
s
) ha
ab
y
m
omnode reboots,
c
4. You will observe the following error messages aseach
small normal.
ide
just the very first time, into the cluster. They
care
u
e
G
t
@
n
b
/usr/cluster/bin/scdidadm: Could not load DID
instance
list.
ku tude
a
/usr/cluster/bin/scdidadm: Cannot open /etc/cluster/ccr/did_instances.
y
in ( tadditional
is S error message every time a
5. You will observe theufollowing
d
h
e is also normal.
al cluster.
s
node boots into
the
This
m
u
ka e to
g
s on clprivnet0 - will use link layer
n(18)
ip: joining multicasts
banfailed
e
c
a
li
b
broadcasts for umulticast
e
l
k
a erab
y
l
u nsf
d
b
a
tra
g
n
n
a
no Task 5 Configuring a New Cluster The One-Nodeat-a-Time Method
If you chose to install your cluster in Task 4 Configuring a New Cluster

The All-Nodes-at-Once Method do not do this task.
Perform the following steps to configure the first node in your cluster (the
one that will be assigned node ID 1). You must wait for this node to
complete and reboot into the cluster before configuring other nodes.
1.
Start the scinstall utility (/usr/cluster/bin/scinstall).
2.

a.
Choose Option 1 from the Main Menu, Create a new cluster or

add a cluster node.

4-67

b.
Choose Option 2 from the Install Menu, Create just the first
node of a new cluster on this machine.
c.
From the Type of Installation Menu, choose Option 1, Typical.
d.
Furnish your assigned cluster name.
e.
Allow the cluster check to run.
f.
Furnish the name of the other nodes to be added later.
g.
Verify the list of node names.
h.
Select the transport adapters. If you are asked (with a tagged

VLAN-capable adapter) if it is a dedicated cluster transport
adapter, answer Yes.
as
h
i.
Agree to use the lofi method for the global device file system.
)
my
j.
For a two-node cluster, do not disable the automaticm
quorum
selection (answer No).
co e
m
cs Guid
k. Reply yes to the automatic reboot question.
e
@ ent
b
u
l.
Examine the scinstall command
options
k tud for correctness.
a
y
Accept them if they seem (appropriate.
n this S
i
d
You must wait for
luthis node
a
seto complete rebooting to proceed to
m
u
the second a
node.
k e to
g
n ethe
3. You willaobserve
nsfollowing error messages as the node reboots,
b
c
a
just
first
li time, into the cluster. They are all normal.
b the very
e
u
l
k
/usr/cluster/bin/scdidadm:
a erab Could not load DID instance list.
y
l
u nsf
/usr/cluster/bin/scdidadm:
Cannot open /etc/cluster/ccr/did_instances.
d
b
a
a
r
g on-t 4. You will observe the following additional error message every time a
n
a
n
node boots into the cluster. This is also normal.
ab
ip: joining multicasts failed (18) on clprivnet0 - will use link layer
broadcasts for multicast
Perform the following steps on each additional node in the cluster. If you
have more than two nodes, you could perform these steps on all the other
nodes simultaneously, although then it will be hard to predict which node
gets which node ID. If you want this to be predictable, do one node at a
time.
1.
Start the scinstall utility on the new node.
2.

a.
4-68
Choose Option 1 from the Main Menu.


b.
Choose Option 3 from the Install Menu, Add this machine as a

node in an existing cluster.
c.
From the Type of Installation Menu, choose Option 1, Typical.
d.
Provide the name of a sponsoring node.
e.
Provide the name of the cluster that you want to join.

Type cluster show -t global on the first node (the
sponsoring node) if you have forgotten the name of the cluster.
f.
Use auto-discovery for the transport adapters.
g.
Agree to use the lofi method for the global device file system.
h.
Reply yes to the automatic reboot question.
s
) ha
ab
y
m
j.
You will see the same normal boot error messages
omonthe
c
additional nodes as on the first node.

csm Guide
e
@ ent
b
u
k ontthe
Note You see interconnect-related
udexisting nodes beginning
aerrors
y
(
S
when a new node adds itselfin
to the cluster
is until the new node completes
d
h
t
u
the first portion of its reboot
e
al operation.
s
m
u
ka e to
g
an cens
b
a
li an Automatically Selected Quorum
Task u
6b Verifying
e
l
k
a e(Two-Node
ab
Device
Cluster)
r
y
l
f
u
s
bd tran
a
- Perform the following steps to verify automatic quorum selection on a
ang non
i.
Examine and approve the scinstall command-line options.
two-node cluster:
1.
In a two node cluster, assuming you have allowed automatic

quorum selection, wait after the second node boots until you see
console messages (on all nodes) indicating that the quorum device
has been selected.
2.
On either node, type the clquorum status command. Verify that

your quorum device was chosen and that you have the expected
number of quorum votes.

4-69
Task 7 Configuring a Quorum Device (Three-Node

Cluster or Two-Node Cluster With No Automatic
Selection)
Perform the following steps for quorum selection on a three-node cluster:
1.
In a three node-cluster, wait after the third node boots until you see
console messages (on all nodes) indicating that the quorum votes are
being reset.
2.
On Node 1, type the cldevice list -v command, and record the

DID devices you intend to configure as quorum disks. For example,
if you had three nodes in a Pair + 1 configuration, you would want
two quorum devices.
s
) ha
y
m
Quorum disks: _______ (d4, d6, and so on)

om
c
csm Guide
e
t be local
@devices
Note Pay careful attention. The first few DID
might
n
b
e
u
disks, such as the boot disk and a CD-ROM
ud6). Examine the
ak (target
t
y
(
S
standard logical path to make sure
the DIDisdevice you select is a disk in a
inmore
d
h one node.
tthan
storage array and is connected
to
u
l
e
a
am to us
k
3. On Node n
1,g
type the clsetup
command. Navigate to the section for
e
s
a
n
adding
ba newlicquorum
e device and supply the name of the first DID
a
b
udevice b(global
le device) that you selected in the previous step. You
k
a
a
should
see
output similar to the following.
y fer
l
u
s
bd tran clquorum add d12
a
May 3 22:29:13 vincent cl_runtime: NOTICE: CMM:
ang non
ab
Cluster members: vincent theo apricot.

May 3 22:29:13 proto192 cl_runtime: NOTICE: CMM: node
reconfiguration #4 completed.
4.
If you have three nodes in a Pair +N configuration, you should add a

second quorum device.
5.
Reply yes to the reset installmode question (two-node cluster only).

You should see a Cluster initialization is complete message.
6.
4-70
Type q to quit the clsetup utility.

Task 8 Verifying the Cluster Configuration and Status
Perform the following steps on all nodes to verify the completed cluster
installation:
1.
On any node, type the clquorum status command.

You should see the correct node and disk quorum votes.
2.
On any node, type the clinterconnect status command.

You should see active redundant interconnect paths between all pairs
of nodes.
3.
On any node, type the cldevice list -v command.
s
Each shared (dual-ported) DID device should show a logicalypath
) ha
from each cluster node.
m
m
co e
4. On any node, type the cluster show command.
m
cs Guidand quorum
The cluster status, node names, transporte
configuration,
@ ent
b
device information should be complete.
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

4-71
Exercise Summary
Exercise Summary

Experiences
Interpretations
Conclusions
Applications
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
4-72

Module 5

Objectives
s
) ha
ab
y
m
om
Identify the cluster daemons
c
csm Guide
Use cluster commands
e
@ ent
b
View and administer cluster global
properties
u
ak Stud
y
(
View and administer quorum
in this
d
u
View and administer
e and settings
al diskuspaths
m
o
ka e tinterconnect
View and
administer
components
g
n
s
ba
en command and the Sun Cluster Manager
Use
the clsetup
c
a
i
l
b
u
le basic cluster startup and shutdown operations, including
k
b
Perform
a
a
r
l y sfebooting
nodes in non-cluster mode and placing nodes in
u
d
n
b
maintenance state
a
tra
g
n
n
a
Modify the private network settings from non-cluster mode
no
5-1
Relevance
Relevance

What are various ways about getting more information about

various Sun Cluster commands themselves?
Why are you sometimes required to add a new quorum device before
deleting an old one?
What are some of the advantages of clsetup and Sun Cluster

Manager?
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
5-2
What is the value of being able to assign authorization for non-root

users to do some cluster management?

820-7458 (for Sun Cluster 3.2 11/09)
Sun Cluster Quorum Server User Guide, part number 819-5360.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

5-3
Identifying Cluster Daemons

When a cluster node is fully booted into a cluster, several cluster daemons
are added to the traditional Solaris OS.
None of these daemons require any manual maintenance, regardless of
which version of Solaris OS you are running. However, behind the scenes,
the Solaris 10 OS manages daemons a bit differently.
Solaris 9 OS Daemons are launched by traditional Solaris OS boot

scripts (and are thus guaranteed to be running by the time you get
the console login prompt after a boot). A special cluster specific
daemon monitor, rcp.pmfd, is required for restarting some daemons.
y
m
om
c
csm Guide
e
@ ent
b
u
ak running
The following is taken from a clusterynode
tudSolaris 10 OS:
(
S
in this
d
u
al use
# ps -ef|grep clust
m
ka? e0:01
to cluster
root
4
0
0 03:27:08
g
an ce?ns 0:00 /usr/cluster/lib/sc/cl_execd
root 1569 1568
0 03:30:27
b
a
li ? 0:00 /usr/cluster/lib/sc/clexecd
root 1570
1 b
0 03:30:27
e
u
l
k
b
root 1568
?
0:00 /usr/cluster/lib/sc/cl_execd
a1 0era03:30:27
y
l
f
root 1546
1 s 0 03:30:27 ?
0:00 /usr/cluster/lib/sc/failfastd
n
bdu 1570
a
root a1571
0
03:30:27
?
0:00 /usr/cluster/lib/sc/clexecd
r
t
g
n
n
root
1
0 03:29:37 ?
0:00 /usr/cluster/lib/sc/qd_userd
a 1513
no
abroot
1566
1
0 03:30:27 ?
0:00 /usr/cluster/lib/sc/pmmd
s
) ha
Solaris 10 OS Daemons are launched by the Solaris 10 OS Service

Management Facility (SMF).Therefore, at boot time you might see a
console login prompt before many of these daemons are launched.
SMF itself can restart some daemons.
root 1585
1
0 03:30:28 ?
0:00
/usr/cluster/lib/sc/ifconfig_proxy_serverd
root 1586
1
0 03:30:28 ?
0:00
/usr/cluster/lib/sc/rtreg_proxy_serverd
root 3072 1566
0 03:30:41 ?
0:01 /usr/cluster/lib/sc/rgmd -z
global
root 1956
1
0 03:30:34 ?
0:00 /usr/lib/inet/xntpd -c
/etc/inet/ntp.conf.cluster
root 2836
1
0 03:30:39 ?
0:00 /usr/cluster/lib/sc/cl_eventd
root 2824
1
0 03:30:39 ?
0:00 /usr/cluster/lib/sc/scqdmd
root 2904
1
0 03:30:40 ?
0:00 /usr/cluster/lib/sc/rpc.fed
root 2821
1
0 03:30:39 ?
0:00 /usr/cluster/lib/sc/scdpmd
root 2845
1
0 03:30:40 ?
0:00 /usr/cluster/bin/cl_pnmd
root 3057
1
0 03:30:41 ?
0:00 /usr/cluster/lib/sc/rgmd
root 2805
1
0 03:30:39 ?
0:00 /usr/cluster/lib/sc/cl_ccrad
5-4


root 2852
1
0 03:30:40 ?
0:00 /usr/cluster/lib/sc/rpc.pmfd
root 2876
1
0 03:30:40 ?
0:00 /usr/cluster/lib/sc/scprivipd
root 2812
1
0 03:30:39 ?
0:00 /usr/cluster/lib/sc/sc_zonesd
root 2882
1
0 03:30:40 ?
0:00 /usr/cluster/lib/sc/cznetd
root 2982
1
0 03:30:40 ?
0:00
/usr/cluster/lib/sc/pnm_mod_serverd
root 2922
1
0 03:30:40 ? 0:00 /usr/cluster/lib/sc/cl_eventlogd
cluster This is a system process (created by the kernel) to

encapsulate the kernel threads that make up the core kernel range of
operations.
There is no way to kill this process (even with a KILL signal) because
it is always in the kernel.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
clexecd This is used by cluster
threads
ud to execute userland
akkernel
t
y
(
S
commands (such as the irun_reserve
n this and dofsck commands). It is
d
also used to run cluster
commands
remotely (like the cluster
u
l
e
a
s
shutdown command).
am to u
k
Thisa
daemon
ng eregisters
se with failfastd so that the failfast device
n
b
a willlicpanic the kernel if this daemon is killed and not restarted
driver
b
e
u
lseconds.
in 30
k
b
a
a
r
l y sfecl_eventd
u
This daemon registers and forwards cluster events
d
b tran (such as nodes
a
entering and leaving the cluster). There is also a
g onn
protocol
whereby
user applications can register themselves to receive
a
n
ab
failfastd This daemon is the failfast proxy server. Other

daemons that require services of the failfast device driver register
with failfastd. The failfast daemon allows the kernel to panic if
certain essential daemons have failed.
cluster events.
The daemon is automatically respawned if it is killed.
qd_userd This daemon serves as a proxy whenever any quorum

device activity requires execution of some command in userland (for
example, a NAS quorum device).
If you kill this daemon, you must restart it manually.
scqdmd This daemon monitors the health of the quorum device.

The daemon is automatically respawned if it is killed.
rgmd This is the resource group manager, which manages the state
of all cluster-unaware applications.
The failfast driver panics the kernel if this daemon is killed and not
restarted in 30 seconds.

5-5
rpc.fed This is the fork-and-exec daemon, which handles requests

from rgmd to spawn methods for specific data services.
The failfast driver panics the kernel if this daemon is killed and not
restarted in 30 seconds.
rpc.pmfd This is the process monitoring facility. It is used as a

general mechanism to initiate restarts and failure action scripts for
some cluster framework daemons (in Solaris 9 OS), and for most
application daemons and application fault monitors (in Solaris 9 and
10 OS)
The failfast driver panics the kernel if this daemon is stopped and
not restarted in 30 seconds.
s
) ha
cl_pnmd This is the public network management daemon, which

manages network status information received from the local IPMP
daemon running on each node and facilitates application failovers
caused by complete public network failures on nodes.
y
m
om
c
sm uide
c
e
G
It is automatically restarted if it is stopped.
t
@
n
b
u events
de into a binary log
kcluster
cl_eventlogd This daemon logs
u
a
t
y
( course,
Sthere is no published
file. At the time of writing ifor
n this
s
i
d
h
t
interface to this log. lu
e
a
s
uif it is stopped.
am
It is automatically
restarted
o
k
t
ngThisedaemon
se provides access from userland
a
n
cl_ccrad
b
c
a
liapplications
b
management
to the CCR.
e
u
l
k
b
a
ya Itfeis rautomatically
restarted if it is stopped.
l
u
s
d
ab -tran scdpmd This daemon monitors the status of disk paths, so that
g
n non
they can be reported in the output of the cldev status command.
a
b
a
sc_zonesd This daemon monitors the state of Solaris 10 non-global

zones so that applications designed to failover between zones can
react appropriately to zone booting and failure.
scprivipd This daemon provisions IP addresses on the

clprivnet0 interface, on behalf of zones.
pnm_mod_serverd, ifconfig_proxy_serverd These daemons

run in the global zone to provide required network services to zones
in a zone cluster.
They are automatically restarted if stopped.
5-6

rtreg_proxy_serverd This daemon runs in the global zone to

provide resource type registration services to zones of a zone cluster.
It is automatically restarted if stopped.
pmmd This daemon provides cross-cluster starting, restarting, and

monitoring for some essential cluster processes such as rgmd that
need to be aware of process status on different nodes.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

5-7
Using Cluster Commands

The Sun Cluster 3.2 command-line commands have an object-oriented

nature. That is:
The name of the command is related to the cluster object you are
trying to display or administer.
Every command requires a sub-command that indicates what it is you

actually want to do.
Every command has a common style of syntax whereby:
When using sub-commands to operate on (or delete) a specific

object, you give the name of the object as the last item on the
command line, or you give a + to indicate a wildcard. The
wildcard might still be limited by other command-line
arguments (specific group or subtypes, if appropriate).
s
) ha
y
m
om
c
csm Guiordestatus,
e
When using subcommands to display configuration
@ ent but you can
the default is to show all objects u
(ofbthat category),
d a specific object.
akto show
tuonly
give an optional last argument
y
(
S
in this
d
u
al use
m
Commands Relating
ka toeBasic
to Cluster Administration
g
an cens
b
a
li
The following
commands
are presented in more detail in this module
b
e
u
l
k
b
because
they
relate
to
configuration,
status, or administration that is
a era
y
l
f
useful
point in the course. Every command has a full name. Some
ns at thishave
bdu trcommands
a
an abbreviated name (long and short names are hard
a
g
n
n
no linked to the same executable):
aba
5-8
clnode Configuration, status, and settings for nodes
clquorum (clq) Configuration, status, settings, adding, and

deleting quorum devices (includes node quorum information)
clinterconnect (clintr) Configuration, status, settings, adding,

and deleting private networks
cldevice (cldev) Configuration, status, and settings for

individual devices (disk, CDROM, and tape)
cluster:
Administering cluster global settings
Showing configuration and status of everything; can be limited

by other arguments to certain types or groups of information

Additional Commands
The following commands relate to administration of device groups and
cluster application resources, which are the topics of Modules 6 through
10. No additional information is presented about them in this module:
cldevicegroup (cldg) Configuration, status, settings, adding and

deleting device groups (including VxVM and Solaris Volume
Manager device groups)
clresourcegroup (clrg) Configuration, status, settings, adding

and deleting application resource groups
clresource (clrs) Configuration, status, settings, adding, and

deleting individual resources in application resource groups
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
Cluster Command Self-Documentation
u
al use
m
ka commands
to have excellent man pages, they are also
g
e
While all the
cluster
s
ban liceinnthat:
self-documenting
a
b
u
lerun a command without any sub-command, the usage
k
b
If a
you
a
r
l y sfemessage
always lists the possible sub-commands:
u
d
n
b
a
tra
#g clquorum
n
n
no (C961689) Not enough arguments.
aba clquorum:
clreslogicalhostname (clrslh) and clressharedaddress

(clrssa) Configuration, status, settings, adding, and deleting IP
resources in application resource groups (these commands simplify
tasks that can also be accomplished with clresource)
clquorum:
Usage:
(C101856) Usage error.
clquorum <subcommand> [<options>] [+ | <devicename> ...]

clquorum [<subcommand>] -? | --help
clquorum -V | --version
Manage cluster quorum

SUBCOMMANDS:
add
disable
enable
export
list
remove
Add a quorum device to the cluster configuration

Put quorum devices into maintenance state
Take quorum devices out of maintenance state
Export quorum configuration
List quorum devices
Remove a quorum device from the cluster configuration

5-9

reset
show
status
Reset the quorum configuration

Show quorum devices and their properties
Display the status of the cluster quorum
If you run a sub-command that requires arguments, the usage message
gives you more information about the particular sub-command. It
does not give you all the information about the names of properties
that you might need to set (for that, you have to go to the man pages):
# clquorum add
clquorum: (C456543) You must specify the name of the quorum
device to add.
clquorum: (C101856) Usage error.
Usage:
clquorum add [<options>] <devicename>

clquorum add -a [-v]
clquorum add -i <configfile> [<options>] + |
<devicename> ...
s
) ha
y
m
om
c
csm Guide
e
Add a quorum device to the cluster configuration
@ ent
b
u
ak Stud
y
(
OPTIONS:
in this
d
u
l text.
e
ahelp
s
-?
Display this
m
u
ka e to
g
-a
Automatically
an cens add a Shared Disk quorum device for 2b
a
li
node cluster.
b
e
u
l
k
a erab
y
l
f | <clconfiguration>}
{u -i
s
d
n
b
Specify XML configuration as input.
a
tra
g
n
n
no
aba
-p <name>=<value>
Specify the properties.
-t <type>
Specify the device type.
-v
Verbose output.
Note If you perform the man cluster command on the Solaris 10 OS,
you might get the wrong man page because there is a man page for a
completely unrelated SQL command called cluster. To fix that, you can
put /usr/cluster/man as the first directory in your MANPATH variable, or
you can type man -s1cl cluster.
5-10

Viewing and Administering Cluster Global Properties

The cluster show -t global command, as demonstrated, shows only
the cluster global properties:
# cluster show -t global
Cluster ===
Cluster Name:
clusterid:
installmode:
heartbeat_timeout:
heartbeat_quantum:
private_netaddr:
private_netmask:
max_nodes:
max_privatenets:
num_zoneclusters:
global_fencing:
Node List:
orangecat
0x49D36C52
disabled
10000
1000
172.16.0.0
255.255.240.0
64
10
12
480
pathcount
vincent, theo
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b the
u
Renaming
le Cluster
k
b
a
a
l y sfer
u
d
ab -tranYou can rename the cluster using the cluster rename command. The
g
cluster name is not particularly important and is not required as an
n non
a
b
argument in any other commands:
a
# cluster rename -c blackcat
Cluster ===
Cluster Name:
clusterid:
installmode:
heartbeat_timeout:
heartbeat_quantum:
private_netaddr:
.
.
# cluster rename -c orangecat
blackcat
0x49D36C52
disabled
10000
1000
172.16.0.0

5-11
Setting Other Cluster Properties

The heartbeat_quantum controls the timing of cluster heartbeats on the
private network (in milliseconds). The heartbeat_timeout describes the
number of milliseconds of missing heartbeat required by a node to declare
a single interconnect dead or to declare the other node(s) dead and start
reconfiguration.
You usually do not change these values, although it is possible to make
them bigger if your nodes are very far apart, or smaller, if for some reason
you are unsatisfied with the 10-second timeout.
s
) ha
The cluster will enforce that heartbeat_timeout is at least five times as

big as heartbeat_quantum, as illustrated in the following example:
y
m
om
# cluster set -p heartbeat_timeout=4000
c
cluster: heartbeat timeout 4000 out of range 5000 - 60000.

sm uide
c
e
# cluster set -p heartbeat_quantum=800
G
t
@
n
b
# cluster set -p heartbeat_timeout=4000
ku tude
a
y
n ( this S
i
d
lu se
Cluster ===
a
am to u
k
Cluster Name:
orangecat
ng ense
a
b
clusterid:
0x49D36C52
c
a
li
b
e
u
l
installmode: k
disabled
b
a
a
r
y
heartbeat_timeout:
4000
ul nsfe
d
heartbeat_quantum:
800
b
a
a
r
t
g
private_netaddr:
172.16.0.0
n
n
o
a
n
255.255.240.0
ab private_netmask:
max_nodes:
max_privatenets:
num_zoneclusters:
global_fencing:
Node List:
64
10
12
480
pathcount
vincent, theo
Note Modifying the private_netaddr and private_netmask

properties is a special case, in that it is done only when the entire cluster
is down, and all nodes are booted in non-cluster mode. This will be
covered later in the module.
5-12

Viewing and Administering Nodes

The clnode command can be used to show status and configuration of
nodes. While it shows a variety of data for each node in the cluster, there
is only a limited amount that you can actually change with the clnode
command.
Viewing Node Status and Configuration

The status and show sub-commands show, by default, all nodes. You
could also show a single node by giving its name as the last commandline argument:
s
) ha
y
m
om
c
Cluster Nodes ===

csm Guide
e
@ ent
b
u
--- Node Status --ak Stud
y
(
in this Status
d
Node Name
u
al use
-------------m
a
o
k
t
vincent
Online
g
e
n
s
theo
Online
ba licen
a
b
u
le
k
b
# clnode show
a
a
l y sfer
u
d
an===
Cluster
ab Nodes
r
t
g
n non
a
b
vincent
a Node Name:
# clnode status
Node ID:
Enabled:
privatehostname:
globalzoneshares:
defaultpsetmin:
quorum_vote:
quorum_defaultvote:
quorum_resv_key:
Node Name:
Node ID:
Enabled:
privatehostname:
1
yes
clusternode1-priv
disabled
1
1
1
1
0x49D36C5200000001
ce2, ce3
theo
2
yes
clusternode2-priv

5-13

globalzoneshares:
defaultpsetmin:
quorum_vote:
quorum_defaultvote:
quorum_resv_key:
disabled
1
1
1
1
0x49D36C5200000002
ce2, ce3
Modifying Node Information

Most of the information shown by clnode show cannot be modified by
the clnode command. Some can be modified by other commands
(clinterconnect for adding and deleting transport adapters, for
example).
s
) ha
y
m
om
c
The reboot_on_path_failure command is described

sm later
idinethe
c
u
e
G
module.
t
@
n
b
ku tude
a
y
You could set the privatehostname
( toiswhatever
S you want. This name
n address
iIP
automatically resolves to u
the
associated
with the nodes
d
h
t
l
e
a
clprivnet0 adapter.
is the
single private IP address whose traffic is
usall
amThistacross
o
k
automatically distributed
physical private networks.
g
e
n
s
ba licen
# clnode set -p privatehostname=vinny-priv
vincent
a
b
e
# clnode showkvincent
u
l
a erab
y
l
f
uNodesn===
s
d
Cluster
b
a
tra
g
n
n
o
a Node nName:
vincent
ab
Node ID:
Enabled:
privatehostname:
globalzoneshares:
defaultpsetmin:
quorum_vote:
quorum_defaultvote:
quorum_resv_key:
1
yes
vinny-priv
disabled
1
1
1
1
0x49D36C5200000001
ce2, ce3
# getent hosts vinny-priv

172.16.4.1
vinny-priv
# clnode set -p privatehostname=clusternode1-priv vincent
5-14

Note If it seems that the OS can resolve private hostnames that no

longer exist (because you changed them), it is because of the OS namecaching daemon (nscd). You can use nscd -i hosts to clear this cache.
Viewing Software Release Information on a Node

You can use clnode show-rev -v to see installed cluster package release
information on a node, or on all nodes. This is useful information to have
when talking to technical support personnel.
s
) ha
# clnode show-rev -v
Solaris Cluster 3.2u3 for Solaris 10 sparc
SUNWscu:
3.2.0,REV=2006.12.05.22.58, 125511-02 126106-37 800149-01
SUNWsccomu:
3.2.0,REV=2006.12.05.22.58, 125511-02 126106-37 800149-01
SUNWsczr:
3.2.0,REV=2006.12.05.22.58, 125511-02 126106-37 800149-01
SUNWsccomzu:
3.2.0,REV=2006.12.05.22.58, 125511-02 126106-37 800149-01
SUNWsczu:
3.2.0,REV=2006.12.05.22.58, 125511-02 126106-37 800149-01
SUNWscsckr:
3.2.0,REV=2006.12.05.22.58, 125992-05 800149-01
SUNWscscku:
3.2.0,REV=2006.12.05.22.58, 125992-05 800149-01
SUNWscr:
3.2.0,REV=2006.12.05.22.58, 125511-02 126106-37 800149-01
SUNWscrtlh:
3.2.0,REV=2006.12.05.22.58, 126106-37 800149-01
SUNWscnmr:
3.2.0,REV=2009.09.25.13.28
SUNWscnmu:
3.2.0,REV=2009.09.25.13.28
SUNWscdev:
3.2.0,REV=2006.12.05.22.58, 125511-02 126106-37 800149-01
SUNWscgds:
3.2.0,REV=2006.12.05.22.58, 126106-37 800149-01
SUNWscsmf:
3.2.0,REV=2006.12.05.22.58, 126106-37 800149-01
SUNWscman:
3.2.0,REV=2006.09.26.13.45, 128556-04 800141-01
SUNWscsal:
3.2.0,REV=2006.12.05.22.58, 126106-37 800149-01
SUNWscsam:
3.2.0,REV=2006.12.05.22.58, 126106-37 800149-01
SUNWscvm:
3.2.0,REV=2006.12.05.22.58, 800149-01
SUNWmdmr:
3.2.0,REV=2006.12.05.22.58, 800149-01
SUNWmdmu:
3.2.0,REV=2006.12.05.22.58, 125514-05 800149-01
SUNWscmasa:
3.2.0,REV=2006.12.05.22.58, 126106-37 800149-01
SUNWscmasar:
3.2.0,REV=2006.12.05.22.58, 126106-37 800149-01
SUNWscmasasen: 3.2.0,REV=2006.12.05.22.58, 126106-37 800149-01
SUNWscmasau:
3.2.0,REV=2006.12.05.22.58, 125511-02 126106-37 800149-01
SUNWscmautil: 3.2.0,REV=2006.12.05.22.58, 125508-08 126106-37 800149-01
SUNWscmautilr: 3.2.0,REV=2006.12.05.22.58, 800149-01
SUNWjfreechart: 3.2.0,REV=2006.12.05.22.58, 139921-02 800149-01
SUNWscspmr:
3.2.0,REV=2006.12.05.22.58, 800149-01
SUNWscspmu:
3.2.0,REV=2006.12.05.22.58, 125508-08 126106-37 800149-01
SUNWscderby:
3.2.0,REV=2006.12.05.22.58, 126106-37 800149-01
SUNWsctelemetry: 3.2.0,REV=2006.12.05.22.58,125511-02 126106-37800149-01
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

5-15
You can also directly examine the file /etc/cluster/release to get

quick information about the release of the cluster software installed on a
particular node:
# cat /etc/cluster/release
Sun Cluster 3.2u3 for Solaris 10 sparc
Copyright 2009 Sun Microsystems, Inc. All Rights Reserved..
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
5-16

Viewing and Administering Quorum

The clquorum (clq) command is used to view configuration and status of
quorum devices and node votes and to add and delete quorum devices.
Viewing Quorum Status and Configuration

The status, list, and show sub-options show the status and
configuration of quorum devices and node-related quorum information.
You could reduce the amount of information by adding a type-restriction
option (-t shared_disk, for example), or adding as the very last
argument the name of a particular quorum device or node.
s
) ha
y
m
om
c
Cluster Quorum ===

csm Guide
e
@ ent
b
u
--- Quorum Votes Summary --ak Stud
y
(
in this
d
Needed
Present
Possible
u
al use
------------ m-------2
3 g ka 3 to
n ense
a
b
a
lic
b
e
u
l
k byraNode
b ----- Quorum Votes
a
y
l
e
f
du ransPresent
bName
Node
Possible
Status
a
t
g
n
--------------------------n
o
a
n
b
1
1
Online
a vincent
# clq status
theo
Online
--- Quorum Votes by Device --Device Name

----------d2
Present
------1
Possible
-------1
Status
-----Online
# clq show d2
Quorum Devices ===
Quorum Device Name:
d2

5-17

Enabled:
Votes:
Global Name:
Type:
Access Mode:
Hosts (enabled):
yes
1
/dev/did/rdsk/d2s2
shared_disk
scsi2
vincent, theo
Adding and Removing (and Replacing) Quorum

Devices
s
) ha
There is no specific command to replace or repair a quorum device. You

just add a new one and remove the old one.
y
m
om requires
c
A two-node cluster, which absolutely requires a quorum device,
e
m
d
s
i
you to perform repairs in that order (add and then
cremove).
uIf you have
ethe
G
t
more than two nodes you could actually perform
operations
in any
@
n
b
e
u
order:
ak Stud
y
(
in this
# clq add d5
d
u
# clq remove d2
al use
m
# clq status -t shared_disk ka
to
g
e
ban licens
Cluster Quorum ===
a
b
u
le
k
b
a
a
r
--- Quorum Votes
--l y by
eDevice
f
u
s
d
abName -tran Present
g
Device
Possible
Status
n
n
o
a
n
------------------ab----------d5
Online
Installing a Quorum Server (Outside the Cluster)

When you install the packages SUNWscqsr and SUNWscqsu on a machine
outside the cluster, a quorum server instance is configured automatically
on the default quorum server port (9000). This single instance can serve as
the quorum device for as many clusters as you like.
See appendix for more information about starting, stopping, examining
data, and clearing out data in quorum servers.
5-18

Adding a Quorum Server Device to a Cluster

On the cluster side, you need to specifically add a quorum server device
to serve the cluster. This is likely to be your only quorum device (after you
remove other quorum devices, if necessary) because it always has the
number of votes equal to one fewer than the number of node votes.
On the cluster side, you assign a random ascii device name to the quorum
server device (in this example, qservydude).
# clq add -t quorum_server -p qshost=clustergw \
-p port=9000 qservydude
# clq remove d5
s
) ha
y
m
# clq status
om
c
csm Guide
e
Cluster Quorum ===
@ ent
b
u
ak Stud
--- Quorum Votes Summary --y
(
in this
d
u
Needed
Present
Possible
al use
m
-----------ka -------to
e
2
3 ng
3
ba licens
a
b
u
le
k
b
a
a
--- Quorum
l yVotessfebyr Node --u
d
abName-tran Present
Node
Possible
Status
g
n
n
o
a
n
--------------------------ab
vincent
theo
1
1
1
1
Online
Online

----------qservydude
Present
------1
Possible
-------1
Status
-----Online

5-19

# clq show qservydude
Quorum Devices ===

Quorum Device Name:
Enabled:
Votes:
Global Name:
Type:
Hosts (enabled):
Quorum Server Host:
Port:
qservydude
yes
1
qservydude
quorum_server
vincent, theo
192.168.101.1
9000
s
) ha
y
m
om
c
Before you can use a Netapp NAS device as a quorum

you must
deregister
sm
i
c
u
e
the NAS device.
G
t
@
n
b
delooks similar to the
kucommand
u
a
t
y
Registering a NAS device with the clnas
n ( this S
i
following:
d
lu se
a
u mynetapp
# clnas add -t netapp
am -utoroot
k
Please enternpassword:
g nse
a
b
ce
a
i
l
b
There
command for Sun NAS devices as well, although it
uis a registration
le
k
b
a
a
is
strictly
a
prerequisite
only for specifying data fencing for the Sun NAS.
y fer
l
u
s
n is no username and password for the Sun NAS device:
bd trThere
a
a
g onn
a
n
# clnas add -t sun mysunnas
ab
Registering NAS Devices
Adding NetApp NAS iSCSI LUN as a Quorum Device

After the NetApp NAS device is registered, you can use the clsetup
utility to establish one of the iSCSI LUNs on the device as a quorum.
Recall that the iSCSI protocol is used on the NetApp filer NAS device only
for the quorum mechanism. From the clsetup dialog the interaction
looks similar to the following:
What is the type of device you want to use?
1) Directly attached shared disk
2) Network Attached Storage (NAS) from Network Appliance
3) Quorum Server
5-20


q) Return to the quorum menu
Option: 2
>>> Add a Netapp NAS Quorum Device <<<
A Netapp NAS device can be configured as a quorum device for Sun
Cluster. The NAS configuration data includes a device name, which is
given by the user and must be unique across all quorum devices, the
filer name, and a LUN id, which defaults to 0 if not specified.
Please refer to the clquorum(1M) man page and other Sun Cluster
documentation for details.
s
) ha
The NAS quorum device must be setup before configuring it with Sun
Cluster. For more information on setting up Netapp NAS filer,
creating the device, and installing the license and the Netapp
binaries, see the Sun Cluster documentation.
y
m
om
c
csm Guide
e
t
Is it okay to continue (yes/no) [yes]? yes b@
n
e
u
ak Stud
y
(
What name do you want to use for this
in quorum
is device? netappq
d
h
t
u
l
e mynetapp
a[netapps]?
s
What is the name of the filer
m
u
ka e to
g
ns [0]? 0
What is the LUN id
filer
baonn the
e
c
a
li
b
e
u
l
k
Is it okay
a to eproceed
ab with the update (yes/no) [yes]? yes
r
y
l
f
s
n
bdu add
a
a
r
clquorum
-t
netapp_nas -p filer=mynetapp -p lun_id=0 netappq
t
g
n
n
no
aba
Adding Sun NAS iSCSI LUN as a Quorum Device
Sun NAS devices also support using an iSCSI LUN as a quorum device.
Unlike NetApp, Sun NAS uses the standard Solaris 10 iscsiadm client to
make Sun NAS iSCSI LUNs appear as if they were regular shared disk
devices.
In order for the node to get visibility to a Sun NAS iSCSI LUN you would
need to administer it on each node:
# iscsiadm modify discovery -s enable
# iscsiadm list discovery
Discovery:
Static: enabled
Send Targets: disabled

5-21

iSNS: disabled
# iscsiadm add static-config iqn.LUNName,IPAddr_of_NASDevice

# devfsadm -i iscsi
At that point, from a single cluster node, you could repopulate the DID
device space and create a normal-looking shared disk device quorum:
# cldev populate
# cldev list -v
# clq add d#
s
) ha
y
m
m
Registering NAS Mounted Directories (for m
Data
co e
d
Fencing: both NetApp and Sun NAS) ecs Gui
@ ent
b
u
d
atokregister
tuon
y
Use the clnas add-dir subcommand
the NAS device the
(
S
n
s
i
i
specific directories that are being
used to
thserve cluster data. The Sun
ludis then
e
a
Cluster client implementation
able
to perform data fencing on
s
m
u
a
o
these specific directories.
k et
g
n
a cens
b
a
li NAS directories for failure fencing looks similar to
Registering
b thelespecific
u
k
the
a following:
ab
r
y
l
e
f
ns add-dir -d /vol/vol_01_03 mynetapp
bdu tr#aclnas
a
- # clnas add-dir -d /vol/vol_01_04 mynetapp
ang non
ab
You can verify the configuration of the NAS device in the CCR using the
clnas show command as in the following example:
# clnas show
Filers of type "netapp":
Filer name:
type:
password:
userid:
directories:
directories:
5-22
mynetapp
netapp
*******
root
/vol/vol_01_03
/vol/vol_01_04

Viewing and Administering Disk Paths and Settings

The cldevice (cldev) command can:
Display disk path information (which nodes are connected to which

disks) and the mapping between DID device names and c#t#d#
Display disk path status information. Disk paths are monitored by a

daemon scdpmd
Allow you to change disk monitoring settings:
By default, all paths are monitored.
We will see at least one reason you may like to turn off
monitoring for non-shared disks.
s
) ha
y
m
om
c
csm Guide
e
Displaying Disk Paths
@ ent
b
u
ak Stud
y
(
cldev list -v is the best summary
in thiofs node-to-disk paths, and the
d
u
corresponding DID device
e
al numbers:
s
m
u
k-va e to
# cldev list
g
DID Device
an cens Full Device Path
b
a
------------------------li
b
e
u
l
k
a d1 erab
y
l
d2
f
u ns
d
b
a
tra d3
g
n
d4
n
o
a
n
b
d4
a
Allow you to change fencing properties
d5
d5
d6
.
. //Omitted for brevity
.
d17
d18
d18
d19
d19
d20
d21
d22

5-23
Displaying Disk Path Status
By default, the daemon scdpmd probes disk paths periodically, once every
few minutes.
Disk path status changes are logged into /var/adm/messages with the
syslogd LOG_INFO facility level. All failures are logged by using the
LOG_ERR facility level.
The cldevice status command shows the status of disk paths as last
recorded by the daemon. That is, you could yank out a disk or sever a
disk path and the status might still be reported as Ok for a couple of
minutes, until the next time the daemon probes the paths:
s
) ha
y
m
om
c
Device Instance
Node
sm Status
ide
c
u
e
-----------------G-----t
@
n
b
/dev/did/rdsk/d1
vincent
u ude Ok
k
a
t
y
(
S
n
/dev/did/rdsk/d10
Ok
di theo
his
t
u
l
e
Ok
a usvincent
m
a
o
k et
g
/dev/did/rdsk/d11
theo
Ok
n
ba licens
vincent
Ok
a
b
e
u
l
k rab
a
y
/dev/did/rdsk/d12
theo
Ok
l
e
f
u
s
d
vincent
Ok
ab -tran
g
n non /dev/did/rdsk/d13
a
b
theo
Ok
a
# cldev status
5-24
vincent
Ok
/dev/did/rdsk/d14
theo
vincent
Ok
Ok
/dev/did/rdsk/d15
theo
vincent
Ok
Ok
/dev/did/rdsk/d16
theo
vincent
Ok
Ok
/dev/did/rdsk/d17
theo
vincent
Ok
Ok
/dev/did/rdsk/d18
theo
Ok


vincent
Ok
/dev/did/rdsk/d19
theo
vincent
Ok
Ok
/dev/did/rdsk/d2
vincent
Ok
/dev/did/rdsk/d20
theo
Ok
/dev/did/rdsk/d21
theo
Ok
/dev/did/rdsk/d4
theo
vincent
Ok
Ok
s
) ha
y
m
om
c
/dev/did/rdsk/d6
theo
csm GuiOkde
e
@ ent Ok
vincent
b
u
ak Stud
y
(
/dev/did/rdsk/d7
Ok
in theo
is
d
h
t
vincent
Ok
u
l
e
a
s
am to u theo
k
/dev/did/rdsk/d8
Ok
g nse
n
a
vincent
Ok
ab lice
b
u
le
k
b
/dev/did/rdsk/d9
theo
Ok
a
a
y fer
l
u
vincent
Ok
s
bd tran
a
g on- You can limit the output to a specific DID device by giving its name as an
n
a
n
ab
/dev/did/rdsk/d5
theo
vincent
Ok
Ok
argument. You can also display only those paths that have a specific
status, as in the following example:
# cldev status -s Fail
Device Instance
---------------
Node
----
Status
------
Changing Disk Path Monitoring Settings

By default, all paths to all disks are monitored.

5-25
You can unmonitor specific paths, or re-monitor them if you have

previously unmonitored them. By default you affect the path from all
connected nodes to the DID device that you mention. You can limit your
action to a path from a specific node by using the -n option. For example,
the following commands unmonitor all paths to disk device d8, and
specifically the path from node theo to disk device d9:
# cldev unmonitor d8
# cldev unmonitor -n theo d9
# cldev status -s Unmonitored
Device Instance
--------------/dev/did/rdsk/d8
Node
---theo
vincent
Status
-----Unmonitored
Unmonitored
s
) ha
y
m
om
/dev/did/rdsk/d9
theo
Unmonitored
c
sm uide
c
e
G+ as the
As is the convention for all cluster commands,@
you can use
t
n
b
wildcard. So cldev monitor + would turn
de back on for all
ku monitoring
u
a
t
y
devices, regardless of their previous(state.
n this S
i
d
lu se
a
kam e to u Devices and Enabling
Unmonitoring Allg Non-Shared
reboot_on_path_failure
ban licens
a
b
u
le
k
b
a
a
l yIf yousfset
erthe node property reboot_on_path_failure to enabled (the
u
d
default
an is disabled), then a node automatically reboots if all the
ab -trmonitored
g
disk paths from that node are broken and there is another node
n
n
o
a
n
that still has working paths to at least one of the shared devices.
ab
This is a way to ensure HA even when there are multiple simultaneous

disk path failures. Typically, with a single-path failure (to an entire array,
for example), everything would still be fine because your data would
always be mirrored across multiple controllers, or multi-pathed, or both.
But if a particular node loses access to all disk paths, you might decide (by
setting the property) that it is best to just have the node reboot, so that at
least any clustered applications running there can fail over to other nodes.
It does not make sense to enable the reboot_on_path_failure if you are
still monitoring non-shared (local) disks. The whole point is to reboot if
you lose contact with all multi-ported storage, but the actual feature will
not reboot a node unless it detects failure on all monitored storage. So, if
you want to enable this feature, you must unmonitor any local disks.
# cldev unmonitor d1 d2 d4
5-26

# clnode set -p reboot_on_path_failure=enabled vincent

# cldev unmonitor d20 d21
# clnode set -p reboot_on_path_failure=enabled theo
Viewing Settings Related to SCSI-2 and SCSI-3 Disk

Reservations
A global setting controls what form of SCSI reservations, if any, are used
with disks. The default value, as demonstrated in the command example,
is pathcount. With this value, disks with exactly two paths are fenced
with SCSI-2 reservations (add SCSI-2 PGRE when used as a quorum
device). Disks with more than two paths are fenced with SCSI-3
reservations, which already implement the persistence needed for quorum
devices. Nothing more is needed.
s
) ha
y
m
om
c
# cluster show|grep global_fencing

csm Guide
e
global_fencing:
pathcount
@ ent
b
u
ak property.
tud The default value for
Each individual disk has its own(y
fencing
S
n tthat
imeans
every disk is global, which
d
hisfencing for that disk follows the
u
l
e
global property.
a us
m
a
k e to
# cldev show d7
g
ban licens
a
b ===
DID Device Instances
u
le
k
b
a
a
l yName:sfer
u
DID Device
/dev/did/rdsk/d7
d
n
b
a
a
r
Device
-t Path:
g Full
n
n
o
Full
Device
Path:
a
n
ab
Replication:
default_fencing:
none
global
Modifying Properties to use SCSI-3 Reservations for

Disks With Two Paths
The alternate value for the global_fencing property is prefer3.
The prefer3 global value implies that the cluster should try to use SCSI-3
fencing for every device that is still following the global policy.
When you change the global policy from pathcount to prefer3:

5-27
The cluster probes the current two-path devices by sending a sample

SCSI-3 ioctl command. The cluster assumes that any device that
responds appropriately to such a command can support SCSI-3.
The cluster immediately begins to use SCSI-3 fencing for any nonquorum disk that seems to support SCSI-3.
If there is a quorum disk, currently configured as a SCSI-2 quorum, at

the time the global policy is changed to prefer3, that quorum disk
will not have its behavior changed to SCSI-3. You would have to
delete the quorum disk and add it back (or add a different quorum
device) to get SCSI-3 behavior on the quorum disk.
s
) ha
The following command changes the global policy. You can see that SCSI2 behavior is left intact for an existing quorum disk. There is no output or
change in per-disk properties for other, non-quorum disks.
y
m
om
c
# cluster set -p global_fencing=prefer3
e
m uidremains
Warning: Device instance d4 is a quorum device - fencingcs
protocol
e
G
PATHCOUNT for the device.
t
@
n
b
Updating shared devices on node 1
ku tude
a
y
( is S
n
i
d
# cldev show d4
lu se th
a
am to u
k
DID Device Instances ===
ng ense
a
b
lic
DID Device Name: b a
/dev/did/rdsk/d4
e
u
l
k
b
Full Device Path:
a era
y
l
f
Full Device
Path:
bdu trans
Replication:
none
a
g
n
n
default_fencing:
pathcount
o
a
n
b
a
# clq show d4
Quorum Devices ===
Quorum Device Name:
Enabled:
Votes:
Global Name:
Type:
Access Mode:
Hosts (enabled):
5-28
d4
yes
1
/dev/did/rdsk/d4s2
shared_disk
scsi2
vincent, theo


The per-disk policy for the existing quorum device has been set to
pathcount so that it does not follow the global default of prefer3. For
other disks, the individual default_fencing property remains with the
value global and the cluster immediately uses SCSI-3 the next time they
need to be fenced.
Getting Quorum Device to Use SCSI-3 Policy

To change the policy for an existing quorum device, you complete the
following steps:
1.
Remove it as a quorum device (but you might need to add another

quorum device first, if this is a two-node cluster).
ab
as
h
)
2. Change the per-disk default_fencing policy back to the value
y
m
global.
m
o
c
want to
3. Add it back as a quorum device (at which point
you might
e
m
d
s
i
c asGa uplace-holder).
remove any other quorum device that is e
acting
t
@
n
b
e
u
akthe procedure:
The folowing example demonstrates
tud
y
(
S
in this
d
# clq add d5
u
l
e
s
# clq remove d4 ma
u
kadefault_fencing=global
to
# cldev setg-p
d4
e
n
s
a
n
Updating
shared
devices
on
node
1
ce devices on node 2
ab shared
i
l
Updating
b
uclq add
led4
k
b
#
a
a
l y #sclq
er remove d5
f
u
d
ab -tran# clq show d4
g
an non
Quorum Devices ===
Quorum Device Name:

Enabled:
Votes:
Global Name:
Type:
Access Mode:
Hosts (enabled):
d4
yes
1
/dev/did/rdsk/d4s2
shared_disk
scsi3
vincent, theo

5-29
Eliminating SCSI Fencing for Particular Disk Devices
You can turn off SCSI fencing for particular disks devices. This feature,
introduced in Sun Cluster 3.2 1/09 (Update 2), allows the use of Serial
Advanced Technology Attachment (SATA) devices as shared storage
devices. These devices do not support SCSI fencing of any sort.
It is not recommended in any way that you eliminate fencing for devices
that support fencing.
s
) ha
The per-disk default_fencing property has values that specify that you
do not want fencing for that disk:
y
m
m existing
nofencing (turns off fencing after scrubbing the diskcofoany
e
m
reservation keys)
d
s
i
u
ec anyt G
nofencing-noscrub (turns off fencingbwithout
scrubbing)
@
n
e
u
d
k
ya Softufencing for a particular
(
The following example shows the
elimination
n
di e this
u
disk device:
l
a us
m
a
# cldev set -p default_fencing=nofencing
d5
k e to
g
n
s
Updating shared
devices
on
node
1
a cen
bshared
a
Updating
li devices on node 2
b
e
u
l
k
a erab
y
l
u nsf
d
b
a
tra
g
n
n
no
aba
5-30

Eliminating SCSI Fencing Globally

It is likely that if none of your shared disk devices support fencing (all are
SATA drives, for example), you would have had to turn off fencing
globally during scinstall time.
In the unlikely case that you need to turn off fencing on all devices after
Sun Cluster is already running, you can use the new values of the clusterwide global_fencing property. The values are the same as those for the
per-disk fencing property, listed previously.
The following example shows the elimination of all disk fencing globally.
As in the earlier examples of switching from scsi2 to scsi3 fencing, a disk
which is already a quorum device requires special manipulation:
s
) ha
ab
y
m
om
.
c
vincent:/# cluster set -p global_fencing=nofencing

sm uide
c
e
Warning: Device instance d4 is a quorum device
G - fencing
t
@
n
b
protocol remains SCSI-3 for the
device.
aku1 Stude
Updating shared devices on (node
y
inon node
Updating shared devices
d
his2
t
u
l
e
vincent:/# clq addad5
m tod4us
aremove
vincent:/# clq
k
vincent:/#
show
ngcldev
se d4
a
n
b
e
a
lic
b
e
u
l
kDID Device
b Instances ===
a
a
r
y
feDevice Name:
ul nDID
s
d
/dev/did/rdsk/d4
b
a
a
r
t
g
Full
Device
Path:
an non
Full Device Path:
Replication:
default_fencing:
none
scsi3
vincent:/# cldev set -p default_fencing=global d4

vincent:/# clq add d4
vincent:/# clq remove d5
Software Quorum for Disks with no SCSI Fencing

As shown in the previous section, you are allowed to add a disk on which
SCSI fencing has been eliminated as a quorum device.

5-31
Sun Cluster will quietly implement its own software quorum mechanism
to reliably and atomically simulate the SCSI-2 or SCSI-3 race for the
quorum device. The persistent reservations will be implemented using
PGRE, exactly as when a SCSI-2 device is used as a quorum device.
The following command shows the verification that a quorum device is
using the software quorum mechanism, since its SCSI fencing has been
eliminated:
# clq show d4
Quorum Devices ===
Quorum Device Name:
Enabled:
Votes:
Global Name:
Type:
Access Mode:
Hosts (enabled):
d4
yes
1
/dev/did/rdsk/d6s2
shared_disk
sq_disk
vincent, theo
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
5-32

Viewing and Administering Interconnect Components

The clinterconnect (clintr) command allows you to display the
configuration and status of the private networks that make up the cluster
transport. In addition, it allows you to configure new private networks
and/or remove private network components without having to reboot
any cluster nodes.
Viewing Interconnect Status
s
) ha
The clintr status command shows status of all private network paths
between all pairs of nodes:
y
m
om
c

csm Guide
e
@ ent
b
u
Endpoint1
Endpoint2
Status
k tud
a
y
( is S
-------------n--------i
d
h
vincent:ce3
theo:ce3
Path
online
t
u
l
e
a
s
vincent:ce2
Path online
am to utheo:ce2
k
ng ense
a
b
a
ic
lsoftware
Nobparticular
administration is required to repair a broken
e
u
l
k
b
interconnect
path..
If
a
cable
breaks, for example, the cluster immediately
a era
y
l
f
s a path failure. If you replace the cable, the path immediately goes
bdu tranreports
back
on line.
a
g
n
n
no
aba
# clintr status
Adding New Private Networks
You can cable a new private network and get it defined in the cluster
without any reboots or interruption to any existing service.
The private network definitions in the cluster configuration repository are
somewhat complex. You must perform one of the following actions:
For a private network defined without a switch (two-node cluster

only):
Define the two adapter endpoints (for example, theo:ce4 and

vincent:ce4)
Define the cable between the two endpoints

5-33
For a private network defined with a switch:
Define the adapter endpoints for each node
Define the switch endpoint (cluster assumes any endpoint not

in the form of node:adapter is a switch)
Define cables between each adapter and the switch
For example, the following commands define a new private network for a
two-node cluster. The definitions define a switch. This does not mean that
the switch needs to physically exist, it is just a definition in the cluster.
#
#
#
#
#
clintr
clintr
clintr
clintr
clintr
add
add
add
add
add
vincent:ce4
theo:ce4
switch3
vincent:ce4,switch3
theo:ce4,switch3
s
) ha
y
m
om
c
csm Guide
# clintr status
e
@ ent
b
u
ak Stud
y
(
in this
d
u
Endpoint1
Endpoint2
Status
l
e
a
s
u
-------------am t--------o
k
vincent:ce3 ng
Path online
se theo:ce3
a
n
vincent:ce2
theo:ce2
Path online
b
e
c
a
i
l
b
vincent:ce4
theo:ce4
Path online
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
5-34

Using the clsetup Command

The clsetup command is a menu-driven utility meant to guide you
through many common (but not all) administrative operations. The
clsetup command leads you through a series of menus and dialogues,
and at the end calls the lower-level administrative commands for you.
In general, it always shows the lower-level commands as it runs them, so
it has educational value as well.
# clsetup
*** Main Menu ***
s
) ha
y
m
om
c
sm uide
c
e
1) Quorum
G
t
@
n
b
2) Resource groups
ku tude
3) Data Services (ya
n this S
i
4) Cluster interconnect
d
lu and
e volumes
5) Device groups
a
s
m
u
6) Private
ka ehostnames
to
g
n Newenodes
s
ncluster
ba7)
8) Other
tasks
c
a
li
b
e
u
l
k rab
a
y
l
e ?) Help with menu options
f
u
s
d
q) Quit
ab -tran
g
an non
ab

5-35
Comparing Low-level Command and clsetup Usage
Even if you use the clsetup utility, you still need to know about how
things are done in the Sun Cluster software environment.
For example, if you go to the cluster interconnect submenu, you see the
following:
*** Cluster Interconnect Menu ***
1)
2)
3)
4)
5)
6)
7)
8)
Add a transport cable

Add a transport adapter to a node
Add a transport switch
Remove a transport cable
Remove a transport adapter from a node
Remove a transport switch
Enable a transport cable
Disable a transport cable
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
?) Help
al use
m
q) Return
kato the
toMain Menu
g
e
ban licens
Option:
a
b
u
le
k
b
a
a
l yFor example,
er if you wanted to permanently delete an entire private
f
u
s
d
an you need to perform the following tasks in the correct order:
ab -trnetwork,
g
n non 1. Disable the cable or cables that define the transport.
a
b
a
This is a single operation for a crossover-cable definition, or multiple

operations if a switch is defined (one cable per node connected to the
switch).
2.
Remove the definition(s) of the transport cable or cables.
3.
Remove the definition of the switch.
4.
Remove the definitions of the adapters, one per node.
Nothing bad happens if you try to do things in the wrong order; you are
just informed that you missed a step. But this would be the same with the
command-line or with clsetup.
The clsetup command saves you from needing to remember the
commands, sub-commands, and options.
5-36

Sun Cluster Manager
Sun Cluster Manager

The Sun Cluster Manager is an optional Web-based administrative tool for
the Sun Cluster 3.2 environment. You can choose whether or not to install
Sun Cluster Manager along with the Sun Cluster framework from the Java
ES installer.
Sun Cluster Manager runs underneath the Sun Java Web Console. Sun
Java Web Console is a single sign-on interface to many Solaris OS
applications that are implemented as Web back-ends. When you log in to
the Sun Java Web Console, you might see other Web applications available
besides Sun Cluster Manager.
s
) ha
y
m
om
c
csm Guide
e
tthe Java language to
@ with
You can use any modern Web-browser u
enabled
n
b
e
d Manager.
access the Sun Java Web Console y
and
akthe Sun
tuCluster
(
S
in this
d
u
Usage of Sun Cluster Manager
al usisemuch like the usage of clsetup in that:
m
ka need
ttoounderstand the cluster tasks before
You stillgreally
e
s
accomplishing
ban licenanything.
a
b It saves
u
leyou from having to remember the syntax of the lower-level
k
b
a
a
r
l y sfecommands.
u
d
ab -tran It accomplishes all its actions through the lower-level commands,
g
n non
and shows the lower-level commands as it runs them.
a
b
a
The Java ES installer also automatically installs or upgrades the Sun Java
Web Console, if needed, regardless of whether you choose to install Sun
Cluster Manager.
Sun Cluster Manager has additional benefits:
It offers some graphical topology views of the Sun Cluster nodes

with respect to device groups, resource groups, and so forth.
It highlights faulted components in real time. The browser refreshes

automatically as component status changes.
Many of the exercises in the modules following this one offer you the
opportunity to view and manage device and resource groups through Sun
Cluster Manager. However, this course does not focus on the details of
using Sun Cluster Manager as the method of accomplishing the task.
Similar to clsetup, after you understand the nature of a task, Sun Cluster
Manager can guide you through it. This is frequently easier than
researching and typing the syntax correctly using the lower-level
commands.

5-37
Sun Cluster Manager
Logging Into the Sun Java Web Console
Use the following URL on a Web browser enabled with Java technology to
access the Sun Java Web Console.
https://nodename:6789
Figure 5-1 shows the Sun Java Web Console login window. Cookies must
be enabled on your Web browser to allow Sun Java Web Console login
and usage.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Figure 5-1
Sun Java Web Console Login Window
Log in with the user name of root and the root password.
Alternatively, you can create a non-root user or role authorized to have
full or limited access through RBAC. By default, any user will be able to
log in to Sun Cluster Manager through the Sun Java Web Console and
have view-only access.
5-38

Sun Cluster Manager
Accessing Sun Cluster Manager

The Sun Cluster Manager application is always available from within the
Sun Java Web Console. You can have other administrative applications
available as well. You can choose to open the Sun Cluster Manager in its
own browser window or in the same window. Figure 5-2 shows the screen
you access after logging into the Sun Java Web Console.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Figure 5-2
Applications Available Underneath Sun Java Web Console

5-39
Sun Cluster Manager
Navigating the Sun Cluster Manager Main Window
Sun Cluster Manager navigation is simple, with the tree-like navigation

bar on the left, and the display corresponding to your current selection in
the main frame.
Topological views are available through a tab in the main window.
Figure 5-3 shows the initial screen.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Figure 5-3
5-40
Sun Cluster Manager Main Window

Running Cluster Commands as Non-root User or Role Using Role Based Access
Running Cluster Commands as Non-root User or Role

Using Role Based Access Control (RBAC)
Sun Cluster 3.2 has a simplified RBAC structure that can allow you to
assign cluster administrative privileges to non-root users or roles. There
are only three authorizations that pertain to Sun Cluster:
solaris.cluster.read
This authorization gives the ability to do any status, list, show
subcommands. By default every user has this authorization, since it
is in the Basic Solaris User profile
solaris.cluster.modify
s
) ha
y
m
om
c
solaris.cluster.admin
csm Guide
e
t
This gives the ability to do switch,b@
online,
offline,
enable,
n
e
u
d
k
disable subcommands. ya
tu
(
S
n
di e this
u
l
a usRBAC authorizations of a much finer
m
Note Sun Cluster
3.1 required
a
k authorizations
to
g
granularityn
(separate
to manage quorum, interconnect,
e
sso on). These old
a
n
b
e
resource
groups,
and
authorizations are syill accessible
c
a
i
l
b
and
for back-compatibility, but they are no longer required.
u are supported
le
k
b
a
a
l y sfer
u
d
ab -tranLike any RBAC authorizations, the cluster authorizations can be:
g
n non
a
Assigned directly to a user
b
a
This gives the ability to do add, create, delete, remove and
related subcommands
Assigned directly to a role (allowed users have to assume the role,

and then they would get the authorization)
Assigned to a rights profile that is then given to a user or a role
There is a review of RBAC concepts in the appendix.

Note The Sun Java Web Console detects if the user you use for its own
login has the ability to assume any roles. If it does, the Web Console gives
you additional options to assume the role. The information is passed
down to the Sun Cluster Manager, which follows the RBAC
authorizations.

5-41
Controlling Clusters
Basic cluster control includes starting and stopping clustered operation on

one or more nodes and booting nodes in non-cluster mode.
Starting and Stopping Cluster Nodes

The Sun Cluster software starts automatically during a system boot
operation. Use the standard init or shutdown command to shut down a
single node. Use the cluster shutdown command to shut down all
nodes in the cluster.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak installation,
Note After an initial Sun Cluster software
tud there are no
y
(
S
s concerned.
configured resource groups with
inwhichtto
d
hibe
u
l
e
a us
m
a
k e to
g
ban alicCluster
ens
ShuttingaDown
b
u
le
k
b
a
a
r down the entire cluster with the cluster shutdown
l yYou can
eshut
f
u
s
d
from any active cluster node. When your cluster is in
an
ab -trcommand
g
production
and running clustered applications, you will have a goal of
n
n
o
a
n
never having to do this. The whole purpose of the Sun Cluster
ab
Before shutting down an individual node, you should switch resource

groups to the next preferred node and then run shutdown or init on that
node.
environment is that you should always be able to keep at least one node
running.
vincent:/# cluster shutdown -y -g0
Apr 9 03:59:49 vincent cl_runtime: NOTICE: CMM: node
reconfiguration #20 completed.
Apr 9 03:59:49 vincent cl_runtime: NOTICE: CMM: Quorum
device /dev/did/rdsk/d4s2: owner set to node 1.
vincent:/# /etc/rc0.d/K05stoprgm: Calling clzc halt -n
vincent +
/etc/rc0.d/K05stoprgm: Calling scswitch -S (evacuate)
/etc/rc0.d/K05stoprgm: disabling failfasts
Apr 9 03:59:58 vincent syseventd[413]: SIGHUP caught reloading modules
Apr 9 03:59:59 vincent syseventd[413]: Daemon restarted
5-42

svc.startd: The system is coming down. Please wait.
svc.startd: 150 system services are now being stopped.
Apr 9 04:00:05 vincent cl_eventlogd[3143]: Going down on
signal 15.
Apr 9 04:00:37 vincent syslogd: going down on signal 15
Apr 9 04:00:37 rpc.metad: Terminated
svc.startd: The system is down.
syncing file systems...NOTICE: clcomm: Path vincent:ce2 theo:ce2 being drained
NOTICE: clcomm: Path vincent:ce3 - theo:ce3 being drained
NOTICE: CMM: Node theo (nodeid = 2) is down.
NOTICE: CMM: Cluster members: vincent.
TCP_IOC_ABORT_CONN: local = 000.000.000.000:0, remote =
172.016.004.002:0, start = -2, end = 6
TCP_IOC_ABORT_CONN: aborted 0 connection
NOTICE: CMM: node reconfiguration #22 completed.
done
NOTICE: CMM: Quorum device /dev/did/rdsk/d4s2: owner set to
node 1.
WARNING: CMM: Node being shut down.
Program terminated
{0} ok
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

5-43
Booting Nodes Into Non-Cluster Mode

Occasionally, you might want to boot a node without it joining the cluster.
This might be to debug some sort of problem preventing a node from
joining a cluster, or to perform maintenance. For example, you upgrade
the cluster software itself when a node is booted into non-cluster mode.
Other nodes might still be up running your clustered applications.
To other nodes that are still booted into the cluster, a node that is booted
into non-cluster node looks like it has failed completely. It can not be
reached across the cluster transport.
s
) ha
To boot a node to non-cluster mode you supply the -x boot option, which
gets passed through to the kernel.
y
m
om
c
m -x
ide
Booting a SPARC Platform Machine With
csthe
u
e
G
t
@
n
b
Command
ku tude
a
y
in ( isthsimple:
is S
For a SPARC-based machine,ubooting
d
al use
m
{1} ok boot -x
ka e to
g
Resetting ...
ban licens
a
b
u
le
k
b
a
a
Rebooting with
command:
boot
-x
y fer
l
u
s
d /pci@1f,700000/scsi@2/disk@0,0:a
Boot device:
File and args: -x
n
b
a
a
r
t
SunOS
Generic_141444-09 64-bit
g Release
n- 5.10 Version
n
o
a
Copyright
1983-2009
Sun
Microsystems,
Inc. All rights reserved.
n
ab
Use is subject to license terms
Hostname: vincent
Reading ZFS config: done.
Mounting ZFS filesystems: (5/5)
vincent console login: root
Password:
# cluster status
cluster: (C152734) This node is not in cluster mode.
5-44

Booting an x86 Platform Machine With the -x

Command
For an x86 machine, booting is a little more complicated.
For versions of Solaris 10 OS supported with Sun Cluster 3.2 (Update 3
and above), the Gnu boot menus (GRUB) are automatically used. As the
machine boots, you see the following menu, with the normal OS
highlighted:
GNU GRUB version 0.95 (615K lower / 2095552K upper memory)
s
) ha
+----------------------------------------------------------------------+
| Solaris 10 11/09 s10x_u8wos_08 X86
|
| Solaris failsafe
|
+---------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press enter to boot the selected OS, 'e' to edit the
commands before booting, or 'c' for a command-line.
The highlighted entry will be booted automatically in 20 seconds.
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
l OS highlighted,
e
a10
s
With the normal Solaris
press the e key to edit the boot
m
u
a the following
o
k
t
parameters. You
see
screen:
ng ense
a
b
a
lic lower / 2095552K upper memory)
b
e
u
l
GNU GRUB version
0.95
(615K
k rab
a
y
ul nsfe
d
+----------------------------------------------------------------------+
b
a (hd0,0,a)
tra
| root n
|
g
n
o
a
n
|
ab | kernel /platform/i86pc/multiboot
| module /platform/i86pc/boot_archive
|
+----------------------------------------------------------------------+
Use the ^ and v keys to select which entry is highlighted.
Press 'b' to boot, 'e' to edit the selected command in the
boot sequence, 'c' for a command-line, 'o' to open a new line
after ('O' for before) the selected line, 'd' to remove the
selected line, or escape to go back to the main menu.

5-45
You need to use the arrows to highlight the line that begins with kernel,
then press the e again to edit that specific line and add the -x
[ Minimal BASH-like line editing is supported. For the first word, TAB
lists possible command completions. Anywhere else TAB lists the possible
completions of a device/filename. ESC at any time exits. ]
grub edit> kernel /platform/i86pc/multiboot -x
Press return to get back to the screen listing boot parameters, where you
will see your -x in the appropriate place. Now you can press b to boot.
s
) ha
y
m
om
c
+----------------------------------------------------------------------+
csm Guide |
| root (hd0,0,a)
e
@ ent
b
| kernel /platform/i86pc/multiboot -x
|
u
d
k
u
a
| module /platform/i86pc/boot_archive
|
t
(y is S
n
+----------------------------------------------------------------------+
i
h highlighted.
tis
lud entry
Use the ^ and v keys to select which
e
a
us command in the
Press 'b' to boot, 'e' to k
edit
amthetoselected
g command-line,
boot sequence, 'c' forna
'o' to open a new line
se
a
n
b
e
after ('O' for before)
line, 'd' to remove the
a the
lic selected
b
e
selected line,
or
escape
to
go
back
to the main menu.
u
l
k
b
a
a
l y5.10sVersion
er Generic_141445-14 64-bit
f
u
d
SunOS Release
an Sun Microsystems, Inc. All rights reserved.
ab 1983-2009
r
t
Copyright
g
n is subject
on to license terms.
a
Use
n
b
a
GNU GRUB version 0.95 (615K lower / 2095552K upper memory)
Hostname: gabi
gabi console login: root

Password:
Jul 10 00:31:50 gabi login: ROOT LOGIN /dev/console
Last login: Mon Jul 10 00:19:32 on console
Sun Microsystems Inc.
SunOS 5.10
Generic Jan 2005
# cluster status
cluster: (C152734) This node is not in cluster mode.
5-46

Placing Nodes Into Maintenance State

If you anticipate that a node will be down for an extended period, you can
place the node into maintenance state from an active cluster node. The
maintenance state disables the nodes quorum vote. You cannot place an
active cluster member into maintenance state. A typical command is as
follows:
theo:/# clq disable vincent
The clquorum status command shows that the possible vote for theo
is now set to 0.
s
) ha
theo:/# clq status
y
m
Cluster Quorum ===

om
c
--- Quorum Votes Summary --csm Guide

e
@ ent
b
u
Needed
Present akPossible
tud
y
(
S
------------------in th1is
d
1
1
u
al use
m
ka e to
g
s Node --nby
--- Quorum
ban Votes
e
c
a
li
b
e
u
l
k
Present
Possible
Status
a NodeerName
ab
y
l
f
------------------s
bdu tran--------vincent
0
0
Offline
a
g
n
n
theo
1
1
Online
no
aba
----------d4
Present
------0
Possible
-------0
Status
-----Offline
In addition, the vote count for any dual-ported quorum device physically
attached to the node is also set to 0.
You can reset the maintenance state for a node by rebooting the node into
the cluster. The node and any dual-ported quorum devices regain their
votes.

5-47
Maintenance Mode Example
To see the value of placing a node in maintenance state, consider the

following topology (shown in Figure 5-4), which was described in
Module 3, Preparing for Installation and Understanding Quorum
Devices.
Switch
Switch
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak SQD(1)
tud
y
(
QD(1)
in this
d
u
al use
m
ka e Quorum
to
Figure 5-4 Clustered-Pair
Devices
g
n
s
a
n
ab lice
b
Imagine
u thereblise no way that you can add a quorum device between
k
a
ra 3 as we described in an earlier module (maybe you have no
ynodes f2eand
l
u
s
n available storage controllers).
bd trmore
a
a
g onn
a
n
Now imagine that node 4 has died. At that point in time, the cluster
ab
Node 1
(1)
Node 2
(1)
Node 3
(1)
Node 4
(1)
survives, but you can tell that if you lose node 3, you will lose the whole
cluster, because you would have only three of the total possible six
quorum votes.
If you put node 4 into maintenance mode, you would eliminate its
quorum vote, and the quorum vote of the shared quorum device between
nodes three and four. There would temporarily be a total possible value of
four quorum votes, making the required number of votes 3. This would
allow you to survive the death of node 3.
5-48

Modifying Private Network Address and Netmask

The final task presented in this module is unique in that it must be
accomplished while all nodes are in multi-user, non-cluster mode.
In this mode, if you run clsetup, it recognizes that the only possible task
is to change the private network information, and it guides you through
this task. You can choose a different network number, and you can give a
different anticipated maximum number of nodes and subnets and use a
different suggested netmask.
You run this from one node only, and it automatically propagates to the
other nodes.
s
) ha
ab
y
m
m
o
c
same
Note In the current revision nodes can communicate
using the
e
m
d
s
i
Remote Procedure Calls that are used use to communicate
ec t Guduring
@
scinstall. No other remote shell accessbis required
n between nodes. In
e
u
d
k
previous revisions of Sun Cluster 3.2,
you
have
to
set
up rsh or ssh
a Stu
y
(
equivalence for root manually.
in this
d
u
al use
m
# clsetup
ka e to
g
ban licens
a
*** Main Menuub
***
le
k
b
a
a
l yfromsfone
er of the following options:
u
Select
d
ab -tran
g
an non1) Change Network Addressing and Ranges for the Cluster Transport
2) Show Network Addressing and Ranges for the Cluster Transport
q) Quit
Option:
>>> Change Network Addressing and Ranges for the Cluster Transport <<<
Network addressing for the cluster transport is currently configured
as follows:
Private Network ===

5-49

private_netaddr:
private_netmask:
max_nodes:
max_privatenets:
num_zoneclusters:
172.16.0.0
255.255.240.0
64
10
12
Do you want to change this configuration (yes/no) [yes]?
yes
The default network address for the cluster transport is 172.16.0.0.

Do you want to use the default (yes/no) [yes]?
What network address do you want to use?
no
s
) ha
192.168.5.0
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
Maximum number of nodes anticipated ifor
n future
is growth [2]? 4
d
h
t
u
e
al anticipated
s
Maximum number of private networks
for future growth
m
u
a
o
k
t
[2]?4
ng ense
a
b
c about number of virtual clusters
a
liask
// Note this script
does not
b
e
u
l
k rab
a
y
fe of 255.255.255.0 to meet anticipated future
ula netmask
Specify
s
d
n
b
a
requirements
tra of 4 cluster nodes and 4 private networks.
g
n
n
o
aba Whatn netmask do you want to use [255.255.255.0]? <CR>
.
.
.
.
.
.
Is it okay to proceed with the update (yes/no) [yes]?
yes
cluster set-netprops -p private_netaddr=192.168.5.0 -p

private_netmask=255.255.255.0 -p max_nodes=8 -p max_privatenets=4
Command completed successfully.

After you reboot into the cluster, your new private network information is
automatically applied by the cluster. For cluster-aware applications, the
same clprivnet0 private hostnames (by default, clusternode1-priv,
and so on) now resolve to the new addresses.
5-50

Exercise: Performing Basic Cluster Administration

Task 1 Verifying Basic Cluster Configuration and Status
Task 2 Reassigning a Quorum Device
Task 3 Adding a Quorum Server Quorum Device
Task 4 Preventing Cluster Amnesia
Task 5 Changing the Cluster Private IP Address Range
Task 6 (Optional) Navigating Sun Cluster Manager
s
) ha
y
m
om
c
e
idthe
csmtool
Boot all nodes into the cluster and run the cconsole
on
u
e
G
t
@
administration workstation.
n
b
ku tude
a
y
n ( this S
i
d
Note During this exercise,
you see italicized names, such as
lu when
e
a
s
IPaddress, enclosure_name,
am to unode1, or clustername embedded in a
k
command string,
g substitute
e the names appropriate for your cluster.
n
s
a
n
ab lice
b
u
le
k
b
a
a
l y 1sferVerifying Basic Cluster Configuration and
Task
u
d
an
ab Status
r
t
g
n non
a
b
a
Preparation
Perform the following steps to verify the basic status of your cluster:
1.
Verify the cluster global properties and node properties:

# clnode show
2.
Use the clquorum (clq) command to verify the current cluster

membership and the quorum status.
# clq status

5-51
3.
4.
Record the quorum configuration from the previous step.

Quorum votes needed:
_____
Quorum votes present:
_____
Quorum votes possible:
_____
Verify all cluster disk paths.

# cldev status
5.
Verify the interconnect status:

# clintr status
6.
Verify the revision of the currently installed Sun Cluster software on

each node.
s
) ha
y
m
om
c
Task 2 Reassigning a Quorum Device csm uide

e tG
@
n
b
e
u
d
k
Pretend that your quorum device (or one
of
your
quorum
devices) has
a Stu
y
(
failed. Perform the following tasks
in fromthany
is one node in the cluster to
reassign the quorum device:lud
a use
m
1. Verify the status
kaof your
tocurrent quorum device(s):
g
e
an cens
# clqbstatus
a
li d#
#bcldev e
status
u
l
k
a Show
aallb disk paths:
r
y
2.
l
e
f
bdu trans# cldev list -v
a
ang non 3. Choose a different shared disk to be the new quorum device: Add
# clnode show-rev -v
ab
your new quorum device:

# clq add new-d-#
4.
Remove your old broken quorum device:

# clq remove old-d-#
5.
Verify the quorum status again:

# clq status
Task 3 Adding a Quorum Server Quorum Device

Your instructor will have the quorum server server-side software available
so that you can add a quorum-server device in the cluster.
5-52

Your instructor will provide you the quorum server IP address/name

information. You can just add the device using the IP address, or you can
use the name if it is resolvable on your cluster nodes:
Perform the following steps on any one cluster node:
1.
Add the quorum server quorum device. The name you give to the
device on the cluster side (the last argument to the command) is
unimportant, you can use anything you want.
# clq add -t quorum_server \
-p qshost=qserver_name_or_IP -p port=9000 qservdude
# clq status
as
h
2. Delete your disk quorum device(s). If you have a quorum server
)
y
m
device you need no other quorum devices:
om
c
# clq remove old-d-# [old-d#]
csm Guide
e
# clq status
@ ent
b
u
akAddSback
3. (Optional): Reverse the process.
tudthe disk quorum devices
y
(
and remove the quorum
inserverthquorum
is device. If you have a cluster
d
u
l having
with non-storage a
nodes,
a
quorum server quorum device is
e
s
m
u
not necessarily
owhat you want, because it will allow the nonka exactly
talive
g
e
storagennode
to s
stay
even if both storage nodes have failed.
a
n
b
e
#a clq add
licd# [d#]
b
e
u
l
k #raclq
b remove qservdude
a
y
l
e
f
bdu trans
a
ang non
ab

5-53
Task 4 Preventing Cluster Amnesia
Perform the following steps to demonstrate how the cluster prevents

cluster amnesia, using persistent reservations on the quorum device. You
can review the information about what you will see by checking
Module 3, Preparing for Installation and Understanding Quorum
Devices.
1.
Verify that all nodes are active cluster members. If you have a
three-node cluster (Pair +1), shut down the node not attached to
storage with init 0.
2.
Shut down Node 1 with init 0.
3.
Shut down Node 2.
s
) ha
y
m
om
c
4. Boot Node 1.
e
csm G
uidis because
Node 1 should hang waiting for operational e
quorum.
This
t the quorum
@ enfrom
b
the reservation key for Node 1 has been
removed
u
ak Stud
disk(s).
y
(
in should
d
hiscomplete the cluster software
5. Now boot Node 2. Both
nodes
t
u
l
e
ajoin inuclustered
s
startup sequence and
operation.
m
a
o
k
t
ng ense
a
b
a
lic the Cluster Private IP Address
Task 5 uChanging
b
e
l
k rab
a
y
Range
ul nsfe
d
b
a
a
trPerform
g
n
the following steps:
n
o
a
n
b
a
# init 0
1.
Shut down the entire cluster by typing on any one node:
# cluster shutdown -y -g0
5-54
2.
Boot all nodes to non-cluster mode using boot -x
3.
Choose (any) single node on which to perform the commands.
4.
On the chosen node only, run the procedure to change the cluster
private IP address range:
a.
Execute the clsetup command.
b.
Choose option 1 to change the configuration.
c.
Verify (answer yes) when asked if you want to change the

settings.
d.
Answer no when asked if you want to accept the default base

address.


e.
Enter a new available private address base address. Consult

your instructor if you are not sure. Do not accidentally conflict
with the public network address space.
f.
Answer no when asked if you want to accept the default

netmask.
g.
Enter 4 for the maximum number of nodes.
h.
Enter 4 for the maximum number of private networks.
i.
Accept the suggested netmask (255.255.255.0).
j.
Confirm that you want to proceed with the update.
5.
When the command succeeds, reboot all nodes back into the cluster.
6.
Verify that the private network addresses have changed.
s
) ha
y
m
om
c
m uide
csglobal.
b. Verify the output of cluster show e-t
G
t
@
n
b
ku tude
a
y
n ( this S
i
d
lu se
a
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba
a.
Use ifconfig -a and verify the information for the physical

private network adapters and for the clprivnet0 adapter.

5-55
Task 6 (Optional) Navigating Sun Cluster Manager

1.
In a Web browser window on your administrative workstation, type

the following URL:
where nodename is the name of one of your nodes currently booted
into the cluster.
If you do not succeed in reaching the Sun Java Web Console, you
might need to disable or set exceptions for the proxy settings in your
Web browser. Ask your instructor if you need help.
s
) ha
y
m
om
c
sm uide
c
e
a.
Run the browser on your remote display.
G
t
@
n
b
de through ssh.
ku thetutraffic
b. Run the browser locally anda
tunnel
y
n ( this S
i
d
lu se
a
Note Consult your instructor
u how this will be accomplished.
am toabout
k
ntheg user
se
a
n
2. Log inbas
root with the root password.
e
a
ic
l
b
e
u
l
3. k Navigate
b to the Sun Cluster Manager application.
a
a
r
y
fe
yourself with Sun Cluster Manager navigation and the
ul 4. nsFamiliarize
d
b
a
topological
views.
a
r
g on-t
n
a
n
ab
If you are running the class with remote equipment, you might be
able to use a Web browser to access Sun Cluster Manager in one of
two ways:
5-56

Exercise Summary
Exercise Summary

Experiences
Interpretations
Conclusions
Applications
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

5-57
Exercise Summary
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
5-58

Module 6
Using VERITAS Volume Manager With Sun

Cluster Software
s
) ha
Objectives
ab
y
m
om
c
e
csm GVolume
uid Manager
Describe the most important concepts of e
VERITAS
tthe Sun Cluster 3.2
@
(VxVM) and issues involved in using
VxVMein
n
b
u
ak Stud
software environment
y
(
in this and shared storage disk
Differentiate between
bootdg/rootdg
d
u
al use
groups
m
a
o
kVERITAS
tVolume
g
Initialize
a
Manager disk
e
n ens
a
b
a
ic basic objects in a disk group
Describe
lthe
b
e
u
l
k Describe
a
ab the types of volumes that will be created for Sun Cluster 3.2
r
y
l
e
f
bdu trans software environments
a
Describe the general procedures and restrictions for installing and
ang non
administering VxVM in the cluster
Use basic commands to put disks in disk groups and build volumes
Describe the two flags used to mark disks under the typical hot
relocation
Register, manage, and resynchronize VxVM disk groups as cluster

device groups
Create global and failover file systems on VxVM volumes
Describe the procedure used for mirroring the boot drive
6-1
Relevance
Relevance

Why is it so important to mirror JBOD disks with VxVM if the

cluster already provides node-to-node failover?
Are there any VxVM feature restrictions when VxVM is used in the
Sun Cluster software environment?
In what way does Sun Cluster provide high availability for disk
groups?
Can your prevent Sun Cluster from providing a global device service
for your volumes?
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
6-2

820-7458 (for Sun Cluster 3.2 11/09).
Sun Cluster 3.2 11/09)uide for Solaris OS, part number 820-4676 (for
Sun Cluster 3.2 1/09).
s
) ha
y
m
Veritas Storage Foundation Installation Guide (version 5.0

omfor Solaris),
c
Symantec Corporation (available with Veritas m

Storage Foundation
e
d
s
i
software).
ec t Gu
@
b den
u
k
ya Stu
(
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Using VERITAS Volume Manager With Sun Cluster Software

6-3
Introducing VxVM in the Sun Cluster Software Environment
Introducing VxVM in the Sun Cluster Software

Environment
This module does not intend to replace a full class in VxVM
administration. Rather, it briefly introduces the most important concepts
of VxVM and focuses on issues involved in using VxVM in the
Sun Cluster 3.2 environment.
Only the following versions of VxVM are supported in Sun Cluster 3.2
software:
For Sun Cluster 3.2 updates earlier than Update 2 (1/09)
VxVM 4.1 MP1 Solaris 9 OS/Solaris 10 OS (SPARC)
VxVM 4.1 MP2 for Solaris 10 OS (x86)
s
) ha
y
m
om
c
VxVM 5.0 Solaris 9 OS/Solaris 10 OS (SPARC)

csm Guide
e
For Sun Cluster 3.2 Update 2 (1/09)
@ ent
b
u
k tud10 OS (SPARC and
VxVM 5.0 MP3 RP1 Solarisy9aOS/Solaris
(
x86)
n this S
i
d
lu 3 s(11/09)
e
a
For Sun Cluster 3.2
Update
m
u
a
kMP3
toand RP3 Solaris 9 OS/Solaris 10 OS
VxVM
5.0
RP2
g
e
s
(SPARC
ban liand
enx86)
c
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab
6-4

Exploring VxVM Disk Groups

The assignment of disk drives or logical units (LUNs) into disk groups is
the most important organizational concept for managing VxVM in the
cluster environment.
Every disk or LUN used by VxVM must be a member of exactly one disk
group. Disks in the same disk group act as an organizational unit.
Shared Storage Disk Groups
ab
as
h
All of the data associated with applications that run in the Sun Cluster
3.2
)
yto at
software environment must be in storage that is physically ported
m
least two nodes. Disk groups are created using only disks
oinmmultiple
c
e
arrays that are connected to the same collection ofsnodes.
m
d
i
ec t Gu
@
b den
u
k
VERITAS Management on (Local
tu (Optional in
ya Disks
S
n
VxVM 4.x and Above)ludi
his
t
e
a us
m
a
k VxVM
torequired that each node have an independent
Prior to VxVM
4.0,
g
e
ns In these earlier versions you could not
disk group
rootdg.
bannamed
e
c
a
li else in VxVM until rootdg was configured. You
b anything
configure
e
u
l
k
a needed
atobhave this disk group even if you did not intend VxVM to
r
y
l
e
f
s
your boot disk. In that case, you needed at least one (preferably
bdu tranmanage
a
- two, for HA) sacrificial local disks so that you could have a rootdg.
ang non
Starting in VxVM 4.x, it is optional to have any disk group at all configured
for the local disks. You can choose not to configure any VxVM
configurations on local disks, and you are still able to create your shared
storage disk groups.
If you want to have VxVM 4.x or 5.x manage your boot disk (that is, to
encapsulate your boot disk), then you have this disk (and presumably
another because there is no reason to encapsulate root if you are not going
to mirror it) in a dedicated local VxVM disk group. This group can have
any group name. VxVM automatically sets up a symbolic link so that
whatever the actual group name is there will be a link to it called bootdg.

6-5
Typical Sun Cluster VxVM Configuration
Figure 6-1 shows a typical organization of disks into disk groups in a

simple Sun Cluster software environment. While the name of the actual
disk group underlying the bootdg link can be anything, the tool that is
used in the Sun Cluster environment to encapsulate the root disk (clvxvm
encapsulate), which is described later in the module, still names it
rootdg.
Diskgroup nfsdg Diskgroup webdg

Node 1
Node 2
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
Boot Disks
Boot Disks
an cens
b
a
i
(bootdg->rootdg
[optional ])
(bootdg->rootdg
[optional])
l
b
e
u
l
k
a 6-1
abTypical Disk Organization in a Sun Cluster Software
r
y
l
e
Figure
f
Environment
bdu trans
a
g
n non
a
b
a
s
) ha
6-6

Shared Storage Disk Group Ownership

While the disks in disk groups used for Sun Cluster software application
data must be physically connected to at least two nodes, disk groups are
owned, or imported, by only one node at a time. The node that is currently
importing a disk group is the one that does the following:
It physically reads and writes data to the drives within the disk
group.
It manages the disk group. VxVM commands pertaining to that

diskgroup can be issued only from the node importing the disk
group.
ab
as
It can voluntarily give up ownership of the disk group by deporting
h
)
y
the disk group.
m
om
c
csm Guide
Sun Cluster Management of Disk Groups
e
@ ent
b
u
ak Stuthed cluster is responsible for
y
After a disk group is managed by
the cluster,
(
n import
ito
issuing all VxVM commands
d
hisand deport disk groups. The node
t
u
l
e
adisk group
currently importingm
the
s becomes the primary device group
u
a
o
k et
server.
g
n
a cens
b
a
li disk groups so that the Sun Cluster software has
Registering
VxVM
b
e
u
l
k
a knowledge
ab of them is described inRegistering VxVM Disk Groups on
r
y
l
e
f
page
6-32.
bdu trans
a
ang non
Sun Cluster Global Devices Within a Disk Group
While the fact that only one node imports a disk group remains true in the
Sun Cluster software environment, Sun Cluster softwares global device
infrastructure makes it appear that devices in the disk group are accessible
from all nodes in the cluster, including nodes that are not even physically
connected to the disks in the disk group.
All nodes that are not the current primary for a particular disk group
actually access the device data through the cluster transport. This applies
even for other nodes that are physically connected to the disks.

6-7
VxVM Cluster Feature Used Only for Oracle RAC
You might see references throughout the VERITAS documentation to a

Cluster Volume Manager (CVM) feature, and creating disk groups with a
specific shared flag.
This is a separately licensed feature of VxVM that allows simultaneous
access to disk group volumes by multiple nodes, directly through the
storage medium.
In the Sun Cluster 3.2 software, this VxVM feature can be used only by the
Oracle Real Application Clusters (RAC) application. The feature is not
supported on x86, however, you can run RAC using Solaris Volume
Manager or without a volume manager.
s
) ha
y
m
m
o
c
Do not confuse the usage of shared storage disk group throughout

this
e
m
d
s
i
module with the CVM feature of VxVM.
ec t Gu
@
b den
u
k
ya Stu
(
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
6-8

Initializing a VERITAS Volume Manager Disk

Before a disk can be put into a VxVM disk group, it must be initialized by
VxVM. The disk is divided into two sections called the private region and
the public region.
The private and public regions are used for the following different
purposes:
The private region is used to store configuration information.
The public region is used for data storage.
s
) ha
y
Traditional Solaris OS Disks and Cross-Platform
Data
m
m
Sharing (CDS) Disks
co
csm Guide
e
VxVM has two different ways of initializing
t is the traditional
@disks.enOne
b
u
way which uses separate Solaris OS disk
slices
for
the
k tud private region (slice
a
y
3) and the public region (slice 4).( This is a S
n this disk layout specific to the
i
d
Solaris OS.
lu se
a
u
am
o
k
t
As shown in Figure
6-2,
the
private
region is small. For example, in VxVM
ng ense
a
5.0 it is b
32MB.
a
lic
b
e
u
l
k rab
Private region
a
y
l
e
f
bdu trans
a
ang non
ab
Configuration and
Management Information
Data storage
Figure 6-2
Public region
VxVM Disk Initialization (Traditional Solaris OS Only Disks)

6-9
Starting in VxVM 4.0, the new default way to partition disks is called
Cross-Platform Data Sharing (CDS) disks. The VxVM configuration on
these disks can be read by VxVM running on all its supported platforms,
not just on the Solaris OS. As shown in Figure 6-3 this layout combines
the configuration and data storage into one large partition (slice 7)
covering the entire disk.
Private
CDS
private
and
public
regions
both
managed
internally
in s7
s
) ha
y
m
om
c
Public
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
Figure 6-3 CDS Disk Layout
u
al use
m
ka e to you are not able to access VxVM disks
g
In the Sun Cluster
environment
s
nthan
ban other
e
from anyaservers
in the same cluster, it does not matter
c
liyou choosethose
b
e
which
disk
layout
for
your
data disks.
u
l
k
b
a
a
l y sfer
u
d
ancannot use CDS disks for the bootdg.
ab -trYou
g
n non
a
b
While initializing a disk and putting it in a disk group are two separate
a
operations, the vxdiskadd utility can perform both steps for you. The
defaults are always to use CDS disks, but the vxdiskadd command can
guide you through setting up your disks either way
Note You cannot use the CDS layout for EFI disks. The EFI label is
required for disks of size 1 Terabyte or greater. VxVM can deal with these
properly, but only using the sliced layout that is specific to the Solaris OS.
6-10

Reviewing the Basic Objects in a Disk Group

This section reviews some of the basic naming terminology of objects in
disk groups. Many of the objects (such as subdisks and plexes) are often
automatically created for you when you use the recommended commands
(such as vxassist) to create volumes.
Although the GUI for VxVM furnishes useful visual status information,
the most reliable and the quickest method of checking status is from the
command line. Command-line status tools are easy to use in script files,
cron jobs, and remote logins.
Disk Names or Media Names
s
) ha
y
m
om
c
Each disk that you put in a diskgroup has a logical

name that
can
m
ideyou
csindependent
u
assign. This logical name can be anything, andeis
of
its
G
t
@
n
b
Solaris OS logical device name (c#t#d#).
ku tude
a
y
S (DIDs). Its own logical
n ( tDisk
s IDs
i
i
VxVM does not use the SundCluster
h
u
e
naming scheme servesaalsimilarspurpose.
m
u
ka e to
g
ban licens
a
Subdisk
b
u
le
k
b
a
a
l y Assubdisk
er is a contiguous piece of a physical disk that is used as a
f
u
d
ab -tranbuilding block for a volume. The smallest possible subdisk is 1 block and
g
n non
the largest is the whole public region of the disk.
a
b
a
Plex
A plex is a collection of subdisk objects, and describes the layout of the
subdisks as either concatenated, striped, or RAID5. A plex, or data plex,
has the following characteristics:
A plex is a copy of the data in the volume.
A volume with one data plex is not mirrored.
A volume with two data plexes is mirrored.
A volume with five data plexes is mirrored five ways.

6-11
Volume
The volume is the actual logical storage device created and used by all
disk consumers. The disk consumers may be file systems, swap space, or
applications, such as Oracle using raw volumes.
Only volumes are given block and character device files.
The basic volume creation method (vxassist) lets you specify parameters
for your volume. This method automatically creates the subdisks and
plexes for you.
Layered Volume
s
) ha
y
m
om
c
A layered volume is a technique used by VxVM that lets

myouucreate
de RAID
ivolumes,
csmirrored
e
1+0 configurations. For example, you can create two
G
t
@
n
b
and then use those as components in a larger
striped
volume.
This striped
u ude
k
a
volume contains a single plex composed
of
sub-volumes
rather
than
t
y
(
S
n
subdisks.
di e this
u
l
a us
m
a
While this sounds complicated,
k e to the automation involved in the
g
recommended
anvxassist
nscommand makes such configurations easy to
e
create. ab
c
li
b
e
u
l
k
a erab
y
l
u nsf
d
b
a
tra
g
n
n
a
no
ab
6-12

Exploring Volume Requirements in the Sun Cluster Environment
Exploring Volume Requirements in the Sun Cluster

Environment
The only requirement for creating a volume in shared storage disk groups
for the Sun Cluster environment, is that you must mirror across controllers
unless there is full redundancy provided by the controller itself, such as
hardware RAID with multipathing.
With that in mind, any of the following are acceptable volumes to hold
your Sun Cluster application data.
Simple Mirrors
s
) ha
y
m
m arrays
ostorage
c
Figure 6-4 demonstrates subdisks from disks in different
e
m
d
s
i
forming the plexes of a mirrored volume.
ec t Gu
@
b den
u
k
ya Stu
(
n
di e this
u
l
a us
m
a
k evolume
to nfsvol
g
an cens
b
a
c1t0d0
c2t0d0
li
b
e
u
l
disk01
disk02
k
b
disk01-01
disk02-01
a
a
r
y
ul nsfe
d
b
a
tra
plex1
plex2
g
n
n
o
a
n
ab
Figure 6-4
Forming the Plexes of a Mirrored Volume

6-13
Mirrored Stripe (Mirror-Stripe)
Figure 6-5 demonstrates subdisks from disks in the same array striped
together. This is mirrored with a similar configuration to the other array.
volume nfsvol
s
) ha
y
m
disk01-01
disk02-01
om
c
disk03-01
disk04-01
sm uide
c
e
G
plex1
plex2
t
@
n
b
ku tude
a
y
n ( this S
i
d
lu se
a
am to u
k
ng ense
c1t1d0
c2t1d0
a
b
c
a
(disk03)
(disk04)
i
l
b
e
u
l
k
Figure
a 6-5
abMirror-Stripe Volume
r
y
l
e
f
bdu trans
a
g onn
a
n
ab
c1t0d0
(disk01)
6-14
c2t0d0
(disk02)

Striped Mirrors (Stripe-Mirror)
Figure 6-6 shows two mirrored sub-volumes, both mirrored across storage
arrays, striped together to form the final volume.
volume nfsvol
subvolume
s
) ha
y
c2t0d0
m
disk01-01
disk02-01
om disk02
c
sm uide
c
e
G
plex1
plex2
t
@
n
b
subvolume ku
de
u
a
t
y
n ( this S
i
d
disk03-01
lu sedisk04-01
a
am to u
k
e
plex2
ng ensplex1
c1t1d0
c2t1d0
a
b
disk03
c
a
disk04
li
single striped plex
b
e
u
l
k rab
a
y
l Figure
e 6-6 Stripe-Mirror Volume
f
u
s
d
ab -tran
g
Fortunately, as demonstrated later in the module, this is easy to create.
n non
a
b
a
c1t0d0
disk01
Often, the RAID 1+0 configuration is preferred because there might be

less data to resynchronize after disk failure, and you could suffer
simultaneous failures in each storage array. For example, you could lose
both disk01 and disk04, and your volume will still be available.

6-15
Dirty Region Logs for Volumes in the Cluster

Dirty region log (DRL) is an optional feature you can add on a per-volume
basis. The DRL is an extra plex for the volume that does not hold data but
rather holds bitmaps of which regions of the data might be dirty. That is,
they might need to be resynchronized in the volume after a Solaris OS
crash.
DRLs have no effect whatsoever on resynchronization behavior following
disk failure.
Without a DRL, a mirrored volume must be completely resynchronized
after a Solaris OS crash. That is because on recovery (on another node, in
the cluster environment), VxVM has no way of knowing which bits of
data it might have been in the middle of writing when the crash occurred.
s
) ha
y
m
om
c
m datauimarked
de in
cofsthe
With the DRL, VxVM knows exactly which regions
e
Gnot marked in
t
@
the DRL bitmap might need to be resynchronized.
Regions
n
b
dtoe be resynchronized.
kunot need
the bitmap are known to be clean, anda
do
u
t
y
n ( this S
i
d
luvolume
You use a DRL for any large
cluster. The whole purpose of
a
sein the
m
u
the cluster is making a
everything
behave
as
well
as possible after crashes.
o
k
t
ng ense
a
b
a
Size of
DRL
lic
b
e
u
l
k rab
a
y
are
fe small. Use the vxassist command to size them. The size of the
ul DRLs
s
d
n
b
a that the command chooses depends on the version of VxVM, but
a
trlogs
g
n
might
be, for example, 26 kilobytes (Kbytes) of DRL per 1 Gbyte of
n
a
no
ab
storage.
If you grow a volume, just delete the DRL, and then use the vxassist
command to add it back again so that it is sized properly.
Location of DRL
You can put DRL subdisks on the same disk as the data if the volume will
not be heavily written. An example of this is Web data.
For a heavily written volume, performance is better with a DRL on a
different disk than the data. For example, you could dedicate one disk to
holding all the DRLs for all the volumes in the group. They are so small
that performance on this one disk is still fine.
6-16

Viewing the Installation and bootdg/rootdg Requirements in the Sun Cluster
Viewing the Installation and bootdg/rootdg

Requirements in the Sun Cluster Environment
The rest of this module is dedicated to general procedures for installing
and administering VxVM in the cluster. It describes both easy automatic
ways, and more difficult manual ways for meeting the requirements
described in the following list:
The vxio major number must be identical on all nodes This is the
entry in the /etc/name_to_major file for the vxio device driver. If
the major number is not the same on all nodes, the global device
infrastructure cannot work on top of VxVM.
as
h
VxVM must be installed on all nodes physically connected to shared
)
y
storage On non-storage nodes, you can install VxVM ifm
you will
use it to encapsulate and mirror the boot disk.
om
c
e
idManager
If not (for example, you might be using Solaris
csmVolume
u
e
Gnode requires
software to mirror root storage), then
a non-storage
t
@
n
b
only that the vxio major number
deto the
kube added
u
a
t
y
/etc/name_to_major file.
n ( this S
i
d
luon allsstorage
A license is required
nodes not attached to Sun
e
a
m
u
StorEdge A5x00
o If you choose not to install VxVM on a nonka arrays.
t
g
e
storage
node you do not need a license for that node.
ban licens
a
b
u
le
k
b
a
a
Requirements
for bootdg/rootdg
l y sfer
u
d
ab -tran
g
n non
You can have an optional bootdg (pointing to an actual arbitrary group
a
b
a
name) on VxVM 4.x and above. You will likely have this only if you want
to encapsulate and mirror the OS disk.

6-17
Cluster-Specific Issues When Encapsulating the Boot

Disk
There are several cluster-specific issues with encapsulating the boot disk.
These caveats all involve the fact that among the file systems being
encapsulated on your boot disk is the /global/.devices/node@# file
system. The requirements are the following:
Unique volume name across all nodes for the

/global/.devices/node@# volume
Unique minor number across all nodes for the

/global/.devices/node@# volume
s
) ha
y
m
om
c
csm Guide
e
t the clvxvm
@
n
b
If you want to encapsulate the boot disk, you
shouldeuse
u
ak ofSthis
utility which automates the correct creation
tudvolume (with different
y
(
in on each
volume names and minor numbers
d
hisnode). This is described later
t
u
l
in this module.
e
a us
m
a
k e to
g
s
ban licinenSun
DMP Restrictions
Cluster 3.2
a
b
e
u
l
k rab
a
y
fe of VxVM supported by Sun Cluster 3.2 (VxVM 4.1 and 5.0)
versions
ul In n
s
d
b
a cannot permanently disable the DMP feature. Even if you take steps
a
tryou
g
n
n
to
disable DMP, it automatically re-enables itself each time the system is
o
a
n
b
a
booted.
The reason for these restrictions is that this file system is mounted as a
global file system. The typical Solaris OS /etc/mnttab logic still demands
that each device and major/minor combination be unique.
Having multiple paths from the same node to the same storage under
control of VxVM DMP is still not supported. You can:
6-18
Have no multiple paths at all from a node to the same storage
Have multiple paths from a node under the control of:
Sun StorEdge Traffic Manager software
Sun Dynamic Link Manager, for Hitachi storage
EMC PowerPath software, for EMC storage

Installing Supported Multipathing Software

The best strategy is to make sure your multipathed storage systems are
under control of the multipath solution supported by the cluster before
you ever even install Sun Cluster software. To review, the installation and
configuration order would be:
1.
Install Solaris OS and patches
2.
Install and configure any required multipathing software
3.
Install and configure Sun Cluster software
4.
Install and configure VxVM
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

6-19
Installing VxVM in the Sun Cluster 3.2 Software Environment
Installing VxVM in the Sun Cluster 3.2 Software

Environment
This section describes the following:
Using the installer or installvm utilities to install and initialize

VxVM 4.1 or 5.0
Installing the packages by hand
Using the clvxvm utility to synchronize vxio major numbers and to

encapsulate the root disk
Addressing VxVM cluster-specific issues if you installed VxVM

before the cluster was installed
s
) ha
y
m
om
c
Using the installer or installvm Utilitysm

c Guide
e
@ ent
b
u
The VERITAS software comes with two text-based
k tudinstallers, installer
a
y
(which can install all of the VERITAS
products)
( is S and installvm (which is
n
i
specifically for volume manager).
tofh these utilities have the
lud Both
e
a
following characteristics:
am to us
k
g the
e
They can n
install
on multiple servers (multiple nodes) if
ssoftware
a
n
b
e
rsh/rcp
is enabled between the nodes (or you can
a or ssh/scp
licit separately
b
e
choose
to
run
on each node).
u
l
k
b
a
a
l y sThey
er will install quite of few packages (30 packages for example, for
f
u
d
ab -tran VxVM 5.0), even if you choose only the required packages. These
g
include Web servers and Java environments to run the GUI backan non
ab
ends. This is many more than the minimal software required to run
VxVM in the cluster.
6-20
They guide you through entering licenses and initializing the

software at the end of the installation.

Installing the Packages by Hand
The following are the only VxVM packages required on all the nodes:
VRTSvxvm
VRTSvlic
The VxVM 4.1 and 5.0 packages are distributed on the media as .tar.gz
files. The installer and installvm utilities extract these correctly.
If you want to install VxVM without using the utilities you copy and
extract these archives manually and then install them with pkgadd on
each node. You then have to manually use vxinstall to license and
initialize the software.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

6-21
Example of installvm Screens
The installvm guides you logically through the tasks. The initial screens
look similar to the following:
Veritas Volume Manager 5.0 Installation Program
Copyright (c) 2006 Symantec Corporation. All rights reserved. Symantec, the
Symantec Logo are trademarks or registered trademarks of Symantec Corporation or
its affiliates in the U.S. and other countries. Other names may be trademarks of
their respective owners.
The Licensed Software and Documentation are deemed to be "commercial computer
software" and "commercial computer software documentation" as defined in FAR
Sections 12.212 and DFARS Section 227.7202.
y
m
om
c
Enter the system names separated by spaces on which to install VxVM: vincent
m utheo
s
ide
c
Initial system check:
e
G
t
@
n
b
e
installvm requires that ssh commands used between systems
dwithout
kuexecute
u
a
t
y
prompting for passwords or confirmations. If installvm
hangs
or
asks
for a login
( is S
n
i
password or hangs, stop installvm and run it again
with
the
ssh
configured
for
thoption.
ludthes-rsh
password free logins, or configure rsh andause
e
am to u
k
Checking ssh communication with
theo
SunOS 5.10
g ns.............................
e
n
a
Checking VxVM installation
on
vincent
.........................
not
b on ltheo
e ............................ not installed
c
a
i
Checking VxVM installation
installed
b
e
u
l
k
a erab
y
l
f
uinstalled
s
d
VxVM can b
be
without optional packages to conserve disk space.
n
a
tra
g
n
n
a
nopackages are typically installed to simplify future upgrades.
abAdditional
s
) ha
Logs for installvm are being created in /var/tmp/installvm-DTVRAi.
1)
2)
3)
Required Veritas Volume Manager packages - 839 MB required

All Veritas Volume Manager packages - 863 MB required
Storage Foundation Enterprise packages - 1051 MB required
Select the packages to be installed on all systems? [1-3,q,?] (3)
6-22

Using the clvxvm Utility

The clvxvm utility automates VxVM cluster-specific configuration,
including optional root encapsulation. The clvxvm checks that VxVM
itself is already installed and that the VxVM general configuration
(vxinstall, which is automated by the installer and installvm
utilities) is complete.
Running clvxvm initialize

If you choose this option, then clvxvm negotiates a vxio major number
and edits the /etc/name_to_major file, if necessary. If it has to modify
the file, then it tells you that you need to reboot.
s
) ha
ab
y
m
om
c
Running clvxvm encapsulate
csm Guide
e
If you choose this option, clvxvm negotiates
t major number. It
@the vxio
n
b
e
u
then encapsulates your root disk by performing
the
following
tasks:
k tud
a
y
( in aisdisk
S group named rootdg, and
ndisk
It encapsulates your boot
i
d
h
u se t
llink.
creates the bootdg
a
u
o
kamvolume
t
It gives g
different
names for the volumes containing the
e
n
s
/global/.devices/node@#
file systems on each node.
ba licen
a
b It editslethe vfstab file for this same volume, replacing the DID
u
k
a edevice
ab with the original c#t#d#s#. This allows VxVM encapsulation
r
y
l
f
bdu trans to properly recognize this is a partition on the OS disk.
a
It installs a script to reminor the bootdg on reboot.
ang non
It reboots you into a state where VxVM on your node is fully

operational. It reboots the first time using -x into non-clustered
mode because the second reboot that is performed by VxVM
automatically takes you back into the cluster.
Note VxVM itself (not our utilities) still puts in a nologging option for
the root file system when you encapsulate. This is a leftover from an old
problem that has been fixed. You must remove the nologging option
from /etc/vfstab manually and reboot one more time.

6-23
Manually Using vxdiskadm to Encapsulate the OS Disk
Use the following steps to make boot-disk encapsulation work correctly

on the cluster. You should use clvxvm encapsulate to automate all of
this, however this information is given for completeness:
1.
Give a different name for the root disk on each node. Do not use the
default disk name on each node.
The result is that the volume name for the
/global/.devices/node@# file system is different on each node.
2.
Make sure you do not enter any shared disk storage drives in the
bootdg.
s
) ha
y
m
om
c
e
csmoutGthat
uidit must
Without making this change, VxVM cannot figure
e
t
@VxVMenvolume
edit that line after the reboot to put inuthe
name.
b
d
k
tu reboot after that
ya aSsecond
4. Reboot with boot -x. VxVM arranges
(
n
which brings you back u
into
di theecluster.
his
t
l
a uallsbut one of the nodes because of the
m
5. That second reboot
fails on
a
k numbers
tofor bootdg (the conflict is really just for
g
conflict innminor
e
ba licens
the /global/.devices/node@#
volume).
a
b
e give the root password to go into single user mode.
uOn those
lnodes,
k
b
a
a
l y sUse
erthe following command to fix the problem:
f
u
d
ab -tran # vxdg reminor bootdg 50*nodeid
g
n non
a
For example, use 50 for Node 1, 100 for Node 2, and so forth. This
b
a
3.
Before you reboot manually, put back the normal

/dev/dsk/c#t#d#s# and /dev/rdsk/c#t#d#s# on the line in the
/etc/vfstab file for the /global/.devices/node@# file system.
provides a unique set of minor numbers for each node.
6.
Reboot the nodes for which you had to repair the minor numbers.
Configuring a Pre-Existing VxVM for Sun Cluster 3.2

Software
The last possibility is that you have a potential Sun Cluster 3.2 software
node where the proper version of VxVM was already installed and
initialized before the Sun Cluster environment was installed. The following
subsections describe how to provision your VxVM to deal with the
cluster-specific issues mentioned on the previous pages. They assume you
will deal with these issues before you install the Sun Cluster software.
6-24

Fixing the vxio Major Number
If you need to change the vxio major number to make it agree with the
other nodes, do the following:
1.
Manually edit the /etc/name_to_major file.
2.
Unmirror and unencapsulate the root disk, if it is encapsulated.
3.
Reboot.
4.
Reencapsulate and remirror the boot disk, if desired.
Fixing the /globaldevices Volume Name and Minor Number
s
) ha
y
m
om
c
csm Guide
e
If this is on a volume manager device:
@ ent
b
u
aksameSname
1. Rename the volume if it has(y
the
tud as the /globaldevices
in tvolume
is of any other node. You might
or /global/.devices/node@#
d
h
u
l
e
need to manuallyaedit the s
/etc/vfstab file.
m
u
a
o
k eoperation
t
2. Performg
a reminor
on the rootdg if it is using the same set
n
s
a
n
of b
minor
numbers
as
another
node. Follow the same procedure as
e
c
a
i
l
b Step 5leon page 6-24.
u
k
a erab
y
l
u nsf
d
b
a
tra
g
n
n
a
no
If you are still at a point before cluster installation, you need to have a
/globaldevices placeholder file system or at least a correctly sized
device ready for cluster installation.
ab

6-25
Creating Shared Disk Groups and Volumes

The examples in this section are not trying to exhaustively cover all the
possible ways of initializing disks and creating volumes. Rather, these are
demonstrations of some simple ways of initializing disks, populating disk
groups, and creating the mirrored configurations described earlier in the
module.
Listing Available Disks
s
) ha
The following command lists all of the disks visible to VxVM:
y
m
om
c
ide
csm Ginvalid
u
e
@ ent invalid
b
u
invalid
k tud
a
y
invalid
n ( this S
i
d
invalid
lu se
a
invalid
am to u
k
invalid
g
e
n
s
invalid
ba licen
a
invalid
b
e
u
l
k
invalid
a erab
y
l
invalid
f
u ns
d
b
invalid
a
tra
g
n
invalid
n
o
a
n
b
invalid
a
vincent:/# vxdisk -o alldgs list
DEVICE
TYPE
DISK
c0t0d0s2
auto:sliced
rootdg_1
c0t1d0s2
auto:none
c1t0d0s2
auto:none
c1t1d0s2
auto:none
c1t2d0s2
auto:none
c1t3d0s2
auto:none
c1t8d0s2
auto:none
c1t9d0s2
auto:none
c1t10d0s2
auto:none
c1t11d0s2
auto:none
c2t0d0s2
auto:none
c2t1d0s2
auto:none
c2t2d0s2
auto:none
c2t3d0s2
auto:none
c2t8d0s2
auto:none
c2t9d0s2
auto:none
c2t10d0s2
auto:none
-
GROUP
rootdg
-
STATUS
online
online
online
online
online
online
online
online
online
online
online
online
online
online
online
online
online
invalid
invalid
The command vxdisk list with the option -o alldgs scans every disk
to see if there is any disk group information, even about disk groups not
currently imported on this node. In this output you can see that there is
no disk group information configured on other disks other than the first
disk.
Disks with the invalid state are not yet initialized by VxVM.
6-26

Initializing Disks and Putting Them into a New Disk

Group
One of the simplest ways to do this is using the dialogue presented by the
vxdiskadd command. It guides you through initializing disks and putting
them in existing or new groups.
# vxdiskadd c1t0d0 c1t1d0 c2t0d0 c2t1d0
.
Verifying Disk Groups Imported on a Node
s
) ha
y
m
om
# vxdg list
c
NAME
STATE
ID
sm uide
c
e
G
rootdg
enabled
1152724944.22.vincent
t
@
n
b
nfsdg
enabled,cds
1152725899.26.vincent
ku tude
a
y
n ( this S
i
d
u contains
lgroup
e the name of the node on which it
Note The ID of a disk
a
s
m
u
was created. Later,
kayou might
to see the same disk group with the same ID
g
e
imported a
b onnalidifferent
ens node.
c
a
b
u
le
k
b
# vxdisk list
a
a
y TYPEfer
DEVICE ul
DISK
GROUP
STATUS
s
d
n
b
c0t0d0s2
rootdg_1
rootdg
online
a
traauto:sliced
g
n
c0t1d0s2
auto:none
online invalid
n
o
a
n
b
auto:cdsdisk
nfs1
nfsdg
online
a c1t0d0s2
The following command shows which groups are imported:
c1t1d0s2
c1t2d0s2
c1t3d0s2
c1t8d0s2
c1t9d0s2
c1t10d0s2
c1t11d0s2
c2t0d0s2
c2t1d0s2
c2t2d0s2
c2t3d0s2
c2t8d0s2
c2t9d0s2
c2t10d0s2
c2t11d0s2
auto:cdsdisk
auto:none
auto:none
auto:none
auto:none
auto:none
auto:none
auto:cdsdisk
auto:cdsdisk
auto:none
auto:none
auto:none
auto:none
auto:none
auto:none
nfs3
nfs2
nfs4
-
nfsdg
nfsdg
nfsdg
-
online
online
online
online
online
online
online
online
online
online
online
online
online
online
online

invalid
invalid
invalid
invalid
invalid
invalid
invalid
invalid
invalid
invalid
invalid
invalid
6-27
The following shows the status of an entire group where no volumes have
been created yet:
# vxprint -g nfsdg
TY NAME
ASSOC
dg nfsdg nfsdg
KSTATE
-
LENGTH
-
PLOFFS
-
STATE
-
TUTIL0
-
PUTIL0
-
dm
dm
dm
dm
71061376
71061376
71061376
71061376
nfs1
nfs2
nfs3
nfs4
c1t0d0s2
c2t0d0s2
c1t1d0s2
c2t1d0s2
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
6-28

Building a Simple Mirrored Volume
The following example shows building a simple mirror, with a subdisk

from a disk in one controller mirrored with one from another controller.
# vxassist -g nfsdg make nfsvol 100m layout=mirror nfs1 nfs2
# vxprint -ht -g nfsdg
// with -ht, vxprint prints out a long legend, which is omitted for
brevity.
.
.
dg nfsdg
default
default
30000
1152725899.26.vincent
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
d
v nfsvol
ENABLED ACTIVE y204800
fsgen
ak StuSELECT
(
pl nfsvol-01 nfsvol
ENABLED ACTIVE
204800
CONCAT
RW
n
i
is
d
h
t
u
sd nfs1-01
nfsvol-01 nfs1
0l
c1t0d0
ENA
e204800 0
aACTIVE
s
m
u
pl nfsvol-02 nfsvol
ENABLED
204800
CONCAT
RW
ka e0 to
g
sd nfs2-01
nfsvol-02 nnfs2
204800
0
c2t0d0
ENA
s
a
n
b
e
a
lic
b
e
u
l
k raaMirrored
b
a
Building
Striped Volume (RAID 0+1)
y
l
e
f
u
s
bd tran
a
g on- The following example rebuilds the mirror. Each plex is striped between
n
a
n
two disks in the same array so mirroring is still across controllers.
ab
dm
dm
dm
dm
nfs1
nfs2
nfs3
nfs4
c1t0d0s2
c2t0d0s2
c1t1d0s2
c2t1d0s2
auto
auto
auto
auto
65536
65536
65536
65536
71061376
71061376
71061376
71061376
# vxassist -g nfsdg remove volume nfsvol

# vxassist -g nfsdg make nfsvol 100m layout=mirror-stripe mirror=ctlr
# vxprint -ht -g nfsdg
.
.
dg nfsdg
default
default
30000
1152725899.26.vincent
dm
dm
dm
dm
c1t0d0s2
c2t0d0s2
c1t1d0s2
c2t1d0s2
auto
auto
auto
auto
65536
65536
65536
65536
71061376
71061376
71061376
71061376
ENABLED
ENABLED
ACTIVE
ACTIVE
204800
204800
SELECT
STRIPE
nfs1
nfs2
nfs3
nfs4
v nfsvol
pl nfsvol-01
nfsvol

fsgen
2/128
RW
6-29

sd
sd
pl
sd
sd
nfs1-01
nfs3-01
nfsvol-02
nfs2-01
nfs4-01
nfsvol-01
nfsvol-01
nfsvol
nfsvol-02
nfsvol-02
nfs1
nfs3
ENABLED
nfs2
nfs4
0
0
ACTIVE
0
0
102400
102400
204800
102400
102400
0/0
1/0
STRIPE
0/0
1/0
c1t0d0
c1t1d0
2/128
c2t0d0
c2t1d0
ENA
ENA
RW
ENA
ENA
Building a Striped Mirrored Volume (RAID 1+0)

The final example rebuilds the volume as a stripe of two mirrors. As
mentioned earlier in this module, this layout is often preferred for faster
resynchronization on disk failure and greater resiliency.
s
) ha
# vxassist -g
# vxassist -g
# vxprint -ht
.
.
dg nfsdg
nfsdg remove volume nfsvol

nfsdg make nfsvol 100m layout=stripe-mirror mirror=ctlr
-g nfsdg
pl
sd
pl
sd
nfsvol-P01
nfs1-02
nfsvol-P02
nfs2-02
nfsvol-L01
nfsvol-P01
nfsvol-L01
nfsvol-P02
ENABLED
nfs1
ENABLED
nfs2
ACTIVE
0
ACTIVE
0
102400
102400
102400
102400
CONCAT 0
CONCAT
0
c1t0d0
c2t0d0
v
pl
sd
pl
sd
nfsvol-L02
nfsvol-P03
nfs3-02
nfsvol-P04
nfs4-02
nfsvol-L02
nfsvol-P03
nfsvol-L02
nfsvol-P04
ENABLED
ENABLED
nfs3
ENABLED
nfs4
ACTIVE
ACTIVE
0
ACTIVE
0
102400
102400
102400
102400
102400
SELECT
CONCAT
0
CONCAT
0
fsgen
RW
c1t1d0
ENA
RW
c2t1d0
ENA
y
m
om
c
csm Guide
e
@ ent
default
default 30000
1152725899.26.vincent
b
u
ak Stud
y
(
dm nfs1
c1t0d0s2 auto
65536 in 71061376
is -d
h
t
dm nfs2
c2t0d0s2 auto
65536
71061376
u
e
al
dm nfs3
c1t1d0s2 auto am
65536 us
71061376 k 65536
to 71061376 dm nfs4
c2t1d0s2 auto
g
e
ban licens
a
v nfsvol
- ub
ENABLED
ACTIVE
204800
SELECT nfsvol-0 fsgen
e
l
k
b
pl nfsvol-03 ynfsvol
RW
a era ENABLED ACTIVE 204800 STRIPE 2/128
l
f
u
sv nfsvol-S01
nfsvol-03
nfsvol-L01
1
102400
0/0
2/2
ENA
s
bd trnfsvol-03
an
a
sv nfsvol-S02
nfsvol-L02 1
102400
1/0
2/2
ENA
g
n
n
no abva nfsvol-L01
ENABLED ACTIVE
102400 SELECT fsgen
6-30

RW
ENA
RW
ENA
Examining Hot Relocation
Examining Hot Relocation

Hot relocation is controlled by the vxrelocd daemon, which is started by
default after you initialize the volume manager.
When a disk that is a component of a mirrored volume breaks, VxVM
looks for space to substitute for the broken half of the mirror. If one large
disk breaks, VxVM can concatenate many new subdisks from other disks
in the diskgroup to recreate the broken plex. It never uses any disk space
from disks in the surviving plex.
The SPARE and NOHOTUSE Flags
s
) ha
y
m
om
c
csm Guide
SPARE
e
t disks to use for
@
n
b
Disks marked with the SPARE flag
are
the
preferred
e
u
k be used
astill
tudto build normal volumes.
hot relocation. These disks can
y
(
S
in this
d
NOHOTUSE
u
al use
m
Disks marked
ka withe the
to NOHOTUSE flag are excluded from
g
consideration
ban licenforshot relocation.
a
b following
u
le example, disk nfs2 is set as a preferred disk for hot
k
In
the
b
a
a
l y relocation,
er and disk nfs1 is excluded from hot relocation usage:
f
u
s
d
ab -tran# vxedit -g nfsdg set spare=on nfs2
g
n non
a
# vxedit -g nfsdg set nohotuse=on nfs1
b
a
Under normal hot relocation, disks can be marked with one of the two
following flags, but not both:
The flag settings are visible in the output of the vxdisk list and
vxprint commands.

6-31
Registering VxVM Disk Groups

After you create a new VxVM disk group and volumes, you must
manually register the disk group using either the clsetup utility or the
cldevicegroup create command for the disk group to be managed by
the cluster.
When a VxVM disk group is registered in the Sun Cluster software
environment, it is referred to as a device group.
Until a VxVM disk group is registered, the cluster does not detect it.
While you can build volumes and perform all normal VxVM
administration tasks, you are not able to use the volume manager devices
in the cluster.
s
) ha
y
m
om
c
If you create a new volume or delete a volume in a VxVM

e that
m diskuigroup
d
s
c
is already registered with the cluster, you must synchronize
e t Gthe disk
@
device group by using clsetup. Such configuration
changes
include
n
b
e
u
d
k
adding or removing volumes and changing
u owner, or
(ya isthe
Stgroup,
permissions of existing volumes.in
Synchronization
after volumes are
d
h
t
u
l the sglobal
created or deleted ensuresathat
e namespace is in the correct state.
m
u
ka shows
toa disk group that is known to VxVM, but
g
e
The following n
example
ns cldg is just the short form of
ba to lthe
e
not yet known
cluster.
c
a
i
b
e
cldevicegroup.
u
l
k
a erab
y
l
# vxdg list
u nsf
d
b
NAME a
ID
ra
tSTATE
g
n
n
rootdg
1152724944.22.vincent
a
no enabled
abnfsdg
enabled,cds
1152725899.26.vincent
# cldevicegroup show
# cldg status
Cluster Device Groups ===
--- Device Group Status --Device Group Name
-----------------
6-32
Primary
-------
Secondary
---------
Status
------

Using the cldg create Command to Register Disk

Groups
The following command registers a VxVM disk group as a cluster device
group. You must type this command on the node currently importing the group.
# cldg create -t vxvm -n vincent,theo -p preferenced=true \
-p failback=false nfsdg
This command uses the following parameters:
The -n option (node list) should list all the nodes and only the nodes
physically connected to the disk groups disks.
s
) ha
ab
y
m
om
c
e
idnode
csa m
u
The failback=true/false affects whether
preferred
takes
e
G
t
@
n
b
back its device group when it joins
e The default value is
u the cluster.
d
k
u
a
false. If it is true, it only works
if
preferenced
is also set to true.
t
y
(
S
n
di e this
u
l
a us
m
a
Using the clsetup
Command
to Register Disk Groups
k e to
g
an cens
b
a
li can guide you through the previous options in a
Thebclsetup utility
e
u
l
k
environment. From the main menus, selecting Menu
a menu-driven
a5b(Device
r
y
l
e
f
Option
Groups and Volumes) gives you the following submenu:
u ns
d
b
a
aDevice-trGroups
***
Menu ***
g
n
n
a
no
The preferenced=true/false affects whether the nodelist indicates

an order of failover preference. On a two node cluster this option is
only meaningful if failback is true.

1)
2)
3)
4)
5)
6)
7)
8)
Register a VxVM disk group as a device group

Synchronize volume information for a VxVM device group
Unregister a VxVM device group
Add a node to a VxVM device group
Remove a node from a VxVM device group
Change key properties of a device group
Set a VxVM disk group as a local disk group
Reset a local disk group to a VxVM disk group
?) Help

6-33
Viewing and Controlling Registered Device Groups
Use the cldg show command to show the properties of a device group (or
all device groups). Use cldg status to see the status.
# cldg show nfsdg
Device Groups ===
Device Group Name: nfsdg
Type:
failback:
Node List:
preferenced:
numsecondaries:
diskgroup name:
VxVM
false
vincent, theo
true
1
nfsdg
s
) ha
y
m
om
c
csm Guide
# cldg status
e
@ ent
b
u
ak Stud
y
(
in this
d
u
--- Device Group Status --al use
m
ka Secondary
to
g
Device Group Name
Primary
Status
e
n ens--------a
b
---------------------------c
a
i
l
b
nfsdg
theo
Online
u vincent
le
k
b
a
a
l y sfer
u
d
an By default, even if there are more than two nodes in the node list
ab -trNote
g
n non for a device group, only one node shows up as secondary. If the primary
a
b
a
fails, the secondary becomes primary and another (spare) node becomes
secondary.
The numsecondaries=# parameter is used on the cldg command line to

allow more than one secondary.
When VxVM disk groups are registered as Sun Cluster software device
groups and have the status Online, never use the vxdg import and vxdg
deport commands to control ownership of the disk group. This causes
the cluster to treat the device group as failed.
Instead, use the following command syntax to control disk group
ownership:
# cldg switch -n node_to_switch_to nfsdg
6-34

Managing VxVM Device Groups

The cldevicegroup (cldg) command can be used to perform clusterspecific changes to VxVM device groups.
Resynchronizing Device Groups

After a VxVM device group is registered with the cluster, it must be
resynchronized any time a new volume is created or deleted in the disk
group. This instructs the Sun Cluster software to scan the disk group and
build and remove the appropriate global device files:
s
) ha
y
m
om
c
Making Other Changes to Device Groups

csm Guide
e
@ ent
b
u
The properties of existing VxVM y
device
ak groups
tudcan also be changed. For
(
S
example, the failback and preferenced
in this properties of a group can be
d
u
modified after it is registered.
al use
m
o
ka -p
tfailback=false
# cldg set -p preferenced=false
nfsdg
g
e
n
s
ba licen
a
b If you
u
le do this command on the node that is not primary for the
k
Note
b
a
a
l y device
er group, you may get a warning that makes it seem like the
f
u
s
d
ab -trancommand failed. It is just a warning, and the command succeeds.
g
an non
# cldg sync nfsdg
ab
Putting a Device Group Offline and Back Online

You can take a VxVM device group out of service, as far as the cluster is
concerned, for emergency repairs.
To put the device group offline all of the VxVM volumes must be unused
(unmounted or otherwise not open). You then issue the following
command, from any node:
# cldg offline nfsdg
You will rarely take a device group out of service because almost all
repairs can still be done while the device group is in service.

6-35

Type the following command to put the device group online on the first
node in the list:
# cldg online nfsdg

Alternatively, you can choose exactly which node will become the device
group primary.
# cldg online -n theo nfsdg
Note You do not need to perform both commands. One or the other will
do.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
6-36

Using Global and Failover File Systems on VxVM Volumes

Sun Cluster 3.2 supports running data services on the following categories
of file systems:
Global file systems Accessible to all cluster nodes simultaneously,

even those not physically connected to the storage
Failover file systems Mounted only on the node running the failover
data service, which must be physically connected to the storage
UFS and VxFS are two file system types that can be used as either global
or failover file systems. These examples and the exercises assume you are
using UFS.
s
) ha
y
m
m
co fileesystems
Note There is a convention that the mount point form

global
be
d
s
i
c is just
somewhere underneath the /global directory.eThis
ua convention.
G
t
@
b den
u
k
ya Stu
(
n
di e this
u
l
a us
Creating File Systems
m
a
k e to
g
The distinction
ban libetween
ens a global and a failover file system is not made at
c
a
btime oflefile system creation. Use newfs as normal to create a UFS file
the
u
k
a system
aforb the cluster on a volume in a registered disk group:
r
y
l
e
f
bdu tran#snewfs /dev/vx/rdsk/nfsdg/nfsvol
a
g onn
a
n
ab
Mounting File Systems
For UFS or VxFS, the distinction between a global and a failover file
system is made in the /etc/vfstab mount-at-boot and options columns.
A global file system entry should look similar to the following, and it
should be identical on all nodes (including nodes not physically
connected to the storage):
/dev/vx/dsk/nfsdg/nfsvol /dev/vx/rdsk/nfsdg/nfsvol /global/nfs ufs 2 yes global
A local failover file system entry looks like the following, and it should be
identical on all nodes who may run services which access the file system
(can only be nodes physically connected to the storage):
/dev/vx/dsk/nfsdg/nfsvol /dev/vx/rdsk/nfsdg/nfsvol /localnfs ufs 2 no -

6-37
Note The logging option is the default for all OS versions supported
with Sun Cluster 3.2. You do not need to specify it explicitly.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
6-38

Mirroring the Boot Disk With VxVM
Mirroring the Boot Disk With VxVM

If you have encapsulated your boot drive with VxVM, mirroring the boot
drive is a simple procedure that can be undertaken any time while the
cluster is online, without any reboot. To mirror the boot drive, complete
the following steps:
1.
Initialize and add another local drive to bootdg, if this has not been
done yet. This drive must be at least as big as the boot drive. Give
the drive the VxVM logical name rootmir, for example. This is done
as part of the vxdiskadd dialogue.
Make sure that you do not make the disk a CDS disk (choose
sliced).
s
) ha
y
m
omvolumes
2. Use the vxmirror command, or the vxdiskadm Mirror
on a
c
e
m
disk option to mirror all the volumes on the
drive.
csboot G
uid
e
@ ent
b
u
Caution Do not mirror each volume
This
ud withbe vxassist.
akseparately
tcannot
y
(
S
leaves you in a state that the mirrored
drive
unencapsulated.
in this the second disk you will not be
dunencapsulate
That is, if you ever need lto
u
a use
able to.
m
ka e to
g
nsdrive correctly as shown, the second drive can also
If you mirror
root
ban the
e
c
a
li That is because correct restricted subdisks are laid
bunencapsulated.
be
e
u
l
k
a down
onto
ab the new drive. These can later be turned into regular Solaris
r
y
l
e
f
bdu tranOSs partitions because they are on cylinder boundaries.
a
ang non
# vxmirror -g bootdg rootdg_1 rootmir
# vxdiskadd c0t8d0
ab
.
.
3.
The previous command also creates aliases for both the original root
partition and the new mirror. You need to manually set your bootdevice variable so you can boot off both the original and the mirror.
# eeprom|grep vx
devalias vx-rootdg_1 /pci@1f,4000/scsi@3/disk@0,0:a
devalias vx-rootmir /pci@1f,4000/scsi@3/disk@8,0:a
# eeprom boot-device="vx-rootdg_1 vx-rootmir"
4.
Verify that the system has been instructed to use these device aliases
on boot:
# eeprom|grep use-nvramrc
use-nvramrc?=true

6-39
Exercise: Configuring Volume Management

Task 1 Selecting Disk Drives
Task 2 Using pkgadd to Install and Initialize VxVM Software (on

All Storage Nodes)
Task 3 Using clvxvm to Verify the vxio Major Number
Task 4 Adding vxio on Any Non-Storage Node on Which You

Have Not Installed VxVM
Task 5 Rebooting All Nodes
s
) ha
y
m
Task 7 Configuring webdg

om
c
Task 8 Registering Demonstration Disk Groups

csm Guide
e
@ ent
Task 9 Creating a Global nfs File System
b
u
akSystem
tud
Task 10 Creating a Global web
File
y
(
S
inSystems
d
his
t
Task 11 Testing Global
File
u
l
e
a us
m
a
Task 12 Managing
Disk
k e toDevice Groups
g
n ensViewing and Managing VxVM Device Groups
Task 13
ba(Optional)
a
lic Manager
Using
SuneCluster
b
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
a
no
Task 6 Configuring nfsdg
ab
6-40

Preparation
Record the location of the VxVM software you will install during this
exercise.
Location: _____________________________
During this exercise, you create two data service disk groups. Each data
service disk group contains a single mirrored volume. Encapsulating the
boot disk is an optional exercise at the end. The setup is shown in
Figure 6-7.
Node 1
Boot Disk
Boot Disk
bootdg
bootdg
Node 2
c0
s
) ha
y
m
ccccc
om
c1
c2
c2
c
c c1
(Encapsulated (bootdg->rootdg): Optional)

csm Guide
e
t
A
A
Bb@
B
n
e
u
ak Stud
y
(
Disk Group
Quorum Disk
n this
and Volumesudi
al use
m
Array A Disk
ka e to Array B Disk
g
an cens
b
a
Array B Disk
bArray A Disk le li
u
k
a eArray
aAb
r
y
Array B
l
f
u
s
d
ab -tran
g
n non
a
b
a
c0
nfsdg
nfsvol
webdg
webvol
Figure 6-7
Configuring Volume Management
Note During this exercise, when you see italicized names, such as
IPaddress, enclosure_name, node1, or clustername embedded in a
command string, substitute the names appropriate for your cluster.

6-41
Task 1 Selecting Disk Drives
Before proceeding with this exercise, you must have a clear picture of the
disk drives that are used throughout this course. You may have one or
more disks configured as quorum devices (or you may have a quorum
server quorum device). It is perfectly acceptable to use a quorum disk as
one of your disk group drives as well. In this exercise, you must identify
the boot disk and two disks in each storage array for use in the two
demonstration disk groups, nfsdg and webdg.
Perform the following step to select disk drives:
s
) ha
Use the cldev list -v, clq status, and df -h commands to

identify and record the logical addresses of disks for use during this
exercise. .
y
m
om
c
Table 6-1
csm Guide
e
@ entDID # (d#)
c#t#d#
b
u
ak Stud
y
(
Boot disk
in this
d
u
al use
Quorum disk(s)
m
ka e to
g
an cens
b
a
nfsdgbdisk 1
li
e
u
l
k
anfsdg edisk
a2b(different controller
r
y
l
f
ns m disk 1, if possible)
bdu trafrom
a
g onn
a
n
ab
webdg disk 1
webdg disk 2 (different controller

from disk 2, if possible)
6-42

Task 2 Using pkgadd to Install and Initialize VxVM

Software (on All Storage Nodes)
In this lab, just to save time, you add VxVM packages manually, rather
than using installer or installvm. This is completely supported.
You need to install VxVM only on nodes connected to the shared storage.
If you have a non-storage node (Pair +1 cluster), you do not need to install
VxVM on that node.
Perform the following steps on all cluster nodes that are connected to
shared storage:
s
) ha
y
m
# cd vxvm_sw_dir/volume_manager/pkgs
om
c
cs\m Guide
# cp VRTSvlic.tar.gz VRTSvxvm.tar.gz
e
@ ent
VRTSvmman.tar.gz /var/tmp
b
u
ak Stud
y
(
# cd /var/tmp
in this
d
u
# gzcat VRTSvlic.tar.gz
xf al use|| tar
m
# gzcat VRTSvxvm.tar.gz
tar
xf a
o
k
t
# gzcat
| tar xf ngVRTSvmman.tar.gz
se
a
n
b
e
a the new
2. b Add
lic VxVM packages. Answer yes to all questions
e
u
l
k (including
b conflicts in directories).
a
a
r
y
ul nsfe# pkgadd -d /var/tmp VRTSvlic VRTSvxvm VRTSvmman
d
b
a
tra 3. Add the VxVM 5.0 patches for MP3 and RP1 in that order.
g
n
n
a
no
1.
ab
Spool and expand the VxVM packages.
At the time of writing the course, the MP3 patch is 122058-11 and the
RP1 patch is 139352-02.
# cd location_of_vxvm_patches_for_course
# patchadd -M . 122058-11 139352-02
4.
Run vxinstall to initialize VxVM:

a.
Enter the license information as directed by your instructor
b.
Answer No when asked about enclosure-based naming.
c.
Answer No when asked about a default disk group.

6-43
Task 3 Using clvxvm to Verify the vxio Major Number
Run the following commands on all nodes on which you have installed
VxVM:
# clvxvm initialize
Note At the time of writing this course, this fails when you have ZFS
root. Please manually verify that the vxio major number in
/etc/name_to_major is the same on all cluster nodes. If it is not, edit the
file so it is the same on all nodes. Be careful that you dont conflict with
any numbers already in a file (you can just choose a major number higher
than that in any of the files).
s
) ha
y
m
om
c
e
idon
Task 4 Adding vxio on Any Non-Storage
csmNode
u
e
G
t
@
Which You Have Not Installed VxVM
n
b
ku tude
a
y
( is S
n which
i
If you have a non-storage nodedon
hyou have not installed VxVM,
t
u
l
e
edit /etc/name_to_majoraand addsa line at the bottom containing the
uon the storage nodes:
am as tused
same vxio major number
o
k
ng ense
a
vxio
same_major_number_as_other_nodes
b
a
lic
b
e
u
l
k rab
a
y
e
ul Note
f Be careful and assure that on the non-storage node the major
s
d
n
b
a
a
is not already used by some other device driver on another line in
trnumber
g
n
n
o
a
the file. If it is, you will have to change the vxio major number on all
n
ab
nodes, picking a number higher than currently exists on any node.
Task 5 Rebooting All Nodes

Reboot all your nodes. In the real production environment, you would
always do this one node at a time to maintain the HA of any cluster
services that might already be configured.
Note Technically the storage nodes would proceed correctly even
without a reboot. A non-storage node with the manually created or fixed
vxio entry absolutely requires a reboot here. In addition Sun Cluster
manager will fail to operate properly unless you reboot here.
6-44

Task 6 Configuring nfsdg
Perform the following steps to configure two demonstration disk groups,

with each containing a single mirrored volume:
1.
On Node 1, create the nfsdg disk group with your previously

selected logical disk addresses.
# vxdiskadd c#t#d# c#t#d#

Which disk group [<group>,none,list,q,?] (default: none) nfsdg
Create a new group named nfsdg? [y,n,q,?] (default: y) y
Create the disk group as a CDS disk group? [y,n,q,?] (default: y) y
Use default disk names for these disks? [y,n,q,?] (default: y) y
Add disks as spare disks for nfsdg? [y,n,q,?] (default: n) n
Exclude disks from hot-relocation use? [y,n,q,?] (default: n) y
Add site tag to disks? [y,n,q,?] (default: n) n
.
.
Continue with operation? [y,n,q,?] (default: y) y
.
Do you want to use the default layout for all disks being initialized?
[y,n,q,?] (default: y) y
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
an cens
b
a
Caution
If you
li are prompted about encapsulating the disk, you should
b
e
u
l
k
If you are prompted about clearing old disk usage status from a
a replyerno.
abtraining
y
l
previous
class, you should reply yes. In your work environment,
f
u ns
d
b
careful when answering these questions because you can destroy
a
tra be
g
n
n
critical
data or cluster node access information.
o
a
n
b
a
2.
Verify the status of the disk group and the names and ownership of
the disks in the nfsdg disk group.
# vxdg list
# vxdisk list
3.
On Node 1, verify that the new nfsdg disk group is globally linked.
# ls -l /dev/vx/dsk/nfsdg
lrwxrwxrwx
1 root
root
12 Dec
/dev/vx/dsk/nfsdg ->
/global/.devices/node@1/dev/vx/dsk/nfsdg
4.
2008
On Node 1, create a 500-Mbyte mirrored volume in the nfsdg disk

group.
# vxassist -g nfsdg make nfsvol 500m layout=mirror

6-45

5.
Use the vxprint command on both nodes. Notice that Node 2 does
not see the disk groups created and still imported on Node 1.
6.
Issue the following command on the node that currently owns the
disk group:
# newfs /dev/vx/rdsk/nfsdg/nfsvol
It should fail with a no such device or address error.
Task 7 Configuring webdg

1.
s
) ha
On Node 1, create the webdg disk group with your previously

selected logical disk addresses. Answers in the dialog can be similar
to those in the previous task.
y
m
om
c
# vxdiskadd c#t#d# c#t#d#

csm Guide
e
Which disk group [<group>,none,list,q,?]
@ ent
b
(default: none) webdg
u
ak volume
tud in the webdg disk
2. On Node 1, create a 500-Mbyte(y
mirroredS
in this
d
group.
u
e
al make
s
m
u
# vxassist -g
webdg
webvol 500m layout=mirror
a
o
k
t
ng ecommand
se on both nodes. Notice that Node 2 does
3. Use the a
vxprint
n
b
asee the disk
not
lic groups created and still imported on Node 1.
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
a
no
ab
6-46

Task 8 Registering Demonstration Disk Groups
Perform the following steps to register the two new disk groups with the
Sun Cluster framework software:
1.
On Node 1, use the cldg create utility to manually register the

nfsdg disk group.
# cldg create -t vxvm -n node1,node2 \

-p preferenced=true -p failback=true nfsdg
Note Put the local node (Node 1) first in the node list.
as
h
2. On Node 1, use the clsetup utility to register the webdg disk
group.
)
y
m
# clsetup
om
c
3. From the main menu, complete the followingsm

ide
c steps:
u
e
G
a. Select option 5, Device groups and
volumes.
t
@
n
b
e 1, Register a VxVM
doption
ku select
b. From the device groupsymenu,
u
a
t
S
disk group as a device
n (group.
s
i
i
d
h
t
luquestions
e
a
4. Answer the clsetup
as follows:
s
m
u
a
o
k VxVM
tdisk group you want to register? webdg
Name ofgthe
e
n
s
a want
Dobyou
ento configure a preferred ordering (yes/no)
c
a
i
l
b [yes]?
e yes
u
l
k
b
ra both nodes attached to all disks in this group
ya feAre
l
u
s
bd tran (yes/no) [yes]? yes
a
ang non
ab
Note Read the previous question carefully. On a cluster with more than
two nodes, you will be asked if all nodes are connected to the disks. In a
Pair +1 configuration, for example, you need to answer no and respond
when you are asked which nodes are connected to the disks.
Which node is the preferred primary for this device
group? node2
Enable "failback" for this disk device group (yes/no)
[no]? yes
5.
From either node, verify the status of the disk groups.

# cldg status

6-47
Perform the following steps on Node 1 to create and mount a

demonstration file system on the nfsdg disk group volume:
1.
On Node 1, create a file system on nfsvol in the nfsdg disk group.

# newfs /dev/vx/rdsk/nfsdg/nfsvol
2.
On all nodes, including any non-storage nodes, create a global mount

point for the new file system.
# mkdir /global/nfs
On all nodes, add a mount entry in the /etc/vfstab file for the new
file system with the global mount option.
s
) ha
y
m
om
c
csm Guide
e
Note Do not use the line continuation character
@ (\)einntthe vfstab file.
b
u
ak fileSsystem.
tud
y
(
3. On Node 1, mount the /global/nfs
in this
d
u
# mount /global/nfs
al use
m
a systemtois mounted and available on all nodes.
4. Verify that thekfile
g
n ense
a
# mount
b
a /global/nfs
lic
#bls
e
u
l
k lost+found
a
ab
r
y
l
e
f
bdu trans
a
g Task
n- 10 Creating a Global web File System
n
o
a
n
ab
/dev/vx/dsk/nfsdg/nfsvol /dev/vx/rdsk/nfsdg/nfsvol \
Perform the following steps on Node 1 to create and mount a

demonstration file system on the webdg disk group volume:
1.
On Node 2, create a file system on webvol in the webdg disk group.

# newfs /dev/vx/rdsk/webdg/webvol
2.
On all nodes, create a global mount point for the new file system.
# mkdir /global/web
6-48


3.
On all nodes, add a mount entry in the /etc/vfstab file for the new
/dev/vx/dsk/webdg/webvol /dev/vx/rdsk/webdg/webvol \
/global/web ufs 2 yes global
Note Do not use the line continuation character (\) in the vfstab file.
4.
On Node 1, mount the /global/web file system.

# mount /global/web
5.
Verify that the file system is mounted and available on all nodes.
# mount
# ls /global/web
lost+found
s
) ha
y
m
om
c
csm Guide
e
Task 11 Testing Global File Systems
@ ent
b
u
ak theSgeneral
tud behavior of globally
y
(
Perform the following steps to confirm
in Cluster
dSun
his3.2 software environment:
available file systems in lthe
t
u
e
a us
am into
1. On Node 2,kmove
tothe /global/nfs file system.
g
e
# cdan
ns
b /global/nfs
e
c
a
li1, try to unmount the /global/nfs file system (umount
b On Node
2.
e
u
l
k
a e/global/nfs).
ab
You should get an error that the file system is busy.
r
y
l
f
u
s
bd tran3. On Node 2, move out of the /global/nfs file system (cd /) and try
a
g onto unmount it again on Node 1.
n
a
n
b
a
4.
Mount the /global/nfs file system again on Node 1.
5.
Try unmounting and mounting /global/nfs from all nodes.
Task 12 Managing Disk Device Groups

In the Sun Cluster 3.2 software environment, VERITAS disk groups
become cluster device groups when they are registered. In most cases, they
should not be managed using VERITAS commands. Some administrative
tasks are accomplished using a combination of Sun Cluster and VERITAS
commands. Common tasks are:
Adding volumes to a device group
Removing volumes from a device group

6-49
Adding a Volume to a Disk Device Group
Perform the following steps to add a volume to an existing device group:

1.
Make sure the device group is online (to the Sun Cluster software).
# cldg status
2.
On the node that is primary for the device group, create a 100-Mbyte
test volume in the nfsdg disk group.
# vxassist -g nfsdg make testvol 100m layout=mirror
3.
Verify the status of the volume testvol.

# vxprint -g nfsdg testvol
4.
s
) ha
Prove that the new volume can not be used in the cluster (you will
get an error message):
y
m
# newfs /dev/vx/rdsk/nfsdg/testvol
om
c
5. Synchronize the changes to the nfsdg disk group

configuration,
ide and
csm
u
e
G
prove that the volume is now usable:
t
@
n
b
ku tude
# cldg sync nfsdg
a
y
# newfs /dev/vx/rdsk/nfsdg/testvol
n ( this S
i
d
lu se
a
u
am
Removing a Volume
Fromto
a Disk Device Group
k
ng ense
a
b
a a volume
To remove
lic from a disk device group, perform the following
b
e
u
l
steps
k on the
bnode that currently has the related disk group imported:
a
a
r
y
fe
ul 1. nsUnmount
any file systems that are related to the volume.
d
b
a
a
r
t
ang non 2. On Node 1, remove the test volume, testvol, from the nfsdg disk
ab
group.
# vxassist -g nfsdg remove volume testvol

3.
Synchronize the changes to the nfsdg disk group configuration:

# cldg sync nfsdg
6-50

Migrating Device Groups

You use the cldg show command to determine current device group
configuration parameters. Perform the following steps to verify device
group behavior:
1.
Verify the current demonstration device group configuration.

# cldg show
2.
From either node, switch the nfsdg device group to Node 2.

# cldg switch -n node2name nfsdg
Type the init 0 command to shut down Node 1.
s
) ha
y
m
m
o
c
with how
4. Disable the device group failback feature. It is m
inconsistent
e
d
s
i
u be configured
the device groups work with the applications
will
ec that
G
t
@
in Modules 9 and 10:
b den
u
k
tu
# cldg set -p failback=false
ya nfsdg
(
S
n
# cldg set -p failback=false
di e thiswebdg
u
l
# cldg show
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
3.
Boot Node 1. The nfsdg disk group should automatically migrate

back to Node 1. Verify this with cldg status from any node.
ab

6-51
Task 13 (Optional) Viewing and Managing VxVM

Device Groups Using Sun Cluster Manager
In this task, you view and manage VxVM device groups through Sun
Cluster Manager. Perform the following steps on your administration
workstation or display station.
1.
In a Web browser, log in to Sun Java Web Console on any cluster

node:
2.
Enter the Sun Cluster Manager Application.
3.
Open up the arrow pointing to the Storage folder on the left.
4.
From the subcategories revealed for Storage, click Device Groups.
s
) ha
y
m
om for
5. You will see error messages about the singleton device
groups
c
each DID. This is normal.

csm Guide
e
tthe bottom of the
@ (near
6. Click the name of your VxVM deviceu
groups
n
b
e
list).
ak Stud
y
(
in tothswitch
is the primary for the device
7. Use the Switch Primaries
button
d
u
l
e
a us
group.
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab
6-52

Exercise Summary
Exercise Summary

Experiences
Interpretations
Conclusions
Applications
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

6-53
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
Module 7
Using Solaris Volume Manager With Sun

Cluster Software
s
) ha
Objectives
ab
y
m
om
c
m uide
cspartitioning
Describe Solaris Volume Manager traditional
and soft
e
G
t
@
partitions
n
b
ku tude
a
y
Differentiate between shared
disksets
( is Sand local disksets
n
i
th multi-owner diskset
Describe Solaris Volume
lud Manager
e
a
us(metadb) management issues and the role
amdatabase
o
Describe volume
k
t
ngVolume
seManager mediators
of Solaris
a
n
b
e
c Solaris Volume Manager software
a
lithe
b
Configure
e
u
l
k rab
a
y
l
eCreate the local metadb replicas
f
u
s
d
ab -tran Add disks to shared disksets
g
an non
Build Solaris Volume Manager mirrors with soft partitions in

disksets
Use Solaris Volume Manager status commands
Perform Sun Cluster software-level device group management
Create global file systems
Mirror the boot disk with Solaris Volume Manager
7-1
Relevance
Relevance

Is it easier to manage Solaris Volume Manager devices with or

without soft partitions?
How many different collections of metadbs will you need to

manage?
What is the advantage of using DID devices as Solaris Volume

Manager building blocks in the cluster?
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
7-2
How are Solaris Volume Manager disksets registered in the cluster?

820-7458 (for Sun Cluster 3.2 11/09).
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Solaris Volume Manager Administration Guide, part number 816-4520.
Using Solaris Volume Manager With Sun Cluster Software

7-3
Exploring Solaris Volume Manager Disk Space Management
Exploring Solaris Volume Manager Disk Space

Management
Solaris Volume Manager has two distinct ways of managing disk space.
The original Solstice DiskSuite software (Solaris Volume Managers
precursor product) had the restriction that Solaris OS partitions were the
smallest granularity building blocks for volumes. The current Solaris
Volume Manager supports both the traditional way of managing space
and the new method called soft partitioning.
a
Solaris Volume Manager Partition-Based Disk Space
s
a
Management
y) h
m
m
comanagement
Figure 7-1 demonstrates Solaris Volume Manager disk space

e
m
d
s
i
that equates standard disk partitions with building
foruvirtual
cblocksG
e
t
@
volumes.
b den
u
k
tu
ya d18 submirror
Volume
(
S
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
Volume d6 submirror
k
b
a
a
r
y
e 7 (metadb)
ul nsfSlice
d
b
a
tra
g
n
Slice 3
n
a
no
ab
Slice 4
Volume d12 submirror
Slice 6
Physical Disk Drive
Figure 7-1
7-4
Solaris Volume Manager Space Management

The following are limitations with using only partitions as Solaris Volume
Manager building blocks:
The number of building blocks per disk or LUN is limited to the

traditional seven (or eight, if you want to push it) partitions. This is
particularly restrictive for large LUNs in hardware RAID arrays.
Disk repair is harder to manage because an old partition table on a

replacement disk must be recreated or replicated manually.
Solaris Volume Manager Disk Space Management

With Soft Partitions
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
Soft partitions add an enormous
in amount
is of flexibility to the capability of
h
t
Solaris Volume Manager:lud
a use
m
a to manage
They allowkyou
to volumes whose size meets your exact
g
e
requirements
ns physically repartitioning Solaris OS disks.
ban licewithout
a
b They lallow
e you to create more than seven volumes using space from
u
k
b
ya fearasingle large drive or LUN.
l
u
bd trans They can grow if there is space left inside the parent component. The
a
g onspace does not have to be physically contiguous. Solaris Volume
n
a
n
b
Manager finds whatever space it can to allow your soft partition to
a
Soft partitioning supports the creation of virtual partitions within Solaris
OS disk partitions or within other volumes. These virtual partitions can
range in size from one block to the size of the base component. You are
limited in the number of soft partitions only by the size of the base
component.
grow within the same parent component.

7-5

There are two ways to deal with soft partitions in Solaris Volume
Manager:
Make soft partitions on disk partitions, and then use those to build
your submirrors as shown in Figure 7-2.

Slice 7 (metadb)
soft partition d18
slice s0
s
) ha
V olume d20 submirror

soft partition d28
y
m
om
c
Physical Disk Drive

csm Guide
e
@ ent
Figure 7-2 Building Submirrors From SoftbPartitions
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
soft par tition d38
7-6

Build mirrors using entire partitions, and then use soft partitions to
create the volumes with the size you want as shown in Figure 7-3.
Slice 7 (metadb)
Slice 7 (metadb)
slice s0
slice s0
Physical disk drive
submirror
Physical disk drive
submirror
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this volume
d
u
al use
m
soft partition
ka e to
g
ban licens
a
b
u
le Using Soft Partitions to Create Volumes
k
b
Figure
7-3
a
a
l y sfer
u
d
ab -tranAll the examples later in this module use the second strategy. The
g
n non
a
advantages of this are the following:
b
a
Volume management is simplified.
Hot spare management is consistent with how hot spare pools work.
The only disadvantage of this scheme is that you have large mirror
devices even if you are only using a small amount of space. This slows
down resynching of mirrors.

7-7
Exploring Solaris Volume Manager Disksets
Exploring Solaris Volume Manager Disksets

When using Solaris Volume Manager to manage data in the Sun Cluster
3.2 environment, all disks that hold the data for cluster data services must
be members of Solaris Volume Manager shared disksets.
Only disks that are physically located in the shared storage will be
members of the shared disksets. Only disks that are in the same diskset
operate as a unit. They can be used together to build mirrored volumes,
and primary ownership of the diskset transfers as a whole from node to
node. You should always have disks from different controllers (arrays) in
the same diskset, so that you can mirror across controllers.
s
) ha
y
m
om
c
e
csm Grequires
uid that each
To create any shared disksets, Solaris Volume e
Manager
@ entdisks. The only
host (node) must have a local diskset on
non-shared
b
u
k theythave
requirement for these local disks is
ud local diskset metadbs,
athat
y
(
S
which are described later. Figure
in 7-4thshows
is the contents of two shared
d
u
disksets in a typical cluster,
ewith the local disksets on each node.
al along
s
m
u
ka e to
g
Diskset nfsds Diskset webds
ban licens
a
b
u
le
k
b
a
a
r 1
l y sfeNode
Node 2
u
d
n
b
a
tra
g
n
n
a
no
Shared disksets are given a name that often reflects the intended usage of
the diskset (for example, nfsds).
ab
Boot Disks
(Local Diskset)
Figure 7-4
7-8
Boot Disks
(Local Diskset)
Two Shared Disksets

Solaris Volume Manager Multi-Owner Disksets (for Oracle RAC)
Solaris Volume Manager Multi-Owner Disksets (for Oracle

RAC)
Solaris Volume Manager has a multi-owner diskset feature that is
analogous to the VERITAS Cluster Volume Manager (CVM) feature. That
is, a multi-owner diskset allows more than one node to physically access
the storage, simultaneously.
In the current implementation, multi-owner disksets are only for use with
Oracle RAC software. In fact, you cannot create a multi-owner diskset
until you have enabled a layer of software underlying Oracle RAC known
as the RAC framework. This is described in more detail, and you will get
a chance to work with it, if you choose, in one of the optional exercises in
Module 11.
s
) ha
y
m
om
c
Managing multi-owner disksets is identical to managing

sm uother
idedisksets,
c
e
except that you use a -M option when you create the tdiskset.
G
@
n
b
ku tude
a
y
n ( this S
i
d
lu se
a
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba

7-9
Using Solaris Volume Manager Database Replicas (metadb replicas)
Using Solaris Volume Manager Database Replicas

(metadb replicas)
Solaris Volume Manager requires that certain partitions serve as volume
database replicas, storing the Solaris Volume Manager configuration and
state information in a raw format (non-file system). These are normally
small, dedicated Solaris OS partitions. In the local disksets, Solaris Volume
Manager allows the creation of metadbs on a large partition and then the
use of the same partition as a component in a volume. The reasons why
you would never want to do this on the boot disks are described later in
this module.
as
There are separate sets of metadb replicas for the local disksets and for) h
y
each shared diskset. For example, in Figure 7-4 on page 7-8 (which
shows
m
two nodes and two shared disksets), there are four distinctccollections
om of
metadb replicas.
csm Guide
e
t
n
b@same
You can put several copies of the metadbk
on
the
partition.
You might
e
u
d
u
a
t
do this to balance the numbers of metadb
across disks or
(y ireplicas
S subsections.
n
s
i
partitions, for reasons described
in
the
following
lud se th
a
am to u
k
ng ense
Local ReplicaaManagement
b
a
lic
b
e
u
l
k managing
When
a
ab local replicas:
r
y
l
e
f
bdu transYou must add local replicas manually.
a
You can put local replicas on any dedicated partitions on the local
ang non
ab
7-10
disks. You might prefer to use slice 7 (s7) as a convention because

the shared diskset replicas have to be on that partition.
You must spread local replicas evenly across disks and controllers.
If you have less than three local replicas, Solaris Volume Manager
logs warnings. You can put more than one copy in the same partition
to satisfy this requirement. For example, with two local disks, set up
a dedicated partition on each disk and put three copies on each disk.

Local Replica Mathematics
The math for local replicas is as follows:
If fewer than 50 percent of the defined metadb replicas are available,

Solaris Volume Manager ceases to operate.
If fewer than or equal to 50 percent of the defined metadb replicas

are available at boot time, you cannot boot the node. However, you
can boot to single-user mode, and just use the metadb command to
delete ones that are not available.
s
) ha
Note Override the behavior of the last bullet item by inserting the line:
y
m
om
c
e nothing
idhas
in the /etc/system file. This flag is a misnomer
in thatuit
csm
e
G boot fully with
to do with using SVM to mirror root. It simply
does n
lett you
@
b
de available.
kulocal tmetadbs
exactly 50% (but not less than 50% a
) of
u
y
n ( this S
i
d
lu se
a
u
am to Management
Shared Diskset kReplica
ng ense
a
b
a the following
lic
Consider
issues when managing shared diskset replicas:
b
e
u
l
k
a eThere
ab are separate metadb replicas for each diskset.
r
y
l
f
bdu trans They are automatically added to disks as you add disks to disk sets.
a
g onn
They will be, and must remain, on slice 7.
a
n
ab
set md:mirrored_root_flag=1
You must use the metadb command to remove and add replicas if
you replace a disk containing replicas.
Shared Diskset Replica Quorum Mathematics

The math for shared diskset replica quorums is as follows:
If fewer than 50 percent of the defined replicas for a diskset are

available, the diskset ceases to operate.
If exactly 50 percent of the defined replicas for a diskset are

available, the diskset still operates, but it cannot be taken or switched
over.

7-11
Shared Diskset Mediators
When you have nodes connected to exactly two storage arrays, the
implication of the replica mathematics described in the previous section is
that if one storage array fails, your diskset can keep operating, but cannot
transfer primary control from node to node.
This is unacceptable in the Sun Cluster environment, because it can take a
while to fix a broken array or controller, and you still want to be able to
gracefully survive a node failure during that period.
The Sun Cluster 3.2 software environment includes special add-ons to
Solaris Volume Manager called mediators. Mediators allow you to identify
the nodes themselves as tie-breaking votes in the case of failure of exactly 50
percent of the metadb replicas of a shared diskset. The mediator data is
stored in the memory of a running Solaris OS process on each node. If you
lose an array, the node mediators will change to golden status, indicating
they count as extra votes for the shared diskset quorum mathematics (one
from each node). This allows you to maintain normal diskset operations
with exactly 50 percent of the metadb replicas surviving. You can also lose
a node at this point (you would still have at least one golden mediator).
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
7-12

Installing Solaris Volume Manager and Tuning the md.conf File
Installing Solaris Volume Manager and Tuning the

md.conf File
In the Solaris 9 OS and the Solaris 10 OS, the packages associated with the
standard Solaris Volume Manager functionality are part of the base
OS.Support for shared diskset mediators is in the packages SUNWmdmr and
SUNWmdmu. These packages are automatically installed as part of the
cluster framework.
Modifying the md.conf File (Solaris 9 OS Only)
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
Table 7-1 Modifications to the md.conf File
ak Stud
y
(
in this
Default
Variable
Description lud
Value
a use
m
o
ka e tnumber
g
nmd
128
The
maximum
n this
s to limitofthevolumes.
a
n
b
e
uses
setting
names
of
volumes as well.
c this number too high can use the
a Setting
i
l
b
a lot of inodes for device
u
le
k
b
a
a
files
in
your
/global/.devices/node@#
file system. The
y fer
l
u
s
maximum value is 8192.
bd tran
a
gmd_nsets
n- 4
The maximum number of disksets. This includes the local
n
o
a
n
b
diskset. This number should be set to the number of shared
a
Based on your planned implementation, you might need to update Solaris

Volume Managers kernel configuration file, /kernel/drv/md.conf. Two
variables might need to be updated. These maximums include your local
disksets. The modifications are summarized in Table 7-1.
disksets you plan to create in your cluster, plus one. The

maximum value for md_nsets is 32.
Keep this file identical on all nodes of the cluster. Changes to this file take
effect after you call devfsadm or perform a boot -r.
Solaris 10 dynamically creates device files as required. The absolute upper
limit on number of disksets (32) and number of volumes per diskset
(8192) is still the same.

7-13
Initializing the Local metadb Replicas on Local Disks

Solaris Volume Manager management cannot be performed until you
initialize the local metadb replicas on each node.
These instructions assume that you have a small partition to use for the
metadb replicas on your boot disk. In this example, the partition is s7.
Make sure you initialize local metadb replicas correctly and separately on
each node.
Using DIDs Compared to Using Traditional c#t#d# ) has

y
m
m
In the Sun Cluster environment, you can add any metadbc
oropartition
by
e
m
component using either its cluster DID (/dev/did/rdsk/d#s#)
or
d
s
i
ec t Gu
using the traditional c#t#d#.
@
b den
u
k
tu it you are restricted
ya Without
Use the DID naming for all shared disksets.
(
S
n
s node. You might assume
to having identical controllerunumbers
di e on
hieach
t
l
a be surprised
this will always be true,mthen
when you add a new node or
s
u
a
o
repair a node.
k et
g
n
a cens
b
a
li c#t#d# naming scheme for local metadb replicas and
Use the
b traditional
e
u
l
k
devices.
recovery easier in the case that you need to access
a eThis
abmakes
r
y
l
these
structures
when
booted in non-clustered mode. Omit the /dev/rdsk
f
u ns
d
b
aabbreviate traditional names in all of the Solaris Volume Manager
a
trto
g
n
commands.
n
a
no
ab
Leaving Dedicated Partitions Even When Using ZFSroot

You must have local metadb replicas on dedicated local disk partitions
even when you use ZFS root. Dedicate a small partition for metadbs
outside of your ZFS configuration, as discussed in the example in Module
3, if you need to use Solaris Volume Manager for your data.
7-14

Adding the Local metadb Replicas to the Local Disks

Use the metadb -a command to add local metadb replicas. As you add
the first ones, you must use the -f (force) option. The following example
creates three copies on each of two local disks. If these disks were
mirrored using ZFS root, you would have to make sure you had a slice 7
of sufficient size dedicated for metadas on each one.
# metadb -a -f -c 3 c0t0d0s7
Repartitioning a Mirror Boot Disk and Adding metadb

a
Replicas
as
)h
y
midentical
This procedure assumes you have a second local disk, which
has
m
co command
geometry to the first. In that case, you can use the fmthard
to
e
m
d
s
i
replicate the boot disks partition table onto the
disk:
u
ecsecond
G
t
@
b den
# prtvtoc /dev/rdsk/c0t0d0s0|fmthard -s - /dev/rdsk/c0t8d0s0
u
k
tu
yain place.
fmthard: New volume table of contents now
(
S
n
di e this
u
l
a us
# metadb -a -c 3 c0t8d0s7
m
a
k e to
g
an cens
b
a
Using the
li or metadb -i Command to Verify
b metadb
e
u
l
k
metadb
a eReplicas
ab
r
y
l
f
bdu trans
a
g on- The output of the metadb command includes one line for each replica. The
n
a
n
-i option adds the legend information about the flag abbreviations.
ab
# metadb -i
flags
first blk
block count
a
u
16
8192
/dev/dsk/c0t0d0s7
a
u
8208
8192
/dev/dsk/c0t0d0s7
a
u
16400
8192
/dev/dsk/c0t0d0s7
a
u
16
8192
/dev/dsk/c0t8d0s7
a
u
8208
8192
/dev/dsk/c0t8d0s7
a
u
16400
8192
/dev/dsk/c0t8d0s7
r - replica does not have device relocation information
o - replica active prior to last mddb configuration change
u - replica is up to date
l - locator for this replica was read successfully
c - replica's location was in /etc/lvm/mddb.cf
p - replica's location was patched in kernel
m - replica is master, this is replica selected as input

7-15

W
a
M
D
F
S
R
replica
replica
replica
replica
replica
replica
replica
has device write errors

is active, commits are occurring to this replica
had problem with master blocks
had problem with data blocks
had format problems
is too small to hold current data base
had device read errors
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
7-16

Creating Shared Disksets and Mediators

Use the metaset command to create new empty disksets and to add disk
drives into the diskset. You must use the -a -h options of the metaset
command first to create an empty diskset. Then you can add disks. The
first host listed as you create a new diskset is the first one to be the owner
of the diskset. You can add mediators with the -a -m options.
All the examples shown in this module of diskset operations use the DID
names for disks instead of the c#t#d# names. You might need to run the
cldev list command often to map between the two.
# cldev list -v c1t3d0 c2t3d0
DID Device
Full Device Path
------------------------d9
d9
d17
d17
# cldev list -v d9 d17
DID Device
Full Device Path
------------------------d9
d9
d17
d17
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
n -a -h vincent theo
# metaset
anfsds
ab -t-s
r
g
#
metaset
-s
nfsds
-a -m vincent theo
n
n
o
a
n
ab
# metaset -s nfsds -a /dev/did/rdsk/d9 /dev/did/rdsk/d17
# metaset
Set name = nfsds, Set number = 1
Host
vincent
theo
Owner
Yes
Mediator Host(s)
vincent
theo
Aliases
Drive Dbase
d9
Yes
d17
Yes

7-17
# metadb -s nfsds
flags
a
u
a
u
r
r
# medstat -s nfsds
Mediator
vincent
theo
first blk
16
16
Status
Ok
Ok
block count
8192
/dev/did/dsk/d9s7
8192
/dev/did/dsk/d17s7
Golden
No
No
Mediators become golden if exactly 50 percent of the metadb replicas fail.
s
) ha
y
Automatic Repartitioning and metadb Placement on
m
m
Shared Disksets
co
csm Guide
e
t
When a disk is added to a diskset, it is automatically
as
@ erepartitioned
n
b
u
follows:
ak Stud
y
(
in(starting
A small portion of the drive
d
hiatscylinder 0) is mapped to slice
t
u
l
e
7 or slice 6 to be usedafor statesdatabase replicas (at least 4 Mbytes in
am to u
size).
k
ng ense
a
b
a
lic
b
e
u
l
Note
Cluster
3.2 is the first Sun Cluster version to fully support
k Sun
b
a
a
r
y
disks
with
Extensible
Firmware Interface (EFI) labels, which are required
l
e
f
u
s
d
andisks of size greater than 1 Terabyte (Tbyte). These disks have no slice
ab -trfor
g
an non 7. Solaris Volume Manager automatically detects an EFI disk, and uses
ab
slice 6 for the metadb partition.
One metadb is added to slice 7 or 6 as appropriate.
Slice 7 or 6 is marked with the flag bits 01 (this shows up as wu when

you view the disk from the format command writable but
unmountable).
The rest of the drive is mapped to slice 0 (on a standard VTOC disk,
even slice 2 is deleted).
The drive is not repartitioned if the disk already has no slice 2 and slice 7
already has the following characteristics:
7-18
It starts at cylinder 0.
It has at least 4 Mbytes (large enough to hold a state database).
The flag bits are already 01 (wu in format command).


Regardless of whether the disk is repartitioned, diskset metadb replicas
are added automatically to slice 7 or slice 6 as appropriate (as shown in
Figure 7-5). If you have exactly two disk controllers, you should always
add an equivalent numbers of disks or LUNs from each controller to each
diskset to maintain the balance of metadb replicas in the diskset across
controllers.
Slice 7 (metadb)
s
) ha
y
m
om
c
Slice 0
csm Guide
e
@ ent
b
u
ak Stud
y
(
Physical disk drive din
his
t
u
l
e
a us
m
Figure 7-5 Automatically
a
k e toAdding a metadb to Slice 7
g
ban licens
a
b If itleis impossible to add the same number of disks from each
u
Note
k
a controller
abto a diskset, you should manually run the command
r
y
l
e
f
s
-s setname -b in order to balance the number of metadb
bdu tranmetadb
a
g
replicas
across
controllers.
n
n
o
a
n
ab

7-19
Using Shared Diskset Disk Space
Using Shared Diskset Disk Space

This section shows the commands that implement the strategy described
earlier in the module.
Always use slice 0 as is in the diskset (almost the entire drive or

LUN).
Build sub-mirrors out of slice 0 of disks across two different

controllers.
Build a mirror out of those sub-mirrors.
Use soft partitioning of the large mirrors to size the volumes

according to your needs.
s
) ha
y
m
Figure 7-6 demonstrates the strategy again in terms of volumem

and
o to(d#)
c
DID (d#s#). While Solaris Volume Manager would allow

you
use
e
c#t#d#, always use DID numbers in the cluster toeguarantee
csm Guaidunique,
t
@ of eallnnodes.
agreed-upon device name from the point of b
view
u
ak Stud
y
(
in this
d
u
e
d10 (soft partition) al
d11 (soft partition)
s
m
u
a
o
k et
g
n
ba licens
a
d100 (mirror)
b
e
u
l
k
b
ya fera
l
u
bd trans
a
d101 (stripe / concat)
d102 (stripe / concat)
ang non
ab
Volumes
DIDs
/dev/did/rdsk/d9s0
Figure 7-6
7-20
/dev/did/rdsk/d17s0
Strategy for Building Volumes

Building Volumes in Shared Disksets With Soft Partitions of Mirrors
Building Volumes in Shared Disksets With Soft Partitions

of Mirrors
The following are two ways to indicate which diskset you are referring to
for each metainit command:
Use -s disksetname with the command
Use disksetname/d# for volume operands in the command
This module uses the former model for all the examples.
s
) ha
The following is an example of the commands that are used to build the
configuration described in Figure 7-6 on page 7-20:
ab
y
m
m
# metainit -s nfsds d101 1 1 /dev/did/rdsk/d9s0
o
c
nfsds/d101: Concat/Stripe is setup

e
m
d
s
i
# metainit -s nfsds d102 1 1 /dev/did/rdsk/d17s0
ec t Gu
@
nfsds/d102: Concat/Stripe is setup
b den
u
k
# metainit -s nfsds d100 -m d101
ya Stu
(
nfsds/d100: Mirror is setup
n
i
this
# metattach -s nfsds d100 d102 alud
e
us
nfsds/d100: submirror nfsds/d102
am istoattached
k
# metainit -s nfsds d10n-p
200m
g d100
e
s
a
n
d10: Soft Partition
is
setup
b
e
a d11 -plicd100 200m
b
# metainit -s nfsds
u
lesetup
k
b
a
a
d11: Soft Partition
is
l y sfer
u
d
ab -tranThe following commands show how a soft partition can be grown if there
g
an non
is space in the parent volume, even non-contiguous space. You will see in
the output of metastat on the next page how both soft partitions contain
noncontiguous space, because of the order in which they are created and
grown.
# metattach -s nfsds d10

nfsds/d10: Soft Partition
/ metattach -s nfsds d11
400m
has been grown
400m
has been grown
Note The volumes are being increased by 400 Mbytes, to a total size of
600 Mbytes.

7-21
Using Solaris Volume Manager Status Commands

No commands display information about volumes or metadb replicas in
multiple disksets at the same time. If you do not specify a diskset name
(with -s), you get output only about the local diskset on the node on
which you entered the command.
Checking Volume Status

The following metastat command output is for the mirrored volume and
soft partitions built in the previous example:
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
d count
k tBlock
u
a
y
409600
n ( this S
i
d
819200
lu se
a
am to u
k
nfsds/d100: Mirror
ng ense
Submirror 0: nfsds/d101
a
b
lic
State: Okay b a
e
u
l
knfsds/d102
Submirror 1:
a
ab
r
y
l
e
State:
Resyncing
f
ns
bduintrprogress:
a
Resync
0 % done
a
g
n
Pass:
1
n
o
aba Readn option: roundrobin (default)
# metastat -s nfsds
Device: nfsds/d100
State: Okay
Size: 1228800 blocks (600 MB)
Extent
Start Block
0
409664
1
1638528
Write option: parallel (default)

Size: 71118513 blocks (33 GB)
nfsds/d101: Submirror of nfsds/d100
State: Okay
Stripe 0:
Device
Start Block Dbase
d9s0
0
No
State Reloc Hot Spare

Okay
No
nfsds/d102: Submirror of nfsds/d100

State: Resyncing
Stripe 0:
7-22

Device
d17s0
Start Block
0
Dbase
No
State Reloc Hot Spare

Okay
No

Device: nfsds/d100
State: Okay
Size: 1228800 blocks (600 MB)
Extent
Start Block
0
32
1
819296
Block count
409600
819200
Device Relocation Information:

Device
Reloc Device ID
d17
No
d9
No
-
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

7-23
Managing Solaris Volume Manager Disksets and Sun Cluster Device Groups
Managing Solaris Volume Manager Disksets and Sun

Cluster Device Groups
When Solaris Volume Manager is used in the cluster environment, the
commands are tightly integrated into the cluster. Creation of a shared
diskset automatically registers that diskset as a cluster-managed device
group:
# cldg status
s
) ha
--- Device Group Status ---
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
# cldg show nfsds
in this
d
u
al use
Device Groups ===
m
ka e to
g
Device Group Name:
nfsds
an cens
b
a
i
l
Type:
SVM
b
e
u
l
k
failback:
false
a erab
y
l
Node List:
vincent, theo
f
u ns
d
b
preferenced:
true
a
tra
g
n
numsecondaries:
1
n
noname:
nfsds
aba diskset
Device Group Name
----------------nfsds
Primary
------vincent
Secondary
--------theo
Status
-----Online
Addition or removal of a directly connected node to the diskset using

metaset -s nfsds -a -h newnode automatically updates the cluster
to add or remove the node from its list. There is no need to use cluster
commands to resynchronize the device group if volumes are added and
deleted.
In the Sun Cluster environment, use only the cldg switch command
(rather than the metaset -[rt] commands) to change physical
ownership of the diskset. As demonstrated in this example, if a mirror is
in the middle of synching, this forces a switch anyway and restarts the
synch of the mirror all over again.
# cldg switch -n theo nfsds
7-24

Managing Solaris Volume Manager Disksets and Sun Cluster Device Groups
Dec 10 14:06:03 vincent Cluster.Framework: stderr: metaset: vincent:
Device busy
[a mirror was still synching, will restart on other node]

# cldg status
--- Device Group Status --Device Group Name
----------------nfsds
Primary
------theo
Secondary
--------vincent
Status
-----Online
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

7-25
Managing Solaris Volume Manager Device Groups
Managing Solaris Volume Manager Device Groups

Solaris Volume Manager is cluster aware. As such, there is no need to
register Solaris Volume Manager disksets with the cldg create
command. The cldg set command can be used to perform
cluster-specific changes to Solaris Volume Manager device groups.
Device Group Resynchronization

Solaris Volume Manager device groups are automatically resynchronized
when new volumes are added or removed within an existing diskset.
s
) ha
y
m
Other Changes to Device Groups

om
c
sm uide
c
e
G
The properties of existing Solaris Volume Manager
device
groups can be
t
@
n
b
e could be
changed. For example, the failback property
ku oftuadgroup
a
y
modified with the following command:
n ( this S
i
d
lu nfsds
# cldg set -p failback=true
e
a
s
m
u
ka e to
g
ns Offline
ban lGroup
e
Putting a Device
c
a
i
b
e
u
l
k
a canetake
aba Solaris Volume Manager device group out of service, as far
r
y
l
You
f
bdu trasanthes cluster is concerned, for emergency repairs.
a
ang non
ab
To put the device group offline, all of the Solaris Volume Manager
volumes must be unused (unmounted, or otherwise not open). Then you
can issue the following command:
# cldg offline nfsds
You rarely need to take the device group offline because almost all repairs
can be done while the device group is in service.
To place the device group back online, type one of the following
commands (you do not need both):
# cldg online nfsds
# cldg online -n node_to_switch_to nfsds
7-26

Using Global and Failover File Systems on Shared Diskset Volumes
Using Global and Failover File Systems on Shared Diskset

Volumes
Sun Cluster 3.2 supports running data services on the following categories
of file systems:
Global file systems These are accessible to all cluster nodes

simultaneously, even those not physically connected to the storage.
Failover file systems These are mounted only on the node running
the failover data service, which must be physically connected to the
storage.
ab
as
h
The file system type can be UFS or VxFS regardless of whether you
are
)
y in
m
using global or failover file systems. The examples and the labexercises
om
this module assumes that you are using UFS.
c
csm Guide
e
@ ent
b
Creating File Systems
u
ak Stud
y
(
dinand
his file system is not made at the
t
The distinction between lglobal
failover
u
e
a Use
time of file system creation.
m
usnewfs as normal to create a UFS file
a
o
k
t
system on thegvolume: e
n ens
ba/dev/md/nfsds/rdsk/d10
# newfs
a
lic
b
e
u
l
k rab
a
y
l
fe
uMounting
s
d
n
File Systems
b
a
a
r
t
g
an non
The distinction between a global and a failover file system is made in the
/etc/vfstab mount-at-boot and options columns.
A global file system entry should look similar to the following, and it
should be identical on all nodes including nodes not physically connected
to the storage:
/dev/md/nfsds/dsk/d10 /dev/md/nfsds/rdsk/d10 /global/nfs ufs 2 yes global
A failover file system entry looks similar to the following, and it should
be identical on all nodes that might run services that access the file system
(they can only be nodes that are physically connected to the storage).
/dev/md/nfsds/dsk/d10 /dev/md/nfsds/rdsk/d10 /localnfs ufs 2 no -

7-27
Using Solaris Volume Manager to Mirror the Boot Disk

If Solaris Volume Manager is your volume manager of choice for the
shared storage, you might also use Solaris Volume Manager to manage
and mirror the boot drive.
Note If you already have ZFS-root, of course you will not need or be
able to mirror your root drive with Solaris Volume Manager. This
discussion is for those running Solaris OS prior to Solaris 10 10/08, or still
interested in using a legacy configuration in Solaris 10 10/08 and later.
The following example contains the following scenario:
s
) ha
y
m
The boot drive is mirrored after cluster installation.

om
c
e root
All partitions on the boot drive are mirrored. The
exampleid
has
sm
c
u
e
(/), swap, and /global/.devices/node@1. That is,
Gyou will be
t
@
n
b
manually creating three separate mirror
devices.
ku tude
a
y
( of ithe
Sboot disk and the new
The geometry and partitionin
tables
s
d
h
t
mirror are identical. Copying
table from one disk to the
lu sthee partition
a
other was previously
demonstrated
in Repartitioning a Mirror Boot
u
am
o
k
t
Disk and Adding
metadb
Replicas
on page 7-15.
g nse
n
a
b bootlicdisk
e and mirror have three copies of the local metadb
Bothathe
b
ureplicas.bThis
le was done previously.
k
a
a
l y sSoft
erpartitions are not used on the boot disk. That way, if you need to
f
u
d
ab -tran back out, you can just go back to mounting the standard partitions
g
by editing the /etc/vfstab manually.
an non
ab
The /global/.devices/node@1 is a special, cluster-specific case. For

that one device you must use a different volume d# for the top-level
mirror on each node. The reason is that each of these file systems
appear in the /etc/mnttab of each node (as global file systems) and
the Solaris OS does not allow duplicate device names.
Verifying Partitioning and Local metadbs

The bold sections of the following output emphasize the partitions that
you will mirror:
# df -k
/dev/dsk/c0t0d0s0
/devices
7-28
33453987 7360376 25759072

0
0
0
23%
0%
/
/devices


ctfs
proc
mnttab
swap
objfs
fd
swap
swap
0
0
0
1920832
0
0
1926208
1919440
0
0
0
0
0
0
1472 1919360
0
0
0
0
6848 1919360
80 1919360
0%
0%
0%
1%
0%
0%
1%
1%
/system/contract
/proc
/etc/mnttab
/etc/svc/volatile
/system/object
/dev/fd
/tmp
/var/run
/dev/did/dsk/d11s3
491520
3735
438633
1%
491520
3677
438691
1%
/dev/did/dsk/d1s3
s
) ha
y
m
# swap -l
om
c
swapfile
dev swaplo blocks
free
/dev/dsk/c0t0d0s1
32,1
16 8170064 8170064
csm Guide
e
@ ent
b
u
# metadb
ak Stud
y
(
flags
first blk
in block
iscount
d
h
t
a m pc luo
16
8192
/dev/dsk/c0t0d0s7
u
e
al us8192
a
pc luo
8208am
/dev/dsk/c0t0d0s7
k e to 8192
a
pc luo
16400
/dev/dsk/c0t0d0s7
g
n
s
a
n
a m pc luo
8192
/dev/dsk/c0t8d0s7
ce
ab 16
i
l
a
pc luo
8208
8192
/dev/dsk/c0t8d0s7
b
u
le16400
k
b
a
pc
luo
8192
/dev/dsk/c0t8d0s7
a
a
y fer
l
u
s
bd tran
a
g onBuilding
n
Volumes for Each Partition Except for Root
a
n
ab
For all boot disk partitions except root, follow the same general strategy:
1.
Create simple sub-mirror volumes for the existing partition and the
other disk partition. Use the -f option for the existing partition.
2.
Create a mirror using only the sub-mirror mapping to the existing

partition. Do not attach the other half of the mirror until after a
reboot. Make sure the d# chosen for the /global/.devices/node@#
is different across your nodes.
3.
Edit the /etc/vfstab file manually to use the new mirror volume
instead of the original partition.
4.
Wait to reboot until you do all partitions.

7-29
Building Volumes for Root Partition
For the root boot disk partition, follow the same general strategy:
1.
Create simple sub-mirror volumes for the existing partition and for
the other disk partition. Use the -f option for the existing partition.
2.
Create a mirror using only the sub-mirror mapping to the existing

partition. Do not attach the other half of the mirror until after a
reboot.
3.
Use the metaroot command, which will automatically edit the

/etc/vfstab and /etc/system files.
s
) ha
y
m
om
c
sm uide If
Examples are shown for root, swap, and /global/.devices/node@#.
c
e
Gbe the same as
you had any other OS disk partitions, the procedure
would
t
@
n
b
for the non-root slices. Only for /global/.devices/node@#
do you need
ku tude
a
y
to worry about making a different mirror
volume
name
on
each
node.
( is S
n
i
lud se th
# metainit -f d11 1 1ac0t0d0s0
u
d11: Concat/Stripe
is setup
am
o
k
t
# metainit d12
ng1 e1 nc0t8d0s0
se
a
b
d12: Concat/Stripe
is
setup
c
a
i
l
b
# metainit
d10
u
le -m d11
k
b
a
a
d10:
Mirror
y fer is setup
l
u
d10
ns
bd tr#ametaroot
a
ang non # metainit -f d21 1 1 c0t0d0s1
Running the Commands
ab
d11: Concat/Stripe is setup

# metainit d22 1 1 c0t8d0s1
# metainit d20 -m d21
d10: Mirror is setup
# vi /etc/vfstab
(change the correct line manually)
/dev/md/dsk/d20 -
swap
# metainit -f d31 1 1 c0t0d0s3

7-30

no
# metainit d32 1 1 c0t8d0s3

//following volume name (d30) must be different on each
// node
# metainit d30 -m d31
d10: Mirror is setup
# vi /etc/vfstab
(change correct line (for /global/.devices/node@# manually)
/dev/md/dsk/d30 /dev/md/rdsk/d30 /global/.devices/node@1
ufs 2 no global
s
) ha
ab
y
m
om
c
Rebooting and Attaching the Second Submirror
csm Guide
e
@ entot each volume. The
After a reboot, you can attach the secondbsubmirror
u
k background,
synchronizing of the mirrors runsyin
athe
tud and can take a long
(
S
time.
in this
d
u
al use
# init 6
m
ka e to
# df -k
g
s
/dev/md/dsk/d10
26843000
25510407
5%
/
an cen1064163
b
a
/devices
0
0
0%
/devices
li 0
b
e
u
l
k
ctfs
0
0
0
0%
/system/contract
a erab
y
l
proc
0
0
0
0%
/proc
f
u ns
d
b
mnttab
0
0
0
0%
/etc/mnttab
a
tra
g
n
swap
1920832
1472
1919360
1%
/etc/svc/volatile
n
a
no
objfs
0
fd
0
swap
1926208
swap
1919440
/dev/md/dsk/d30
95702
/dev/did/dsk/d22s5
95702
# swap -l
swapfile
/dev/md/dsk/d20
0
0
0
0
6848 1919360
80 1919360
5317
80815
0%
0%
1%
1%
7%
5010
6%
81122
/system/object
/dev/fd
/tmp
/var/run
dev swaplo blocks

free
85,20
16 8170064 8170064
# metattach d10 d12

# metattach d20 d22
# metattach d30 d32

7-31
Exercise: Configuring Solaris Volume Manager

Task 1 Initializing the Solaris Volume Manager Local metadb

Replicas
Task 2 Selecting the Solaris Volume Manager Demo Volume Disk

Drives
Task 3 Configuring Solaris Volume Manager Diskset and Volumes

for NFS
Task 4 Configuring Solaris Volume Manager Demonstration

Volumes
y
m
om
c
Task 6 Creating a Global web File System
csm Guide
e
@ ent
b
u
ak Stud
y
(
Task 9 Viewing and Managing
in Solaris
is Volume Manager Device
d
h
t
u
Groups Using Sun Cluster
e
al Manager
s
m
u
ka e to
g
n
s
Preparationaba licen
b
u
le
k
b
a
a
l yThis sexercise
er assumes you are running the Sun Cluster 3.2 software
f
u
d
on Solaris 10 OS.
an
ab -trenvironment
g
an non
s
) ha
ab
At least one local disk on each node must have a small unused slice that
you can use for the local metadb replicas. The examples in this exercise
use slice 7 of the boot disk.
If you happen to be using ZFS root, slice 7 should have been reserved
ahead of time to accomodate this lab.
7-32


During this exercise, you create two data service disksets that each
contain a single mirrored volume as shown in Figure 7-7.
Node 1
Boot Disk
Boot Disk
c0
Node 2
c0
(replicas) (replicas)
c1
c2
c1
c2
Quorum Disk
Diskset
and Volumes
nfsds
s
) ha
y
m
om
c
webds
Mirror Disk
Primary Disk
sm uide
c
e
d100
G
t
@
n
b
Array A
Array
B
ku tude
a
y
S
n ( Volume
s
i
i
Figure 7-7 Configuring Solaris
Manager
d
h
lu se t
a
am to u
k
g nse
Note a
During
ban lithis
e exercise, when you see italicized names, such as
c
b
IPaddress,
u
leenclosure_name, node1, or clustername embedded in a
k
b
a
a
command
y fer string, substitute the names appropriate for your cluster.
l
u
bd trans
a
g onn
a
n
ab
Primary Disk
Mirror Disk
d100

7-33
Task 1 Initializing the Solaris Volume Manager Local

metadb Replicas
Before you can use Solaris Volume Manager to create disksets and
volumes, you must initialize the state database and create one or more
replicas.
Perform the following steps on each node of the cluster:
1.
Verify that the boot disk has a small slice s7 reserved for metadbs. In
the standard setup, the boot disk (c#t#d# below) should be
configured as the ZFS root pool.
# zpool status
# prtvtoc /dev/rdsk/c#t#d#s0
s
) ha
y
m
om
c
e
csm Gnodes.
uid Check
Caution Your boot disk path may be different onedifferent
@ ent
carefully on each node.
b
u
ak Stud
y
(
2. Use the metadb command to
increateththree
is replicas on the unused boot
d
u
l
disk slice.
e
a us
m
a
# metadb -ak-f -c t3oc#t#d#s7
e
ng verify
sthat
a
n
3. On allbnodes,
the replicas are configured and operational.
e
c
a
i
l
b
u# metadb
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n nTask
on 2 Selecting the Solaris Volume Manager Demo
a
b
a
Volume Disk Drives
Perform the following steps to select the Solaris Volume Manager demo
volume disk drives:
7-34
1.
If you happen to also be using VxVM, type vxdisk -o alldgs

list to ensure that you pick drives that do not conflict with VxVM.
2.
On Node 1, type the cldev list -v command to list all of the

available DID drives and make note of the DID numbers of disks
used for VxVM, if any. Make sure you do not use those disks in this
exercise.
3.
Record the logical path and DID path numbers of four disks that you
will use to create the demonstration disksets and volumes in
Table 7-2. Remember to mirror across arrays.

Note You need to record only the last portion of the DID path. The first
part is the same for all DID devices: /dev/did/rdsk.
Table 7-2 Logical Path and DID Numbers
Diskset
Volumes
example
d100
nfsds
d100
webds
d100
Primary
Disk
Mirror
Disk
c2t3d0 d4
c3t18d0 d15
s
) ha
y
m
om
c
csm Guide
e
t
Note Make sure the disks you select arebnot
They must be
@localendevices.
u
dual-hosted and available to more than
ud host.
ak oneStcluster
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

7-35
Task 3 Configuring Solaris Volume Manager Diskset

and Volumes for NFS
Perform the following steps to create demonstration disksets and volumes

for use in later exercises:
1.
On Node 1, create the nfsds diskset, and configure the nodes that
are physically connected to it.
# metaset -s nfsds -a -h node1 node2
2.
Add the same nodes as diskset mediators to each diskset.

# metaset -s nfsds -a -m node1 node2
s
) ha
y
m
3. Add the two disks chosen previously, one from each array,
mto the
o
c
nfsds diskset.
e
m
d
s
i
ec t \Gu
# metaset -s nfsds -a /dev/did/rdsk/dx
@
b den
/dev/did/rdsk/dy
u
k
ya Stu
(
4. Verify the status of the new n
diskset.
di e this
u
# metaset -s nfsds
l
a us
m
# medstat -s a
nfsds
k e to
# cldg status
g
an cens
b
a
5. Create
li on each of your disks in the nfsds diskset.
b a submirror
e
u
l
k
a # emetainit
ab -s nfsds d0 1 1 /dev/did/rdsk/dxs0
r
y
l
f
bdu trans# metainit -s nfsds d1 1 1 /dev/did/rdsk/dys0
a
g on- 6. Create a mirror volume, d99, using the d0 submirror.
n
a
n
ab
# metainit -s nfsds d99 -m d0
7.
Attach the second submirror, d1, to the mirror volume d99.

# metattach -s nfsds d99 d1
8.
Create a 500 Mbyte soft partition, d100, on top of your mirror. This is
the volume you will actually use for your file system data.
# metainit -s nfsds d100 -p d99 500m
9.
Verify the status of the new volume.

# metastat -s nfsds
7-36

Task 3 Configuring Solaris Volume Manager Diskset

and Volumes for Web
Perform the following steps to create demonstration disksets and volumes
for use in later exercises:
1.
On Node 1, create the webds diskset, and configure the nodes that
are physically connected to it.
# metaset -s webds -a -h node1 node2
2.
Add the same nodes as diskset mediators to each diskset.

# metaset -s webds -a -m node1 node2
s
) ha
y
m
3. Add the two disks chosen previously, one from eachoarray,

m to the
c
webds diskset.
e
m
d
s
i
ec t Gu\
# metaset -s webds -a /dev/did/rdsk/dx
@
b den
/dev/did/rdsk/dy
u
k
ya Stu
(
4. Verify the status of the new
diskset.
n
i
is
# metaset -s a
webds
lud se th
# medstat -s
amwebdsto u
k
# cldg
ngstatus
se
a
n
b
e
a a submirror
5. b Create
on each of your disks in the webds diskset.
lic
e
u
l
k
b
a e#rametainit
-s webds d0 1 1 /dev/did/rdsk/dws0
y
l
f
u
s
bd tran # metainit -s webds d1 1 1 /dev/did/rdsk/dzs0
a
g on- 6. Create a mirror volume, d99, using the d0 submirror.
n
a
n
ab
# metainit -s webds d99 -m d0
7.
Attach the second submirror, d1, to the mirror volume d99.

# metattach -s webds d99 d1
8.
Create a 500 Mbyte soft partition, d100, on top of your mirror. This is
the volume you will actually use for your file system data.
# metainit -s webds d100 -p d99 500m
9.
Verify the status of the new volume.

# metastat -s webds

7-37
Perform the following steps on Node 1 to create a global file system in the
nfsds diskset:
1.
On Node 1, create a file system on d100 in the nfsds diskset.

# newfs /dev/md/nfsds/rdsk/d100
2.
Note If you have already done the VxVM exercises, and already have a
mounted /global/nfs file system, you can choose a different mount
point for this one.
s
) ha
y
m
# mkdir /global/nfs
om
c
3. On all nodes, add a mount entry in the /etc/vfstab

sm fileufor
idethe new
c
e
G
t
@
n
b
/dev/md/nfsds/dsk/d100 /dev/md/nfsds/rdsk/d100
\
ku tude
a
y
n ( this S
i
d
lu se
a
m to u
aline
k
Note Do not use
the
continuation character (\) in the vfstab file.
g
e
n
s
ba 1, mount
en the /global/nfs file system.
c
a
i
l
4. On
Node
b
u
le/global/nfs
k
b
a
a
#
mount
l y sfer
u
d
ab -tr5.an Verify that the file system is mounted and available on all nodes.
g
# mount
an non
ab
7-38
# ls /global/nfs
lost+found

Task 6 Creating a Global web File System
Perform the following steps on Node 1 to create a global file system in the
webds diskset:
1.
On Node 1, create a file system on d100 in the webds diskset.

# newfs /dev/md/webds/rdsk/d100
2.
Note If you have already done the VxVM exercises, and already have a
mounted /global/web file system, you can choose a different mount
point for this one.
s
) ha
ab
y
m
# mkdir /global/web
om
c
3. On all nodes, add a mount entry in the /etc/vfstab

sm ufile
idefor the new
c
e
G
t
@
n
b
/dev/md/webds/dsk/d100 /dev/md/webds/rdsk/d100
\
ku tude
a
y
/global/web ufs 2 yes global
n ( this S
i
d
lu se
a
m to u
athe
k
Note Do not
use
line continuation character (\) in the vfstab file.
g
e
n
s
a
n
b
e
a Nodeli1,cmount the /global/web file system.
4. b On
u
le /global/web
k
b
a
a
#
mount
l y sfer
u
d
ab -tran5. Verify that the file system is mounted and available on all nodes.
g
# mount
an non
# ls /global/web
lost+found

7-39
Perform the following steps to confirm the general behavior of globally

available file systems in the Sun Cluster 3.2 software environment:
1.
On Node 2, move into the /global/nfs file system.

# cd /global/nfs
2.
On Node 1, try to unmount the /global/nfs file system. You should

get an error that the file system is busy.
3.
On Node 2, move out of the /global/nfs file system (cd /) and try
to unmount it again on Node 1.
4.
Mount the /global/nfs file system on Node 1.
5.
Try unmounting and mounting /global/nfs from all nodes.
2.
Verify the current demonstration device group configuration.
s
) ha
y
m
om
c
sm uide
c
e
G
t
@
n
b
ku tude
a
y
Perform the following steps to migrate
n ( tahdisk
s Sdevice group (diskset)
i
i
d
between cluster nodes:
lu se
a
uare online (to the Sun Cluster software).
am groups
1. Make sure thekdevice
o
t
ng ense
# cldgastatus
b
a
lic
b
e
u
l
k rab
a
y
fe
You can bring a device group online to a selected node as follows:
ul Note
s
d
n
b
# cldg switch -n node_to_switch_to devgrpname
a
a
r
t
g
an non
ab
# cldg show
3.
Shut down Node 1.

The nfsds and webds disksets should automatically migrate to
Node 2 (verify with the cldg status command).
4.
Boot Node 1. Both disksets should remain mastered by Node 2.
5.
Use the cldg switch command from either node to migrate the
nfsds diskset to Node 1.
# cldg switch -n node1 nfsds
7-40

Task 9 Viewing and Managing Solaris Volume

Manager Device Groups Using Sun Cluster Manager
In this task you view and manage Solaris Volume Manager device groups
through Sun Cluster Manager. Perform the following steps on your
administration workstation or display station.
1.

node:
2.
Enter the Sun Cluster Manager Application.
3.
Open up the arrow pointing to the Storage folder on the left.
4.
From the subcategories revealed for Storage, click Device Groups.
s
) ha
y
m
omgroups
5. You will see error messages about the singleton device
c
for
e
m
d
s
each DID. This is normal.
i
ec t Gu
@
6. Click the name of your Solaris Volume
n device groups (near
b Manager
e
u
d
k
the bottom of the list).
ya Stu
(
n
7. Use the Switch Primaries
di button
htoisswitch the primary for the device
t
u
l
e
a us
group.
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

7-41
Exercise Summary
Exercise Summary

Experiences
Interpretations
Conclusions
Applications
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
7-42

Module 8
Using ZFS With Sun Cluster Software

Objectives
s
) ha
y
m
m
odata
Understand when ZFS can be used for Sun Cluster
c
e
m
d
s
i
Build ZFS storage pools and file systems ec
u
G
t
@
b den
Use ZFS root in the Sun Cluster environment
u
k
ya Stu
(
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
8-1
Relevance
Relevance

If ZFS is so powerful and so easy to configure, why would you use

any other volume manager in the Sun Cluster environment?
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
8-2

820-7458 (for Sun Cluster 3.2 11/09).
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Solaris ZFS Administration Guide , part number 819-5461

8-3
ZFS for Sun Cluster Failover Data
ZFS for Sun Cluster Failover Data

Beginning in the initial release of Sun Cluster 3.2 ZFS became available as
a failover file system for application data. You will be able to store data
only for failover applications (not scalable applications), as such, you will
be able to run the applications only on nodes physically connected to the
storage.
Using ZFS with Sun Cluster: Volume Management
s
) ha
When you use ZFS, you will generally not need any other software
volume manager (neither VxVM nor Solaris Volume Manager), nor will
you need to perform any cluster device group management.
y
m
om
c
ZFS automatically manages its storage in units calledszpools.

zpool
m uThe
ideraidz
cof mirroring.
e
layer provides optional data protection in the form
(a
G
t
@
n
b
ZFS-specific variation of RAID 5), and raidz2
e of RAID 6).
u (a variation
d
k
u
a
t
(y is S
n
i
Within a zpool you can build any
number
d e th of file systems. All the file
u
l
systems in a pool share allaof the storage
s in the pool. With ZFS, you never
m
u
a
need to worry about
the
amount
of
space available to a particular file
o
k et
g
system. Just grow
any time, and the added space will
n theepool
ns toat all
babe
automatically
available
the filesystems within.
c
a
li
b
e
u
l
k rab
a
y
fe
ul ZFS
s
d
n
Using
with Sun Cluster: Eliminating /etc/vfstab
b
a
a
r
t
g
n nEntries
on
a
b
a
The configuration database, that ZFS automatically maintains within the
pool, contains all the mount information for file systems within the pool.
You never need to create any associated vfstab entries.
8-4

Building ZFS Pools and File Systems

In the cluster, you use traditional disk paths (not DID devices) as
components of pools. The pools can still fail over even if the device paths
have different names on different nodes (the model is precisely analogous
to how VxVM has always automatically discovered disk groups).
vincent:/# zpool create marcpool mirror c1t0d0 c2t0d0
vincent:/# zpool status
pool: marcpool
state: ONLINE
scrub: none requested
config:
s
) ha
y
m
NAME
STATE
READ WRITE CKSUM
om
c
marcpool
ONLINE
0
0
0
csm Guide
e
mirror
ONLINE
0
0
0
@ ent
b
c1t0d0 ONLINE
0
0
0
u
k tud
c2t0d0 ONLINE
0
0(ya 0
n this S
i
d
lu se
errors: No known data errors
a
am to u
k
g npool
When a ZFS
nstorage
se is created, a default root file system is created
a
b
e
for theapool. The
licdefault mount point for that file system is /poolname.
b
e
u
l
kNow we
bcan create additional filesystems within the pool. ZFS
a
a
r
y
automatically
creates mount points and mounts the file systems. You do
l
e
f
u
s
d
ab -trannot need to add /etc/vfstab entries.
g
nvincent:/#
on zfs create marcpool/myfs1
a
n
b
a
vincent:/# zfs create marcpool/myfs2
vincent:/# zfs list -r

NAME
USED
marcpool
158K
marcpool/myfs1
18K
marcpool/myfs2
18K
vincent:/# df -h
Filesystem
.
.
marcpool
marcpool/myfs1
marcpool/myfs2
marcpool
AVAIL REFER
19.6G
21K
19.6G
18K
19.6G
18K
size
used
20G
20G
20G
19K
18K
18K
MOUNTPOINT
/marcpool
/marcpool/myfs1
/marcpool/myfs2
avail capacity
20G
20G
20G
1%
1%
1%
Mounted on
/marcpool
/marcpool/myfs1
/marcpool/myfs2

8-5
The mount points default to /poolname/fsname, but you can change them
to whatever you want:
vincent:/# zfs set mountpoint=/oracle marcpool/myfs1
vincent:/# zfs set mountpoint=/shmoracle marcpool/myfs2
vincent:/# df -h |grep marcpool
marcpool
20G
18K
20G
1%
/marcpool
marcpool/myfs1
20G
18K
20G
1%
/oracle
marcpool/myfs2
20G
18K
20G
1%
/shmoracle
Growing a ZFS Storage Pool
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
When you grow a ZFS storage pool, y
the
ud check that you are
aksystem
twill
(
S
using the same kind of underlying
as
in RAID
isdevices already in the pool.
d
h
t
u
While it is not recommended,
you
can
specify
pools with mixed RAID
l
e
a
s
moption
uto the zpool command.
behavior by using thea-f
o
k
t
ng ense
a
b
The following
a example
lic shows adding new mirrored disks into a mirrored
b
e
pool:
u
l
k rab
a
y
fe add marcpool mirror c1t3d0 c2t3d0
ul # zpool
s
d
n
b
a
tra
g
n
n
o
nQuotas
aba
and Reservations
You can grow a ZFS storage Pool by adding new devices any time. There
is no such thing as growing or shrinking individual ZFS file systems, and
you get way more flexibility by being able to set quotas and reservations
on the file systems, as shown in the next section.
You control the amount of space associated with each individual file
system within the pool using quotas and reservations.
The default (without quotas and reservations) is that all file systems share
all the space in the pool. Any file system can hog all the space in the pool,
and any file system can be deprived of space by another file system being
a hog.
A quota sets a maximum amount of space for a file system (to prevent it
being a hog).
8-6

A reservation sets a guaranteed amount of space for the file system. This is
just a guaranteed amount, not translating to any particular physical file
blocks.
Quotas and reservations can be grown at any time, within the bounds of
the total size of the pool. Quotas and reservations can be shrunk at any
time, as long as the current usage of a file system is below the new
lowered values. This scheme gives way more flexibility than traditional
resizing of legacy types of file systems.
The following is an example of setting a quota and a reservation on a file
system.
# zfs set quota=20g marcpool/myfs2
# zfs set reservation=6g marcpool/myfs2
s
) ha
ab
y
m
om
c
ZFS Snapshots
csm Guide
e
@ ent
b
u
ak snapshot
ZFS has an instantaneous point-in-time
tud feature.
y
(
S
in this
d
u
e any any additional space pool. If there
Initially, snapshots do not
al consume
s
m
u
are no changes to
o file system , the parent and the snapshot
kathe parent
t
g
e
point to the
ZFS implements a copy-on-write policy for
nsameedisk
nsablocks.
bain
the parent;
this
way
ZFS
snapshot
appears to grow over time due to it
c
a
i
ldifferent
b
e
u
l
occupying
the
blocks
(the
original
blocks) from the parent after
k
b
a
a
parent
l y the
er has made modifications.
f
u
s
d
n
alist
ab zfs
r
theo:/#
t
g
USED AVAIL REFER MOUNTPOINT
anNAME non
orapool
orapool/oracle
2.70G
2.70G
30.5G
30.5G
24.5K
2.70G
/orapool
/oracle
theo:/# zfs snapshot orapool/oracle@thursday_1feb09

theo:/# zfs list
NAME
orapool
2.70G 30.5G 24.5K /orapool
orapool/oracle
2.70G 30.5G 2.70G /oracle
orapool/oracle@thursday_1feb09
299K
- 2.70G The parent file system can be rolled back. If you want to roll back to a
snapshot that is not the most recent one, you must specify a -r option to
the rollback subcommand which will automatically destroy the
snapshots taken after the one you are rolling back to.
theo:/# zfs rollback orapool/oracle@thursday_1feb09

8-7
Zpool Ownership: In General and in Sun Cluster
Zpool Ownership: In General and in Sun Cluster

The commands that ZFS uses to transfer ownership of a pool from one
host to another are zpool export and zpool import. The pool is always
the unit of ownership. A new node importing a pool need never have
seen it or heard of it before; the zpool import command will
automatically scan the connected storage looking for pools:
vincent# zpool export marcpool
theo# zpool import
zpool import
pool: marcpool
id: 17715066435682450109
state: ONLINE
action: The pool can be imported using its name or numeric
identifier.
config:
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
marcpool
ONLINE
u
ak ONLINE
mirror
tud
y
(
S
c0t2d0
in thisONLINE
d
u
c1t2d0
al use ONLINE
m
ka marcpool
to
g
e
theo# zpool n
import
ba licens
a
b
u
le
k
b
a
a
r
ZFS
Failover in the Cluster
l yPool
eAutomatic
f
u
s
d
ab -tran
g
n non The Sun Cluster device group management layer does not manage ZFS
a
b
storage pools in any way.
a
Automating the failover of ZFS storage in the cluster is achieved using
application-level resources, which are covered in a later module in this
book.
8-8

ZFS Root in Sun Cluster.

Solaris 10 10/08 introduced the ability to use ZFS as the root file system.
Sun Cluster 3.2 1/09 (update 2) and later support this feature. ZFS root
has little bearing on proper cluster operations. ZFSs advantages in
simplicity and flexibility, and the ability to manage and rollback
snapshots, may make ZFS root a preferred choice even if you require
other volume managers to manage data.
Recall the /global/.devices/node@# file system must be a UFS. When
you have ZFS root it will likely reside on a lofi device using
/.globaldevices as the backing store.
s
) ha
ab
y
m
m
o
c
ZFS Root Pool and Sun Cluster

e
m
d
s
i
ec t Gu
@
nof entire disks. The
b consist
ZFS pools other than the root pool typically
e
u
d
k
a it beSbuilt
tu on top of Solaris
root pool is different and requires
ythat
(
n
partitions. This is because Solaris
di ecannot
his boot from an EFI-labeled disk.
t
u
l
a us
m
a
In typical non-cluster
k scenarios,
to the Solaris installer provisions slice 0 of
g
e
the boot disk
an tocmap
nsthe entire disk, and creates the root pool using slice
b
e
a
0. b
li
e
u
l
k
a erab
y
l
f only variation that you need to consider in Sun Cluster is the need for
u nThe
s
d
b
Volume Manager dedicated metadb partitions, if you require
a
tra Solaris
g
n
Solaris
Volume
Manager for any or all of your cluster data. Recall from
n
a
no
Module 3 that the Solaris jumpstart facility has a natural way of reserving
space for such a dedicated partition:
# example Jumpstart profile -- ZFS with
# space on s7 left out of the zpool for SVM metadb
cluster SUNWCXall
filesys c0t0d0s7 32
pool
rpool
auto
2G
2G c0t0d0s0
Currently the interactive install does not have the ability to create a
partition that is not part of the root pool. In this case, you would need
separate local disks if you required metadbs.

8-9
A disk that will be used as a mirror in the root pool would likely be
configured identically. For example, if after installation you were going to
add a disk of identical geometry to your root pool, you could copy the
partitioning of the original root pool disk and then add the mirror:
# prtvtoc /dev/rdsk/c0t0d0s2|fmthard -s - /dev/rdsk/c0t1d0s2
# zpool attach rpool c0t0d0s0 c0t1d0s0
# installboot -F zfs /usr/platform/ùname -i`/lib/fs/zfs/bootblk \
/dev/rdsk/c0t1d0s0
Note You must install the ZFS boot block manually, as shown in the last
command above, when you attach a mirror to a ZFS root pool. If you
choose to mirror the root pool as you install Solaris (using the interactive
installer or Jumpstart), the boot block will automatically be placed on both
disks.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
Typical ZFS Root Configuration
m
ka e to
g
nsis automated by the Solaris installer. There are no
Provisioning
root
baofnZFS
e
c
a
li for Sun Cluster, besides the recommendation for
brequirements
special
e
u
l
k
a eminimum
ab swap space. Note that swap space and dump space are
750MB
r
y
l
f
provisioned on ZFS volumes (zvols).
ns
bdu trautomatically
a
a
g status
n#n
zpool
o
a
n
ab pool: rpool
state: ONLINE
scrub: resilver completed after 0h4m with 0 errors on Fri Apr 10
03:34:06 2009
config:
NAME
rpool
mirror
c0t0d0s0
c0t1d0s0
STATE
ONLINE
ONLINE
ONLINE
ONLINE
READ WRITE CKSUM

0
0
0
0
0
0
0
0
0
0
0
0
errors: No known data errors
8-10

# zfs list -r rpool

NAME
rpool
rpool/ROOT
rpool/ROOT/s10s_u6wos_07b
rpool/dump
rpool/export
rpool/export/home
rpool/swap

8.84G 58.1G
94K /rpool
4.83G 58.1G
18K legacy
4.83G 58.1G 4.67G /
2.00G 58.1G 2.00G 38K
58.1G
20K /export
18K
58.1G
18K /export/home
2G 60.1G
3.56M -
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
# swap -l
in this
d
u
swapfile
dev al swaplo
e blocks free
s
m
u
/dev/zvol/dsk/rpool/swap 256,1
16
4194288 4194288
a
o
k
t
ng ense
a
b
# dumpadm
a kernellicpages
b
Dump content:
u
le
k
b
a
a
Dump
device:
/dev/zvol/dsk/rpool/dump
(dedicated)
y fer
l
u
s
Savecore
/var/crash/host01
n
bd directory:
a
a
r
t
Savecore
enabled:
yes
g onn
a
n
ab
Note The Solaris jumpstart automatically creates the /export and

/export/home file systems. If you do not want to use them, you can just
destroy them at your leisure.

8-11
ZFS Root and Snapshots

You can take snapshots of the root file system or of any other file system
within the root pool. The operation looks like any other snapshot
operation and you can give your snapshot any name you like:
# zfs snapshot rpool/ROOT/s10s_u8wos_08b@installM
There is a special procedure for rolling back snapshots of the root

filesystem:
1.
Boot or reboot the system using -F failsafe option:

ok boot -F failsafe
s
) ha
y
m
om
c
csm Guide
3. Rollback your snapshot
e
@ ent
b
4. Reboot the system
u
ak Stud
Rebooting with command: boot -F failsafe n (y
is and args: -F failsafe
di e thFile
Boot device: /pci@1f,700000/scsi@2/disk@0,0:a
u
l
a us 64-bit
SunOS Release 5.10 Version Generic_141444-09
m
a
k e toInc. All rights reserved.
Copyright 1983-2009 Sun Microsystems,
g
ns
Use is subject to license
banterms.
e
c
a
li
Hardware watchdog enabled
b
e
u
l
k
Configuring devices.
a erab
y
l
f OS instances...
Searchingdu
for installed
s
n
b
a
tra
g
n
n
ROOT/s10s_u6wos_07b
was found on rpool.
a
no
abDo
you wish to have it mounted read-write on /a? [y,n,?] y
2.
Confirm that the system mounts the root pool for you under /a
when it volunteers to do so.
mounting rpool on /a
Starting shell.
# zfs list
NAME
USED AVAIL REFER
rpool
8.84G 58.1G
94K
rpool/ROOT
4.83G 58.1G
18K
rpool/ROOT/s10s_u8wos_08b
4.83G 58.1G 4.67G
rpool/ROOT/s10s_u8wos_08b@installM 170M
- 4.64G
rpool/dump
2.00G 58.1G 2.00G
rpool/export
38K 58.1G
20K
rpool/export/home
18K 58.1G
18K
rpool/swap
2G 60.1G 3.56M
# zfs rollback rpool/ROOT/s10s_u8wos_08b@installM
# reboot
8-12
MOUNTPOINT
/a/rpool
legacy
/a
/a/export
/a/export/hom
-


Task 1 Create a ZFS Pool and File System for Cluster Data
Task 2 Create a Snapshot and then Modify your Data
Task 3 Manually Migrate your ZFS Pool to Another Node
Task 4 Examine ZFS Root and Mirror Your Root Pool
Task 1 Create a ZFS Pool and File System for

Cluster Data
s
) ha
y
m
oa m
1. Select disk drives from the shared storage to use in
new ZFS pool
c
m
for data. Select a disk from shared storage. Make
de you do
s sureuithat
c
e
Ggroups. Note the
not use any disks already in use in existing
device
t
@
n
b
e step 2). The following
logical device name (referred to k
asuc#t#d#din
u
a
t
y
example checks against VxVM,
Solaris
Volume
Manager, and ZFS
( is S
n
i
disks.
lud se th
a
m twith
udisks that may already be in use with
acareful
o
k
Be especially
g nManager.
nVolume
se
Solaris
The metaset command lists their DID
a
b
e
c
a
i
l
device
name.
Make
sure
you
understand the c#t#d# associated with
b
e
u
l
k
a ethose
ab DID devices and choose other disks for ZFS. Consult with your
r
y
l
f
instructor if you are unsure.
bdu trans # vxdisk -o alldgs list
a
ang non
# zpool status
ab
# metaset
# cldev list -v
2.
From one node connected to the storage, create a zpool that provides
mirroring for your two disks:
# zpool create nfspool mirror c#t#d# c#t#d#
# zpool status
3.
Create a ZFS file system within the pool:

# zfs create nfspool/nfs
# zfs list -r nfspool
# df -h
4.
Change the mount point.

# zfs set mountpoint=/testnfs nfspool/nfs
# df -h

8-13

5.
Populate your file system with some simple files
# cd /testnfs
# touch file1 file2 file3
# vi letter_to_mom
Hi Mom, ZFS is much better than that legacy stuff
Task 2 Create a Snapshot and then Modify your Data

1.
Snapshot your ZFS file system:

# zfs snapshot nfspool/nfs@SAVEMYWORK
# zfs list -r nfspool
y
m
om
2. Modify the content of your file system:
c
# cd /testnfs
csm Guide
e
# touch file4
@ ent
b
u
# rm file2
ak Stud
y
(
# vi letter_to_mom
din ethan
histhat legacy stuff
Hi Mom, ZFS is muchlubetter
t
a usabout my class
Here is more information
m
a
k e to
g
an cens
b
a
li Migrate your ZFS Pool to Another
Task 3uManually
b
e
l
k
a erab
Node
y
l
u nsf
d
b
a
tr1.a Export your pool:
g
n
n
no
aba
# cd /
s
) ha
# zpool export nfspool

# zpool status
# zfs list
2.
On the other node that is connected to storage, import the pool:

#
#
#
#
#
3.
Recover the snapshot that had migrated along with the pool:
#
#
#
#
8-14
zpool import
zpool import nfspool
zpool status
zfs list -r nfspool
df -h
cd /
zfs rollback nfspool/nfs@SAVEMYWORK
cd /testnfs
ls

Task 4 Examine ZFS Root and Mirror Your Root Pool
1.
Examine your ZFS root configuration on one of your cluster nodes.

#
#
#
#
2.
zpool status
zfs list -r rpool
swap -l
dumpadm
Find another unused local disk (same controller as your root disk).
Consult your instructor if necessary.
# format
3.
Modify the partitioning of your second disk to match that of the first
disk, assuming they are of identical geometry:
s
) ha
ab
y
m
om(argument
Warning make sure the first disk in the command below
to
c
e
m
prtvtoc) is the original root disk, and the second
one (argument
to
csthis
uid you
e
G
fmthard) is your new intended mirror. If you
get
backwards
will
t
@
n
b
have trouble!
ku tude
a
y
S
n| (fmthard
s
i
i
d
h
# prtvtoc /dev/rdsk/origdisk-c#t#d#s0
-s - /dev/rdsk/newdiskt
u
l
e
a
c#t#d#s0
am to us
k
4. Add the
to
ngmirror
seyour root pool. Make sure you use slice 0, and not
a
n
b
e
slice
2.
a
lic
b
e
u
l
k #razpool
b attach rpool origdisk-c#t#d#s0 newdisk-c#t#d#s0
a
y
#
zpool
status
l
e
f
u
s
d
ab -tran5. Install the boot block on your second disk:
g
an# installboot
-F zfs /usr/platform/ùname -i`/lib/fs/zfs/bootblk \
non
/dev/rdsk/c#t#d#s0

8-15
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
8-16

Module 9
Managing the Public Network With IPMP

Objectives
s
) ha
y
m
om
Define the purpose of IPMP
c
m uide
Define the concepts of an IPMP group ecs
t Gon a single
@
n
b
List examples of network adaptersuin
IPMP e
groups
ak Stud
Solaris OS server
y
(
inthe in.mpathd
d
his daemon
t
Describe the operation
of
u
l
e
a us
m
a
List the options
to
k ethetoifconfig command that support IPMP and
g
configure
IPMP using /etc/hostname.xxx files
ban licens
a
b Configure
e IPMP manually with ifconfig commands
u
l
k
b
ra a forced failover of an adapter in an IPMP group
ya fePerform
l
u
s
bd tran Describe the integration of IPMP into the Sun Cluster software
a
environment
ang non
ab
9-1
Relevance
Relevance

Should you configure IPMP before or after you install the Sun
Cluster software?
Why is IPMP required even if you do not have redundant network

adapters?
Is the configuration of IPMP any different in the Sun Cluster

environment than on a stand-alone Solaris OS?
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
9-2
Is the behavior of IPMP any different in the Sun Cluster

environment?

819-2971
Sun Cluster Software Installation Guide for Solaris OS, part number 8192970
Sun Cluster Concepts Guide for Solaris OS, part number 819-2969
Solaris Administration Guide: IP Services, part number 816-4554
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

9-3
Introducing IPMP
Introducing IPMP
IPMP has been a standard part of the base Solaris OS since Solaris 8 OS
Update 3 (01/01).
IPMP enables you to configure redundant network adapters, on the same
server (node) and on the same subnet, as an IPMP failover group.
The IPMP daemon detects failures and repairs of network connectivity for
adapters, and provides failover and failback of IP addresses among
members of the same group. Existing TCP connections to the IP addresses
that fail over as part of an IPMP group are interrupted for short amounts
of time without data loss and without being disconnected.
s
) ha
y
m
In the Sun Cluster 3.2 software environment, you must use IPMP
om to
c
manage any public network adapters on which you will

e the IP
mbe placing
d
s
i
c
u
addresses associated with applications running inethe cluster.
G
t
@
n
b
e
ku tubydmechanisms
a
These application IP addresses are implemented
known
y
(
S
n
s
as LogicalHostname and SharedAddress
The configuration of
dipublic
hiresources.
t
u
l
these resources requires that
the
network
adapters
being used are
e
a us
m
under the control ofkIPMP.
9, Introducing Data Services,
a Module
to and
g
e
Resource Groups,
and
HA-NFS,
Module 10, Configuring Scalable
n ens
a
b
Services and
Relationships, provide detail
a Advanced
lic Resourceof Group
b
e
about
the
proper
configuration
applications
and their IP addresses in
u
l
k
b
a
a
r
software environment.
l ythe Sun
eCluster
f
u
s
d
ab -tran
g
an non
ab
9-4

Describing General IPMP Concepts

IPMP allows you to group together network adapters for redundancy.
Members of the same IPMP group are identified by a common group
name. This can be any alphanumeric name. The group name is
meaningful only inside a single Solaris OS server.
Defining IPMP Group Requirements

You must observe the following rules when configuring an IPMP group:
s
) ha
A network adapter can be a member of only one group.
When both IPv4 and IPv6 are configured on a physical adapter, the
group names are always the same (and thus need not be specified
explicitly for IPv6).
y
m
om
c
csm Guide
e
All members of a group must be on the
subnet.
t The members,
n
b@same
e
if possible, should be connectedktouphysically
separate
switches on
d
u
a
t
y
the same subnet.
n ( this S
i
d
lu subnets
Adapters on different
e must be in different groups.
a
s
m
u
kamultiple
You can have
togroups on the same subnet.
g
e
s
You
angroup with only one member. However, there is no
bacann have
e
c
a
li and there is no automatic failover.
b redundancy
e
u
l
k
a eEach
abEthernet adapter must have a unique MAC address. This is
r
y
l
f
bdu trans achieved by setting the local-mac-address? variable to true in the
a
g onOpenBoot PROM. This is automatically set correctly by
n
a
n
b
scinstall.
a
Network adapters in the same group must be of the same type. For
example, you cannot combine Ethernet adapters and Asynchronous
Transfer Mode (ATM) adapters in the same group.
In Solaris 9 OS, when more than one adapter is in an IPMP group,

each adapter requires a dedicated test IP address, or test interface.
This is an extra static IP for each group member specifically
configured for the purposes of testing the health of the adapter using
ping traffic.
The test interface enables test traffic on all the members of the IPMP
group. This is the reason that local-mac-address?=true is
required for IPMP.

9-5
Solaris 10 OS does not require test addresses, even with multiple

adapters in a group. If you configure adapters in Solaris 10 OS IPMP
groups without test addresses, the health of the adapter is
determined solely by the link state of the adapter.
Note Using IPMP without test addresses reduces network traffic and
reduces the administrative strain of allocating the addresses. However, the
testing is less robust. For example, this author has personally experienced
adapters with a valid link state but broken receive logic. Such an adapter
would be properly faulted using test addresses. The remaining examples
in the module focus on creating IPMP configurations with test addresses.
s
) ha
If both IPv4 and IPv6 are configured on the same adapters, it is

possible to have test addresses for both IPv4 and IPv6, but it is not
necessary. If a failure is detected (even if you have only an IPv4 test
address), all IPv4 and IPv6 addresses (except the test address) fail
over to the other physical adapter.
y
m
om
c
csm Guide
e
@ ent
b
u
If you do choose to have an IPv6 test
address,
k tudthat test address will
a
y
( is(theSaddress automatically
always be the link-local IPv6naddress
i
d
assigned to the adapter,
only
th be used on the local subnet).
uthat can
l
e
a
s IPV6 (Optional) on page 9-16 to see
See Configuring m
Adapters u
for
a
o
k
t
how to set up
g IPv6nsoneyour adapters.
n
a
ab lice
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab
9-6

Configuring Standby Adapters in a Group

In an IPMP group with two or more members, you can configure an
adapter as a standby adapter for the group. Standby adapters have the
following properties:
You are allowed (and must) configure only the test interface on the
adapter. Any attempt to manually configure any other addresses
fails.
Additional IP addresses are added to the adapter only as a result of a

failure of another member of the group.
The standby adapter is preferred as a failover target if another

member of the group fails.
s
) ha
y
m
om
c
csm Guide
e
t
n
b@groups
e
u
Note The examples of two-memberkIPMP
in
d the Sun Cluster
u
a
t
y
software environment will not use
any standby
S adapters. This allows the
n (additional
s
i
i
d
h
Sun Cluster software to balance
IP addresses associated with
u se t
l
a
applications across both
u of the group.
am members
o
k
t
g nse IPMP group in the Sun Cluster software
If you had
anthree-member
a
b
ce up one of the members as a standby would still be a
a
i
l
environment,
setting
b
u option.
le
k
b
valid
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
You must have at least one member of the group that is not a
standby adapter.

9-7
Examining IPMP Group Examples

The following sections describe and illustrate examples of network

adapters in IPMP groups on a single Solaris OS server.
Single IPMP Group With Two Members and No

Standby
Figure 9-1 shows a server with two member adapters in a single IPMP
group. These two adapters must be on the same subnet and provide
failover for each other.
s
) ha
y
m
om
c
Group:
csm Guide
therapy
e
@ ent
b
u
qfe0
qfe4 ak
tud
y
(
S
in this
d
u
e
al failover
s
mutual
m
u
ka e to
g
ban licens
a
Solaris OS Server
b
e
u
l
k
b
ya fera
l
u
bd trans
a
g on- Figure 9-1 Server With Two Member Adapters in a Single IPMP Group
n
a
n
ab
9-8

Single IPMP Group With Three Members Including a

Standby
Figure 9-2 shows a server with three members of a single IPMP group. In
the example, one of the adapters in the group is configured as a standby.
Group:
therapy
qfe0
qfe4
qfe8 (standby)
s
) ha
y
m
om
failover
c
csm Guide
e
Solaris Operating System
@Server ent
b
u
ak Stud
y
(
din Members
his of a Single IPMP Group
Figure 9-2 Server WithluThree
t
e
a us
m
a
k e to
g
s Different Subnets
Two IPMPbGroups
an cenon
a
li
b
e
u
l
k
a Figure
a9-3bshows how different IPMP groups must be used for adapters
r
y
l
e
f
bdu tranons different subnets.
a
g onn
a
n
ab
Group:
therapy
qfe0
qfe4
qfe1
Group:
mentality
qfe5
Solaris OS Server
Figure 9-3
Two IPMP Groups on Different Subnets

9-9
Two IPMP Groups on the Same Subnet
Figure 9-4 shows two different IPMP groups configured on the same
subnet. Failover still occurs only within each particular group.
Group:
therapy
qfe0
qfe4
Group:
mentality
qfe1
qfe5
s
) ha
y
m
Solaris OS Server
om
c
csm Guide
e
@ ent
b
u
k Subnet
aSame
tud
Figure 9-4 Two IPMP Groups on (the
y
S
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
9-10

Describing IPMP
Describing IPMP
The in.mpathd daemon controls the behavior of IPMP. This behavior can
be summarized as a three-part scheme:
Network path failure detection
Network path failover
Network path failback
The in.mpathd daemon starts automatically when an adapter is made a

member of an IPMP group through the ifconfig command.
s
) ha
ab
y
m
om
c
de
The following paragraphs describe the functionality
csmofGtheuiin.mpathd
e
daemon in a configuration using test addresses.
@ Inenthet Solaris 10 OS,
b
u
without test addresses, adapter failure
akdetection
tudand repair is based solely
y
(
S
on the link state of the adapter.
in this
d
u
ein.mpathd daemon sends Internet
alused,uthe
s
When test addresses
are
m
a
o
kProtocol
t(ICMP)
Control Message
echo probes (pings) to the targets
g
e
n
s
a
n
connected
the link
all adapters that belong to a group to detect
ce Theontest
abandtorepair.
i
l
failures
address is used as the source address of these
b
e
u
l
k
b
ra
ya pings.
l
e
f
u
s
bd tran
a
- Note Using test addresses, even in Solaris 10, is more robust. You can
ang non
Network Path Failure Detection
have broken adapters, ports, or cables and still have a valid link state.
The in.mpathd daemon automatically chooses targets in the following

order:
1.
2.
Targets are chosen from the routing table in memory in the following
order:
a.
host
b.
network
c.
default
If no targets are discovered through the routing table, the targets are
discovered by a ping command to the 224.0.0.1 (all-hosts) multicast
IP address.

9-11
Describing IPMP
To ensure that each adapter in the group functions properly, the

in.mpathd daemon probes all the targets separately through all the
adapters in the multipathing group, using each adapters test address. If
there are no replies to five consecutive probes, the in.mpathd daemon
considers the adapter as having failed. The probing rate depends on the
failure detection time (FDT). The default value for failure detection time is
10 seconds. For a failure detection time of 10 seconds, the probing rate is
approximately one probe every two seconds.
You might need to manipulate the choice of ping targets. For example,
you might have default routers that are explicitly configured not to
answer pings.
as
h
One strategy is to enter specific static host routes to IP addresses that are
)
y the
on the same subnet as your adapters. These will then be chosen first
as
m
targets. For example, if you wanted to use 192.168.1.39 and

om
c
192.168.1.5 as the targets, you could run these commands:

csm Guide
e
# route add -host 192.168.1.39 192.168.1.39
t
@ e-static
n
b
u
# route add -host 192.168.1.5 192.168.1.5
ak Stud-static
y
(
ianboot tscript.
d
his The Solaris Administration
You can put these commands
in
u
l
e
a us listed in the resources section at the
Guide: IP Services fromadocs.sun.com
m
k suggests
to making a boot script
beginning of thisgmodule
e
/etc/rc2.d/S70ipmp.targets.
This works on both Solaris 9 and Solaris
ban licens
a
10. ub
le
k
b
a
a
l y sfer
u
d
an Path Failover
abNetwork
r
t
g
n non
a
b
a
After a failure is detected, failover of all network access occurs from the
failed adapter to another functional adapter in the group. If you have
configured a standby adapter, the in.mpathd daemon chooses that for
failover of IP addresses and multicast memberships. If you have not
configured a standby adapter, in.mpathd chooses the adapter with the
least number of IP addresses. The new adapter assumes all of the IP
addresses of the failed adapter, except for the test address.
Network Path Failback

The in.mpathd daemon detects if the failed path has been repaired. At
this time the IP addresses moved during the failover will be returned to
their original path, assuming failback is enabled.
9-12

Configuring IPMP
Configuring IPMP
This section outlines how IPMP is configured through options to the
ifconfig command.
You will specify the correct IPMP-specific ifconfig options in the
/etc/hostname.xxx files. After these are correct, you never have to
change them again.
Examining ifconfig Options for IPMP
ab
as
h
Several new ifconfig options have been added for use with IPMP.) They
y
are the following:
m
om in a
c
group groupname Adapters on the same subnet

are placed
m group
idename. The
failover group by configuring them withe
the
cssame
u
Gonly within a
t
@
group name can be any name and must
be unique
n
b
e
ku tifud
single Solaris OS (node). It is irrelevant
you have the same or
a
y
S nodes of a cluster.
different group names across
different
n ( the
s
i
i
d
h
t
lu option
e
-failover Useathis
to demarcate the test address for each
s
minterface
u is the only interface on an adapter that
atest
adapter. The
o
k
t
ngfaileover
seto another adapter when a failure occurs.
doesanot
n
b
a
lic This option, while not required, is typically used on
deprecated
b
e
u
l
k the
a
abtest interfaces. Any IP address marked with this option is not
r
y
l
e
f
used as the source IP address for any client connections initiated
bdu trans from
a
this machine.
ang non
standby This option, when used with the physical adapter

(-failover deprecated standby), turns that adapter into a
standby-only adapter. No other virtual interfaces can be configured
on that adapter until a failure occurs on another adapter within the
group.
addif Use this option to create the next available virtual interface
for the specified adapter. The maximum number of virtual interfaces
per physical interface is 8192.
removeif Use this option to remove a virtual interface, by just

giving the IP address assigned to the interface. You do not need to
know the virtual interface number.

9-13
Configuring IPMP
Putting Test Addresses on Physical or Virtual

Interfaces
IPMP was designed so that the test address for each member of an IPMP
group can be either on the physical interface (for example, qfe0) or on a
virtual interface (for example, qfe0:1, created with the
ifconfig xxx addif command).
There were at one time many existing IPMP documents written with the
test IP on the physical interface because it looks cleanest to have only the
virtual interfaces failing over to other adapters and the IP on the physical
interface always staying where it is.
s
) ha
y
m
om
c
csm Guide
e
t specifically
@applications,
Note The bug concerns failure of certainuRPC
n
b
e
ak S
when the deprecated flag is on the physical
adapter.
tud You should always
y
(
use the deprecated flag with test
inaddresses;
is therefore, you should ensure
d
thadapter.
u
the test addresses are not a
onl the physical
The examples in this
e
s
m
u
module follow the convention
kathe choice.
toof placing the test interface on a virtual
g
e
interface, whenngiven
ba licens
a
b
u
le
k
b
a
a
l y ifconfig
er
f
Using
Commands to Configure IPMP
u
s
d
b tran
a
g onn
a
While you will configure your /etc/hostname.xxx files once and never
n
ab
The convention changed due to a bug relating to having a deprecated
flag on the physical adapter. Refer to bug #4710499 for more information.
have to worry about IPMP configuration again, these examples show

using the ifconfig command directly with the IPMP-specific options.
The /etc/hostname.xxx files still need to exist for subsequent reboots.
This /etc/hosts fragment shows the dedicated IP address for the test
interface for each adapter. While it is not required to have host names for
these, it is a good practice to put names in /etc/hosts to keep track of
the addresses.
# cat /etc/hosts
.
#physical host address
172.20.4.192
vincent
# test addresses for vincent (node 1)
172.20.4.194
vincent-qfe1-test
9-14

Configuring IPMP
172.20.4.195
vincent-qfe2-test
# ifconfig qfe1 vincent group therapy netmask + broadcast + up

# ifconfig qfe1 addif vincent-qfe1-test -failover deprecated \
netmask + broadcast + up
# ifconfig qfe2 plumb
# ifconfig qfe2 vincent-qfe2-test group therapy -failover deprecated \
# ifconfig -a
.
qfe1: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> mtu 1500 index 2
inet 172.20.4.192 netmask ffffff00 broadcast 172.20.4.255
groupname therapy
ether 8:0:20:f1:2b:d
qfe1:1:flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAI
LOVER> mtu 1500 index 2
qfe2:flags=9040843<UP,BROADCAST,RUNNING,MULTICAST,DEPRECATED,IPv4,NOFAILO
VER> mtu 1500 index 3
groupname therapy
ether 8:0:20:f1:2b:e
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer the /etc/hostname.xxx Files for IPMP
u
Configuring
d
ab -tran
g
n
n# catno/etc/hostname.qfe1
a
b
a
vincent group therapy netmask + broadcast + up

addif vincent-qfe1-test -failover deprecated netmask + broadcast + up
# cat /etc/hostname.qfe2
vincent-qfe2-test group therapy -failover deprecated \

9-15
Configuring IPMP
Configuring Adapters for IPV6 (Optional)
If you want to support application-specific IP addresses that use IPv6, you

must manually configure your adapters for IPv6.
You can choose to configure IPV6 with or without an IPV6 test address.
You already have an IPV4 test address, so you do not need to also have an
IPV6 test address. If you do not want an IPV6 test address, create empty
file names for the IPV6 interfaces. Upon reboot, IPV6 is enabled on the
interfaces with the same group name as the IPV4 interfaces, without a test
address.
# touch /etc/hostname6.qfe1
# touch /etc/hostname6.qfe2
# init 6
s
) ha
y
m
om
c
Alternatively, you can choose to use an IPV6 test address.

de
sm Theuionly
c
e
address that can serve this purpose is the link-local address,
G which is
t
@
n
b
automatically assigned to the interface and
can
be
used
only
on the local
u ude
k
a
t
subnet; therefore, you can still have(very
y simple
S IPV6 adapter
n
s
i
i
configuration files:
lud se th
a
# vi /etc/hostname6.qfe1
am to u
-failover up g k
n ense
a
# vi /etc/hostname6.qfe2
b
a up lic
b
-failover
u
le
k
b
a
#
init
6
a
l y sfer
u
d
ab -tran
g
n nUsing
on IPV6 Test Address Only
a
b
a
It is possible to configure IPV6 test addresses only; that is, configuring
IPV4 without any test addresses and then configuring IPV6 with test
addresses, as in the example in the previous section.
The advantage is do not have to use any space in your IPV4 public
network address range for test addresses.
The disadvantage is you would have to have some external machines
outside the cluster also configured with IPV6 to answer the ping.
9-16

Configuring IPMP
The in.mpathd Configuration File

The in.mpathd daemon uses the settings in the /etc/default/mpathd
configuration file to invoke multipathing. Changes to this file are read by
the in.mpathd daemon at startup and on a SIGHUP. This file contains the
following default settings and information:
#
# Time taken by mpathd to detect a NIC failure in ms.
# The minimum time that can be specified is 100 ms.
#
FAILURE_DETECTION_TIME=10000
#
# Failback is enabled by default. To disable failback
# turn off this option
#
FAILBACK=yes
#
# By default only interfaces configured as part of
# multipathing groups are tracked. Turn off this
# option to track all network interfaces on the system
#
TRACK_INTERFACES_ONLY_WITH_GROUPS=yes
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b Generally,
Note
u
le you do not need to edit the default
k
b
a
a
/etc/default/mpathd
configuration file.
y fer
l
u
s
bd tran
a
g on- The three settings you can alter in this file are:
n
a
n
ab
FAILURE_DETECTION_TIME You can adjust the value of this

parameter. If the load on the network is too great, the system cannot
meet the failure detection time value. Then the in.mpathd daemon
prints a message on the console, indicating that the time cannot be
met. It also prints the time that it can meet currently. If the response
comes back correctly, the in.mpathd daemon meets the failure
detection time provided in this file.
FAILBACK After a failover, failbacks take place when the failed

adapter is repaired. However, the in.mpathd daemon does not fail
back the addresses if the FAILBACK option is set to no.
TRACK_INTERFACES_ONLY_WITH_GROUPS In stand-alone servers,

you can set this to no so that in.mpathd monitors traffic even on
adapters not in groups. In the cluster environment make sure you
leave the value yes so that private transport adapters are not probed.

9-17
Performing Failover and Failback Manually
Performing Failover and Failback Manually

You can do a forced failover of an adapter in an IPMP group. The

following is an example:
# if_mpadm -d qfe1
This command causes IPMP to behave as if qfe1 had failed. All IP
addresses, except the test interface, are failed over to another member of
the group. The adapter is marked down and will not be accessed by
in.mpathd until reenabled.
To reenable an adapter after this operation, use:
s
) ha
y
m
om
c
This command allows the adapter to take back the IPsaddresses

idtoewhich
cis m
e
it was originally assigned, assuming FAILBACK=yes
set G
in u
the
@ ent
b
/etc/default/mpathd file.
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
# if_mpadm -r qfe1
9-18

Configuring IPMP in the Sun Cluster 3.2 Environment

There are no special cluster-related tools for configuring IPMP in the
Sun Cluster 3.2 environment. IPMP is configured on each node exactly as
in a nonclustered environment.
Configuring IPMP Before or After Cluster Installation

If you have /etc/hostname.xxx files without IPMP groups, scinstall
automatically rewrites the files such that each adapter is a member of an
IPMP group, as in the following example:
s
) ha
ab
y
m
om
c
ide by hand
You still need to correctly customize multi-adapter
IPMP u
groups
csm
e
Gscinstall uses
t
to use the appropriate test addresses. Thebfile
created
by
@
n
deas the test address, but
kuaddress)
the data address (for example, the node
u
a
t
y
( iTos customize
S
you will want a different test n
address.
your configuration,
i
d
h
t
you can choose to wait until
scinstall,
as you are doing in this
u after
e
albe
s
module (where youm
will
modifying
some
files
rewritten by
u
a
o
k
t
scinstall), or
g youncan
econfigure IPMP before scinstall.
n
s
a
ceto make IPMP configuration part of your Solaris OS
ab choose
i
l
b
You
might
e
u
lsoftware
k
b
a
JumpStart
installation, by copying in the correct
a
y fer
l
u
/etc/hostname.xxx
files as part of a JumpStart software finish script.
s
bd tran
a
ang non
vincent:/# cat /etc/hostname.qfe1
vincent group sc_ipmp0 -failover
Using Same Group Names on Different Nodes
It makes no difference to IPMP whether you use the same group names or
different group names for IPMP across nodes. IPMP itself is aware only of
what is going on within the local Solaris OS.
It is a helpful convention to use the same group names on all nodes that
are connected to the same subnet.

9-19
Understanding Standby and Failback
It is unlikely that you want to use a standby adapter in a two-member

IPMP group in the cluster environment. If you do, the Sun Cluster
software will be unable to load-balance additional application-related IP
addresses across the members of the group.
Keep FAILBACK=yes in the /etc/default/mpathd file. Sun Cluster
software is automatically trying to load-balance additional IP addresses
across the members of the group. It makes sense that you want them to
rebalance when a repair is detected.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
9-20

Integrating IPMP Into the Sun Cluster 3.2 Software Environment
Integrating IPMP Into the Sun Cluster 3.2 Software

Environment
There is nothing special about IPMP itself within the cluster. In the cluster
environment, as in a noncluster environment, in.mpathd is concerned
with probing network adapters only on a single Solaris OS, and manages
failovers and failbacks between the adapters in a group.
The Sun Cluster software environment, however, needs additional
capability wrapped around IPMP to do the following:
Make the status of IPMP groups on each node available throughout

the cluster
s
) ha
y
m
om
c
csm Guide
e
t
@ cluster
n
bthese
e
u
Clearly, IPMP itself is unaware of anykof
d requirements.
u
a
t
y
Instead, the Sun Cluster 3.2 software
uses
a
cluster-specific
public
( is S
n
i
network management daemon
thto perform this cluster integration.
lud s(pnmd)
e
a
am to u
k
nofgthe
se Daemon in Sun Cluster 3.2
Capabilities
pnmd
a
n
b
e
a
lic
Software
b
e
u
l
k rab
a
y
fe cluster environment, the pnmd daemon has the following
ul nInsthe
d
b
a
tra capabilities:
g
n
n
no
aba
Communicate with pnmd on other nodes to transmit the status of
Facilitate application failover in the case where all members of an

IPMP group on one node have failed, but the corresponding group
on the same subnet on another node has a healthy adapter
IPMP groups
Facilitate application failover
When pnmd detects that all members of a local IPMP group have failed, it
consults a file named /var/cluster/run/pnm_callbacks. This file
contains entries that would have been created by the activation of
LogicalHostname and SharedAddress resources. (There is more
information about this in Module 9, Introducing Data Services, Resource
Groups, and HA-NFS, and Module 10, Configuring Scalable Services
and Advanced Resource Group Relationships.)

9-21
It is the job of the hafoip_ipmp_callback, in the following example, to

decide whether to migrate resources to another node.
# cat /var/cluster/run/pnm_callbacks
therapy orangecat-nfs.mon \
/usr/cluster/lib/rgm/rt/hafoip/hafoip_ipmp_callback mon nfs-rg orangecatnfs
Summary of IPMP Cluster Integration

Figure 9-5 summarizes the IPMP and pnmd elements of public network
management in the cluster.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
in.mpathd
akin.mpathd
tud
y
(
S
in pnmdthis
d
pnmd
u
al use
m
ka e to
g
hafoip_ipmp_callback
n ens
baresources
a
failover
lic
b
e
u
l
k rab
a
y
e
f9-5
ul Figure
s
d
IPMP Cluster Integration
n
b
a
a
r
t
g onn
a
n
ab
9-22

Viewing Cluster-wide IPMP Information With the

clnode status -m Command
In the Sun Cluster 3.2 software environment, running the clnode status
-m command from any node shows the status of IPMP group members on
all the nodes.
# clnode status -m
--- Node IPMP Group Status --Node Name
--------vincent
vincent
theo
theo
Group Name
---------therapy
therapy
therapy
therapy
Status
-----Online
Online
Online
Online
Adapter
------qfe2
qfe1
qfe2
qfe1
Status
-----Online
Online
Online
Online
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

9-23
Exercise: Configuring and Testing IPMP

Perform the following tasks on all cluster nodes. It is assumed that the
Sun Cluster software is installed and operational, and that the only IPMP
configuration is the single-adapter group called sc_ipmp0 automatically
created by scinstall.
Task 1 Verifying the local-mac-address? Variable
Task 2 Verifying the Adapters for the IPMP Group
Task 3 Verifying or Entering Test Addresses in the /etc/hosts File
s
) ha
y
m
Task 5 Rebooting and Verifying That IPMP Is Configured

om
c
e
Task 6 Verifying IPMP Failover and Failbackcsm uid
e
G
t
@
n
b
ku tude
a
y
Preparation
n ( this S
i
d
lu se
a
No preparation is required
am fortothisu exercise.
k
ng ense
a
b
b a le licthe local-mac-address? Variable
Task 1kuVerifying
a erab
y
l
u nsf
d
b
Perform
the following steps on each node in the cluster:
a
a
r
t
g
n non 1. Verify that the EEPROM local-mac-address? variable is set to
a
b
a
Task 4 Creating /etc/hostname.xxx Files
true:
# eeprom "local-mac-address?"
The variable is set to true by the scinstall utility at cluster
installation time, so the only reason it should be false now is if it
were changed manually. If it needs to be changed to true, do so
now.
9-24

Task 2 Verifying the Adapters for the IPMP Group

1.
Make sure you know which are the redundant adapters on the
public network. You might have already written this down in the
exercises for Module 3, Preparing for Installation and
Understanding Quorum Devices.
2.
Your primary public network adapter should be the only one

currently configured on the public net. You can verify this with:
# ls -l /etc/hostname.*
# ifconfig -a
s
) ha
ab
y
m
a. Making sure it is not configured as a private transport

om
c
b. Making sure it can snoop public network

idetraffic:
csmbroadcast
u
e
G
# ifconfig ifname plumb b@
t
n
ku tude
# snoop -d ifname (ya
S
npingth-sis pubnet_broadcast_addr
i
(other window or node)#
d
lu se
a
am to u
k
ng enorseEntering Test Addresses in the
Task 3 Verifying
a
b
a File
lic
b
/etc/hosts
e
u
l
k rab
a
y
ul nItsisfea good idea, although not required, to have test IP addresses listed in
d
b
a
tra the hosts file. While one node does not need to know anything about
g
n
n
a
no
3.
You can verify your secondary public network adapter by:
another nodes test addresses, it is advisable to have all test addresses in

the hosts file for all nodes to indicate that these addresses are reserved
and what they are reserved for.
1.
Verify with your instructor which IP addresses should be used for

the test interfaces for each adapter on each node.

9-25
2.
Enter the IP addresses in /etc/hosts on each node if they are not

already there.
The following is only an example. It really does not matter if you use
the same names as another group working on another cluster if the
IP addresses are different:
# IPMP TEST ADDRESSES
172.20.4.194
vincent-qfe1-test
172.20.4.195
vincent-qfe2-test
172.20.4.197
theo-qfe1-test
172.20.4.198
theo-qfe2-test
Task 4 Creating /etc/hostname.xxx Files
s
) ha
y
m
Perform the following task:

om
c
sm uide
c
e
On each node, create the appropriate /etc/hostname.xxxG
to place
@ ent files
adapters in IPMP groups. The group nameuisbunimportant.
The following
ak that
tuared configured on your
are just examples, so use the adapter(ynames
S
in this
public network.
d
u
al use
m
ka utility
toshould have already modified the
g
e
Note The scinstall
ns(one of the two below) that already existed at the
ban licefile
/etc/hostname.xxx
a
b
u
le was used to configure your cluster. Your adapter was
time
scinstall
k
b
a
a
in
l yplaced
eran IPMP group called sc_ipmp0. You will be completely
f
u
s
d
this file to place the adapter in a real, multi-adapter IPMP
an
ab -troverwriting
g
group,
as
in
the
examples. While scinstall places the -failover flag on
an non
ab
the node address, you will be using it only with dedicated test addresses.
# vi /etc/hostname.qfe1
vincent group therapy netmask + broadcast + up
addif vincent-qfe1-test -failover deprecated netmask + broadcast + up
# vi /etc/hostname.qfe2
vincent-qfe2-test group therapy -failover deprecated \
9-26

Task 5 Rebooting and Verifying That IPMP Is

Configured
The following steps can be performed on one node at a time, so that the
cluster stays active the whole time:
1.
Reboot the node.
2.
Verify the new IPMP configuration with ifconfig -a.
3.
Verify IPMP cluster-wide status with clnode status -m.
Task 6 Verifying IPMP Failover and Failback
s
) ha
y
m
Perform the following steps on at least one of your nodes: m

co eand
keep it
m
1. From outside of the cluster, launch ping -ssnodename,
d
i
ec t Gu
running.
@
n
e
ubcluster
d
k
2. If you have physical access to a
your
hardware,
unplug the
u
t
y
(
S
Ethernet cable from the inetwork
n thadapter
is that currently has the node
d
physical interface on
it.
u
al use
m
a physical
If you havekno
to access, you can sabotage your adapter with:
g
e
# b
ifconfig
modinsert ldterm@2
an ceadapter_name
ns
a
i
l
b Observe
3.
u
le the node messages (on the console or in the
k
b
a
a
/var/adm/messages
file).
y fer
l
u
s
bd tran4. Observe the output of the clnode status -m command.
a
g onn
a
5. Observe the behavior of your command from step 1 (keep it
n
ab
running).
If you have physical access, reattach the broken cable. If you have no
physical access, use the following to repair your sabotage:
# ifconfig adapter_name modremove ldterm@2
6.
Observe the messages and the behavior of your command from

step 1.
7.
Observe the output of the clnode status -m command.

9-27
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
9-28

Module 10
Introducing Data Services, Resource

Groups, and HA-NFS
s
) ha
Objectives
ab
y
m
om
c
iindea cluster to
csmservice
u
Describe how data service agents enable aedata
G
t
@
operate properly
n
b
ku tude
a
y
List the components of a data
service
( is Sagent
n
i
d e thinstallation, and registration
Describe data service
lupackaging,
a
m topurpose
us of resource groups
aprimary
Describe the
k
ng ebetween
se failover and scalable data services
a
n
Differentiate
b
a
lic
b
e
u
l
k Describe
b how to use special resource types
a
a
r
y
ul nsfeList the components of a resource group
d
b
a
tra
Differentiate between standard, extension, and resource group
g
n
n
a
no
properties
Register resource types
Configure resource groups and resources
Perform manual control and switching of resources and resource

groups
10-1
Relevance
Relevance

What is a data service agent used for in the Sun Cluster 3.2
software?
What is involved in fault monitoring a data service?
What is the implication of having multiple applications in the same

failover resource group?
How do you distinguish the resources representing different

instances of the same application?
y
m
om
What are the specific requirements for setting up NFS
in
the Sun
c
e
m
Cluster 3.2 software environment?
d
s
i
ec t Gu
@
b den
u
k
ya Stu
(
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
10-2

820-7458 (for Sun Cluster 3.2 11/09).
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Introducing Data Services, Resource Groups, and HA-NFS

10-3
Introducing Data Services in the Cluster

The Sun Cluster 3.2 software framework makes applications HA,
minimizing application interruptions after any single failure in the cluster.
In addition, some applications, such as Apache web server software, are
supported not only with HA features, but also in a scalable configuration.
This configuration allows the service to run on multiple nodes of the
cluster simultaneously while providing a single IP address to the client.
Solaris 10 OS Non-Global Zones Act as Virtual Nodes

a
s
a
for Data Services
)h
y
m
m and
omodule
On the Solaris 10 OS, the entire framework described in this
c
e
m
d
s
the next module treats every non-global zone as a virtual
node.
That
is,
i
c applications.
u
e
G
non-global zones can host clustered failover and
scalable
@ ent
b
u
k tud
yathe
(
S
The intention of this feature is ton
allow
flexibility
of configuring nons
i
i
d
h
global zones on multiple nodes,
t your application fail over (or
lu andsehaving
a
be load balanced) between
non-global
zones
on different nodes. You
m to u
aapplications
k
might prefer to have
running
in
non-global zones for a
g
e
n
s
a including
variety of reasons,
n the security sandbox that they provide (you
b
e
c
a
i
l
could b
give application
the root password to a non-global
eare onlyadministrators
u
l
k
b
zone,
and
they
super
within
that
zone).
a era
y
l
f
ns it is not strictly the intention of this feature, you could even have
bdu trWhile
a
a
ang non applications fail over between non-global zones on the same physical
ab
node. With the zone integration feature, a data-service developer with

access to only a one-node cluster can experiment with real application
failover between non-global zones on the node.
Some applications, such as NFS, are not supported in non-global zones.

This is typically an application or OS restriction, not a cluster restriction.
Off-the-Shelf Application
For most applications supported in the Sun Cluster software environment,
software customization is not required to enable the application to run
correctly in the cluster. A Sun Cluster 3.2 software agent is provided to
enable the data service to run correctly in the cluster environment.
10-4

Application Requirements
You should identify requirements for all of the data services before you
begin Solaris OS and Sun Cluster software installation. Failure to do so
might result in installation errors that require you to completely reinstall
the Solaris OS and Sun Cluster software.
Determining the Location of the Application Binaries

You can install the application software and application configuration files
on one of the following locations:
The local disks of each cluster node
s
) ha
y
m
om
c
csm Guide
e
t you might
@ enzones,
b
If you are running applications inunon-global
k local tstorage
ud visible to the noninstall the application only inya
the
(
S
global zones.
in this
d
u
e file system using shared storage
al or failover
s
A global file system
m
u
ka e to binaries on a shared file system, you have
g
If you n
put
the application
ntosmaintain and manage. However, you must shut
baone lcopy
e
only
c
a
b downlethe idata service in the entire cluster to upgrade the application
u
k
a esoftware.
ab If you can spare a small amount of downtime for upgrades,
r
y
l
f
bdu trans place a single copy of the application and configuration files on a
a
shared global or failover file system.
ang non
Placing the software and configuration files on the individual cluster
nodes lets you upgrade application software later without shutting
down the service. The disadvantage is that you have several copies
of the software and configuration files to maintain and administer.
ab

10-5
Sun Cluster 3.2 Software Data Service Agents
The word agent is an informal name for a set of components, written

specifically for Sun Cluster 3.2 software, that enable a data service in a
cluster to operate properly. Figure 10-1 shows the relationship between a
standard application and the data service agent.
Data Service
Agent
Standard
Application
start
stop
y
m
om
c
fault
sm uide
monitorsec
G
t
@
n
b
ku tude
a
y
n ( this S
i
d
e Data Service Agent
alu usand
Figure 10-1 Standard m
Application
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
10-6

Reviewing Components of a Data Service Agent
Reviewing Components of a Data Service Agent

Typical components of a data service agent include the following:
Methods to start and stop the service in the cluster
Fault monitors for the service
Methods to start and stop the fault monitoring
Methods to validate configuration of the service in the cluster
A registration information file that allows the Sun Cluster software

to store all the information about the methods into the CCR
You only need to reference a resource type to refer to all the
components of the agent.
s
) ha
ab
y
m
m
ofrom
c
Sun Cluster software provides an API that can be called
shell
e
m
d
s
i
programs, and an API that can be called from C or
programs.
Most of
u
ec C++
G
the program components of data service b
methods
that
are
supported
by
t
@
n
e
u
Sun products are actually compiledaCkand C++
tudprograms.
y
(
S
in this
d
u
al use
Fault Monitor Components
m
ka e to
g
ns
ban licecomponents
Fault a
monitoring
specific to data services in Sun Cluster 3.2
b
e
software
are
run
on
the
local
node
only. This is the same node that is
u
l
k
b
a
a
the data service. Fault monitoring components are intended to
l y running
erapplication
f
u
s
d
detect
failure and can suggest either application restarts or
b tranfailovers in the cluster.
a
ang non
The actual capabilities of these fault monitors are application-specific and
often poorly documented, the general strategy for fault monitors in the
Sun Cluster 3.2 environment is to monitor the health of the following:
The daemons, by placing them under control of the process monitoring

facility (rpc.pmfd). This facility calls action scripts if data service
daemons stop unexpectedly.
The service, by using client commands.
Note Data service fault monitors do not need to monitor the health of
the public network itself because this is already done by the combination
of in.mpathd and pnmd, as described in Module 8, Managing the Public
Network With IPMP.

10-7
Introducing Data Service Packaging, Installation, and Registration
Introducing Data Service Packaging, Installation, and

Registration
Agent packages released along with Sun Cluster are part of the Java ES
medium. You can install them using the Java ES installer at the same time
that you install the Sun Cluster framework, you can invoke the Java ES
installer at a later time to install them, or you can just use the pkgadd
command.
Some agents are supplied with the application software, rather than with
the cluster software.
as
h
If you intend to run certain clustered applications only in Solaris 10 non)
y
m
global zones, you might install the agents only in these non-global
zones.
oinmthe global
Alternatively, you might choose to install the agents always
c
to all
e
m
zone, without the -G option, and have them automatically
propagate
d
s
i
ec t Gu
existing and future non-global zones.
@
b den
u
k
ya Stu
(
n
Data Service Packages u
and
di Resource
his Types
t
l
e
a us
m
a
k encapsulates
to
Each data servicegagent
all the information about the agent
e
n
s
a
n
as a resource
type.
ab lice
b
u this resource
le type is registered with the cluster software, you do not
k
b
a
When
a
r
l yneedstofeknow
u
the location or names of the components of the agent. You
d
n need to reference
b tronly
a
a
the applications resource type to determine all the
ang non correct information about methods and fault monitors for that component.
ab
Note The package that you add by using the pkgadd command to install
an agent, and the corresponding resource type might have different
names. For example, when you install the SUNWschtt package, a resource
type called SUNW.iws becomes available.
10-8

Introducing Resources, Resource Groups, and the Resource Group Manager
Introducing Resources, Resource Groups, and the

Resource Group Manager
Data services are placed under the control of the cluster by configuring
the services as resources within resource groups. The rgmd daemon is the
resource group manager, which controls all activity having to do with
resources and resource groups, as shown in Figure 10-2. That is, the rgmd
daemon controls all data service activity within the cluster.
Resource
Group
Manager
s
) ha
y
m
om
controls
c
sm uide
c
e
G
Resource
t
@
n
b
Group
ku tude
a
y
n ( this Scontains
i
contains
d
lu se
a
am to u
k
Resource
Resource
g nse
n
a
ab lice
b
u
is a
leis a
k
b
a
a
l y sfer
u
d
Resource Type
Resource Type
ab -tran
g
n
n
no
aba
Figure 10-2 Resource Group Manager

10-9
Resources
In the context of clusters, the word resource refers to any element above
the layer of the cluster framework that can be turned on or off, and can be
monitored within the cluster.
An example of a resource would be an instance of a running data service.
For example, an Apache web server with a single httpd.conf file, counts
as one resource.
Each resource is an instance of a specific resource type. For example, each
Apache web server is an instance of SUNW.apache resource type.
s
) ha
y
m
om
c
csm Guide
A particular resource is identified by the following:
e
@ ent
b
u
Its resource type
ak Stud
y
(
A unique name, which is used
in Sun Cluster utilites
in as identification
is
d
h
t
u
e
al are
A set of properties,
which
parameters
that define the
s
m
u
a
o
k a particular
configuration of
t resource
g
e
n
s
a cen
b
a
limultiple resources of the same type in the cluster, either
You can
b configure
e
u
l
k
in
athe same
abor different resource groups.
r
y
l
e
f
s
nexample,
bdu trFor
a
a
you might want to run two failover Apache web server
g on- application services
n
that reside, by default, on different nodes in the
a
n
ab
cluster, but could fail over to the same node if there were only one node
Other types of resources represent IP addresses and data storage that are
required by the application service.
available. In this case, you have two resources, both of type SUNW.apache,
in two different resource groups.
Resource Groups
Resource groups are collections of resources. Resource groups are either
failover or scalable.
10-10

Failover Resource Groups

For failover applications in the cluster, the resource group becomes the unit
of failover. That is, the resource group is the collection of services that
always run together on one node of the cluster at a given time, and
simultaneously fail over or be switched over to another node or nonglobal zone.
Scalable Resource Groups

Scalable resource groups describe the collection of services that run
simultaneously on one or more nodes or zones.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
A scalable application in a scalable resource group also refers to an

application making use of the load-balancing services built into Sun
Cluster software. This is described further in Module 11. Applications
which are in scalable groups but do not make use of the built-in loadbalancing service are referred to as multimaster applications.

10-11
Number of Data Services in a Resource Group
Multiple data services can be configured to run in the same resource

group.
For example, you can run two Sun Java System Web Server application
services, a Sun Java System Directory Server software identity
management service, and a Sybase database as four separate resources in
the same failover resource group. These applications always run on the
same node at the same time, and always simultaneously migrate to
another node. Or you can put all four of the preceding services in separate
resource groups. The services could still run on the same node, but could
fail over and switch over independently.
s
) ha
m
Benefits of Putting All Services in a Single Resourcem
Group
co e
m
s
If you put all data services in the same failover c
resource
group
uid of a
e
G
two-node cluster, the node that is not currently
t the resource group
@ hosting
n
b
e
u
is in a pure backup mode. Some customers
prefer
services this
ud to deployprovides
ak this
tconfiguration
y
(
S
way, in a single resource group,
because
more
n
s
i
i
d
h
predictable behavior. If the
node currently
running all the data services is
t
luthere
e
a
s
performing optimally,
and
is
a
failover,
you can predict that the new
m to u
asame
k
node will have
the
performance,
assuming
equivalent servers.
g
e
n
s
ba licen
a
b
u
le Manager
k
b
Resource
Group
a
a
l y sfer
u
d
ab -tranThe rgmd daemon maintains all the information about data services that
g
an non
are known to the cluster as resources in resource groups. The rgmd
ab
10-12
daemon launches all methods that start resources in resource groups and
performs all failovers.

Describing Failover Resource Groups

The following example is provided to help you understand the general
concept of resources as part of specific resource groups.
A resource name must be globally unique, not merely unique within a
particular resource group.
Figure 10-3 shows a typical failover resource group.
s
) ha
Application Resource
y
m
om
c
csm Guide
e
b@ dent
Data Storage Resource ku
ya Stu
(
n
Resource iType:
d e this
u
SUNW.HAStoragePlus
l
a us
m
a
k e to
g
ns Host Resource
ban liceLogical
a
b
u
le
k
b
a
a
Resource Type:
y fer
l
u
SUNW.LogicalHostname
s
bd tran
a
g onn
a
n
ab
Failover Resource Group
Resource Type:
SUNW.nfs
Figure 10-3 Failover Resource Group

This failover resource group would be defined with a unique name in the
cluster. Configured resources are added to the resource group. By placing
these resources into the same failover resource group, they move together.

10-13
Resources and Resource Types
Each instance of a data service under cluster control is represented by a

resource within a resource group.
Resources exist only inside resource groups. There is no such thing as a
disembodied resource that is not a member of a resource group.
Each resource has a resource type that describes the type of resource it is
(for example, SUNW.nfs for an NFS resource).
At least one defined resource type exists for each supported service in the
cluster. Some applications that are typically considered to be a single
entity, such as an instance of the ORACLE database, actually require two
different resources with different types: the ORACLE server and the
ORACLE listener.
s
) ha
y
m
om
c
csm Guide
e
ta few other
@services,
In addition to resource types that relate tou
data
n
b
e
d These resource
special resource types relate to IP addresses
ak and
tustorage.
y
(
S
types are described in Using Special
in Resource
is Types on page 10-15.
d
h
t
u
al use
m
ka e to
g
Resource TypenVersioning
ba licens
a
b 3.2lesoftware gives you the ability to use different versions of
u
Sun
Cluster
k
a etypes.
ab For example, old and new versions of data service agents
r
y
l
resource
f
s
nco-exist
bdu trcan
as separate types. Individual resources of an original resource
a
a
g
an non type can be upgraded to a new type on a resource-by-resource basis.
ab
Officially, the version number is appended to the resource type name. For
example, the official name of the NFS resource type is SUNW.nfs:3.2.
When using a resource type name, the version suffix can be dropped if
there is no ambiguity. For that matter, the vendor prefix can also be
dropped. For example, when you initially install Sun Cluster 3.2 software,
all of the following names can be used to refer to the nfs resource type:
10-14
SUNW.nfs:3.2
SUNW.nfs
nfs:3.2
nfs

Using Special Resource Types

The following sections describe special resource types that you include in
resource groups to complete the configuration.
The SUNW.LogicalHostname Resource Type

Resources of type SUNW.LogicalHostname represent one or more IP
addresses on a particular subnet that will become the logical IP addresses
used to access the services in that resource group.
ab
as
h
That is, each IP address described by a SUNW.LogicalHostname resource
)
y If the
migrates from node to node, along with the services for that group.
m
group is mastered by a non-global zone rather than a physical

the
om node,
c
IP address is automatically configured in the appropriate

m zone.
ideThe clients
cscluster.
u
use these IP addresses to access the services inethe
G
t
@
n
b
de transparent to the
ku relatively
u
a
t
A large part of what makes cluster
failover
y
S services of the group. The
n ( along
s
i
i
client is that IP addresses migrate
with
d
h
t
lu logical
e
clients always use the a
same
IP address to contact a particular
s
m tregardless
u
aservice,
instance of a data
of which physical node or zone is
o
k
g
e
actually running
service.
ban licthe
ens
a
b
u
le
k
b
a
a
The
Resource Type
l y SUNW.SharedAddress
er
f
u
s
d
ab -tran
g
Resources of type SUNW.SharedAddress represent a special type of IP
an non
address that is required by scalable services. This IP address is configured
on the public net on only one node or zone with failover capability, but
provides a load-balanced IP address that supports scalable applications
that run on multiple nodes or zones simultaneously. This subject is
described in greater detail in Module 11.

10-15
The SUNW.HAStoragePlus Resource Type
The SUNW.HAStoragePlus resource type manages the following types of

storage:
Global raw devices
Global file systems
Traditional (non-ZFS) failover file systems
ZFS zpools, including all the datasets within
Global Raw Devices and Global File Systems
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
k tud
In fact, managing global storage withyaaSUNW.HAStoragePlus
resource
(
S
n
is optional, but serves the following
di euseful
hispurposes:
t
u
l
The START methodmofaSUNW.HAStoragePlus
is used to check if the
us
a
o
k
t
global devices
g or nfilesesystems in question are accessible from the
n
a
node where
b the
eresource group is going online.
c
a
i
l
b
u
Usually,
you
le use the Resource_dependencies standard property to
k
b
a
a
l y sconfigure
er a dependency so that the application resources depend on
f
u
d
SUNW.HAStoragePlus resource. In this way, the resource group
ab -tran the
g
manager
does not try to start services if the storage the services
n
n
o
a
n
depend on is not available.
ab
Global raw devices and global file system management, which include
failing over from a failed node, are part of the Sun Cluster software
framework. So it might seem redundant to provide a resource type that
also manages these global devices and file systems.
The SUNW.HAStoragePlus resource type can also be used to attempt

co-location of resource groups and device groups onto the same
node, thus enhancing the performance of disk-intensive data
services.
Failover File Systems

You must configure a SUNW.HAStoragePlus to manage a failover file
system. It is the resource methods that provide the failover of the file
system.
A failover file system must fail over together as the service fails over, but
can fail over only to nodes physically connected to the storage.
10-16


The configuration of a global or failover file system looks very similar.
Both use the FilesystemMountPoints property to identify the file system
or file systems in question. The SUNW.HAStoragePlus methods
distinguish between global and failover file systems using the
/etc/vfstab entry.
Guidelines for Using Global and Failover File Systems

These are some general guidelines about how to choose between global
and failover file systems.
When to Use a Global File System
s
) ha
y
m
Use a global file system if you want to support any of the

following:
om
c
e
m
A scalable service
d
s
i
c Gu
e
t physically
@
A failover service that must fail over
to a node
not
n
b
e
u
connected to the storage
ak Stud
y
(
s for different failover services
incontains
A single file system that
d
hidata
t
u
l
e
a on udifferent
that may be running
nodes
s
m
a
o
k dataefrom
t a node that is not currently running the
Access to
the
g
n
s
service
ba licen
a
b
u
le
k
b
a
a
If
your
storage
is physically connected to all possible nodes for the
y fer
l
u
s
group containing the HAStoragePlus resource, you can migrate
bd tranresource
a
the
underlying
ownership of the storage along with the service, as a
ang non
performance benefit.
ab
When to Use a Failover File System

Use a failover file system if all of the following are true:
The file system is for a failover service only.
The Nodelist property for the resource group contains only nodes
that are physically connected to the storage; that is, if the Nodelist
for the resource groups is the same as the Nodelist for the device
group.
Only services in a single resource group are using the file system.
If these conditions are true, a failover file system provides a higher level
of performance than a global file system, especially for services that are
file system intensive.

10-17
HAStoragePlus Resources in Zones

When a traditional (non-ZFS) file system that is controlled by an
HAStoragePlus resource is contained in a resource group that is mastered
by a non-global zone, the file system is always mounted in the
appropriate global zone, and then made available to the non-global zone
through a loopback mount.
HAStoragePlus and ZFS

The same HAStoragePlus resource that provides failover of traditional
filesystems can also provide ZFS failover. For ZFS, an HAStoragePlus
instance represents one or more zpools and all the file systems within.
s
) ha
y
m
m have to
oyou
To manage failover of ZFS filesystems within the cluster, all
c
e
m
d
s
do is configure an HAStoragePlus instance with the
value
of
the
Zpools
i
c over.GYou
u do not
efail
property indicating one or more pools that should
t
@
n All of the ZFS
bwhatsoever.
e
u
need to configure any /etc/vfstab entries
d
k
tuconfiguration database.
yathe zpool
mount information is self-contained(in
S
n
di e this
u
l
a us
m
a
Note A single HAStoragePlus
k e to instance can refer to multiple traditional
g
(non-ZFS) filesystems,
an cenorsmultiple ZFS zpools, but not both. When you
b
a
use thebZpools property
the values of the properties for traditional
li
e
u
l
k
filesystems
and AffinityOn) are ignored.
b
a era(FilesystemMountPoints
y
l
f
bdu trans
a
- ZFS technology is zone-aware, and ZFS zpools that are configured into a
ang non non-global zone via HAStoragePlus are only exposed in that zone. They
ab
are not visible in the global zone.
Generic Data Service

The Generic Data Service (GDS) is a resource type for making simple
applications highly available or scalable using the resource type
SUNW.gds by plugging them into the Sun Cluster RGM framework. The
GDS composes a fully functional Sun Cluster Resource Type complete
with callback methods and a Resource Type Registration file.
10-18


Basically, the concept is that many different applications can share the
same resource type. All you have to do is have a resource type where what
is being launched is not implied by the resource type itself. Instead, what is
being launched, how to probe this application, and how to stop this application
are just properties that can be set differently for each instance of
SUNW.gds.
Many Sun-supported data services (for example, DHCP, Samba, and
many more) do not actually supply new resource types. Rather, they
supply configuration scripts that configure resources to represent the
applications as instances of SUNW.gds.
s
) ha
In the lab exercises for this module, you will get an opportunity to use an
instance of the SUNW.gds type to put your own customized application
under control of the Sun Cluster software.
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

10-19
Understanding Resource Dependencies and Resource Group Dependencies
Understanding Resource Dependencies and Resource

Group Dependencies
You can declare dependency relationships between resources.

Relationships between individual resources can be between resources in
the same or different resource groups. You can also configure group
dependencies that take into account only the state of a group rather than
the state of its resources.
There are four levels of resource dependencies.
Regular Resource Dependencies
s
) ha
y
m
If Resource A depends on resource B, then there is one very hard

omand fast
c
rule (known henceforth as the prime directive of regular dependencies):
sm uide
c
e
Resource A cannot be brought online unless resource
GB is already
t
@
n
b
online.
ku tude
a
Note this has some implications:
y
n ( this S
i
d
The cluster must serialize
lu sestarting A and B (waiting to start A
a
until B is online)
u where otherwise they could be started
am intcases
o
k
in parallel.
ng ense
a
b
aThe cluster
lic must serialize stopping A and B (waiting to stop B
b
e
u
l
k runtil
b A is stopped) in cases where otherwise they could be
a
a
y
ul nsfe stopped in parallel.
d
b
a
a
trYou
g
n
are allowed to specifically, manually, disable resource B, even if
n
o
a
n
b
resource A is still running. This is somewhat counterintuitive, but because
a
you are explicitly issuing the command (clrs disable) to disable the
dependee, you can do it. Note the prime directive (the first bullet item
above), does not explicitly forbid you from stopping the resources in the
wrong order.
Weak Dependencies (Weaker Than Regular Dependencies)

If resource A has a weak dependency on resource B, rgmd preserves the order
of the stops and starts as if it were a regular dependency, but does not
enforce the dependency in other ways. If B fails to start, the rgmd will still
try to start A. You can manually disable either resource or its group freely.
10-20

Understanding Resource Dependencies and Resource Group Dependencies
Restart Dependencies (Stronger Than Regular Dependencies)

Restart dependencies have all the attributes of regular dependencies, with
the additional attribute that if the RGM is told to restart resource B, or is
informed that some agent did the restart of resource B itself, the RGM will
automatically restart resource A.
Similar to the case of regular dependencies, you are allowed to manually,
specifically disable the dependee, resource B, while leaving resource A
running. However, when you manually reenable resource B, the restart
dependency begins, and the RGM restarts resource A
Offline-Restart Dependencies (Slight Variation of Restart

Dependencies)
y)
has
ab
m
m
o
cdependee,
In this slight variation, when RGM is informed thatm
the
de issues a
s theuiRGM
coffline,
resource B, is offline, or when it is put explicitly
e
G
t
@
restart on resource A. Resource A explicitly
depends
on resource B;
n
b
deB actually restarts.
ku until
therefore, resource As start will be
blocked
u
a
t
y
n ( this S
i
d
lu se
a
m to u
Resource Group
aDependencies
k
ng ense
a
b
a grouplicdependencies imply an ordering relationship between
Resource
b
u
leIf resource group G has a group dependency on resource
two groups:
k
b
a
a
l y group
erH, group G cannot be brought online unless group H is online.
f
u
s
d
H cannot be brought offline unless group G is offline. This type of
ab -tranGroup
g
dependency considers the state of the group only rather than the
an non
resources inside it.
Note Resource group dependencies are somewhat limited. They are

enforced between groups running on different nodes when you manually
try to start or stop resource groups in the wrong dependency order,
regardless of which nodes the groups are mastered.
They consider only the group state and not individual resource state;
therefore, resource group dependencies are made obsolete by the various
flavors of resource dependencies, now that they can be between resources
in different groups.

10-21
Configuring Resource and Resource Groups Through Properties
Configuring Resource and Resource Groups Through

Properties
You configure specific resources within the cluster by defining values for
resource properties and resource group properties. Properties consist of a
set of name=value pairs.
These properties are used by the data service agent and by rgmd. There is
no way they can be accessed directly by the application software itself,
because it is cluster-unaware software.
s
) ha
Some properties are essential for running the service in the cluster. Scripts
specifically used to run the service in the cluster can read these property
values and use them to locate configuration files, or can use them to pass
command-line parameters to the actual services.
y
m
om
c
sm uide
c
e
G
Other properties are essential only for data service
fault tmonitoring.
@
n
b
Misconfiguring these properties might leave
de running fine, but
ku thetuservice
a
y
cause fault monitoring to fail.
n ( this S
i
d
lu dozens
e of properties. Fortunately, many
a
s
Each resource can literally
have
m
u
kafor particular
to types of resources are automatically
important properties
g
e
provided with
anreasonable
nsdefault values. Therefore, most administrators
b
e
c
a
li environments never need to deal with the
of SunbCluster software
e
u
l
k
properties.
a erab
y
l
u nsf
d
b
a
tra Resource Properties
g
n
n
Standard
no
aba
The names of standard resource properties can be used with any type of
resource.
You can access a full list of standard resource properties and their general
meanings by typing:
# man r_properties
Of the dozens of properties listed, only a handful are critical for a
particular resource type. Other properties can be ignored, or can have
default values that are unlikely to be changed.
10-22

Some Significant Standard Resource Properties
The following properties are standard resource properties; that is, they
can have meaningful values for many different types of resources.
The Resource_dependencies Property

Resource dependencies, which imply ordering, are configured using
standard properties.
Resource_dependencies=nfs-stor
The Resource_dependencies_weak Property
s
) ha
ab
y
m
m
To set up a weak dependency, so that resource A has a weak
odependency
c
e
on resource B, set the Resource_dependencies_weak
propertyidon
m
s
u
ec nottaGreal
resource A. (Recall that this implies an ordering, but
@
b den
dependency.)
u
k
ya Stu
(
Resource_dependencies_weak=resB
n
di e this
u
l
a us
m
a
The Resource_dependencies_restart
and
k e to
g
Resource_dependencies_offline_restart
Properties
ban licens
a
bthe twolkinds
e of restart dependencies, set the
u
To k
set
b
a
a
or
l yResource_dependencies_restart
er
f
u
s
d
property on resource A.
an
ab -trResource_dependencies_offline_restart
g
Resource_dependencies_restart=resB
an non
Resource_dependencies_offline_restart=resB

10-23
The Failover_mode Property

The Failover_mode property describes what should happen to a resource
if the resource fails to start up or shut down properly. Table 10-1 describes
how values of the Failover_mode property work.
Table 10-1 The Failover_mode Value Operation
Value of the
Failover_mode
Property
Failure to Start
Failure to
Stop
Can Fault
Monitor
Cause
RGM to
Fail RG
Over?
Can Fault
Monitor Cause
RGM to Restart
the Resource?
Yes
Yes
No
No
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak YesStud Yes
SOFT
The whole resource The
y
(
group is switched
STOP_FAILED
in this
d
u
l is setson
to another node.
e
aflag
m
u
ka ethetoresource.
g
anresource
HARD
The whole
Yes
Yes
ns The node
b
e
c
a
i
l
group
b is switched
enode. reboots.
u
l
k
b
to
another
ya fera
l
u
RESTART_ONLY
No
Yes
bd transOther resources in The
a
the
same
resource
STOP_FAILED
g
n non
a
group can still start flag is set on
b
a
NONE
LOG_ONLY
Other resources in
the same resource
group can still start
(if nondependent).
The
STOP_FAILED
flag is set on
the resource.
(if nondependent).
the resource.
Other resources in
the same resource
group can still start
(if non-dependent).
The
STOP_FAILED
flag is set on
the resource.
If the STOP_FAILED flag is set, it must be manually cleared by using the

clrs clear command before the service can start again.
# clrs clear -n node -f STOP_FAILED resourcename
10-24

Extension Properties
Names of extension properties are specific to resources of a particular
type. You can get information about extension properties from the man
page for a specific resource type. For example:
# man SUNW.apache
# man SUNW.HAStoragePlus
For example, if you are setting up the Apache web server, you must create
a value for the Bin_dir property, which points to the directory containing
the apachectl script that you want to use to start the web server.
The HAStoragePlus type has the following important extension
properties regarding filesystems.
s
) ha
ab
y
m
m
o
c
FilesystemMountPoints=list_of_storage_mount_points
e
m
d
s
i
AffinityOn=True/False
ec t Gu
@
-------------------b den
u
k
Zpools=list_of_zpools
ya Stu
(
n
di properties
his to identify which storage
t
u
l
Use the first of these extension
e
a The
m
ussecond extension property is a parameter
resource is being described.
a
o
k
t
g nframework
that tells the
cluster
to switch physical control of the storage
se the service.
annode
bthe
e
groupato
running
Switching control to the node that
c
i
l
b
isurunning lthe
e service optimizes performance when services in a single
k
b
a
a
failover
resource
group are the only services accessing the storage. The
y fer
l
u
s
property (Zpools) is used only for specifying pool names for a
bd tranthird
a
failover
ZFS resource, in which case the first two properties are not used.
ang non
Many resource types (including LDAP and DNS) have an extension
property called Confdir_list that points to the configuration.
Confdir_list=/global/dnsconflocation
Many have other ways of identifying their configuration and data. There
is no hard-and-fast rule about extension properties.
Resource Group Properties

Certain properties apply to an entire resource group. You can get
information about these properties by typing:
# man rg_properties

10-25
Some Significant Resource Group Properties
The following properties are associated with a resource group, rather than
an individual resource.
The RG_dependencies Property

The RG_dependencies property will be set on a resource group to
describe its dependency on another group. If resource group A has the
property RG_dependencies=rgB, then resource group A cannot be
brought online unless resource group B is online. Resource group B must
be brought online somewhere, but not necessarily on the same node.
s
) ha
y
m
om
c
ide or
The Nodelist property for a resource group describes
what u
nodes
csm
e
G
zones the resource group can run on.
t
@
n
b
ku tude
a
y
(an ordered
S list, from most preferred
The value of this property is always
n
s
i
i
d
h
t zone acts as a virtual node
node to least preferred node.
luA non-global
e
a
s
from the point of view of
a
resource
group
node list. As mentioned before,
m to u
a
k
while it is not strictly
intention
of the cluster zone integration feature,
g the
e
n
s
a
n
it is possible
for
the
same
groupss
node list to include multiple zones on
b lice
a
the same
b physical
e node.
u
l
k
b
ya fera
l
u
nsFailback Property
bd trThe
a
a
ang non
The Nodelist Property
ab
If the Failback property is TRUE (not the default), the resource group
automatically switches back when a preferred node or zone (earlier in the
node list) joins the cluster.
The Implicit_network_dependencies Property

The Implicit_network_dependencies property is TRUE by default. This
property makes all the services in the resource group dependent on all the
LogicalHostname and SharedAddress resources. In other words, no
services can start if the logical IP addresses cannot be brought online. You
can set the Implicit_Network_Dependencies property to FALSE if you
have a service that does not depend on logical IP addresses.
10-26

The Pingpong_interval Property
This property controls resource group behavior in a couple of different

ways:
If a resource group fails to start twice on the same particular node or

zone (failure of START methods of same or different resources in the
group) within the interval (expressed in seconds), then the RGM
does not consider that node or zone a candidate for the group
failover.
If the fault monitor for one particular resource requests that a group be
failed off a particular node or zone, and then the fault monitor for
the same resource requests another failover that would bring the
group back to the original node or zone, the RGM rejects the second
failover if it is within the interval.
s
) ha
y
m
om
c
idefaulty start
csm
u
e
Note The Pingpong_interval property is meant
to prohibit
G
t
@
n
b
scripts or properties and faulty fault monitors,
or
problem
applications,
u ude
k
a
t
from causing endless ping-ponging
(y between
S nodes or zones.
n
s
i
i
lud se th
a
u
am
o
The Pathprefix
Property
k
t
ng ense
a
b
a
ThebPathprefix
licproperty points to a directory in a shared-storage file
e
u
l
ksystemrathat
b is used for the administrative purposes of services in the
a
y
fe group.
ul nresource
s
d
b
a
tra Currently, the only data service that must have the Pathprefix property
g
n
n
no
set is NFS, which uses the property to find its dfstab file. NFS needs the
aba
dfstab file so that it knows what to share. NFS also uses the same arena
to store file lock state information. File lock state information is typically
stored in the /etc directory on a stand-alone NFS server, but it must be in
the shared storage in the cluster.
An NFS resource must have its dfstab file named:
value_of_Pathprefix_for_RG/SUNW.nfs/dfstab.resource_name
When you create an NFS resource, the resource checks that the
Pathprefix property is set on its resource group, and that the dfstab file
exists.

10-27
Specifying Non-Global Zone Names in Place of Node Names
Specifying Non-Global Zone Names in Place of Node

Names
You may specify non-global zone names in the place of node names in the
following contexts:
When specifying or modifying the node list for a resource type
When specifying or modifying the node list for a resource group
When switching a resource group to a particular node
When enabling or disabling a resource or resource flag on a

particular node
s
) ha
y
m
om
c
idae
The form nodename:zonename is used to specify
a zoneuon
csm
e
G
t
@
particular node.
n
b
desame or different
kunodes
u
a
t
You can easily give zones on different
the
y
S
n ( tothbeisdifferent
i
zone names. They are considered
virtual nodes by the
d
u
l
e
cluster either way. a
m to us
a
k
The syntax g
-n nodename
e -z zonename is identical to
n
s
a
n
-n nodename:zonename.
ab lice
b
uThe syntax
le-n node1,node2,... -z zonename
k
b
a
a
l y sisfea rshorthand for -n node1:zonename,node2:zonename,... In
u
d
ab -tran other words, it is a shorthand you when specifying multiple zones
g
n non
on different nodes that have the same zone name.
a
b
a
The commands to perform these operations are presented in the next few
sections, but there are some general rules in using zone names:
10-28

Using the clresourcetype (clrt) Command

You use clresourcetype (clrt) to register or unregister a resource type.
Registering a resource type makes it known to the CCR and available to
be used for new resources.
If you register a type and do not specify a node list, you are allowing that
type to be used on all nodes and non-global zones, even ones that are
added and created in the future. In other words, the default is to place no
restriction on where resources of this registered type could run. The
assumption is that you will have the data service agent installed on all
nodes.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
Registering Resource
Types
al use
m
ka e to
g
The following
s show registration of resource types. The first
an examples
nnode
bspecifies
e
example
a
list of all current and future nodes. The second
c
a
li
b
e
u
l
kexample
blimits the node list to only the nodes mentioned.
a
a
r
y
fe register SUNW.nfs
ul n#sclrt
d
b
a
tra # clrt register -n vincent,theo SUNW.apache
g
n
n
no
aba
If you do specify a node list, you are restricting the resource type to certain
nodes or zones. The cluster then allows you to specify only those nodes or
zones for the resource groups that contain resources of that type. The only
property of a registered resource type that you can modify with the clrt
command is the node list.
Yo
Note You can specify a zone with the syntax nodename:zonename.

10-29
Viewing Registered Resource Types

clrt list -v shows the list of registered resource types and their node
lists. clrt show can display more about a specific registered type or all
registered types:
vincent:/# clrt list -v
Resource Type
------------SUNW.LogicalHostname:3
SUNW.SharedAddress:2
SUNW.HAStoragePlus:8
SUNW.nfs:3.2
SUNW.apache:4.1
Node List
--------<All>
<All>
<All>
<All>
vincent theo
s
) ha
y
m
vincent:/# clrt show apache

om
c
csm Guide
Registered Resource Types ===
e
@ ent
b
u
Resource Type:
SUNW.apache:4.1
ak Stud
y
(
RT_description:
s Server on Sun Cluster
inApache
iWeb
d
h
t
u
RT_version:
e
al u4.1
s
m
API_version:
2
ka e to /opt/SUNWscapc/bin
g
RT_basedir:
ban licens
Single_instance:
False
a
b
Proxy:
False
u
le
k
b
a
a
Init_nodes:
All potential masters
y fer
l
u
s
Installed_nodes:
vincent theo
bd tran
a
Failover:
False
g onn
a
SUNWscapc
n
ab Pkglist:
RT_system:
Global_zone:
False
False
Note The Failover (False) means that the resource type is not limited
to being failover only (that is, it can be failover or scalable). Global_zone
(False) means that methods for the resource type will run in a nonglobal zone when the type is configured in a non-global zone.
Unregistering Types
You can unregister only types for which there are no remaining resources:
# clrt unregister nfs
10-30

Configuring Resource Groups Using the clresourcegroup (clrg) Command
Configuring Resource Groups Using the

clresourcegroup (clrg) Command
You must create a resource group before creating resources in it. In
addition, you can only delete a resource group that contains no resources.
Use the following command syntax:
clrg create [-n nodelist] [-p property [...]] RG_name
clrg delete RG_name
clrg set [-p property [...]] RG_name
For example, the following command creates a resource group nfs-rg
that runs on two physical nodes (not on non-global zones). The
Pathprefix property is set, in anticipation of placing an NFS resource in
the group:
s
) ha
y
m
om
c
# clrg create -n vincent,theo -p Pathprefix=/global/nfs/admin

de
csm Guinfs-rg
e
t
@Both especify
n
b
u
The following two commands are identical.
a new resource
k tud
a
y
group that specifies two non-global
zones
on
two
different
nodes as the
( is S
n
i
potential masters. You can
duse theesecond
th syntax because the non-global
lusame
a
zones happen to have
the
name:
am to us
k
# clrg create -n vincent:frozone,theo:frozone
ng en-zsefrozone ora-rgora-rg
a
b
# clrg create -n vincent,theo
a
lic
b
e
u
l
k rab
a
y
fe If you omit the node list when creating a group, it defaults to all
ul nNote
s
d
b
a
tra physical nodes of the cluster, and none of the non-global zones.
g
n
n
a
no
ab
Displaying Group Configuration Information

You can display resource group information with clrg list or clrg
show. Some examples follow.
# clrg show nfs-rg
Resource Groups and Resources ===
Resource Group:
RG_description:
RG_mode:
RG_state:
Failback:
nfs-rg
<NULL>
Failover
Unmanaged
False

10-31
Configuring Resource Groups Using the clresourcegroup (clrg) Command

Nodelist:
# clrg show ora-rg
vincent theo

Resource Group:
RG_description:
RG_mode:
RG_state:
Failback:
Nodelist:
ora-rg
<NULL>
Failover
Unmanaged
False
vincent:frozone theo:frozone
# clrg show -p pathprefix nfs-rg
s
) ha
y
m
om
c
Resource Group:
nfs-rg
Pathprefix:
/global/nfs/admin
csm Guide
e
@ ent
b
u
# clrg show -v nfs-rg
ak Stud
y
(
in this
d
u
al use
m
ka e to
Resource Group:
nfs-rg
g
n
s
a
n
RG_description:
<NULL>
ab lice
RG_mode:
Failover
b
e
u
l
k
b
RG_state: ya
Unmanaged
ra
l
e
f
u
RG_project_name:
default
s
bd tran
a
RG_affinities:
<NULL>
g
n
n
RG_SLM_type:
manual
no
aba Auto_start_on_new_cluster:
True
Failback:
Nodelist:
Maximum_primaries:
Desired_primaries:
RG_dependencies:
Implicit_network_dependencies:
Global_resources_used:
Pingpong_interval:
Pathprefix:
RG_System:
Suspend_automatic_recovery:
10-32
False
vincent theo
1
1
<NULL>
True
<All>
3600
/global/nfs/admin
False
False

Configuring a LogicalHostname or a SharedAddress Resource
Configuring a LogicalHostname or a SharedAddress

Resource
The commands clreslogicalhostname (clrslh) and
clressharedaddress (clrssa) are special commands for creating and
modifying LogicalHostname and SharedAddress resources, respectively.
Specifically, these commands deal with two special properties of these IP
resources and can simplify the administration of these properties:
Hostnamelist: List of IPs (on a single subnet) corresponding to this

resource
as
h
Multiple logical IP resources on different subnets must be separate
)
y
resources, although they can still be in the same resourcem
group.
om
c
Each name in Hostnamelist can be an IPV4 address,

ane
IPV6
m
d
s
i
address, or both.
ec t Gu
@
ncommands assume
b these
If you do not specify the Hostnamelist,
e
u
d
k
tuof Hostnamelist.
that the resource name is also
yathe value
(
S
n
is to use on each node, in the
dofi which
hadapters
NetIfList: Indication
t
u
l
e
a us
format: ipmp_grp@node_id,ipmp_grp@node_id,..
m
a
k e toNetIfList, the commands try to figure out if
g
If you n
do
not specify
nsIPMP group per node on the subnet indicated by
bais only
e
there
one
c
a
li If so, the value of NetIfList is automatically
b Hostnamelist.
e
u
l
k
a ederived.
ab When you use the -N shorthand of the clrslh command
r
y
l
f
bdu trans you can use the node names, but the actual value of the property will
a
contain the node ids.
ang non
ab
Examples of Using clrslh to Add a LogicalHostname

In the first example, all the defaults are used. The HostnameList value is
the same as the name of the resource.
# clrslh create -g nfs-rg orangecat-nfs
The second example provides values for the two special properties. The
command has shorthand options (-h and -N) for supplying the values:
# clrslh create -g nfs-rg -h orangecat-nfs,myother-ip \
-N therapy@vincent,therapy@theo orangecat-nfs

10-33
Configuring Other Resources Using the clresource (clrs) Command
Configuring Other Resources Using the clresource

(clrs) Command
Use the following command syntax to create, delete, and change
properties of resources.
clrs create -t resource_type_name -g RG_name [-p property []] res-name
clrs set [-p property []] res-name
clrs delete res-name
For example, the following adds an HAStoragePlus and an nfs resource:
# clrs create -t HAStoragePlus -g nfs-rg -p AffinityOn=true \
-p FilesystemMountPoints=/global/nfs nfs-stor
s
) ha
y
m
omnfs-res
# clrs create -t nfs -g nfs-rg -p Resource_dependencies=nfs-stor
c
e
m
d
s
i
ec t Gu
@
b den
u
k
tu
Displaying Resource Configuration
Information
ya S
(
n
di e this
u
l
a clrsusshow to display the configuration of
You can use clrs list
and
m
a
k theeIPtoresources):
any resource (including
g
ban licens
# clrs list -v
a
b
le Type
Resource Name ku Resource
Resource Group
b
a
a
r
y
-------------------------------------l
e
f
u
s
d
nfs-stor
nfs-rg
ab -tran SUNW.HAStoragePlus:8
g
orangecat-nfs
SUNW.LogicalHostname:3
nfs-rg
n
n
o
a
n
SUNW.nfs:3.2
nfs-rg
abnfs-res
# clrs show nfs-res
Resources ===
Resource:
Type:
Type_version:
Group:
R_description:
Resource_project_name:
Enabled{vincent}:
Enabled{theo}:
Monitored{vincent}:
Monitored{theo}:
10-34
nfs-res
SUNW.nfs:3.2
3.2
nfs-rg
default
True
True
True
True

Configuring Other Resources Using the clresource (clrs) Command

# clrs show
-p resource_dependencies nfs-res
Resources ===
Resource:
Resource_dependencies:
nfs-res
nfs-stor
--- Standard and extension properties ---
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

10-35
Complete Resource Group Example for NFS

The following example is based on the assumption that the NFS agent has
been added. You should also assume that the dfstab file has been
configured in the subdirectory SUNW.nfs of the directory referenced by
the groups Pathprefix property.
1.
Add (register) the resource types:

# clrt register SUNW.nfs
# clrt register SUNW.HAStoragePlus
2.
Create the resource group:

# clrg create -n vincent,theo \
-p Pathprefix=/global/nfs/admin nfs-rg
s
) ha
y
m
om
c
# clrslh create -g nfs-rg orangecat-nfs m

s
ide
c
u
e
G
4. Add the SUNW.HAStoragePlus resource: @
t
n
b
# clrs create -t HAStoragePlus
de \
ku-g nfs-rg
u
a
t
y
S
-p AffinityOn=true \ n (
s
i
i
d
h
-p FilesystemMountPoints=/global/nfs
nfs-stor
lu se t
a
u many services have properties that
5. Add the NFS service.
am Although
o
k
t
point to configuration
ng ensefiles or binaries, NFS does not because it uses
a
b
the Pathprefix
of the resource group. Dependency on the
a
lic property
b
e
u
storage
is
expressed
by
using
the standard property. Dependency on
l
k
b
a
a
r
resource is implied.
l y sthe
eLogicalHostname
f
u
d
ab -tran # clrs create -t nfs -g nfs-rg \
g
-p Resource_dependencies=nfs-stor nfs-res
an non
ab
3.
Add the logical host name resource:
6.
Bring the resource group online; these state manipulations are

covered in the next section:
# clrg online -M nfs-rg
10-36

Modifying Properties With clrs set -p ...

Sun Cluster 3.2 software allows the developer of a resource type to place
restrictions on when particular properties can be set or changed. Each
property for a particular type has a tunability characteristic that can be one
of the following:
at_creation You can only set the property as you do the clrs
create command to add the resource.
when_disabled You can change with the clrs set command if

the resource is disabled.
anytime You can change anytime with the clrs set command.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

10-37
Controlling the State of Resources and Resource Groups

The following examples demonstrate how to use the clrg and clrs
commands to control the state of resource groups, resources, and data
service fault monitors.
Introduction to Resource Group State

When a resource group is first created it is in an unmanaged state. Once it
is brought into a managed state, the cluster provides automatic control and
failover for its resources. The managed/unmanaged state is a persistent
state that survives cluster reboots.
s
) ha
y
m
A resource group that is mangaged but offline can be thought

oofmas
c
econtrol
currently not online on any nodes, but still subject to
automatic
sm
iitdonline
u
and recovery. That is, the cluster will still manageeitcand bring
if
G
t
@
n
b
the entire cluster reboots, or if the cluster reconfigures
u ude because any node
k
a
fails or joins.
t
y
(
S
n
di e this
u
l
a us
m
a
Introduction to Resource
k e toState
g
an cens
b
a
li have a persistent disabled/enabled state flag that
Individual
b resources
e
u
l
k
survives
of its groups state, and survives cluster reboots. A
a etransitions
abis disabled
r
y
l
resource
that
when its group is unmanaged or offline will still
f
u ns
d
b
a disabled (and will not run) when the group goes online. If you change
a
trbe
g
n
the
enabled/disabled state of a resource while the group is online, it turns
n
a
no
ab
on and off the resource. If you do so while the group is not online, it
affects what will happen to the resource when the group goes online.
If a resource is enabled, then another similar state flag can disable and
enable just the fault monitoring for the resource.
When a resource is created using clrs create or clrslh create or
clrssa create it is automatically put into the enabled state, regardless
of the state of its resource group.
10-38

Resource Group Operations
Use these commands to do the following:
Online an offline resource group onto its preferred node or a specific

node. With -M, it will bring the group into the managed state first if
it is unmanaged. Without -M, the group must already be in the
managed state. With -e, all disabled resources are enabled. Without
-e, the enabled/disabled state of each resource is preserved.
# clrg online [-M] [-e] nfs-rg
# clrg online [-M] [-e] -n node nfs-rg
Switch a failover resource group to a specific node. (It might be

online already on another node, or offline.) With -M, it will bring the
group into the managed state first if it is unmanaged. Without -M,
the group must already be in the managed state. With
-e, all disabled resources are enabled. Without -e, the
enabled/disabled state of each resource is preserved.
s
) ha
y
m
om
c
csm Guide
e
t
n
b@nfs-rg
# clrg switch [-M] [-e] -nknode
e
u
d
a The
tugroup
y
(
S
Remaster a failover resource
group.
switches to its
n this
i
d
preferred node, even
luif it issealready running on another node.
a
u
# clrg remaster
am nfs-rg
o
k
t
nagresource
segroup. The group remains in the managed state, so
Offline
a
n
b
e
itais still subject
lic to automatic recovery:
b
e
u
l
k #raclrg
b offline nfs-rg
a
y
l
e
f
bdu trans Restart a resource group:
a
g onn
# clrg restart nfs-rg
a
n
b
a
Resource Operations
Disable a resource and its fault monitor:

# clrs disable nfs-res
Enable a resource and its fault monitor:

# clrs enable nfs-res
Clear the STOP_FAILED flag:

# clrs clear -n node -f STOP_FAILED resourcename

10-39
Fault Monitor Operations

Disable the fault monitor for a resource:

# clrs unmonitor nfs-res
Enable the fault monitor for a resource:

# clrs monitor nfs-res
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
10-40

Summary of Resource Group and Resource

Transitions
The diagram in Figure 10-4 summarizes the resource and resource group
transitions. Note the following:
The -M switch for clrg switch or clrg online manages a group

and then brings it online.
By default, all transitions preserve the state of the enabled/disabled

flags (for the service and its fault monitoring). You can force all
resources to enabled by adding a -e option to clrg
switch/online.
s
) ha
y
m
m
Group online
o
c
resources running
e
m
d
s
if
enabled
i
res1: disabled/enabled
ec t Gu
@
b den
u
k
tu clrg switch -n node rg
ya Sres
(
clrs
disable/enable
n
di e this
u
l
ma to us clrg online -n node rg
clrg offlinekrg
a
(enabled/disabled
clrg online rg (brings it on preferred node)
g nse
n
state
preserved)
(enabled/disabled state of resource preserved)
a
b lice
a
b
u
le
k
Group: Offline
b
a
a
r
y
No resources running
l
e
f
u
s
d
ab -tran
Resources may be
g
an non
enabled or disabled
ab
clrg unmanage rg
(must disable each
resource first)
clrg online -M rg
clrg online -M
-n node rg
clrs disable/enable res (affects whether they

will run when group
switched on)
clrg manage rg
Group: Unmanaged
clrs disable/enable res
Figure 10-4 Resource Group and Resource Transitions

10-41
Suspended Resource Groups
Suspended Resource Groups

When you suspend a resource group, you disable any automatic recovery,
failover, or restarting of the resource group or the resources within. That
is:
You can still transition the resource group any way you like using
the commands presented on the previous pages.
You can still enable/disable any individual resources, using the

commands presented on the previous pages. If the resource group is
online, the resources will go on and off accordingly.
The fault monitors for resources will still be running.
s
) ha
y
m
om
c
csm Guide
e
t group is to
@ resource
The reason you might want to suspend an online
n
b
e
u
perform maintenance on itthat is, start
udsome applications
akand Sstop
t
y
(
manually, but while preserving the
online
status of the group and other
in can
isbe
d
h
t
components, so that dependencies
still
honored correctly.
u
l
e
a
s
am to u
k
g suspend
The reason younmight
an offline resource group is so that it does
sewhen
a
n
not go online
automatically
you did not intend it to do so.
b
e
c
a
i
l
b
u
le
k
b
a
a
you put a resource group offline (but it is still
l yFor example,
erandwhen
f
u
s
d
managed
not
suspended),
a node failure still causes the group to go
b tran
a
- online.
ang non
ab
Resources will not automatically be restarted by fault monitors, nor

will resource groups automatically fail over, even if an entire node
fails.
To suspend a group, type:

# clrg suspend grpname
To remove the suspension of a group, type:
# clrg resume grpname
To see whether a group, is currently suspended, use clrg status , as
demonstrated on the next page.
10-42

Displaying Resource and Resource Group Status Using the clrg status and clrs
Displaying Resource and Resource Group Status Using

the clrg status and clrs status Commands
There are separate commands to show the status of resource groups and
resources. When viewing resources, you can still use -g grpname to limit
the output to a single group.
Example of Status Commands for a Single Failover

Application
s
) ha
# clrg status
y
m
Cluster Resource Groups ===

om
c
Group Name
Node Name
Suspended
Status
csm Guide
e
@ ent
-------------------------- b-----u
nfs-rg
vincent
No
Offline
ud
ak S
t
y
(
theo
Noin
is Online
d
h
t
u
al use
# clrs status -g nfs-rg
m
ka e to
g
Cluster Resources ===
ban licens
a
b
u
le State
k
b
Resource Name
Node
Name
Status Message
a
a
r
y f--------l
e
u
-----------------------------s
bd tran vincent
a
nfs-stor
Offline
Offline
g
n
n
theo
Online
Online
no
aba
orangecat-nfs
vincent
theo
Offline
Online
Offline
Online - LogicalHostname online.
nfs-res
vincent
theo
Offline
Online
Offline
Online - Service is online.

10-43
Using the clsetup Utility for Resource and Resource Group Operations
Using the clsetup Utility for Resource and Resource

Group Operations
The clsetup utility has an extensive set of menus pertaining to resource

and resource group management. These menus are accessed by choosing
Option 2 (Resource Group) from the main menu of the clsetup utility.
The clsetup utility is an intuitive, menu-driven interface that guides you
through the options without having to remember the exact command-line
syntax. The clsetup utility calls the clrg, clrs, clrslh, and clrssa
commands that have been described in previous sections of this module.
as
h
The Resource Group menu for the clsetup utility looks similar to the
)
y
m
following:
m
o
c
*** Resource Group Menu ***

e
m
d
s
i
ec t Gu
@
b den
u
k
ya Stu
(
1) Create a resource group
n
s
i
higroup
t
2) Add a network resource tolu
a dresource
e
a tousa resource group
m
3) Add a data service resource
a
k e to
4) Resource type registration
g
5) Online/Offline
a resource group
banorliSwitchover
ens
c
a
b
6) Suspend/Resume
recovery
for
a
resource group
e
u
l
k
b
7) Enable/Disable
ya fera a resource
l
u
8)
Change
d ransproperties of a resource group
b9)
a
properties of a resource
-t
g 10)onChange
n
Remove
a resource from a resource group
a
n
ab
11) Remove a resource group
12) Clear the stop_failed error flag from a resource
?) Help
s) Show current status
q) Return to the main menu
Option:
10-44

Using the Data Service Wizards in clsetup and Sun Cluster Manager
Using the Data Service Wizards in clsetup and Sun

Cluster Manager
The data service wizards, new to Sun Cluster 3.2, are yet another way to
configure application resource groups and resources.
These wizards guide you through the entire task of integrating some of
the popular applications into the cluster. They are available through the
Tasks item in Sun Cluster Manager and the Data Services submenu of
clsetup:
*** Data Services Menu ***
s
) ha
ab
y
m
om
c
1) Apache Web Server

2) Oracle
csm Guide
e
3) NFS
@ ent
b
u
4) Oracle Real Application
Clusters
akServer
tud
y
(
S
5) SAP Web Application
in Storage
d
his
t
6) Highly Available
u
l
e
aHostname
7) Logical
m
us
a
o
k
t
8) g
Shared Address
n
se
a
n
b
e
a ?) Help
lic
b
e
u
l
k rabq) Return to the Main Menu
a
y
ul nsfe
d
b
Option:
a
tra
g
n
n
a
no
The wizards are intended as an alternative to the Resource Group menu

items presented on the previous page.
The NFS wizard, for example, can create and online the NFS resource
group and resources, just like the Resource Group menu items could. It
can also create the vfstab entry for your storage (it will not actually build
your volume or file sytstem), and provision the proper share command
for you. The idea is that it can guide you through the task in a wizard-like
fashion, without even mentioning the terminology of resources and
resource groups.

10-45
Exercise: Installing and Configuring HA-NFS

Task 1 Installing and Configuring the HA-NFS Agent and Server
Task 2 Registering and Configuring the Sun Cluster HA-NFS Data

Services
Task 3 Verifying Access by NFS Clients
Task 4 Observing Sun Cluster HA-NFS Failover Behavior
Task 5 Generating Cluster Failures and Observing Behavior of the

NFS Failover
s
) ha
y
m
m
Task 7 Putting Failover Application Data in a ZFS FileoSystem
c
e
m
Task 8 Making a Customized Application Fail
Over
With
a
Generic
d
s
i
ec t Gu
Data Service Resource
@
b den
u
k
Task 9 Viewing and Managing Resources
a Stuand Resource Groups
y
(
Using Sun Cluster Managerin
d e this
u
l
a us
m
a
o
Preparation ng k se t
ba licen
a
b
e are explained in this section:
u
ltasks
The
following
k
b
a
a
l y sPreparing
er to register and configure the Sun Cluster HA for NFS data
f
u
d
ab -tran service
g
an non
Task 6 Configuring NFS to Use a Failover File System
ab
Registering and configuring the Sun Cluster HA for NFS data service
Verifying access by NFS clients
Observing Sun Cluster HA for NFS failover behavior
Note During this exercise, when you see italicized terms, such as
IPaddress, enclosure_name, node1, or clustername, embedded in a
10-46

Task 1 Installing and Configuring the HA-NFS Agent

and Server
In earlier exercises in Module 6, Using VERITAS Volume Manager With
Sun Cluster Software, or Module 7, Using Solaris Volume Manager With
Sun Cluster Software, you created the global file system for NFS.
Confirm that this file system is available and ready to configure for Sun
Cluster HA for NFS.
1.
Install the Sun Cluster HA-NFS data service agent on all nodes:
# cd sc32_location/Solaris_sparc/Product
# cd sun_cluster_agents/Solaris_10/Packages
# pkgadd -d . SUNWscnfs
s
) ha
ab
y
m
om
c
sm uide
c
e
2. Modify the /etc/default/nfs file on all nodes. Change
G the line that
t
@
n
b
currently reads:
ku tude
a
y
GRACE_PERIOD=90
n ( this S
i
d
luperiodsto
and lower the grace
e 10:
a
m
u
GRACE_PERIOD=10
ka e to
g
ban licens
a
b Thisleshould speed up HA-NFS switchovers and failovers. If your
u
Note
k
a clienterand
abserver are both running Solaris 10 OS, you will be using NFS
y
l
f
s 4 by default. This is a stateful protocol which will intentionally
bdu tranVersion
a
- delay resumption of NFS activity so that clients have a chance to reclaim
ang non
their state any time the server is recovering (which will include any
cluster switchover or failover). The GRACE_PERIOD controls the length of
the delay.
3.
Verify that the /global/nfs file system is mounted and ready for
use.
# df -h

10-47

4.
Add an entry to the /etc/hosts file on each cluster node and on the
administrative workstation for the logical host name resource
clustername-nfs. Substitute the IP address supplied by your
instructor.
IP_address
clustername-nfs
Note In the RLDC, the /etc/hosts file on the vnchost already contains
the appropriate entry for each cluster. Verify that the entry for your cluster
exists on the vnchost, and use the same IP address to create the entry on
your cluster nodes.
Perform the remaining steps on just one node of the cluster.
s
) ha
y
m
om
c
# cd /global/nfs
csm Guide
e
@ ent
# mkdir admin
b
u
# cd admin
ak Stud
y
(
# mkdir SUNW.nfs
in this
d
u
6. Create the dfstab.nfs-res
ein the
al ufile
s
m
/global/nfs/admin/SUNW.nfs
directory. Add the entry to share the
ka e to
g
/global/nfs/data
an censdirectory.
b
a
#bcd SUNW.nfs
li
e
u
l
k
b
a # eviradfstab.nfs-res
y
l
f
bdu transshare -F nfs -o rw /global/nfs/data
a
g onn
a
7. Create the directory specified in the dfstab.nfs-res file.
n
ab
5.
Create the administrative directory that contains the dfstab.nfsres file for the NFS resource.
#
#
#
#
cd /global/nfs
mkdir /global/nfs/data
chmod 777 /global/nfs/data
touch /global/nfs/data/sample.file
Note You are changing the mode of the data directory only for the
purposes of this lab. In practice, you would be more specific about the
share options in the dfstab.nfs-res file.
10-48


HA-NFS Data Services
Perform the following steps on one node only to configure HA-NFS in the
cluster:
1.
Register the NFS and SUNW.HAStoragePlus resource types.

# clrt register SUNW.nfs
# clrt list -v
2.
Create the failover resource group. Note that on a three-node cluster

the node list (specified with -n) can include nodes not physically
connected to the storage if you are using a global file system.
s
) ha
y
m
om
c
m uide
csresource
3. Create the logical host name resource to the
group.
e
G
t
@
b den
# clrslh create -g nfs-rg clustername-nfs
u
k
tu If all of your nodes are
ya resource.
(
S
4. Create the SUNW.HAStoragePlus
n
di (two-node
hiscluster, for example), you should
connected to the storage
t
u
l
e
a us to true. If you have a third, nonstorage
set the value ofm
AffinityOn
a
k esettothe value of AffinityOn to false.
node, you
should
g
an create
ns-t HAStoragePlus -g nfs-rg \
e
#ab
clrs
c
li
b -p AffinityOn=[true|false]
\
e
u
l
k
b
a
a
-p
FilesystemMountPoints=/global/nfs
nfs-stor
y fer
l
u
s
bd tran5. Create the SUNW.nfs resource.
a
# clrs create -t nfs -g nfs-rg \
ang non
# clrg create -n node1,node2,[node3] \
-p Pathprefix=/global/nfs/admin nfs-rg
ab
6.
Bring the resource group to a managed state and then online.

# clrg online -M nfs-rg
Note The very first time you do this you may see a console warning
message concerning lockd: cannot contact statd. Both daemons
are restarted (successfully) and you can ignore the message.
7.
Verify that the data service is online.

# clrs list -v
# clrs status
# clrg status

10-49
Task 3 Verifying Access by NFS Clients
Perform the following steps to verify that NFS clients can access the file
system of the Sun Cluster HA for NFS software package:
1.
On the administration workstation, verify that you can access the

cluster file system.
(# or $) ls /net/clustername-nfs/global/nfs/data
sample.file
2.
On the administration workstation, copy the test.nfs file from the

lab files location into your home directory.
3.
Edit the $HOME/test.nfs script and verify that the logical host name
and NFS file system names are correct.
s
) ha
y
m
When this script is running, it creates and writes a file containing

a
m
oalso
c
timestamp to your NFS-mounted file system. The script

displays
e
m how
idlong
cstimes
the file to standard output (stdout). This script
the
u
e
G
t
@
NFS data service is interrupted during b
switchovers
n and takeovers.
e
u
d
k
ya Stu
(
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab
10-50

Task 4 Observing Sun Cluster HA-NFS Failover

Behavior
Now that the Sun Cluster HA for NFS environment is working properly,
test its HA operation by performing the following steps:
1.
On the administration or display station, start the $HOME/test.nfs

script.
2.
On one node of the cluster, determine the name of the node currently
primary for the nfs-rg resource group.
3.
On one node of the cluster, transfer control of the NFS service from
one node to another.
s
) ha
y
m
Substitute the name of your offline node for dest-node.

om
c
e
4. Observe the messages displayed by the test.nfs
csm Gscript.
uid
e
t the switchover
@ during
5. How long was the data service interrupted
n
b
e
u
from one physical host to another?
ak Stud
y
(
in this
__________________________________________________
d
u
al and
seshare commands on all nodes to verify
u
6. Use the cldgam
status
k ethetonodes are now mounting and exporting.
g
whichn
file
systems
a cens
b
a
li
b __________________________________________________
e
u
l
k
a e__________________________________________________
ab
r
y
l
f
bdu trans __________________________________________________
a
- 7. Use the ifconfig command on all nodes to observe the additional
ang non
# clrg switch -n dest-node nfs-rg
ab
IP address associated with the Logical Hostname resource

configured as a virtual interface on one of the adapters in your IPMP
group.
# ifconfig -a
8.
On one node of the cluster, use the clrg switch command to

transfer control of the NFS service back to its preferred host.
# clrg switch -n dest-node nfs-rg

10-51
Task 5 Generating Cluster Failures and Observing

Behavior of the NFS Failover
Generate failures in your cluster to observe the recovery features of Sun
Cluster 3.2 software. If you have physical access to the cluster, you can
physically pull out network cables or power down a node. If you do not
have physical access to the cluster, you can still bring a node to the ok
prompt using a break signal through your terminal concentrator.
Try to generate the following failures:
Node failure (power down a node or bring it to the ok prompt)
s
Single public network interface failure (pull a public network cable h
,a
)
or sabotage the adapter using ifconfig modinsert as youm
didyin
Module 8)
om
c
Multiple public network failure on a single node

csm Guide
e
@ ent
b
u
Try your tests while the resource groupais
nonsuspended
k in its tnormal,
ud group:
y
(
S
state. Repeat some tests after suspending
the
resource
in this
d
u
# clrg suspend nfs-rg
al use
m
ka e to
g
s your results, remove the suspension:
When you are
nwith
bansatisfied
e
c
a
li nfs-rg
#bclrg resume
e
u
l
k
a erab
y
l
u nsf
d
b
a
tra
g
n
n
a
no
ab
10-52

Task 6 Configuring NFS to Use a Failover File

System
In this task, you configure NFS to use a failover file system rather than a
global file system.
1.
Disable the NFS resource group.

# clrg offline nfs-rg
2.
If you have a nonstorage third node, delete the nonstorage node from
the group node list, and set the affinity:
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
3. Unmount the global NFS
infile system.
d
his
t
u
l
e
a us
# umount /global/nfs
m
a
k editethe
to/etc/vfstab file to make /global/nfs a
4. On eachgnode,
n system
ns(if you have a non-storage node, you can either
local
bafile
e
c
a
liedit, or just remove the line completely):
b makelthis
e
u
k
a ea.rabChange yes to no in the mount-at-boot column.
y
l
u nsf
d
b
b. Remove the word global from the mount options. Replace it
a
tra
g
n
n
with a minus sign (-). Make sure you still have seven fields on
no
aba
that line.
#
#
#
#
#
#
5.
clrg
clrs
clrs
clrs
clrs
clrs
remove-node -n nonstorage-node-name nfs-rg

disable nfs-res
disable nfs-stor
set -p AffinityOn=TRUE nfs-stor
enable nfs-stor
enable nfs-res
Restart the NFS resource group.

# clrg online nfs-rg
6.
Observe the file system behavior. The file system should be mounted
only on the node running the NFS service and should fail over
appropriately.
Note Ordinarily, just by convention, you wouldnt have a failover file

system mounted under a subdirectory of /global. However, it is all just a
convention, and keeping the same mount point simplifies the flow of
these exercises.

10-53
Task 7 Putting Failover Application Data in a ZFS File

System
In this task, you will migrate your server-side data to a failover ZFS
filesystem.
Note This task assumes that there are no longer any non-storage nodes
included in your NFS resource group. This should be the case if you have
completed Task 6. If you have a non-storage node in your resource group
(and have not done Task 6), remove it from the group using clrg
remove-node -n nonstorage-node-name.
s
) ha
y
m
om
c
m uide
csZFS
1. Identify the node that is currently importingethe
storage pool
G
t
@
n
(nfspool) that you created in the ZFS
module.
b
ku tude
a
y
# zpool status
n ( this S
i
d
lu se
a
2. On the node identified
step
u 1, verify that the nfspool/nfs dataset
am int(this
o
is mounted g
at k
/testnfs
will match the destination for the cpio
e
n
s
a
n
done in
ce
ab stepl4).
i
b
u# df -hble| grep nfspool/nfs
k
a
rayour NFS failover resources (and current failover filesystem)
y3. Move
l
e
f
u
s
bd tran to the node identified in step 1:
a
g onn
a
# clrg switch -n node_importing_the_zpool nfs-rg
n
ab
Peform the following steps only on the single node that is currently the
primary for the NFS resource group:
4.
From the node identified in step 1, disable your NFS service and
migrate the data to your ZFS file system:
#
#
#
#
#
10-54
clrs disable nfs-res

clrs delete nfs-res
cd /global/nfs
find . -print|cpio -pdmuv /testnfs
cd /

5.
Disable your old (non-ZFS) NFS resource, and remount the ZFS
under /global/nfs. If you have already done task 6 then the
traditional /global/nfs is already a failover file system, and it will
be unmounted by the clrs disable (so the umount is there just in
case you had not done task 6).
#
#
#
#
#
6.
clrs disable nfs-stor

clrs delete nfs-stor
umount /global/nfs
zfs set mountpoint=/global/nfs nfspool/nfs
df -k
Recreate your NFS resources using the ZFS storage:

# clrs create -g nfs-rg -t HAStoragePlus \
-p Zpools=nfspool nfs-stor
# clrs create -g nfs-rg -t nfs \
# clrg status
# clrs status
# df -k
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
d file system from your
aklegacy
tuNFS
y
7. Remove or comment out the
old
(
S
in this
/etc/vfstab on bothdnodes.
u
e behavior of the NFS application,
al andufailover
s
m
8. Observe switchover
a
know
to the zpool containing your ZFS file system.
whichn
will
include
g
e
ba licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

10-55
Task 8 Making a Customized Application Fail Over

With a Generic Data Service Resource
In this task, you can see how easy it is to get any daemon to fail over in
the cluster, by using the Generic Data Service (so that you do not have to
invent your own resource type).
Perform the following steps on the nodes indicated:
1.
On all nodes (or do it on one node and copy the file to other nodes in
the same location), create a daemon that represents your customized
application:
# vi /var/tmp/myappdaemon
#!/bin/ksh
while :
do
sleep 10
done
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
k nodes.
aall
tud
2. Make sure the file is executable(y
on
S
n failover
inew
d
his resource group for your
t
3. From any one node, create
a
u
l
e
a us
application:
m
a
k -n node1,node2,[node3]
to
g
e
# clrg create
myapp-rg
n ens
a
b
a one node,
4. From
licregister the Generic Data Service resource type:
b
e
u
l
k # clrt
a
abregister SUNW.gds
r
y
l
e
f
bdu tr5.ansFrom one node, create the new resource and enable the group:
a
g onn
# clrs create -g myapp-rg -t SUNW.gds \
a
n
b
a
-p Start_Command=/var/tmp/myappdaemon \
-p Probe_Command=/bin/true -p Network_aware=false \
myapp-res
# clrg online -M myapp-rg
6.
10-56
Verify the behavior of your customized application.

a.
Verify that you can manually switch the group from node to
node.
b.
Kill the daemon. Wait a little while and note that it restarts on
the same node. Wait until clrs status shows that the
resource is fully online again.
c.
Repeat step b a few times. Eventually, the group switches over

to the other node.

Task 9 Viewing and Managing Resources and

Resource Groups Using Sun Cluster Manager
Perform the following steps on your administration workstation:
1.

node:
2.
Log in as root and enter the Sun Cluster Manager Application.
3.
Click the Resource Groups folder on the left.
4.
Investigate the status information and graphical topology

information that you can see regarding resource groups.
s
) ha
y
m
om
c
6. Select the check box next to a resource group, m

and use the
e Switch
d
s
i
c
u
Primary button to switch the primary node
group.
e for at resource
G
@
n
b
ku tude
a
y
n ( this S
i
d
lu se
a
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba
5.
Go back to the Status Panel for Resource Groups.

10-57
Exercise Summary
Exercise Summary

Experiences
Interpretations
Conclusions
Applications
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
10-58

Module 11
Configuring Scalable Services and

Advanced Resource Group Relationships
s
) ha
Objectives
ab
y
m
om
c
csm Guide
Describe the characteristics of scalable services
e
@ ent
b
u
Describe the function of the load-balancer
ak Stud
y
(
Create the failover resource
in group
isfor the SharedAddress resource
d
h
t
u
egroup for the scalable service
alresource
Create the scalable
s
m
u
kathe eSharedAddress
to
Describeghow
resource works with scalable
n
s
a
n
services
ab lice
b
u Addbauxiliary
le
nodes
k
a
a
r
y
ul nsfeCreate these resource groups
d
b
a
tra
Control scalable resources and resource groups
g
n
n
a
no
View scalable resource and group status
Configure and run Apache as a scalable service in the cluster
11-1
Relevance
Relevance

Web servers are supported as scalable applications. But will every

application within a web server necesarily behave properly in a
scalable environment?
Why might you want a client affinity for load balancing?
What reason might you have to have separate resource groups with
a requirement that group A run only on a node also running group
B? Why not just combine them into one group?
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
11-2

820-7458 (for Sun Cluster 3.2 11/09).
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Configuring Scalable Services and Advanced Resource Group Relationships

11-3
Using Scalable Services and Shared Addresses
Using Scalable Services and Shared Addresses

The scalable service architecture allows some services, such as Apache

web server, to run simultaneously on multiple nodes or non-global zones,
while appearing to the client to be running on a single server.
Clients connect to such a service using a single IP address called a shared
address. Typically, clients do not know which node or zone they connect to,
nor do they care.
Figure 11-1 shows the architecture of scalable services and shared
addresses.
y
m
om
c
csm Guide
e
Client 1
Client 2
@Client e3 nt
b
u
Requests
Requests
ak Stud
y
Network
(
in this
d
u
Node 1
al use
m
a
to
Global Interface g k
e
Web
xyz.coman
b licens
Page
a
bRequest le Transport
u
k
a Distribution
ab
r
y
Node 2
Node 3
l
e
f
u
s
d
ab -tran
g
n non
HTTP
a
b
HTTP
HTTP
Application
a
s
) ha
Application
Application
<HTML>
<HTML>
<HTML>
</HTML>
</HTML>
</HTML>
Globally Available HTML Documents
Figure 11-1 Scalable Services, Shared Addresses Architecture
11-4

Exploring Characteristics of Scalable Services
Exploring Characteristics of Scalable Services

Similar to the failover services described in the previous modules, the
applications used as scalable services in the Sun Cluster 3.2 environment
are generally off-the-shelf applications that are not specifically compiled
or released to run in the cluster. The application binaries running on the
various nodes are unaware of each other. This works because of the Sun
Cluster 3.2 SharedAddress mechanism.
File and Data Access
abes
Similar to the failover services, all the data for the scalable service must
h
)
y
in the shared storage.
m
om
c
e must use
Unlike the failover service, a file system-oriented scalable
service
sm can
idstill
cresource
u
e
the global file system. A SUNW.HAStoragePlus
be
G
t
@
n
b
configured to manage dependencies between
eservice and the storage.
u uthe
d
k
a
t
(y is S
n
i
th
ludData
File Locking for Writing
e
a
am to us
k
ngbarriers
seto taking just any application and turning it into a
One of the
big
a
n
b
e
a servicelicis that a service that is cluster-unaware may have been
scalable
b
u
le a way as to ignore file-locking issues.
k
b
writtenain
such
a
l y sfer
u
d
ab -tranIn Sun Cluster 3.2 software, an application that performs data
g
n non
modification without any type of locking or file synchronization
a
b
a
mechanism generally cannot be used as a scalable service.
While Sun Cluster 3.2 software provides the global data access methods, it
does not automatically call any file-locking primitives for you.
Web servers that do file writing in common gateway interface (CGI)
scripts, although not written specifically for a scalable platform, are
usually written in such a way that multiple instances can be launched
simultaneously even on a stand-alone server. Therefore, they already have
the locking in place to make them ideal to work as scalable services in Sun
Cluster 3.2 software.
Web servers that perform file writing using Java servlets must be
examined much more closely. A servlet might use thread synchronization
rather than file locking to enforce serial write access to a critical resource.
This does not translate properly into a scalable service.

11-5
Using the SharedAddress Resource
Using the SharedAddress Resource

The glue that holds a scalable service together in Sun Cluster 3.2 software
is the SharedAddress resource.
This resource provides not only a single IP address that makes the
scalable service look similar to a single server from the point of view of
the client, but also provides the load balancing of requests to all the nodes
or zones on which a scalable service is active.
Client Affinity
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
A single SharedAddress
ka resource
to can provide standard (per connection
g
e
load balancing)
thentype
s of sticky load balancing required by
banAand
e
shoppingacarts.
scalable
services agent registers which type of load
c
li
b
e
u
l
balancing
k rais brequired, based on a property of the data service.
a
y
ul nsfe
d
b
a Load-Balancing
tra
g
Weights
n
n
o
a
n
ab
Certain types of applications require that load balancing in a scalable

service be on a per client basis, rather than a per connection basis. That is,
they require that the same client IP always have its requests forwarded to
the same node or zone. The prototypical example of an application
requiring such client affinity is a shopping cart application, where the state
of the clients shopping cart is recorded only in the memory of the
particular node where the cart was created.
Sun Cluster 3.2 software also lets you control the weighting that should be
applied for load balancing. The default weighting is equal (connections or
clients, depending on the stickiness) per node or zone, but through the
use of properties described in the following sections, a better node or zone
can be made to handle a greater percentage of requests.
Note You could have a single SharedAddress resource that provides
load balancing for the same application using multiple IP addresses on
the same subnet. You could have multiple SharedAddress resources that
provide load balancing for the same application using IP addresses on
different subnets. The load balancing properties discussed on this page
are set per scalable application, not per address.
11-6

Exploring Resource Groups for Scalable Services

A scalable service requires the creation of two resource groups. A failover
resource group contains the SharedAddress resource. It is online, or
mastered, by only one node at a time. The node that masters the
SharedAddress resource is the only one that receives incoming packets
for this address from the public network. A scalable resource group
contains an HAStoragePlus resource and the actual service resource.
The HAStoragePlus resource is added to guarantee that the storage is
accessible on each node before the service is started on that node.
Figure 11-2 is a block diagram showing the relationship between scalable
and failover resource groups.
s
) ha
y
m
m
oGroup
c
Scalable Resource Group
Failover Resource
e
m
d
s
i
ec t Gu
@
n
b SUNW.SharedAddress
e
u
SUNW.HAStoragePlus
d
k
tu
ya SResource
(
Resource
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
bResource_dependencies
u
le
k
b
a
a
l y sfer
u
d
ab -tran SUNW.apache Resource
g
n non
a
b
a
Figure 11-2 Scalable and Failover Resource Groups

11-7
Resources and Their Properties in the Resource

Groups
Table 11-1 and Table 11-2 demonstrate the properties and contents of the
two resource groups required to implement a scalable service:
Resource group name: sa-rg
Properties:[Nodelist=vincent,theo Mode=Failover
Failback=False...]
Resource group contents: See Table 11-1.
Table 11-1 sa-rg Resource Group Contents
s
) ha
y
m
m
o
c
apache-lh
SUNW.SharedAddress
HostnameList=apache-lh
e
m
d
s
i
Netiflist=therapy@1,therapy@2
ec t Gu
@
b den
u
k
ya Stu
Resource group name: apache-rg
(
n
di e this Mode=Scalable
u
Properties: [Nodelist=vincent,theo
l
a Maximum_primaries=2]
m
us
Desired_primaries=2
a
o
k
t
g contents:
ngroup
se See Table 11-2.
Resource
a
n
b
e
a
lic
b
e
u
l
Table 11-2 apache-rg
k Resource
b Group Contents
a
a
r
y
fe
ul
d
Resource
Name ansResource Type
Properties
b
a
r
t
g
n non
a
web-stor
SUNW.HAStoragePlus FilesystemMountPoints=/global/web
b
a
Resource Name
Resource Type
Properties
AffinityOn=False
apache-res
11-8
SUNW.apache
Bin_dir=/global/web/bin
Load_balancing_policy=LB_WEIGHTED
Load_balancing_weights=3@1,1@2
Scalable=TRUE
Port_list=80/tcp
Resource_dependencies=web-stor,
apache-lh

Understanding Properties for Scalable Groups and Services
Understanding Properties for Scalable Groups and

Services
Certain resource group properties and certain resource properties are of
particular importance for scalable services.
The Desired_primaries and Maximum_primaries

Properties
These are group properties that indicate how many nodes the service
should run on. The rgmd tries to run the service on Desired_primaries
nodes, but you can manually bring it online to more nodes to using clrg
online, up to Maximum_primaries.
s
) ha
y
m
om
c
ideis
csm property
u
If these values are greater than 1, the Mode=Scalable
e
G
t
@
automatically set.
n
b
ku tude
a
y
n ( this S
i
d
Note If you create a resource
using clrg create -S, the
lu sgroup
e
a
u
Desired_primaries
properties are both
amand tMaximum_primaries
o
k
automatically
ngset etonthe
senumber of nodes in the nodelist for the group.
a
b
a
lic
b
e
u
l
k rab
a
y
l Load_balancing_policy
fe
Property
uThe
s
d
n
b
a
tra
g
n
n
This is a property of the data service resource. It has one of the following
no
aba
values:
Lb_weighted Client connections are all load balanced. This is the

default. Repeated connections from the same client might be serviced
by different nodes.
Lb_sticky Connections from the same client IP to the same server

port all go to the same node. Load balancing is only for different
clients. This is only for the ports listed in the Port_list property.
Lb_sticky_wild Connections from the same client to any server

port go to the same node. This is good when port numbers are
generated dynamically and not known in advance.

11-9
Understanding Properties for Scalable Groups and Services
The Load_balancing_weights Property

This property controls the weighting of the load balancer. A value such as
5@1,3@2 indicates node 1 will handle five requests (or clients, if using
sticky policies) for every three handled by node 2. The default is that all
nodes handle the same volume.
The Resource_dependencies Property

You must set this property on a scalable resource. Its value must include
the SharedAddress resource (in the other resource group). This is used by
the data service agent to register the data service with the load balancer
associated with the SharedAddress resource.
s
) ha
y
m
om
c
The same Resource_dependencies property of the scalable

e also
m uservice
d
s
i
c
typically includes the HAStoragePlus resource inethe same
Ggroup.
t
@
n
b
ku tude
a
y
n ( this S
i
d
lu se
a
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
a
no
ab
11-10

Adding Auxiliary Nodes for a SharedAddress Property
Adding Auxiliary Nodes for a SharedAddress Property

The SharedAddress logic includes a routine whereby the actual IP
address associated with the resource is configured on the public network
adapter (IPMP groups) on the primary node and is configured as a virtual
address on the loopback network on all other nodes in its nodelist.
This enables scalable services that are running on nodes other than the
primary SharedAddress node to still bind to the IP address used for the
SharedAddress.
However, the SharedAddress resource must know about all possible
nodes on which any scalable service associated with it might run.
s
) ha
ab
y
m
If the node list for the SharedAddress resource group is o

am
superset of the
c
But
node list of every scalable service that might run onm
it, you are
fine.
e
d
s
i
c the primary
u for the
you might want to restrict which nodes mightebe
G
t
@
SharedAddress, while still allowing a u
larger
nlist for the scalable
b node
e
d
k
services dependent on it.
ya Stu
(
n
di hasea tspecial
his auxiliary nodes property that
u
l
The SharedAddress resource
a us
m
allows you to augment
node list of its group, just for the purposes of
a
knodesethe
toscalable
g
supporting n
more
for
services. This is set with the -X option
s
a
n
b
e
to clrssa
a create.
lic
b
e
u
l
k rab
a
y
l Insthe
e following example, you want only nodes vincent and theo to be
f
u
d
primaries for the SharedAddress (to actually host it on the public net
ab -tranthe
g
and
do the load balancing). But you might be supporting scalable services
an non
that run on vincent, theo, and apricot:
# clrg create -n vincent,theo sa-rg

# clrssa create -g sa-rg -X apricot apache-lh

11-11
Reviewing Command Examples for a Scalable Service
Reviewing Command Examples for a Scalable Service

The following examples assume that the Apache web server agent was
added from the Sun Cluster 3.2 Data Services CD using the Java ES
installer or the pkgadd command.
1.
Register the resource types:

# clrt register SUNW.apache
2.
Create the failover resource group for the SharedAddress by typing:

# clrg create -n vincent,theo sa-rg
3.
Create the SharedAddress resource:
y
m
om
4. Bring the resource group online:
c
# clrg online -M sa-rg

csm Guide
e
5. Create the scalable resource group: ub@ ent
d
ak S
y
# clrg create -S -n vincent,theo
\ tu
(
in this
apache-rg
d
u
e
al resource:
s
m
u
6. Create the HAStoragePlus
ka-t SUNW.HAStoragePlus
to
g
e
# clrs create
-g apache-rg \
n ens
a
b
-pa FilesystemMountPoints=/global/web
\
lic
b
e
u
-p
AffinityOn=false
\
l
k rab
a
y
l
eweb-stor
f
u
s
d
ab -tr7.an Create the Apache service:
g
an non
# clrs create -t SUNW.apache -g apache-rg \
s
) ha
# clrssa create -g sa-rg apache-lh
ab
-p Resource_dependencies=web-stor,apache-lh \
-p Bin_dir=/global/web/bin \
-p Port_list=80/tcp \
-p Scalable=TRUE \
apache-res
8.
Online the resource group:

# clrg online apache-rg
11-12

Controlling Scalable Resources and Resource Groups

The following examples demonstrate how to use the clrg and clrs
commands to control the state of resource groups, resources, and data
service fault monitors.
Resource Group Operations

Use the following commands to:
Online an offline resource group onto its preferred nodes.

# clrg online [-M] [-e] apache-rg
s
) ha
ab
y
m
om
c
csm Guide
e
# clrg online -n node,node [-M]
t
@[-e]eapache-rg
n
b
u
akto theSspecified
Switch a scalable resource group
tud nodes. On any other
y
(
nodes, it is switched off.
in this
d
u
l [-e]
e-n node,node,.. apache-rg
a[-M]
# clrg switch
s
m
u
a
kscalable
to
Remaster
a
resource
group. The group switches to its most
g
e
n
s
a
n
preferred
b linodes
e (the number of nodes is controlled by
c
a
b
It may go offline on some nodes if it was
u Desired_primaries).
le
k
b
a
a
previously
running
on
more nodes or on nodes that were less
y fer
l
u
s
bd tran preferred.
a
# clrg remaster apache-rg
ang non
Online a resource group on specific nodes. Any nodes not mentioned

are not affected (if the group is running on other nodes, it just stays
running there).
Offline a resource group (it goes off of all nodes):

# clrg offline apache-rg
Offline a resource group on specific nodes. Any nodes not mentioned

are not affected (if the group is running on other nodes, it just stays
running there).
# clrg offline -n node,node apache-rg
Restart a resource group:

# clrg restart apache-rg
Restart a resource group on a particular node or nodes:

# clrg restart -n node,node,... apache-rg

11-13
Resource Operations
Disable a resource and its fault monitor on all nodes:

# clrs disable apache-res
Disable a resource and its fault monitor on specified nodes:

# clrs disable -n node,node,... apache-res
Enable a resource and its fault monitor on all nodes:

# clrs enable apache-res
s
) ha
Enable a resource and its fault monitor on specified nodes:
y
m
om
c
Fault Monitor Operations

csm Guide
e
@ ent
b
u
ak Stud
y
(
in on allthnodes:
is
d
Disable just the fault monitor
u
l
e
aapache-res
m
us
# clrs unmonitor
a
o
k
t
Disable a
just
ngthe efault
semonitor on specified nodes:
n
b
a unmonitor
lic -n node,node,... apache-res
#bclrs
e
u
l
k Enable
a
aba fault monitor on all nodes:
r
y
l
e
f
bdu trans# clrs monitor apache-res
a
g onn
Enable a fault monitor on specified nodes:
a
n
ab
# clrs enable -n node,node,... apache-res
# clrs monitor -n node,node,... apache-res
11-14

Using the clrg status and clrs status Commands for a Scalable Application
Using the clrg status and clrs status Commands for

a Scalable Application
Use the status subcommands as follows to show the state of resource
groups and resources for a scalable application:
# clrg status
Cluster Resource Groups ===
Group Name
---------sa-rg
Node Name
--------vincent
theo
Suspended
--------No
No
Status
-----Online
Offline
s
) ha
y
m
om
c
web-rg
vincent
No
Online
theo
No
Online
csm Guide
e
@ ent
b
u
ak Stud
y
(
# clrs status -g sa-rg,web-rg
in this
d
u
al use
Cluster Resources ===
m
ka e to
g
Resource Name
NodebName
an censState Status Message
a
li
-----------------------------b--------e
u
l
k
orangecat-web
Online Online - SharedAddress online.
a vincent
ab
r
y
l
e
theo
Offline Offline
f
u ns
d
b
a
tra vincent
g
n
web-stor
Online Online
n
o
a
n
b
theo
Online Online
a
apache-res
vincent
theo
Online
Online
Online - Service is online

Online - Service is online.

11-15

Sun Cluster 3.2 software offers a series of advanced resource group
relationships called resource group affinities.
Resource group affinities provide a mechanism for specifying either a
preference (weak affinities) or a requirement (strong affinities) that certain
resource groups either run on the same node (positive affinities) or do not
run on the same node (negative affinities).
In this section, the words source and target are used to refer to the resource
groups with an affinities relationships. The source is the group for which
the value of RG_affinities is set, and the target is the group referred to
by the value of the property. So, in the following example:
s
) ha
y
m
om
# clrg set -p RG_affinities=++rg1 rg2
c
csm Guide
e
rg2 is referred to as the source and rg1 as the @
target.
t
n
b
e
u
ak Stud
y
(
din Weak
hisNegative Affinities
Weak Positive Affinitiesluand
t
e
a us
m
a
kof affinities
toplace only a preference, not a requirement,
g
The first two kinds
e
ns group.
banoflthe
e
on the location
source
c
a
i
b
e
u
l
k
a ea rweak
ab positive affinity says that the source group prefers to switch
y
Setting
l
f
nsnode already running the target group, if any. If the target group is
bdu trtoathe
a
- not running at all, that is fine. You can freely switch online and offline
ang non either group, and freely, explicitly, place either on any node that you want.
ab
Setting a weak negative affinity means that the source group prefers any
other node besides the target. Once again, it is just a preference. You can
freely, explicitly, place either group on whatever node you want.
Weak positive and weak negative group affinities are denoted by a single
plus or single minus sign, respectively.
In this example, group rg2 is given a weak positive affinity for rg1. This
does not affect the current location of either group.
# clrg set -p RG_affinities=+rg1 rg2
WARNING: resource group rg2 declares a weak positive
affinity for resource group rg1; but this affinity is not
satisfied by the current state of the two resource groups
11-16


The following will be affected by a weak positive affinity (if the target
group is actually online):
Failover of the source group

If the target is online, when the source group needs to fail over it will
fail over to the node running the target group, even if that node is
not a preferred node on the source groups node list.
Putting the resource group online onto a nonspecified node:

# clrg online source-grp
Similary, when a source group goes online and you do not specify a
specific node, it will go onto the same node as the target, even if that
node is not a preferred node on the source groups node list.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
# clrg switch -n specific-node
aksrc-grp
tud
y
(
S
in this
d
u
l
There can be multiple a
resource
groups
e as the value of the property. In
s
m
u
other words, a source
can
have
more
than one target. In addition, a source
a
o
k
t
can have both
positive
weak negative affinities. In these cases,
ngweak
se aand
a
n
b
e
the source
prefers
to
choose
node
satisfying the greatest possible
c
a
i
l
b
number
of lweak
e affinities. For example, select a node that satisfies two
u
k
b
a
a
weak
positive
affinities and two weak negative affinities rather than a
y fer
l
u
s
that satisfies three weak positive affinities and no weak negative
bd trannode
a
affinities.
g onn
a
n
ab
However, weak affinities are not enforced when you manually bring or
switch a group onto a specific node. The following command will
succeed, even if the source group has a weak affinity for a target running
on a different node.
Strong Positive Affinities

Strong positive affinities (indicated with a ++ before the value of the
target) place a requirement that the source run on the same node as the
target. This example sets a strong positive affinity:
# clrg set -p RG_affinities=++rg1 rg2
The following applies:
The only node or nodes on which the source can be online are nodes
on which the target is online.
If the source and target are currently running on one node, and you
switch the target to another node, it drags the source with it.

11-17
If you offline the target group, it will offline the source as well.
An attempt to switch the source to a node where the target is not

running will fail.
If a resource in the source group fails, the source group still cannot
fail over to a node where the target is not running. (See the next
section for the solution to this example.)
The source and target are closely tied together. If you have two failover
resource groups with a strong positive affinity relationship, it might make
sense to make them just one group. So why does strong positive affinity
exist?
s
) ha
The relationship can be between a failover group (source) and a

scalable group (target). That is, you are saying the failover group
must run on some node already running the scalable group.
y
m
om
c
You might want to be able to offline the source group

leave
e the
m but
d
s
i
c
u
target group running, which works with thiserelationship.
G
t
@
n
b
e resources in the
u to put
For some reason, you might not be k
able
dsome
u
a
t
y
( check
same group (the resources might
and
S reject you if you try to
n
s
i
i
d
h
put it in the same group
as
another
resource).
But you still want
u se t
l
a
them all to be running
am ontotheu same node or nodes.
k
ng ense
a
b
a Affinity
lic
Strong Positive
With Failover Delegation
b
e
u
l
k
b
ya fera
l
u
ns variation on strong positive affinity is set with the +++ syntax:
bd trAaslight
a
g on- # clrg set -p RG_affinities=+++rg1 rg2
n
a
n
ab
The only difference between the +++ and the ++ is that here (with +++), if
a resource in the source group fails and its fault monitor suggests a
failover, the failover can succeed. What happens is that the RGM moves
the target group over to where the source wants to fail over to, and then
the source is dragged correctly.
11-18

Strong Negative Affinity
A strong negative affinity is set using the following syntax:

# clrg set -p RG_affinities=--rg1 rg2
Here, the source cannot run on the same node as the target. It absolutely
refuses to switch to any node where the target is running.
If you switch the target to the node where the source is running, it chases
the source out of the way and the source switches to a different node, if
any. If there are no more nodes, the source switches offline.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
You need to be careful with strong negative
d because the HA of
k taffinities
u
a
y
the source group can be compromised.
n ( this S
i
d
lu se
a
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba
For example, if you have a two-node cluster with both the source and
target groups online (on different nodes), and one node crashes
(whichever it is), only the target group can remain running because the
source group absolutely, categorically refuses to run on the same node.

11-19
Exercise: Installing and Configuring Sun Cluster Scalable Service for Apache
Exercise: Installing and Configuring Sun Cluster Scalable

Service for Apache
Task 1 Preparing for Apache Data Service Configuration
Task 2 Configuring the Apache Environment
Task 3 Testing the Server on Each Node Before Configuring the

Data Service Resources
Task 4 Registering and Configuring the Sun Cluster Apache Data

Service
s
) ha
y
m
Task 6 Observing Cluster Failures

om
c
m uide
csRelationships
e
G
t
@
n
b
ku tude
a
y
Preparation
n ( this S
i
d
lu se
a
The following tasks are
uin this section:
amexplained
o
k
t
g SunnCluster
nfor
se HA for Apache registration and
Preparing
a
b
e
a
configuration
lic
b
e
u
l
k Registering
a
ab and configuring the Sun Cluster HA for Apache data
r
y
l
e
f
bdu transservice
a
g onVerifying Apache web server access and scalable capability
n
a
n
b
a
Task 5 Verifying Apache Web Server Access
Note During this exercise, when you see italicized names, such as
IPaddress, enclosure_name, node1, or clustername embedded in a
11-20

Task 1 Preparing for Apache Data Service

Configuration
1.
Install the Sun Cluster Apache data service software package:

# pkgadd -d . SUNWscapc
2.
Create an entry in /etc/hosts for the shared address you will be

configuring with the Apache web server:
IP_address
clustername-web
s
) ha
ab
y
m
om entry
Note In the RLDC, the vnchost already contains thec
appropriate
dethe
for each cluster. Verify that the entry for your cluster
smexistsuion
c
e
G on your cluster
vnchost, and use the same IP address to create
the entry
t
@
n
b
nodes. In a non-RLDC environment, create
/etc/hosts
entry on your
de
ku the
u
a
t
y
administrative workstation as well.
n ( this S
i
d
lu se
a
am to u
k
Task 2 Configuring
ng ensethe Apache Environment
a
b
a
lic
b
e
u
l
kOn (any)
bone node of the cluster, perform the following steps:
a
a
r
y
ul n1.sfeMake a resource-specific copy of the /usr/apache2/bin/apachectl
d
b
a
tra
script and edit it:
g
n
n
o
a
n
# mkdir /global/web/bin
# cp /usr/apache2/bin/apachectl /global/web/bin
# vi /global/web/bin/apachectl
a.
Add a line to make an Apache runtime directory. This directory

is deleted every time you reboot. Letting this script create it
resolves the problem. Just add the line in bold as the second line
of the file:
#!/bin/sh
mkdir -p /var/run/apache2
b.
Locate the line:

HTTPD='/usr/apache2/bin/httpd'
And change it to:
HTTPD='/usr/apache2/bin/httpd -f /global/web/conf/httpd.conf'

11-21
2.
Copy the sample /etc/apache2/httpd.conf-example to

/global/web/conf/httpd.conf.
# mkdir /global/web/conf
# cp /etc/apache2/httpd.conf-example /global/web/conf/httpd.conf
3.
Edit the /global/web/conf/httpd.conf file, and change the

following entries, as shown in Table 11-3. The changes are shown in
their order in the file, so you can search for the first place to change,
change it, then search for the next, and so on.
Table 11-3 Entries in the /global/web/conf/httpd.conf File

Old Entry
New Entry
KeepAlive On
KeepAlive Off
s
) ha
y
m
Listen 80
Listen clustername-web:80 om
c e
m
ServerName 127.0.0.1
ServerName clustername-web
s
c Guid
e
@ ent
DocumentRoot "/var/apache2/htdocs" DocumentRoot b
"/global/web/htdocs"
u
ak"/global/web/htdocs">
tud
y
<Directory "/var/apache2/htdocs">
<Directory
(
S
in this
d
u
ScriptAlias /cgi-bin/
e /cgi-bin/
al ScriptAlias
s
m
u
"/var/apache2/cgi-bin/" [one kline]
"/global/web/cgi-bin/"
[one line]
a
to
g
e
<Directory "/var/apache2/cgi-bin">
ban licens <Directory "/global/web/cgi-bin">
a
b
u
le
k
b
a
a
l y4. sCreate
er directories for the Hypertext Markup Language (HTML)
f
u
d
ab -tran and CGI files and populate with the sample files.
g
# cp -rp /var/apache2/htdocs /global/web
n non
a
b
# cp -rp /var/apache2/cgi-bin /global/web
a
5.
Copy the file called test-apache.cgi from the classroom server to

/global/web/cgi-bin. You use this file to test the scalable service.
Make sure that test-apache.cgi can be executed by all users.
# chmod 755 /global/web/cgi-bin/test-apache.cgi
11-22

Task 3 Testing the Server on Each Node Before

Configuring the Data Service Resources
Perform the entire task, steps 1-7, on one node at a time.
1.
Temporarily configure the logical shared address (on one node).
# ifconfig pubnet_adapter addif clustername-web netmask + broadcast + up

2.
Start the server (on that node).
# /global/web/bin/apachectl start
3.
Verify that the server is running.
as
h
?
0:00 /usr/apache2/bin/httpd
)
y
start
m
?
om
c
start
csm Guide
e
?
@ ent
b
start
u
d
ak 0:01
?
tu/usr/apache2/bin/httpd
y
(
S
start
in this
d
u
al? use 0:00 /usr/apache2/bin/httpd
m
ka e ?start
to
g
n ens
a
b
start
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba
vincent:/# ps -ef|grep apache2
webservd 4604 4601
0 10:20:05
-f /global/web/conf/httpd.conf -k
webservd 4603 4601
0 10:20:05
webservd 4605 4601
0 10:20:05
root 4601
1
0 10:20:04
webservd 4606 4601
0 10:20:05
webservd 4602 4601
0 10:20:05

11-23
4.
Connect to the server from the web browser on your administration

or display station. Use http://clustername-web (Figure 11-3).
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
dinPagee this
Figure 11-3 Apache Serverlu
Test
a us
m
a
k e to
g
n econnect,
ns you might need to disable proxies or set a
Note If you
bacannot
c
a
i
bexception
proxy
inl your web browser.
e
u
l
k
a erab
y
l
f the Apache web server.
u 5. nsStop
d
b
a
tra
g
#n
/global/web/bin/apachectl
stop
n
o
a
n
ab
6. Verify that the server has stopped.
# ps -ef | grep apache2
root 8394 8393 0 17:11:14 pts/6
7.
0:00 grep apache2
Take down the logical IP address.
# ifconfig pubnet_adapter removeif clustername-web
11-24


Apache Data Service
Perform the following steps only on (any) one node of the cluster:
1.
Register the resource type required for the Apache data service.
# clrt register SUNW.apache
2.
Create a failover resource group for the shared address resource. If

you have more than two nodes, you can include more than two
nodes after the -n.
# clrg create -n node1,node2 sa-rg
3.
s
) ha
Add the SharedAddress logical host name resource to the resource

group.
ab
y
m
# clrssa create -g sa-rg clustername-web com
e
m
d
s
i
4. Bring the failover resource group online. c
e t Gu
@
# clrg online -M sa-rg
b den
u
k
u all nodes of the cluster.
ton
ya to run
5. Create a scalable resource group
(
S
n
i nodes.)
(The example assumes
dtwo
his
t
u
l
e
a-S -n unode1,node2
s
# clrg create
web-rg
m
a
o
k
t
g nresource
6. Add anstorage
se to the scalable group.
a
b
e
a -tlicSUNW.HAStoragePlus \
# clrs create -gbweb-rg
u
le
k
-p FilesystemMountPoints=/global/web
-p AffinityOn=false web-stor
b
a
a
r
y
l
e
f
bdu tran7.s Add an application resource to the scalable resource group.
a
ang non
# clrs create -g web-rg -t SUNW.apache -p Bin_dir=/global/web/bin \
-p Scalable=TRUE -p Resource_dependencies=clustername-web,web-stor \
apache-res
8.
Bring the scalable resource group online:
# clrg online -M web-rg

9.
Verify that the data service is online.
# clrg status
# clrs status

11-25
Task 5 Verifying Apache Web Server Access
Perform the following steps to verify the Apache web server access and
scalable utility:
1.
Connect to the web server from a browser anywhere else besides a

cluster node using http://clustername-web/cgi-bin/testapache.cgi.
2.
Repeatedly press the Refresh or Reload button on the browser. The

test-apache.cgi script shows the name of the cluster node that is
currently servicing the request. The load balancing is not performed
on a round-robin basis, so you might see several consecutive
requests serviced by the same node. Over time, however, you should
see the load balancing be 50-50.
s
) ha
y
m
om
c
e
Task 6 Observing Cluster Failures ecsm uid
G
t
@
n
b
u at autime,
de and fix each one
k(one
a
Cause as many of the following failures
t
y
( can,isgiven
S your physical access to
n you
before going on to the next one)
as
i
d
h
lu se t
the cluster.
a
am to u
k
Perform the following
ng ensteps
se to observe the behavior of the scalable
a
b
service: a
lic
b
e
u
l
k Fail raasingle
b public network interface on the node where the
a
1.
y
l
e
f
bdu transSharedAddress resource is online.
a
g on- 2. Fail all public network interfaces on that node.
n
a
n
ab
3. Reboot one of the nodes running the scalable service.
4.
11-26
Fail (or bring to the OK prompt) one of the nodes running the
scalable service.


Relationships
In this task, you observe the effects of configuring variations of the
RG_affinities property.
1.
If you have not yet done so, complete Task 8 in Module 10.
2.
Put your myapp-rg resource group offline.

# clrg offline myapp-rg
3.
s
) ha
Create a strong negative affinity with the Apache resource group as

the source and myapp-rg as the target. This means that Apache
categorically refuses to run on any node where myapp-rg is running.
y
m
om
c
# clrg set -p RG_affinities=--myapp-rg

smweb-rg
ide
c
u
e
G is running.
4. Switch the myapp-rg onto some node@
where Apache
t
n
b
Observe what happens to Apache.
dethe console messages as
kuObserve
u
a
t
y
well.
n ( this S
i
d
# clrg switch -nlusomenode
myapp-rg
e
a
s
# clrg status
am to u
k
g myapp-rg
5. Switch
nthe
se to another node where Apache is running.
a
n
b
e
a
Observe
what
lic happens to Apache. Does it come back online on the
b
e
u
l
k first
bnode?
a
a
r
y
ul nsfe# clrg switch -n othernode myapp-rg
d
b
# clrg status
a
tra
g
n
n
a
6. Switch the myapp-rg offline. Can Apache come back online on more
no
ab
nodes? Now remove the relationship.

# clrg status
# clrg set -p RG_affinities=""
7.
web-rg
Set a weak positive affinity so that myapp-rg (source) always prefers

to run on the same node as nfs-rg (target).
# clrg set -p RG_affinities=+nfs-rg myapp-rg
8.
Print out the value of the Nodelist for myapp-rg.

# clrg show -p Nodelist myapp-rg
9.
Switch your nfs-rg so that it is not on the preferred node of myapprg:

# clrg switch -n nonpreferrednode nfs-rg

11-27
10. Bring myapp-rg online without specifying the node. Where does it
end up? Why does it not end up on the first node of its own node
list?
# clrg online myapp-rg

# clrg status
11. Switch the myapp-rg so it is no longer on the same node as nfs-rg.
Can you do it? Is a weak affinity a preference or a requirement?
# clrg switch -n othernode myapp-rg
# clrg status
12. Switch the myapp-rg offline and now change the affinity to a strong
positive affinity with delegation. What is the difference between ++
and +++ (use +++)? The answer lies a few steps further on.
s
) ha
y
m
om
c
13. Bring myapp-rg online without specifying the node.

idedoes it
csm Where
u
e
end up?
G
t
@
n
b
# clrg online myapp-rg
ku tude
a
y
# clrg status
n ( this S
i
d
14. Switch the myapp-rgaso
luit is noselonger on the same node as nfs-rg.
u
Can you do it? am
o
k
t
e
# clrg aswitch
myapp-rg
ng e-n
sothernode
n
b
# clrg
status
c
a
li
b
e
u
l
k
b
15.
if you switch the target group nfs-rg?
a What
ahappens
r
y
l
e
f
switch -n othernode nfs-rg
bdu trans## clrg
a
clrg
status
g onn
a
n
ab
# clrg set -p RG_affinities=+++nfs-rg myapp-rg
16. What happens if RGM wants to failover the source myapp-rg

because its fault monitor indicates application failure?
Kill myappdaemon on whichever node it is running a few times (be

patient with restarts) and observe.
11-28

Exercise Summary
Exercise Summary

Experiences
Interpretations
Conclusions
Applications
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

11-29
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
Module 12
Performing Supplemental Exercises for

Sun Cluster 3.2 Software
s
) ha
Objectives
y
m
om
c
ide
csm
u
Configure a scalable application in non-global
zones
e
G
t
@
n
b
Configure HA-Oracle in a Sun Cluster
software
de environment as a
ku 3.2
u
a
t
y
failover application
n ( this S
i
d
Configure Oracle a
RAC
lu 11gsinea Sun Cluster 3.2 software environment
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
a
no
ab
12-1
Relevance
Relevance

content this module
Why is it more difficult to install Oracle software for a standard

Oracle HA database on the local disks of each node rather than
installing it in the shared storage?
Is there any advantage to installing software on the local disks of

each node?
How is managing Oracle RAC different from managing the other

types of failover and scalable services presented in this course?
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
12-2

s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Performing Supplemental Exercises for Sun Cluster 3.2 Software

12-3
Exercise 1: Running a Clustered Scalable Application in Non-global Zones
Exercise 1: Running a Clustered Scalable Application in

Non-global Zones
This exercise demonstrates running a clustered application in non-global

zones.
Task 1 Configuring and Installing the Zones
Task 2 Booting the Zones
Task 3 Modifying Your Resource Groups So Apache Runs in the

Non-global Zones
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
Preparation
u
ak Stud
y
(
s background information in
din eread
hithe
Before continuing with this lexercise,
t
u
a us
this section.
m
a
k e to
g
ns Clustered Applications in Zones
banRunning
e
Background
c
a
li
b
e
u
l
k
a ofethe
abclustered applications are supported in non-global zones. The
r
y
Many
l
f
ns group manager (RGM) is fully knowledgeable about zones and
bdu trresource
a
a
g on- treats them as if they were virtual nodes as far as configuring resource
n
a
n
groups.
ab
Task 4 Verify That the Apache Application Is Running Correctly in

the Non-global Zones
If you already have an application running across the physical nodes in

the cluster (global zone), it is a relatively simple operation to move it into
a non-global zone.
The same HAStoragePlus resource that is used to control global or
failover storage in global zones can be configured into a resource group
that runs across non-global zones.
HAStoragePlus methods automatically arrange for non-global zones to
gain access to your failover or global file system through simple loopback
mounts.
12-4

Task 1 Configuring and Installing the Zones

Perform the following steps on all nodes. It can take a while to build your
zones, so get them up to where they are running the installation stage
(step 3) simultaneously, so as not to waste time.
1.
Allocate a dedicated public net IP address for the zone. Make sure
you use an IP that is not going to conflict with any other node or
previously used application IP or IPMP test address. Make sure you
use a different IP for the zone on each node. You will identify the
address in step 2. You are not required to put it in any /etc/hosts
file.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
2. Configure the zone.
u
ak Stud
y
(
in this
d
# zonecfg -z myzone
u
e
alzoneuconfigured
s
myzone: No such
m
ka toe begin
to configuring a new zone.
Use 'create'
g
zonecfg:myzone>
an cens create
b
a
set zonepath=/myzone
li
b zonecfg:myzone>
e
u
l
k
set autoboot=true
a ezonecfg:myzone>
ab
r
y
l
zonecfg:myzone>
add net
f
u ns
d
b
zonecfg:myzone:net> set address=x.y.z.w
a
tra
g
n
zonecfg:myzone:net> set physical=node_pubnet_adapter
n
o
a
n
b
zonecfg:myzone:net> end
a
Note Only scalable load-balanced applications require a dedicated nonglobal zone IP address. You could run failover applications in non-global
zones without dedicating public network IP addresses for the zones.
zonecfg:myzone> commit
zonecfg:myzone> exit
3.
Install the zone.
# zoneadm -z myzone install

Preparing to install zone <myzone>.
Creating list of files to copy from the global zone.
.
.
.
The file </myzone/root/var/sadm/system/logs/install_log> contains a log
of the zone installation.

12-5
Task 2 Booting the Zones

On each node, perform the following steps:
1.
Boot the zone.

# zoneadm -z myzone boot
2.
Connect to the zone console and configure the zone. It will look
similar to a standard Solaris OS that is booting after a sysunconfig:
# zlogin -C myzone
[Connected to zone 'myzone' console]
s
) ha
Wait until the SMF services are all loaded, and navigate through the
configuration screens. Get your terminal type correct, or you may
have trouble with the rest of the configuration screens. vt100 usually
works. Do not choose sun.
y
m
om
c
csm Guide
e
t
n
b@
e
u
Note While the zone on each node willkhave
the
same
d zone name
u
a
t
y
(SolarisishostS name in the configuration
(myzone), give each zone a different
n
i
d
h
screens (myzone-nodename),lu
so you cantdistinguish
between them (and so
e
a
s
that the Apache proof ofm
concept CGI
program
will
make
sense).
u
a
o
k
t
g nse
nhave
a
Whenb
you
efinished the system configuration in the zone, it will
c
a
i
l
reboot
You can stay connected to the zone console.
b automatically.
e
u
l
k
b
a /etc/hosts file in the zone (on each node) and add the IP
rthe
y3.a Edit
l
e
f
u
s
bd tran address for clustername-web. Use the same name and IP address
a
g onthat you have in the /etc/hosts file in the global zone.
n
a
n
ab
4. Edit the /etc/default/login file in the zone to allow root logins:
# vi /etc/default/login
[Comment out the CONSOLE=/dev/console line]
# exit
5.
12-6
Disconnect from the zone console using ~.

Task 3 Modifying Your Resource Groups So Apache

Runs in the Non-global Zones
Perform the following in the global zone on the nodes indicated:
1.
From any one node, offline both the scalable and shared address
resource groups:
# clrg offline web-rg sa-rg
2.
On all nodes (still in the global zone), manually do a loopback mount

of the /global/web filesystem into the zone. This will allow the
apache resource to validate correctly when the node list of its group
is changed to the zones. Once the resource group is running in the
zones, the HAStoragePlus methods will automatically be doing the
mounts.
s
) ha
ab
y
m
om
c
# mkdir -p /myzone/root/global/web
# mount -F lofs /global/web /myzone/root/global/web

csm Guide
e
@ ent
3. From any one node, modify the node
lists:
b
u
ak Stud sa-rg
# clrg set -n node1:myzone,node2:myzone[,node3:myzone]
y
(
in this
# clrg set -n node1:myzone,node2:myzone[,node3:myzone]
web-rg
d
u
l
e
a us and then remanage all the groups.
4. From any one
node, unmanage
m
a
k toeproperly
to reconfigure the shared address in all the
This is required
g
zones,
banandlicstart
ensthe load-balanced service:
a
b # clrs
u
le disable -g sa-rg,web-rg +
k
b
a
a
l y sfe#r clrg unmanage sa-rg web-rg
u
d
b tran # clrg online -Me sa-rg web-rg
a
ang non
Note You may see console messages about the scalable service being
registered with the physical node name, although the application will
really be running in the zone. This is normal.

12-7
Task 4 Verify That the Apache Application Is Running

Correctly in the Non-global Zones
Perform the following steps to verify the correct operation of Apache, and
to demonstrate that the load balancing is working when the scalable
application is running across non-global zones.
1.
On any node, from the global zone, verify the cluster application
status:
# clrg status
# clrs status
2.
On each node, log in to the non-global zone and verify that the
Apache application is running within the zone:
s
) ha
y
m
om
c
(in the non-global zone)

sm uide
c
e
G
# ps -ef|grep apache2
t
@
n
b
# exit
ku tude
a
y
S point a Web browser
3. On the administration workstation
n ( thorisdisplay,
i
d
to http://clustername-web/cgi-bin/test-apache.cgi.
Press
lu se
a
m totimes
u to demonstrate that load balancing is
reload or refreshaseveral
k
g thenszones.
working across
e It is not round-robin load balancing;
nyou
a
b
e
therefore,
might
get
a response from the same zone several times
c
a
i
l
b
uin a row.ble
k
a
y fera
l
u
bd trans
a
g onn
a
n
ab
# zlogin myzone
12-8

Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2 Software as a Failover
Exercise 2: Integrating Oracle 11g Into Sun Cluster 3.2

Software as a Failover Application
In this exercise you integrate Oracle 11g into Sun Cluster 3.2 software as a
failover application.
Task 1 Creating a Logical IP Entry in the /etc/hosts File
Task 2 Creating oracle and dba Accounts
Task 3A Creating a Shared Storage File System for Oracle Software

(VxVM)
s
) ha
y
m
om
c
e
m Oracle
idSoftware
Task 3C: Create a Shared Storage File System
csfor
u
e
G
(ZFS)
t
@
n
b
de
ku Environment
u
a
Task 4 Preparing the oracle
User
t
y
( is S
nControl
i
d
Task 5 Disabling Access
th of X Server on the Admin
u
l
e
a
Workstation m
us
a
o
k
t
Task 6ng
Running
sethe runInstaller Installation Script
a
n
b
e
a 7 Preparing
Task
an Oracle Instance for Cluster Integration
lic
b
e
u
l
k
a eTask
ab8 Registering the SUNW.HAStoragePlus Type
r
y
l
f
bdu trans Task 9 Installing and Registering Oracle Data Service
a
g onn
Task 10 Creating Resources and Resource Groups for Oracle
a
n
ab
Task 3B Creating a Shared Storage File System for Oracle Software

(SVM)
Task 11 Verifying That Oracle Runs Properly in the Cluster
Preparation
Before continuing with this exercise, read the background information in
this section.

12-9
Background
It is relatively straightforward to configure the Oracle 11g Database

software as a failover application in the Sun Cluster 3.2 environment. Like
the majority of failover applications, the Oracle software is clusterunaware. You will be installing and configuring the software exactly as
you would on a stand-alone system and then manipulating the
configuration so that it listens on a failover logical IP address.
Installing an Application on Local Storage (Each Node) or Shared

Storage
s
) ha
For ease of installation and management, this lab takes the strategy of
installing the Oracle binaries themselves directly in shared storage.
Therefore, you only need to perform the Oracle installation once. The
disadvantage is that there would be no way to perform maintenance or
patching on the software while keeping your application available on
another node.
y
m
om
c
csm Guide
e
@ ent
b
u
d
tucopies
yakseparate
(
S
In production, you might want toninstall
of the software
s
i
i
d
h
locally on each node (with the
course,
being in the shared
t
lu data,toseofinstall,
a
storage). While it is more
difficult
you
get the advantage of
m to u
arolling
k
being able to perform
maintenance
(including
patching) of the
g
e
n
s
a node
software onbone
while
n keeping your database alive on the other
e
c
a
i
l
node. b
u
le
k
b
a
a
l y sfer
u
d
n of File System and Storage
b trChoice
a
a
g onn
a
n
A failover file system is optimal for performance (although actual failover
ab
time is longer, the application runs faster after it is up and running).
However, you cannot use a failover file system if you want your
application to fail over to a nonstorage node.
This exercise uses a failover file system. You will not be able to use a nonstorage node (just pretend it does not exist), but it is unlikely anyone
would ever use a global file system or a non-storage node for HA-Oracle.
In this exercise there are options to use VxVM, SVM, or ZFS. Choose only
one of tasks, 3A, 3B, or 3C depending on your choice of storage
management.
12-10

Task 1 Creating a Logical IP Entry in the /etc/hosts

File
Create an entry for a logical IP address for Oracle in the /etc/hosts
file on all storage nodes as follows:
# vi /etc/hosts
...
x.y.z.w
ora-lh
Task 2 Creating oracle and dba Accounts
s
) ha
y
m
To create the oracle user and dba group accounts, perform

omthe following
c
steps on all storage nodes:

csm Guide
e
1. Create the dba group account:
@ ent
b
u
# groupadd -g 8888 dba yak
tud
(
S
n arethspecifying,
iYou
is
2. Create the oracle user.
but not creating, the
d
u
l
e
a ituwill
home directorym
because
be
located
in
shared
storage that is not
s
a
o
yet created.k
t
g
e
n
s
n 8888 -d /oracle oracle
# useradd -s /bin/ksh
ba-g dba
e-u
c
a
i
l
b Createlethe oracle user password:
u
3.
k
a erab
y
l
u nsf # passwd oracle
d
b
New password: oracle
a
tra
g
n
Re-enter new Password: oracle
n
a
no
ab

12-11
Task 3A Creating a Shared Storage File System for

Oracle Software (VxVM)
Perform the following steps on any node physically connected to the
storage. Only step 6 is performed on all storage nodes.
Note Just to speed up the lab, we will create an unmirrored volume. You
can always add another disk and mirror your volume later at your leisure,
or pretend that you were going to.
1.
Select a disk from shared storage. Make sure that you do not use any
disks already in use in existing device groups. Note the logical
device name (referred to as c#d#t# in step 2). The following
example checks against VxVM, Solaris Volume Manager, and ZFS
disks.
s
) ha
y
m
om
c
sm uide
c
e
# vxdisk -o alldgs list
G
t
@
n
b
# zpool status
ku tude
a
# metaset
y
n ( this S
i
# cldev list -v
d
lu thissdisk
e
a
2. Create a disk group
using
m
u
ka e to -i c#t#d#
g
# /etc/vx/bin/vxdisksetup
n oradg
ns ora1=c#t#d#
bainit
e
# vxdg
c
a
li
b a volume
e
u
l
3.
Create
to hold the Oracle binaries and data.
k
b
a
a
r
y
ul ns#fevxassist -g oradg make oravol 6g
d
b
a
tr4.a Register the new disk group (and its volume) with the cluster. The
g
n
n
no
nodelist property contains all nodes physically connected to the
aba
storage.
# cldg create -t vxvm -n node1,node2 oradg

5.
Create a UFS file system on the volume:

# newfs /dev/vx/rdsk/oradg/oravol
6.
Create a mount point and an entry in /etc/vfstab on all storage

nodes:
# mkdir /oracle
# vi /etc/vfstab
/dev/vx/dsk/oradg/oravol /dev/vx/rdsk/oradg/oravol /oracle ufs 2 no -
7.
On the node on which you created the storage group, mount the file
system and change its ownership:
# mount /oracle
# chown oracle:dba /oracle
12-12

Task 3B Creating a Shared Storage File System for

Oracle Software (SVM)
Perform the following steps on any node physically connected to the
storage. Only step 5 is performed on all storage nodes. This lab assumes
that local metadbs have been created already.
Note Just to speed up the lab, we will create unmirrored volumes. You
can always add another disk and mirror the parent volume later at your
leisure, or pretend that you were going to.
1.
s
) ha
Select a disk from shared storage.. Make sure that you do not use any
disks already in use in existing device groups. Note the DID device
name (referred to as d# in step 2). The following example checks
against VxVM, Solaris Volume Manager, and ZFS disks.
y
m
om
c
sm uide
c
e
G
t
@
n
b
# zpool status
ku tude
a
# metaset
y
( is S
# cldev list -v din
u se th
l
a
2. Create a diskset
u disk.
amusingtothis
k
# metaset
orads
se -a -h node1 node2
ang ce-s
#ab
metaset
-snorads -a /dev/did/rdsk/d#
li
b Createlea soft
u
3.
partition (d100) of a mirror volume for Oracle. (SVM
k
b
a
a
r
y
ul nsfestill calls it a mirror, although we only have one submirror. This
d
b
gives us flexibility to add the other half of the mirror at our leisure
a
a
r
t
g
later.)
n non
a
b
a
# metainit -s orads d11 1 1 /dev/did/rdsk/d#s0
# metainit -s orads d10 -m d11
# metainit -s orads d100 -p d10 6g
4.
Create a file system:

# newfs /dev/md/orads/rdsk/d100
5.
Create a mount point and an entry in /etc/vfstab on all storage

nodes:
# mkdir /oracle
# vi /etc/vfstab
/dev/md/orads/dsk/d100 /dev/md/orads/rdsk/d100 /oracle ufs 2 no -
6.
On the node on which you created the storage, mount the file system
and modify its permission:
# mount /oracle
# chown oracle:dba /oracle

12-13
Task 3C: Create a Shared Storage File System for

Oracle Software (ZFS)
Perform the following steps on any one node physically connected to the
storage.
Note Just to simplify, we will create a ZFS storage pool with one disk.
You can always add another disk and mirror the existing disk later.
1.
Select a disk from shared storage.. Make sure that you do not use any
disks already in use in existing device groups. Note the device name
(referred to as c#t#d# in step 2). The following example checks
against VxVM, Solaris Volume Manager, and ZFS disks.
s
) ha
y
m

om
c
# zpool status
csm Guide
e
# metaset
@ ent
b
# cldev list -v
u
aksystem
tuford Oracle. Verify that
y
(
S
2. Create a ZFS storage pool and
a
file
inand change
d
his its permissions:
your file system is mounted
t
u
l
e
a uc#t#d#
s
m
# zpool create
orapool
a
o
k
t
# zfs create
orapool/orafs
e
ng e-onsmountpoint=/oracle
a
b
# df
-h
a oracle:dba
lic
#bchown le
/oracle
u
k
b
a
a
l y sfer
u
d
a4n Preparing the oracle User Environment
abTask
r
t
g
n non
a
b
a
To configure environment variables required for the Oracle software, run

the following commands from the cluster node which owns the Oracle
storage:
1.
Switch to the oracle user:

# su - oracle
2.
Edit the .profile file as shown. Be sure to substitute the proper

value for your DISPLAY variable.
$ vi .profile
ORACLE_BASE=/oracle
ORACLE_HOME=$ORACLE_BASE/product/11.1.0/db_1
ORACLE_SID=MYORA
PATH=$PATH:$ORACLE_HOME/bin
DISPLAY=display-name-or-IP:#
export ORACLE_BASE ORACLE_HOME ORACLE_SID PATH DISPLAY
12-14

$ exit
Task 5 Disabling Access Control of X Server on the

Admin Workstation
To allow client GUIs to be displayed, run the following command on
the admin workstation:
(# or $) /usr/openwin/bin/xhost +
s
) ha
Task 6 Running the runInstaller Installation Script

y
m
m
co ethe
perform
To install the Oracle software on the local file system,

m
d
s
i
u
following steps on the node that owns the Oracle
ec storage:
G
t
@
b den
u
1. Switch to the oracle user:
k
ya Stu
(
# su - oracle
n
i
is
d
h
t
u
l
e
2. Change directoryato the location
s of the Oracle software:
m
u
a
o
k et
$ cd ORACLE-11gR1-db-software-location
g
n
ns
3. Run
script:
bathe runInstaller
e
c
a
i
l
b
u $ ./runInstaller
le
k
b
a
a
r
l y 4.sfeRespond
to the dialogs by using Table 12-1.
u
d
b tran
a
g 12-1
anTable
non The runInstaller Script Dialog Answers
ab
Dialog
Action
Select Installation Method
Click Advanced Installation (near the bottom).

[The basic installation makes some wrong choices
for Sun Cluster.]
Click Next.
Click OK when you get the error about the
previous inventory location (there obviously was
none).

12-15
Table 12-1 The runInstaller Script Dialog Answers (Continued)

Dialog
Action
Specify Inventory Directory and

Credentials
Change the full path of the inventory directory to:

/oracle/oraInventory .
Verify the group name (dba) and click Next.
Select Installation Type
Select the Custom radio button, and click Next.
Install Location
Verify and click Next.

[The path is taken from the $ORACLE_HOME that
you set in your profile, so if it looks wrong, you
need to quit and start again.]
s
) ha
y
m
othemOracle
A popup will warn you against using
c
e
m
d
s
Base as the home directory. Click
Yes.
i
ec t Gu
@
n
Product Specific Prerequisite
The tests will either u
allbsucceed...or..
e
d
k
Checks
you will be warned
swap space
tu
ya about
(
S
n
requirements.
di e this
u
l
a us
m
Click
Next,
and if you get a pop-up warning
a
k window,
toclick
g
e
Yes to proceed.
n ens
a
b
a
lic Deselect the following (so our install goes faster):
Available Product Components
b
e
u
l
k rab
a
Enterprise Edition Options
y
l
e
f
u
s
bd tran
Oracle Programmer 11.1.0.6.0
a
g
n
n
a
no
ab
Click Next.
Privileged Operating System

Groups
Verify that they all say dba, and click Next.
Create Database
Verify that the Create a Database radio button is

selected, and click Next.
Summary
Verify, and click install.
Oracle Net Configuration Assistant

Welcome
Verify that the Perform Typical Configuration

check box is not selected, and click Next.
Listener Configuration, Listener

Name
Verify that the Listener name is LISTENER, and

click Next.
Select Protocols
Verify that TCP is among the Selected Protocols,

and click Next.
12-16


Dialog
Action
TCP/IP Protocol Configuration
Verify that the Use the Standard Port Number of

1521 radio button is selected, and click Next.
More Listeners
Verify that the No radio button is selected, and

click Next.
Listener Configuration Done
Click Next.
Naming Methods Configuration
Verify that the No, I Do Not Want to Configure

Additional Naming Methods radio button is
selected, and click Next.
s
) ha
y
m
om
DBCA Step 1: Database Templates
Select the General Purpose or Transaction
c
Processing radio button, and

Next.
ide
csmclick
u
e
G Name text
t
@
Step 2: Database Identification
Type MYORA in the b
Global
Database
n
de in the SID text
ku it istechoed
field, observeathat
u
y
field, and
n (clickthNext.
sS
i
i
d
lu sConfigure
e
a
Step 3: Management Options
Uncheck
Enterprise Manager and
m
u
a
o
k click
t Next.
g
e
n
s
Step 4: Database Credentials
ba licen Select the Use the Same Administrative Password
a
b
for All Accounts radio button.
u
le
k
b
a
a
l y sfer
u
Enter cangetin as the password, and click Next.
d
b tran
a
gStep 5:oStorage
n- Options
Verify that File System is selected, and click Next.
n
a
n
b
a
Done
Click Finish.
Step 6: Database File Locations
Select or verify the Use Database File Locations

from Template radio button and click Next.
Step 7: Recovery Configuration
Uncheck all the boxes, and click Next.
Step 8: Database Content
Uncheck Sample Schemas, and click Next.
Step 9: Initialization Parameters
On the Memory tab, select the Typical radio

button.
Make the memory size as small as they will let
you (you can tell using the slider next to Memory
Size box). Click Next.
Step 10: Security Settings
Leave the radio button at Keep the enhanced....

and click Next.

12-17

Dialog
Action
Step 11: Automatic Maintenance

Tasks
Uncheck the checkbox (were not hanging around

too long after these labs) and click Next.
Step 12: Database Storage
Click Next.
Step 13: Creation Options
Verify that Create Database is checked, and click

Finish.
Confirmation
Verify summary of database creation options,

parameters, character sets, and data files, and
click OK.
s
) ha
y
m
Note This might be a good time to takem

o a break.
c
The database configuration assistant

takes e
15 to
m
d
s
i
20 minutes to complete. ec
u
G
t
@
b den
u
k
Database Configuration Assistant
Click Exit.
ya Stu
(
n
i
Script (setup privileges)
Open a d
terminal
window
his as user root on the
t
u
l
e
installer
a node
s and run:
m
u
a
o
k et
g
/oracle/oraInventory/orainstRoot.sh
n
a
ns
b
e
c
a
li Now run
b
e
u
l
k
a erab
/oracle/product/11.1.0/db_1/root.sh
y
l
f
u
s
bd tran
a
Accept the default path name for the local bin
ang non
directory.
ab
Click OK in the script prompt window.

End of Installation
12-18
Click Exit and confirm.

Task 7 Preparing an Oracle Instance for Cluster

Integration
On the same node from which the Oracle software was installed, perform
the following steps as indicated to prepare the Oracle instance for Sun
Cluster 3.x software integration:
1.
Configure an Oracle user for the fault monitor (as user oracle):
$ sqlplus /nolog
SQL> connect / as sysdba
SQL> create user sc_fm identified by sc_fm;
SQL> grant create session, create table to sc_fm;
SQL> grant select on v_$sysstat to sc_fm;
SQL> alter user sc_fm default tablespace users quota 1m on users;
SQL> quit
s
) ha
ab
y
m
om
c
e (as user
2. Test the new Oracle user account used by thesm
c faultGmonitor
uid
e
oracle):
@ ent
b
u
$ sqlplus sc_fm/sc_fm
ak Stud
y
SQL> select * from sys.v_$sysstat;
(
in this
SQL> quit
d
u
e oracle). You can be creative about
al (asuuser
s
m
3. Create a sample
table
ka e to
your data
values:
g
an cens
$ sqlplus /nolog ab
li
b sysdba
e
l
SQL> connect k
/ uas
atableemytable
ab (Name VARCHAR(20), age NUMBER(4));
r
y
SQL> create
l
f
s mytable values ('vincent', 14);
du rainto
SQL>binsert
n
a
-t into mytable values ('theo', 14);
g insert
n
o
anSQL>
n
SQL> commit;
SQL> select * from mytable;
SQL> quit
Note This sample table is created only for a quick verification that the
database is running properly.
4.
Shut down the Oracle instance:

$ sqlplus /nolog
SQL> connect / as sysdba
SQL> shutdown
SQL> quit
5.
Stop the Oracle listener:

$ lsnrctl stop

12-19
6.
Configure the Oracle listener (as user oracle):

$ vi $ORACLE_HOME/network/admin/listener.ora
(a) Modify the HOST variable to match logical host name ora-lh.
(b) Add the following lines at the top:
SID_LIST_LISTENER =
(SID_LIST =
(SID_DESC =
(SID_NAME = MYORA)
(ORACLE_HOME = /oracle/product/11.1.0/db_1)
(GLOBALDBNAME = MYORA)
)
)
s
) ha
y
m
$ vi $ORACLE_HOME/network/admin/tnsnames.ora om
c e
m
Modify the HOST variable to match logical host
name
s
c Gora-lh
uid
e
@ ent
b
u
ak Stud
y
Task 8 Registering the SUNW.HAStoragePlus
Type
(
n this
i
d
lu se
a
m to u
Perform the followingatask:
k
ng(if not
se in a previous exercise), register the
If necessary
done
a
n
b
e
a
SUNW.HAStoragePlus
type from any one node in the cluster:
lic
b
e
u
l
k
a # eclrt
abregister SUNW.HAStoragePlus
r
y
l
f
bdu trans
a
on- 9 Installing and Registering Oracle Data
ang nTask
7.
ab
Configure the tnsnames.ora file (as user oracle):
Service
Perform the following steps as indicated to install and register the Oracle
data service:
1.
Install the data service agent on all storage nodes:

# pkgadd -d . SUNWscor
2.
Register the SUNW.oracle_server and SUNW.oracle_listener

resource types from one cluster node:
# clrt register SUNW.oracle_server
# clrt register SUNW.oracle_listener
12-20

Task 10 Creating Resources and Resource Groups

for Oracle
Perform the following steps from the node which owns the Oracle storage
to create the resource group and resources necessary for the Oracle data
service:
1.
Create an empty resource group:

# clrg create -n node1,node2 ora-rg
2.
Add a LogicalHostname resource:

# clrslh create -g ora-rg ora-lh
3.
Add an HAStoragePlus resource.

For VxVM or SVM:
s
) ha
ab
y
m
# clrs create -g ora-rg -t HAStoragePlus c\om
e
-p FilesystemMountPoints=/oracle \ sm
d
i
ec t Gu
-p AffinityOn=true ora-stor
@
b den
u
k
For ZFS:
ya Stu
(
n
i -tthHAStoragePlus
# clrs create -g ora-rg
\
is
dora-stor
u
l
-p Zpools=orapool
e
a us
m
a
k e to
g
4. Putbthe
anresource
nsgroup online (required by the dependency
e
c
a
li in the next step):
b specification
e
u
l
k
b online -M ora-rg
a e#raclrg
y
l
f
bdu tran5.s Create an oracle_server resource by creating and running a script.
a
(the command is too long to type interactively in some shells):
ang non
a.
Write a script containing the clrs command:
# vi /var/tmp/setup-ora-res.sh
#!/bin/sh
/usr/cluster/bin/clrs create -g ora-rg \
-t oracle_server \
-p Resource_dependencies=ora-stor \
-p ORACLE_sid=MYORA -p ORACLE_home=/oracle/product/11.1.0/db_1 \
-p Alert_log_file=/oracle/diag/rdbms/myora/MYORA/trace/alert_MYORA.log \
-p Connect_string=sc_fm/sc_fm \
ora-server-res
b.
Run the script:
# sh /var/tmp/setup-ora-res.sh

12-21
6.
Create an oracle_listener resource:
# clrs create -g ora-rg -t oracle_listener \

-p ORACLE_home=/oracle/product/11.1.0/db_1 \
-p Listener_name=LISTENER -p Resource_dependencies=ora-stor \
ora-listener-res
7.
Verify the application:

# clrg status
# clrs status
# ps -ef | grep oracle
Task 11 Verifying That Oracle Runs Properly in the

a
s
a
Cluster
y) h
m
m
co software
Perform the following steps as indicated to verify that the Oracle
e
m
d
s
i
properly runs in the cluster:
ec t Gu
@
n running the
b nodedcurrently
1. Switch user to oracle user on the cluster
e
u
k
ora-rg resource group:
ya Stu
(
n
di e this
# su - oracle
u
l
a theucluster
s node currently running the oram
2. Query the database
from o
a
k
t
rg resource
nggroup:
se
a
n
b
e
a
$ sqlplus
SYS/cangetin@MYORA
as sysdba
lic
b
e
u
l
* from mytable;
k SQL>raselect
b
a
y
SQL>
quit
ul nsfe
d
b
a
tr3.a As user root (from any cluster node), switch the resource group to
g
n
n
another node:
a
no
ab
# clrg switch -n other-node ora-rg

4.
Query the database from the cluster node currently running the orarg resource group (as user oracle):
$ sqlplus SYS/cangetin@MYORA as sysdba
SQL> quit
5.
12-22
Verify that the database fails over properly under various failure
conditions:
a.
Complete failure of the node on which the resource group is

running (bring it to the OK prompt).
b.
Complete public network failure on the node on which the

resource group is running.

Exercise 3: Running Oracle 11g RAC in Sun Cluster 3.2 Software
Exercise 3: Running Oracle 11g RAC in Sun Cluster 3.2

Software
In this exercise, you run Oracle 11g RAC in Sun Cluster 3.2 software.
Task 1 Selecting the Nodes That Will Run Oracle RAC
Task 1.5 Unconfigure any Nonstorage Node From the Cluster
Task 2 Creating User and Group Accounts
Task 3A (If Using VxVM) Installing RAC Framework Packages for

Oracle RAC With VxVM Cluster Volume Manager
s
) ha
y
m
om
c
csm Guide
e
t
@
n
b
Task 4 Installing Oracle Distributed
Lock Manager
e
u
k tud
athe
y
Task 5 Creating and Enabling
(
S Framework Resource Group
n thisRAC
i
d
lu sCreating
Task 6A (If using a
VxVM):
Shared Volume Manager
e
m
u
Framework Resource
Group
ka e to
g
n Using
Taska6B(If
nsSolaris Volume Manager) Creating Shared Volume
b
e
c
a
i
Resource Group
lFramework
b Manager
e
u
l
k
a eTask
ab7A Creating Raw Volumes (VxVM)
r
y
l
f
bdu trans Task 7B Creating Raw Volumes (Solaris Volume Manager)
a
g onn
Task 8 Configuring Oracle Virtual IPs
a
n
ab
Task 3B (If Using Solaris Volume Manager) Installing RAC

Framework Packages for Oracle RAC With Solaris Volume Manager
Multi-Owner Disksets
Task 9 Configuring the oracle User Environment
Task 10A Creating the dbca_raw_config File (VxVM)
Task 10B Creating the dbca_raw_config File (Solaris Volume

Manager)
Task 11 Disabling Access Control on X Server of the Admin

Workstation
Task 12 Installing Oracle CRS Software
Task 13 Installing Oracle Database Software
Task 14 Create Sun Cluster Resources to Control Oracle RAC

Through CRS
Task 15 Verifying That Oracle RAC Works Properly in a Cluster

12-23
Preparation
Before continuing with this exercise, read the background information in

this section.
Background
Oracle 11g RAC software in the Sun Cluster environment encompasses
several layers of software, as follows:
RAC Framework
s
) ha
This layer sits just above the Sun Cluster framework. It encompasses
the UNIX distributed lock manager (udlm) and a RAC-specific
cluster membership monitor (cmm). In the Solaris 10 OS, you must
create a resource group rac-framework-rg to control this layer (in
the Solaris 9 OS, it is optional).
y
m
om
c
csm Guide
e
Shared Volume Manager framework
@ ent
b
u
k thetRAC
ud framework starting
afrom
This framework has been split off
y
(
S
in Sun Cluster 3.2 11/09 (Update
in 3).
d
hItisis required specifically and
t
u
l
only for shared volume
manager
support.
e
a us
m
a
kReadyeServices
to (CRS)
Oracle Cluster
g
s
an cenOracles
CRSaisbessentially
own implementation of a resource group
i
l
b
umanager.
leThat is, for Oracle RAC database instances and their
k
b
a
a
and related resources, CRS takes the place of the
l y sassociated
erClusterlisteners
f
u
d
Sun
resource
group manager.
b tran
a
g onOracle Database
n
a
n
b
a
The actual Oracle RAC database instances run on top of CRS. The
database software must be installed separately (it is a different
product) after CRS is installed and enabled. The database product
has hooks that recognize that it is being installed in a CRS
environment.
Sun Cluster control of RAC

The Sun Cluster 3.2 environment features new proxy resource types
that allow you to monitor and control Oracle RAC database
instances using standard Sun Cluster commands. The Sun Cluster
instances issue commands to CRS to achieve the underlying control
of the database.
12-24


The various RAC layers are illustrated in Figure 12-1.
Oracle RAC DB
Sun Cluster RAC proxy resource lets

you use Sun Cluster commands to control
database by "proxying" through CRS
Oracle CRS
Shared VM
Framework
RAC
Framework
s
) ha
y
m
m
o
c
RAC Database Storage

e
m
d
s
i
ec t Gu
@
In the Sun Cluster environment, you have
n choices of where
b the following
e
u
d
k
to store your actual data for RACydatabases:
a Stu
(
n
i
Raw devices using the
Cluster
dVxVM
his Volume Manager (CVM)
t
u
l
e
a us
feature
m
a
k usingethe
toSolaris Volume Manager multi-owner diskset
g
Raw devices
ban licens
feature
a
b Raw devices
u
le using Oracles Automatic Storage Managment (ASM)
k
b
a
a
r
l y sfeRaw
u
devices using no volume manager (assumes hardware RAID)
d
b tran
a
g onShared QFS file system on raw devices or Solaris Volume Manager
n
a
n
b
multi-owner devices
a
Figure 12-1 Oracle RAC Software Layers
On a supported NAS device
Note Use of global devices (using normal device groups) or a global file
system is not supported. The rationale is that if you used global devices or
a global file system, your cluster transport would now be used for both
the application-specific RAC traffic and for the underlying device traffic.
The performance detriment this might cause might eliminate the
advantages of using RAC in the first place.

12-25

In this lab, you can choose to use either VxVM or Solaris Volume
Manager. The variations in the lab are indicated by tasks that have an A or
B suffix. You will always choose only one such task. When you have built
your storage devices, there are very few variations in the tasks that
actually install and test the Oracle software.
This lab does not actually address careful planning of your raw device
storage. In the lab, you create all the volumes using the same pair of disks.
Task 1 Selecting the Nodes That Will Run Oracle

RAC
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
Note All the cluster command examples
d the rest of the RAC
k shown
anodes.
tufor
y
(
S
exercise assume you have two storage
You
can make the
n
s
i
i
d
h
t than two storage nodes on
appropriate substitutions if lyou
more
u have
e
a
s
which you choose to install
RAC.
There
m to u are some storage volumes that are
a
k
created on a per-node
g basis,
e this will be noted further in the sections
n
s
a
n
where you b
ce storage volumes.
a create lyour
i
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
Select the nodes on which the Oracle 11g RAC software will be installed
and configured. The nodes you select must all be attached to shared
storage.
ab
12-26

Task 1.5 Unconfigure any Nonstorage Node From

the Cluster
VxVMs CVM feature, in particular, is very nasty and kernel-panicky if
you still have nonstorage nodes in the cluster, even if you do not intend to
use them for Oracle RAC. Perform the following steps on the indicated
nodes if you have a nonstorage node:
1.
Reboot the nonstorage node to noncluster mode.
2.
On the nonstorage node, edit /etc/vfstab and comment out lines

containing a global filesystem except the line for
/global/.devices/node@# (you can leave that one).
as
h
3. On one of the nodes remaining in the cluster, type the following
to
)
y
m
allow the nonstorage node to remove itself:
om
c
# claccess allow -h name_of_non_storage_node
# clrg remove-node -n name_of_non_storage_node

csm Guide+
e
t lab in this
@ theenzones
b
If for some reason you already completed
u
d will need to clean up
akthisSlab,
module, and are also attempting
tuyou
y
(
the node lists for the apache
in application
d
his like this:
t
u
l
e
a -nus\
# clrg remove-node
m
a
k e to
name_of_non_storage_node:myzone
sa-rg web-rg
g
n
s
a nonstorage
4. On
bthe
en node, type the following to remove itself from the
c
a
i
l
b cluster:le
u
k
b
a e#raclnode
y
remove -n name_of_any_other_node_remaining
l
f
u
s
d
ab -tran
g
an non
ab

12-27
Task 2 Creating User and Group Accounts
Perform the following steps on all selected cluster nodes:

Note If you have already completed the HA-Oracle lab, then delete the
oracle user and dba groups from all nodes before proceeding.
1.
Create the dba group account:

# groupadd -g 7777 dba
2.
Create the oracle user account:
# useradd -g dba -u 7777 -d /oracle -m -s /bin/ksh oracle
s
) ha
y
m
om
# passwd oracle
c
New Password: oracle

sm uide
c
e
Re-enter new Password: oracle
G
t
@
n
b
ku tude
a
y
n ( this SRAC Framework
Task 3A (If Using VxVM)udiInstalling
e VxVM Cluster Volume
al With
s
Packages for Oracle
RAC
m
u
ka e to
g
Manager
ban licens
a
b the following
u
le
Perform
steps on all selected cluster nodes
k
b
a
a
r
y
fe the appropriate packages from the software location:
ul 1. nsInstall
d
b
a
tra
g
#n
cd
sc32_location/Solaris_sparc/Product
n
no
ab#a cd sun_cluster_agents/Solaris_10/Packages
3.
Provide a password for the oracle user account:
# pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWcvm SUNWcvmr SUNWscor

2.
Enter a license for the CVM feature of VxVM. Your instructor will
tell you from where you can paste a CVM license.
# vxlicinst
3.
Verify that you now have a second license for the CVM_FULL feature:
# vxlicrep
4.
Refresh the VxVM licenses that are in memory:

# vxdctl license init
12-28

Task 3B (If Using Solaris Volume Manager) Installing

RAC Framework Packages for Oracle RAC With
Solaris Volume Manager Multi-Owner Disksets
Perform the following steps on all selected cluster nodes. For Solaris
Volume Manager installations, no reboot is required.
1.
Install the appropriate packages from the software location:
# pkgadd -d . SUNWscucm SUNWudlm SUNWudlmr SUNWscor
s
) ha
y
m
# metadb
om
c
e
3. Add metadbs on the root drive, only if they s
idexist:
c domnotGyet
u
e
# metadb -a -f -c 3 c#t#d#s7
@ ent
b
u
ak Stud
y
(
inDistributed
d
Task 4 Installing Oracle
Lock Manager
his
t
u
l
e
a
am to us
k
Perform thenfollowing
g nssteps
e on all selected cluster nodes as user root.
a
b
e
a thelicORCLudlm package:
1. b Install
e
u
lORACLE_clusterware_location/udlm
k
b
a
#
cd
a
l y sfe#r cp ORCLudlm.tar.Z /var/tmp
u
d
ab -tran # cd /var/tmp
g
n non
# zcat ORCLudlm.tar.Z|tar xf a
b
a
# pkgadd -d . ORCLudlm
2.
List the local metadbs:
2.
The pkgadd command prompts you for the group that is to be DBA
for the database. Respond using dba:
Please enter the group which should be able to act as
the DBA of the database (dba): [?] dba

12-29
Task 5 Creating and Enabling the RAC Framework

Resource Group
On any one of the selected nodes, run the following commands as root:
# clrt register rac_framework
# clrt register rac_udlm
# clrg create -S -n node1,node2 rac-framework-rg
# clrs create -g rac-framework-rg -t rac_framework rac-framework-res
# clrs create -g rac-framework-rg -t rac_udlm \
-p Resource_dependencies=rac-framework-res rac-udlm-res
s
) ha
y
m
m
If all goes well, you will see a message on the console: co
e
m
d
s
i
u
Unix DLM version(2) and SUN Unix DLM Library Version
ec (1):compatible
G
t
@
b den
u
k
ya Stu
(
n
Task 6A (If using VxVM): Creating
di e thisShared Volume
u
l
a us Group
Manager FrameworkmResource
a
k e to
g
n ens nodes, run the following commands as root:
On any oneb
ofathe selected
a
lic
b
e
u
l
# clrt registerak
vucmm_framework
ab
r
y
l
e
# clrt register
vucmm_cvm
f
s
du r-S
n
# clrgab
create
-n node1,node2 shvm-framework-rg
a
g on-t
n
a
n
-g shvm-framework-rg -t vucmm_framework vucmm-framework-res
ab# clrs create
# clrg online -M rac-framework-rg
# clrs create -g shvm-framework-rg -t vucmm_cvm\

-p Resource_dependencies=vucmm-framework-res vucmm-cvm-res
# clrg online -M shvm-framework-rg
Following that there will be about 100 lines of console output as CVM
initializes. If your nodes are still alive (not kernel panic), you are in good
shape.
12-30

Task 6B(If Using Solaris Volume Manager) Creating

Shared Volume Manager Framework Resource Group
On any one of the selected nodes, run the following commands as root to
create the RAC framework resource group. As you create the group, use
only those nodes on which you have selected to run RAC (the ones
connected to the physical storage):
# clrt register vucmm_framework
# clrt register vucmm_svm
# clrg create -S -n node1,node2 shvm-framework-rg
as
h
# clrs create -g shvm-framework-rg -t vucmm_framework vucmm-framework-res
)
y
m
om
# clrs create -g shvm-framework-rg -t vucmm_svm \
c
-p Resource_dependencies=vucmm-framework-res vucmm-svm-res
csm Guide
e
@ ent
b
# clrg online -M shvm-framework-rg
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab

12-31
Task 7A Creating Raw Volumes (VxVM)
To create the raw volumes for the Oracle 11g RAC database, perform the
following steps from the single node that is the CVM Master.
1.
Determine which node is the CVM Master by running the following

command on all nodes:
# vxdctl -c mode
2.
Select a single disk from shared storage for a new disk group. Make
sure you do not use any disks already in use in existing device
groups. The following example checks against VxVM, ZFS and
Solaris Volume Manager disks, just in case you are running any
combination.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
n willthcreate
Note Just to speed up the lab,
iwe
is unmirrored volumes. You
d
u
l
can always add another disk
and
mirror
all
volumes later at your leisure,
e
a us
m
a going
or pretend that youkwere
to to.
g
e
n ens
baaCVM
3. Create
a
licshared disk group using this disk:
b
e
u
l
k # /etc/vx/bin/vxdisksetup
c#t#d#
a
ab-s init ora-rac-dg -i
r
y
l
e
#
vxdg
ora1=cAtAdA
f
bdu tr4.ansCreate the volumes for the database. You can use the provided
a
g onn
a
create_rac_vxvm_volumes script. This script is reproduced here, if
n
ab
#
#
#
#
vxdisk -o alldgs list

zpool status
metaset
cldev list -v
you want to type it in yourself rather than use the script:
vxassist
vxassist
vxassist
vxassist
vxassist
vxassist
vxassist
vxassist
vxassist
vxassist
vxassist
vxassist
vxassist
vxassist
12-32
-g
-g
-g
-g
-g
-g
-g
-g
-g
-g
-g
-g
-g
-g
ora-rac-dg
ora-rac-dg
ora-rac-dg
ora-rac-dg
ora-rac-dg
ora-rac-dg
ora-rac-dg
ora-rac-dg
ora-rac-dg
ora-rac-dg
ora-rac-dg
ora-rac-dg
ora-rac-dg
ora-rac-dg
make
make
make
make
make
make
make
make
make
make
make
make
make
make
raw_system 750m
raw_spfile 100m
raw_users 120m
raw_temp 100m
raw_undotbs1 312m
raw_undotbs2 312m
raw_sysaux 550m
raw_control1 110m
raw_control2 110m
raw_redo11 120m
raw_redo12 120m
raw_redo21 120m
raw_redo22 120m
raw_ocr 256m

vxassist -g ora-rac-dg make raw_css_voting_disk 256m

Note You need to create 3 additional volumes for each additional node
(more than 2) that you may be using. For example, for a third node create
the volumes raw_undotbs3 (size 312m), raw_redo13 (120m), and
raw_redo23 (120m)
5.
Zero out the configuration and voting devices. This eliminates

problems that might arise from data left over from a previous class:
# dd if=/dev/zero of=/dev/vx/rdsk/ora-rac-dg/raw_ocr bs=1kk

# dd if=/dev/zero of=/dev/vx/rdsk/ora-rac-dg/raw_css_voting_disk bs=1kk
s
) ha
y
m
om
c
csm Guide
e
@ withenthet newly created
b
6. Change the owner and group associated
u
ak here):
tud
volumes (assumes sh or ksh
syntax
y
(
S
in this
d
# cd /dev/vx/rdsk/ora-rac-dg
u
l
e
s
# for vol inm
*a
u
a
to
> do g k
e
s
> b
vxedit
set user=oracle group=dba $vol
an c-genora-rac-dg
a
i
l
>
done
b
u
le
k
b
a
a
l y sfer
u
d
an7B Creating Raw Volumes (Solaris Volume
ab Task
r
t
g
an nonManager)
Note Do not be concerned with error messages that appear at the end of
running the above commands. You will probably get errors indicating that
the end of the device does not align exactly on a 1kk (1MB) boundary.
ab
To create the raw volumes for the Oracle 11g RAC database, perform steps
15 from any one of the selected cluster nodes. Step 6 is the only step that
is run on all selected nodes.
1.
Select a single disk from shared storage for a new disk set. Make sure
you do not use any disk already in use in existing device groups.
Note the device identifier (DID) number of your selected disk
(referred to as d# in step 2). The following example checks against
VxVM, ZFS, and Solaris Volume Manager disks, just in case you are
running any combination.
#
#
#
#
vxdisk -o alldgs list

zpool status
metaset
cldev list -v

12-33
Note Just to speed up the lab, we will create unmirrored volumes. You
can always add another disk and mirror the parent volume later at your
leisure, or pretend that you were going to.
2.
Create a multi-owner diskset using this disk.

# metaset -s ora-rac-ds -a -M -h node1 node2
# metaset -s ora-rac-ds -a /dev/did/rdsk/d#
3.
Create a big mirror from which to partition all the necessary data
volumes (still call it a mirror although it has only one submirror;
then you could easily just add the second half of the mirror later at
your leisure)
s
) ha
y
m
om
# metainit -s ora-rac-ds d11 1 1 /dev/did/rdsk/d#s0
c
# metainit -s ora-rac-ds d10 -m d11

csm Guide
e
t provided
4. Create the volumes for the database. You
can usenthe
@
b
e
u
k script
create_rac_svm_volumes script.aThis
here,
tudis reproduced
y
(
S
including the comments indicating
which
volumes
you
are
creating.
in if tyou
d
hiswant to type them in yourself
The commands are highlighted,
u
l
e
a
rather than use the
script. us
m
a
k d100
to(750mb)
g
e
# systemnvolume
ba -slicora-rac-ds
ens
metainit
d100 -p d10 750m
a
b
e
u
l
k # spfile
a
ab volume d101 (100mb)
r
y
l
e
f
bdu transmetainit -s ora-rac-ds d101 -p d10 100m
a
ang non
# users volume d102 (120mb)
ab
metainit -s ora-rac-ds d102 -p d10 120m

# temp volume d103 (100mb)
# undo volume #1 d104 (312mb)
# undo volume #2 d105 (312mb)
# sysaux volume d106 (550mb)
# control volume #1 d107 (100mb)
12-34

# control volume #2 d108 (100mb)

# redo volume #1.1 d109 (120m)
# redo volume #2.2 d112(120m)
s
) ha
ab
y
m
om
c
# OCR Configuration volume d113 (256mb)

sm uide
metainit -s ora-rac-ds d113 -p d10
c256m
e
G
t
@
n
b
ku tude
# CSS voting disk d114 (256mb)
a
y
metainit -s ora-rac-ds
-pSd10 256m
n (d114
s
i
i
d
h
lu se t
a
u
am
o
k
t
Note You need
to
create
3
additional
volumes for each additional node
g nse
n
a
(more than
2)
that
you
may
be
using.
For
example, for a third node create
b lice
a
d115
b (312mb)
e for the undo #3 volume , d116 (120m) for redo #1.3, and
u
l
k
b
(120mb)
ra for redo #2.3.
ya d117
l
e
f
u
s
bd tran
a
ang non
5.
From either node, zero out the configuration and voting devices. This
eliminates problems that might arise from data left over from a
previous class:
# dd if=/dev/zero of=/dev/md/ora-rac-ds/rdsk/d113 bs=1kk

# dd if=/dev/zero of=/dev/md/ora-rac-ds/rdsk/d114 bs=1kk
Note Do not be concerned with error messages that appear at the end of
running the above commands. You will probably get errors indicating that
the end of the device does not align exactly on a 1kk (1MB) boundary.
6.
On all selected nodes, change the owner and group associated with the
newly created volumes.
# chown oracle:dba /dev/md/ora-rac-ds/dsk/*
# chown oracle:dba /dev/md/ora-rac-ds/rdsk/*

12-35
Task 8 Configuring Oracle Virtual IPs

Configure virtual IPs for use by the Oracle RAC database. Oracle CRS
controls these IPs, failing over both of them to a surviving node if one of
the nodes crashes. When one of these IPs fails over, Oracle clients do not
successfully contact the database using that IP. Instead, they get a
connection refused indication, and have their client software set to
automatically try the other IP.
Perform the following task on all selected nodes as root (you can edit the
hosts file on one node and copy it over or paste in your entries):
s
) ha
Edit the /etc/hosts file and create public network entries for new
virtual IPs. You will have one IP per node that you are using with
Oracle RAC. If you use consecutive IP addresses, then the
configuration assistant in the next task automatically guesses the
second IP when you type in the first. Make sure you do not conflict
with any other physical or logical IPs that you or any other students
have been using during the week.
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
# vi /etc/hosts
in this
d
x.y.z.w
nodename1-vip
u
al use
x.y.z.w+1
nodename2-vip
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab
12-36

Task 9 Configuring the oracle User Environment
Perform the following steps on all selected cluster nodes:

1.
Switch to the oracle user:

# su - oracle
2.
Edit .profile (or start by copying the profile.RAC provided) to

include environment variables for Oracle 11g RAC:
$ vi .profile
ORACLE_BASE=/oracle
ORACLE_HOME=$ORACLE_BASE/product/11.1.0/db_1
CRS_HOME=$ORACLE_BASE/product/11.1.0/crs
TNS_ADMIN=$ORACLE_HOME/network/admin
DBCA_RAW_CONFIG=$ORACLE_BASE/dbca_raw_config
#SRVM_SHARED_CONFIG=/dev/md/ora-rac-ds/rdsk/d101
SRVM_SHARED_CONFIG=/dev/vx/rdsk/ora-rac-dg/raw_spfile
DISPLAY=display-station-name-or-IP:display#
if [ `/usr/sbin/clinfo -n` -eq 1 ]; then
ORACLE_SID=sun1
fi
if [ `/usr/sbin/clinfo -n` = 2 ]; then
ORACLE_SID=sun2
fi
PATH=/usr/ccs/bin:$ORACLE_HOME/bin:$CRS_HOME/bin:/usr/bin:/usr/sbin
export ORACLE_BASE ORACLE_HOME TNS_ADMIN DBCA_RAW_CONFIG CRS_HOME
export SRVM_SHARED_CONFIG ORACLE_SID PATH DISPLAY
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran3. You might need to make a modification in the file, depending on
g
your choice of VxVM or Solaris Volume Manager. Make sure the line
n non
a
b
beginning SRVM_SHARED_CONFIG lists the correct option and that the
a
other choice is deleted or commented out.
4.
Make sure your actual X-Windows display is set correctly on the line
that begins with DISPLAY=.
5.
Read in the contents of your new .profile file and verify the
environment.
$ . ./.profile
$ env
6.
Enable rsh for the oracle user.

$ echo + >/oracle/.rhosts

12-37
Task 10A Creating the dbca_raw_config File (VxVM)

The dbca_raw_config file defines the locations of the raw devices of
the Oracle 11g RAC database. You must either copy the existing
dbca_raw_config.vxvm file that your instructor has prepared for
you or manually create the file as user oracle on all selected cluster
nodes.
$ vi /oracle/dbca_raw_config
system=/dev/vx/rdsk/ora-rac-dg/raw_system
spfile=/dev/vx/rdsk/ora-rac-dg/raw_spfile
users=/dev/vx/rdsk/ora-rac-dg/raw_users
temp=/dev/vx/rdsk/ora-rac-dg/raw_temp
undotbs1=/dev/vx/rdsk/ora-rac-dg/raw_undotbs1
undotbs2=/dev/vx/rdsk/ora-rac-dg/raw_undotbs2
sysaux=/dev/vx/rdsk/ora-rac-dg/raw_sysaux
control1=/dev/vx/rdsk/ora-rac-dg/raw_control1
control2=/dev/vx/rdsk/ora-rac-dg/raw_control2
redo1_1=/dev/vx/rdsk/ora-rac-dg/raw_redo11
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
bAdd entries
u
le for undotbs3, redo1_3, and redo2_3 using the
Note
k
b
a
a
l yappropriate
er additional volumes names you created if you have a third
f
u
s
d
an Extrapolate appropriately for each additional node.
ab -trnode.
g
n non
a
b
a
12-38

Task 10B Creating the dbca_raw_config File (Solaris

Volume Manager)
The dbca_raw_config file defines the locations of the raw devices of
the Oracle 11g RAC database. You must either copy the existing
dbca_raw_config.svm file that your instructor has prepared for you
or manually create the file as user oracle on all selected cluster
nodes.
$ vi /oracle/dbca_raw_config
system=/dev/md/ora-rac-ds/rdsk/d100
spfile=/dev/md/ora-rac-ds/rdsk/d101
users=/dev/md/ora-rac-ds/rdsk/d102
temp=/dev/md/ora-rac-ds/rdsk/d103
undotbs1=/dev/md/ora-rac-ds/rdsk/d104
undotbs2=/dev/md/ora-rac-ds/rdsk/d105
sysaux=/dev/md/ora-rac-ds/rdsk/d106
control1=/dev/md/ora-rac-ds/rdsk/d107
control2=/dev/md/ora-rac-ds/rdsk/d108
redo1_1=/dev/md/ora-rac-ds/rdsk/d109
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y Note
er Add entries for undotbs3, redo1_3, and redo2_3 using the
f
u
s
d
ab -tranappropriate additional volumes names you created if you have a third
g
n non
node. Extrapolate appropriately for each additional node.
a
b
a
Task 11 Disabling Access Control on X Server of the

Admin Workstation
To allow client GUIs to be displayed, run the following command on
the admin workstation or display station:
(# or $) /usr/openwin/bin/xhost +

12-39
Task 12 Installing Oracle CRS Software

Install the Oracle CRS software from the first selected cluster node. The
installer automatically copies over the binaries to the other nodes using
rsh at the end of the installation.
Perform the following steps on the first selected cluster node as the
oracle user:
1.
Change directory to the Oracle 11g CRS 11.1.0.6 software location:

$ cd ORACLE11gR1-CRS-software-location
2.
Run the runInstaller installation program by:

$ ./runInstaller
s
) ha
y
m
om
c
Table 12-2 Oracle CRS Installation Dialog Actions

csm Guide
e
@ ent
b
u
Dialog
Action
ak Stud
y
(
Welcome
Click Next. din
his
t
u
l
e
a us
m
a
Click
OK
get the error about the previous
k ewhen
to you(there
g
inventory
location
obviously was none).
n ens
a
b
a
lic the path to /oracle/oraInventory. Verify the
Specify Inventory directory
Change
b
e
u
l
and credentialsyak
ab group name (dba) and click Next.
r
l
e
f
s
du Details
n
Specify
Change the Path to:
bHome
a
a
r
t
g
n non
a
b
/oracle/product/11.1.0/crs
a
3.
Respond to the dialog boxes by using Table 12-2.
Click Next.
Checks
Most likely, these checks will all succeed.

Click Next.
If you happen to get a warning, click Next, and if you
get a pop-up warning window, click Yes to proceed.
12-40

Table 12-2 Oracle CRS Installation Dialog Actions (Continued)

Dialog
Action
Specify Cluster Configuration
Enter the name of the cluster (this is actually

unimportant).
For each node listed, verify the private host name
(clusternode1-priv, etc). These should have been
determined automatically.
For each node listed, verify that the Virtual Host Name
column contains nodename-vip similar to what you
entered in the /etc/hosts file.
s
) ha
y
m
om
c
e be
If you get some error concerning s
am
node that
idcannot
c
u
e
G
clustered (null), it is probably
because
you do not have
t
@
n
b
an /oracle/.rhosts
file.
You
must
have
one even on
u ude
k
a
t
the node on which
(y youisareSrunning the installer.
n
i
th section!
Specify Network Interface
Be very
careful
this
lud with
e
a
Usage
am to us
k
mark
in the instructions in this box,
ng To
se anitadapter
a
n
highlight
and
click
Then choose the appropriate
b
e
cradio button and clickEdit.
a
i
l
b
OK
u
le
k
b
a
a
l y sfer
u
Mark your actual public adapters as public.
d
b tran
a
ang non
Mark only the clprivnet0 interface as private.
Click Next.
ab
Mark all other adapters, including actual private network

adapters, as Do Not Use.
Click Next.
Specify Oracle Cluster
Registry (OCR) Location
Choose the External Redundancy radio button.

VxVM: Enter /dev/vx/rdsk/ora-rac-dg/raw_ocr
SVM: Enter /dev/md/ora-rac-ds/rdsk/d113
Click Next.

12-41
Table 12-2 Oracle CRS Installation Dialog Actions (Continued)

Dialog
Action
Specify Voting Disk Location
Choose the External Redundancy radio button.

VxVM: Enter /dev/vx/rdsk/ora-rac-dg/raw_css_voting_disk
SVM: Enter /dev/md/ora-rac-ds/rdsk/d114
Click Next.
Summary
Verify, and click Install.
Execute Configuration Scripts
IMPORTANT: You must wait for the scripts below to

complete on one node before starting another.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
/oracle/oraInventory/orainstRoot.sh
ak Stud
y
/oracle/product/11.1.0/crs/root.sh
(
in this
d
u
e the voting device and
al scriptusformats
The second
m
ka thee CRS
enables
to daemons on each node. Entries are put
g
so that the daemons run at boot time.
ban inlic/etc/inittab
ens
a
b
u
le On the first node, you may have to wait the whole 600
k
b
a
a
l y sfer
seconds for the verification to complete (have a snack
u
d
n
b
courtesy of your training center).
a
tra
g
n
n
a
no
On all selected nodes, one at a time, starting with the
node on which you are running the installer, open a
terminal window as user root and run the scripts:
ab
On all but the first node, the messages:

EXISTING Configuration detected...
NO KEYS WERE Written.....
are correct and expected.
As the root.sh script runs on the last node, various

CRS daemon resources will be configured and started
on all nodes.
Click OK when root.sh finished on the last node.
Configuration Assistants
These should all succeed and pass you to the final

screen.
End of Installation
12-42

Task 13 Installing Oracle Database Software

Install the Oracle database software. The installer automatically copies
over the binaries to the other nodes using rsh at the end of the
installation.
1.
On all selected nodes, become root and change the /oracle

permissions back:
# chown -R oracle:dba /oracle
2.
On the node from which you installed CRS, as the user oracle, run the
the database installer. This is in a different directory than the CRS
software location:
$ cd ORACLE11gR1-db-location
$ ./runInstaller
s
) ha
y
m
m
3. Respond to the dialog boxes by using Table 12-3.co
e
m
d
s
i
ec t Gu
Table 12-3 Install Oracle Database Software Dialog Actions@
b den
u
k
ya Stu
Dialog
Action
(
n
i
is
lud se th
Welcome
Click a
Next.
u
am thetoCustom
k
Select Installation Type
Select
radio button, and click Next.
g nse
n
a
e especially that the destination path is
Install Location
ab licVerify,
b
u
le /oracle/product/11.1.0/db_1
k
b
a
a
l y sfer
u
d
Click Next. When you see the warning about the Oracle
ab -tran
g
Base, click Yes.
n
n
o
a
n
ab
Specify Hardware Cluster
Installation Mode
Verify that the Cluster Installation radio button is

selected.
Put check marks next to all of your selected cluster

nodes.
Click Next.
Checks
You may get a warning about swap space.

Click Next, and if you get a pop-up warning window,
click Yes to proceed.

12-43
Table 12-3 Install Oracle Database Software Dialog Actions (Continued)

Dialog
Action
Available Product
Components
Deselect the following components (to speed up the

installation):
Enterprise Edition Options
Oracle Programmer 11.1.0.6.0
Click Next.
Privileged Operating System
Groups
Verify that dba is listed in all entries, and click Next.
s
) ha
y
m
Create Database
Verify that Create a Database is selected, and click
om Next.
c
e
m
Summary
Verify, and click Install
d
s
i
ec t Gu
@
Oracle Net Configuration
Verify that the Perform Typical
Configuration
check
n
e
ubNext.
d
k
Assistant Welcome
box is not selected, andaclick
tu
y
(
S
n
Listener Configuration,
Verify the Listener
di name
hisisLISTENER, and click Next.
t
u
l
e
a us
Listener Name
m
a
k thateTCP
to is among the Selected Protocols, and
g
Select Protocols
Verify
ns
ban click
Next.
e
c
a
li
b
e
u
l
k
TCP/IP Protocola
that the Use the Standard Port Number of 1521
ab Verify
r
y
l
e
Configuration
radio
button
is selected, and click Next.
f
u ns
d
b
aListeners-tra
More
Verify that the No radio button is selected, and click
g
n
n
o
a
Next
(be patient with this one, it takes a few seconds to
n
ab
move on)
Listener Configuration Done
Click Next.
Naming Methods
Configuration
Verify that the No, I Do Not Want to Configure

Additional Naming Methods radio button is selected,
and click Next.
Done
Click Finish.
DBCA Step 1: Database

Templates
Select the General Purpose radio button, and click

Next.

Identification
Type sun in the Global Database Name text field (notice

that your keystrokes are echoed in the SID Prefix text
field), and click Next.
12-44


Dialog
Action
DBCA Step 3: Management

Options
Uncheck Configure Enterprise Manager and click

Next.

Credentials
Select the Use the Same Password for All Accounts

radio button.
Enter cangetin as the password, and click Next.
DBCA Step 5: Storage Options
Select the Raw Devices radio button. Verify that the

check box for Raw Devices Mapping File is selected,
and that the value is /oracle/dbca_raw_config.
s
) ha
y
m
om
c
DBCA Step 6: Recovery

Leave the boxes unchecked, and click
ide
csm Next.
u
Configuration
e
G
t
@
n
b
e Next.
Uncheck Sample Schemas,
dclick
ku and
u
a
t
y
Content
n ( this S
i
d
luTypical
eradio button selected.
DBCA Step 8: Initialization
Keep a
the
s
m
u
Parameters
ka e to
g
Use the
slider to choose the lowest possible Memory
ban licSize
ens
a
b
u
le
k
b
a
a
Click Next.
l y sfer
u
d
n
ab Step-t9:raSecurity
DBCA
Verify the radio button is at Keep the enhanced....
g
n
n
Settings
and click Next.
no
aba
Click Next.
DBA Step 10 Automatic

Maintenance Tasks
Uncheck the checkbox (we arent staying around for

maintenance) and click Next.

Storage
Verify that the database storage locations are correct by

clicking leaves in the file tree in the left pane and
examining the values shown in the right pane. These
locations are determined by the contents of the
/oracle/dbca_raw_config file that you prepared in a
previous task. Click Next.
DBCA Step 12: Creation

Options
Verify that the Create Database check box is selected,

and click Finish.
Summary
Verify and click OK.

12-45

Dialog
Action
Database Configuration
Assistant
Click Exit. Wait a while (anywhere from a few seconds

to a few minutes: be patient) and you will get a pop-up
window informing you that Oracle is starting the RAC
instances.
Execute Configuration Scripts
On each of your selected nodes, open a terminal

window as root, and run the script:
/oracle/product/11.1.0/db_1/root.sh
s
Accept the default path name for the local bin directory. ha
)
y
m
Click OK in the script prompt window.

om
c
End of Installation
csm Guide
e
@ ent
b
u
ak Stud to Control
y
(
Task 14 Create Sun Cluster
Resources
in this
d
u
Oracle RAC Through CRS
al use
m
ka e to
g
s on only one of your selected nodes to create
Perform the following
an censteps
b
a
Sun Cluster
resources
li that monitor your RAC storage, and that allow you
bSun Cluster
e
u
l
to
use
k
b to control your RAC instances through CRS:
a
a
r
y
fe the types required for RAC storage and RAC instance
ul 1. nsRegister
d
b
a
tra control:
g
n
n
a
no
ab# clrt register crs_framework
# clrt register ScalDeviceGroup

# clrt register scalable_rac_server_proxy
2.
Create a CRS framework resource (the purpose of this resource is to

try to cleanly shut down CRS if you are evacuating the node using
cluster commands):
# clrs create -g rac-framework-rg -t crs_framework \

-p Resource_dependencies=rac-framework-res \
crs-framework-res
3.
Create a group to hold the resource to monitor the shared storage:
# clrg create -S -n node1,node2 \

-p RG_affinities=++rac-framework-rg \
rac-storage-rg
12-46


4.
Create the resource to monitor the shared storage. Do one of the

following commands:
If you are using VxVM, do this one and skip the next one:
# clrs create -g rac-storage-rg -t ScalDeviceGroup \
-p DiskGroupName=ora-rac-dg \
-p Resource_dependencies=rac-cvm-res{local_node} \
rac-storage-res
If you are using Solaris Volume Manager, perform this one:
# clrs create -g rac-storage-rg -t ScalDeviceGroup \
-p DiskGroupName=ora-rac-ds \
-p Resource_dependencies=rac-svm-res{local_node} \
rac-storage-res
s
) ha
y
m
om
c
# clrg online -M rac-storage-rg

csm Guide
e
6. Create a group and a resource to allow@
you to run
t cluster commands
n
b You
e
u
that control the database throughkCRS.
will
d be warned about the
u
a
t
y
strong affinities, but the command
S succeed.
n ( thisshould
i
d
# clrg create -S -n node1,node2 a
\ lu
e
s
m
u
-p RG_affinities=++rac-framework-rg,++rac-storage-rg
\
a
o
k
t
rac-control-rg
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
a
no
5.
Everyone should continue here. Bring the resource group that

monitors the shared storage online.
ab

12-47
Note For the following command, make sure you understand which
node has node id 1, which has node id 2, and so forth, so that you match
correctly with the names of the database sub-instances. You can use
clinfo -n to verify the node id on each node.
# vi /var/tmp/cr_rac_control
clrs create -g rac-control-rg -t scalable_rac_server_proxy \
-p DB_NAME=sun \
-p ORACLE_SID{name_of_node1}=sun1 \
-p ORACLE_SID{name_of_node2}=sun2 \
-p ORACLE_home=/oracle/product/11.1.0/db_1 \
-p Crs_home=/oracle/product/11.1.0/crs \
-p Resource_dependencies_offline_restart=rac-storage-res{local_node} \
-p Resource_dependencies=rac-framework-res \
rac-control-res
y
m
om
c
csm Guide
e
# sh /var/tmp/cr_rac_control
@ ent
b
u
ak Stud
y
(
# clrg online -M rac-control-rg
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
12-48

Task 15 Verifying That Oracle RAC Works Properly in

a Cluster
Run the following commands as indicated to verify that the Oracle
software properly runs in the cluster:
Note So that the flow of the following task makes sense, make sure you
understand which node has node id 1, which has node id 2, and so forth.
1.
Switch to the oracle user (all selected cluster nodes):

# su - oracle
s
) ha
y
m
$ sqlplus SYS@sun1 as sysdba

m
o
c
Enter password: cangetin

e
m
d
s
i
ec t Gu
@
SQL> create table mytable (Name VARCHAR(20), age
n
b NUMBER(4));
e
u
d
k
SQL> insert into mytable values ('vincent',
14);
a Stu
y14);
(
SQL> insert into mytable values ('theo',
n
di e this
u
SQL> commit;
l
a us
SQL> select * from mytable; am
k e to
SQL> quit
g
n ens
bathe
3. From
a
ic node, query the other database sub-instance and
lother
b
e
u
l
k verify
b that the data is there:
a
a
r
y
ul nsfe$ sqlplus SYS@sun2 as sysdba
d
b
a
tra
g
n
n
a
no
2.
ab
On node 1, connect to database sub-instance sun1 and create a table.

SQL> quit
4.
On either node, take advantage of the Sun Cluster control resource to

use Sun Cluster commands to shut down the database on one of the
nodes (the resource accomplishes the actual control through CRS).
# clrs disable -n node2 rac-control-res
5.
On the node for which you shutdown Oracle, verify as the oracle
user that the instance is unavailable:
$ crs_stat -t

12-49

6.
On either node, reenable the instance through the Sun Cluster

resource:
# clrs enable -n node2 rac-control-res

7.
On that (affected node), you should be able to repeat step 5. It might

take a few attempts before the database is initialized and you can
successfully access your data.
8.
Cause a crash of node 1:

# <Control-]>
telnet> send break
9.
On the surviving node, you should see (after 45 seconds or so), the
CRS-controlled virtual IP for the crashed node migrate to the
surviving node. Run the following as user oracle:
s
) ha
y
m
om
c
m uinode2
ora.node2.vip application
ONLINE sONLINE
de
ora.node1.vip application
ONLINEec ONLINE
node2
G
t
@
n
b
u that
10. While this virtual IP has failed over,kverify
dethere is actually no
u
a
t
y
( iCRS.
S This virtual IP fails over
failover listener controlled by
Oracle
n
s
i
d
h
merely so a client quickly
gets a TCP
t disconnect without having to
luClient
e
a
s
wait for a long time-out.
software
then has a client-side option
m to u
a
k
to fail over g
to the other
e instance.
n
s
a
n
$ sqlplus
as sysdba
ce
ab SYS@sun1
i
l
b
uSQL*Plus:
le Release 11.1.0.6.0 - Production on Tue Mar 25
k
b
a
a
10:56:18
2009
y fer
l
u
s
bd tran Copyright (c) 1982, 2005, ORACLE. All rights reserved.
a
ang non
$ crs_stat -t|grep vip
ab
Enter password:
ERROR:
ORA-12541: TNS:no listener
Enter user-name: ^D
11. Boot the node that you had halted, by typing boot or go at the OK
prompt in the console. If you choose the latter, the node will panic
and reboot.
12. After the node boots, monitor the automatic recovery of the virtual
IP, the listener, and the database instance on the surviving node:
$ crs_stat -t
$ /usr/cluster/bin/clrs status
It can take several minutes for the full recovery. Keep repeating the
steps.
12-50


13. Verify the proper operation of the Oracle database by contacting the
various sub-instances as the user oracle on the various nodes:

SQL> quit
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

12-51
Exercise Summary
Exercise Summary

Experiences
Interpretations
Conclusions
Applications
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
12-52

Appendix A
Terminal Concentrator
This appendix describes the configuration of the Sun Terminal
Concentrator (TC) as a remote connection mechanism to the serial
consoles of the nodes in the Sun Cluster software environment.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
A-1
Viewing the Terminal Concentrator

The Sun Terminal Concentrator (Annex NTS) has its own internal
operating system and resident administration programs.
Note If any other TC is substituted, it must not send an abort signal to
the attached host systems when it is powered on.
Figure A-1 shows that the TC is a self-contained unit with its own
operating system.
s
) ha
y
m
Self -Load Power-On

om
c
PROM
sm uide
c
e
Operating System
G
t
@
n
b
oper.52.enet
ku tude
a
y
Serial Ports
n7 ( 8this S
i
5
6
1
4
d
2
3
lu se
a
Serial port A
am to u
k
ng ense
Setup port
a
b
a Node
Node 2
lic1
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
traSetup Device
g
n
n
no
aba
Memory
Network
Interface
Figure A-1
Terminal Concentrator Functional Diagram
Note If the programmable read-only memory (PROM) operating system

is older than version 52, you must upgrade it.
A-2

Operating System Load

You can set up the TC to load its operating system either internally from
the resident PROM or externally from a server. In the cluster application,
it is always set to load internally. Placing the operating system on an
external server can decrease the reliability of the terminal server.
When power is first applied to the TC, it performs the following steps:
1.
It runs a PROM-based self-test and displays error codes.
2.
It loads a resident PROM-based operating system into the TC

memory.
s
) ha
ab
y
m
Setup Port
om
c
e
smis used
idonly
cthat
u
Serial port 1 on the TC is a special purpose port
during
e
G
t
@
n
b
initial setup. It is used primarily to set u
up the IP e
address and load
d
k
u
a
sequence of the TC. You can access
port
1
from
either
a tip connection or
t
y
(
S
n
from a locally connected terminal.
di e this
u
l
a us
m
a
k e toSetup Programs
g
ban licens
a
bmust configure
u
le
k
You
the TC nonvolatile random access memory (NVRAM)
b
a
a
r
y
l
e
with
the
appropriate
IP address, boot path, and serial port information.
f
u ns
d
b
a
tra Use the following resident programs to specify this information:
g
n
n
a
addr
no
seq
image
admin
A-3
Setting Up the Terminal Concentrator

The TC must be configured for proper operation. Although the TC setup

menus seem simple, they can be confusing and it is easy to make a
mistake. You can use the default values for many of the prompts.
Connecting to Port 1
To perform basic TC setup, you must connect to the TC setup port.
Figure A-2 shows a tip hardwire connection from the administrative
console. You can also connect an American Standard Code for Information
Interchange (ASCII) terminal to the setup port.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
e1
al touPort
s
m
Figure A-2 Setup Connection
ka e to
g
ban licens
a
bSetupleMode
Enabling
u
k
a erab
y
l
f Setup mode, press the TC Test button shown in Figure A-3 until
u To nenable
s
d
b
a TC power indicator begins to blink rapidly, then release the Test
a
trthe
g
n
n
no button and press it again briefly.
aba
1
STATUS
POWER
UNIT
NET
Power indicator
Figure A-3
ATTN
LOAD
ACTIVE
Test button
Terminal Concentrator Test Button
After you have enabled Setup mode, a monitor:: prompt should appear
on the setup device. Use the addr, seq, and image commands to complete
the configuration.
A-4

Setting the Terminal Concentrator IP Address

The following example shows how to use the addr program to set the IP
address of the TC. Usually this is set correctly when your cluster arrives,
but you should always verify that it is correct.
monitor:: addr
Enter Internet address [192.9.22.98]:: 129.150.182.100
Enter Subnet mask [255.255.255.0]::
Enter Preferred load host Internet address
[192.9.22.98]:: 129.150.182.100
Enter Broadcast address [0.0.0.0]:: 129.150.182.255
Enter Preferred dump address [192.9.22.98]::
129.150.182.100
Select type of IP packet encapsulation
(ieee802/ethernet) [<ethernet>]::
Type of IP packet encapsulation: <ethernet>
s
) ha
y
m
om
c
csm Guide
e
Load Broadcast Y/N [Y]:: y ub@ ent
ak Stud
y
(
in this
d
u
l
Setting the TerminalaConcentrator
Load Source
e
s
m
u
ka e to
g
The following
an example
ns shows how to use the seq program to specify the
b
e
c
a
type
li mechanism to be used:
b of loading
e
u
l
k
a emonitor::
ab
seq
r
y
l
f
u
s
a list of 1 to 4 interfaces to attempt to use for
bd tran Enter
a
downloading
code or upline dumping. Enter them in the
g onn
a
order they should be tried, separated by commas or
n
ab
spaces. Possible interfaces are:
Ethernet: net
SELF: self
Enter interface sequence [self]::
The self response configures the TC to load its operating system

internally from the PROM when you turn on the power. The PROM image
is currently called oper.52.enet.
Enabling the self-load feature negates other setup parameters that refer to
an external load host and dump host, but you must still define these
parameters during the initial setup sequence.
A-5
Note Although you can load the TCs operating system from an external
server, this introduces an additional layer of complexity that is prone to
failure.
Specifying the Operating System Image

Even though the self-load mode of operation negates the use of an
external load and dump device, you should still verify the operating
system image name as shown by the following:
monitor:: image
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
monitor::
y
(
in this
d
u
al use
m
Note Do not define
kaa dump
toor load address that is on another network
g
e
because you a
receive additional
questions about a gateway address. If you
nspress Control-C
b n you
e
make a mistake,
can
to abort the setup and start
c
a
li
b
e
u
l
again.
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
o
the Serial Port Variables
nSetting
aba
Enter Image name ["oper.52.enet"]::
Enter TFTP Load Directory ["9.2.7/"]::
Enter TFTP Dump path/filename
["dump.129.150.182.100"]::
The TC port settings must be correct for proper cluster operation. This
includes the type and mode port settings. Port 1 requires different type
and mode settings. You should verify the port settings before installing the
cluster host software. The following is an example of the entire procedure:
admin-ws# telnet terminal_concentrator_name
Trying terminal concentrator IP address ...
Connected to terminal concentrator IP address.
Escape character is '^]'.
Rotaries Defined:
cli
Enter Annex port name or number: cli
Annex Command Line Interpreter * Copyright 1991
Xylogics, Inc.
annex: su
A-6


Password: type the password
annex# admin
Annex administration MICRO-XL-UX R7.0.1, 8 ports
admin: show port=1 type mode
Port 1:
type: hardwired
mode: cli
admin:set port=1 type hardwired mode cli
admin:set port=2-8 type dial_in mode slave
admin:set port=1-8 imask_7bits Y
admin: quit
annex# boot
bootfile: <CR>
warning: <CR>
s
) ha
ab
y
m
Note Do not perform this procedure through the special m

osetupport; use
c
public network access.

csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
Setting the Port Password
d
u
al use
m
a
krecommended
to security feature is to set per port
An optional and
g
e
ns an extra password challenge as you access a
passwords.
provides
banThese
e
c
a
li mode remotely through the telnet command.
b port line slave
serial
u
k
a erab
y
l
f can set different (or the same) port passwords on each port. You must
u nYou
s
d
b
a
tra set the enable_security parameter to Y to enable all the passwords.
g
n
n
a
no
If you ever forget a port password but know the TC root password, you
can just reset the passwords to whatever you want.
admin-ws# telnet terminal_concentrator_name
Trying terminal concentrator IP address . . .
Connected to terminal concentrator IP address.
Rotaries Defined:
cli
Xylogics, Inc.
annex: su
Password: type the password
annex# admin
A-7
admin:
admin:
admin:
admin:
admin:
set port=2 port_password homer

set port=3 port_password marge
reset 2-3
set annex enable_security Y
reset annex security
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
A-8

Setting a Terminal Concentrator Default Route

If you access a TC from a host that is on a different network, the TCs
internal routing table can overflow. If the TC routing table overflows,
network connections can be intermittent or lost completely.
Figure A-4 shows that you can correct this problem by setting a default
route within the TC config.annex configuration file.
Administrative Console
Router
hme0
hme1
s
) ha
129.50.2.12
y
m
om
c
Network
sm uide
c129.50.2
129.50.1
e
G
t
@
Terminal Concentrator (129.50.1.35)
n
b
ku tude
a
config.annex
y
n ( this S
i
d
%gateway
lu129.50.1.23
e metric 1 active
a
net default gateway
s
m
u
ka e to
g
ban licens
a
b Nodele1
u
Node 2
Node 3
Node 4
k
b
a
a
r
y
ul nsfe
d
b
a
tra Figure A-4 Terminal Concentrator Routing
g
n
n
no
aba
129.50.1.23
129.50.2.17
A-9
Creating a Terminal Concentrator Default Route
To create a default route for the TC, you must edit an electrically erasable
programmable read-only memory (EEPROM) file in the TC named
config.annex. You must also disable the TC routing function. The
following is a summary of the general process:
admin-ws# telnet tc1.central
Trying 129.50.1.35 ...
Connected to 129.50.1.35.
Escape character is ^].
[Return] [Return]
...
annex: su
Password: root_password
annex# edit config.annex
(Editor starts)
Ctrl-W:save and exit Ctrl-X:exit Ctrl-F:page down
Ctrl-B:page up
%gateway
net default gateway 129.50.1.23 metric 1 active ^W
annex# admin set annex routed n
You may need to reset the appropriate port, Annex
subsystem or reboot the Annex for changes to take
effect.
annex# boot
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non Note You must enter an IP routing address appropriate for your site.
a
b
a
While the TC is rebooting, the node console connections are not available.
A-10

Using Multiple Terminal Concentrators
Using Multiple Terminal Concentrators

A single TC can provide serial port service to a maximum of eight nodes.
If it is necessary to reconfigure the TC, the node connection to port 1 must
be switched with a serial connection to the configuration device, and the
TC placed into setup mode. After configuration is complete, the normal
node connection to port 1 is replaced and the TC rebooted.
The maximum length for a TC serial port cable is approximately 348 feet.
As shown in Figure A-5, it might be necessary to have cluster host
systems separated by more than the serial port cable limit. You might
need a dedicated TC for each node in a cluster.
s
) ha
y
m
om
c
Router
csm Guide
e
@ ent
b
u
ak Network
tud
y
(
S
in this
d
Network
u
al use
m
ka e to
TC
g
n
s
a
n
ab lice TC
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
Node 2
g
n
n
o
a
n
ab
Node 1
Administrative Console
Figure A-5
Multiple Terminal Concentrators
A-11
Troubleshooting Terminal Concentrators

Occasionally, it is useful to be able to manually manipulate the TC. The
commands to do this are not well documented in the cluster manuals.
Using the telnet Command to Manually Connect to a

Node
If the cconsole tool is not using the TC serial ports, you can use the
telnet command to connect to a specific serial port as follows:
# telnet tc_name 5002
s
) ha
y
m
m have
You can then log in to the node attached to port 5002. After o
you
c
finished and logged out of the node, you must breaksthe

telnet
e
m
d
i
c thenGtype
u quit. If
connection with the Control-] keyboard sequencee
and
t
@
nbe used by other
b cannot
you do not, the serial port remains locked u
and
e
d
k
applications, such as the cconsole tool.
ya Stu
(
n
di e this
u
l
a us
m
Using the telnet Command
a
k e to to Abort a Node
g
an cens
b
a
If you b
have to abort
li a cluster node, you can either use the telnet
e
u
l
k
command
connect directly to the node and use the Control-] keyboard
a eraortobyou
y
l
sequence,
can use the Control-] keyboard sequence in a cluster
f
u ns
d
b
a window. When you have the telnet prompt, you can abort the
a
trconsole
g
n
node
with the following command:
n
a
no
ab
telnet > send brk

ok
Note You might have to repeat the command multiple times.
A-12

Connecting to the Terminal Concentrator

Command-Line Interpreter
You can use the telnet command to connect directly to the TC, and then
use the resident command-line interface (CLI) to perform status and
administration procedures.
# telnet IPaddress
Trying 129.146.241.135...
Connected to 129.146.241.135
Xylogics, Inc.
annex:
s
) ha
ab
y
m
om
c
csm Guide
e
t
@ eCommand
n
b
Using the Terminal Concentrator
help
u
ak Stud
y
(
ina terminal
d
his concentrator, you can get online
t
After you connect directly
into
u
l
e
a us
help as follows: am
k e to
g
s
annex:
nhangup
ban help
e
c
a
annex:
help
li
b
e
u
l
k
a erab
y
l
f
uIdentifying
s
d
n
and Resetting a Locked Port
b
a
a
r
t
g
an non
If a node crashes, it can leave a telnet session active that effectively locks
the port from further use. You can use the who command to identify which
port is locked, and then use the admin program to reset the locked port.
The command sequence is as follows:
annex: who
annex: su
Password:
annex# admin
admin : reset 6
admin : quit
annex# hangup
A-13
Erasing Terminal Concentrator Settings
Using the TC erase command can be dangerous. Use it only when you
have forgotten the superuser password. It returns all settings to their
default values. When the addr command is then run to give the TC its IP
address, the password will be set to this IP. For security reasons, the
erase command is available only through the port 1 interface. A typical
procedure is as follows:
monitor :: erase
1) EEPROM(i.e. Configuration information)
2) FLASH(i.e. Self boot image)
s
) ha
y
m
om
c
Caution Do not use option 2 of the erase command;

it destroys
idethe TC
csm
u
e
boot PROM-resident operating system.
G
t
@
n
b
ku tude
a
y
n ( this S
i
d
lu se
a
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
no
aba
Enter 1 or 2 :: 1
A-14

Appendix B
Configuring Multi-Initiator SCSI

This appendix contains information that can be used to configure
multi-initiator Small Computer System Interface (SCSI) storage devices,
including the Sun StorEdge MultiPack desktop array and the Sun
StorEdge D1000 array.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
B-1
Multi-Initiator Overview
Multi-Initiator Overview
This section applies only to SCSI storage devices and not to Fibre Channel
storage used for the multihost disks.
In a stand-alone server, the server node controls the SCSI bus activities
using the SCSI host adapter circuit connecting this server to a particular
SCSI bus. This SCSI host adapter circuit is referred to as the SCSI initiator.
This circuit initiates all bus activities for this SCSI bus. The default SCSI
address of SCSI host adapters in Sun systems is 7.
Cluster configurations share storage between multiple server nodes.
When the cluster storage consists of singled-ended or differential SCSI
devices, the configuration is referred to as multi-initiator SCSI. As this
terminology implies, more than one SCSI initiator exists on the SCSI bus.
s
) ha
y
m
om
c
m uide
ca sSCSI
The SCSI specification requires that each device on
have a
e
Gbus
t
@
unique SCSI address. (The host adapter is u
also
a device
on
the
SCSI bus.)
n
b
e
d
k
The default hardware configuration in
tu environment
yaa multi-initiator
(
S
results in a conflict because all SCSI
host
adapters
default to 7.
n
i
is
d
h
t
u
al use
m
To resolve this conflict,
on each
SCSI bus leave one of the SCSI host
ka address
toof
g
e
adapters with the
SCSI
7, and set the other host adapters to
n ens
a
b
unused SCSI
Proper planning dictates that these unused
a addresses.
lic both
b
e
SCSI
addresses
include
currently and eventually unused addresses.
u
l
k
b
a
a
l yAn example
er of addresses unused in the future is the addition of storage
f
u
s
d
by
installing
new drives into empty drive slots. In most configurations,
b tran
a
- the available SCSI address for a second host adapter is 6.
ang non
ab
B-2
You can change the selected SCSI addresses for these host adapters by
setting the scsi-initiator-id OpenBoot PROM property. You can set
this property globally for a node or on a per host adapter basis.
Instructions for setting a unique scsi-initiator-id for each SCSI host
adapter are included in the chapter for each disk enclosure in Sun
Cluster Hardware Guides.

Installing a Sun StorEdge Multipack Device

This section provides the procedure for an initial installation of a
Sun StorEdge MultiPack device.
Use this procedure to install a Sun StorEdge MultiPack device in a cluster
prior to installing the Solaris Operating System and Sun Cluster
software. Perform this procedure with the procedures in Sun Cluster 3.2
Installation Guide and your server hardware manual.
1.
Ensure that each device in the SCSI chain has a unique SCSI address.
The default SCSI address for host adapters is 7. Reserve SCSI address
7 for one host adapter in the SCSI chain. This procedure refers to the
host adapter you choose for SCSI address 7 as the host adapter on
the second node. To avoid conflicts, in step 7 you change the
scsi-initiator-id of the remaining host adapter in the SCSI chain
to an available SCSI address. This procedure refers to the host
adapter with an available SCSI address as the host adapter on the
first node. Depending on the device and configuration settings of the
device, either SCSI address 6 or 8 is usually available.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
Caution Even k
though
a a tslot
o in the device might not be in use, you
g
e
should avoid
setting
scsi-initiator-id
for the first node to the SCSI
n ens
a
b
address
a for thatlicdisk slot. This precaution minimizes future complications
b
u
leadditional disk drives.
if you install
k
b
a
a
l y sfer
u
d
more information, refer to OpenBoot 3.x Command Reference Manual
ab -tranFor
g
and
the labels inside the storage device.
n
n
o
a
n
ab
2.
Install the host adapters in the nodes that will be connected to the
device.
For the procedure on installing host adapters, refer to the documentation

that shipped with your nodes.
3.
Connect the cables to the device, as shown in Figure B-1 on page B-4.

B-3
Make sure that the entire SCSI bus length to each device is less than
6 meters. This measurement includes the cables to both nodes, as
well as the bus length internal to each device, node, and host
adapter. Refer to the documentation that shipped with the device for
other restrictions regarding SCSI operation.
Node 1
Host Adapter A
Host Adapter B
Node 2
Host adapter B
Host Adapter A
SCSI IN
9-14
1-6
SCSI IN
IN
9-14
1-6
IN
s
) ha
y
m
m
o
c
SCSI OUT
SCSI OUT
e
m
d
s
i
ec2 t Gu
Enclosure 1
Enclosure
@
b den
u
k
a Stu
SCSI (y
n
cables
di e this
u
l
a us
m
a
Figure B-1 Example
of
a
Sun
MultiPack Desktop Array
k Mirrored
toStorEdge
g
e
Enclosure
Pair
ban licens
a
b
u
le AC power cord for each device of the pair to a different
k
4.
Connect
the
b
a
a
l y spower
er source.
f
u
d
ab -tr5.an Without allowing the node to boot, power on the first node. If
g
n non
a
necessary, abort the system to continue with OpenBoot PROM
b
a
OUT
OUT
Monitor tasks.
6.
Find the paths to the host adapters.

ok show-disks
Identify and record the two controllers that will be connected to the
storage devices, and record these paths. Use this information to
change the SCSI addresses of these controllers in the nvramrc script.
Do not include the trailing /disk or /sd entries in the device
paths.
7.
Edit the nvramrc script to set the scsi-initiator-id for the

host adapter on the first node.
For a list of nvramrc editor and nvedit keystroke commands, see
the The nvramrc Editor and nvedit Keystroke Commands on
page B-11.
B-4

The following example sets the scsi-initiator-id to 6. The

OpenBoot PROM Monitor prints the line numbers (0:, 1:, and so
on).
nvedit
0: probe-all
1: cd /pci@1f,4000/scsi@4
2: 6 encode-int " scsi-initiator-id" property
3: device-end
4: cd /pci@1f,4000/scsi@4,1
6: device-end
7: install-console
8: banner <Control-C>
ok
s
) ha
y
m
om
c
Note Insert exactly one space after the first double

idemark and
csmquotation
u
e
before scsi-initiator-id.
G
t
@
n
b
ku tude
a
8. Store the changes.
y
S
n ( ththeisnvedit
i
d
The changes you make
through
command are done on a
u se
l
a
temporary copy
script. You can continue to edit this
mof thetonvramrc
u
a
k
copy without
you complete your edits, save the changes.
g risk.
eAfter
n
s
a
n
If you
are
not
sure
about
the changes, discard them.
b lice
a
b To
u
lediscard the changes, type:
k
b
a
a
l y sfer ok nvquit
u
d
ok
ab -tran
g
n non
a
To store the changes, type:
b
a
ok nvstore
ok
9.
Verify the contents of the nvramrc script you created in step 7.

ok printenv nvramrc
nvramrc = probe-all
cd /pci@1f,4000/scsi@4
6 encode-int " scsi-initiator-id" property
device-end
cd /pci@1f,4000/scsi@4,1
device-end
install-console
banner
ok

B-5

10. Instruct the OpenBoot PROM Monitor to use the nvramrc script.
ok setenv use-nvramrc? true

use-nvramrc? = true
ok
11. Without allowing the node to boot, power on the second node. If
Monitor tasks.
12. Verify that the scsi-initiator-id for the host adapter on the
second node is set to 7.
ok cd /pci@1f,4000/scsi@4
ok .properties
scsi-initiator-id
00000007
...
y
m
13. Continue with the Solaris OS, Sun Cluster software, and
volume
m
o
c
management software installation tasks.

e
m
d
s
i
ec t Gu
@
b den
u
k
ya Stu
(
n
di e this
u
l
a us
m
a
k e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
B-6

Installing a Sun StorEdge D1000 Array

This section provides the procedure for an initial installation of a Sun
StorEdge D1000 array.
Use this procedure to install a Sun StorEdge D1000 array in a cluster prior
to installing the Solaris OS and Sun Cluster software. Perform this
procedure with the procedures in Sun Cluster 3.1 Installation Guide and
your server hardware manual.
1.
Ensure that each device in the SCSI chain has a unique SCSI address.
The default SCSI address for host adapters is 7. Reserve SCSI
address 7 for one host adapter in the SCSI chain. This procedure
refers to the host adapter you choose for SCSI address 7 as the host
adapter on the second node. To avoid conflicts, in step 7 you change
the scsi-initiator-id of the remaining host adapter in the SCSI
chain to an available SCSI address. This procedure refers to the host
adapter with an available SCSI address as the host adapter on the
first node. SCSI address 6 is usually available.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
l
edevice might not be in use, you should
aslot
s
m
u
Note Even though
a
in
the
a
kscsi-initiator-id
to
avoid setting
the
for the first node to the SCSI
g
e
nthat disk
s This precaution
a
n
b
address
for
slot.
minimizes future complications
e
c
a
i
l
b
ifuyou install
leadditional disk drives.
k
b
a
a
r
l y For
emore
f
u
s
d
information, refer to OpenBoot 3.x Command Reference Manual
b tranand the labels
a
inside the storage device.
g onn
a
n
ab
2.
Install the host adapters in the node that will be connected to the
array.
For the procedure on installing host adapters, refer to the
documentation that shipped with your nodes.
3.
Connect the cables to the arrays, as shown in Figure B-2.

B-7
Make sure that the entire bus length connected to each array is less
than 25 meters. This measurement includes the cables to both nodes,
as well as the bus length internal to each array, node, and the host
adapter.
Node 1
Host Adapter A
Host Adapter B
Node 2
Host Adapter B
Host Adapter A
Disk array 1
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ud Mirrored Pair
akD1000
tArray
y
(
S
Figure B-2 Example of a Sun StorEdge
in this
d
u
al cordufor
seeach array of the pair to a different
m
4. Connect the ACa
power
k e to
g
power source.
an cens
b
a
5. Power
b onlethelifirst node and the arrays.
u
k
a Find
athebpaths to the host adapters.
6.
r
y
l
e
f
bdu transok show-disks
a
g onn
Identify and record the two controllers that will be connected to the
a
n
ab
storage devices and record these paths. Use this information to
Disk array 2
change the SCSI addresses of these controllers in the nvramrc script.

Do not include the /disk or /sd trailing parts in the device paths.
7.
Edit the nvramrc script to change the scsi-initiator-id for the

host adapter on the first node.
For a list of nvramrc editor and nvedit keystroke commands, see
The nvramrc Editor and nvedit Keystroke Commands on
page B-11.
B-8

The following example sets the scsi-initiator-id to 6. The

OpenBoot PROM Monitor prints the line numbers (0:, 1:, and
so on).
nvedit
0: probe-all
1: cd /pci@1f,4000/scsi@4
3: device-end
4: cd /pci@1f,4000/scsi@4,1
6: device-end
7: install-console
8: banner <Control-C>
ok
s
) ha
y
m
om
c
Note Insert exactly one space after the first double

idemark and
csmquotation
u
e
before scsi-initiator-id.
G
t
@
n
b
ku tude
a
8. Store or discard the changes.
y
S
n ( thiscopy
i
d
The edits are done lon
a
temporary
of the nvramrc script. You
u se
a
can continue tom
edit this u
copy without risk. After you complete your
a
o
k
t
edits, save
g thenchanges.
e If you are not sure about the changes,
n
s
a
discard
them.
ab lice
b
u
To
lestore the changes, type:
k
b
a
a
l y sfer ok nvstore
u
d
ok
ab -tran
g
n non
a
To discard the changes, type:
b
a
ok nvquit
ok
9.
Verify the contents of the nvramrc script you created in step 7.

ok printenv nvramrc
nvramrc = probe-all
cd /pci@1f,4000/scsi@4
device-end
cd /pci@1f,4000/scsi@4,1
device-end
install-console
banner
ok

B-9

10. Instruct the OpenBoot PROM Monitor to use the nvramrc script.
ok setenv use-nvramrc? true

use-nvramrc? = true
ok
11. Without allowing the node to boot, power on the second node. If
Monitor tasks.
12. Verify that the scsi-initiator-id for each host adapter on the
second node is set to 7.
ok cd /pci@1f,4000/scsi@4
ok .properties
scsi-initiator-id
00000007
differential
isp-fcode
1.21 95/05/18
device_type
scsi
...
s
) ha
y
m
om
c
csm Guide
e
t and volume
@software,
13. Continue with the Solaris OS, Sun Cluster
n
b
e
u
k tud
management software installationatasks.
y
(
n this S
i
d
lu serefer to Sun Cluster 3.1 Installation
For software installation procedures,
a
Guide.
am to u
k
ng ense
a
b
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
a
no
ab
B-10

The nvramrc Editor and nvedit Keystroke Commands

The OpenBoot PROM Monitor builds its own device tree based on the
devices attached to the system when the boot sequence is invoked. The
OpenBoot PROM Monitor has a set of default aliases for the commonly
occurring devices in the system.
An nvramrc script contains a series of OpenBoot PROM commands that
are executed during the boot sequence. The procedures in this guide
assume that this script is empty. If your nvramrc script contains data,
add the entries to the end of the script. To edit an nvramrc script or merge
new lines in an nvramrc script, you must use nvedit editor and nvedit
keystroke commands.
s
) ha
y
m
Table B-1 and Table B-2 on page B-12 list useful nvramrcoeditor
m and
c
nvedit
nvedit keystroke commands, respectively. For an m
entire list of
e
d
s
i
c OpenBoot
editor and nvedit keystroke commands, refereto
u 3.x
G
t
@
Command Reference Manual.
b den
u
k
ya Stu
(
Table B-1 The nvramrc Editor
Commands
n
di e this
u
l
a us
Command Description
m
a
k e to
g
nvedit an Enters
nsthe nvramc editor. If the data remains in the
b ltemporary
e
c
a
buffer from a previous nvedit session,
i
b
e
u
l
resumes
editing
previous contents. Otherwise, reads the
k
b
a
a
r
y
contents of nvramrc into the temporary buffer and begin
ul nsfe
d
b
editing it. This command works on a buffer, and you can
a
a
r
t
g
save the contents of this buffer by using the nvstore
an non
ab
command.
nvstore
Copies the contents of the temporary buffer to nvramrc,

and discards the contents of the temporary buffer.
nvquit
Discards the contents of the temporary buffer, without

writing it to nvramrc. Prompts for confirmation.
nvrecover
Attempts to recover the content of the nvramrc if the

content was lost as a result of the execution of
set-defaults, then enters the nvramrc editors as with
nvedit. This command fails if nvedit is executed
between the time the content of nvramrc was lost and the
time the content of the nvramrc was executed.
nvrun
Executes the contents of the temporary buffer.

B-11

Table B-2 lists more useful nvedit commands.
Table B-2 The nvedit Keystroke Commands

Keystroke
Description
Â
Moves to the beginning of the line
^B
Moves backward one character
^C
Exits the script editor
^F
Moves forward one character
^K
Deletes until end of line
^L
Lists all lines
s
) ha
y
m
^N
Moves to the next line of the nvramrc editing buffer
om
c
m stays
Ô
Inserts a new line at the cursor positionsand
idonethe
c
u
e
G
current line
t
@
n
b
de editing buffer
kofuthe nvramrc
^P
Moves to the previous y
line
u
a
t
( is S
n line
i
^R
Replaces the current
d
lu se th
a
m tocharacter
u
Delete
Deletes
aprevious
k
ng eannew
se line at the cursor position and advances to
Return baInserts
a thelnext
ic line
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
tra
g
n
n
a
no
ab
B-12

Appendix C
Quorum Server
This appendix describes basic operation of the quorum server software.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
C-1
Quorum Server Software Installation
Quorum Server Software Installation

Before configuring a quorum-server quorum device into the cluster, you
obviously need to get the server software installed and rolling.
The quorum server packages are not part of the cluster software core
framework itself. The names of the packages are SUNWscqsr and
SUNWscqsu.
The quorum server software is one of the choices in the Java Enterprise
System (Java ES) installer, though you may prefer to just add the two
packages by hand.
s
) ha
y
m
om
c
The actual quorum server daemon, /usr/cluster/lib/sc/scqsd
e
csm Gserver,
uid
The utility to start, stop, query, and clear theequorum
tclqs)
@ name
n
/usr/cluster/bin/clquorumserver
(short
b
e
u
ak Stud
y
A single configuration file, /etc/scqsd/scqsd.conf
(
in this
d
u
An rc script (Solaris 9)
SMF service
10) to automatically
alordaemon
seat boot(Solaris
m
u
start the quorumaserver
time.
k e to
g
ban licens
a
NoteubRegardless
le of the OS that the quorum server is running on, you
k
b
a
a
always
use
the
y fer clquorumserver (clqs) command if you need to start and
l
u
nsthe daemon manually. This includes starting the daemon the first
bd trstop
a
a
- time after installation, without having to reboot. See page 4.
ang non
The package installation will install:
ab
An entry in /etc/services for the default port:

scqsd
C-2
9000/tcp
# Sun Cluster quorum server daemon

Configuring the Quorum Server
Configuring the Quorum Server

The configuration file is installed with a single quorum server instance
running on port 9000:
# cat /etc/scqsd/scqsd.conf
.
.
#
# To configure more instances, add commands to this file.
# Lines beginning with # are treated as comments and ignored.
#
/usr/cluster/lib/sc/scqsd -d /var/scqsd -p 9000
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
You can change the port number or data directory for the single instance,
or you could add more lines representing more daemons. There isnt a
particular practical reason to run multiple daemons, since the single
instance can handle as many clusters as you like. If you do run multiple
daemons, each must have a different port and directory. You can
optionally associate each instance with a text identifier (using -i
textid). If there is no text identifier (and there is none in the default line
already in the configuration file), the server instance will be identified by
its port number.
Quorum Server
C-3
Starting and Stopping a Quorum Server Daemon

Use clqs start and clqs stop. You either give a single daemon
identifier (port number or text identifier), or you can use the wildcard + to
indicate all server daemons in the configuration file:
# clqs start +
# ps -ef|grep scqsd
root 13108 13107 0 22:40:22 ?
-d /var/scqsd -p 9000
root 13107
1 0 22:40:22 ?
-d /var/scqsd -p 9000
0:00 /usr/cluster/lib/sc/scqsd
0:00 /usr/cluster/lib/sc/scqsd
as
h
Normally, you will not need to run these commands very much, since
the
)
y
m
boot/shutdown scripts will do the job for you.
om
c
m uide
csSide)
Displaying Quorum Server Data (on the Server
e
G
t
@
n
b
u the uquorum
de server machine)
k(on
The following is an example of querying
a
t
y
S can happily add access to
the state of the quorum server. Once
you
n (again,
s
i
i
d
h
t as you like:
the quorum server from as many
lu clusters
e
a
s
am to u
# clqs show
k
g ===
n9000
se
=== Quorum Server on port
a
n
b
e
a
lic
b
e
u
l
--- Cluster aorangecat
k rab(id 0x4480DAF7) Reservation --y
dul ransfe
Nodeab
ID:
2
t
g
0x4480daf700000002
n Reservation
on key:
a
n
b
a
---
Cluster orangecat (id 0x4480DAF7) Registrations ---
Node ID:
Registration key:
1
0x4480daf700000001
Node ID:
Registration key:
2
0x4480daf700000002
---
Cluster chicos (id 0x448DE82F) Reservation ---
Node ID:
Reservation key:
---
C-4
1
0x448de82f00000001
Cluster chicos (id 0x448DE82F) Registrations ---


Node ID:
Registration key:
---
Cluster food (id 0x448E2AE6) Reservation ---
Node ID:
Reservation key:
---
1
0x448de82f00000001
1
0x448e2ae600000001
Cluster food (id 0x448E2AE6) Registrations ---
Node ID:
Registration key:
1
0x448e2ae600000001
Node ID:
Registration key:
2
0x448e2ae600000002
s
) ha
y
m
m
o
c
are
For each cluster, the command displays the registrants
(nodesethat
m
d
s
i
c Gu holder. The
currently not fenced out of the cluster) and theereservation
t to ever perform
@
reservation holder would be equivalent to
the last node
n
b
e
u
any quorum fencing in the cluster;yit
aisk really
tuthedregistrants that are more
(
S
interesting.
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab
Quorum Server
C-5
Clearing a Cluster From the Quorum Server
Clearing a Cluster From the Quorum Server

If you remove a quorum server object cleanly from the cluster side, its
data will be removed cleanly from the quorum server state.
If you rebuilt a cluster without ever deleting old quorum information, you
may have to clean up the quorum server manually. In the previous
output, I have since actually rebuilt the cluster chicos and I am no longer
using the quorum server on behalf of that cluster. In order to clear out the
cluster information, obtain the cluster name and ID from the output of
clqs show, and run:
s
) ha
# clqs clear -c chicos -I 0x448DE82F 9000

The quorum server to be cleared must have been removed from the cluster.
Clearing a valid quorum server could compromise the cluster quorum.
Do you want to continue? (yes or no): yes
y
m
om
c
csm Guide
e
@ ent that is actually
b
Note Be careful not to clear quorum server
information
u
ud out a JBOD disk
ak Styanking
still in use. It is exactly equivalent to(y
physically
s words, the cluster itself
in tInhiother
device, which is a quorum disk
device.
d
u
l
e
using the quorum server as
a a quorum
s device will not find out until it is
m
u
a
too late!
o
k et
g
n
ba licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
an non
ab
C-6

Appendix D
Role-Based Access Control

This appendix provides a brief review of role-based access control
(RBAC).
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
D-1
Brief Review of RBAC Terminology

Because RBAC can be confusing, the following sections provide a review
of basic RBAC concepts to help you understand its integration into the
cluster.
Role
A role is like a user, except for the following:
You can not log in to it directly. You must log in as a user and then
use the su command to switch to the role.
s
) ha
y
m
om
c
e
A role is meant to replace an old-style privileged user,
dthe
smwhere
i
c
u
e
intention is that multiple administrators assume this identity
G when they
t
@
n
b
need to do privileged tasks. The advantage
is
that
you
must
log in as a
u ude
k
a
t
regular user and also know the password
for the role and be allowed to
n (y this S
i
assume the role to assume the d
identity.
lu se
a
uand shadow databases just like regular
athempasswd
o
Roles have entries in
k
t
ngroles
seassigned one of the pf-shells so they can run
users. Most often,
are
a
n
b
e
acommands,
privileged
lic but this is not a requirement.
b
e
u
l
k rab
a
y
ul nsfe
d
Authorization
b
a
tra
g
n
n
a
no
ab
Only certain users are allowed to switch to the role.
Authorizations are meant to convey certain rights to users or roles, but

authorizations are interpreted only by RBAC-aware applications. In other
words, having a particular authorization means nothing unless the
software you are running is specifically instrumented to check for your
authorization and grant you a particular right to do a task.
Authorizations have names like solaris.cluster.modify and
solaris.cluster.read. The names form a natural hierarchy, so that
assigning solaris.cluster.* has the meaning that you expect.
D-2

Assigning Authorizations
Authorizations can be assigned to users or roles, or they can be part of
rights profiles, which are collections of authorizations and command
privileges. The rights profiles could then be assigned to users or roles,
rather than the individual authorizations.
Command Privileges
Command privliges have been enhanced in the Solaris 10 OS:
Solaris 9 OS and Earlier
s
) ha
y
m
om with a
Command privileges allow users or roles to run certainccommands
e is
different real user ID (uid), effective user ID (euid),
smor both.
idThis
c
u
e
G bit, this RBAC
similar to turning the setuid bit on. But unlike
the setuid
t
@
n
b
mechanism lets you restrict which users
deare allowed to obtain
ku or roles
u
a
t
y
command privileges.
n ( this S
i
d
lu se
a
Starting With Solaris
am 10toOSu
k
ng enthe
seSolaris 9 command privileges feature to support
baaugments
Solarisa10
lic
b
fine
grainedle
privileges over a specific command. For example, a user can be
u
kgiven rthe
a
abprivilege to run a specific command and have root-like ability to
y
l
e
f
s all files regardless of their permissions, but still be denied the
bdu tranread
a
- privilege to write all files.
ang non
ab
Profiles
Profiles are collections of authorizations and command privileges.
Authorizations can be assigned directly to a user or role or through
inclusion in a profile assigned to the user or role.
Command privileges can be assigned only to a user or role through its
inclusion in a profile.

D-3
The Basic Solaris User Profile

The Basic Solaris User profile is a magical profile in that it is
automatically assigned to every user, and there is no way to remove it. If
there are authorizations included in the Basic Solaris User profile that you
want to remove from some users, the solution is to do the following:
Remove the offending authorizations from the Basic Solaris User

profile.
Create a custom profile that includes only the offending

authorizations, and assign that to the users to whom you want to
give those authorizations.
y
m
om
c
ide
csmprivileges
u
Figure 12-2 shows the relationships between command
e
G
authorizations, profiles, users, and roles. ub@ ent
ak Stud
y
(
in su this
users
roles
d
u
l
e
a
am to us
k
ng ense
a
b
profiles
a
lic
b
e
u
l
k rab
a
y
ul nsfe
d
b
a
auths
command privs
tra
g
n
n
o
a
n
s
) ha
RBAC Relationships
ab
Figure 12-2 RBAC Relationships
D-4

Simplified RBAC Authorizations in the Sun Cluster 3.2 Environment
Simplified RBAC Authorizations in the Sun Cluster 3.2

Environment
All the software is fully instrumented to understand RBAC
authorizations. The only authorizations you need (discussed in Module 5)
are:
solaris.cluster.read (part of Basic Solaris User)
solaris.cluster.admin
solaris.cluster.modify
s
) ha
RBAC in the Sun Cluster 3.1 Environment (for y

m
m
Backward Compatibility)
o
ab
c e
m
s
c Gauthorization
uid
e
The Sun Cluster 3.1 environment had a finer granularity
t but not required:
@
n
b
system. It is still recognized (for backward
compatibility)
e
u
ak Stud
y
(
solaris.cluster.system.{read,
in this admin, modify}
d
u
e denotes three separate authorizations:
This, just for easeaofl notation,
s
m
u
ka e to
solaris.cluster.system.read
g
ban licens
solaris.cluster.system.admin
a
b
u
le
k
b
solaris.cluster.system.modify
a
a
l y sfer
u
d
b tran solaris.cluster.node.{read,admin,modify}
a
solaris.cluster.quorum.{read.admin.modify}
ang non
solaris.cluster.device.{read,admin,modify}
solaris.cluster.transport.{read,admin, modify}
solaris.cluster.resource.{read,admin,modify}
solaris.cluster.network.{read,admin,modify}
solaris.cluster.install
solaris.cluster.appinstall
solaris.cluster.gui

D-5
Simplified RBAC Authorizations in the Sun Cluster 3.2 Environment
Assigning Sun Cluster Command Privileges to a User
The following example assigns Sun Cluster admin privileges (but not
modify privileges) directly to a user (rather than through a profile).
# usermod -A solaris.cluster.admin youzer
Assigning Sun Privileges to a Role

The following example assigns the Sun Cluster admin privileges to a new
role. Various users can be given permission to assume the role. When they
assume the role, they have the authorizations implied by the role.
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
# roleadd -u 9301 -d /export/home/clusoper \

-A solaris.cluster.admin clusoper
# usermod -R clusoper youzer
# usermod -R clusoper shmouzer
D-6

Appendix E
Logical Domains (LDoms) Guidelines

This appendix describes basic guidelines for configuring LDoms for usage
with Sun Cluster.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
E-1
LDom Support
LDom Support
LDoms are supported as Sun Cluster nodes, beginning with Sun Cluster
3.2 2/08 (Update 1). The following qualifications apply:
Sun Cluster is supported in guest LDoms and IO (service) LDoms
You can mix and match LDoms and physical machines (or hardware
domains) in the same cluster.
Multiple LDoms on the same physical server can be members of

different clusters
Multiple LDoms on the same physical server can be members of the

same cluster. The entire cluster consists of LDoms on the same
physical machine. This is supported but cannot provide true high
availability.
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
s
) ha
E-2

LDom Configuration for Sun Cluster

This discussion revolves around the typical deployment of a guest LDom
as a cluster node, as illustrated in Figure E-1. It is not important that both
are all nodes of the cluster are LDoms; the discussion centers on the
correct implementation of LDoms for use of cluster nodes. The fact that a
peer node in a cluster may also be an LDom or may be a physical machine
has no bearing whatsoever on the correct deployment of one particular
guest LDom.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
Figure E-1
Guest LDom as a cluster node
Guidelines for LDom Configuration

Here are the rules and guidelines for the various elements:

E-3
Public Network
You must configure the standard bridged networking in the I/O

domain; in this configuration the I/O domain uses a vnet adapter to
bridge the guest LDom (cluster node) out to the same subnet. An LDom
that is a cluster node cannot hide behind routing or NAT provided by its
I/O domain, because it needs to be in the same subnet space as another
node (whether that node be another LDom or a physical node) so that IP
addresses can properly fail over on the public network.
IPMP Groups
s
) ha
IPMP groups must be configured in the guest LDom that is serving as the
cluster node, regardless of whether groups are configured in the I/O
domain. The guest domains IPMP group is independent of any that you
configure in an I/O domain. IP address failover (LogicalHostname)
works only when IPMP is configured on the public network of the cluster
node.
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
Cluster Interconnect
in this
d
u
al use
m
You must configurektwo
domain physical adapters as virtual
a separate
to I/O
g
e
switches bridging
the
Sun
Cluster
node
(guest LDom) to the private
n ens
a
b
network. a
There is aicspecial flag mode=sc for these switches to indicate
lof traffic.
b
e
priority
handling
u
l
k
a erab
y
l
f
u Thenfollowing
s
d
example is run in the control domain. The name primary is
b
a domain name
a
r
t
the
for the I/O domain itself (which happens to be the
g
n
n
o
a
n
control domain). A virtual switch is added and then a virtual adapter is
ab
provided for the switch for the guest domain.
# ldm list
NAME STATE
FLAGS
CONS
VCPU MEMORY
UTIL UPTIME
primary active -n-cv- SP
4
4G
0.4% 12h 40m
myLDom1 active -n---- 5000
4
2G
0.2% 8h 3m
# ldm add-vsw net-dev=e1000g2 mode=sc privnet1 primary
# ldm add-vnet vnet2 privnet1 myLDom1
The guest domain will see an adapter named vnet2 which can be
configured as a cluster transport.
E-4

Boot Disk Device for a Guest LDom
The I/O domain can provide almost any kind of local virtual disk as the
boot disk for the guest domain (Sun Cluster node):
file
zvol
disk partition
entire physical disk
Just about the only gotcha, which this author came about unfortunately
and accidentally, is that if the guest domain is provided physical disk(s),
and both the I/O domain and guest domain use ZFS-root, make sure to
name the guests root pool differently from the I/O domain. If the names
conflict the I/O domain will be confused about which disks are in its root
pool when it boots. The guest domain will not be confused (it can only see
the disks/virtual disks that it has been provided), but its boot disk can
already be corrupted by the I/O domain if there is a ZFS pool name
conflict.
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
e Devices for a Guest LDom
al Data)
s
Shared Storage (Cluster
m
u
ka e to
g
For shared
andatacdevices
ns you must map entire physical disks/LUNs into
b
e
a
li This is the only way that quorum and data fencing is
thebguest domain.
e
u
l
k
a goingertoabwork properly. Do not create any other kind of virtual disk
y
l
f for Sun Cluster data:
u ndevice
s
d
b
a
tra # ldm add-vdsdev /dev/rdsk/c2t2d0 clustdata1@primary-vds0
g
n
n
no
aba
The guest domain (cluster node) will now have access to the disk.
These disks are given names such as c0d1 in the guest domain. As cluster
disks, they will work just fine. They will be assigned DID numbers just
like any other disk, be usable as quorum devices, and so on.
Running Volume Management in the Guest Domain

Outside of a cluster environment, it is conceivable to perform volume
management in an I/O domain and give to the guest domain virtual disks
that are already built on top of the I/O domains volumes.

E-5
When a guest domain is a Sun Cluster node, all volume management

(VxVM, SVM, or ZFS) must be performed in the guest domain. This is
required so that the Sun Cluster device group layer (or, in the case of ZFS,
the HAStoragePlus resource) can implement device group highavailability in the standard way.
Installing Sun Cluster in the Guest Domain

While adapter names (vnet1, vnet2) and disk names may look a little
different, there are not any special procedures for installing Sun Cluster in
a guest domain. Make sure you pick the correct devices for the private
network. You should not be able to tell whether a peer node in the same
cluster is a physical machine or another LDom on a different physical
machine.
s
) ha
y
m
m
o
c
Virtual Console for Guest LDom

e
m
d
s
i
ec t Gu
@
A control domain provides a virtual network
nservice (vnts) for
bterminal
e
u
d
k
the virtual console for the guest domain.
ya Stu
(
n
s
da iTCPeport,
histarting
t
u
l
Each guest LDom is provided
at 5000.
a us
m
ka e to
g
By default this
service is sconfigured to listen only on the localhost
banthislicconfiguration,
en
address. Using
you must connect using a traditional
a
b
e
u
l
network
connection
to
the
control
domain, log in there, and then issue
k
b
a
a
r
500x to access the guest domain virtual console.
l ytelnet
elocalhost
f
u
s
d
an
ab -trThis
g
is all fine and good and of no direct impact on Sun Cluster. You may,
n non
a
b
however,
like to configure the vntsd to listen directly on a remotely
a
accessible control domain IP address so you can get through to the guest
LDom virtual console directly rather than having to log into the control
domain.
You can configure the service by setting a property using svccfg. In this
example, we are setting the vntsd daemon to listen on a public IP
address:
# svccfg -s vntsd setprop vntsd/listen_addr=192.168.101.30
Now you can directly access the guest LDom virtual console externally
using telnet 192.168.101.30 500x.
E-6

Guidelines for Multiple LDoms on Same Machine (in Different Clusters)
Guidelines for Multiple LDoms on Same Machine (in

Different Clusters)
There are some special guidelines for this scenario, as illustrated below:
Multiple LDoms in different clusters can use the same physical

public network and private network adapters. You do not need to
create separate virtual switches in the I/O domain on behalf of each
LDom; you can use the same one and create different virtual
adapters on the virtual switch for each LDom. You must use a
different IP address space in each cluster for the private network
(cluster interconnect).
aass
h
You must use export different physical data disks to LDoms acting
)
y
m
nodes in different clusters.
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a

E-7
Guidelines for Multiple LDoms on Same Machine (in Different Clusters)
s
) ha
y
m
om
c
csm Guide
e
@ ent
b
u
ak Stud
y
(
in this
d
u
al use
m
ka e to
g
ban licens
a
b
u
le
k
b
a
a
l y sfer
u
d
ab -tran
g
n non
a
b
a
E-8


Sun Cluster 3.2 Student Guide

Încărcat de

Informații document

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Sun Cluster 3.2 Student Guide

Încărcat de

Drepturi de autor:

Formate disponibile

Unauthorized reproduction or distribution prohibited Copyright 2010, Oracle and/or its affiliates

Copyright 2009, Oracle and/or its affiliates. All rights reserved.

U.S. GOVERNMENT RIGHTS

This page intentionally left blank.

This page intentionally left blank.

About This Course ...........................................................Preface-xxiii

Exploring the Sun Cluster Software HA Framework................. 1-28

Accessing the Node Console on Servers with Virtual Consoles

Sun Cluster 3.2 Administration

Allocating a Different VLAN ID for the Private Network 3-46

Sun Cluster 3.2 Administration

Verifying DID Devices ........................................................... 4-53

Setting Other Cluster Properties........................................... 5-12

Viewing and Administering Disk Paths and Settings ................ 5-23

Sun Cluster 3.2 Administration

Task 1 Selecting Disk Drives .............................................. 6-42

Sun Cluster 3.2 Administration

Using Shared Diskset Disk Space .................................................. 7-20

Using ZFS With Sun Cluster Software............................................8-1

Building ZFS Pools and File Systems.............................................. 8-5

Managing the Public Network With IPMP ......................................

Two IPMP Groups on the Same Subnet .............................. 9-10

Sun Cluster 3.2 Administration

Integrating IPMP Into the Sun Cluster 3.2 Software Environment ..

Standard Resource Properties ............................................. 10-22

Configuring Scalable Services and Advanced Resource Group

Sun Cluster 3.2 Administration

Performing Supplemental Exercises for

Sun Cluster 3.2 Software............................................................... 12-1

Sun Cluster 3.2 Administration

Task 7 Configuring Oracle Virtual IPs............................ 12-36

Configuring Multi-Initiator SCSI ..................................................... B-1

Assigning Authorizations....................................................... D-3

Sun Cluster 3.2 Administration

About This Course

Use resources and resource groups to manage clustered applications

Use storage resources to manage both global and failover file

Configure a failover application (Network File System (NFS))

Configure a load-balanced application (Apache)

Configure applications in non-global zones

Configure HA-Oracle and Oracle Real Application Clusters (RAC)

Solaris Volume Manager

Sun Cluster 3.2 Administration

Topics Not Covered

Topics Not Covered

Solaris 10 Operating System (OS) Advanced Features or

About This Course

How Prepared Are You?

How Prepared Are You?

Can you explain virtual volume management terminology, such as

Can you perform basic Solaris OS administration tasks, such as using

Do you have prior experience with Sun hardware and the

Are you familiar with general computer hardware, electrostatic

Sun Cluster 3.2 Administration

Title, function, and job responsibility

Experience related to topics presented in this course

Reasons for enrolling in this course

Expectations for this course

About This Course

How to Use Course Materials

How to Use Course Materials

Goals You should be able to accomplish the goals after finishing

Objectives You should be able to accomplish the objectives after

Lecture The instructor will present information specific to the