Documente Academic
Documente Profesional
Documente Cultură
html
unixwerk Setu p a Tw o - N o d e C l u s t er w i th H A C M P
Contents
AIX
1. Introduction
BSD 2. Setup and Preparation
Storage setup
HP-UX Network setup
3. Installation
Linux Prerequisite filesets
HACMP Filesets
Solaris
4. Cluster Topology Configuration
Define the Cluster
Others
Define the Cluster Nodes
Un*x Define Cluster Sites
Define a Cluster Network
Topics Add a Communication Interface for Heartbeat
Add Persistent IP Addresses
Storage Configuration
Disk Heartbeat
5. Resource Group Configuration
Application Volume Groups
Guestbook Application Server
Contact Cluster Service Address
Define Resource Group(s)
Archive Create LVs and Filesystems for Applications
A. Failover Test
B. Disk Heartbeat Check
C. Useful Commands
D. clstat and snmp
E. Related Information
1. Introduction
This article describes how to setup a two-nodes-cluster with IBM's standard cluster solution for
AIX. Although the name has changed to Power HA with Version 5.5 and to Power HA System
Mirror with version 7 IBM's cluster solution is still widely known as HACMP. This article refers to
version 5.5.
Storage setup
The reason why we create a cluster is to make an application high available. Therefore we need
storage from two independent sites (read it as storage from two different datacenters). In this
article we have to sites: Datacenter1 and Datacenter2. Each filesystem will be mirrored over the
two sites. All storage has to be visible on both nodes.
In addition we need two (very small) LUNs for disk heartbeat. 512MB to 1G LUN size is sufficient.
Network setup
In our setup we have two nodes: barney and shakira. We need a boot address only used for
cluster intercommunication, a service address, and a persistent address which is equal to the
hostnames of our nodes. All cluster addresses have to be present in the /etc/hosts file on
both nodes:
node1+node2# vi /etc/hosts
#### HACMP
# Boot address
172.18.1.4 barneyboot
172.18.1.6 shakiraboot
# Service/Cluster address
10.111.111.70 haservice1
# Node/Persistent address
10.111.111.4 barney
10.111.111.6 shakira
####
3. Installation
bos.net.nfs.server
bos.clvm
rsct.compat.basic.hacmp
rsct.compat.clients.hacmp
| bos.clvm ALL
| + 6.1.1.1 Enhanced Concurrent Logical Volume Manager
| + 6.1.4.0 Enhanced Concurrent Logical Volume Manager
| > + 6.1.4.2 Enhanced Concurrent Logical Volume Manager
| rsct.compat.basic ALL
| + 2.5.4.0 RSCT Event Management Basic Function
| > + 2.5.4.0 RSCT Event Management Basic Function (HACMP/ES Support)
| + 2.5.4.0 RSCT Event Management Basic Function (PSSP Support)
| rsct.compat.clients ALL
| + 2.5.4.0 RSCT Event Management Client Function
| > + 2.5.4.0 RSCT Event Management Client Function (HACMP/ES Support)
node1+node2# cd /path/to/bffs
node1+node2# smitty install_latest
| > cluster.es.client ALL
| + 5.5.0.0 ES Client Libraries
| + 5.5.0.4 ES Client Libraries
| + 5.5.0.0 ES Client Runtime
| + 5.5.0.5 ES Client Runtime
| + 5.5.0.0 ES Client Utilities
| + 5.5.0.5 ES Client Utilities
| + 5.5.0.0 ES Communication Infrastructure
| + 5.5.0.5 ES Communication Infrastructure
| + 5.5.0.0 Web based Smit
| + 5.5.0.5 Web based Smit
| > cluster.es.server ALL
| + 5.5.0.0 ES Base Server Runtime
| + 5.5.0.6 ES Base Server Runtime
| + 5.5.0.0 ES Server Diags
| + 5.5.0.5 ES Server Diags
| + 5.5.0.0 ES Server Events
| + 5.5.0.6 ES Server Events
| + 5.5.0.0 ES Server Utilities
| + 5.5.0.6 ES Server Utilities
| + 5.5.0.0 ES Cluster Simulator
| + 5.5.0.4 ES Cluster Simulator
| + 5.5.0.0 ES Cluster Test Tool
| + 5.5.0.3 ES Cluster Test Tool
| + 5.5.0.0 ES Two-Node Configuration Assistant
| > cluster.es.cfs ALL
| + 5.5.0.0 ES Cluster File System Support
| + 5.5.0.4 ES Cluster File System Support
| > cluster.es.nfs ALL
| + 5.5.0.0 ES NFS Support
| + 5.5.0.1 ES NFS Support
| > cluster.es.cspoc ALL
| + 5.5.0.0 ES CSPOC Commands
| + 5.5.0.6 ES CSPOC Commands
| + 5.5.0.0 ES CSPOC Runtime Commands
| + 5.5.0.5 ES CSPOC Runtime Commands
| + 5.5.0.0 ES CSPOC dsh
| > cluster.license ALL
| + 5.5.0.0 HACMP Electronic License
| > cluster.man.en_US.es ALL
| + 5.5.0.0 ES Man Pages - U.S. English
| + 5.5.0.1 ES Man Pages - U.S. English
Note: In the above fileset list HACMP update filesets for SP6 are included. If you installed HACMP
from a base CD it's strongly recommended to update HACMP with the latest fixes. Base versions
of HACMP are not known to be excessively tested.
node1+node2# cd /path/to/update
node1+node2# smitty update_all
Basically the cluster configuration has to be done on only one of our nodes. Only the initial
definition and startup has to be done on both nodes. Please mind the command prompt in the
below commands. It indicates whether something has to be done on one node or on both nodes.
[Entry Fields]
* Cluster Name [Cluster1]
[Entry Fields]
* Node Name [barney]
Communication Path to Node [barneyboot] +
First site:
Add a Site
[Entry Fields]
* Site Name [Datacenter1] +
* Site Nodes barney +
* Dominance [Yes] +
* Backup Communications [none]
Second site:
Add a Site
[Entry Fields]
* Site Name [Datacenter2] +
* Site Nodes shakira +
* Dominance [No] +
* Backup Communications [none] +
The home node of our service shall be barney - that's why we set the Dominance to Yes for
barney and to No for shakira.
The network topology is used by HACMP for the heartbeat. First we configure heartbeat over
ethernet:
+--------------------------------------------------------------------------
| Select a Network Type
|
| Move cursor to desired item and press Enter.
|
| # Discovery last performed: (January 30 10:02)
| # Discovered IP-based Network Types
| ether
|
| # Discovered Serial Device Types
| rs232
|
| # Pre-defined IP-based Network Types
| XD_data
| XD_ip
| atm
| ether
| fddi
| hps
| ib
| token
|
| # Pre-defined Serial Device Types
| XD_rs232
| diskhb
| rs232
| tmscsi
| tmssa
|
| F1=Help F2=Refresh F3=Cancel
| F8=Image F10=Exit Enter=Do
| /=Find n=Find Next
+--------------------------------------------------------------------------
If you trust the automatic discovery select ether under "Discovered IP-based Network Types" -
if not select ether under "Pre-defined IP-based Network Types". The latter always work - so it
might be the better choice. In the next screen put in the correct netmask and activate the use of IP
aliases for IP takeover:
+--------------------------------------------------------------------------
| Select a Network Name
|
| Move cursor to desired item and press Enter.
|
| ALL
| net_ether_01
|
| F1=Help F2=Refresh F3=Cancel
| F8=Image F10=Exit Enter=Do
| /=Find n=Find Next
+--------------------------------------------------------------------------
[Entry Fields]
* IP Label/Address [barneyboot]
* Network Type ether
* Network Name net_ether_01
* Node Name [barney]
Network Interface [en8]
[Entry Fields]
* IP Label/Address [shakiraboot]
* Network Type ether
* Network Name net_ether_01
* Node Name [shakira]
Network Interface [en8]
[Entry Fields]
* Node Name barney
* Network Name [net_ether_01]
* Node IP Label/Address [barney]
Prefix Length []
We miss a default route here. Since the persistent IP is defined within HACMP there is no default
route defined in the ODM. However, after a reboot the system comes up with boot and persistent
address. So we define a default route on both nodes:
This sets 10.111.111.1 as the default gateway. We will activate the route later with the cluster
start. In normal operation you don't have to touch the default route anymore.
Storage Configuration
First we set PVIDs on every LUN we want to use for HACMP and run cfgmgr on the other node.
On node2 we have to remove the hdisks first an run cfgmgr again. Now we see the same PVIDs
as on node1:
Now we connect the LUNs to our cluster sites. For every LUN do the following:
+-------------------------------------------------------------------------
| Site Names
|
| Move cursor to desired item and press Enter.
|
| Datacenter1
| Datacenter2
|
| F1=Help F2=Refresh F3=Cancel
| F8=Image F10=Exit Enter=Do
| /=Find n=Find Next
+-------------------------------------------------------------------------
Select site Datacenter1 and select all LUNs located there in the next screen:
[Entry Fields]
* Site Name Datacenter1
* Disks PVID
<F4> gives you a list of all LUNs configured for HACMP - select the ones for site Datacenter1:
+-------------------------------------------------------------------------
| Disks PVID
|
| Move cursor to desired item and press F7.
| ONE OR MORE items can be selected.
| Press Enter AFTER making all selections.
|
| > 00f6418384f345d0 ( hdisk1 on all selected nodes )
| > 00f6418384f34621 ( hdisk2 on all selected nodes )
| > 00f6418384f3466c ( hdisk3 on all selected nodes )
| > 00f6418384f346b0 ( hdisk4 on all selected nodes )
| > 00f6418384f346f2 ( hdisk5 on all selected nodes )
| > 00f6418384f34739 ( hdisk11 on all selected nodes )
| 00f6418384f44fca ( hdisk6 on all selected nodes )
| 00f6418384f45015 ( hdisk7 on all selected nodes )
| 00f6418384f45054 ( hdisk8 on all selected nodes )
| 00f6418384f4508f ( hdisk9 on all selected nodes )
| 00f6418384f450ca ( hdisk10 on all selected nodes )
| 00f6418384f450ff ( hdisk12 on all selected nodes )
|
| F1=Help F2=Refresh F3=Cancel
| F7=Select F8=Image F10=Exit
| Enter=Do /=Find n=Find Next
+-------------------------------------------------------------------------
Disk Heartbeat
Two of our LUNs are dedicated to disk heartbeat. Typically you use small LUN sizes here. If
you're not sure which LUNs are the heartbeat LUNs check with " bootinfo -s hdisk<X> ".
To protect the LUNs for disk heartbeat we create volume groups for them - a separate VG for
each LUN:
+-------------------------------------------------------------------------
| Node Names
|
| Move cursor to desired item and press F7.
| ONE OR MORE items can be selected.
| Press Enter AFTER making all selections.
|
| > barney
| > shakira
|
| F1=Help F2=Refresh F3=Cancel
| F7=Select F8=Image F10=Exit
| Enter=Do /=Find n=Find Next
+-------------------------------------------------------------------------
+-------------------------------------------------------------------------
| Physical Volume Names
|
| Move cursor to desired item and press F7.
| ONE OR MORE items can be selected.
| Press Enter AFTER making all selections.
|
| 00f6418384f345d0 ( hdisk1 on all selected nodes )
| 00f6418384f34621 ( hdisk2 on all selected nodes )
| 00f6418384f3466c ( hdisk3 on all selected nodes )
| 00f6418384f346b0 ( hdisk4 on all selected nodes )
| 00f6418384f346f2 ( hdisk5 on all selected nodes )
| 00f6418384f34739 ( hdisk11 on all selected nodes )
| 00f6418384f44fca ( hdisk6 on all selected nodes )
| 00f6418384f45015 ( hdisk7 on all selected nodes )
| 00f6418384f45054 ( hdisk8 on all selected nodes )
| 00f6418384f4508f ( hdisk9 on all selected nodes )
| 00f6418384f450ca ( hdisk10 on all selected nodes )
| 00f6418384f450ff ( hdisk12 on all selected nodes )
|
| F1=Help F2=Refresh F3=Cancel
| F7=Select F8=Image F10=Exit
| Enter=Do /=Find n=Find Next
+-------------------------------------------------------------------------
We select the small LUN from Datacenter1 and fill the empty fields in the next screen:
[Entry Fields]
Node Names barney, shakira
PVID 00f6418384f34739
VOLUME GROUP name [hacmp_hb1]
Physical partition SIZE in megabytes 4
Volume group MAJOR NUMBER [38]
Enhanced Concurrent Mode true
Enable Cross-Site LVM Mirroring Verification false
Warning:
Changing the volume group major number may result
in the command being unable to execute
successfully on a node that does not have the
major number currently available. Please check
for a commonly available major number on all nodes|
before changing this setting.
The same procedure has to be done for the second disk heartbeat LUN. We call the second
volume group " hacmp_hb2 ".
Before we go on with the disk heartbeat configuration we let HACMP discover first...
+--------------------------------------------------------------------------
| Select Point-to-Point Pair of Discovered Communication Devices to Add
|
| Move cursor to desired item and press F7. Use arrow keys to scroll.
| ONE OR MORE items can be selected.
| Press Enter AFTER making all selections.
|
| # Node Device Device Path Pvid
| barney tty0 /dev/tty0
| shakira tty0 /dev/tty0
| > barney hdisk11 /dev/hdisk11 00f6418384f34739
| barney hdisk12 /dev/hdisk12 00f6418384f450ff
| > shakira hdisk11 /dev/hdisk11 00f6418384f34739
| shakira hdisk12 /dev/hdisk12 00f6418384f450ff
|
| F1=Help F2=Refresh F3=Cancel
| F7=Select F8=Image F10=Exit
| Enter=Do /=Find n=Find Next
+--------------------------------------------------------------------------
We choose the first pair of disks and repeat the procedure for the second pair.
+--------------------------------------------------------------------------
| Node Names
|
| Move cursor to desired item and press F7.
Select both nodes as shown in the screen above and select the hdisks you need in the next
screen. Choose one set of disks from Datacenter1 and one set of disks from Datacenter2.
Unfortunately in this screen the location is not indicated. In this example we just select all
available disks:
+--------------------------------------------------------------------------
| Physical Volume Names
|
| Move cursor to desired item and press F7.
| ONE OR MORE items can be selected.
| Press Enter AFTER making all selections.
|
| > 00f6418384f345d0 ( hdisk1 on all selected nodes )
| > 00f6418384f34621 ( hdisk2 on all selected nodes )
| > 00f6418384f3466c ( hdisk3 on all selected nodes )
| > 00f6418384f346b0 ( hdisk4 on all selected nodes )
| > 00f6418384f346f2 ( hdisk5 on all selected nodes )
| > 00f6418384f44fca ( hdisk6 on all selected nodes )
| > 00f6418384f45015 ( hdisk7 on all selected nodes )
| > 00f6418384f45054 ( hdisk8 on all selected nodes )
| > 00f6418384f4508f ( hdisk9 on all selected nodes )
| > 00f6418384f450ca ( hdisk10 on all selected nodes )
|
| F1=Help F2=Refresh F3=Cancel
| F7=Select F8=Image F10=Exit
| Enter=Do /=Find n=Find Next
+--------------------------------------------------------------------------
The next screen asks for the type of volume group. These days scalable VGs seem to be the best
choice:
+--------------------------------------------------------------------------
| Volume Group Type
|
| Move cursor to desired item and press Enter.
|
| Legacy
| Original
| Big
| Scalable
|
| F1=Help F2=Refresh F3=Cancel
| F8=Image F10=Exit Enter=Do
After we selected disks and VG type we choose a name for the volume group:
[Entry Fields]
Node Names barney, shakira
PVID 00f6418384f345d0 00f6>
VOLUME GROUP name [appl01vg]
Physical partition SIZE in megabytes 128
Volume group MAJOR NUMBER [42]
Enable Cross-Site LVM Mirroring Verification true
Warning:
Changing the volume group major number may result
in the command being unable to execute
successfully on a node that does not have the
major number currently available. Please check
for a commonly available major number on all nodes
before changing this setting.
After confirming with <ENTER> we are done with the VG and can go on with the
Application Server
For the application servers we first need application start and stop scripts. The scripts are usually
provided by the application owners and should match at least two conditions:
Once the scripts are in place we can configure the application server:
In the above example the start/stop scripts are stored in a folder /etc/hacmp. But you can
place them anywhere in the local filesystem tree. Don't place them on shared filesystems! Since
the scripts are local we have to copy them over to the other node:
Remember: we already defined the service address in /etc/hosts with the initial network
setup.
+--------------------------------------------------------------------------
| Select a Service IP Label/Address type
|
| Move cursor to desired item and press Enter.
|
| Configurable on Multiple Nodes
| Bound to a Single Node
|
| F1=Help F2=Refresh F3=Cancel
| F8=Image F10=Exit Enter=Do
| /=Find n=Find Next
+--------------------------------------------------------------------------
As the said before, the service address needs to move with the application - so we select
"Configurable on Multiple Nodes" here.
[Entry Fields]
* IP Label/Address haservice1
* Network Name net_ether_01
Alternate HW Address to accompany IP Label/Address []
[Entry Fields]
* Resource Group Name [RG_01]
In this panel we initially define name of the resource group (RG_01 here). The policy definitions
on the bottom are typical to two-node clusters. But you could choose different values here. For
HACMP insiders: The above setup is the classic cascading setup.
+--------------------------------------------------------------------------
| Change/Show Resources and Attributes for a Resource Group
|
| Move cursor to desired item and press Enter.
|
| RG_01
|
| F1=Help F2=Refresh F3=Cancel
| F8=Image F10=Exit Enter=Do
| /=Find n=Find Next
+--------------------------------------------------------------------------
In the above smit panel we assign our service address and the application server we just created (
Application Server) and set the varyon policy to forced.
[Entry Fields]
* Verify, Synchronize or Both [Both]
* Automatically correct errors found during [No]
verification?
At this point the cluster is synchronized and in a consistent state. Both nodes have the same
information about the cluster setup.
[Entry Fields]
* Start now, on system restart or both now
Start Cluster Services on these nodes [barney]
BROADCAST message at startup? true
Startup Cluster Information Daemon? True
Reacquire resources after forced down ? false
Ignore verification errors? false
Automatically correct errors found during Interactively
cluster start?
To activate the route we defined earlier ( Add Persistent IP Addresses) we issue the command
Once the cluster is up we go on with creating LVs and filesystems. If you don't want to use inline
jfs2 logs, first a log device has to be created (if you don't do this a log LV called loglv00 will be
automatically created with the first filesystem). The procedure to create a log LV is the same as for
a regular filesystem with two exceptions:
Refer to the next section on how to create the LV applvg01_jfs2log and remember to set
the right Logical volume TYPE.
Now we are ready to create the application filesystems. The below example shows how to create
one filesystem. Repeat the steps until all filesystems are setup. Remember to create a jfs2log for
each volume group first (if you don't use inline logs).
+--------------------------------------------------------------------------
| Shared Volume Group Names
|
| Move cursor to desired item and press Enter. Use arrow keys to scroll.
|
| #Resource Group Volume Group
| RG_01 appl01vg
|
| F1=Help F2=Refresh F3=Cancel
| F8=Image F10=Exit Enter=Do
| /=Find n=Find Next
+--------------------------------------------------------------------------
Select one pair of disks from both sites - mark with <F7>:
+--------------------------------------------------------------------------
| Physical Volume Names
|
| Move cursor to desired item and press F7.
| ONE OR MORE items can be selected.
| Press Enter AFTER making all selections.
|
| Auto-select
| > barney hdisk1 Datacenter1
| barney hdisk2 Datacenter1
| barney hdisk3 Datacenter1
| barney hdisk4 Datacenter1
| barney hdisk5 Datacenter1
| > barney hdisk6 Datacenter2
| barney hdisk7 Datacenter2
| barney hdisk8 Datacenter2
| barney hdisk9 Datacenter2
| barney hdisk10 Datacenter2
|
| F1=Help F2=Refresh F3=Cancel
| F7=Select F8=Image F10=Exit
|
+--------------------------------------------------------------------------
Warning: Don't use Auto-select here - although we assigned LUNs to sites it's not guaranteed
that CSPOC selects LUNs from different sites!
[Entry Fields]
Resource Group Name RG_01
VOLUME GROUP name appl01vg
Reference node barney
* Number of LOGICAL PARTITIONS [80]
PHYSICAL VOLUME names hdisk1 hdisk6
Logical volume NAME [applv01]
Logical volume TYPE [jfs2]
+--------------------------------------------------------------------------
| Logical Volume Names
|
| Move cursor to desired item and press Enter.
|
| applv01 barney,shakira
|
| F1=Help F2=Refresh F3=Cancel
| F8=Image F10=Exit Enter=Do
| /=Find n=Find Next
+--------------------------------------------------------------------------
[Entry Fields]
Node Names barney,shakira
LOGICAL VOLUME name applv01
* MOUNT POINT [/appl01/fs01]
PERMISSIONS read/write
Mount OPTIONS []
Block Size (bytes) 4096
Inline Log? no
Repeat the steps until all filesystems are setup. Our cluster is ready for use now.
Appendix
A. Failover Test
1. Manual Failover
The manual failover is the most important test for a cluster configuration. This test can be invoked
on one node by
[Entry Fields]
* Stop now, on system restart or both now
Stop Cluster Services on these nodes [barney]
BROADCAST cluster shutdown? true
* Select an Action on Resource Groups Move Resource Groups
When stopping the cluster on node 1 the first thing executed is the cluster stop script. It brings
down the applications and unmounts all application filesystems. If your application stop script is
not able to stop all application processes some filesystems can't be unmounted and the failover
fails.
When all resources are down on node 1 HACMP starts to bring up all resources on node 2. The
application start script is the last thing hacmp does.
Check that your application is working properly and that all clients can connect. If so the first
phase of the failover test is completed.
2. Manual Failback
Switch the resources back to the home node. Again check if everything is fine.
3. Automatic Failover
This test simulates a hardware failure on the active node. The easiest way to simulate is to issue
the command
node1# halt -q
Receive Mode:
Waiting for response . . .
Magic number = 0x87654321
Magic number = 0x87654321
Magic number = 0x87654321
Magic number = 0x87654321
Magic number = 0x87654321
Magic number = 0x87654321
Magic number = 0x87654321
Magic number = 0x87654321
Link operating normally
and on the other node we set the same disk to transmit mode...
Transmit Mode:
Magic number = 0x87654321
Detected remote utility in receive mode. Waiting for response . . .
Magic number = 0x87654321
Magic number = 0x87654321
Link operating normally
The last line in the above output indicates that the disk heartbeat is working properly.
C. Useful Commands
This is only a brief and selective list of commands that might be useful when working with HACMP
# /usr/sbin/cluster/utilities/clRGinfo
-----------------------------------------------------------------------
Group Name State Node
-----------------------------------------------------------------------
RG_01 ONLINE barney
OFFLINE shakira
# smitty clstop
# smitty clstart
# /usr/sbin/cluster/clstat -a
# /usr/sbin/cluster/utilities/cllistlogs
/var/hacmp/log/hacmp.out
/var/hacmp/log/hacmp.out.1
/var/hacmp/log/hacmp.out.2
# stopsrc -s snmpd
# startsrc -s snmpd
E. Related Information
AIX 6.1 Information Center > PowerHA SystemMirror > HACMP Version 6.1
IBM Redbook: PowerHA for AIX Cookbook
Certification Study Guide: HACMP for AIX
Also on unixwerk
HACMP: Cluster Commandline
How to Add a New VG to an Active HACMP Resource Group
How to Add a New Resource Group to an Active Cluster
How to Add a Node to an HACMP Cluster
How to Remove a Node from an HACMP Cluster