APG40, Node, Change, APG40C/4

OPERATIONAL INSTRUCTION

Page 1 of 16

9/154 31-CRZ 222 05 Uen A

Ericsson AB 2006 - all rights reserved

APG40, Node, Change, APG40C/4

Contents
1
1.1

Introduction
Scope

2
2.1
2.2

Procedure
Prerequisites
Actions

3
3.1
3.2
3.3

Additional Information
Update AM FW
Delete customer data on the faulty node
Update RAID FW

Glossary

5
5.1
5.2

References
Operational Instructions
Printout Descriptions

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 2 of 16

1 Introduction
1.1 Scope
This Operational Instruction describes the procedure to change a node in the Adjunct Processor
(AP) cluster. This procedure must be performed on site.

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 3 of 16

2 Procedure
2.1 Prerequisites
2.1.1

Conditions

The following conditions must apply before this procedure can be completed:

The alarm AP FAULT is received or there is a work order received.

A technician is available on site.
The node to replace is faulty. The other node is the active working node.
The user account names and passwords that were set in the system at the time the backup
was taken are also available.
The user name and password valid for a user with administrative rights on the replaced
node.
The new node (spare part) must contain a correct LBB software installation, to be able to
restore the node.
A valid backup of the node to be replaced available on the APG or on DAT-Tape or on FTP
site.

2.1.2

Data

The following data should be known before starting this procedure:

The user name and password valid for a user with administrative rights on the active
node.
The hostname of the active node.
The public IP address of the active node.
The hostname of the faulty node.
The hostname of the replacement node.
The public IP address of the replacement node.
The subnet mask of the replacement node.
The gateway address of the replacement node.
2.1.3

Special Aids
A VGA monitor.
A keyboard with PS/2 or USB connector.
A mouse with PS/2 or USB connector.
An adapter to connect the keyboard and monitor to the same cable.
A grounding strap. To be used when exchanging the equipment.

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 4 of 16

2.2 Actions
Find out which Node to Replace
1. Find out which node to replace by examining the alarm printout from AP FAULT or the
work order.
Logon to Faulty Node
2. Connect to the faulty node by inserting the adapter to the mouse/keyboard input on the
AP. Connect the keyboard and the mouse to the adapter and connect the monitor to the
monitor input. Logon as a user with administrative rights.
Note:
If it is not possible to login to the faulty node then proceed with Step 6. In that case
stopping the cluster server is not possible as in steps from Step 3 to Step 5.
Stop Cluster Service
3. Stop ACS_PRC_EventAnalyser on the faulty node. Enter the following command in a
Command Prompt window:
cluster res ACS_PRC_EventAnalyser_<0/1> /off /wait
Note:
Select either '0' or '1' depending on the last letter of the faulty AP node's hostname.
If the last letter is 'a' select '0' and if the last letter is 'b' select '1'.
4. Stop acs_prc_clustercontrol on the faulty node. Enter the following command in a
Command Prompt window:
net stop acs_prc_clustercontrol
5. Stop Cluster Service on the faulty node. Enter the following command in a Command
Prompt window:
net stop clussvc
Note:
Confirm the question, cluster service will also stop "ACS_FCH_Server" service with
"yes".
Logon to Active Node
6. Connect to the active node by inserting the adapter to the mouse/keyboard input on the
AP. Connect the keyboard and the mouse to the adapter and connect the monitor to the
monitor input. Logon as a user with administrative rights.
Check Drive Status
7. Ensure that all the drives residing in the active node have the status optimal and also
check their capacity. Enter the following command in a Command Prompt window:
megarc -dispcfg -a0
Note:

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 5 of 16

All three of them should be in the state optimal. Under the column Capacity you can
see the capacity. Take a note of this value. If the drives on the active node have a
status other than optimal then this Operational Instruction cannot correct the
problem.
8. Are all the drives in the active node optimal?
Yes
Go to Step 9.
No
Go to Step 69.
9. Check which firmware version that is installed. Enter the following command in a
Command Prompt window:
megarc -ctlrinfo -a0
Note:
Under the column firmware version you can see the version. Note this down.
Note down Heartbeat IP addresses
10. Enter the following command in a Command Prompt window:
ipconfig
11. Note down the IP addresses of Heartbeat 1 and Heartbeat 2.
Bring Cluster into Safe State
12. Set cluster's hardware and software in a state that allows replacement of the faulty node.
Enter the following command in the active node:
fcc_save_to_remove other
Note:
Confirm the question with "yes".
Note:
If you were unable to perform the steps from Step 3 to Step 5 the command may
give an error message since the command will try to stop the cluster service on the
faulty node.
Note:
If the fcc_save_to_remove command hang (for more than 10 minutes) or does not
shut down the node properly then log on to the faulty (passive) node and shut it
down with START->Shut Down and select Shut down.
Disconnect Faulty Node
13.

Warning!

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 6 of 16

In case of a suspected hardware fault, the node shall be exchanged. It's not
allowed to exchange boards since that will violate any warranties.

Warning!
There is a special procedure for removing the SCSI cables. This procedure must
be followed to ensure that the SCSI bus will be terminated correctly.
Wait until the Manual Intervention Allowed (MIA) LED is lit and then disconnect the faulty
node. Disconnect the cables from the faulty node in the following order:
Table 1

1
2
3
4
5
6
7
8
9

Power cable
Public ethernet cable
IPN ethernet cable
External alarms cable
Alarm panel cable
Both Heartbeat ethernet cables
Both Serial cables (to CPU board and AM board)
Monitor cable
Keyboard/Mouse adapter cable

Note:
The SCSI cables should be removed in the correct order. See next step.
Disconnect SCSI Cables
14. Always disconnect the SCSI connectors of the active node before removing the SCSI
connectors of the powered-down (faulty) node. Disconnect the SCSI cables in the
following order:
Table 2

1
2
3
4

Disconnect
Disconnect
Disconnect
Disconnect

the
the
the
the

upper SCSI connector on the active node

lower SCSI connector on the active node
upper SCSI connector on the faulty node
lower SCSI connector on the faulty node

Replace Node
15. Replace the faulty node with the replacement node.
16. Connect the cables in the following order:
Table 3

1
2
3
4
5

Both Heartbeat cables

Public ethernet cable
Monitor cable
Keyboard/mouse adapter cable
Power cable

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 7 of 16

Logon to Replaced Node

17. Logon as a user with administrative rights.
Stop Cluster Service
18. Stop the Cluster Service on the replaced node. Enter the following commands in a
Command Prompt window:
net stop clussvc
Check Heartbeat IP addresses
19. Check the IP addresses of Heartbeat 1 and Heartbeat 2. Enter the following command in a
Command Prompt window:
ipconfig
20. Were the Heartbeat IP addresses of the replaced node conflict with the Heartbeat IP
addresses of the active node?
Yes
Go to Step 21.
No
Go to Step 22.
21. Change the IP addresses of the Heartbeat 1 to "192.168.202.3" and Heartbeat 2 to
"192.168.201.3".
Note:
Go to Start->all programs->accessories->communications->network connections.
Select the corresponding Heartbeat connection and double click on it. Click the
properties button in the general tab. Select Internet Protocol (TCP/IP) in the general
tab and double click on it.
Check Disk Capacity
22. Check which capacity the drives have on the replaced node. Enter the following command
in a Command Prompt window:
megarc -LogPhysInfo -a0
Note:
Under the column Capacity you can see the capacity. Take a note of this value.
23. Was the capacity of the disks equal to or bigger than the disks in the active node?
Yes
Go to Step 25.
No
Go to Step 24.
Note:
Refer to Step 7 for the Disk Capacity in the active node.

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 8 of 16

24. This spare part cannot be used. The disks in the spare part must have equal or bigger
capacity than the disks in the active node. Go to Step 69.
Unpack Backup to System Disk
25. On which media do you have a valid backup?
DAT Tape
Go to Step 26.
FTP site
Go to Step 34.
The shared disk M:\Images\node(a/b)
Go to Step 35.
26. In the active node, insert a tape containing the backup.
27. Check that the correct backup is inserted in the tape drive. Enter the following command in
a Command Prompt window:
emfls
28. Is the backup valid?
YES
Go to Step 29.
NO
Go to Step 25.
29. Copy the backup file to be restored from the tape containing the backup file to M:\Images.
Enter the following command in a Command Prompt window:
emfcopy -m FROMTAPE <BACKUPFILENAME>.zip M:\Images\node(a/b)
Note:
The backup should be valid for the node to be restored.
Note:
Select either a or b in (a/b) depending on the last letter of the replaced AP node's
hostname.
30. Enter the following command in a Command Prompt window:
emfinfo -n <NODENAME>
Note:
<NODENAME> is the name of the node on which the copy operation was initiated.
31. Is the copy operation finished?
YES
Go to Step 32.
NO
Go to Step 30.
32. When copying is finished, eject the tape.

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 9 of 16

33. Go to Step 35.

34. Use FTP to transfer the compressed backup. The backup should be transferred to the
directory M:\Images\node(a/b). The backup should be valid for the node to be restored.
Select either a or b in (a/b) depending on the last letter of the restored AP node's
hostname. See AP, File, Transfer and then return to Step 35 in this Operational Instruction.
35. Access the Replaced Node.
36. Enter the following command :
net use Z: \\<heartbeat net>\images/<user> <password>
Note:
<heartbeat net> is the IP address of either heartbeat 1 or Heartbeat 2 on the
working AP node where the shared directory 'images' is located. When prompted,
logon with a user with administrative rights.
Note:
Refer to Step 10 for the IP addresses of Heartbeat 1 and Heartbeat 2 on the active
node.
37. Enter the following command in a Command Prompt window:
cd /d c:\Program Files\pkware\pkzipc
38. Enter the following command:
pkzipc -extract Z:\node(a/b)\<NAMEOFBACKUP>.zip "program
files\ap\acs\bin\burrestore.exe"
Note:
The backup should be valid for the replaced node.
Note:
Select either a or b in (a/b) depending on the last letter of the replaced AP node's
hostname.
39. Enter the following command:
burrestore -a Z:\node(a/b)\<NAMEOFBACKUP>.zip -d D:\
Note:
The backup should be valid for the node to be restored.
Note:
Select either a or b in (a/b) depending on the last letter of the replaced AP node's
hostname.
Note:
Confirm the question with "yes". The prompt is not returned until the backup is
restored.
Check RAID FW
40. In a Command Prompt window, enter the following command in replaced node:
megarc -ctlrInfo -a0

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 10 of 16

41. Was the RAID firmware (FW) version in the replaced node equal to the one in the active
node?
YES
Go to Step 43.
NO
Go to Section 3.3 and then return to Step 42.
Note:
Refer to Step 9 for the RAID FW version in the active node.
42. In a Command Prompt window, enter the following command:
shutdown /r /t 1
Check AM FW
43. Check which AM FW version is used on the active node. Enter the following command in
the active node:
fcc_amversion own
44. Check which AM FW version is used on the replaced node. Enter the following command in
the replaced node:
fcc_amversion own
45. Are the AM FW versions identical on the active and on the replaced node?
YES
Go to Step 50.
NO
Go to Step 46.
Note:
Compare the printouts received in Step 43 and Step 44.
46. Enter the following command:
dir /d c:\Program Files\Force\am_services
47. Is the correct version available on the replaced node?
YES
Go to Section 3.1 and return to Step 50.
NO
Go to Step 48.
Note:
The file used when upgrading the AM FW should be named
<lbb_am_appl_x_y_r.bin> where 'x' and 'y' is the AM FW version and should
correspond to the printout received in Step 43.
48. Copy the correct 'lbb_am_appl_x_y_r.bin' file from the active node. Enter the following
command:

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 11 of 16

copy \\<active node hostname>\C$\Program

Files\Force\am_services\<lbb_am_appl_x_y_r.bin> d:\Program
Files\Force\am_services
49. Go to Section 3.1 and return to Step 50.
Restoring From Backup
50. Enter the following command in the replaced node:
burrestore -r <NAME> -s
Note:
The NAME is to recognize the present LBB OS.
51. Confirm the question with "yes".
Shutdown Replaced Node
52. In a Command Prompt window, enter the following command:
shutdown /s /t 1
Note:
Wait until "It is now safe to turn off your computer" message appears.
53. Disconnect the power cable from the replaced node.
Reconnect SCSI Cables
54. Always connect the SCSI connectors of the replaced (powered-down) node before
connecting the SCSI connectors of the active node. Connect the SCSI cables in the
following order:
Table 4

1
2
3
4

Connect
Connect
Connect
Connect

the
the
the
the

upper SCSI connector on the replaced node

lower SCSI connector on the replaced node
upper SCSI connector on the active node
lower SCSI connector on the active node

Note:
The SCSI cables must be crossconnected (i.e. SCSI 1 on one node to SCSI 0 on the
other. The same for SCSI 0 to SCSI 1).
Reconnect Other Cables
55. Reconnect the remaining cables in the following order:
Table 5

1
2
3
4
5

IPN ethernet cable

External alarms cable
Alarm panel cable
Both Serial cables (to CPU board and AM board)
Power cable

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 12 of 16

Logon to Replaced Node

56. Logon as a user with administrative rights.
Setup Services
57. Enter the following command in a Command Prompt window:
setupservices
58. Enter the following command in a Command Prompt window:
prcboot
Note:
The node will reboot and start up again.
59. Logon as a user with administrative rights.
Logon to Active Node
60. Connect to the active node by inserting the adapter to the mouse/keyboard input on the
AP. Connect the keyboard and the mouse to the adapter and connect the monitor to the
monitor input. Logon as a user with administrative rights.
Integrate Shared Mirrored Disk Drive
61. Integrate the shared mirrored disk drive. Enter the following command in a Command
Prompt window:
fcc_integrate other
62. Confirm action by entering 'y' followed by <ENTER>.
Note:
The replaced node should now join the cluster and the mirroring of the data disk will
start. The mirroring process will take around 1-2 hours. The lower figure is for idle
disks with no traffic to them and the higher figure is for maximum specified disk
traffic.
Check RAID Mirroring
63. Ensure that the mirroring of the RAID is still ongoing. Enter the following command:
megarc -dispcfg -a0
Note:
One of the RAID-1 should be in the status "rebuild" and the other two should be in
the status "rebuild/ pending".
Log Off from Active Node
64. Log off from the active node using START-> Log Off.

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 13 of 16

Logon to Replaced Node

65. Connect to the replaced node by inserting the adapter to the mouse/keyboard input on the
AP. Connect the keyboard and the mouse to the adapter and connect the monitor to the
monitor input. Logon as a user with administrative rights.
Wait for RAID Mirroring to Complete
66. Before proceeding with the next step, make sure that the mirroring of the RAID is
completed i.e. the RAID is optimal.
megarc -ctlrinfo -a0
Note:
The mirroring process will take around 1-2 hours. The lower figure is for idle disks
with no traffic to them and the higher figure is for maximum specified disk traffic.
Backup System
67. Go to Operational Instruction AP, System Backup and Verify, Initiate.
Perform the steps in that Operational Instruction, and return to Step 68 in this Operational
Instruction.
68. Go to Step 70.
Consult Next Level of Maintenance Support
69. Consult the next level of maintenance support. Further action is outside the scope of this
Operational Instruction.
Prepare Faulty Node
70. Perform the actions in Section 3.2 and then return to Step 71.
Job Completed
71. Make a report.
Note:
See Operational Instruction Report of Finished Work.
72. The job is completed.

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 14 of 16

3 Additional Information
3.1 Update AM FW
1. Start the AM configuration tool by using START->Programs->AM Configuration->AM
Configuration.
2. Select the tab Installation.
3. Press UpgradeAMgo.. button.
4. In the AMgo Upgrade window press the Browse button.
5. Select the file lbb_am_appl_x_y_r.bin (where 'x' and 'y' is the AM FW version).
6. Press the Open button.
7. Press the Start button.
8. Answer OK on the pop up menu and wait until the firmware is upgraded.
9. When the upgrade status has changed to Success the upgrade is finished.
10. Press OK to exit the AM configuration tool.
11. The AM firmware is now updated.

3.2 Delete customer data on the faulty node

Before the faulty node is sent for repair, the customer data must be deleted from the data disks.
If possible, perform these actions off site when this OPI has been completed successfully.
1. Connect only the keyboard and monitor to the faulty node, and then connect the power
cable.
2. Wait until Ctrl+M appears. Press Ctrl+M to activate the Raid Bios Configuration menu.
3. Use down arrow key to reach Initialize and press <Enter> to open submenu to display all
logical drives.
4. Use down arrow key to browse and select each drive by pressing <Space> key.
5. Press <F10> to initialize.
6. Select the "Yes" button with the tab key and press <ENTER> to confirm selection. Process
will take few minutes to complete.
7. Confirm the action with OK, the system reboot.

3.3 Update RAID FW

1. Enter the command in the Command Prompt window: cd /d c:\program
files\force\raid\firmware
2. Enter the command in the Command Prompt window: NTflash /r /f mflashw.exe

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 15 of 16

4 Glossary
-

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016

APG40, Node, Change, APG40C/4

Page 16 of 16

5 References
5.1 Operational Instructions
AP, System Backup and Verify, Initiate
Report of Finished Work

5.2 Manual Pages

burrestore (1m)
prcboot(1m)

5.3 Printout Descriptions

AP FAULT

file://C:\Program Files\ALEX\Lib\ALEX_TMP\1005\alex0285.htm

2/5/2016