Software Patching Guide: Hig1800 Element Management System (Ems)

hiG1800 Element Management System (EMS)
Software Patching Guide
Provides software patching instructions for a

Nokia Siemens Networks hiG1800 Gateway GenView EMS
hiG1800 Element Management System Software Patching Guide

A30828-X1122-A462-01-76P1
Release 7.2
Rev 4
12/22/09
The information in this document is subject to change without notice and describes only the
product defined in the introduction of this documentation. This documentation is intended for
the use of Nokia Siemens Networks customers only for the purposes of the agreement under
which the document is submitted, and no part of it may be used, reproduced, modified or
transmitted in any form or means without the prior written permission of Nokia Siemens
Networks. The documentation has been prepared to be used by professional and properly
trained personnel, and the customer assumes full responsibility when using it. Nokia Siemens
Networks welcomes customer comments as part of the process of continuous development
and improvement of the documentation.
The information or statements given in this documentation concerning the suitability, capacity,
or performance of the mentioned hardware or software products are given "as is" and all
liability arising in connection with such hardware or software products shall be defined
conclusively and finally in a separate agreement between Nokia Siemens Networks and the
customer. However, Nokia Siemens Networks has made all reasonable efforts to ensure that
the instructions contained in the document are adequate and free of material errors and
omissions. Nokia Siemens Networks will, if deemed necessary by Nokia Siemens Networks,
explain issues which may not be covered by the document.
Nokia Siemens Networks will correct errors in this documentation as soon as possible. IN NO
EVENT WILL NOKIA SIEMENS NETWORKS BE LIABLE FOR ERRORS IN THIS
DOCUMENTATION OR FOR ANY DAMAGES, INCLUDING BUT NOT LIMITED TO
SPECIAL, DIRECT, INDIRECT, INCIDENTAL OR CONSEQUENTIAL OR ANY LOSSES,
SUCH AS BUT NOT LIMITED TO LOSS OF PROFIT, REVENUE, BUSINESS
INTERRUPTION, BUSINESS OPPORTUNITY OR DATA,THAT MAY ARISE FROM THE USE
OF THIS DOCUMENT OR THE INFORMATION IN IT.
This documentation and the product it describes are considered protected by copyrights and
other intellectual property rights according to the applicable laws. The wave logo is a
trademark of Nokia Siemens Networks Oy. Nokia is a registered trademark of Nokia
Corporation. Siemens is a registered trademark of Siemens AG. Other product names
mentioned in this document may be trademarks of their respective owners, and they are
mentioned for identification purposes only.
Copyright © Nokia Siemens Networks 2009. All rights reserved.
2 hiG1800 Element Management System Software Patching Guide

A30828-X1122-A462-01-76P1 12/22/09
Contents
1 Introduction .................................................................................................................. 5
1.1 Purpose and Scope........................................................................................................... 5
1.2 Acronyms .......................................................................................................................... 5
1.3 Terminology ...................................................................................................................... 5
2 EMS/hiG1800 Patching Overview................................................................................ 6
2.1 TYPES OF APPLICATION BINARIES .............................................................................. 6
2.2 EMS Active/Standby Applications ..................................................................................... 6
2.3 EMS Load Sharing Applications........................................................................................ 6
2.4 EMS Platform Applications................................................................................................ 7
2.5 hiG1800 Platform Applications.......................................................................................... 7
2.6 Patch Bundle Naming Explanation.................................................................................... 7
2.7 hiG1800 Patch Naming Explanation ................................................................................ 7
3 Patch Bundle Installation Prerequisites ..................................................................... 8
3.1 Prerequisites Overview EMS BUNDLE ............................................................................. 8
3.2 Pre-Patch Checks ............................................................................................................. 9
3.3 Copy New patching files from FTP server....................................................................... 12
3.4 Run CHECK DATABASE scripts .................................................................................... 13
3.5 Backup Current msfswcontrolfile..................................................................................... 14
3.6 Verify current patch level on EMS ................................................................................... 14
3.7 Verify current patch level on hiG1800 ............................................................................. 15
3.8 Backup current database on both EMS nodes................................................................ 15
3.9 Install Tekbase 1.8 (or Latest Version) ........................................................................... 15
4 Patching EMS Server ................................................................................................. 17
4.1 Applying Patches To EMS .............................................................................................. 17
4.1.1 Verify patches are present on EMS ...................................................................... 17
4.1.1.1 Apply Patch Bundles to EMS Nodes ....................................................... 18
5 Patching the hiG1800 ................................................................................................. 19
5.1 Applying hiG1800 Patches To EMS Nodes .................................................................... 19
5.1.1 Check Node Link Status ....................................................................................... 19
5.1.2 Generating the MsfSwControlFile ......................................................................... 20
5.1.3 Predownload MsfSwControlFile and New hiG1800 Patch Load.......................... 21
5.1.4 Patching the Standby Packet and Control (PAC) card ......................................... 22
5.1.5 Monitor Call rate with RTHM................................................................................. 23
5.1.6 Patching VS (Voice Server) Cards (If Applicable)................................................. 23
5.1.7 Signaling Gateway (SG) Server Cards (If Applicable) .......................................... 24
5.1.8 Patching Standby GigE-2 Card (If Applicable)...................................................... 24
5.1.9 Patching ES CARDs (If Applicable) ...................................................................... 25
5.1.10 Load Standby SST Card with DSP/PCM Binaries (If applicable).......................... 26
5.1.11 Patching Standby QOC3 Cards (If Applicable) ..................................................... 27
5.1.12 Have Customer Execute Critical Call-Through Test Plan ..................................... 29
5.1.13 Upgrade Remaining Half of (hiG1800) (If Decision to Proceed) ........................... 29
5.1.14 Patching the Newly Standby QOC3 Cards (If Applicable) .................................... 29
5.1.15 Load Newly Standby SST Card with DSP/PCM Binaries (If applicable)............... 31
5.1.16 Patching Newly Standby GigE-2 Card (If Applicable) ........................................... 32
hiG1800 Element Management System Software Patching Guide 3

A30828-X1122-A462-01-76P1 12/22/09
5.1.17 Patching the Newly Standby PAC card................................................................. 33
6 Activate New Patch on STANDBY EMS NODE (2) ................................................... 35
6.1 Apply Patch To Standby Node ........................................................................................ 35
7 Switchover Preparation ............................................................................................. 38
7.1 Switchover Standby EMS Node (2) To Active................................................................. 38
7.2 Critical Call Test Plan...................................................................................................... 38
8 Activate New Patch on Standby EMS Node (1)........................................................ 39
8.1 Apply Patch To Standby EMS Node ............................................................................... 39
9 Switchover Preparation ............................................................................................. 42
9.1 Switchover Standby Node (1) to Active........................................................................... 42
9.2 Critical Call Test Plan...................................................................................................... 42
10 Post-Patch checks ..................................................................................................... 44
10.1 Verify Alarms................................................................................................................... 44
10.2 Run Check DB Health ..................................................................................................... 44
10.3 Run Database Backup .................................................................................................... 45
10.4 Run Switchcheck Script .................................................................................................. 45
10.5 EMS Post Checks ........................................................................................................... 45
10.6 Review Summaries ......................................................................................................... 47

A30828-X1122-A462-01-76P1 12/22/09
1 Introduction
1.1 Purpose and Scope

This document is used to perform a patch application on the EMS/hiG1800 System.
1.2 Acronyms
Acronym Description
SWOPs Software Operations
hiG1800 NSN Media Gateway Distributed Switching Solution
TAC Technical Assistance Center
CCC Customer Call Center
MOP Method of Procedure
CSR Customer Service Request
1.3 Terminology
Term Definition
Software The act of upgrading a switching product from one major release to another..
Upgrade
Patching Activity The act of patching software on a switching product.

A30828-X1122-A462-01-76P1 12/22/09
2 EMS/hiG1800 Patching Overview
The EMS/hiG1800 is patched essentially by replacing the individual program’s binaries with new
versions. Since every function on the EMS/hiG1800 either has a redundant Hot Standby or is
load shared across multiple nodes, once the patched binaries are put into place, the individual
application restart is required to activate the new binaries, so as to not require downtime.
Patch removal is simply a reversal of what is done during patching. Instead of utilizing the new
application binaries from the patch, the pre-requisite (the previous patch) is utilized to get the
replacement application binary from. Deactivation of a removed patch follows the same
procedures as patch application.
The following sections outline the different subsystem patching methodology as well as the
technical means by which they are accomplished.
2.1 TYPES OF APPLICATION BINARIES

• EMS ACTIVE/STANDBY APPLICATIONS
• EMS LOAD SHARING APPLICATIONS
• EMS PLATFORM APPLICATIONS
• EMS MISCELLANEOUS APPLICATIONS
• hiG1800 PLATFORM APPLICATIONS
2.2 EMS Active/Standby Applications

EMS Active/Standby applications run in the EMS complex as a pair regardless of the number of
EMS’s in the complex. Examples of Active/Standby Applications are: CsSp (Subscriber Profile
Manager), SgwIsup (Signaling Gateway ISUP Application) and EMSCfg (EMS Management
Application).
The base application binaries are initially placed in the /opt/SANTone/msc/active/bin directory
upon initial installation of the SANTone package. Patches to these binaries are placed in the
directory: /Santera/msc/active/patch. Binaries have names, such as SgwIsup_07023037, and a
UNIX symbolic link with the application name pointing to the patched file. Running applications
can be replaced now with a new binary prior to an application restarting (i.e., the patching tools
place a new patch in the /Santera/msc/active/patch directory called SgwIsup_07023039, then
link SgwIsup to it. Once this occurs, the standby application should be identified and restarted
first using the EMS GUI. Once it has successfully restarted, the active side will restart causing
the old standby to become active. At this point, the new version of the application is running.
2.3 EMS Load Sharing Applications

EMS load sharing applications are applications that run in the EMS complex individually on
every EMS node in the EMS complex. Like Active/Standby Applications, base application
binaries are initially placed in the /opt/SANTone/msc/active/bin directory upon initial installation
of the SANTone package. Patches to these binaries are placed in the directory
/Santera/msc/active/patch. Activation of these binaries is accomplished exactly as
A30828-X1122-A462-01-76P1 12/22/09
Active/Standby applications with the exception; the user simply restarts the applications one
node at a time for every EMS in the complex.
2.4 EMS Platform Applications

EMS platform applications are applications that run in the EMS complex individually on every
EMS node in the EMS complex. Examples of the Platform Applications are BsPm (Base
Platform Manager) and BsDm (Base Platform Distribution Manager). The base platform
applications in practice act as an EMS load sharing application in the sense that they run
independent from any mate. However, from a patching perspective they are special and require
special application instructions. To activate patches to these applications requires the
EMS/hiG1800 platform be completely stopped and restarted. However, due to the fact that
every application either has a mate on another EMS in the complex or is load shared across
another EMS in the complex, activation does not cause an impact to live traffic.
2.5 hiG1800 Platform Applications

hiG1800 Patch Bundles are not placed into the same patch directory as the EMS. The patch
application process un-bundles the binaries and places them in the
/opt/SANTone/msf/<load_id>/BIN.
2.6 Patch Bundle Naming Explanation

CBUNDLE
A cumulative bundle is a bundle package that contains every released file at the point the
CBUNDLE is created.
IBUNDLE
An incremental bundle is a bundle package that contains files that are new in that update. There
may be many incremental bundles. The incremental bundle is intended to be small.
2.7 hiG1800 Patch Naming Explanation
0703.xx.xx.install
When an MG load is release it will have the above naming convention. This is an executable
install file.

A30828-X1122-A462-01-76P1 12/22/09
3 Patch Bundle Installation Prerequisites
3.1 Prerequisites Overview EMS BUNDLE
Before installing patch bundles on the EMS/hiG1800 product the following rules must be
followed:
1. A patch bundle can only be installed using the “installbundle” script contained in the
patch bundle itself.
2. A patch bundle can only be removed using the “uninstallbundle” script contained in the
patch bundle itself.
3. Once a patch bundle is installed the binaries and scripts contained in that patch bundle
should not be removed, altered, or modified in any way unless they are completely
removed using the “uninstallbundle” script contained in the patch bundle itself.
4. If a binary or script is removed, altered, or modified by the user after the patch bundle
was installed using the “installbundle” script it must be restored to its original condition
before installing new patch bundles.
5. If an incremental bundle is removed it must be replaced (reinstalled) before continuing
patch bundle installation.
6. Before installing the next subsequent patch release level (e.g. from 07.01.20.00 to
07.01.40.00) ALL incremental bundles must be installed on the current patch release
level. Please see section 1 of most recent RN’s for most current software
released for each software stream.
See example below:
Current release: 07.02.40.10
Currently installed bundles:
CBUNDLE113
IBUNDLE114
IBUNDLE115
Not yet installed bundles:
IBUNDLE116
IBUNDLE117
Customer wants to upgrade to: 07.02.40.30

Available bundle: CBUNDLE309
THESE BUNDLES MUST BE INSTALLED IN THIS ORDER:
IBUNDLE116
IBUNDLE117
CBUNDLE212
CBUNDLE309
All bundles are in the “tar.gz” format. The “tar.gz” file is a compressed file containing all files
located in the associated patch bundle sub-directory. A “tar” file is a UNIX command that allows
the user to create a single archive file containing many files. Such archiving allows the user to
maintain directory relationships and facilitates transferring complex programs with many
separate but integrated parts that must have their relationships preserved. This single “tar.gz”
file is the file that is transferred to and installed on the system.

A30828-X1122-A462-01-76P1 12/22/09
3.2 Pre-Patch Checks
Execution
Steps Node User Estimated time of completion: 15 min.
1 EMS GUI santera Login to the EMS GUI 7.2

2 EMS GUI santera From the EMS check Call/hr and success rate average
• Utilizing the EMS, monitor Call/hr, success rate average with the Real
Time Health Monitor
• System Status: Real Time Health Monitor Tab: Select Status: Start
graph with collected data.
3 EMS GUI santera Verify and Capture the alarms that are present in the switch from EMS
GUI
a) Fault Tab
b) Freeze
c) Clear All Alarms (right click on alarms)
d) Click ‘Yes’
e) Click ‘OK’
f) Refresh Button
g) Tools: Request Active Alarms
h) Click ‘Yes’
i) Print Screen and Save File
Verify Santera Application processes are Active or Hot Standby.

4 EMS 1 & santera
Su to root but do not use the – this will pick up santera properties.
2
su
Run Command: dis_appmgr_index
Example:
bocalaba1% dis_appmgr_index
FaultMgr AppMgr Act/Stby Index View on bocalaba1

AppMgr AppMgr Routing Active Node Standby Node In
Id Name Group Switchover
------------------------------------------------------------------------
4 OamFault 1 1 (ACT) 2 (HOT) NO
40 OamCfg 1 1 (ACT) 2 (HOT) NO
41 OamTrap 1 1 (ACT) 2 (HOT) NO
43 OamPfm 1 1 (ACT) 2 (HOT) NO
46 OamMgMgr 1 1 (ACT) 2 (HOT) NO
Verify that there are no processes exits are occurring. Check counters too for recent
5 EMS 1 & ROOT
exits.
2
Run Command: ptree_view
Example:
ptree View of Apps on bocalaba1

A30828-X1122-A462-01-76P1 12/22/09
9254 BsPmMonitord -l 0 -d /Santera -c RT
9257 BsPm
9261 BsDm
9263 OamNm
9272 OamDbMaint
9307 BsBootpd
9310 OamFault
9315 CsTftpd
9318 OamCfg
9319 OamTrap
9320 OamPfm
9322 OamMt
9327 OamMgMgr
9339 OamDbAudit
9343 /usr/java/jre/bin/java -DNameService -
Djava.endorsed.dirs=/opt/SANTone/ems/acti
9346 /usr/java/jre/bin/java -DNotificationService -
Djava.endorsed.dirs=/opt/SANTone/
9355 java -server -classpath
/opt/SANTone/ems/active/jacorb/etc:/opt/SANTone/ems/act
9369 java -classpath
/opt/SANTone/ems/active/jacorb/etc:/opt/SANTone/ems/active/jaco
9388 /usr/bin/perl /Santera/msc/active/patch/OamSysWatch
Displays Application Status.

RunCommand: pm_process_view
Example: Application 47 OamLr is Locked and Disable. This application will not run.

PM's View of Applications on bocalaba1

Id Execute Name Admin St. Oper St. Sch Pr Cnt Start Time
1 BsPm UNLOCKED ENABLED RT 30 1 Thu Oct 22 06:02:47 2009
2 BsDm UNLOCKED ENABLED RT 20 1 Thu Oct 22 06:02:48 2009
3 BsBootpd UNLOCKED ENABLED RT 10 1 Thu Oct 22 06:02:53 2009
4 OamFault UNLOCKED ENABLED RT 20 1 Thu Oct 22 06:02:55 2009
15 CsTftpd UNLOCKED ENABLED RT 10 1 Thu Oct 22 06:02:57 2009
40 OamCfg UNLOCKED ENABLED RT 10 1 Thu Oct 22 06:02:59 2009
41 OamTrap UNLOCKED ENABLED RT 15 1 Thu Oct 22 06:03:01 2009
43 OamPfm UNLOCKED ENABLED RT 10 1 Thu Oct 22 06:03:03 2009
44 OamNm UNLOCKED ENABLED RT 20 1 Thu Oct 22 06:02:49 2009
45 OamMt UNLOCKED ENABLED RT 10 1 Thu Oct 22 06:03:05 2009
46 OamMgMgr UNLOCKED ENABLED RT 15 1 Thu Oct 22 06:03:07 2009
47 OamLr LOCKED DISABLED TS 0
48 OamDbAudit UNLOCKED ENABLED RT 1 1 Thu Oct 22 06:03:09 2009
56 NamingService UNLOCKED ENABLED TS 0 1 Thu Oct 22 06:03:11 2009
57 NotificationService UNLOCKED ENABLED TS 0 1 Thu Oct 22 06:03:13 2009
58 CorbaEmsServer UNLOCKED ENABLED TS 0 1 Thu Oct 22 06:03:15 2009
59 CliServer UNLOCKED ENABLED TS 0 1 Thu Oct 22 06:03:17 2009
70 OamDbMaint UNLOCKED ENABLED RT 10 1 Thu Oct 22 06:02:50 2009
71 OamSysWatch UNLOCKED ENABLED RT 25 1 Thu Oct 22 06:03:20 2009
117 BsSnmpMonitord UNLOCKED ENABLED RT 25 0 Thu Oct 22 06:02:31 2009
120 timestend UNLOCKED ENABLED RT 5 0 Mon Sep 21 15:11:46 2009
122 snmpdm UNLOCKED ENABLED RT 0 0 Thu Oct 22 06:02:32 2009
6 ALL EMS ROOT Check Disk Space

Run Command: df –k
bocalaba1# df ‐k
/dev/dsk/c0t1d0s4 5161014 1981221 3128183 39% /var (less than 50%)
swap 20538664 20256 20518408 1% /tmp (less than 50%)
swap 20518432 24 20518408 1% /var/run (less than 50%)
/dev/dsk/c0t3d0s7 9029429 7983012 956123 90% /dbbackup (cleans up manually)
/dev/dsk/c0t3d0s6 123779200 65557 122475851 1% /billing (less than 50%)
/dev/dsk/c0t1d0s7 8658869 2430418 6141863 29% /space (less than 50%)

A30828-X1122-A462-01-76P1 12/22/09
/dev/dsk/c0t1d0s6 111744451 6174382 104452625 6% /stats (less than 50%)
7 EMS GUI santera Continue to Monitor throughout the procedure.

• Utilize the EMS to continue to monitor Call/hr, success rate average
with the Real Time Health Monitor at the top of the EMS Configuration
tab and Fault tab->Alarms Windows.

A30828-X1122-A462-01-76P1 12/22/09
3.3 Copy New patching files from FTP server
Execution
Ste Node User Estimated time of completion: 30min to 1hr. (depends on transfer speed)
ps
1 EMS 1 santera Use scp or ftp to copy patch files from patch repository to EMS1
login to EMS1
ssh ems1
cd /stats/bundles
ftp public server
cd /location of EMS patches
bin
prompt
mget *bundle418*gz
cd /location hiG1800patch
mget 0702.xx.xx.install
cd /location latest TEKbase (if applicable)
get TEKbase1.8 (need for installing new bundles during patch or
upgrade, please always use the latest TEKbase version)
2 EMS 1 santera Copy files to EMS2

scp patch_files EMS2:/stats/bundles/

A30828-X1122-A462-01-76P1 12/22/09
3.4 Run CHECK DATABASE scripts
Step Node User Execution

Estimated time of completion: 10 min.
1 EMS 1 santera On active EMS node
su - santera
a) cd /opt/SANTone/msc/active/installation/switchCheck
b) switchCheck -b
NOTE
If errors occur in checkDbHealth portion refer to
/space/Santera/db/log/checkDbHealth.log
Some of the errors that are displayed can be ignored such as alarm card
errors. You are looking for differences in the database and if the
checkDbHealth passed.
Note: If applications are not in HOT Standby status.

Please contact TAC.
2 EMS 2 santera a) su - santera
b) cd /opt/SANTone/msc/active/TimesTen/scripts
c) checkDbHealth <localEMSnode> <peerEMSnode> -all
Example: checkDbHealth <EMS2> <EMS1> -all
NOTE
If errors are exhibited, refer to /space/Santera/db/log/checkDbHealth.log
and call your next level of support.

A30828-X1122-A462-01-76P1 12/22/09
3.5 Backup Current msfswcontrolfile

1 EMS 1 santera cd to the location of the MsfSwControfile
a) cd /space/Santera/msf/1/07.02.40.00
b) ls –ltr (and look for the MsfSwControlFile.txt file)
c) mv MsfSwControlFile.txt MsfSwControlFile.old.txt
2 EMS 2 santera cd to the location of the MsfSwControfile
a) cd /space/Santera/msf/1/07.02.40.00
b) ls –ltr (and look for the MsfSwControlFile.txt file)
c) mv MsfSwControlFile.txt MsfSwControlFile.old.txt
3 EMS santera Repeat steps above if there are multiple MGs.

Example: /space/Santera/msf/2/07.02.40.00
3.6 Verify current patch level on EMS

1 EMS 1 santera
a) cd /stats/bundlelogs/<software release>/
i. Ex: /stats/bundlelogs/07.02.40.00
b) cat BUNDLE_SEQUENCE
i. Contents of the file should look like following:
# BundleId UpdateId Had DB? Had EMS? Had MG? Had Eng?
CBUNDLE107 07.02.40.32 yes yes yes yes
2 EMS 2 santera Repeat above step on EMS2

A30828-X1122-A462-01-76P1 12/22/09
3.7 Verify current patch level on hiG1800

1 EMS 1 santera Verify in EMS GUI what the software release is currently on both CM cards.
OR
View the last update level to verify that the hiG1800 is ready to be patched
as well. The location of the hiG1800 Application is
/opt/Santone/msf/<software update release level> in R.7.2.
Display hiG1800 loads before updating.
ttIsqlCS -connStr dsn=systemconfigurationcs -e "select

MSFRELNODENUM,MSFRELCMCSLOTID,MSFRELSWVERSION,
MSFRELSWUPDATELEVEL from Santera.msfreleasecontrol ; quit"
Example Output
select msfrelnodenum,msfrelcmcslotid,msfrelswversion,msfrelswupdatelevel
from santera.msfreleasecontrol;
< 1, 7, 07.02.40.00 , 07.02.40.32 >
< 1, 8, 07.02.40.00 , 07.02.40.32 >
< 2, 7, 07.02.40.00 , 07.02.40.32 >
< 2, 8, 07.02.40.00 , 07.02.40.32 >
4 rows found.
3.8 Backup current database on both EMS nodes

1 EMS 1 santera a) cd /opt/SANTone/msc/active/cron
b) dbbackup.sh
c) ls –lrt /dbbackup (to check current database
backup)
2 EMS 2 a) cd /opt/SANTone/msc/active/cron
b) dbbackup.sh
backup)
3.9 Install Tekbase 1.8 (or Latest Version)

Install TEKbase – 1.8 or latest TEKbase version.
1 ALL santera
EMS
NODES
a) Go into the <site>: #cd /stats/bundles

A30828-X1122-A462-01-76P1 12/22/09
b) <site># chmod 755 TEKbase-1.8.install
c) <site># TEKbase-1.8.install <enter>
d) <site># . /Santera/.zshenv (when file is completed: copy and
paste this command and execute it adds installbundle to path)
e) ls (and check to see if TEKbase-1.8.install is in the directory)
f) verify file is in dbbackup directory: <site># ls /dbbackup/recovery
Revised:
a) cd /stats/bundles (or your software directory)

b) % gunzip TEKbase-1.8.install (if TEKbase is .gz)
c) % chmod 777 TEKbase-1.8.install
d) # TEKbase-1.8.install <enter>
e) # . /Santera/.zshenv <enter> (when file is completed: copy and
paste this command and execute it adds installbundle to path)
f) % ls /dbbackup/recovery (verify file is in dbbackup directory)
Note:
% = santera
# = root

A30828-X1122-A462-01-76P1 12/22/09
4 Patching EMS Server
4.1 Applying Patches To EMS
• Verify EMS patch is present on both EMS nodes

• Installbundle EMS patch
• Install hiG1800 patch
• Activate hiG1800 patch
• Customer call thru test plan
• Activate EMS patch
• Run customer defined critical call through test plan
CAUTION
Alarms that may occur during patching include:

• Alarms indicating that hiG1800 cards are being reset are down.
• Alarms indicating that EMS processes/nodes are down.
• Alarms indicating that SS7 Link(s) are down.
NOTE
In the tables below, bold text should be typed.
4.1.1 Verify patches are present on EMS

1 EMS 1 santera SSH to EMS node 1 as ‘santera’
ssh santera@ems1
Connects to the EMS<node>. Current directory is: /space/Santera.
2 EMS 1 santera Verify appropriate patch(es) ##PATCHID## is (are) in

/stats/bundles
a) cd /stats/bundles
b) ls -la
Ensure patch ##PATCHID## is in the list of files returned by the
command.
Example CBUNDLE309-07.02.40.30.tar.gz 0702.47.00.install
3 EMS 2 santera SSH to EMS node 2 as ‘santera’

ssh santera@ems2
Connects to the EMS<node>. Current directory is: /space/Santera.

A30828-X1122-A462-01-76P1 12/22/09
4.1.1.1 Apply Patch Bundles to EMS Nodes

1 EMS Santera Login to EMS as santera
Nodes
a) ssh santera@EMS1
b) ssh santera@EMS2
2 EMS Santera To verify installed patches:

Nodes
# cat /stats/bundlelogs/07.02.40.00/BUNDLE_SEQUENCE
3 EMS Santera To unzip and untar a patch bundle:

Nodes
a) cd /stats/bundles
b) gunzip < bundlename.tar.gz | tar xvf –
4 EMS ROOT Switch user to root
Nodes
a) su
b) <root password>
c) zsh
5 EMS ROOT To install the patch bundle:
Nodes
a.) cd /stats/bundles/ <bundlename >
b.) ls –la
c.) installbundle
/opt/TEKutils/active/GenBundle/installbundle (execute this

command)
6 EMS Perform Patch Application Instructions as Directed.
Nodes Make a Note of the Special Application.

A30828-X1122-A462-01-76P1 12/22/09
5 Patching the hiG1800
This section covers the patch application steps for the hiG1800. These steps will be repeated
for each hiG1800 that is a part of the switching complex.
The upgrade of the hiG1800s are split into 2 phases. The first phase is when all of the standby
cards are upgraded to the latest release and switched over first before doing any of the
redundant cards. Performing the upgrade in phases allows for a simpler and quicker way to
downgrade the gateway since can perform switchovers to revert back to the previous release
rather than having to downgrade the standby cards and doing switchover on the standby cards.
The cards that use a load share protection scheme are patched during phase one. The cards
are completed one at a time to avoid affecting service. The second phase is when the newly
standby cards on the old release will be upgraded.
5.1 Applying hiG1800 Patches To EMS Nodes

1 EMS 1 Root
EMS 2 a) cd /stats/bundles.
b) On the switch, make sure the install file is executable.
c) chmod 755 0702.47.00.install
d) Execute the install file.
e) 0702.47.00.install
f) Once the script is finished, there will be a 0702.47.00 directory
in /opt/SANTone/msf.
g) This load can now be selected from the EMS GUI and used as
an argument to generate the MsfControl file.
5.1.1 Check Node Link Status

1 BsPm ROOT Ensure All Nodes and Links show correct information.
Monitor
Using BsPmMonitorGui on Active EMS node, click FaultMgr>Display Node All
Ensure MSC node 1 is UNLOCKED ON_LINE
Ensure MSC node 2 is UNLOCKED ON_LINE
Ensure MSF node X is UNLOCKED ON_LINE depending
Click OK
Note: This output will vary depending on the customer
switch configuration.

A30828-X1122-A462-01-76P1 12/22/09
Example Output
Node Node Admin Node Node ESA In Node Group

Type Id State State Sub-type Update Activation Group Type
----------------------------------------------------------------------------------------------------------------
MSC 1 UNLOCKED ON_LINE EMS --- No 0 1 CORE
MSC 2 UNLOCKED ON_LINE EMS --- No 0 1 CORE
MG 1 UNLOCKED ON_LINE --- --- --- --- 1 CORE
MG 2 UNLOCKED ON_LINE --- --- --- --- 1 CORE
5.1.2 Generating the MsfSwControlFile

1 EMS 1 Santera As santera generate the MsfControlFile on each node. Use the name of the file version
of the patch just installed. Verify it is the name in the /opt/SANTone/msf directory.
In command below node is the hiG1800 number. Repeat for each hiG1800. For the 2nd
one node=2. Only run once for each hiG1800. Do not increment slot use either 7 or 8
not both.
/space/Santera/msc/07.02.40.00/patch/scfmgr --common --
baserelease=07.02.40.00 --patchrelease=0702.47.00 --node=1 --slot=7
cd /space/Santera/msf/1/07.02.40.00
ls –l MsfSwControlFile.txt
cat MsfSwControlFile.txt
Repeat for 2nd hiG1800 node 2.
Note: Compare the size and contents of the file with the
other EMS node.

A30828-X1122-A462-01-76P1 12/22/09
2 EMS 2 Santera As santera generate the MsfControlFile on each node. Use the name of the file version
of the patch just installed. Verify it is the name in the /opt/SANTone/msf directory.
In command below node is the hiG1800 number. Repeat for each hiG1800. For the 2nd
one node=2. Only run once for each hiG1800. Do not increment slot use either 7 or 8
not both.
Repeat for 2nd hiG1800 node 2.
Note: Compare the size and contents of the file with the
other EMS node.
5.1.3 Predownload MsfSwControlFile and New hiG1800 Patch Load

1 EMS GUI GUI From EMS, Predownload 1st PAC card
a) Click on 1st PAC card .

b) Click third mouse buton :Software Release:Pre-download MG Sw Ctl file
c) Click Wait for “Successful” click OK
Monitor standby CMM window from the terminal server

(telnet termsvr (or IP) 2100)
2 EMS GUI GUI From EMS, Predownload 2nd PAC card
d) Click on 2nd PAC card .

e) Click third mouse buton :Software Release:Pre-download MG Sw Ctl file
f) Click Wait for “Successful” click OK

(telnet termsvr (or IP) 2600)

A30828-X1122-A462-01-76P1 12/22/09
5.1.4 Patching the Standby Packet and Control (PAC) card

1 EMS GUI GUI From EMS, lock standby PAC card
g) Click on Standby PAC card

h) Click third mouse button: Admin State: Lock
i) Click Yes to Accept warning

(telnet termsvr (or IP) 2100 or 2600)
2 EMS GUI GUI From EMS, set SwUgrade flag for standby PAC card
a) Click on Standby PAC card

b) Click on Software Release Tab
c) Select the latest release level from the Update Level field to pick up new
Software for the hiG1800
d) Click on SwUpgrading flag (to set checkmark) for standby PAC card
e) Click Apply at the bottom of the window
SoftwareVersion = 07.02.40.00 Update Level (Patches) = 0702.47.00

DatabaseFile = MsfDbControlFile.txt SoftwareFile =
MsfSwControlFile.txt SWUpgrading = ON
The upgrade flag is turned ON to indicate to the hiG1800 to not sync database
from its mate CM card but get the new release database from EMS. The
Software Revision must be set to 07.02.40.00 otherwise the Regeneration of
the Software Control File will not occur.
3 EMS GUI GUI From EMS, unlock standby PAC card
a) Click on standby PAC card

b) Click third mouse button: Admin State: Unlock
c) Monitor standby from terminal server window.
d) From CMM window
o Verify CMM goes in-service by typing the following “mcinit”
CAUTION
Wait for the status to be "System Ready"
4 EMS GUI GUI From EMS, ensure standby PAC card displays “Hot Standby”
5 EMS GUI GUI From EMS, switchover to standby PAC card
a) Click on Active PAC card

b) Click third mouse button: Operation: SwitchOver
c) Click Yes to Accept warning
6 EMS GUI GUI From EMS, Clear the SwUpgrade flag for new active PAC card
a) Click on new active PAC card

A30828-X1122-A462-01-76P1 12/22/09
b) Click on Software Release Tab: SwUpgrade flag (to uncheck) active card
c) Click on Apply
5.1.5 Monitor Call rate with RTHM
CAUTION
If calls drop significantly and do not recover have customer make test calls. If necessary, call next level
of support.
5.1.6 Patching VS (Voice Server) Cards (If Applicable)

1 EMS GUI GUI From EMS, upgrade the protecting VS Voice Server Card
a) Click on standby Voice Server card

b) Click third mouse button: Operation: Reset
c) Click ‘Yes’
2 EMS GUI GUI From EMS, switchover traffic to the protecting VS (only on Standby/Active
Protection shceme.)
a) Click on active VS card

b) Click third mouse button: Operation: Switchover
CAUTION
Verify the switchover was successful. Active card should become Standby and
Protecting card should become Active
3 EMS GUI GUI From EMS, reset the protecting VS card
a) Click on protecting Voice Server card

CAUTION
Monitor VS for it to finish booting
4 EMS GUI GUI From EMS, repeat steps 1 thru 3 for each provisioned VS
cards

A30828-X1122-A462-01-76P1 12/22/09
5.1.7 Signaling Gateway (SG) Server Cards (If Applicable)

1 EMS GUI GUI From EMS, upgrade the protecting Signaling Gateway (SG) Server Cards
a) Click on standby SG card

c) Click ‘Yes’
2 EMS GUI GUI From EMS, switchover traffic to the protecting SG Cards
a) Click on active SG card

CAUTION
Verify the switchover was successful. Active card should become Standby and
Protecting card should become Active
3 EMS GUI GUI From EMS, reset protecting Signaling Gateway (SG) Server card
a) Click on standby protecting Signaling Server card

CAUTION
Monitor Signaling Server Card for it to finish booting
4 EMS GUI GUI From EMS, repeat steps 1 thru 3 for each provisioned
Signaling Gateway (SG) Server cards
5.1.8 Patching Standby GigE-2 Card (If Applicable)

1 EMS GUI GUI From EMS, lock and reset standby GigE-2 card
a) Click on standby GigE-2

b) Click third mouse button: Admin State: Lock
c) Click ‘Yes’
d) Click on standby GigE-2 card
2 EMS GUI GUI From EMS, unlock GigE-2 cards
a) Click on standby GigE-2 card

b) Click third mouse button: Administration: Unlock
c) Click ‘Yes’

A30828-X1122-A462-01-76P1 12/22/09
CAUTION
Monitor GigE-2 for it to finish unlocking
3 EMS GUI GUI From EMS, switchover GigE-2 cards
a) Click on active GigE-2 card

CAUTION
Wait for GigE-2 switchover to complete
4 Repeat all the above steps for any other standby GIGE pairs that are
provisioned.
5.1.9 Patching ES CARDs (If Applicable)

1 EMS GUI GUI From EMS, upgrade first ES card using EMS Gui
a) Click on first ES card to be upgraded

b) Click third mouse button: Admin State: Shutdown
* Wait for ES card to transition to Locked
c) Click on standby ES card
d) Click third mouse button: Operation: Reset
e) Click ‘Yes’
2 EMS GUI GUI From EMS, reset and Unlock ES card
• Click third mouse button: Admin State: Unlock
CAUTION
Monitor ES for it to display “Active”
3 Repeat steps for all provisioned ES cards

4 EMS GUI GUI Validate that ALL ES cards are upgraded and on the correct load.

A30828-X1122-A462-01-76P1 12/22/09
5.1.10 Load Standby SST Card with DSP/PCM Binaries (If applicable)

1 EMS GUI GUI From EMS, load the standby SST with the new DSP/PCM binaries
a) Click on standby SST

c) Click ‘Yes’
e) Click ‘Yes’
CAUTION
Wait 2 minutes for the SST to completely stabilize
f) Click third mouse button: Admin State: Unlock
2 EMS GUI GUI From the EMS GUI, Nodes->MG Nodes->Node Number->SST DSP Groups,
reload each Standby DSP Group type.
a) Right Click on Group (for example DTMF)

b) Click on Operations->Reload Standby
SST DSP Groups

- DTMF -> OPERATIONS -> Reload Standby
- MF -> OPERATIONS -> Reload Standby
- Tone -> OPERATIONS -> Reload Standby
- CallerID/FSK -> OPERATIONS -> Reload Standby
- Bridge -> OPERATIONS -> Reload Standby
- Highway Monitor -> OPERATIONS -> Reload Standby
- Tone Sequencer -> OPERATIONS -> Reload Standby
-Tone Detector Network -> OPERATIONS -> Reload Standby
- Measurement -> OPERATIONS -> Reload Standby
- Cellular Text Modem -> OPERATIONS -> Reload Standby
3 EMS GUI santera From EMS, switchover the upgrade SST
a) Click on active SST

c) Click ‘Yes’

A30828-X1122-A462-01-76P1 12/22/09
5.1.11 Patching Standby QOC3 Cards (If Applicable)

1 EMS GUI GUI Align ports on Protecting card to “Hot Standby”
On Protecting QOC3 card, take note of the OC3 Ports and their
status. All ports must be in a “Hot Standby” mode.
a) Click on the Protecting OC3 Card

b) Click on each OC3 Port
c) Validate all provisioned ports on the card are “Hot Standby”. If not,
proceed with step d otherwise proceed to step 2.
CAUTION
All Ports not on “Hot Standby” must have their mated port investigated for issues.
Failure to do this could cause call degradation.
d) Check alarms, errors and logs for any issues that may have caused the
port to not be Active.
NOTE
Once it is determined no issues reside on the Working Card Ports, switch them
over to Active.
e) Click on Protecting QOC3 Card

f) Right click on the Active OC3 Port: Operation: Switchover.
g) Select ‘Yes’
CAUTION
Failure to monitor ports (logs, alarms and calls) could result in service degradation.
h) Monitor the ports logs, alarms and look for any call failures for 5 minutes.
Revert the port back if issues can’t be resolved in a timely manner.
i) Perform f through k for all Protecting ports not “Hot Standby”.
2 EMS GUI GUI Upgrade the Protecting QOC3 card by resetting it from EMS.
CAUTION
Failure to follow these steps precisely could result in service degradation.
a) Right click on Protecting QOC3 card: Operation: Reset

b) Click Yes
c) Wait for Operation State to become “Enabled”.
CAUTION
Failure to monitor card for call failures and adverse logs or alarms, could result in
service degradation.

A30828-X1122-A462-01-76P1 12/22/09
d) Monitor QOC3 for load completion, adverse alarms and logs or call
failures.
3 Validate stability of newly upgraded card. Card is Enabled and Hot Standby.
4 EMS GUI GUI Align ports on Protected card
On Protected QOC3 card, take note of the OC3 Ports and their status. All ports
must be in a “Hot Standby” mode.
a) Click on Protected QOC3 Card

b) Click on the OC3 Port
CAUTION
NOTE
Once it is determined no issues reside on the Working Cards Ports, switch them
over to Active.
e) Click on Protected QOC3 Card

g) Select ‘Yes’
CAUTION
h) Monitor the ports logs, alarms and look for any call failures for 5 minutes.
Revert the port back if issues can’t be resolved in a timely manner.
5 EMS GUI GUI Validate all QOC3 cards have correct software load and bootloaders.
Connect to each QOC3 cpu and display the bootloader information.
CAUTION
Failure to validate software could result in service degradation.
i. send 22,0:connect
j. info
k. disconnect

A30828-X1122-A462-01-76P1 12/22/09
5.1.12 Have Customer Execute Critical Call-Through Test Plan
CAUTION
Make sure all test calls pass before proceeding. Evaluate EMS/hiG1800 switch performance. Pay special
attention to Cards switching over on their own, processes exiting, and call rate completion dropping.
Now is the time to decide on proceeding with the upgrade or falling back to the OLD Release on the
hiG1800. If necessary, call next level of support.
5.1.13 Upgrade Remaining Half of (hiG1800) (If Decision to Proceed)
WARNING
Failure to upgrade all active cards could result in call processing degradation.
5.1.14 Patching the Newly Standby QOC3 Cards (If Applicable)

1 EMS GUI GUI Align ports on Protecting card to “Hot Standby”
On Protecting QOC3 card, take note of the OC3 Ports and their
status. All ports must be in a “Hot Standby” mode.
a) Click on the Protecting OC3 Card

b) Click on each OC3 Port
CAUTION
NOTE
Once it is determined no issues reside on the Working Card Ports, switch them
over to Active.
e) Click on Protecting QOC3 Card

g) Select ‘Yes’

A30828-X1122-A462-01-76P1 12/22/09
CAUTION
h) Monitor the ports logs, alarms and look for any call failures for 15
minutes. Revert the port back if issues can’t be resolved in a timely
manner.
i) Perform f through k for all Protecting ports not “Hot Standby”.
2 EMS GUI Upgrade the Protecting QOC3 card by resetting it from EMS.
CAUTION
Failure to follow these steps precisely could result in service degradation.
a) Right click on Protecting QOC3 card: Operation: Reset

b) Click Yes
c) Wait for Operation State to become “Enabled”.
CAUTION
Failure to monitor card for call failures and adverse logs or alarms, could result in
service degradation.
d) Monitor QOC3 for load completion, adverse alarms and logs or call
failures.
3 Validate stability of newly upgraded card. Card is Enabled and Hot Standby.
4 EMS GUI GUI Align ports on Protected card
On Protected QOC3 card, take note of the OC3 Ports and their status. All ports
must be in a “Hot Standby” mode.
a) Click on Protected QOC3 Card

b) Click on the OC3 Port
CAUTION
NOTE
Once it is determined no issues reside on the Working Cards Ports, switch them
over to Active.
e) Click on Protected QOC3 Card

A30828-X1122-A462-01-76P1 12/22/09
g) Select ‘Yes’
CAUTION
h) Monitor the ports logs, alarms and look for any call failures for 15
minutes. Revert the port back if issues can’t be resolved in a timely
manner.
5 EMS GUI GUI Validate all QOC3 cards have correct software load and bootloaders.
Connect to each QOC3 cpu and display the bootloader information.
CAUTION
Failure to validate software could result in service degradation.
a. send 22,0:connect
b. info
c. disconnect
6 EMS GUI GUI Make sure both Active and Standby QOC3 cards are on the NEW release.
5.1.15 Load Newly Standby SST Card with DSP/PCM Binaries (If applicable)

1 EMS GUI GUI From EMS, load the standby SST with the new DSP/PCM binaries
a) Click on standby SST

c) Click ‘Yes’
e) Click ‘Yes’
CAUTION
Wait 5 minutes for the SST to completely stabilize
f) Click third mouse button: Admin State: Unlock
2 EMS GUI GUI From the EMS GUI, Nodes->MG Nodes->Node Number->SST DSP Groups,
reload each Standby DSP Group type.
a) Right Click on Group (for example DTMF)

b) Click on Operations->Reload Standby
SST DSP Groups

- DTMF -> OPERATIONS -> Reload Standby
- MF -> OPERATIONS -> Reload Standby
- Tone -> OPERATIONS -> Reload Standby
- CallerID/FSK -> OPERATIONS -> Reload Standby

A30828-X1122-A462-01-76P1 12/22/09
- Bridge -> OPERATIONS -> Reload Standby
- Highway Monitor -> OPERATIONS -> Reload Standby
- Tone Sequencer -> OPERATIONS -> Reload Standby
-Tone Detector Network -> OPERATIONS -> Reload Standby
- Measurement -> OPERATIONS -> Reload Standby
- Cellular Text Modem -> OPERATIONS -> Reload Standby
3 EMS GUI GUI Make sure both Active and Standby SST cards are on the NEW
release.
5.1.16 Patching Newly Standby GigE-2 Card (If Applicable)

1 EMS GUI GUI From EMS, lock and reset standby GigE-2 card
a) Click on standby GigE-2

c) Click ‘Yes’
d) Click on standby GigE-2 card
e) Click third mouse button: Operation: Reset
f) Click ‘Yes’
2 EMS GUI GUI From EMS, unlock GigE-2 cards
a) Click on standby GigE-2 card

b) Click third mouse button: Administration: Unlock
c) Click ‘Yes’
CAUTION
Monitor GigE-2 for it to finish unlocking
3 EMS GUI GUI

Repeat all the above steps for any other standby GIGE
pairs that are provisioned.
4
Make sure both Active and Standby GIGE cards are on the NEW release.

A30828-X1122-A462-01-76P1 12/22/09
5.1.17 Patching the Newly Standby PAC card

1 EMS GUI GUI
NOTE
The SwUpgrade flag is NOT set for this PAC.
From EMS, lock standby PAC card
a) Click on Standby PAC card

d) Click Yes to Accept warning
e) Click on Software Release Tab
f) Select the latest release level from the Update Level field to pick up new
Software for the hiG1800.
g) Click Apply at the bottom of the window
SoftwareVersion=07.02.40.00 Update Level=07.02.40.39 ( select

the lastest)
DatabaseFile=MsfDbControlFile.txt
SoftwareFile=MsfSwControlFile.txt

(telnet termsvr (or IP) 2100 or 2600)
2 EMS GUI GUI From EMS, reset standby PAC card

d) Monitor standby CMM window from the terminal server window.
* Note the new software being TFTPed from the EMS
* Verify CMM goes in-server by typing the following periodically “mcinit”
CAUTION
3 EMS GUI GUI From EMS, unlock standby PAC card

b) Click third mouse button: Admin State: Unlock
c) Monitor standby from the terminal server.
d) From CMM window
e) Verify CMM goes in-service by typing the following “mcinit”
CAUTION
4 EMS GUI GUI From EMS, ensure standby PAC card displays “Hot Standby”
5 Make sure both Active and Standby PAC cards are on the NEW
release.

A30828-X1122-A462-01-76P1 12/22/09
THE hiG1800 PATCHING HAS BEEN COMPLETED!!!
IF THE SWITCH HAS MULTIPLE hiG1800s, REPEAT
SECTION 5.1. OTHERWISE, CONTINUE TO THE NEXT
SECTION TO UPGRADE THE FIRST EMS NODE.

A30828-X1122-A462-01-76P1 12/22/09
6 Activate New Patch on STANDBY EMS NODE (2)
6.1 Apply Patch To Standby Node

1 EMS 1 ROOT Align all active EMS processes on Node 1 as Active (ACT) and Node 2 as Standby
Standby (HOT).
Node If applications are not aligned, go to EMS GUI and switchover the AppMgrId.
Example:
nechbrgems2% dis_appmgr_index
FaultMgr AppMgr Act/Stby Index View on nechbrgems2

Id Name Group
Switchover
-----------------------------------------------------------------------
-----
4 EMSFault 1 2 (ACT) 1 (HOT) NO
40 EMSCfg 1 2 (ACT) 1 (HOT) NO
41 EMSTrap 1 2 (ACT) 1 (HOT) NO
43 EMSPfm 1 2 (ACT) 1 (HOT) NO
46 EMSMgMgr 1 2 (ACT) 1 (HOT) NO
2 EMS 2 GUI Shutdown the standby node using EMS GUI:
Standby
Node a) Click Nodes
b) EMS Nodes (MSC)
c) Node Number (2)
d) Click right mouse button: Admin State: Shutting
Down
e) Click on YES button
Run Command: # ptree_view
o Wait until only the Critical Applications are running.

* BsPm
* BsDm
* EMSNm
* EMSDbMaint
Allow the processes 1-2 minutes to finish shutdown. If processes do not shutdown
after 2 mintutes, lock the standby EMS node from the EMS GUI.
Once the shutdown is complete, the node will automatically be locked. Any
applications which were active on this node will be activated on the other EMS node
3 EMS 2 ROOT Shutdown EMS Applications.

Standby
Node Run Command:# stopPlatform
or
Run Command: # svcmgmt disable platform

* Should indicate “Nothing Is Running”

A30828-X1122-A462-01-76P1 12/22/09
4 EMS 2 ROOT Perform Special Application Instructions as Directed
Standby
Node
(If Applicable).
5 EMS 2 ROOT Start EMS Applications.

Standby
Node Run Command: # startPlatform
or
Run Command: # svcmgmt enable platform

Example:
nechbg2mgc1% ptree_view
ptree View of Apps on nechbg2mgc1
773 BsPmMonitord -l 0 -u 9999 -d /Santera -c RT

776 BsPm
791 BsDm
798 EMSNm
811 BsBootpd
817 EMSFault
821 CsTftpd
825 EMSCfg
828 EMSTrap
829 EMSPfm
833 EMSMt
837 EMSMgMgr
862 EMSDbAudit
868 /usr/java/jre/bin/java -DNameService -Djava.endorsed.dirs=/opt/SANTone/ems/acti
871 /usr/java/jre/bin/java -DNotificationService -Djava.endorsed.dirs=/opt/SANTone/
884 java -server -classpath /opt/SANTone/ems/active/jacorb/etc:/opt/SANTone/ems/act
914 java -classpath /opt/SANTone/ems/active/jacorb/etc:/opt/SANTone/ems/active/jaco
917 /usr/bin/perl /Santera/msc/active/patch/EMSSysWatch
975 EMSDbMaint
1518 sh -c ttStatus > /tmp/ttStatusMaint
1519 /opt/SANTone/msc/active/TimesTen/bin/ttStatusCmd
Ensure the applications for the standby node have started and are stable.
Once booted, the new software will be running on the standby node. Since the
standby node is locked, no links with other node will be established. All applications
will come up in the default state of In Service or Standby. The process EMSCfg will
also come up standby and will not send routing to any of the applications.
6 EMS 2 GUI Using the EMS GUI, UNLOCK the standby EMS. This will cause a node sync to be
Standby performed by the active EMSCfg to bring this node online.
Node
a) System
b) Nodes
c) EMS (MSC)
d) Node # (example: 2 ewlinofxp -2)
e) Right Click: Admin State: Unlocked
If a process remains in COLD state for an extended period of time, restart the process
and monitor it again for going HOT.
Example:

A30828-X1122-A462-01-76P1 12/22/09

Id Name Group
Switchover
-----------------------------------------------------------------------
-----

A30828-X1122-A462-01-76P1 12/22/09
7 Switchover Preparation
At this point, the standby EMS node should be
patched by now.
7.1 Switchover Standby EMS Node (2) To Active

1 EMS 2 GUI Activate Standby EMS Node.
Standby
Node a) Click Nodes
b) EMS Nodes (MSC)
c) Node Number (2)
d) Click on third mouse: Operations: Activate
EMSFault should be Active on the EMS Node 2 that is running the new software
release.
7.2 Critical Call Test Plan
CAUTION
Have customer execute critical call test plan.
Make sure all test calls pass before proceeding. If necessary, call next level of support.

A30828-X1122-A462-01-76P1 12/22/09
8 Activate New Patch on Standby EMS Node (1)
8.1 Apply Patch To Standby EMS Node

1 EMS 1 ROOT Align all active EMS processes on Node 2 as Active (ACT) and Node 1 as Standby
Standby (HOT).
Node If applications are not aligned, go to EMS GUI and switchover the AppMgrId.
Example:

Id Name Group
Switchover
-----------------------------------------------------------------------
-----
2 EMS 1 GUI Shutdown the standby node using EMS GUI:

Standby
Node a) Click Nodes
b) EMS Nodes (MSC)
c) Node Number (1)
d) Click right mouse button: Admin State: Shutting
Down
e) Click on YES button
o Wait until only the Critical Applications are running.

* BsPm
* BsDm
* EMSNm
* EMSDbMaint
Allow the processes 1-2 minutes to finish shutdown. If processes do not shutdown
after 2 mintutes, lock the standby EMS node from the EMS GUI.
Once the shutdown is complete, the node will automatically be locked. Any
applications which were active on this node will be activated on the other EMS node
3 EMS 1 ROOT Shutdown EMS Applications.

Standby
Node Run Command:# stopPlatform

A30828-X1122-A462-01-76P1 12/22/09
or
Run Command: # svcmgmt disable platform

* Should indicate “Nothing Is Running”
4 EMS 1 ROOT Perform Special Application Instructions as Directed

Standby
Node
(If Applicable).
5 EMS 1 ROOT Start EMS Applications.
Standby
Node Run Command: # startPlatform
or
Run Command: # svcmgmt enable platform

Example:
nechbg2mgc1% ptree_view
ptree View of Apps on nechbg2mgc1
773 BsPmMonitord -l 0 -u 9999 -d /Santera -c RT

776 BsPm
791 BsDm
798 EMSNm
811 BsBootpd
817 EMSFault
821 CsTftpd
825 EMSCfg
828 EMSTrap
829 EMSPfm
833 EMSMt
837 EMSMgMgr
862 EMSDbAudit
868 /usr/java/jre/bin/java -DNameService -Djava.endorsed.dirs=/opt/SANTone/ems/acti
871 /usr/java/jre/bin/java -DNotificationService -Djava.endorsed.dirs=/opt/SANTone/
884 java -server -classpath /opt/SANTone/ems/active/jacorb/etc:/opt/SANTone/ems/act
914 java -classpath /opt/SANTone/ems/active/jacorb/etc:/opt/SANTone/ems/active/jaco
917 /usr/bin/perl /Santera/msc/active/patch/EMSSysWatch
975 EMSDbMaint
1518 sh -c ttStatus > /tmp/ttStatusMaint
1519 /opt/SANTone/msc/active/TimesTen/bin/ttStatusCmd
Ensure the applications for the standby node have started and are stable.
Once booted, the new software will be running on the standby node. Since the
standby node is locked, no links with other node will be established. All applications
will come up in the default state of In Service or Standby. The process EMSCfg will
also come up standby and will not send routing to any of the applications
6 EMS 1 GUI Using the EMS GUI, UNLOCK the standby EMS. This will cause a node sync to be
Standby performed by the active EMSCfg to bring this node online.
Node
a) System
b) Nodes
c) EMS (MSC)
d) Node # (example: 1 ewlinofxp -1)
e) Right Click: Admin State: Unlocked
If a process remains in COLD state for an extended period of time, restart the process
and monitor it again for going HOT.
Example:

A30828-X1122-A462-01-76P1 12/22/09

Id Name Group
Switchover
-----------------------------------------------------------------------
-----

A30828-X1122-A462-01-76P1 12/22/09
9 Switchover Preparation
At this point, the standby EMS node (1)should be patched by now.
9.1 Switchover Standby Node (1) to Active

1 EMS 1 GUI Activate Standby EMS Node.
Standby
Node a) Click Nodes
b) EMS Nodes (MSC)
c) Node Number (1)
d) Click on third mouse: Operations: Activate
EMSFault should be Active on the EMS Node 1 that is running the new software
release.
9.2 Critical Call Test Plan
CAUTION
Have customer execute critical call test plan.
Make sure all test calls pass before proceeding. If necessary, call next level of support.
PATCHING EMS NODES IS COMPLETE!!!

A30828-X1122-A462-01-76P1 12/22/09
Post Patch Checks:
a) Verify alarms
b) Run checkDbHealth
*<EMS1> <EMS2> -all
*<EMS2> <EMS1> -all
c) Run dbbackup
d) Verify Binaries
e) Run switchcheck –b

A30828-X1122-A462-01-76P1 12/22/09
10 Post-Patch checks
10.1 Verify Alarms
1 EMS GUI
Check if there are new alarms that was generated after the
patch. Contact TAC is an alarm was caused by our patching.
10.2 Run Check DB Health

1 EMS 1 Santera a) su - santera
c) checkDbHealth <localMGCnode> <peerMGCnode>
SanteraDatabases
Example: checkDbHealth <MGC1> <MGC2> -all
NOTE
If errors are exhibited, refer to /space/Santera/db/log/checkDbHealth.log and
call your next level of support.
2 EMS 2 Santera a) su - santera
c) checkDbHealth <localMGCnode> <peerMGCnode>
SanteraDatabases
Example: checkDbHealth <MGC2> <MGC1> -all
NOTE
If errors are exhibited, refer to /space/Santera/db/log/checkDbHealth.log and
call your next level of support.

A30828-X1122-A462-01-76P1 12/22/09
10.3 Run Database Backup
1 EMS 1 Santera Back up database.
EMS 2
a) cd /opt/SANTone/msc/active/cron
b) dbbackup.sh
backup)
Note: ‘crontab –l’ is an alias for the command above.
10.4 Run Switchcheck Script

1 EMS 1 Santer Execute on Active OAM node.
EMS 2 a
%
/opt/SANTone/msc/active/installation/switchCheck/switchCh
eck -b
10.5 EMS Post Checks
Execution
Steps Node User Estimated time of completion: 5min.
1 EMS 1 santera Login to the EMS

2 EMS 1 santera From the EMS look at call/h and success rate average
Utilizing EMS continue to monitor Call/hr, success rate average with the Real
Time Health Monitor at the top of the EMS Configuration tab.
3 EMS 1 santera Verify the alarms that are present in the switch.
Utilizing EMS Fault tab>Alarms and verify no new alarms are present
4 EMS 1 santera Continue to Monitor throughout the procedure.

Utilizing EMS continue to monitor Call/hr, success rate average with the Real
Time Health Monitor at the top of the EMS Configuration tab and Fault tab-
>Alarms Windows for 10 minutes.
5 All All
CAUTION

A30828-X1122-A462-01-76P1 12/22/09
Ensure ALL hiG1800 cards are patched to the correct load. Verify all cards
are completed. Failure to do so could result in call processing degradation.
6 All All Ensure you logout of the switch

A30828-X1122-A462-01-76P1 12/22/09
10.6 Review Summaries
Meeting Minutes Summary Report
Review conclusion by Moderator ====
Action Items
Action Assignee Exp. Resolution Date
Fix the following suggestions
Conclusion: Approved with comments (see below)

General Comments:
Sec/Page Defect/Comment

A30828-X1122-A462-01-76P1 12/22/09

Software Patching Guide: Hig1800 Element Management System (Ems)

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Software Patching Guide: Hig1800 Element Management System (Ems)

Încărcat de

Drepturi de autor:

Formate disponibile

hiG1800 Element Management System (EMS)

Software Patching Guide

Provides software patching instructions for a

hiG1800 Element Management System Software Patching Guide

Copyright © Nokia Siemens Networks 2009. All rights reserved.

2 hiG1800 Element Management System Software Patching Guide

hiG1800 Element Management System Software Patching Guide 3

4 hiG1800 Element Management System Software Patching Guide

1.1 Purpose and Scope

hiG1800 Element Management System Software Patching Guide 5

2.1 TYPES OF APPLICATION BINARIES

2.2 EMS Active/Standby Applications

2.3 EMS Load Sharing Applications

2.4 EMS Platform Applications

2.5 hiG1800 Platform Applications

2.6 Patch Bundle Naming Explanation

2.7 hiG1800 Patch Naming Explanation

hiG1800 Element Management System Software Patching Guide 7

Customer wants to upgrade to: 07.02.40.30

8 hiG1800 Element Management System Software Patching Guide

1 EMS GUI santera Login to the EMS GUI 7.2

Verify Santera Application processes are Active or Hot Standby.

FaultMgr AppMgr Act/Stby Index View on bocalaba1

hiG1800 Element Management System Software Patching Guide 9

Displays Application Status.

6 ALL EMS ROOT Check Disk Space

10 hiG1800 Element Management System Software Patching Guide

7 EMS GUI santera Continue to Monitor throughout the procedure.

hiG1800 Element Management System Software Patching Guide 11

2 EMS 1 santera Copy files to EMS2

12 hiG1800 Element Management System Software Patching Guide

Step Node User Execution

Note: If applications are not in HOT Standby status.

Example: checkDbHealth <EMS2> <EMS1> -all

hiG1800 Element Management System Software Patching Guide 13

Step Node User Execution

2 EMS 2 santera cd to the location of the MsfSwControfile

3 EMS santera Repeat steps above if there are multiple MGs.

3.6 Verify current patch level on EMS

Step Node User Execution

2 EMS 2 santera Repeat above step on EMS2

14 hiG1800 Element Management System Software Patching Guide

Step Node User Execution

Display hiG1800 loads before updating.

ttIsqlCS -connStr dsn=systemconfigurationcs -e "select

3.8 Backup current database on both EMS nodes

Step Node User Execution

3.9 Install Tekbase 1.8 (or Latest Version)

Step Node User Execution

hiG1800 Element Management System Software Patching Guide 15

a) cd /stats/bundles (or your software directory)

16 hiG1800 Element Management System Software Patching Guide

• Verify EMS patch is present on both EMS nodes

Alarms that may occur during patching include:

4.1.1 Verify patches are present on EMS

Step Node User Execution

2 EMS 1 santera Verify appropriate patch(es) ##PATCHID## is (are) in

3 EMS 2 santera SSH to EMS node 2 as ‘santera’

hiG1800 Element Management System Software Patching Guide 17

Step Node User Execution

2 EMS Santera To verify installed patches:

3 EMS Santera To unzip and untar a patch bundle:

/opt/TEKutils/active/GenBundle/installbundle (execute this