Change Assessment Document

CR Number

#
1

Basic assessment information

Actual downtime (outage)
(In GMT 24HH:MM format)
within the CR window
27/09/2016 06:30:00 AM IST

Is PRE-Production Testing /
ORT successfully completed?
Do you think this should be
verified by DEV/QA (post
Production deployment)?
Who all are required from
DEV/QA/L1-L2-L3 on standby?

3
4

5
6
7

#
A
A.
1

A.
2

A.
3
A.

CRQ000000008725

NA
No

PIES Sysadmin
Storage Admin
L3 DBAs
Remedy SME
Nexus SME
FNMT SME
Tibco SME
Ebonding SME
SSP SME
IV SMEs
Is the rollback verifiable (only NO
for PRE-PROD CRs)?
Is rollback verified in PRENO
PROD (only for PROD CRs)
1. Avoiding
What is the business benefit
from this change?

Suresh/Tim/Jeremy/Todd
Michael Knoll
Narashim/Tina
Prashanth / Gary / L1L2
Ravikumar / L1L2
Ravikumar / L1L2
Ravikumar / L1L2
Prashanth / Gary / L1L2
Mir
Ankit / Rishabh

the unscheduled reboots.

Risk / Impact information

Risk / Dependencies (in deployment)
1. Switch to previous firmware bundle and fNIC
Describe rollback steps
driver.
2. Start the Oracle Cluster and check all the
databases are up
3. Sanity check on all applications and ticketing
on PNOC Stack , SPP and IV application post
applications restarts
Describe Impact on other PNOC 4.0 Stack. MSFT , SPP and IV
Dependent Application needs restart
system or domain
managers because of this
Remedy
change
Nexus
Tibco
Ebonding
FNMT
IV Mart , Portal
SPP
None
Provide details if any
specific VMs, Devices,
infrastructure etc. is
required
Sanity check on all applications and ticketing on
Describe if any specific

testing required

PNOC Stack , SPP and IV application post

applications restarts
B
Impact on customer (and for how long)
B.1 What features / services will Impact on Remedy, Nexus, Tibco, Ebonding, FNMT, SPP
be impacted because of this Best Case scenario:

No disruption expected in Remedy access,

change?
ticketing and SPP fulfilment.
Example:
Worst Case scenario:
Reports generation

Remedy: We have clustering in remedy so

will be affected for 20
ticketing
will NOT be lost and they will be
minutes,
queued in Tibco. There should not be any issue
Ticketing will be
with User experience also due to clustering in
delayed for 10
place.
minutes etc.)

FNMT & PEMS: There will be downtime of

30 to 60 seconds to restart FNMT and PEMS and
hence there will be loss of events.

eBonding: ebonding will not happen in

worst case

SPP Portal: User should avoid placing

orders during the CR window. Ticketing will be
delayed/queued during the actual downtime
(they will be processed later). This will NOT
affect MSA portal in anyway. SPP portal may be
slow down during switch over..
Impact on assurance Service for all PNOC customers
(IV Mart , Portal)
Both in Best and Worst case

Info vista assurance Portal will not

accessible during the change window during the
Infovista DBs restart

MOP (Method of Procedure)

Task

Resource
(activity
owner)

Communicated to relevant Team

Service Desk , Platops and
Command Center on the
Maintenance Window and alerts
on the components restart
Maintenance on Oracle Node 2
Stop Impacted IV Mart
associated with DBs on Node 2
(IVDB8
IVDB5,IVDB2,IVDB6,ALNPSMA)
Database Backups Need to be
verified
Stop and disable Oracle RAC
clusterware , RAC Databases and
Single Instance DB on Oracle
Node 2
Shutdown Oracle Server

Change Mgmt.
Team

2
3

4
5

Ankit/Rishabh

Michael Knoll
Narashim

Suresh/Tim

Task
time
(in GMT
24:00
hrs.
format)

Contingen Actual
cy Time
Completi
to kick
on Time
rollback

7
8
9
10
11
12
13
14

15

Perform Firmware Upgrade

Power on Server and perform
Driver Upgrade
Verify LUN Path issue and
firmware is upgraded
Start Oracle Cluster on Node2
Start RAC Databases and Single
instance DB on Node2
check apps on node
1(fnmt,remedy,Tibco,IV,ebonding
pems and SPP)
restart anything if necessary
Sanity check on all
applications and ticketing on
PNOC Stack , SPP and IV
application post applications
restarts
If Everything is stable repeat
same on node 1

16
17

Maintenance on Oracle Node 1

Same steps as in Node 2

#
A
B

Other mandatory tasks

Update CMDB
Close CR

Todd
Suresh/Tim
Suresh/Tim
Narashim
Narashim
SMEs need to
check their
applications