SAP Disaster Recovery Test for

Europe Region
September 2014
Objective & Scope
Objective:
Execute a Planned Disaster Recovery Test of SAP Europe
Environments .

Scope:
Test SAP solutions : ECC, CRM and BW.
Perform a controlled switch to Atlanta Site using EMC disk storage.
Interfaces with ECC, CRM and BW are not part of the scope.
Execute functional tests by SAP specialist support with no data
update.
Identify events that could affect the Recovery Time Objective
(RTO).
Schedule date for the drill: Sep 13th. 2014 - 18:00 hrs (CDT)

DR Test Macro Activities


30 mins 90 min 120 min 120 min 180 min 150 min 120 min
30 mins 120 min 240 min 180 min 180 min 240 min 120 min
Worst Case:
Best Case:
12 Hrs
18 Hrs
Macro Activities
Timeframe 30 minutes 90 minutes 120 minutes 120 minutes 120 minutes 150 minutes 180 minutes
Mexico Time
(CDT)
18:00 - 19:30 19:30 21:30 21:30 23:30 23:30 1:30 1:30 4:00 4:00 7:00
Area/Activity
Preparation
(Before Start Drill)
Stop MTY
Operation
Switch and Start
ATL Operation
ATL Test
Stop ATL
Operation
Switch and Start
MTY Operation
MTY Final Test
Enterprise
Computing
Team UNIX &
STORAGE
Verify SAP backups
availability.
Verify volume groups
consistency .
Verify Gold copy
availabililty in ATL.
Stop SAP DBs
Unmount FS
Switch storage to
alternate site in ATL.
Import volume
groups and mount of
FS
Start SAP DBs

Stop SAP DBs
Unmount FS
Switch storage to
MTY site.
Import volumen
groups and mount of
FS
Start SAP DBs
Verify & Monitor
technical environments
and SAP behaviour.
SAP Security
Lock SAP users
accounts from ECC,
CRM & BW
Unlock SAP users
accounts whom will
participate in test phase
Unlock Users accts
from ECC, CRM & BW
Interface
Monitoring Team
Turn down IPC
interfaces
Turn up IPC interfaces
Verify & Monitor PI
Interfaces behavior
ORACLE DB
Admin
Verify & Adjust DB
parameter to set up
DBs in ATL
Intel Team
Change DNS server
assign IP to ATL
servers
Change DNS server
assign IP to Mty
servers
SAP Basis Team
Re-scheduled jobs
Take Snapshot of
SAP FS & parameter
from ECC, CRM &
BW.
Cancel Jobs
Disable RFC
connections.
Shutdown SAP ECC,
CRM & BW.
Start SAP ECC,
CRM & BW
Verify login access to
SAP Apps.
Shutdown SAP Apps
of ECC, CRM & BW.

Start SAP ECC,
CRM & BW
Verify loging access
to SAP ECC, CRM &
BW
Enable RFC
connections
Release Jobs

Information
security
Stop scheduled
backups during the
outage window

Release Automatic
backup process.
SAP Process
Specialist (GBS)
Review Access &
Authority to SAP
Productions
Environments and
transaction codes.
Verify Test Scripts
and Tables to get the
record count during the
drill.
Get record count fo
most critical tables

CheckPoint: Get & verify
record count before start
test phase (check point).
Perform Health Check of
SAP Applications ECC,
CRM & BW.
Perform Data Inquires &
Reports.
Perform simulation of
real transactions without
data update.

CheckPoint: Get &
verify record count after
switch back to MTY site
(check point).
Perform Health Check
of SAP Applications
ECC, CRM & BW.
Verify SAP Applics
before release
production
environments to end
users.
Directory
Team Location Phone Participant eMail
Change Mgt Mty-E3 Alejandro Benitez fabianalejandro.benitez@ext.cemex.com
Intel Team India Siva Balantrapu (*) sibalant@in.ibm.com
UNIX Team India
Bhaskar Pagilla
Vishal Dakarapu (*)
bhaskar.pagilla@in.ibm.com
vdakarap@in.ibm.com
Storage Team India
Mallikarjuna Kalikiri
Ramesh Bammidi (*)
Venkata Remala
mallikreddyk@in.ibm.com
ramesh.bammidi@in.ibm.com
veremala@in.ibm.com
SAP Basis
Mty

India
Ayax Morales (*)
Hernan Vasquez
Carlos Uresti
Gustavo Carrera
Bharat Devineni
Pradeep Muthojo (*)
Rajnikanth Talanki
ayax.morales@neoris.com
hernan.vasquez@neoris.com
carlos.uresti@neoris.com
gustavo.carrera@neoris.com
devineni.bharat@in.ibm.com
prmuthoj@in.ibm.com
rtalanki@in.ibm.com
Interface Monitoring Remote Andras Bolvary andras.bolvary@neoris.com
SAP Process Remote Roger Altamirano (*) rogerap.mx1.ibm.com
SAP Security Remote 81 13113033 Alfredo Murguia (*) alfredoulises.murguia@ext.cemex.com
Oracle DB Remote Luis Paz (*) luispaz@mx1.ibm.com
Backup On site Jose Flores (*) joseguadalupe.floress@ext.cemex.com
DR Coordinator Mty-E3 81 10669483
Ulises Leon (*)
Carlos Robles (*)
ulises.leon@cemex.com
cargera@mx1.ibm.com
IBM Bridge during the Drill: USA Toll-Free: 888-426-6840 , Mty 8329.6380 - Participant Code: 48458602#
THANK YOU
For further reference, please contact:

Carlos G. Robles
carlosgerardo.robless@cemex.com
cargera@ibm.com


Back Up Slides
Risks Mitigation Approaches
Servers List by SAP Environment
BW
CRM
R3
Host IP ATL IP MTY
mxoccsapcwp01 10.16.135.142 10.26.135.29
mxoccsapcwp02 10.16.135.139 10.26.135.30
Host IP ATL IP MTY
mxoccsapccp01 10.16.135.132 10.26.135.31
mxoccsapccp02 10.16.135.143 10.26.135.32
mxoccsapccp03 10.16.135.144 10.26.135.19
Mxoccsapccp04 10.26.135.33
Mxoccsapccp05 10.26.135.34
mxoccsapccp08 10.26.135.38
Host IP ATL IP MTY
mxoccsapcrp01 10.16.135.136 10.26.135.27
mxoccsapcrp02 10.16.135.146 10.26.135.28
mxoccsapcrp03 10.16.135.137 10.26.135.12
mxoccsapcrp04 10.26.135.13
mxoccsapcrp05 10.26.135.14
mxoccsapcrp06 10.26.135.42
mxoccsapcrp07 10.26.135.49

Relevant Information
Finding
(before the Drill)
Whys Action Plan
1. Long time taken to perform switch to Atlanta.

Cause: MXOCCSAPCRP01 have a dual
storage in MTY and single storage in Atlanta.

Time taken to unconfigure dual storage to
single storage around 2:30 Hrs (expected 30
mins).
R1. It takes too much time to turn down configuration
of mirror of CRP01.
R2. Its the first time & there is not reference of time
for this activity.
1. Add detail of activities to
unconfigure Dual Storage for
CRP01 in checklist.
2. Perform this activity of unconfigure
Dual Storage of CRP01 one day
before next Drill.
Relevant Information
Finding
(during the Drill)
Whys Action Plan
1. NFS Mounts missing:
The following NFS mounts were missing:
CCP03 - /sapmnt/CCP & /usr/sap/trans
CWP02 - /sapmnt/CWP
CRP03 - /sapmnt/CRP & /usr/sap/trans
R1. There is not additional detail for this
activity in checklist.
R2. There was not consider needed.
Detail NFS in checklist and follow through
procedure (after switch to Atlanta, NFS exportfs
should be rexeported with correct IPs)
2. In CCP02 the /etc/hosts file contained
duplicate conflicting entries for CCP01, hence
it was resolving to the MTY IP of CCP01
instead of the ATL IP
R1. It was not fixed in Atlanta.
R2. Validation missed of /etc/hosts in Atl
R3. There was not an specific task for
this activity in checklist.

Update checklist to avoid this issue & follow
through
Consider as best practice a previous activity of
clon & update SO in Atlanta, change IPs and
update etchost entries by Unix team.
3. There was a swapping of host CWP01 &
CWP02 in Atlanta that causes CWP01 was
having the profiles of CWP02.
R1. There were not definition where the
LUNS should be mapped to (VIO server
or directly to the host).
R2. Lack documentation of BW resources
from Atlanta.
R3. There was not an specific task for
this activity in the checklist.
Update checklist to avoid this issue & follow
through.
4. CRP had problem with environment of
crpadm users. The home directory of
/home/crpadm was changed in Atlanta by
changing the owner to 'root' . As a result, SAP
was not able to come up.
R1. This is a local FS and there is not
replicated to Atlanta.
R2. There was not an specific task for
this activity in the checklist.
Update checklist to avoid this issue & follow
through
Consider as best practice a previous activity of
clon & update SO in Atlanta, change IPs and
update etchost entries by Unix team.
Relevant Information
Finding
(after the Drill)
Whys Action Plan
1. Missing functional test for next Applications:
CAPEX
RMS for UK & FR (roll out project is in progress)
GBS ADM Assure all production applications are
consider to be tested at the end of
the Drill.