Documente Academic
Documente Profesional
Documente Cultură
1.2 Overview
The scope of this document is the Technical details for FR5.4 Monthly processing from the AMCBANK
(ICRA) Detailed Requirements document. The following are the details of the requirement:
1) Extract and retain Monthly full files from the Source Systems (that is all records) to populate
month-end tables. This process:
a)
Expedi
te
s
th
e
re
co
ve
ry
or
co
rre
cti
on
pr
oc
es
se
s
b)
Facilit
at
es
th
e
re
co
nci
liat
io
n
to
st
an
da
rd
m
on
th
en
d
re
po
rts
or
fin
an
cia
l
re
po
rts
2) Retain copies of the Source System and Monthly extracts as per the following:
Monthly snapshot data should be retained for 13 months
3) The system should also support the purging of data based upon the aforementioned data
retention parameters, or additional parameters established at the local or regional level
This Backup and Restore process is mainly designed to rerun the cycles from the required date. This
result and data files are mainly for Support Teams only and not for the business users.
This activity is not applicable to restore the Initial Cycle data. If there is
any issue with the Initial Cycle data, it is required to start from the
beginning of the cycle.
This activity is not applicable to restore the History tables.
If there is any issue with History data, it is required to rebuild the
warehouse (WH) with Daily backup data process and complete the
required process.
2 Monthly backup process and flow
This topic illustrates a high level process flow diagram for Monthly Data Backup and Restore
processes.
The Backup, Archive scripts and DB Import or Export scripts are placed as part of Monthly Control M
schedule which runs at the end of every month.
Generalized process for backup
AMCBANK_Monthly_BKP_Process.sh
Refer to the following points regarding the script, AMCBANK_Monthly_BKP_Process.sh:
This script exports the table data to .ixf files and the same files are archived by
AMCBANK_Monthly_Archive_Process.sh script.
This script runs on a Monthly basis
This script backups the following Source System tables:
HUB tables
BASEL tables
IFAI tables
HUBODS tables.
Script archives export message files (for future reference), which are
generated while creating the .ixf files.
$prmDirlogbkp/AMCBANK_Monthly_ixf_msgs.$Process_date.tar.gz
Monthly Flat Files - Archive process: It picks all Monthly source flat files from respective
directories based on Source System Input filelist and builds into single tar file for each Source
System. It further compresses all tarred files and moves to Archive directory. Refer to the
following Input files:
Basel_Monthly_filelist
External_Monthly_filelist
HMI_Monthly_filelist
XXXXX_Net_Monthly_filelist
OHC_Monthly_filelist
Parameter_Monthly_filelist
Monthly Output files generated by script:
$prmDirdatabkp/AMCBANK_Monthly_OHC.$Process_date.tar.gz
$prmDirdatabkp/AMCBANK_Monthly_BASEL.$Process_date.tar.gz
$prmDirdatabkp/AMCBANK_Monthly_XXXXX_Net.$Process_date.tar.gz
$prmDirdatabkp/AMCBANK_Monthly_PARAMETER.$Process_date.tar.gz
$prmDirdatabkp/AMCBANK_Monthly_EXTERNAL.$Process_date.tar.gz
$prmDirdatabkp/AMCBANK_Monthly_HMI.$Process_date.tar.gz
Daily Accumulate files - Archive process: Script picks all Daily Archive files (that are generated by
Daily Backup and Archive process) for specified date. It further builds into single tar file for each
Source System.
Suppose specified date is 2010-06 then it picks all 30 June month
Daily archived files for particular Source System.
For example: For BASEL system Daily archived files:
AMCBANK_Daily_BASEL.2010-06-01.tar.gz
AMCBANK_Daily_BASEL.2010-06-02.tar.gz
AMCBANK_Daily_BASEL.2010-06-03.tar.gz
--------------------------------------------------
-------------------------------------------------
AMCBANK_Daily_BASEL.2010-06-29.tar.gz
AMCBANK_Daily_BASEL.2010-06-30.tar.gz
Script tars previous 30 files as one tar file
(AMCBANK_Daily_Accumulate_BASEL.$Process_date.tar) for BASEL
system.
Purging Process for Old files: Apart from archiving the input files,
AMCBANK_Monthly_Archive_Process.sh script also checks the already backed up Monthly
files and deletes them if their Retention Period is more than 13 months. Retention Period is
passed as parameter to the script. The format YYYY-MM refers to the year and month part. For
example: 2010-06 for June 2010.
Pseudo Code for Retention: If mtime** of file is older than the Retention Period, delete the
Archive files, else retain the Archive files in the same location. ** mtime is the modification time
of the file.
2.2.1 HUB, HUB-REM and GPS Source System (iSeries)
The following figure illustrates the HUB, HUB_REM and GPS Monthly Backup process for ICRA:
HUB/HUB-REM/GPS:
HUB-REM and GPS are part of HUB System
On Monthly basis, after the post critical batch backup, HUB team loads HUB tables which are
required for backup and archive process into new staging libraries. New Staging libraries are
created as part of the Backup process.
DBA script (AMCBANK_stg_load_recovery.sh) is run to extract data from the previous
iSeries library to the pSeries Monthly Staging Schema. New pSeries Monthly Staging Schema
is created as part of the Backup & Recovery Process.
DBA Export or Import scripts identifies the critical batch data changes based on the
MODE_FLAG filter which has a value of Blank, P for Primary records and S for shadow
updates or with help of other transaction date/time fields wherever applicable and import the
same to pSeries Monthly Schema.
If the table has MODE_FLAG columns then the script reads all the latest records based on
MODE_FLAG = P and condition
If the table does not have MODE_FLAG Columns and it belongs to the Transaction journal/
table then all the records are filtered based on Posting Date or Capture date
If the table does not have MODE_FLAG Columns and if it does not belong to Transaction
tables then full images are taken as a backup
The pSeries tables are backed up to .ixf files and are archived and stored for 13 months
HUBODS:
For the tables which are directly getting loaded from HUBODS tables, the backup process is
different. New Monthly jobs are created which captures the full image into the reprocessing
tables of the Monthly schema. Please refer to Appendix for the list of new jobs.
New jobs have the PPN_DT condition changed to <= (#prmDate#) so that full image till month
end is captured. Target reprocessing tables is in Write Truncate mode in order to capture the
fresh image of data till month end.
These jobs run after the regular cycle is completed and as part of the Monthly backup process
The previous reprocessing tables are backed to IXF files and is archived and stored for 13
months.
Archive Script generates the following output files for HUB/HUBODS Source System:
AMCBANK_Monthly_HUB_ixf.YYYY-MM.tar.gz
AMCBANK_Monthly_HUBODS_ixf.YYYY-MM.tar.gz
BASEL:
For BASEL we have both Daily files and monthly files as sources. In addition, we need to
backup the HEW_PD and HEW_CRR WH tables data in order to retain proper history.
AMCBANK_Monthly_BKP_Process.sh script exports the HEW_PD and HEW_CRR tables
data to ixf files and moves these files to backup landing directory
AMCBANK_Monthly_Archive_Process.sh script picks up the WH ixf files from the backup
landing directory, tars and zips them to a single monthly ixf file
AMCBANK_Monthly_Archive_Process.sh script picks up the monthly BASEL files from the
landing directory, tars and zips them to a single monthly file
AMCBANK_Monthly_Archive_Process.sh script picks up the archived Daily files from the
Daily archive directory and tars them to an accumulated monthly file
The previously mentioned monthly version files (ixf, Daily and monthly) are archived to monthly
archive directory
Archived Monthly files are retained for 13 months and any files which are older than 13 months
are purged.
Script generates the following output file for BASEL Source System:
AMCBANK_Monthly_BASEL_ixf.YYYY-MM.tar.gz
AMCBANK_Monthly_BASEL.YYYY-MM.tar.gz
AMCBANK_Daily_Accumulate_BASEL.YYYY-MM.tar
XXXXX Net:
For XXXXXnet we have both Daily files as well as Monthly files
The Daily files has complete image till date, so for Monthly backup and archive purpose we
need to consider only the month end Daily files which were archived by the Daily archive
process
Monthly files are readily available in the landing directory
AMCBANK_Monthly_Archive_Process.sh script picks up the month end archived Daily files
from Daily archive directory and tars them to an accumulated Monthly file
AMCBANK_Monthly_Archive_Process.sh script picks up the Monthly files from landing
directory, tars and zips them to a single Monthly file.
Previously Monthly version (Daily and Monthly) files are archived to Monthly archive directory
The Archived Monthly files are retained for 13 months and any files which are older than 13
months are purged.
Script generates the following output files for XXXXX_Net Source System:
AMCBANK_Monthly_XXXXX_Net.YYYY-MM.tar.gz
AMCBANK_Daily_Accumulate_XXXXX_Net.YYYY-MM.tar
2.8 IFAI Source System
The following figure illustrates IFAI Monthly Backup Process for ICRA:
External System:
External Source System (like Insurance) provides the Monthly files in Monthly landing directory.
AMCBANK_Monthly_Archive_Process.sh script picks up the Monthly files from the Monthly
landing directory (needs confirmation), tars and zips them into a single mentioned Monthly files
and place them in Monthly archive directory.
Archived Monthly files are retained for 13 months and any files which are older than 13 months
are purged.
BASEL is also part of External Source System but the backup and archive process differs from
other External Source systems. Please refer to BASEL backup and archive process for more
details.
Script generates the following output file for EXTERNAL Source System:
AMCBANK_Monthly_EXTERNAL.YYYY-MM.tar.gz
sh
/data/ds/<Project>/scripts/AMCBANK_Monthly_Recovery_PrmFileUpdate.sh
/data/ds/<Project>/scripts/<ctcd> bir2_wh_Rec_new.param
InputFile_AMCBANK_Recovery_PrmFileUpdate R
During original run all the external files are read from following
directory
Server Path: \<project>\landing\<CountryCode>\
But during recovery process all the files are available in backup
landing directory, so it is required to update all the parameters to read
the data from the following folder during recovery process.
Server Path: \<project>\landing\<CountryCode>\ <Recovery>
5.5 External
The following figure illustrates External Monthly Recovery Process for ICRA:
5.7 IFAI
This topic provides details about IFAI Recovery Process as displayed in the following figure:
5.8 HMI
The following figure illustrates HMI Recovery Process for ICRA:
Refer to the following points regarding AMCBANK User Reference Data Recovery Process:
1) Validate the impact analysis and identify the required Source System, tables and date for
recover, if AMCBANK User Reference data impacted than follow the steps to recover
2) Make sure all the prerequisites are met.
3) Run the AMCBANK_Monthly_Untar_Process.sh script with
AMCBANKUSERREF_MONTHLY.txt
4) Once Untar script is completed, please check the backup landing directory and make sure the
required files to start the recovery process are restored. Use the Source System wise excel
sheet attached in the appendix for verification
5) Run all the AMCBANK User Reference (Parameter) related jobs pointing the source file
location to the backup landing directory where the restored files are present
6) When other systems are also being recovered, wait till the corresponding interface jobs are
done and then run the Summary and Snap Shot process to load SNAP and SUMMARY tables.
7) Validate and confirm the recovery process and data population before start the ongoing
process
Source_Systemwise_
Files_Tables.xls
Library Analysis.xls
Consolidate
J ob_table_file Details.xls
HUBODS Inventory:
\\
HYDNFIL06.corp.capgemini.com\users$\nanali\My Documents\OHBI\FR5.4\Monthly\HUBODS_Inventory.xls
ICRA Integration
Change Control For FR5.4 Monthly.xls
sh /data/ds/dpr_gacpt_dev/scripts/AMCBANK_Monthly_BKP_Process.sh
/data/ds/dpr_gacpt_dev/scripts/bn bir2_wh.param
InputFile_AMCBANK_Monthly_BKP_Process INST<1 to N>
The following figure illustrates the Backup Process for AMCBANK ICRA:
sh
/data/ds/dpr_gacpt_dev/scripts/AMCBANK_Monthly_Recovery_Process.sh
/data/ds/dpr_gacpt_dev/scripts/bn bir2_wh.param
InputFile_AMCBANK_Monthly_BKP_Process ALL INST<1 to N>
To restore the entire Source System data need to mention the Source
System code as ALL for all instances.
To restore the Source system wise Data use following command lines.
To restore the IFAI Source System:
sh
/data/ds/dpr_gacpt_dev/scripts/AMCBANK_Monthly_Recovery_Process.sh
/data/ds/dpr_gacpt_dev/scripts/bn bir2_wh.param
InputFile_AMCBANK_Monthly_BKP_Process IFAI INST1
sh
/data/ds/dpr_gacpt_dev/scripts/AMCBANK_Monthly_Recovery_Process.sh
/data/ds/dpr_gacpt_dev/scripts/bn bir2_wh.param
InputFile_AMCBANK_Monthly_BKP_Process BASEL INST2
sh
/data/ds/dpr_gacpt_dev/scripts/AMCBANK_Monthly_Recovery_Process.sh
/data/ds/dpr_gacpt_dev/scripts/bn bir2_wh.param
InputFile_AMCBANK_Monthly_BKP_Process HUBODS INST3
sh
/data/ds/dpr_gacpt_dev/scripts/AMCBANK_Monthly_Recovery_Process.sh
/data/ds/dpr_gacpt_dev/scripts/bn bir2_wh.param
InputFile_AMCBANK_Monthly_BKP_Process HUB INST<4 to N>
The following figure illustrates the Recovery Process for AMCBANK ICRA: