PLATFORM REFRESH

INTEGRATION – DATA
PUMPS

7 July, 2014 Xchanging Platform Refresh Programme © 2014 Xchanging Ltd 1

Data Pumps – End to End view

Mainframe
Database
1. Scheduled tasks run on Mainframe to extract data from the operational database
tables in CLASS and POSH and populate “staging” tables with a status flag.
Intermediary
Staging Datastore
(Mainframe)
2. Scheduled tasks (mostly crons) run on Java servers to read from the staging
tables, write to IMR and update staging table status flag. Also clean up old data.
Data Pump Primarily “Class Data Pump” and “ACL Datapump”.

(Non Mainframe)

3. Data is written into IMR (or Pega or other database). Further processes within
the estate carry out checks and processes. (MrACL2, WsGen)
IMR

7 July, 2014 Xchanging Platform Refresh Programme © 2014 Xchanging Ltd 2

 Data Pumps – Most Common pattern – Direct DB

1. Brokers still enter data via UI on the Mainframe into CLASS DB. It populates a “staging” table for copying data
2. The Data Pump is triggered to run and reads from the staging tables, into memory. It then inserts data into IMR DB
3. An ECF User checks the IMR and sees the “pumped” data.
4. 5. Users of other platforms (ECF2-CAS and the Xpresso Tablet App) also see the data on IMR
7 July, 2014 Xchanging Platform Refresh Programme © 2014 Xchanging Ltd 3
Data Pumps – Typical Flow Start

Initialisation

This is the process of the simple Main Loop – while bRunning flag TRUE and retryCount < retryMax

CLASS Data Pump Set Number of Items to be processed to 0

The ACL data pump follows exactly Check

Availability of
Out of Hours
set bRunning flag FALSE
Mainframe
the same pattern with slightly better
DB connection handling / cleardown If getRequestKeys
SELECT ACTION_CODE, UMR, UCR, TR, BUREAU_ID,
STATUS_CODE, CREATED_TIMESTAMP FROM
Data Access Error Returns [DB2-Database].VCLRUDA1
Then increment linkedHashSet WHERE RECORD_TYPE='' AND (STATUS_CODE='' OR DB2
retryCount Of results STATUS_CODE='P')AND UCR >'' ORDER BY Database
ACTION_CODE, CREATED_TIMESTAMP"

The rewrite will combine the two Iterate over the linkedHashList
into one process with better retry Reset

and clean up code.

retryCount
Skip if
Repeat invalid Check Validity
next of data (UMR, UCR,
BureauId and
TrRef) plus lookup
Skip if Policy exists
Locked

Is the
DB record locked
for processing

Process the Action:

Add, Update, Delete, Respond Oracle
Database

If
bRunning Shut
Sleep for set delay
(more to down
do)

7 July, 2014 Xchanging Platform Refresh Programme © 2014 Xchanging Ltd 4

 WsGen Flow Start
WsGen

Initialisation of DB Connection pool

Unlock Records – left locked status A,

set back to P

WSGen provides the interface

for creating workspaces,
Main Loop

projects and transaction

getTransactions
processTransactions

document folders in the IMR initCMSSession – creates an Enterprise Session

CMS based on data copied. 0

Num
Check Admin Group exists
Yes
Transactions >0
>0

It runs standalone but could be No

For each transaction in the list

Admin
combined to run in conjunction
New WorkspaceDTO
Group exists
processTransaction
validateTransaction
with the class data pump which
identifies new claims in the first createPolicy workspace if not found

place. Create Teamroom from policy ID

Set Workteam security policy

It creates Claims as well as createClaim workspace if not found

Policies, Transactions and Create or Update workspace project

Tags within the folders. Set project security policy

Create Transaction Label for document folder

being processed

WsGen also processes

requests from
initCMSSession

processCmrRequestQueue
CmrRequestQueue lockQueue
processEach Req Req Req
Request

Process the Cmr Request

7 July, 2014 Xchanging Platform Refresh Programme © 2014 Xchanging Ltd 5

WsGen Current Package design
The WsGen datapump
contains a number of
generators inheriting from a
base class.

There are a number of other

utilities and wrappers round
the CMS (WorksiteMP API).

The code in the Generators

is similar to the “Create”
classes that are in the
package, which make up
the separate
MarketRepositoryBuilder
utility. They are not invoked
from the generators.

There is one standalone

shutdown client which just
sends a signal to the
WsGen executable to shut it
down cleanly

7 July, 2014 Xchanging Platform Refresh Programme © 2014 Xchanging Ltd 6

Single Fuse ESB server development environment

To commence development a single Linux VM with 2G RAM requires access to the two databases
Connectivity to the CMS API for WsGen can be stubbed initially.

7 July, 2014 Xchanging Platform Refresh Programme © 2014 Xchanging Ltd 7

Current primary Mainframe integrations
EXTERNAL CUSTOMERS
XAG IMR WORKFLOW CWT XDH GeneSys
PROCESS DFV XCS
PROCESS CWS
Workflow PROCESS PROCESS PROCESS
XCS
DP
WORK
XAG XMA IMR XIS FLOW CWS CWT
Workflow PROCESS
DP
PROCESS XDH
eAccount PROCESS GeneSys
ECF s WORK XIS
Tracker WORK XCS Class
FLOW
PROCESS CAS FLOW PROCESS
PROCESS PROCESS
Tracker OSND USER UMR/ Class Class Claims Ref data
ALTAIR
ETL
DP SYNCH WPR ACL DP commen APIX
PROCESS DP DP DP t DP PROCESS

CASA Mainframe B
P
CDW U
R
F Company DDF ATTUNITY CDW
O L Class S
C A XCS Class
CDC ODS I
E M FAC XOL CLASS
S E and TREATY ETL N
S C BATCH CDW E
ENQUIRY
P I other PROCESS ADS S
R C Settlement
S
VTAM LINK

O S Security
C Reference POSH SDW O
E
S
ETL B
SHARED
S
W LORS VOLUME J
P A
LORS E
R L LIDS SDW SDW C
O N ODS ADS
C U
LIDS BATCH T
E ENQUIRY
T PROCSS S
S
S C Settlement other BRIO
I Security
C Reference
S CICS WEB DDF BRIO
TDW
Web
Account Premium Data Multiple times a day PROCESS
Enquiry
Claims Data Once a day
PROCESS

Other Data
ALL CORE SYSTEMS

7 July, 2014 Xchanging Platform Refresh Programme © 2014 Xchanging Ltd 8

Mainframe real-time integration strategy
The Attunity Client and Server should continue to be upgraded to bring them back into support

Attunity would then become a “Tolerate” interface for the foreseeable future

CICS Web should be investigated further as a more standard, lightweight integration method with all
integration to be routed via the Fuse ESB

We also need to prove that CICS Web can be used to invoke REST services on Fuse ESB

We should see if CICS Web can be implemented on Flame (currently it is only on Walnut)

We can look at meeting the ECF requirements for calling non-Mainframe services from the MF

Can the CICS transactions that are currently wrapped by Attunity be called via CICS Web instead?

Assuming CICS Web is successfully implemented.

When a change to an Attunity service is required it should be considered for migration to CICS Web

Media formats could be extended from standard XML or JSON to include Text (i.e. EDI-style messages)

7 July, 2014 Xchanging Platform Refresh Programme © 2014 Xchanging Ltd 9

Future integration with Mainframe via Fuse
EXTERNAL CUSTOMERS
XAG IMR WORKFLOW CWT XDH GeneSys
PROCESS DFV XCS
PROCESS CWS
Workflow PROCESS PROCESS PROCESS
XCS
DP
WORK
XAG XMA IMR XIS FLOW CWS CWT
Workflow PROCESS
DP
PROCESS XDH
eAccount PROCESS GeneSys
ECF s WORK XIS
Tracker WORK XCS Class
FLOW
PROCESS CAS FLOW PROCESS
PROCESS PROCESS
Tracker OSND USER UMR/ Class Class Claims Ref data
PROCESS
DP
Fuse Pipeline Service
SYNCH
DP
WPR
DP
ACL
DP
DP commen
t DP ETL
APIX ALTAIR
PROCESS

CASA Mainframe Fuse B

CDW
P
R
O
F
L
Company
Class
DDF ATTUNITY REST CDW
U
S
C A XCS Class
CICS Web CDC ODS I
E M FAC XOL CLASS
S E and TREATY ETL N
S C BATCH CDW E
ENQUIRY
P I other PROCESS ADS S
R C Settlement
S
VTAM LINK

O S Security
C Reference POSH SDW O
E
S
ETL B
SHARED
S
W LORS VOLUME J
P A
LORS E
R L LIDS SDW SDW C
O N ODS ADS
C U
LIDS BATCH T
E ENQUIRY
T PROCSS S
S
S C Settlement other BRIO
I Security
C Reference
S CICS WEB DDF BRIO
TDW
Web
Account Premium Data Multiple times a day PROCESS
Enquiry
Claims Data Once a day
PROCESS

Other Data
ALL CORE SYSTEMS

7 July, 2014 Xchanging Platform Refresh Programme © 2014 Xchanging Ltd 10

Proposed Data Pumps Rationalisation and Rewrite

Combine Class data pump, WsGen, ACL and MRACL2 into a single camel route rather than four separate jobs. I
have termed this a “Data Pipeline” to distinguish it from data pumps. Note that each component service is
loosely-coupled and reusable, especially important for the CMIS service to create workspace, folders, etc
7 July, 2014 Xchanging Platform Refresh Programme © 2014 Xchanging Ltd 11
Data Pumps Roadmap

Class+WsGen combined into one pipeline service

OSND and Workflow Notification OSND follows pattern of Class+WsGen

ACL, MrACL2 also combined into pipeline

Comments Data Pump to be done

as part of Attunity Upgrade MTBC, Workflow data, User Synch

DRI Report Sender to be Reconciliation jobs – to be discussed as part of IMR Refresh

addressed as part of XDH refresh (EmTrans, RetryUpload, Failed DRI)

7 July, 2014 Xchanging Platform Refresh Programme © 2014 Xchanging Ltd 12