Where Value & Innovation co-exist

Testing Automation

CORE and TAM Road Map

October , 2005

Raja Ramesh
 Where Value & Innovation co-exist

Agenda
 Why do we need test automation?
 What can be test automated?
 Need for automation at SHPS
 Automation objectives
 Automation script architecture
 Automation MANAGER
 CORE automation plans
 TAM automation plans
 Challenges on hand
 Automation demo
 Discussion

October 14, 2008 2

 Where Value & Innovation co-exist

Why do we need test automation?

 Speed up testing to accelerate releases
 Allow testing to happen more frequently
 Reduce costs of testing by reducing manual labor
 Improve test coverage
 Ensure consistency
 Improve the reliability of testing
 Allow testing to be done by staff with less skill
 Define the testing process and reduce dependence on the few who know it
 Make testing more interesting
 Develop programming skills

October 14, 2008 3

 Where Value & Innovation co-exist

What can be test automated?

 Regression testing
 Tests that can run on every build are excellent candidates for automation
 Configuration tests
 If testing required for different platforms, automation may help
 Test bed setup
 Automating setup procedures will save time
 Non-GUI testing
 It’s almost always easier to automate command line and API tests than it
is to automate GUIs
 Build readiness
 QA time can be better utilized if unstable builds can be identified through
automation

October 14, 2008 4

 Where Value & Innovation co-exist

Need for automation at SHPS

# Item CORE TAM
1 Total Applications 41 18 *
2 Test Cases 2006 2055
3 Hours for Manual Execution 171 Man Hours 740 Man Hours
4 Frequency of Regression Testing 3 to 4 time/release ** 1 time/release

5 Sanity testing (Man Hours) 2-3 3-4

6 Frequency of Sanity Testing Every Build Every Build
* TAM CSR is executed for 23 clients and TAM PPWEB executed for 18 client.
** Few application are executed only once a release. These applications occupy 40-50 Man hours of CORE
regression and 10-12 hours of TAM regression.

October 14, 2008 5

 Where Value & Innovation co-exist

Need for automation SHPS cont…

 Multiple regressions is not possible

 CORE takes 21 man-days for one iteration
 TAM takes 93 man-days for one iteration
 Cost of labor
 Reasonable labor is being spent on manual testing
 Limitation on aggressive schedules
 Short release cycles not possible
 Parallel releases not possible
 No tool/mode for confirming build readiness
 Manual cursory testing
 3-4 hours effort
 Prone to skip test scenarios
 Due to repeated testing of test cases, it is possible for the tester to
overlook/assume correctness of some scenarios

October 14, 2008 6

 Where Value & Innovation co-exist

Automation objectives
 Reduce the regression testing time by at least 60-70%
 Provide quick sanity testing facility at onsite
 Improve the regression testing execution frequency
 Develop scripts with minimal maintenance time
 Provide facility for everyone to run automation scripts
 Provide facility to receive automation test results through email

October 14, 2008 7

 Where Value & Innovation co-exist

Automation overview
 Where Value & Innovation co-exist

Script architecture - Guidelines

 External data source
 Generic code
 Flexibility
 Controllability
 Maintainability
 Operability
 Reporting ability

October 14, 2008 9

 Where Value & Innovation co-exist

Script architecture – Baseline suite

Start

.INI
Driver Script Invoke the
File Get Test Data
Application

Get GUI File Application

Path / Exception Script Navigate to the
Data Database Screen
Each Test Case

Database GUI
GUI Verification Yes
Verify?
Database
Next

No
Results

Results Generate Functional Test

Checks
Report Case execution

Stop

October 14, 2008 10

 Where Value & Innovation co-exist

Script architecture – Participant WEB

STAGE 1: DB Module STAGE 2: GUI Module

Start Each Client Each Client

CORE DB

Get Active Links for

Partcipant Menu and Generate SQA
Load all Clients Path
DirectorMenu GUI Map file Database
(Active)

SQA
Insert Data
Database

Next Next

STAGE 3: Integrated Module

Each Client
SQA
Database

Update Stop
Load GUI Map Result
Generate Report (email, excel)
Link Checker

Next

October 14, 2008 11

 Where Value & Innovation co-exist

Script architecture – Cursory testing

Start

Execute
Driver Script

For each Launch the

Modul/App. Module/App.

Execute the
Get Test Cases
Parent TC

Execute rest of SQA

Pass? Yes
the Test Cases Database

No
Next

Genearte Report

Report
(Email, Excel) Stop

October 14, 2008 12

 Where Value & Innovation co-exist

Automation MANAGER
 What is Automation Manager
 A utility to ease the operation folks jobs
 Can be executed by staff with less testing skills
 No dependency on offshore SQA
 ON-FLY configure
 Applications that needs to be tested
 Test cases that needs to be executed (single, multiple or a range)
 Test results communication (email, results in shared folder etc)

October 14, 2008 13

 Where Value & Innovation co-exist

Automation MANAGER – process flow

SCOPE: User / Tester SCOPE: Automation Manager
Start

Choose the .INI file

Initiate (Remote file)
Application to
Automation
execute
Manager
(Single / Multiple)

Update
Trigger
Launch
PsExec
PscExec
Choose the Test Enter email Batch file
(Remote)
Cases for the app. address for (Remote)
(Single / Multiple / receiving test
Range) results

Driver Script Path

SCOPE: WinRunner Server

Trigger
Load SQA
WinRunner Generate Report
Driver Script Database
Batch file

Result

Each Application
Report
Load Retreive Execute (email / Excel) Stop
Application Script Test Data Test Cases

Next

October 14, 2008 14

 Where Value & Innovation co-exist

Automation MANAGER - prototype

Continued…

October 14, 2008 15

 Where Value & Innovation co-exist

Automation manager – prototype cont…

October 14, 2008 16

 Where Value & Innovation co-exist

CORE Automation
 Where Value & Innovation co-exist

Current automation status - CORE

# Application Total Cases Cases Hours for Manual Hours
Automated Manual with current
Execution Automation
1 CSR 173+30 48 10 7
2 HR Access 224+143 59 12 8
3 Case Setup * 119+126 103 22 NA

* For Case Setup, scripting is completed. Test data needs to be added.

1 Total regression test cases 2006*

2 Total test cases automated 107 **
3 Effort required to perform total regression manually (Man hours) 170
4 Effort required to perform regression with current automation (Man hours) 163

* Includes functional & end-to-end cases for all the 41 CORE applications.
** Does not include 103 case setup test cases for which, test data still needs to be added.

October 14, 2008 18

 Where Value & Innovation co-exist

Automation scope - CORE

# Regression Testing Cursory Testing
1 CSR Case Setup
2 HR Access CSR
3 Case Setup HR Access
4 Participant Web * Participant Web
5 Load Passwords CallDoc
6 CallDoc Template Setup
7 Knowledge Base Database checks
8 STD Load
9 AddrPBSI
10 Load PBSI
11 LoadEmailAddress
12 Template Setup
13 Survey Setup

October 14, 2008 19

 Where Value & Innovation co-exist

Proposed automation road map - CORE

PHASE VI PHASE I
- CallDoc, KB, STD Load, - Sanity Test Automation
I
- Load Password, Address PBSI, - Save 3 hours per build
[ Mid of Oct 05]
- Load Email Address, Load PBSI,
PHASE II
- Survey Setup, Template Setup
II - Case Setup & Template Setup
- Save 27 hours per build [Mid of Nov 05] - Save 18 hours per build

PHASE VII PHASE III

III
- Test Cases Version Control - Participant WEB
[End of Oct 05]
- Test Data Automation - Save 07 hours per build
IV PHASE IV
[TBD] - Client Specific
- Save XX hours per build
V
[End of Nov 05] PHASE V
- End to End cases Automation
VI - Save 21 hours per build
[End of Nov 05]

VII
[TBD]

October 14, 2008 20

 Where Value & Innovation co-exist

Road map in detail – CORE

# Phase Applications Cases Expected Start Date End Date
Included Hours
1 Phase1 Cursory tests 30-40 420 9/26/05 10/12/05
2 Phase2 Case Setup 298 192 10/13/05 11/15/05
3 Phase3 Participant Web 160 600 9/12/05 10/31/05
4 Phase4 Client Specific tests TBI TBI TBD TBD
5 Phase5 End-to-end testing 200 480 10/13/05 11/23/05
6 Phase6 CallDoc, KB, STD Load, 220 608 9/19/05 11/28/05
Load Password, Address
PBSI, Load Email
Address, Load PBSI,
Survey Setup, Template
Setup
7 Phase7 Miscellaneous – version TBI TBI TBD TBD
control & test data
reconstruction

* Often, multiple phases are executed in parallel. Team is already working on Phase2, Phase3
& Phase6

October 14, 2008 21

 Where Value & Innovation co-exist

SUMMARY: CORE automation

Total regression cases (all applications functional & end-to-end) 2006
Total time required if regression is done manually (Man Hours) 170
# Phase Total Cases Total Hours Manual efforts still required to
Automated Required Automate complete regression (Man Hours)

1 Before 107 NA 163

Phase1
2 Phase1 30-40 420 NA
3 Phase2 298 192 145
4 Phase3 160 600 138
5 Phase4 TBI TBI NA
6 Phase5 200 480 117
7 Phase6 220 608 90 *
8 Phase7 TBI TBI NA
* There are about 30 applications that are executed only once a release. These applications
occupy 40-50 hours of total regression time.

October 14, 2008 22

 Where Value & Innovation co-exist

TAM Automation
 Where Value & Innovation co-exist

Current automation status - TAM

# Application Total Cases Cases still Hours for Manual Hours
Cases Automated need to be Manual with current
Automated Execution Automation
1 TAM CSR 537 395 54 342 123.2
2 Participant Web 362 342 10 188 31
4 TAM Case Setup 705 00 705 11 NA

* CSR cases are executed for 23 clients and case setup cases are executed for 18 clients

1 Total regression test cases 2055*

2 Total test cases automated 737
3 Effort required to perform total regression manually (Man hours) 740
4 Effort required to perform regression with current automation (Man hours) 364

* Includes functional & end-to-end cases for all the 41 CORE applications.

October 14, 2008 24

 Where Value & Innovation co-exist

Automation scope – TAM

# Cursory Testing Regression Testing
1 Core Case Setup 11 STD Daily TAM CSR
2 TAM Case Setup 12 STD Daily Cigna TAM Case setup
3 TAM CSR 13 STD Daily CNA TAM Participant Web
4 TAM Batch 14 STD Daily LEA
5 Auto Close 15 Intel TAM Out
6 Calculations 16 Participant Web
7 Email Batch 17 HR Access
8 Letter Batch
9 Printed Letters
10 TAM Termination
Batch

October 14, 2008 25

 Where Value & Innovation co-exist

Proposed automation road map - TAM

Current Status

PHASE V - TAM CSR – 70% case automated

- Test Cases Version Control - Participant WEB – 95% cases automated

- Save 40-50 hours of test result - Saving 376 hours per build
review time per build
Current PHASE I
- Sanity Test Automation

PHASE IV - Save 3-4 hours per build

I
-Test Data Automation PHASE II
-Save script maintenance time - TAM Case Setup
II
- Save 13 hours per build

PHASE III
III - End to End cases Automation
- Save 4 hours per build

IV

October 14, 2008 26

 Where Value & Innovation co-exist

Road map in detail – TAM

# Phase Applications Cases Expected Start Date End Date
Included Hours

1 Phase1 Cursory tests 40-50 250

2 Phase2 CSR, Case Setup & HR 769 416
Access
3 Phase3 End-to-end testing 32 150-170
4 Phase4 Test data reconstruction All TBD

5 Phase5 Miscellaneous TBI TBD

* Often, multiple phases are executed in parallel.

October 14, 2008 27

 Where Value & Innovation co-exist

SUMMARY: TAM automation

Total regression cases (all applications functional & end-to-end) 2055
Total time required if regression is done manually (Man Hours) 740
# Phase Total Cases Total Hours Manual efforts still required to
Automated Required Automate complete regression (Man Hours)

1 Before 737 NA 364

Phase1
2 Phase1 40-50 250 NA
3 Phase2 769 416 351
4 Phase3 32 150-170 347 *
5 Phase4 TBI TBD NA
6 Phase5 TBI TBD 300-310

* On average, 200 man hours are spent on testing letters & emails application, which can not
be automated. Also, there are about 10 applications that are executed only once a while.
These applications occupy 10-12 hours of total regression time.

October 14, 2008 28

 Where Value & Innovation co-exist

Challenges on hand
 Research
 Extensive research is required to implement Phase5 & Phase6 (CORE)
 Automatic Test Data
 Writing queries to pick test data from database requires a lot of additional study on the
system and tool.
 Support from other teams
 Phase4 (CORE) requires support from Business development and Tech OPS teams
 Prioritization of application automation
 SQA team needs SHPS support to finalize the list of applications for sanity testing
 WinRunner instances
 Require multiple instances of WinRunner so that multiple tests can be executed
simultaneously
 QuickTest Pro tool availability
 QuickTest Pro, a new regression testing tool is required to automate Knowledge Base

October 14, 2008 29

 Where Value & Innovation co-exist

Automation DEMO
 Where Value & Innovation co-exist

Discussion
 Where Value & Innovation co-exist

Thank you!