Making the QA-QC Marriage Pay Off

SASPIN Presentation - v.1.0 2001

Presenters Data
Jorge Boria wrote his first program in Assembly Language in 1966,
and has since been trying hard to recover from the effort.
He has managed to convince universities to give him titles early and
positions later in life. In the process, he has performed every job
that the community holds unsacred, and some that it holds
repellent, such as manager.
He has done some unspeakable things, like writing books telling
people how things are, about which things he has frequently
changed his mind.
Confronted with the challenges of holding a regular job, he has
become the ultimate gipsy: a software engineering process
consultant.
He is very grateful to TeraQuest for allowing him to play with the big
guns and just loves to mingle with his fellow teraquesters in all
occasions this is possible.
He is extremely happy to be in San Antonio, a city he visits as often
as he can, and thanks you enormously for this. You can find his
contact information in the very last slide of this presentation.
SASPIN Presentation - v.1.0 2001

Agenda
QA vs. Testing
Context of the Case in Point
Dealing with Success
Lessons Learned

SASPIN Presentation - v.1.0 2001

What is Testing?
Testing as a noun is the task of uncovering errors in the
product

Testing is NOT showing that the program works

SASPIN Presentation - v.1.0 2001

What is Testing?
As a noun, testing is a profession requiring many talents

Discipline
Intelligence
Domain knowledge
Good working habits
Teammanship

Testing professionals cover many roles

Requirements reviewers
Test Suite developers
Test Suite executors
Test results reporters
Work product issue reporters
SASPIN Presentation - v.1.0 2001

A Good Tester
is Effective

is Efficient

Develops and uses high yield

test cases

Uses only high yield test

cases

Selects high yield paths from

the products maze

Reports concisely

Makes accurate decisions on

the ROI of testing effort,

Understands possible
workarounds and uses them

Reports the defects so that

they can be reproduced

SASPIN Presentation - v.1.0 2001

Discussion: What is High Yield Testing?

What is a high yield case?

What are the obvious benefits of high yield cases?

How can you promote the use of high yield cases?

SASPIN Presentation - v.1.0 2001

A Good Tester
is Adaptable

is a Team Player

Can switch from one

component to another
Can switch from one product
to another
Can work with new products

Understands and follows the

software process
Understands standards being
implemented by the product
team (motif, COM, product
style guide...)
Understands testers role in
the quality system
Can participate in technical
reviews with profit
Follows up on issues
Can work with a testing
automation specialist

SASPIN Presentation - v.1.0 2001

A Good Tester
is a good error reporter

Executes test cases

properly
Can discern errors from
desired behavior in the
software
Can extrapolate from
testing to business
environment
Produces succinct, no
fat error reports
event driven, complete
descriptions
meaningful in terms of
existing issues history
Judges severity as a user
would

has Solid Domain Knowledge

Writes good test procedures

and cases
Is able to technically support
issues that are challenged
Defends ideas and issue
reports
Can report theoretical flaws in
the product

SASPIN Presentation - v.1.0 2001

What is Quality Assurance?

Schulmeyer (Handbook of SQA)

Systematic activities providing evidence of the fitness for use

of the total product

Software CMM

Reviewing and auditing the products and activities to verify

that they comply with the applicable procedures and standards

and providing the project and other appropriate managers with
the results of these reviews and audits

Example alternate names for SQA

Process Assurance
Process Quality Consulting
Process Quality Assurance
Software Quality Engineering
SASPIN Presentation - v.1.0 2001

10

QC is not Quality Assurance

QC (Testing)
Goal: detect problems
in the work products

QA
Goal: ensure adherence
to processes,
standards, plans
Focus: product content; Focus: project process;
quality control (after
quality assurance (built
the fact)
into the product)
Activities: develop and Activities: guide and
run tests, check results monitor (audit)
processes used; review
are correct
results of tests; ensure
tests were done
SASPIN Presentation - v.1.0 2001

11

Capability Maturity Model (CMM)

Level 5:
Optimizing

Continuous
process
improvement

Level 4:
Managed

Process
control

Level 3:
Defined

Process
definition

Process
discipline

Level 1:
Initial

Level 2:
Repeatable

Change
management

Quantitative
management

Engineering
management

Project
management

SASPIN Presentation - v.1.0 2001

12

Structure of the CMM

Maturity Level
Key Process Areas
Goals
Common Features
Activities
Performed

Commitment
to Perform

Ability to
Perform

Institutionalization
Items

Measurement
and Analysis

Verifying
Implementation

Key Practices

SASPIN Presentation - v.1.0 2001

13

SASPIN Presentation - v.1.0 2001

Process Change Management

Technology Change Management

Defect Prevention

Software Quality Management

Quantitative Process Management

Peer Reviews

Intergroup Coordination

Software Product Engineering

Integrated Software Management

Training Program

Organization Process Definition

Organization Process Focus

Software Configuration Management

Software Quality Assurance

Software Subcontract Management

Software Project Tracking and Oversight

Software Project Planning

Requirements Management

SQA in the SEI CMM

Both a key process area and verification of most others

Verification 3 - The SQA group reviews and/or audits the activities and work products for ...

14

SQA Verification Role

In many key process areas, Verification 3

asks that the SQA group review and/or audit the activities

performed
review and/or audit the work products created
report on the results

This provides the objective view to ensure

the processes are being used

the results are useful

SASPIN Presentation - v.1.0 2001

15

SQA Role Evolves

Initial role of SQA

ensuring compliance to project plans

ensuring compliance to organization policy
Eventual role of SQA is proactive

early involvement with project team, advising on selection and

tailoring of organization processes

reviewing procedures, plans, and standards early in the project
preparing quality plans
partnering in process and product improvement
recommendations

SASPIN Presentation - v.1.0 2001

16

Quality Assurance Purpose

Provide teams and management with visibility into the
process being used and into the products being built
Review and audit the products and activities to verify
they comply with procedures and standards
Provide teams and management with results of reviews
and audits
Assist in defining plans, standards, and procedures
Address non-compliance issues

SASPIN Presentation - v.1.0 2001

17

Problems with QA
QA group (individuals) seen as process police
QAs alignment with corporate goals fuzzy
QAs alerts are not heeded by the development team
QA lacks leverage to enforce process

SASPIN Presentation - v.1.0 2001

18

Agenda
QA vs. Testing
Context of the Case in Point
Dealing with Success
Lessons Learned

SASPIN Presentation - v.1.0 2001

19

Context - the Company

Fast growing federal & state e-commerce company

Ten-fold growth in two years

Went from one product in the first five years to multiple ones
based on initial product

Project size

Median size four person/month effort per project

Largest size today is nine months with about twelve persons
Largest (historically) three years with twenty five people
Cash-cow project

Original development took a mid-size team over two years

Most projects are upgrades of or variations on original product

SASPIN Presentation - v.1.0 2001

20

Context - the Foundation

Y2K Success Story

Project started in mid-1998

Had one project manager and one test manager throughout
Based on strict test-debug process

Formal testing practices were enforced

Testers were mix of permanent personnel and contractors
Product showed no defects over the millennium
But the pizza party was great!
Defects identified were 30% date related, 70% non-Y2K

Developers found defects that had been dormant for

years
In the wake of success, the Quality Department had
established its value and place in the organization

SASPIN Presentation - v.1.0 2001

21

Context - the Process

Post Mortem Project Review
Requirements
(SDS)

UAT Execution
(SDS)

UAT Test Planning and Preparation

Phase End Review

Acceptance
Specifications
(TSD)

Te
st

Sys Test Execution

(SDS)

System Test Planning and Preparation

Te
st

Phase End Review

Acceptance
Coding
(SDS)

Unit Test Planning

and Preparation

Acceptance
Developed Components
(SDS)`

SASPIN Presentation - v.1.0 2001

Unit Test Execution

(SDS)

Ha

Re
po
r

Re
po
r

nd
Of
f

22

Problems in Paradise

Original Sins

Developer-owned code
Success of a non-documented process
Buddy-network of the twelve original
Excellent developers (heroes) promoted to management
(Peters Principle)

SASPIN Presentation - v.1.0 2001

23

Egypt Was No Better

Realization of need to adopt processes
Readiness for change highly challenged:

Explosive Growth of Products and People

Albeit small attrition (+)

Explosive demand (+/-)
Difficulties in recruiting (-)
Overtime the norm (-)

Matrix Organization
Project management vs. Project Administration
Flexible . chaotic team composition
Project leadership power struggles

SASPIN Presentation - v.1.0 2001

24

Yes, We Were Good

Problems

Poor quality
Incomplete products
Hurry up and code
Risk everywhere

You guys hurry up

and code while I
go and get the
requirements!

Good practices which delivered the initial

success were abandoned and forgotten.

SASPIN Presentation - v.1.0 2001

25

Agenda
QA vs. Testing
Context of the Case in Point
Dealing with Success
Lessons Learned

SASPIN Presentation - v.1.0 2001

26

Applying the CMM

Problems with following traditional solutions

Little concern for the here and now

Give me two years and Ill

Institutionalize the practice over a couple of iterations
Introduce the practices parsimoniously
Unstructured organization
Project managers dont manage
Functional managers are not in charge of projects
Functional managers task oriented - too busy getting the
work done to worry about how its getting done

SASPIN Presentation - v.1.0 2001

27

Death of the SQA Role

Post Mortem Project Review
Requirements
(SDS)

UAT Execution
(SDS)

UAT Test Planning and Preparation

Phase End Review

Acceptance
Specifications
(TSD)

Te
st

Sys Test Execution

(SDS)

System Test Planning and Preparation

Te
st

Phase End Review

Acceptance
Coding
(SDS)

Unit Test Planning

and Preparation

Acceptance
Developed Components
(SDS)`

SASPIN Presentation - v.1.0 2001

Unit Test Execution

(SDS)

Ha

Re
po
r

Re
po
r

nd
Of
f

28

SQA vs. SQC

SQC (Testing)

SQA

is perceived as a professional
role
has well-defined technical
goals
has a history of saving the
developers from greater evils
has leverage to stop the show
has authority with the
developers
is aligned with the project
goals

is perceived as managements
eyes and ears
has no history of saving
anyone yet
has fuzzy alignment with the
projects goals (apple-pie and
motherhood quality goals)
has no leverage to stop
anything
has few opportunities to show
proefficiency before the
problem

SQA cannot enforce the process

Maybe Testing can?
SASPIN Presentation - v.1.0 2001

29

Opportunity Calls
Defect
Insertion

Requirements

Design

Construction

Test

Defect
Removal

Requirements

Design

Construction

Test

Late projects were made later because of test

results that forced rework
Design constraints were ignored by new people
Serious configuration management problems:
test A, ship B
NEED TO MOVE THE QUALITY ANALYSIS FORWARD

SASPIN Presentation - v.1.0 2001

30

Changing the Role of QC

Cases
Reqs

QC Analyst
Review requirements for
understandability, completeness,
and testability.
QC Lead

Develop high level objectives from

the requirements for:
O
Functional
O
Performance
O
Error Handling
O
Field Edits
O
Linkages
Generate anomalies against vague,
missing or conflicting requirements

Case3

RM 2

QC Analyst
Create Test Case Shells in
draft mode
Create Operational & Usability
Cases and Procedures (when
possible)

Walk-thru Cases with QC

Lead - Record Anomalies

RM3

QC Analyst
O

O
O
O
O

Finalize estimates & resource plan

Walk-thru objectives, estimates &
resource plan at project status
meeting

Statement of Work Completed

Case2

RM 1

Specifications Completed

Requirements Baselined

Prepare Test Plan

O
Risk Assessment
O
Strategy
O
Resources
O
Walk-thru with
Coordinator, PM, Tech
Lead & QC Analyst

Case1

Create additional Test

Procedures & Cases to
cover objectives
completely
Evaluate Req/Case
Coverage
Develop white-box cases
and procedures
Walk-thru Cases with Test
Lead - Record Anomalies
Walk-thru Cases and
Strategy w/ Team

Code Completed

Developer
Execute Unit Testing

Post
Mortem

Anexsys QC Verification
&
Validation Cycle

Create Unit test report

providing unit test results on:
Statement Coverage
Decision Coverage
Condition Coverage
Path Coverage
Data Flow Coverage
Memory Leakage
Cyclomatic Complexity
Complete Unit Test Report
and create build

Migrate to QA for User Acceptance Test if required by Customer

QC Test Lead
Create System
Test report
identifying final
results,
outstanding
issues, and
migratoin approval

QC Analyst
O
O
O

Execute
Regression Test
Cases
Re-test failed
cases
Record Anomalies

QC Analyst

Code Corrected

Code Migrated

O
O
O
O

Execute Negative
Functionsl Tests
Additional System
Tests
White-box tests
Record Anomalies

SASPIN Presentation - v.1.0 2001

QC Analyst
O
O

Lead customer
walk-thru of
application
Record
Anomalies

QC Analyst
Install build
Execute Integration Tests
Execute Positive Functional
Tests
Record Anomalies

31

Initial Consequences
Programmers claimed

Its not our job

We know you need it, but

Initial chaos

Lots of contention
Finger pointing galore
The requirements document used as scapegoat
Time-crunch in the early steps passed down the food chain
until testing choked
Shortcuts ballooned required test execution time in an
increasingly compressed project schedule

SASPIN Presentation - v.1.0 2001

32

Good Things Happen to Those Who Wait

Requirements are now agreed upon and taken seriously

Since requirements are the litmus-test of acceptance, they

are peer reviewed and agreed upon by testers and developers

alike
Requirements are baselined and put under change control
(even prior to customer acceptance if necessary)
Requirements are accessible to all via Aimware, a Lotus Notes
application

SASPIN Presentation - v.1.0 2001

33

Changes Trickle In
Testing started sharing test cases with analysis and
development teams

Immediate payback with early detection

Shortened the integration phase
SQA seen as the supporting group

Holding developers hands to help them through the processes

Waiving unnecessary procedures (No sacred cows!)
No longer process police
Testing (SQC) as the enforcers (at each transition)

Process a consequence of their need to do good work

Validated by their past success (30%-70% defects in Y2K) and
need to use the process deliverables

SASPIN Presentation - v.1.0 2001

34

Obstinate Facts
Not out of the pit yet, but getting there
Handovers are now smooth

The tested builds are the delivered builds

Testers dont have to deal with installation failures
Products run from the moment they enter testing
With problems
Sometimes severe failures
But they dont freeze on the first test as before

The number of builds between freeze and release is down 50%

Test preparation time vs. test execution time:

from a split of 20%-80% to a more manageable 70%-30%

moving 50% of the total effort out of the projects critical path

Defects caught by developers has increased an order of magnitude

Developers pushing involvement of testers early in review of
requirements and design specifications

SASPIN Presentation - v.1.0 2001

35

Agenda
QA vs. Testing
Context of the Case in Point
Dealing with Success
Lessons Learned

SASPIN Presentation - v.1.0 2001

36

Lessons Learned
Testing can be used

To enforce better development practices

As a natural consequence of poor quality
By a group with legitimate arguments
To avoid common problems in meeting customer needs
Process has found a natural way of being introduced

Software Quality Assurance can be seen as a support group

The process that ensues is reasonable, useful, and crisp
The company is moving towards its goals
Changes to process are still expected and managed
A Stitch in Time Saves Nine

SASPIN Presentation - v.1.0 2001

37

Questions

SASPIN Presentation - v.1.0 2001

38

Quality Review Processes

Requirements Specifications:
Author:
Cust or BA

UAT Test Plan & Cases:

Author:
UAT

11

Req. Verification:
Presenter:
BSA
Summit:
QC
Testability:
QC
Analysis:
All others

System Validation:
Presenter:
QC Lead
Summit:
QA
Coverage:
BSA
Analysis:
All others

System Test Plan and Cases:

Author:
QC

Functional Design Specification:

Author:
BSA
Product Simulation

Functional Design Verification:

Presenter:
Project Leader
Summit:
QA
Testability:
QC
Analysis:
All others

10

System Validation:
Presenter:
QC
Summit:
QA
Coverage:
BA
Analysis:
All others

Usability Test

Internal Design Specification:

Author:
Dev. Mgr.

Internal Design Verification:

Presenter:
BSA
Summit:
QA
Testability:
Developer
Analysis:
All others

Integration Test Plan and Cases:

Author:
QC

Code:
Author:

Developer

Code Verification:
Presenter:
Developer
Summit:
QA
Testability:
Dev. Lead
Analysis:
All others

Unit Test Plan and Cases:

Author:
Developer

System Validation:
Presenter:
QC
Summit:
QA
Coverage:
Dev. Lead
Analysis:
All others

System Validation:
Presenter:
Other Developer
Summit:
QA
Testability:
Developer
Analysis:
All others

SASPIN Presentation - v.1.0 2001

39

Contact Information
Jorge Boria
Senior Management
Consultant
TeraQuest Metrics, Inc.
Office (512) 219-9152
Fax (512) 219-0587
email:
boria@teraquest.com
web: www.teraquest.com

Mike Baldwin
Quality Manager
Anexsys.
Office (512) 219-9152
Fax (512) 219-0587
email:
mbaldwin@anexsys.com
web: www.anexsys.com

SASPIN Presentation - v.1.0 2001

40