Quality Assurance Institute’s

(QAI)
Effective Methods of Software
Testing
A Perspective
Seattle Area Software Quality Assurance Group

John Napier, CSQE, CSTE

BEST Consulting
10/14/08
Purpose
 Value of Professional Associations
 Value of Software Certification programs
 QAI’s “Effective Methods of Software Testing”
Disclaimers

 The opinions expressed in this presentation are

my own, and not those of:
◆ BEST Consulting
◆ The Boeing Company
◆ Quality Assurance Institute
◆ SASQAG
What its like in the trenches...
 No common language
 “In my last job…”
 No individual professional context
 No organizational professional context
 Can’t see the forest...
What’s a person to do?
 Establish a personal professional context
 Get to know “the forest”
 Help create a common language
 Help bring best practices into your organization
The Quality Assurance Institute
What’s QAI?

10/14/08
The Quality Assurance
Institute
 An international association of 1000
organizations banded together to share
best practices, benchmark data and
experiences
 Software quality focus
 Certification Programs
◆ Certified Quality Analyst

◆ Certified Software Quality Engineer

QAI
Certified Software Test
Engineer (CSTE)
 Mandatory Skills  Optional Skills

◆ Test Planning ◆ Risk Analysis

◆ Use of Test Tools ◆ Measurement

◆ Test Performance ◆ Test Case Design
◆ Defect Management ◆ Establishing and Maintaining
a Test Environment
ASQC*
Certified Quality Software
Engineer (CQSE)
 Experience
 Professional Association
 Software Engineering Body of Knowledge
◆ Exam

 Many other ASQC Certification Programs

*American Society for Quality Control

Effective Methods of Software
Testing
Carolyn Fairbanks
Q-Star Consultants
10-14 March 1997
Seattle

10/14/08
Effective Methods of Software
Testing
There are really only two issues
in software testing:
 Requirements
 Organizational Culture
10/14/08
QAI Test Materials Organization
 Test Strategy
 Test Plan

 Supporting Materials
◆ Metrics

◆ Test Toolbox - Techniques

◆ Test Conditions - Techniques

◆ Test Case Design - Techniques

Test Strategy Definition
 1. Type of Project
 2. Type of Software
 3. When Testing will Occur
 4. Critical Success Factors
 5. Who will Conduct the Testing
 6. Tradeoffs
1. Types of Project
 Traditional
 Client/Server
 Prototyping
 Object Oriented
 Purchased / Contracted
2. Types of Software
Batch Diagnostic Software
Process Control Sensor and Signal
Graphic User Interface Simulation
Object Oriented DB Management
Procedure Control Data Presentation
Expert System Decision Analysis
Multi-Media Pattern & Image
Advanced Math Models Computer Operations
Software Development Tools
3. When will Testing Occur?
 Requirements Phase Activities
 Design Phase Activities
 Build Phase Activities
 Test Phase Activities
 Installation Phase Activities
 Maintenance Phase Activities
4. Critical Success Factors
Correctness Methodology
Authorization Reliability
Data Integrity Ease of Use
Audit Trail Maintainable
Continuity of Processing Portable
Service Levels Coupling
Access Control Performance
Ease of Operations
Testing Concerns Matrix
Critical
Success
Factors Life Cycle Stages

Requirements Design

Complete, Clear, Correct, Etc

Correctness
Testable, Feasible
7. Tradeoffs (What can be given
up?)
 Schedule
 Cost / Resources
 Scope
 Technology
 Quality / Critical Success Factors
 Some QAI Philosophy
Test Life Cycle, Test Workbench

10/14/08
Testing Relationship Definitions
-
Testing Lifecycle Concept

Operational or Acceptance
Business Need Test

Define System
Requirements Test
Design Integration
System Test
Build Unit
System Test
Testing Relationship Definitions
-
Continuous Life Cycle Testing
Verify Validate
Operational or Acceptance
Business Need Test

Verify Validate
Define System
Requirements Test

Verify Validate
Design Integration
System Test
Verify Validate
Build Unit
System Test
 Testing Relationship Definitions
-
Test Support Activities
TEST OBJECTIVES
Verify Validate

Operational or Acceptance
Business Need Test

TEST PLANNING Validate

Verify
System
Define TEST EXECUTION
Requirements Test
(DYNAMIC) &
Validate TEST REPORTING
Verify

Design Integration
System Test
TEST EXECUTION Verify Validate
(STATIC)
Build Unit
System Test
QAI Definitions -
“Test Workbench” Concept
Problems

Products Test
Test Validated
for Quality
Processes Test
Test Control
Results

Test Test
Toolbox Measures
The QAI Test Process
1. Set Test Objectives
2. Develop Test Plan
3. Execute Tests
4. Summarize and Report Results
Step 1 - Set Test
Objectives
 Task 1 - Organize Test Team
 Task 2 - Perform Risk Assessment
 Task 3 - Set Test Objectives
 Task 4 - Quality Control
Set Test Objectives
Task 1 - Organize Test Team
 Define Test Responsibilities / Tasks
 Determine People to Perform Tasks

Worksheets 4,5
Set Test Objectives
2. Perform Risk Assessment
 Dimensions of Risk
◆ Criticality / Mission Impact Impact of Failure
◆ System Size, Scale and Complexity
Likelihood
◆ System Environment and Stability
of
◆ Reliability and Integrity Failure
◆ Technology Integration

Simple or Complex Scoring Approach

Set Test Objectives
3. Set Test Objectives
 Using:
Prioritize Each-
◆ System Objectives
Low
◆ Test Strategy Medium
◆ Risk Score Analysis High
◆ Team Assignments
Worksheet 6
 Set Objectives
◆ Brainstorming

◆ Relate Objectives to Function / Output

◆ Relate Objectives to Events / System Input

Set Test Objectives
Quality Control
 Objectives QC Checklist
 Objectives Incident Report (Change Request)

…lots of checklists!
Develop Test Plan
 1. Define Business Processes
 2. Define Structural Attributes
 3. Define Tests
 4. Create Test / Function Matrix
 5. Define Admin Requirements
 6. Formulate System Test Plan
 7. QC System Test Plan
 8. Develop Unit Test Plan
 9. QC Unit Test Plan
Develop Test Plan
1. Define Business Functions
 Describe what the system should do
◆ …stands alone as an activity

◆ …is controllable

◆ …can be a transition

◆ …can be described by a process

Business Function Matrix

Function Description Evaluation Criteria

Worksheet 8
Develop Test Plan
2. Define Structural Attributes
 Reliability  Testability
 Efficiency  Flexibility
 Integrity  Portability
 Usability  Reusability
 Maintainability  Interoperability

Worksheet 9
Structural Attribute Matrix

Attribute Priority Description Evaluation Criteria

Develop Test Plan
3. Define Tests
 Using
◆ Test Objectives Worksheet

◆ Business Functions Worksheet

◆ Structural Attributes Worksheet

 Define Tests
◆ Associate test factors, techniques and test types

◆ Associate evaluation criteria with techniques and

types of tests
◆ Determine type of test which satisfies each objective
Type of Test Techniques
 Functional  Structural
◆ Requirements ◆ Stress

◆ Regression ◆ Execution

◆ Error Handling ◆ Recovery

◆ Manual Support ◆ Operations

◆ Intersystems ◆ Compliance (to process)

◆ Control ◆ Security

◆ Parallel
Test Data Sources
 Test Data / Test Desks
 Production Data
 Simulation / Prototyping
 Review / Confirmation
 Instrumentation / Tracing
 Test Script
Test Definition
 Each Test is defined in terms of:
◆ Controls

◆ Inputs

◆ Outputs

◆ Procedures

...At a high level

Develop Test Plan
4. Create Function / Test Matrix

Tests Test 1 Test 2

Function Ect.

25 17,30
Function #1

Function #2 5,7 45
Develop Test Plan
5. Define Admin Test
Requirements
 Schedule
 Budget
 Resources
 Testing Materials
 Test Training
 Tests to be Conducted
 Defect Estimation

Worksheets 13,14
Develop Test Plan
6. Formalize Test Plan
 Section I - General Information
 Section 2 - Plan
 Section 3 - Specifications and Evaluation
 Section 4 - Test Description

See Test Plan Example

Develop Test Plan
7. Test Plan Quality Control
 Quality Control Checklist
 Test Plan Incident Report (Change Request)

…even more checklists!

Develop Test Plan
8. Develop Unit Test Plan
 Using:
◆ System Test Plan

◆ Program Initiation Document

◆ Program Design / Specification Documents

 Develop
◆ Test Plan Administration

◆ Subfunction / Test Matrix

◆ Interface / Test Matrix

Worksheets 20,21,22
Develop Test Plan
9. Unit Test Plan Quality Control
 Checklists
Step 3 - Execute Tests
 Task 1 - Select Test Tools
 Task 2 - Develop Test Cases
 Task 3 - Execute Tests
 Task 4 - Quality Control
QAI Tool Definition
 What’s a Test Tool?
◆ …any vehicle which assists
in performing a test.
◆ Examples-
 Decision Tables
 Automated capture / playback

 Checklists

 Program Analyzer
Execute Tests
Step 1 - Select Test Tools
 Using:
◆ System Test Plan

◆ Unit Test Plan

◆ Tool Selection Methodology

Execute Tests
Step 2 - Develop Test Cases

Test Condition Expected Result Test Procedures

Worksheet 23
Execute Tests
Step 3 - Execute Tests
 Test Activities Log
 Regression Testing
◆ Full

◆ Partial

Worksheet 24
Step 4 - Summarize and
Report
Test Results
 Task 1 - Record Defects
 Task 2 - Perform Data Reduction
 Task 3 - Develop Findings and Recommendations
 Task 4 - Finalize Test Report
 Task 5 - Test Report Quality Control
◆ Test Defect Report

◆ Test Defect List

◆ Test Defect Log

Worksheets 25,26,27,28
 Supporting Material
 Metrics
 Toolbox Techniques
 Test Condition Analysis
 Test Design Analysis

10/14/08
 Test Metrics
Method -

Goal - Question - Metric

QAI Recommend Test Metrics
 Defect Reports
 Requirements Implemented
 Backlog of Defects (Not Corrected)
 Status of -
◆ Project, Tests, Schedule, Budget

 Selected Test Metrics

QAI Test Metrics Selection

Types General Subject

1 Involvement in Testing
4 Extent of Testing
3 Resources Consumed in Testing
20 Effectiveness of Testing
2 Assessment of Testing
The QAI Test Toolbox
 Data Modeling
 Boundary Value Analysis
 Capture/Playback
 Cause-Effect Graphing
 Change Control Trackers
 Checklists
 Checkpoint Review
 Code Comparison
 Compiler-based Analysis
 Complexity-based Analysis
 Compliance Checkers
 Control Flow Analysis
 Correctness Proofs
 Coverage Based Analysis
 Data Dictionary
 Decision Tables
 Defect Trackers
 Desk Checking
 Equivalence Partitioning
 Error Guessing
The QAI Test Toolbox
 File Comparison
 Flowcharts
 Help Desk Trackers
 Inspections
 Instrumentation
 Integrated Test Facility
 Mapping
 Model Animation
 Model Balancing
 Network Analyzers
 Performance / Timing Analyzers
 Population Analysis
 Risk Matrix
 Scoring
 Snapshots
 Symbolic Execution
 System Logs
 Test Case Generator
 Test Data Generator
 Test Scripts
The QAI Test Toolbox
 Test Stubs and Harnesses
 Test Suite Managers
 Tracing
 Walkthroughs
The QAI Test Toolbox

Test Tools Types of Functional and

Structural Tests

Boundary Analysis X X
Population Analysis X
Checklists X X
Test Condition Analysis
 Sources -
◆ 1. Specification Decomposition

◆ 2. Population Analysis

◆ 3. Business Process Analysis

◆ 4. Structural Analysis

◆ 5. Test Transactions

Test Condition - Expected Result - Test Process

Identify Test Conditions
 1. System Specifications or Production Environment
 2. Population Analysis
◆ Files and Tables

◆ Screens

◆ Field/Data Element

Test Condition - Expected Result - Test Process

Worksheet 4
Identify Test Conditions
3. Test Transaction Types
 Field  Match / Merge
 Record  Stress
 File  Control
 Relationships  Attributes
 Error  States
 Use (Outputs)  Procedures
 Search  Control Flow

Test Condition - Expected Result - Test Process

Identify Test Conditions
 4. Business Cases
◆ …business scenarios…

 5. Structural Analysis
◆ …decision points...

Test Condition - Expected Result - Test Process

Design Test Cases
 Design Test Cases
 Develop Scripts / Sets

Expand Test Conditions

Design Test Cases
 Expand Conditions into Test Cases using -
◆ Boundary Value Analysis

◆ Equivalence Partitioning

◆ Decision Tables

◆ Code Coverage
 Statement Coverage
 Condition Coverage

Expand Test Conditions

Design Test Cases
Test Scripts

Actor/Scene Dialog

…Sam scowls “Where did this

at the report... come from?”

Detailed Test Script Descriptions

Script Reuse
Expand Test Conditions
QAI Summary - Ten Key Test
Concepts
 Use strategic test questions to set direction
 Develop Test Objectives
 Use a cross-functional “test team”
 Prepare a test plan, then execute it
 Utilize both verification and validation methods
 Collect and analyze defects
 Prepare test management reports
QAI Summary - Ten Key Test
Concepts
 Make test efforts reusable
 Feed the continuous improvement process with
information from testing
 Continuously improve the process
Incorporating the Process
Models within Models
…Make them your own

10/14/08
Questio
ns?