Testing

Testing
Once source code has been generated ,software must be tested
to uncover and correct as many errors as possible before
delivering to the customer.
Software testing is a critical element of software quality
assurance and represents the ultimate review of specification
,design and code generation.
Most of the organization spend 30 to 40 percent of total project
effort on testing.

Software Testing Fundamental

During earlier software engineering activities ,the engineer
attempts to build software from an abstract concept to a
tangible product.
Then comes testing.
The engineer creates a series of test cases that are intended to
demolish the software that has been built.
Testing is the one step in the software process which is
reviewed as destructive rather than constructive.

Testing Objectives
There are number of rules that can serve well as testing
objectives.
1. Testing is a process of executing a program with the
intent of finding an error.
2. A good test case is one that has a high probability of
finding an as-yet undiscovered error.
3. A successful test is one that uncovers an as-yetundiscovered error

Testing
Our objective is to design tests that systematically uncover
different classes of errors and to do so with a minimum
amount of time and effort.
If test is conducted successfully(according to the objectives),it
will uncover errors in the software.
Another benefit is ,testing demonstrates that software
functions appear to be working according to specification ,that
behavioral and performance requirements appear to have been
met .
In addition data collected as testing is conducted provide a
good indication of software reliability and some indication of
software quality as a whole.

Testing Principles
Some of the testing principles are:
All tests should be traceable to customer requirements.
The objective of software testing is to uncover errors.
It follows that most severe defects(from customers
point of view) are those that cause the program to fail to
meet the requirements.
Tests should be planned long before testing begins.
Test planning can begin as soon as the requirements
model is complete.
Detailed definition of test cases can begin as soon as the
design model has been solidified.
Therefore all tests can be planned and designed before
any code has been generated

Testing Principles
The Pareto principle applies to software testing.
Pareto principle implies that 80% of all errors
uncovered during testing will likely be traceable to 20%
of all program components,
The problem is to isolate these suspect components and
to thoroughly test them.
Testing should begin in the small and progress toward in
the large.
The fist tests planned and executed generally focus on
individual components.
As testing progresses ,focus shifts in an attempt to find
errors in integrated clusters of components and
ultimately in the entire system.

Testing Principles
Exhaustive testing is not possible.
The no. of path permutations for even moderately sized
programs is exceptionally large.
For this reason ,it is impossible to execute every
combination of paths during testing.
However it is possible to adequately cover program
logic and to ensure that all conditions in the component
level design have been exercised.

To be most effective, testing should be conducted by an

independent third party.
The software engineer who created the system is not the
best person to conduct all tests for the software.

Testability
Software testability is simply how easily a computer program can
be tested .
The characteristics that lead to testable software are:
Operability
The better it works, the more efficiently it can be tested.
The system has few bugs .
No bugs block the execution of tests .
The product evolves in functional stages (allows
simultaneous development and testing)
Observability
what you see is what you test
Distinct output is generated for each input.
System states and variables are visible or queriable
during execution.

Testability
Past system states and variables are visible or queriable
.(e.g transaction logs)
All factors affecting the output are visible.
Incorrect output is easily identified.
Internal errors are automatically detected through selftesting mechanisms.
Internal errors are automatically reported.
Source code is accessible.
Controllability
The better we can control the software ,the more
testing can be automated and optimized.
All possible outputs can be generated through some
combination of input
All code is executable through some combination of
input

Testability
Software and hardware states and variables can be
controlled directly by the test engineer.
Input and output formats are consistent and structured .
Tests can be conveniently specified ,automated and
reproduced.

Decomposability
By controlling the scope of testing ,we can more
quickly isolate problems and perform smarter retesting
The software system is built from independent modules.
Software modules can be tested independently.

Testability

Simplicity
The less there is to test ,the more quickly we can test
it.
Functional simplicity (e.g. the feature set is the
minimum necessary to meet requirements)
Structural simplicity(e.g architecture is modularized to
limit the propagation of faults)
Code simplicity(e.g. a coding standard is adopted for
ease of inspection and maintenance.
Stability
The fewer the changes, the fewer the disruptions to
testing
Change to the software are infrequent.
Changes to the software are controlled.
Changes to the software do not invalidate existing tests.

Testability
Understandability
The more information we have ,the smarter we will
test
The design is well understood.
Dependencies between internal,enternal and shared
component are well understood.
Changes to the design are communicated.
Technical documentation is instantly accessible
Technical documentation is organized.
Technical documentation is specific and detailed .
Technical documentation is accurate

Testing
Attributes of a good test
1. A good test has a high probability of finding an error.
To achieve this goal ,the tester might understand the
software and attempt to develop a mental picture of
how the software might fail.
2. A good test is not redundant.
Testing time and resources are limited
There is no point in conducting a test that has the same
purpose as another.
Every test should have a different purpose .

Testing
3. A good test should be best of breed.
In a group of tests that have a similar intent, time and
resource limitations may mitigate toward the execution
only a subset of these tests.
In such cases ,the test that has the highest likelihood of
uncovering a whole class of errors should be used.
4. A good test should be neither too simple or too complex.
Although it is possible to combine a series of tests into
one test case, the possible side effects associated with
this approach may mask errors.
Each test should be executed separately

Test Case Design

The objectives of testing is to design tests that have the
likelihood of finding the most errors with a minimum amount
of time and effort.
Various test case design methods have evolved for software
and each provide the developer with a systematic approach to
testing.

Test Case Design

Any engineering product can be tested in one of the two ways:

Black box testing.

White box testing.

Testing Techniques
Black-box testing
Knowing the specified function that a product has been
designed to perform ,tests can be conducted that demonstrate
each function is fully operational while at the same time
searching for errors in each function.

They are used to demonstrate that software functions are

operational, that input is properly accepted and output is
correctly produced and that the integrity of external
information (e.g. database) is maintained.
Includes tests that are conducted at the software interface.

Not concerned with internal logical structure of the software

18

Testing Techniques
White-box testing
Knowing the internal workings of a product, tests can be
conducted to ensure that all internal operations are
performed according to specifications and all internal
components have been adequately exercised.

Involves tests that concentrate on close examination of

procedural detail
Logical paths through the software are tested.
Test cases exercise specific sets of conditions and loops.

White-box Testing

White-box Testing
Also called glass-box testing

Uses the control structure part of component-level design to

derive the test cases
These test cases
Guarantee that all independent paths within a module have
been exercised at least once
Exercise all logical decisions on their true and false sides
Execute all loops at their boundaries and within their
operational bounds
Exercise internal data structures to ensure their validity
21

Basis Path Testing

Basis Path Testing is a White-box testing technique proposed
by Tom McCabe.
Enables the test case designer to derive a logical complexity
measure of a procedural design .
Uses this measure as a guide for defining a basis set of
execution paths.
Test cases derived to exercise the basis set are guaranteed to
execute every statement in the program at least one time
during testing.
22

Flow Graph
Flow graph represent control flow in a procedure.
Flow graph depicts logical control flow using the notation
given below

Flow Graph Notation

A circle in a graph represents a node, which stands for a
sequence of one or more procedural statements

A node containing a simple conditional expression is referred

to as a predicate node
Each compound condition in a conditional expression
containing one or more Boolean operators (e.g., and, or) is
represented by a separate predicate node
A predicate node has two edges leading out from it (True
and False)

24

Flow Graph Notation

An edge, or a link, is a an arrow representing flow of control
in a specific direction
An edge must start and terminate at a node
An edge does not intersect or cross over another edge
Areas bounded by a set of edges and nodes are called regions
When counting regions, include the area outside the graph as a
region, too

Flow Graph Example

FLOW CHART

FLOW GRAPH

R4

R3

3
4

R2
7

R1

9
9
11

10

11

10

26

Independent Program Paths

Defined as a path through the program from the start node
until the end node that introduces at least one new set of
processing statements or a new condition (i.e., new nodes)
Must move along at least one edge that has not been traversed
before by a previous path
Basis set for flow graph on previous slide
Path 1: 0-1-11
Path 2: 0-1-2-3-4-5-10-1-11
Path 3: 0-1-2-3-6-8-9-10-1-11
Path 4: 0-1-2-3-6-7-9-10-1-11

The number of paths in the basis set is determined by the

cyclomatic complexity

27

Cyclomatic Complexity
Provides a quantitative measure of the logical complexity of a
program
Defines the number of independent paths in the basis set
Provides an upper bound for the number of tests that must be
conducted to ensure all statements have been executed at least
once
Can be computed three ways
The number of regions
V(G) = E N + 2, where E is the number of edges and N is
the number of nodes in graph G
V(G) = P + 1, where P is the number of predicate nodes in
the flow graph G
Results in the following equations for the example flow graph
Number of regions = 4
V(G) = 14 edges 12 nodes + 2 = 4
V(G) = 3 predicate nodes + 1 = 4
28

Deriving the Basis Set and Test Cases

The basis path design can be applied to the procedural design
or to source code.
Steps in basis path testing are:

1)
2)

3)
4)

Using the design or code as a foundation, draw a

corresponding flow graph
Determine the cyclomatic complexity of the resultant flow
graph
Determine a basis set of linearly independent paths
Prepare test cases that will force execution of each path in
the basis set

29

A Second Flow Graph Example

1
2
3
4

int functionY(void)
{
int x = 0;
int y = 19;

5
6
7
8
9
10

A: x++;
if (x > 999)
goto D;
if (x % 11 == 0)
goto B;
else goto A;

11
12
13

B: if (x % y == 0)
goto C;
else goto A;

14
15

C: printf("%d\n", x);
goto A;

16
17
18

D: printf("End of list\n");
return 0;
}

3
4
5
6

10

13

16

11

17

12
14
15

30

A Sample Function to Diagram and Analyze

1
2
3

int functionZ(int y)
{
int x = 0;

4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

while (x <= (y * y))

{
if ((x % 11 == 0) &&
(x % y == 0))
{
printf(%d, x);
x++;
} // End if
else if ((x % 7 == 0) ||
(x % y == 1))
{
printf(%d, y);
x = x + 2;
} // End else
printf(\n);
} // End while

20
21
22

printf("End of list\n");
return 0;
} // End functionZ

31

A Sample Function to Diagram and Analyze

1
2
3

int functionZ(int y)
{
int x = 0;

4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

while (x <= (y * y))

{
if ((x % 11 == 0) &&
(x % y == 0))
{
printf(%d, x);
x++;
} // End if
else if ((x % 7 == 0) ||
(x % y == 1))
{
printf(%d, y);
x = x + 2;
} // End else
printf(\n);
} // End while

20
21
22

printf("End of list\n");
return 0;
} // End functionZ

4
6

12

13

7
9
10

15
16
18
20

21
32

 Path 1: test case:

value (k) = valid input, where k < i defined
below.
value (i) = -999, where 2 <= I <= 100
expected results: correct average based on
k values and proper totals.

Graph Matrices
Graph matrix:
a date structure used to develop a software tool that assists
in basis path testing .
Graph matrix is a square matrix whose size (i.e. number of
rows and columns )is equal to the number of nodes on the
flow graph.
Each row and column corresponds to an identified node
,and matrix entries correspond to connections (edge)
between the nodes.
It is a tabular representation of flow graph.
An e.g. is shown below

Graph Matrices

Graph Matrices
By adding a link weight to each matrix entry ,the graph matrix
become a powerful tool for evaluating program control
structure during testing.
The link weight provides additional information about control
flow.

Link weight can be 1( a connection exist) or 0(a connection

does not exist)
Other values are:

The probability that a link(edge) will be executed.

The processing time expended during the traversal of a link
The memory required during the traversal of a link.
The resources required during the traversal of a link.

Graph Matrices
Graph matrix can be redrawn as below:

Graph Matrices
In this representation the graph matrix is called connection
matrix.
Each row with two or more entries represents a predicate node.
Cyclomatic complexity can be calculated from the connection
matrix.
Using these techniques ,the analysis required to design test
cases can be partially or fully automated.

Control Structure Testing

White-box testing
Control Structure Testing
Basis path testing
Condition testing
Data flow testing
Loop testing

Condition testing
Condition testing
Is a test case design method that exercises the logical
conditions contained in a program module.
A simple condition is a Boolean variable or a relational
expression ,possibly preceded with one NOT() operator.
A relational expression takes the form:
<relational operator>
Where and are arithmetic expressions and
<relational operator> is one of the following:< ,>,=,,, or

Condition testing

A compound condition is composed of two or more simple

conditions, Boolean operators and parentheses.
Boolean operators allowed in a compound condition include
OR(|),AND(&) and NOT().
A condition without relational expressions is referred to as
Boolean expression.
The possible types of elements in a condition include a
Boolean operator,a Boolean variable,a pair of Boolean
parantheses(surrounding a simple or compound condition), a
relational operator , or an arithmetic expression.

If a condition is incorrect ,then at least on component of the

condition is incorrect.

Condition testing
Therefore ,the types of errors in a condition include the
following:
Boolean operator error(incorrect/missing/extra Boolean
operators)
Boolean variable error
Boolean parentheses error
Relational operator error
Arithmetic expression error

Condition testing
The condition testing method focuses on testing each condition
in the program.
Condition testing strategies generally have two advantages:
First,measurement of test coverage of a condition is simple.
Second,the test coverage of conditions in a program
provides guidance for the generation of additional tests for
the program.
The purpose of condition testing is to detect not only errors in
the conditions of a program but also other errors in the
program.

Condition testing
If a test set for a program P is effective for detecting errors in
the conditions contained in P,it is likely that this test set is also
effective for detecting other errors in P.
Various Condition testing strategies have been proposed:
Branch testing
Simplest condition testing strategy.
For a compound condition C ,the true and false branches
of C and every simple condition in C need to be executed
at least once.
Domain testing
Requires three or four tests to be derived for a relational
expression.

Condition testing
For a relational expression of the form
<relational operator>

Three tests are required to make the value of greater than

,equal to or less than that of .

If <relational operator> is incorrect and and are correct

,then these three tests guarantee the detection of the relational
operator error.
To detect errors in and , a test that makes the value of
greater or less than that of should make the difference
between these two values as small as possible.

Condition testing
Boolean operator testing
For a Boolean expression with n variables ,all of
2 possible tests are required (n>0).
This can detect Boolean operator ,variable and parentheses
error, but it is practical only if n is small.

BRO (Branch and relational operator)testing

The technique guarantees the detection of branch and
relational operator errors in a condition provided that all
Boolean variables and relational operators in the condition
occur only once and have no common variables.

Data Flow testing

The data flow testing method selects test paths of a program
according to the locations of definitions and uses of variables
in the program.
Various data flow testing strategies are there.

Data Flow testing

Assume that each statement in a program is assigned a unique
statement number and that each function does not modify its
parameters or global variables.
For a statement with S as its statement number ,
DEF(S) = {X|statement S contains a definition of X}
USE(S)={X|statement S contains a use of X}

If statement S is an if or loop statement ,its DEF set is

empty and its USE set is based on the condition of
statement S.
The definition of variable X at statement S is said to be
live in statement if there exists a path from statement S
to that contain no other definition of X.

Data Flow testing

A definition-use (DU) chain of variable X is of the form
[X,S, ] ,where S and are statement numbers ,X is in
DEF(S) and USE[ ],and the definition of X in
statement S is live at statement .
One simple data flow testing strategy is to require that every
DU chain can be covered at least once.
This strategy is called DU testing strategy.

Loop Testing
A white-box testing technique that focuses exclusively on the
validity of loop constructs

Four different classes of loops exist

Simple loops
Nested loops
Concatenated loops
Unstructured loops
Testing occurs by varying the loop boundary values
Examples:
for (i = 0; i < MAX_INDEX; i++)
while (currentTemp >= MINIMUM_TEMPERATURE)
50

Classes of loops

Loop Testing
Testing of Simple Loops

The following set of tests can be applied to simple loops

,where n is the maximum number of allowable passes
through the loop.
1)
2)
3)
4)
5)

Skip the loop entirely

Only one pass through the loop
Two passes through the loop
m passes through the loop, where m < n
n 1, n, n + 1 passes through the loop

n is the maximum number of allowable passes through the loop

52

Loop Testing
Testing of Nested Loops
To reduce the number of tests follow the approach given below:

1) Start at the innermost loop; set all other loops to

minimum values
2) Conduct simple loop tests for the innermost loop while
holding the outer loops at their minimum iteration
parameter values; add other tests for out-of-range or
excluded values
3) Work outward, conducting tests for the next loop, but
keeping all other outer loops at minimum values and other
nested loops to typical values
4) Continue until all loops have been tested
53

Loop Testing
Testing of Concatenated Loops
For independent loops, use the same approach as for simple
loops.
If the loops are concatenated and the loop counter for loop1
is used as the initial value for loop2,then the loops are not
independent.
When the loops are not independent ,the approach applied
to nested loops is used.
54

Loop Testing
Testing of Unstructured Loops
Redesign the code to reflect the use of structured
programming practices
Depending on the resultant design, apply testing for simple
loops, nested loops, or concatenated loops

55

Black-box Testing

Black-box Testing
Also called behavioral testing.
Complements white-box testing by uncovering different
classes of errors
Focuses on the functional requirements and the information
domain of the software.
Used during the later stages of testing after white box testing
has been performed.

Enables the tester to identify a set of input conditions that will

fully exercise all functional requirements for a program.

57

Black-box Testing
Black box testing attempts to find errors in the following
categories:
1. Incorrect or missing functions
2. Interface errors
3. Errors in data structures or external base access
4. Behavior of performance errors and
5. Initialization and termination errors.
It disregards the control structure ;attention is focused
on the informational domain.

Questions answered by
Black-box Testing
How is functional validity tested?

How are system behavior and performance tested?

What classes of input will make good test cases?
Is the system particularly sensitive to certain input values?
How are the boundary values of a data class isolated?
What data rates and data volume can the system tolerate?
What effect will specific combinations of data have on
system operation?
59

Black-box Testing
By applying black-box techniques ,we derive a set of test cases
that satisfy the following criteria:
Test cases that reduce, by a count greater than one, the
number of additional test cases that must be designed to
achieve reasonable testing.
Tell us something about the presence or absence of classes
of errors, rather than an error associated only with the
specific task at hand.

Graph Based Testing Methods

The first step in black-box testing is to understand the objects
that are modeled in software and the relationships that connect
these objects.
The next step is to define a series of tests that verify all
objects have the expected relationship to one another.
Software testing includes:
creating a graph of important objects and their relationships
and
then deriving a series of tests that will cover the graph so
that each object and relationship is exercised and errors are
uncovered.
61

Graph Based Testing Methods

The software engineer creates a graph a collection of nodes that represent objects
Links that represent the relationship between the
objects ;
Node weights that describe the properties of a node
(e.g. a specific data value or state behavior)
Link weights that describe some characteristic of a
link

The symbolic representation of a graph is

shown in fig.

Graph Based Testing Methods

Graph Based Testing Methods

Nodes are represented by circles.
Directed links indicates that a relationship moves only in one
direction.
A bidirectional link ,also called a symmetric link implies that a
relationship applies in both direction.
Parallel links are used when a number of different
relationships are established between graph nodes.
Consider a portion for a word processing application where:

Graph Based Testing Methods

The software engineer then derives test cases by traversing a
graph and covering each of the relationships shown.
These test cases are designed in an attempt to find errors in
any of the relationships.

Graph Based Testing Methods

Number of behavioral testing methods that can make use of graphs
are:
Transaction flow modeling
The nodes represent steps in some transaction(e.g. steps required to
make an airline reservation using an on-line service) and the links
represent the logical connection between the steps.(e.g. flight
.information .input is followed by validation/availability processing)
Data flow diagram can be used to create graph of these types.

Finite state modeling

The nodes represent different user observable states of the software.

For e.g. each of the screens that appear as an order entry clerk takes a
phone order.
Links represent the transitions that occur to move from state to state.
E.g order-information is verified during the inventory-availability lookup and is followed by customer-billing information input.
State transition diagrams can be used to assist in creating graph of this
type.

Graph Based Testing Methods

Data flow modeling
The nodes are data objects and the links are the
transformations that occur to translate one data object
into another
Timing modeling
The nodes are program objects and the links are the
sequential connections between those objects.
Link weights are used to specify the required execution
times as the program executes.

Graph Based Testing Methods

Graph-based testing begins with the definition of all nodes and

node weights.

i.e objects and attributes are identified.

Data model can be used as a starting point.
To provide an indication of the start and stop points for the
graph ,it is useful to define entry and exit nodes.

Graph model may have loops i.e. a path through the graph in
which one or more nodes is encountered more than one time.
Loop testing can be used here and the graph will assist in
finding the loops that is to be tested.

Graph Based Testing Methods

Each relationship is studied separately so that test cases can be
derived.
Transitivity: is studied to determine how the impact of
relationships propagates across objects defined in a graph.
for e.g. three objects X,Y and Z
Consider the following relationships:
X is required to compute Y
Y is required to compute Z
Then X is required to compute Z

Graph Based Testing Methods

Symmetry: if the link is bidirectional it is important to test
this relationship.
Reflexive: every node have a relationship that leads back to
itself which has to tested.
As test case design begins, the first objective is to achieve
node coverage

i.e. tests should be designed to demonstrate that no nodes

have been omitted and that node weights are correct.
Then link coverage is addressed.

Equivalence Partitioning
A black-box testing method that divides the input domain of a
program into classes of data from which test cases are derived.

An ideal test case single-handedly uncovers a complete class of

errors, thereby reducing the total number of test cases that must
be developed.
Test case design for equivalence partitioning is based on an
evaluation of equivalence classes for an input condition.

If a set of objects can be linked by relationships that are

symmetric,transitive and reflexive, then an equivalence class is
present.

71

Equivalence Partitioning
An equivalence class represents a set of valid or invalid states
for input conditions.
An input condition is either a specific numeric value, a range
of values, a set of related values or a Boolean condition.
From each equivalence class, test cases are selected so that the
largest number of attributes of an equivalence class are
exercise at once

Guidelines for Defining Equivalence Classes

If an input condition specifies a range, one valid and two invalid

equivalence classes are defined
Input range: 1 10
Eq classes: {1..10}, {x < 1}, {x > 10}

If an input condition requires a specific value, one valid and two

invalid equivalence classes are defined
Input value: 250
Eq classes: {250}, {x < 250}, {x > 250}
If an input condition specifies a member of a set, one valid and one
invalid equivalence class are defined
Input set: {-2.5, 7.3, 8.4}
Eq classes: {-2.5, 7.3, 8.4}, {any
other x}
If an input condition is a Boolean value, one valid and one invalid
class are define
Input: {true condition} Eq classes: {true condition}, {false
73
condition}

Boundary Value Analysis

A greater number of errors occur at the boundaries of the input
domain rather than in the "center.
Boundary value analysis leads to test cases that exercise the
boundary values.

Boundary value analysis is a test case design method that

complements equivalence partitioning
It selects test cases at the edges of a class rather than
selecting any element of an equivalence class.
It derives test cases from both the input domain and output
domain

74

Guidelines for Boundary Value Analysis

1.

If an input condition specifies a range bounded by values a

and b, test cases should be designed with values a and b as
well as values just above and just below a and b

2.

If an input condition specifies a number of values, test case

should be developed that exercise the minimum and maximum
numbers. Values just above and just below the minimum and
maximum are also tested

3. Apply guidelines 1 and 2 to output conditions; produce output

that reflects the minimum and the maximum values expected;
also test the values just below and just above.
4. If internal program data structures have prescribed boundaries
(e.g., an array), design a test case to exercise the data structure
at its minimum and maximum boundaries
75