Dynamic Testing

Test Design Techniques

1
Dynamic Testing

• Testing that involves the execution of the software of the

component or system
• Execution of the test object on a computer
• Need a test bed

2
Test Bed
Test
Test Test Test
Case 1 Case 2 Case n
Driver

Stub1 PoC Test

output
PoO
Stub 2
Test Object comparison

Stub k

Run time environment,

Analysis tools, monitors
Test stubs

PoC – Point of Control PoO – Point of Observation

3
Incremental approach to execute tests

• Steps are:
– Determine conditions and preconditions for the test and the
goals that are to be achieved
– Specify individual test cases
– Determine how to execute the tests (usually chaining together
several test cases)
• Eg: group test cases in such a way that a whole sequence of
test cases is executed (test sequence or test scenario)
• Document it in test procedure specification
• Need test script, e.g., JUnit

4
Techniques for Testing

• They are test case design techniques

• Different techniques are:
– Black Box Testing
– White Box or Glass Box or Open Box Testing

5
Black Box Testing Techniques(1)

• Test object is treated as a black box

• The inner structure and design of the test object is unknown
• Test cases are derived/design using the specification or the
requirements of the test object
• The behavior of the test object is watched from the outside (PoO is
outside the test object)
• It is not possible to control the operating sequence of the object
other than choosing the adequate input test data (PoC is situated
outside of test object)
• Used for higher levels of testing
• Any test design before the code is written (test-first programming,
test-driven development) is black box driven

6
Black Box Testing Techniques(2)

Poc and PoO “outside” the test object

Test input data

PoC Test
Output
PoO
data
Test Object

7
Black Box Testing Techniques: Equivalence
Class Partitioning Method(1)
• The domain of possible input data for each input data
element is divided in equivalence classes
• An equivalence class is a group of data values where the
tester assumes that the test object processes them the
same way
– the behavior of the component or system is assumed to be the
same
– Test of one representative of the equivalence class is seen as
sufficient
• test equivalence class for correct input as well as
incorrect input

8
Black Box Testing Techniques: Equivalence
Class Partitioning Method(2)
• Example of correct (valid)
equivalence class vEC

Parameter Equivalence Class Representative values

Bonus calculation vEC1 : 0 <= x <= 3 2

program, duration
of employment in vEC2 : 3 < x <= 5 4
years

vEC3 : 5 < x <= 8 7

vEC4 : x > 8 12

9
Black Box Testing Techniques: Equivalence
Class Partitioning Method(3)
• Example of incorrect (invalid)
equivalence class iEC

Parameter Equivalence Class Representative values

Bonus calculation iEC1 : x < 0 -3

program, duration of (“negative” – thus incorrect –
employment in years staff membership in a
company)

iEC2 : x > 40 45
(unrealistically long and
incorrect staff membership in
a company

10
Black Box Testing Techniques: Equivalence
Class Partitioning Method(4)
Systematic Derivation of the Test Cases
• Determine the domain of all possible inputs for every input data element
– E.g., function/method parameter at component tests or input screen field at
system test
• Partition the domain into equivalence classes
– First the subdomain of correct inputs
• The test object should process these according to specification
– Then the values outside this domain as equivalence classes for invalid values
• Observe how the test object behaves
• Refine the equivalence classes
– The equivalence classes should be divided until each different requirement
corresponds to an equivalence class
• Choose a representative value for every single equivalence class for testing
• Define the preconditions and the expected result for every test case

11
Black Box Testing Techniques: Equivalence
Class Partitioning Method(5)
• the same principle of dividing into equivalence classes
can also be used for the output data
• Identification of the individual test case is more
expensive, because for every output-representative
the corresponding input value combination causing
this output must be determined
• Regard also the output values for the equivalence
classes with incorrect values

12
Black Box Testing Techniques: Equivalence
Class Partitioning Method(6)
Boundaries of the equivalence classes
• Partitioning into equivalence classes and selecting the
representative values should be carefully done
• Probability of failure detection is highly dependent upon the quality
of the partitioning, as well as which test cases are executed
• The best values are those verifying the boundaries of the
equivalence classes
– Often misunderstanding or inaccuracies in the requirements at
these spots
– E.g., “… more than 32 credits…” might mean
• The value 32 being inside equivalence class (EC : x >= 32) or
• Outside equivalence class (EC : x > 32)
• Additional test case with x=32 might detect a misunderstanding

13
Black Box Testing Techniques: Equivalence
Class Partitioning Method(7)
Equivalence class construction for integer values(1)
• All possible classes are to be identified
• Example : based on integer parameter extras of the
function calculate_price()
– the domain of inputs partitioned into equivalence classes
Parameter Equivalence classes Incorrect values:
Numbers that are greater or
Smaller than the range of the
extras vEC1 : [MIN_INT,…,MAX_INT]
Applicable interval or every
iEC1 : Not a Number Nonnumeric value including
Floating point numbers

14
Black Box Testing Techniques: Equivalence
Class Partitioning Method(8)
Equivalence class construction for integer values(2)
• Further divide the equivalence classes with correct values because
negative and positive values often must be treated differently
• Zero is a further input value that often lead to failures
• Refinement of Equivalence classes and representatives

Parameter Equivalence classes Representatives

extras vEC1 : [MIN_INT,…,-1] -123
vEC2 : [0,…,MAX_INT] 654
iEC1 : Not a Number “f”
• Result: the equivalence class method, taking into account
boundary values, generate the following test values:
(“f”, MIN_INT, -123, -1, 0, 654 MAX_INT)

15
Black Box Testing Techniques: Equivalence
Class Partitioning Method(9)
Guidelines for Determining Equivalence class(1)
 For the input as well as for the outputs, identify the restrictions and
conditions in the specification
 For every restriction or condition, partition into equivalence classes:
• If a continuous numerical domain is specified, then create one
valid and two invalid ECs
• E.g., “the item count can be from 1 to 999”, identify one vEC (1 < item < 999)
and two iEC (item count < 1 and item count > 999)
• If a number of values should be entered, then create one valid
(with all possible correct values) and two invalid ECs (less and
more than)
• E.g., “one through six owners can be listed for the car”, identify one vEC and
two iEC (no owner and more than six owners)

16
Black Box Testing Techniques: Equivalence
Class Partitioning Method(10)
Guidelines for Determining Equivalence class(2)
 For every restriction or condition, partition into equivalence classes:
• If a set of values is specified where each value may possibly be
treated differently, then create one vEC for each value of the set
(containing exactly this one value) and one additional iEC
(containing all possible other values)
• E.g., “type of vehicle must be BUS, TRUCK, TAXI, MOTORCYCLE”, identify
one vEC from the set and one iEC (“BICYCLE”)
• If there is a condition that must be fulfilled, then create one vEC
and one iEC
• E.g., “first character of the identifier must be a letter”, identify one vEC (it is a
letter) and one iEC (it is not a letter)
 If there is any doubt that the values of one EC are treated equally,
then the EC should be further divided into subclasses

17
Black Box Testing Techniques: Equivalence
Class Partitioning Method(11)
Test Cases (1)
• Combination of the representatives
– Rules for test case determination
• The representative values of all vECs should be combined to
test cases, meaning that all possible combinations of vECs will
be covered. Those combinations build “valid test case” or
“positive test case”
– Separate test for invalid value
• The representative value of an iEC can only be combined with
representatives of other vECs. Thus every iEC leads to an
additional “invalid test case” or “negative test case”.
– If a test case combines more than one incorrect value, defect
masking may result
– Thus only one of the possible exceptions is actually triggered and
tested

18
Black Box Testing Techniques: Equivalence
Class Partitioning Method(11)
Test Cases (2)
• Restriction of the number of test cases
– Reducing the number of valid test cases as the result of
multiplicative combination of representatives
• Rules for test case restriction
 Combine the test cases and sort them by frequency of occurrence.
Prioritize the test cases in this order. That way only relevant test
cases are tested
 Test cases including boundary values or boundary value
combinations are preferred
 Combine every representative of one EC with every representative
of other EC (dual combination)
 Ensure as minimum criteria that every representative of an EC
appears in at least one test case
 Representatives of iECs should not be combined with
representatives of other iECs

19
Black Box Testing Techniques: Equivalence
Class Partitioning Method(11)
Test Completion Criteria
• Define as the percentage of executed ECs in comparison
to the total number of specified EC
EC-coverage = (number of tested EC / total number of EC) * 100%

• Test coverage is a measurable criterion for finishing testing

20
Black Box Testing Techniques: Equivalence
Class Partitioning Method(11)
The value of ECP
• Contribute to a complete test where specified conditions and
restrictions are not overlooked
• Minimize the generation of unnecessary test cases
• EC can be determined by for input and output
• Can also be prepared for internal values and states, time dependent
values, and interface parameters
• Can be used in system testing, integration testing, and component
testing
• Only single input or output conditions are considered, while possible
dependencies or interactions between conditions are ignored
• May not consider boundary problems

21
Black Box Testing Techniques: Boundary
Value Analysis (BVA) (1)
• Faults often appear at the boundaries of ECs or boundary
conditions
• This is because of misunderstanding or unclear definition
• Boundary conditions are those situations directly on,
above, and beneath the edges of input ECs and output
ECs
• BVA differs with ECP in two aspects:
– Rather than selecting any element in an EC as representative,
BVA requires the one or more elements be selected such that
each edge of the EC is the subject of a test
– Rather than just focusing attention on the input conditions, test
cases are also derived by considering the output ECs
22
Black Box Testing Techniques: Boundary
Value Analysis (BVA) (2)
Guidelines for BVA
• If an input condition specifies a range of values, write
test cases for the ends of the range, and invalid-input
test cases for situations just beyond the ends.
– E.g., if the valid domain is [-1.0,…,+1.0], write the test cases for
the situations -1.0, +1.0, -1.001, +1.001
• If an input condition specifies a number of values, write
test cases for the minimum and maximum number of
values and one beneath and beyond these values
– E.g., if an input file can contain 1-255 records, write test cases
for 0, 1, 255, 256

23
Black Box Testing Techniques: Boundary
Value Analysis (BVA) (3)
Guidelines for BVA
• If an output domain serve as the basis, then the analysis can be
done as follows:
– The output of the test object is an integer value between 500 and 1000.
Test output that should be achieved: 500, 1000, 499, 1001
• If the permitted number of output values is to be tested, proceed just
as with the number of input values: if outputs of 1 to 4 data values
are allowed, the test outputs to produce are: 1, 4 as well as 0 and 5
data values
• If the input or output of a program is an ordered set (a sequential
file, a linear list or a table), focus attention on the first and last
elements of the set
• If complex data structures are given as input or output, an empty list
or zero matrixes can be considered a boundary value

24
Black Box Testing Techniques: Boundary
Value Analysis (BVA) (4)
Guidelines for BVA
• For numeric calculations, values that are close together,
as well as values that are far apart, should be taken into
consideration as boundary values
• For iECs, BVA is only useful when different exception
handling for the test object is expected depending on an
EC boundary
• Extremely large data structures, lists, tables, etc. should
be chosen. For example, those that exceed buffer, file,
or data storage boundaries; to check behavior in
extreme cases

25
Black Box Testing Techniques: Boundary
Value Analysis (BVA) (5)
Test Cases
• The valid boundaries inside an EC may be combined as
test cases
• The invalid boundaries must be verified separately and
cannot be combined with other invalid boundaries

26
Black Box Testing Techniques: Boundary
Value Analysis (BVA) (6)
Test Completion Criteria

BV-Coverage = (number of tested BV/total number of BV) *100%

The Value of the Technique

• Should be done together with ECP
• Require a lot of creativity to define test data at boundaries

27
Black Box Testing Techniques: State
Transition Testing(1)
• Base on history of execution or event or input, influences
of the output and how the test object behaves
• Test object starts from an initial state and then come into
different states
• Events trigger state transitions
– Event is normally a function invocation
• State transitions can involve actions
• The test object can be a complex system with different
system states, as well as a class in OOS with different
states
• Finite state machine, state diagrams, or state transition
tables model this behavior
28
Black Box Testing Techniques: State
Transition Testing(2)
• Example: Stack

Top
Pop [height > 1]
Push*
Push [height < max -1]
initialize top

Push
[height = max – 1]
push
empty filled full
Pop [height = 1] pop

delete

29
Black Box Testing Techniques: State
Transition Testing(3)
• A possible test case for accepting strings
Precondition: stack is initialized; state is “empty”
Input: push (“hello”)
Expected result: stack contains “hello”
Post-condition: state of the stack is “filled”
• Further test cases for the stack example
– Test case 1: initialize [empty], push [filled], push, push, push [full]
– Test case 2: initialize [empty], push [filled], top, pop [empty], delete
*note: not all states have been reached

30
Black Box Testing Techniques: State
Transition Testing(4)
Test criteria
• Should execute all specified functions of a certain state at least once
• The compliance between the specified and the actual behavior can
thus be checked.
• To identify test cases
– The finite state machine is transformed into a transition tree which includes
certain sequences of transitions

31
Black Box Testing Techniques: State
Transition Testing(5)
Building a transition tree
1. The initial or start state is the root of the tree
2. For every possible transition from the initial state to a following state
in a transition diagram, the transition tree receives a branch from its
root to a node, representing this next state
3. The process for step 2 is repeated for every leaf in the tree (every
newly added node) until one of the following two end-conditions is
fulfilled:
• The corresponding state is already included in the tree on the way from the
root to the node. This end-condition corresponds to one pass of a cycle in the
transition diagram
• The corresponding state is a final state.

32
Black Box Testing Techniques: State
Transition Testing(6)
Building a transition tree (example)
initial
initialize
empty
delete
push
deleted
filled
pop
pop
push push pop
empty filled full filled filled

top pop
push*
full full filled

33
Black Box Testing Techniques: State
Transition Testing(7)
• For the example: there are 8 different paths. Each path
represents a test case
• In addition, the reaction of the state machine for wrong
usage must be checked
– The functions are called in state in which they are not supposed
to be called (test of robustness)
• E.g., to delete a stack while in “full” state
– So, extend the transition tree to include a branch for every
function from every node

34
Black Box Testing Techniques: State
Transition Testing(8)
Building a transition tree (example)
initial
pop initialize
failure
empty
delete
push
failure top
deleted delete
filled
pop failure
pop
push push pop
empty filled full filled filled
delete
top pop
push*
failure
full full filled

35
Black Box Testing Techniques: State
Transition Testing(9)
• Also good for system testing
• E.g., to test GUI by modeling it as a finite state machine
– Screen and user controls as states
– Input reactions as state transition
• Appropriate test cases and test coverage can be
identified from the model

36
Black Box Testing Techniques: State
Transition Testing(10)
Test Cases
• The following information is necessary for a complete definition of a
state-based test case
– The initial state
– The input
– The expected behavior
– The expected final state
• Further, for each expected transition of the test case the following
aspects must be defined
– The state before the transition
– The initiating event that triggers the transition
– The expected reaction triggered by the transition
– The net expected state

37
Black Box Testing Techniques: State
Transition Testing(10)
Test Completion Criteria
• Every state has been reached at least once
• Every transition has been executed at least once
• Every transition violating the specification has been
checked
• Percentages can be defined for the above

38
Black Box Testing Techniques: State
Transition Testing(10)
The value of this technique
• Apply where states are important and where the
functionality is influenced by the state
• State transition testing is important for object-oriented
testing
– Takes into account the special aspects of object orientation

39