CHAPTER 2 BLACK BOX TESTING

Cseg5325

By Esubalew A.

Contents

Testing Techniques Types of Testing Techniques What is black-box testing?

Black-box

testing: process, applicability, prose & Cons

Types of black-box testing

Random

Testing Equivalence partitioning

ECP:

Example, how to select classes, Applicability and Limitations

Boundary
BVA:

value analysis

Examples1,2 & 3, Applicability and Limitations

Testing Techniques

Testing, however creative and seemingly complete, cannot guarantee the absence of all errors. Test-case design is so important because complete testing is impossible; the obvious strategy, then, is to try to make tests as complete as possible. Given constraints on time and cost, the key issue of testing becomes What subset of all possible test cases has the highest probability of detecting the most errors? The study of test-case-design methodologies supplies answers to this question.

Testing Techniques

The main objective of test case design techniques is to design an effective test cases. If test cases are effective there is
a

greater probability of detecting/revealing defects a more efficient use of organizational resources a higher probability for test reuse closer adherence to testing and project schedules and budgets, and the possibility for delivery of a higher-quality software product.

Techniques are the synthesis of best practicenot

Testing Techniques

Test case design techniques are the heart of testing. There are many advantages of using techniques to design test cases.
Support

systematic and meticulous work and make the testing specification effective and efficient; Extremely good for finding possible faults. the design of the test cases may be repeated by others that it is possible to explain how test cases have been designed using techniques.

Testing Techniques

Test case design techniques have a few pitfalls that we need to be aware of.
Even

if we could obtain 100% coverage of what we set out to cover (be it requirements, or statements, or paths), faults could remain after testing simply because the code does not properly reflect what the users and customers want. Validation of the requirements before we start the dynamic testing can mitigate this risk. value sensitivity. Even if we use an input value that gives us the coverage we want it may be a value for which incidental correctness applies.

Types Testing Techniques

What are the approaches a tester should use to design effective test cases? There are two basic strategies that can be used to design test cases.
Black

box /functional/behavioural / specificationdriven/data-driven / input output driven test strategy treats the system as a "black-box", so it doesn't explicitly use knowledge of the internal structure, to design test cases White box /clear box/ glass box /structural test strategy allows one to peek inside the "box", and it focuses specifically on using internal knowledge of the software to guide the selection of test data.

What is black-box testing?

These techniques focus on the functionality. Involves deriving tests from external descriptions of the software, including specifications, requirements, and design.
Unlike

its complement, white box testing, black box testing requires no knowledge of the internal paths, structure, or implementation of the software under test (SUT).

Aims at finding cases in which the software does not behave according to its specifications or requirements The general black box testing process is:
The

requirements or specifications are analyzed.

Black-box testing: Process

Valid

inputs are chosen based on the specification to determine that the software under test (SUT) processes them correctly. Invalid inputs must also be chosen to verify that the SUT detects them and handles them properly. Expected outputs for those inputs are determined. Tests are constructed with the selected inputs. The tests are run. Actual outputs are compared with the expected outputs. A determination is made as to the proper functioning of the SUT.

Black-box testing: Applicability

These test case design techniques can be used in all stages and levels of testingunit, integration, system, and acceptance.

Though the size increases from module to subsystem and then to system the box gets larger, with more complex inputs and more complex outputs, the approach remains the same. Also, as we move up in size, we are forced to the black box approach; there are simply too many paths through the SUT to perform white box testing.

Black-box testing: Pros & Cons

Advantages
Tester can be non-technical. Used to verify contradictions in actual system and the specifications. Test cases can be designed as soon as the functional specifications are complete directs the tester to choose subsets of tests that are both efficient and effective in finding defects - helps maximize the return on our testing investment

Disadvantages
The tester can never be sure of how much of the system under test has been tested. i.e. chances of having unidentified paths during this testing The test inputs needs to be from large sample space. It is difficult to identify all possible inputs in limited testing time.

Types of Black box testing

The goal of smart tester

is to effectively use the resources available by developing a set of test cases that gives the maximum yield of defects for the time and effort spent

To achieve this goal there are various functional test case design techniques. Reading Assignments
Random Testing Equivalence partitioning Boundary value analysis Decision tables Cause-effect graph State transition testing

Error Guessing Classification tree method Pair-wise testing Use case testing Domain analysis

Very often combinations of the methods are used to detect different types of defects.

Random Testing

Each software module or system has an input domain from which test input data is selected. If a tester randomly selects inputs from the domain, this is called random testing.
E.g.

If input domain for a module is 1-100 a tester may choose 24, 53 & 7 as input values. But Are the three values adequate to show that the module meets its specification when the tests are run? Are there any input values, other than those selected, more likely to reveal defects? Should any values outside the valid domain be used as test inputs?

Random Testing

Use of random test inputs may save some of the time and effort that more thoughtful test input selection methods require. However, selecting test inputs randomly has very little chance of producing an effective set of test data. There are tools that generate random test data for stress tests. This type of testing can be very useful especially at the system level. Usually the tester specifies a range for the random value generator, or the test inputs are generated according to a

Equivalence class partitioning

Involves partitioning the input or output domain into equivalence classes. Though not always this is possible most of the times.
Partitioning is splitting the domain into classes such that all members of the domain belong to exactly one class i.e. no member belongs to more than one class and no member falls outside the classes. Class is a portion of the domain. Equivalence refers to the assumption that all the members in a class behave in the same way.

The reason for the equivalence partitioning is that all members in an equivalence class will either fail or pass the same test.

Equivalence class partitioning

When SUT domain is partitioned into equivalence classes, usually both valid and invalid classes are obtained.

Test cases should be designed for both the valid and the invalid classes but sometimes it is not possible to execute test cases based on the invalid equivalence classes.

Partitioning of the input domain for the SUT using this technique has the following advantages:
It eliminates the need for exhaustive testing, which is not feasible. It guides a tester in selecting a subset of test inputs with a high probability of detecting a defect. It allows a tester to cover a larger domain of inputs/outputs with a

Equivalence class partitioning: Example

Look at the following taxation table

Income Up to and including 500 More than 500, but less than 1,300 Tax Percentage 0 30

1,300 or more, but less than Based on the above table 3 valid and 40 invalid equivalent 3 5,000 classes can be found

Valid Equivalent Classes Values between 0 to 500, 500 to 1000 and 1000 to 5000 Invalid Equivalent Classes Values less than 0, greater than 5000 and inputs containing letters

ECP: How to select classes

The following conditions can be used as guidelines for selecting input equivalence classes
If

an input condition for the SUT is specified as

range of values, select one valid equivalence class that covers the allowed range and two invalid equivalence classes, one outside each end of the range. a number of values, then select one valid equivalence class that includes the allowed number of values and two invalid equivalence classes that are outside each end of the allowed number. a set of valid input values, then select one valid equivalence class that contains all the members of the set and one invalid equivalence class for any value outside the set

ECP: How to select classes

If

the input specification or any other information leads to the belief that an element in an equivalence class is not handled in an identical way by the SUT, then the class should be further partitioned into smaller equivalence classes

Consider the specification for square root function

ECP: How to select classes

The input conditions are that the variable x must be a real number and be equal to or greater than 0.0. The conditions for the output variable y are that it must be a real number equal to or greater than 0.0, whose square is approximately equal to x. If x is not equal to or greater than 0.0, then an exception is raised. From this information the tester can easily generate both invalid and valid equivalence classes and boundaries. For example, input equivalence classes for the specification in previous slide is
EC1. The input variable x is real, valid EC2. The input variable x is not real, invalid

ECP: How to select classes

EC3. The value of x is greater than 0.0, valid. EC4. The value of x is less than 0.0, invalid.

After the equivalence classes have been identified in this way, the next step in test case design is the development of the actual test cases A good approach includes the following steps.
Assigned a unique identifier for each equivalent classes Develop test cases for all valid & invalid equivalence classes until all have been covered by a test case

The equivalence partition coverage is measured as the percentage of equivalence partitions that have been exercised by a test

ECP: Applicability and Limitations

can significantly reduce the number of test cases that must be created and executed. most suited to systems in which much of the input data takes on values within ranges or within sets. makes the assumption that data in the same equivalence class is, in fact, processed in the same way by the system. The simplest way to validate this assumption is to ask the programmer about their implementation. Equivalence class testing is equally applicable at the unit, integration, system, and acceptance test levels. All it requires are inputs or outputs that can be partitioned based on the system's requirements.

Boundary value analysis

A boundary value is the value on a boundary of an equivalence class.

Boundary value analysis is hence strongly related to equivalence class partitioning.

Equivalence classes of intervals have boundaries, but those of lists do not. Boundary value analysis is the process of identifying the boundary values. The boundary values require extra attention because defects are often found on or immediately around these. Choosing test cases based on boundary value analysis insures that the test cases are effective.

Boundary value analysis

The rules-of-thumb described below are useful for getting started with boundary value analysis.

If an input condition for the SUT is specified as a range of

values, develop valid test cases for the ends of the range, and invalid test cases for possibilities just above and below the ends of the range.
For

interval classes with precise boundaries, it is not difficult to identify the boundary values. If a class has an imprecise boundary (> or <) the boundary value is one increment inside the imprecise boundary.
If

an input condition for the SUT is specified as a number of values, develop valid test cases for the minimum and maximum numbers as well as invalid test cases that include one lesser and one greater than the maximum and

Boundary value analysis

If

the input or output of the software-under-test is an ordered set, such as a table or a linear list, develop tests that focus on the first and last elements of the set.

The boundary value coverage is measured as the percentage of boundary values that have been exercised by a test. Note that
Equivalence

class partitioning and boundary value analysis apply to testing both inputs and outputs of the softwareunder-test Conditions are not combined for equivalence class partitioning or boundary value analysis.

Example 1 on ECP and BVA

Given the following user requirements find test conditions and test cases for the testing of the requirement.
[UR 631] The system shall allow shipments for which the price is less than or equal to 100.

Lets do this using a template designed by Carsten Jrgensen The design of the test conditions based on equivalence class partitioning and boundary value analysis can be captured in a table such as the following one

Example 1 on ECP and BVA

The fields in the table are:

Test design item number: Unique identifier of the test design item Traces: References to the requirement(s) or other descriptions covered by this test design Based on: Input/Output: Indication of which type of domain the design is based on Assumptions: Here any assumption must be documented. Type: Must be one of VBValid boundary value VCValid class IBInvalid boundary value ICInvalid class Description: The specification of the test condition Tag: Unique identification of the test condition BT = Belongs to: Indicates the class a boundary value belongs to. This can be used to cross-check the boundary values.

Example 1 on ECP and BVA

To design the test cases The first thing well do is fill in the header of the design table. The next thing is identifying the valid class(es).

Example 1 on ECP and BVA

If we want 100% equivalence partition coverage and two value boundary value coverage for the requirement, assuming that invalid values are rejected, we get the following test cases:

We can omit some of the test cases, especially since there are actually 5 different test cases covering the same equivalence class.

Example 2 on BVA

Generate test cases for the following user requirement

[UR 627] The system shall allow the packing type to be specified as either Box or Wrapping paper.

The test design table looks like this after the analysis.

This type of equivalence class does not have boundaries. The test cases could be:

Example 3 on ECP and BVA

Suppose we want to test a module that allows a user to enter new student identifiers into a student data base. The input specification for the module states that a widget identifier should consist of 315 alphanumeric characters of which the first two must be letters Three separate conditions that apply to the input
1.

2.

3.

It must consist of alphanumeric characters The range for the total number of characters is between 3 and 15 The first two characters must be letters

First identify input equivalence classes and give them each an identifier

Example 3 on ECP and BVA

Condition 1: alphanumeric
EC1. Part name is alphanumeric, valid. EC2. Part name is not alphanumeric, invalid.

Condition 2: length b/n 3 and 15

EC3. The widget identifier has between 3 and 15 characters, valid. EC4. The widget identifier has less than 3 characters, invalid. EC5. The widget identifier has greater than 15 characters, invalid.

Condition 3
EC6. The first 2 characters are letters, valid. EC7. The first 2 characters are not letters, invalid.

Example 3 on ECP and BVA

Boundary value analysis is now used to refine the results of equivalence class partitioning. The following bounds groups are identified and represented by abbreviations.
BLBa value just below the lower bound (e.g. 2 ) LBthe value on the lower boundary (e.g. 3 ) ALBa value just above the lower boundary (e.g. 4 ) BUBa value just below the upper bound (e.g. 14 ) UBthe value on the upper bound (e.g. 15 ) AUBa value just above the upper bound (e.g. 16 )

Example 3 on ECP and BVA

Then we will augment these with the results from boundary value analysis. If the tester does not believe in them he or she may omit bounds groups except LB & UB, as they are included in either valid or invalid equivalent classes Finally, the test case design process is to select a set of actual input values that covers all the equivalence classes and the boundaries. The table in the next slide is used to organize the results. The table only describes the tests for the module in terms of inputs derived from equivalence classes and boundaries.

Example 3 on ECP and BVA

BVA: Applicability and Limitations

Boundary value testing can significantly reduce the number of test cases that must be created and executed. It is most suited to systems in which much of the input data takes on values within ranges or within sets. Boundary value testing is equally applicable at the unit, integration, system, and acceptance test levels. All it requires are inputs that can be partitioned and boundaries that can be identified based on the system's requirements.