Software testing research: Achievements, Challenges &

Dreams.

Abstract:
The author of this paper is Antonia .. She is research director in Italian National
Research Council at ISTI. She is also an associate director in Journal and systems.
Over 80 papers have been published by her in international conferences and
journals. She also co-ordinates with the pisatel laboratory, which is sponsored by
Ericsson lab Italy. Her interests are in architecture and component based and
service oriented testing methods and also methods for the analysis of non-
functional properties. Software testing is the process used for the measurement of
the quality of any computer software. We can exhibit all mistakes, errors and flows
in the program. Software Testing is a process of finding errors while executing a
program so that we get a zero defect software.

Introduction:
The checking and observing of the execution of any software to validate the
working of that program. It is done also to check whether the programs behaves
as it is designed to do and identify any malfunctions or errors. It is one of the
essential activities in software engineering. In software testing we scrutinize and
check the execution of particular program. It than provides us with a realistic
feedback of its behaviors. It includes various techniques, activities and poses
many complex challenges. Indeed, by checking and scrutinizing the pervasiveness
and criticality of software we ensure that it is then according to the desired levels
of assurance, quality and depend ability.
But this costs a lot of money. Some recent studies have shown
that the testing presumes almost 50% of the total cost used in the making of
software. It also has to face many other problems and hurdles in its way. This
paper organizes many of the challenges in the software testing in a clear
roadmap. Some unreachable theories and dreams are given and also discussed
the challenges coming in the way of those dreams.
Recently this topic has gained the attention of researchers and many testing
papers are also published. Many theories for the effective testing of softwares are
also given. Many of the workshops and organized specialized events are also
being held.28th International Conferences on software engineering (ICSE 2006)
four out of the twelve of its sessions were held in the research track of “Test and
Analysis”.
This paper discusses multiple natures for software testing. It also
introduces the roadmap proposed for testing. In short, we overview several of the
research challenges.[1]
Software product line testing is discussed in the article. It
includes a set of software systems that shares a common and managed set of
some features which satisfy the needs of any particular market segment or
missions which are developed from common set of assets. Some of the important
past techniques and achievements are discussed. Also, some goals which seemed
unreachable are also discussed. The routes for the achievement of those goals are
also discussed. SPL applies and articulates best techniques and practices. We also
analyze that whether SPL can be used to help achieve the challenges defined by
Bertolino to reach the dream of testing research.[2]
In this paper, various phases and importance of testing in the
software development is explained. It plays a vital role in the software
development cycle and recognizes difficulties in the process.[3]
Software testing is important activity in Life Cycle of software
development. Checking the functionality and the correctness of any program is
called software testing. Software testing does not guarantee the production of
high quality software systems, moreover it is nearly impossible to test any
application because the domains of programs are too large, there are many
possible input paths.[4]

Software Testing is an activity that is performed for evaluating

software quality and also for improving it. Thus, the goal of testing is
systematically and stepwise detection of different classes of errors within a
minimum amount of time and also with a much less amount of effort. Software
testing is also an important component of software quality assurance (SQA) and a
number of software organizations are spending up to 40% of their resources on
testing.[5]

A software development process, also known as a software development life cycle

(SDLC), is a structure imposed on the development of a software product.
Software Engineering (SE) is the application of a systematic, disciplined,
quantifiable approach to the development, operation, and maintenance of software,
and the study of these approaches; that is, the application of engineering to
software because it integrates significant mathematics, computer science and
practices whose origins are in Engineering.[6]

Literature Review:
Summary:
As software engineering is present in every discipline of engineering. This
paper was written on the topic of Software Testing. This topic is mainly on
discussion these days. Software is testing is a broad term which includes varieties
of activities in achieving the desired goals. But testing has to face some problems
and challenges. This research is comprised of some of the important
achievements in the past and also tells about some clear goals which up till now
seem bit unreachable.
As we know that this is a broad term and includes a range of difficulties
challenges and activities. From the testing of service oriented applications to the
testing of small pieces of coding, all are included in the term Software Testing. We
test to specify any robustness in the software when given malicious inputs and
the derivations from the requirements of the user. This activity must be carried
out by following a proper method, planning documentation and technique. In the
paper some of the roadmaps are defined to test any software. These tells the
challenges faced currently and also tells some of its solutions which seem
unreachable. Many testing processes have been introduced in the past by many
researchers. These had then been developed into techniques that had helped in
making us think Testing in a more systematic way. Before we outline the future
ways of software testing, the paper discussed some topics or techniques which
can be used for testing a software. The methods discussed in the paper are;

 Object oriented testing:

As we developed, in the 90’s researchers started to focus on
object-oriented software. These object oriented software introduced new
risks and the difficulties, and hence increased the complexity in the testing
process. So object-oriented testing process had to be introduced.
 Component based testing:
In the lates of 90’s, this development emerged to be the best
approach that yielded bigger software rapidly and with much lesser
resources present.
 Protocol testing:
Protocols are defined as the rules that communicate between all the
components of a distributed system. These need to be precisely specified in
the order. This process aims at the verification and conformance of
protocols implementation against its specifications.
 Reliability testing:
This technique works on the basis that it recognizes that we can hardly
discover the last of the errors, so it uses operational profiles to drive
testing, and then tries to eliminate those failures which would occur more
frequently.

Some dream test theories have also been discussed and mentioned the
challenges they have to face. Those are;
 Universal Test Theory:
First Theory comprehends of a vigorous framework, which the testers can
specify to understand the strengths and limitations occurring in the
software and also guides to select the best one for any given condition. The
problems occurring are;
o Explicit test hypothesis
 o Test Effectiveness
o Compositional testing
o Empirical body of evidence

 Test based modelling:

Second Theory uses the technique of models which are defined in the
software and use it to drive our testing process. Some difficulties and
challenges faced in this theory are;
o Model based testing
o Anti- model testing
 100% Automatic testing:
The Third Theory puts a large emphasis on the creation of our softwares
automatically with the use of modern tools for generating even larger
codes with very less effort. Some challenges faced are;
o Text input generations
o Domain specific approaches
 Efficacy maximized text engineering:
Last Theory is considered to be the most important because the ultimate
goal of researchers these days is to reduce the cost of testing and tools of
the development of good quality software. Some of the challenges faced
are;
o Controlling Evolution
o Testing patterns
o Understanding the cost for testing
[2]Guidelines given by Kitchhenham(2004) are followed in the article. The goal is
to identify reports and initiatives carried out in software engineering related to
software product line testing. Following are the studies conducted for the study
of SPL testing.
 Unit Testing (UT)
 Integration Testing (IT)
 Functional Testing (FT)
 SPL Archive Testing (AT)
  Embedded Systems Testing (ET)
 Testing Process (TP)
 Testing effort in SPL (TE)

Software testing Research and SPL:

Many methods have been proposed. All of them had complexities and
shortcomings. Some challenges are described in this technique;
 Model Based Testing
 Explicit Test Hypothesis
 Empirical Body of evidence
 Test Effectiveness
 Controlling Evolution
 Costs of Testing
 Compositional Testing
 Test oracles
 Test Input Generation
 Online Testing and many more.[2]

[3]Software development undergoes different phases. Some of them are;

1) Requirement/Problem Definition:
The problem is studied and understood to get it solved.
2) Specification:
In this phase we describe the technical requirements for software.
3) Architecture:
We define the structure for understanding of the system software and
elements.
4) Software Design:
This phase involves solving the problems and then planning a solution.
It’s output describes of both, what the system Will and How.
 Some of the designing concepts are;
i. Abstraction
ii. Modularity
iii. Data Structure
iv. Structural Partitioning
5) Software Construction:
Developing of the documentation for system by coding, verifying and
debugging. Main fundamentals are;
i. Construction for Verification
ii. Minimizing Complexity
iii. Anticipating Change
6) Software Testing and Evaluation:
This phase includes testing, verifying and validating the system just
built. Main purpose is to detect the failures, so that the defects can be discovered
and corrected.
7) Debugging:
This is the process for finding and reducing all the defects and errors in
any computer program.
8) Software deployment and Maintenance:
It is general process which is customized according to some specific
characteristics. Some of the main maintenance techniques are follows;
i. Adaptive Maintenance
ii. Preventive Maintenance
iii. Perfective Maintenance
iv. Corrective Maintenance
Now Software Testing also undergoes different phases. Some of them are as
follows;
1) Test Analysis:
Try to understand about project/program code.
 2) Test Design:
Design the test cases based on the requirements of users
3) Test Execution:
Execute the test cases and raises defects.
Various types of testing methods are involved throughout SDLC. These methods
are as follows;
 Acceptance testing
 AD-Hoc testing
 Alpha and Beta testing
 Black Box testing
 White Box testing
 Grey Box testing
 Integration testing
 Regression testing
 Stress testing
 User Acceptance testing
 Security testing
 Performance testing
Software testing is a major part of SDLC and these must be carried out
effectively of the product. All the efficient and effective testing provides timely
visibility and insuring the qualities and readiness.[3]
[4]The main objectives of testing is finding problems and then fixing them to
improve quality. Four of the main objectives of software testing are;
 Demonstration:
Demonstrates functions under special conditions and shows that
products are ready for use
 Detection:
Discovers errors and defect, defines compatibilities and limitations,
quality of software and the system
 Prevention:
Provides information for the reduction of errors and identifies ways for
avoidance of risks and problems
  Improve Quality:
By effective testing errors can be minimized and the quality of the
software can be improved

The three basic techniques discussed are;

1) Black box testing
2) White box testing
3) Grey box testing
 Black Box testing:
It is done on the base of customers’ point of views, only tester knows
the set of inputs and outputs. Code is not examined in black box testing. It has
different techniques involved;
1) Equivalence partitioning
2) Boundary value analysis
3) Fuzzing
4) Cause Effect graph
5) Orthogonal array testing
6) All pair testing
7) State transition testing
Advantages:
1) Knowledge of specific programming language is not needed
2) Done on user’s point of view
3) Tester and the programmers are independent
Disadvantages:
1) Hard to design without any specifications
2) Chance of repeating texts on parts that are already done
3) Some parts at back end are even not tested

White Box testing:

 It focuses on internal logic and the structure of code. With the help of this
technique, every branch of the code can be tested. Different techniques involved
are;
1) Desk checking
2) Code walkthrough
3) Formal inspections
4) Control flow testing
5) Basis path testing
6) Data Flow testing
7) Loop testing
Advantages:
1) Reveals errors of hidden code by removing the extra lines.
2) Developers have to give reasons carefully about Implementation.
Disadvantages:
1) Skilled testers are required because internal structures knowledge is
required
2) Much of the paths remain untested because it is difficult to look in every
corner of code to find hidden errors.

Grey Box testing:

This techniques tests the application with limited knowledge of internal
workings of applications and also has knowledge of fundamental aspects of
system. It attempts and combines the benefit of both black and white box. It is
used mostly in integration testing.
1) Orthogonal Array testing
2) Matrix Testing
3) Regression Testing
4) Pattern Testing
Advantages:
1) Provides combined benefits of black and white box techniques
2) Unbiased Testing
 3) Tester can design excellent tests scenarios.
Disadvantages:
1) Coverage of test is limited because access of source code is not available
2) Many of the program paths remain untested.

Three main techniques of software testing were explained. Software testing

allows the business administration to appropriate and understand risks of
software implementation.[4]
A software testing strategy integrates various software test case design
methods into a well-planned series of steps that result in successful testing of
software. Four are the techniques of testing:
 Unit testing
 Integration Testing
 System Testing
 Acceptance Testing
The Four main objectives of testing are:

1) Detection
2) Prevention
3) Demonstration
4) Improving quality

There are following methodologies for software testing:

 White Box Testing:
 Basis Path Testing
 Loop Testing
 Control Structure Testing
 Black box testing:
 Equivalent Partitioning
 Boundary value Analysis
 Cause-Effect Graphing Techniques
 Grey Box testing
Latest Research and Development in Software Testing:
Newer techniques and methods have been adopted for the purpose of
testing. Some of those techniques are:
 Test driven development:
 Iterative and incremental testing:
 GUI automation test:
 Testability of component software:
 Embedded Software Simulation Test:
 International and domestic research hotspots[5]

Software Engineering processes are composed of many activities,

notably the following:


 Requirements Analysis ,
 Specification
 Software architecture
 Implementation
 Testing
 Documentation
 Training and Support
 Maintenance

Phases of SDLC:

Problem solving in software consists of these activities:

 Understanding the problem
 Deciding a plan for a solution
 Coding the planned solution
 Testing the actual program

For large systems, each activity can be extremely complex and methodologies and
procedures are needed to perform it efficiently and correctly.
Software Development Models:
A) Original Waterfall Models:
The Waterfall model is a conventional, linear, sequentional or
traditional waterfall software life cycle model.
1) Basic Principles:
o Project is divided into sequential phases, with some overlap and splash back
acceptable between phases.
o Emphasis is on planning, time schedules, target dates, budgets and
implementation of an entire system at one time.
2) Advantages:
o Each stage has well defined deliverable or milestone.
o It is simple to use and understand.
3) Disadvantage:
o The biggest disadvantage of the waterfall model is one of its greatest
advantages. You cannot go back a step; if the design phase has gone wrong,
things can get very complicated in the implementation phase.
o Often, the client is not very clear of what he exactly wants from the
software. Any changes that he mentions in between may cause a lot of
confusion.
o Small changes or errors that arise in the completed software may cause a
lot of problems.
B) Prototyping model:
Software prototyping is the development approach of activities during
software development, i.e. incomplete versions of the software program being
developed.

1) Basic Principles:
 Not a standalone, complete development methodology, but rather an
approach to handling selected parts of a larger, more traditional development
methodology.
 Attempts to reduce inherent project risk by breaking a project into smaller
segments and providing more ease-of-change during the development
process.
 User is involved throughout the development process, which increases the
likelihood of user acceptance of the final implementation.
2) Advantages:
 Early visibility of the prototype gives users an idea of what the final system
looks like Encourages active participation among users and producer.
 Enables a higher output for user.
 Cost effective (Development costs reduced)
 Increases system development speed
3) Disadvantages:
 Possibility of causing systems to be left unfinished.
 Possibility of implementing systems before they are ready.
 Producer might produce a system inadequate for overall organization needs.
 Often lack flexibility.
 Not suitable for large applications.

After analysis of all models through the various factors, it has been found that the
original water fall model is used by various big companies for their internal
projects. Prototype model used to develop online systems for transaction
processing. Since significantly reduce rework and lead to the creation of working
model in lower capital cost.[6]

Problems:
This paper discusses many of the problems and challenges occurring in the
way of software testing. Some researchers worked on it and rendered some
techniques but we insist on making changes and bringing about latest
technologies for convenience. But many problems are faced, of which the cost is
the main problem. Other problems also occur in the theories proposed. Some of
most highlighted of those problems are;
 Test Effectiveness
 Compositional testing
 Test oracles
 Domain specific Test Approaches
 Controlling Evolution
 Testing Patterns
 Cost Controlling
 Due to these challenges and problems these theories seem unreachable. We
have developed some ideas for the better testing of softwares but, those ideas
need time and also need huge amount to become functional. They also lack in
one or other fields and does not fulfill our desires. Hence these seem like a
dream. Not vulnerable practical work is done to convert these proposed theories
into reality. More research work is needed to overcome the shortcoming and
challenges in the dream proposed ‘testing’ theories.
[2]All the challenges and problems are enumerated. Some like model
based testing and test input generation seems like fundamentals for SPL testing.
But SPL is still not mature, there is still lack of empirical results in the work done
for SPL testing. [2]
[3]It is a costly process. Surveys tell that software error cost U.S
economy to spend 0.6% of their GDP. And about 80% of the total cost of software
developments are spent in identifying and fixing errors.[3]
Sr. No. Reference Problems Solutions Result
No.
1 1 High cost Quality Cost
Management Minimized
2 1 Time Time Time issue
Consuming Managing resolved
3 1 Lesser Practical More Practical Chances to
Work work achieve
Theories
4 1 Reliability of Adopt reliable Reliable
Testing methods Testing
5 1 Effectiveness of Apply Effective Effective tests
the Tests Techniques
6 2 Not mature More research Maturity
enough attained
7 2 Lack of Empirical More Practical Lackness Co-
Work work oped
8 4 Many paths of Quality Code examined
code remain Assurance completely
unchecked
9 5 Skilled testers Adopt skilled Problem solved
required testers
10 6 Hard to design Keep the users Designing
without demands in the becomes easy
specifications design

References:
[1] L. Baresi and M. Young. Test oracles. Technical report,
Dept. of Comp. and Information Science, Univ. of Oregon,
2001. http://www.cs.uoregon.edu/m˜ ichal/pubs/oracles.html.
[2] E. Bayse, A. R. Cavalli, M. N´u˜nez, and F. Za¨ıdi. A passive
testing approach based on invariants: application to the wap.
Computer Networks, 48(2):235–245, 2005.
[3] B. Beizer. Software Testing Techniques (2nd ed.). Van Nostrand
Reinhold Co., New York, NY, USA, 1990.
[4] A. Belinfante, L. Frantzen, and C. Schallhart. Tools for test
case generation. In [21].
Fu Bo (2007), Automatic Generation Method of Test Data Based on Ant Colony Algorithm,
Computer Engineering and Applications.43(12).
Stacey, D. A.( 2004), Software Testing Techniques Guide to the Software Engineering Body of
Knowledge, Swebok – A project of the IEEE Computer Society Professional Practices
Committee.
R.S. Pressman & Associates, Inc. (2005). Software Engineering: A Practitioner’s Approach, 6/e;
Chapter 14: Software Testing Techniques,
[1] Software Testing. Gregory M. Kapfhammer. The Computer Science and Engineering
Handbook, CRC Press. May, 2004.
[2] Mohd. Ehmer Khan, “Different Approaches to Black Box Testing Technique for Finding
Errors,” IJSEA, Vol. 2, No. 4, pp 31-40, October 2011
[3] Jovanovi?, I. "Software Testing Methods and Techniques. " The IPSI BgD Transactions on
Internet Research: 30
[4] F. Saglietti, N. Oster, and F. Pinte, “White and grey-box verification and validation
approaches for safety- and security-critical software systems,” Information Security Technical
Report, vol. 13, no. 1, pp. 10–16, 2008.
[5] T. Murnane and K. Reed, “On the effectiveness of mutation analysis as a black box testing
technique,” in Software Engineering Conference, 2001. Proceedings. 2001 Australian, 2001, pp.
12 –20.
[6] Nidhra, Srinivas, and Jagruthi Dondeti. "Black box and white box testing techniques- A
Literature." International Journal of Embedded Systems & Applications 2.2 (2012).
[7] Khan, Mohd Ehmer, and Farmeena Khan. "A Comparative Study of White Box, Black Box
and Grey Box Testing Techniques." International Journal of Advanced Computer Sciences and
Applications 3, no. 6 (2012):
[8]Maglyas, A.; Nikula, U.; Smolander, K.,“Comparison of two models of success prediction in
software development projects”, Software Engineering Conference (CEE-SECR), 2010 6th
Central and Eastern European on 13-15 Oct. 2010.
[9] A. M. Davis, H. Bersoff, E. R. Comer, “A Strategy for Comparing Alternative Software
Development Life Cycle Models”, Journal IEEE Transactions on Software Engineering
[10] Software maintenance available at
http://en.wikipedia.org/wiki/Software_maintenance
[11] Software development process available at
http://en.wikipedia.org/wiki/Software_development_pr
ocess
[12] Involve testing throughout the SDLC available at
http://www.silverpath.com
 Orthogonal Array
Diagrams: Testing

Grey Box Testing Matrix Testing

(Fig. 1)
Regression Testing

Pattern Testing

Equivalence
Partitioning

Boundary Value
Analysis

Fuzzing

Black Box Testing Cause-Effect

Graph

Orthogonal Array
Testing
(Fig. 2)
All Pair Testing

State transition
testing
Requirements
analysis

Design
(Fig. 3)

Implementation

Testing

Maintenance

Model:
As we have discussed many of the problems involved in the Software
Testing process. Some of those problems have some solutions which can be
applied to resolve the problems. As software Testing is a much expensive and
time consuming process so we can manage it by quality management of the
issues. Also other problems can be resolved by keen interest in the process. We
can also find solutions by combining many of the proposed solutions and then
superposing their effects for betterment of testing process.
Example:

We have created a login page. We have to check it’s authentication

successfulness.

Description: A registered user must be able to login the website successfully.

Precondition: User must be already registered with an e-mail address and a
password

Test Steps:
1) Navigate to the website.
2) In the provided e-mail field, enter the e-mail of already registered user.
3) In the next provided password field, enter the password.
4) Click the sign-in button.

Navigate to Website

Enter valid e-mail in

the provided field

Enter password in
next provided field

Click Sign-in

Expected Result: A page should be displayed containing the information.