Building a user interface test automation

framework
Izzat Alsmadi and Kenneth Magel
Department of Computer Science
North Dakota State University
Fargo, ND 58105
{ izzat.alsmadi, kenneth.magel}@ndsu.edu

Abstract

This article suggests a fully test automation tool with the least amount of time
and decisions required from testers. Using an automated tool should not be
offset by testers spending a good amount of time for setup and guidance.
The proposed framework for user interface test automation uses the information
stored in the application assembly to build the tested model. The GUI model is
parsed from the application under test at run time. The GUI model is built as an
XML tree. Test cases are then generated from the XML tree using different
proposed algorithms. Some techniques for test case prioritization, critical path
testing, execution and verification are suggested. The goal is to do all those
activities dynamically as part of a GUI test automation framework.

General Terms
Software testing, user interface testing and test
automation.
Keywords
Software test automation, test case generation, Test
execution, GUI testing, GUI modeling, and
verification.
1. Introduction
Software testing is an important stage in software
projects. It is one of the most expensive stages that
usually stops after using the available resources (and
not completed). Test automation takes more time in
the initial stages. Eventually, this is expected to
reduce the cost of testing through reducing the
required time and resources. GUI is increasingly
taking a larger portion of the overall program’s size
as newer programs are more interactive with the user
inputs. and GUI testing is taking a major rule in the
whole project’s validation.
GUI test automation is a major challenge for test
automation activities. Most of the current GUI test
automation tools are partially automated and require
the involvement of users or testers in several stages.
Test automation tools are still complex and
expensive. They don’t fully replace testers. They can
be usually used to re-execute those repeated tasks.
Companies consider taking the decision to buy, or
not, a GUI test automation tool as a tough decision
since they don’t know how much time it will require
upfront for setup. They also don’t know how much of
test automation tasks can be fully or partially
automated. We all agree that complete coverage in
test automation is impossible as we also agree that
complete coverage in testing is impossible too.
No matter what software methodology a company
follows, software testing is going to play a major role
in terms of project success or resource allocation.
Automated test refers to testing by a computer (no
human tester). We choose the option of automating
the GUI testing when tests are repeated several times
[1]. Software test automation tasks include selecting
and generating the test cases, building the test oracle
(test cases with expected results), executing, re-
executing, reporting and validating the results.
GUIs manage controls. Controls are reusable objects
with which users can interact. The term "control" is
used as a generic term for any graphical object or
widget that an application may produce. Controls
have common elements that should be considered
before writing a program that interacts with a GUI
[2].
• Controls are categorized according to Window
classes. Each control belongs to a window class
(making it possible to search them by class).
• Controls have an organizational hierarchy; every
GUI has at least one root control, and every control
may have child controls. Controls and forms make a
tree. We can generate a valid test case by selecting
several controls that are children of each other.

1
• Controls have different ways to be located or
identified at run time. They can be identified by their
names, parents, unique location or by the window
handle associated to them at run time.
We present a GUI testing framework that does not
require the involvement of testers throughout the
different stages. The tests do not need to be revisited
during any change in the GUI structure. Since the
application uses reflection to get the actual model at
run time, those changes are included in the current
test.
Generating test cases can happen from requirements,
design or the actual GUI implementation. Although it
is expected that those three should be consistent and
related, yet they have different levels of abstraction.
Requirements and design are usually of a high level
of abstraction to generate from them the test cases.
On the other hand the task of generating test cases
from the GUI implementation model is delayed until
GUI is implemented, which is usually occurred in the
late implementation. We should not have any
problems in delaying GUI testing giving the fact that
a tool will automate the generation and execution
processes.
A tool is built in C# that uses reflection to serialize
the GUI control components. Certain control
properties are selected to be serialized. These
properties are relevant to the user interface testing.
The application then uses the XML file that is created
to build the GUI tree or the event flow graph. Test
cases are then generated using the GUI XML file.
Generating the test cases takes into consideration the
tree structure to select the test cases that cover unique
branches.
In test execution and verification, test cases are
executed on the application and compared with the
original test suite. The advantages of the object
oriented approach over the widely used
capture/replay back one, is in the fact that the model
is generated at run time which makes it represents the
current state of the GUI model. In record/play back
cases, we have to retest the application in case of any
change in the functionalities or the GUI of the
program. Once utilized, this object oriented approach
is expected to be less expensive as it does not require
users to manually test the application or make
decision for pre- and post-conditions.
2. Related work
There are several GUI test automation tools available
in the market. Some of the larger vendors for such
tools includes: IBM Rational, Mercury, and Segue.
Trying to build a GUI test automation tool is like
trying to make a new operating system in such that
the available resources for the existed ones are hard
2
to compete with. However, GUI test automation
didn’t reach a mature level were those tools can be
implemented in all scenarios with no problems. User
Interfaces evolves rapidly with more rich and new
components. The automation processes in general use
artificial intelligence algorithms to simulate user
activities. The richness and complexity of this space
make it possible for new research ideas to compete
with long time existed test automation tools in the
industry.
There are several research papers about GUI testing
using the data model, [3] [4] [5] [7] [8] [9] [10]. The
overall goals and approach for this work is very
similar to their goals. The GUI testing framework
described, as a GUI test automation structure, is
generic and should be applied to any testing or GUI
testing model. Since research projects in general have
limited resources, each paper discusses a specific
area of GUI test automation. Without having all
components, it is very hard to use such ideas in the
industry. There is a need for using universal
standards for the output formats in all testing
activities. If each tool is producing several
intermediate outputs (generated test cases, recorded
scripts, log files, etc), it will be impossible to use
those tools or ideas in other research projects. Some
papers follow a complex procedure in test case
generation and do not consider any state reductions.
Assuming that changing any property in any GUI
object changes the GUI state is an assumption that
generated a very large number of possible states for
even small applications. State reduction techniques
are considered here to improve the effectiveness of
the track. We intended to follow the same GUI
testing framework for our future work and expect the
overall results to be more practical and easier to
apply on actual projects.
The second category[13][6] is related to semi test
automation using some capture/reply tools like
WinRunner, QuickTest pro, Segui silk, QARun,
Rational Robot, JFCUnit, Abbot and Pounder to
creates unit tests for the AUT. Capture/reply tools
have been existed and used for years. This may make
them currently more reliable and practical as they
have been tested and improved through several
generations and improvements. There are several
problems and issues in using record/play back tools
[11]. The need to reapply all test cases when the GUI
changes, the complexity in editing the scripts code,
and the lack of error handlings are some examples of
those issues. The reuse of test oracles is not very
useful in the case of using a capture/replay tool [12].
We expect future software projects to be more GUI
complicated that may make the test automation data
model more practical. Many researches and
improvements need to be done for the suggested data <Visible>True</Visible>
model to be more practical and usable. <Enabled>True</Enabled>
<ShortCut>None</ShortCut>
3. Goals and approaches </MenuItem>
Figure1: a screen shot from a generated XML file.
Our approach uses two constraints: The tool should
develop all test automation activities without or with
The above information about the controls and their
the least user involvement. The second constraint is
properties are extracted directly from the assembly.
that those dynamically generated test case should
We added two more properties; control level and
provide a value for test adequacy and should
control unit for each control. Those properties are
represent actual test case scenarios.
used along with the parent property to locate each
Since the test cases will be dynamically generated
control during test generation, selection and
with no need for a user to define pre- or post-
execution.
conditions, algorithms are developed for test case
The user has the ability to select certain controls from
generation.
the tree and gives them more than the typical equally
One important aspect of GUI’s that helps in state
distributed weight. This will have an effect on test
reduction is the fact that the GUI is hierarchical by
case selection algorithms. For example, a control that
nature. For example, in MS Notepad, to reach the
is selected manually by the user will be selected
event “Print”, the event “File” should be executed
whenever it is possible or valid to do so.
first.
Rather than defining the GUI state change as a
The GUI is analyzed and its hierarchical tree of
change in any control or control property [2], which
controls is extracted using reflection. The decision to
produces a large amount of possible test cases, we
use the XML file as a saving location for the GUI
define the GUI state as the state that is represented by
model is in the fact that XML is a universal format
the generated XML file from that GUI. If any of the
and it supports hierarchy. This hierarchy can be
parent-child relations in the GUI is changed, or any
extracted easily from the XML file. We used the
property of those parsed controls is changed, then
information about the parent of each control in the
that is considered a GUI state change.
assembly to build the GUI tree. Figure1 is a screen
Some abstraction is used removing those properties
shot from the XML output file extracted from a
that less relevant or important to the GUI state to
simple Notepad application.
reduce the large number of possible states. The
manually selected properties are: the control name,
<Root>GUI-Forms</Root>
parent name, text, locationX (horizontal location),
<Root>Form1</Root>
locationY (vertical location), text, or caption,
<Parent-Form>Form1</Parent-Form>
forecolor, backcolor, enabled, visible, control unit
<Name>Form1</Name>
and control level.
<Control-Level>0</Control-Level>
In the process of developing test generation
<TextBox>
techniques, several test generation algorithms are
<Parent-Form>Form1</Parent-Form>
developed. All algorithms check for a valid selection
<Name>textBox1</Name>
of a tree edge. For example, using Notepad AUT, if
<Control-Level>1</Control-Level>
the current control is “File”, then a test algorithm
<ControlUnit>0</ControlUnit>
may select randomly a valid next control from the
<LocationX>0</LocationX>
children of the File control ( e.g. Save, SaveAs,
<LocationY>24</LocationY>
Open, Exit, Close, Print). In another algorithm, we
<Forecolor>Color [DarkBlue]</Forecolor>
processed the selected test scenarios to ensure that no
<BackColor>Color [Linen]</BackColor>
test scenario will be selected twice in a test suite.
<Enabled>True</Enabled>
Figure2 is a sample output generated from one of the
<Visible>False</Visible>
test generation algorithms. In the algorithm, each test
</TextBox>
case starts from the root or the main entry “Notepad
<MenuItem>
Main”, and then selects two or more controls
<Name>System.Windows.Forms.MenuItem,
randomly from the tree. The algorithm verifies that
Items.Count: 4, Text: File</Name>
the current test case is not existed in the already
<Control-Level>1</Control-Level>
generated test cases.
<ControlUnit>0</ControlUnit>
<Parent>System.Windows.Forms.MainMenu,
1,NOTEPADMAIN,SAVE,SAVEFILEBUTTON1,,,
Items.Count: 3</Parent>
2,NOTEPADMAIN,EDIT,FIND,TABCONTROL1,T
<Text>File</Text>
ABFIND,

3
3,NOTEPADMAIN,VIEW,STATUS BAR,,,
4,NOTEPADMAIN,FIND,TABCONTROL1,TABR
EPLACE,REPLACETABTXTREPLACE,
5,NOTEPADMAIN,FIND,TABCONTROL1,TABR
EPLACE,REPLACETABBTNREPLACE,
6,NOTEPADMAIN,FIND,TABCONTROL1,TABR
EPLACE,REPLACETABLABEL2,
7,NOTEPADMAIN,EDIT,CUT,,,
8,NOTEPADMAIN,EDIT,FIND,TABCONTROL1,T
ABREPLACE,
9,NOTEPADMAIN,OPEN,OPENFILECOMBOBO
X4,,,
Figure2: A sample from a test case generation file.
To evaluate test generation efficiency in the
generated test cases, the total number of arcs visited
in the generated test cases is calculated to the total
number of arcs or edges in the AUT. File-Save, Edit-
Copy, Format-Font are examples of arcs or edges. An
algorithm is developed to count the total number of
edges in the AUT by using the parent info for each
control. (This is a simple approach of calculating test
efficiency. More rigorous efficiency measuring
techniques are planned in future work). Of those
tested applications, about 95% of the application
paths can be tested using 200-300 test cases.
For test verification, a log file is created to track the
events that are executed in the tool during the
execution process. In a simple example, figure 3
shown below, two test cases are generated that write
a text in Notepad and save it to a file. Those test
cases are generated using the tool.
1,NOTEPADMAIN,FILE,NEW,TXTBODY,,
2,NOTEPADMAIN,FILE,SAVE
AS,SAVEFILEBUTTON1,,,
Figure 3: a sample of generated test cases.
The first test case opens a new document and writes
to it. As part of the default input values, we set for
each control a default value to be inserted by the tool
through execution. A textbox writes the word “test”
or the number “0” whenever it is successfully called.
A menu item is clicked, using its parent, whenever it
is successfully called. For example, if Save is called
as a control, File-Save as an event is triggered. We
should have tables for valid and invalid inputs for
each GUI control. The second test case opens the
save file dialogue and clicks the OK or accept button
(Savefilebutton1), to save the document. Here is the
corresponding log file output for the above test cases.
Control Event Date Time
File new Menu Click 10/3/2007 11:51:23 AM
File new Mouse Down 10/3/2007 11:51:23 AM
4
File new Mouse Up 10/3/2007 11:51:23 AM
New txtbody Menu Click 10/3/2007 11:51:23 AM
New txtbody Mouse Down 10/3/2007 11:51:23 AM
New txtbody Mouse Up 10/3/2007 11:51:23 AM
TxtBody Mouse Move 10/3/2007 11:51:23 AM
TxtBody Key Down 10/3/2007 11:51:23 AM
TxtBody Key Up 10/3/2007 11:51:23 AM
(Test) is written in the document 10/3/2007 11:51:23 AM
(Test) is written in the document 10/3/2007 11:51:24 AM
SaveFilebutton1 Mouse Move 10/3/2007 11:51:24 AM
SaveFilebutton1 Mouse Button Down 10/3/2007 11:51:24 AM
SaveFilebutton1 Mouse Button Up 10/3/2007 11:51:24 AM
File SAVE AS Menu Click 10/3/2007 11:51:24 AM
File SAVE AS Mouse Down 10/3/2007 11:51:24 AM
File SAVE AS Mouse Up 10/3/2007 11:51:24 AM
SaveFilebutton1 Mouse Move 10/3/2007 11:51:24 AM
SaveFilebutton1 Mouse Button Down 10/3/2007 11:51:24 AM
SaveFilebutton1 Mouse Button Up 10/3/2007 11:51:24 AM
Figure 4 Log file output of a sample test case.
Since the test execution process is subjected to
several environment factors, the verification process
is divided into three levels.
1. In the first level the tool checks that every
control in the test suite is successfully executed. This
step is also divided into two parts. The first part is
checking that all controls executed are existed in the
test suite. This is to verify that the execution process
itself does not cause any extra errors. The second part
that ensures all controls in the test suites are executed
tests the execution and its results. In the
implementation of this level, some controls from the
test scenarios were not executed. This is maybe the
case of some dynamic execution or time
synchronization issues where some controls are not
available the time they are expected.
2. In the second level the tool checks that the
number of controls matches between the two suites.
3. In the third level, the tool checks that the events
are in the same sequence in both suites.
The verification process is automated by comparing
the test cases’ file with the log file. Time stamp is
important to verify the correct sequence of events.
The controls in the test case suites are written to a
sorted list and the execution sequence is also written
to another sorted lists. To verify the results of the test
execution, the two lists are compared with each other.
Upon testing several applications, a small percent of
controls generated in the test cases and not executed.
Timing synchronization causes some controls to be
unavailable or invisible within their execution time.
Regular users “wait” for a new menu to be opened.
This time varies depending on the application, the
computer and the environment. Time synchronization
and some other dynamic issues are part of the future
research goals.
One approach for the GUI test oracle is to have event
templates. For each test scenario listed, expected
results are written in a way that can be automatically
verified. This requires some level of abstraction controls in the categories with a proportion to their
where similar events are abstract into one event (like weight or occurrence.
the saving process). This proposal does not mean This algorithm is considered as a hybrid technique
exhaustively testing all probable events. By selecting that uses some of the capture/play-back processes. In
critical events for automation and abstracting similar a capture/ play-back tool, the same user session that
events we will get a valuable state reduction that is captured in the manual testing is executed. In this
makes the number of GUI states more practical in approach the controls’ weights are extracted from the
terms of testing. manual testing to guide test case generation and
Some algorithms are developed to calculate critical execution.
paths automatically. Those algorithms are:
1. Critical path using node weight. In this approach, 4. Conclusion and future work
each control has a metric weight that represents the
Developing a user interface test automation tool faces
count of all its children. For example if the children
several challenges. Some of those challenges are:
of File are: Save, SaveAs, Close, Exit, Open, Page
Serializing the GUI widgets, test results’ verification,
Setup, and Print, then its metric weight will be 7
time synchronization issues, handling dialog boxes,
(another alternative is to sum all the decedents). For
testing data dependent systems, and building error
each generated scenario, the weight of that scenario is
logging and recovery procedures.
calculated as the sum of all the weights of its
Some techniques, in test case generation, execution
individual selected controls.
and verification, are explained in principles in this
2. Critical path level reduction. In this approach, a
article. Test case generation from user sessions is
test scenario is randomly selected. The selected
explored that represent real usage of the application
scenario includes controls from the different levels.
and focuses on the application areas where they can
Starting from the lowest level control, we exclude
be heavily used by users. A logging procedure is
from selection all those controls that share the same
implemented to compare the executed suite with the
parent with the selected control. This reduction
generated one. A formal verification procedure is
shouldn’t exceed half of the tree depth.
presented that compare and verifies the output of the
For example if the depth of the tree is four levels, we
execution process with its input. Another track of
should exclude controls from levels three and four.
verification is suggested. This track requires building
We assume that each test scenario should start from
templates for events. For each event pre conditions,
the main entry point and that three controls is the
post conditions and expected results are included.
least required for a test scenario (like Notepad – file –
More elaborations and verifications are required to
exit). We select five consecutive test scenarios using
prove the effectiveness of the suggested approaches.
the same reduction process described above.
Automation of the first few test cases is expensive;
3. The previously described algorithms for critical
beyond that they become much cheaper to develop
paths’ selection depend on statistics pulled from the
and execute.
implementation model. As an alternative, we can
Test automation can only be successful when we
analyze several user captured sessions (e.g. from
keep in mind that testing in general, and particularly
testers or users in beta testing) to automatically
automated testing, is easily made obsolete by some
weight the GUI controls. User session data is the set
changes in the application and environment.
of user actions performed on the AUT from entering
In GUI, it is difficult to reach a high level of test
the application until leaving it.
adequacy in generating test cases that cover all
The session logs all the controls that are executed in
possible combinations. GUI state reduction is needed
the different scenarios. A simple count or percentage
to make testing all or most possible states feasible.
is given to each control depending on how many
In this approach, there is state reduction from
times it is listed in those scenarios. The test scenarios
selecting specific properties to parse. Those
should include all primary and major use cases for
properties are more relevant and critical than the rest
the tested application. The controls’ weights
for the testing process. Total properties of less than
(calculated from user sessions) can drive the test case
10 are selected. The idea of test automation is not to
generation and execution. Theoretically all controls
automate everything; we automate to save time and
should get the same weight in the generated test suite.
effort.
However, in real scenarios this may not be true.
The other issue that causes state reduction is
We may cluster the controls according to their usage
considering the hierarchy. In flat state diagrams, we
from user sessions into three levels; heavily used,
assume that any state is reached from another state
medium and low. Depending on the availability of
once certain preconditions are achieved. In GUI
the resources to testing, we may choose one or two
terms it means that any control is accessible from any
categories and generate test cases that cover those

5
other one. This is not true in GUI states, as usually
controls are only accessible through their parent
controls.
We can also get GUI states reduction by abstracting
the processes. For example, the saving process has
several intermediate processes (file-save-as, file
name, type… ok), but what is important is the end
state where there are certain results if the process is
successful (e.g. a file with certain content is saved to
a certain location) and other results if it is not.
We don’t automate testing everything. We automate
testing those frequently repeated tasks. We don’t
automate those tests that require user validation or
those tests that require special dynamic or time
synchronization consideration (such as connecting to
a database or verifying the overall acceptance of a
GUI).
5. References
1. Yury Makedonov. Managers guide to GUI test
automation. Software test and performance conference.
2005. http://www.softwaretestconsulting.com/
Presentations_slides/Manager_sGuide_GUI_TestAutomati
on_11wh.pdf.
2. George Nistorica. Automated GUI testing.
http://www.perl.com/pub/a/2005/08/11/win32guitest.html.
2005.
3. Q. Xie. Developing Cost-Effective Model-Based
Techniques for GUI Testing. In Proceedings of The
International Conference of Software Engineering 2006
(ICSE’06). 2006.
4. A. M. Memon and Q. Xie . Studying the fault
detection effectiveness of GUI test cases for rapidly
evolving software. IEEE Transactions on Software
Engineering, 31(10):884-896, 2005.
5. A. M. Memom, I Banerejee, and A. Nagarajan. GUI
Ripping: Reverse Engineering Of Graphical User Interfaces
For Testing. In Proceedings of the 10th Working
Conference on Reverse Engineering ( WCRE’03), 1095-
1350/03. 2003.
6. A. K. Ames and H Jie. Critical Paths for GUI
Regression Testing. University of California, Santa Cruz.
http://www.cse.ucsc.edu/~sasha/ proj/ gui_testing.pdf.
2004.
7. A. M. Memon. Developing Testing Techniques for
Event-driven Pervasive Computing Applications.
Department of Computer Science. University of Maryland.
8. A. M. Memon. GUI testing: Pitfall and Process.
Software Technologies. August 2002. Pages 87-88.
9. A. M Memon. A Comprehensive Framework For
Testing Graphical User Interfaces. Ph.D. thesis,
Department of Computer Science, University of Pittsburgh,
July 2001.
10. A. M. Memon, and M. Soffa. Regression Testing of
GUIs. In Proceedings of ESEC/FSE’03. Sep. 2003.
Saket Godase. An introduction to software automation.
http://www.qthreads.com/articles/testing/an_introduction_t
o_software_test_automation.html. 2005.
12. Brian Marick. When should a test be automated.
http://www.testing.com/writings/automate.pdf. (Presented
at Quality Week '98.).
13. L. White and H. Almezen. Generating test cases from
GUI responsibilities using complete interaction sequences.
In Proceedings of the International Symposium on
Software Reliability Engineering, pages 110-121, Oct
2000.
14. A. Mitchell and J. Power. An approach to quantifying
the run-time behavior of Java GUI applications.