Sunteți pe pagina 1din 44

An AUTOMATED TESTING INSTITUTE Publication - www.automatedtestinginstitute.

com

A
S
utomated .......
T
oftware esting MAGAZINE
September 2010

Automation Is What
We Do!
Ending The Developer/
Automator TUG OF WAR

Continuous Int. Development’s Testing With


And Automation Role Reflection
CI teams with How Development Build a Utility To
Automation to improve Helps to improve Dynamically Learn
code quality automation .NET Assemblies

Dynamic Data Entry: 9 Steps To Creating A Dynamic Data Entry Function


Are You Contributing
Content Yet?
The Automated Testing Institute relies heavily on the automated testing
community in order to deliver up-to-date and relevant content. That’s why
we’ve made it even easier for you to contribute content directly to the ATI
Online Reference! Register and let your voice be heard today!

As a registered user you can submit content directly to the site, providing you with
content control and the ability to network with like minded individuals.
Community
Comments Box >> Community Comments Box - This comments box, available on the home page of
the site, provides an opportunity for users to post micro comments in real time.
>> Announcements & Blog Posts - If you have interesting tool announcements, or
you have a concept that you’d like to blog about, submit a post directly to the ATI
Announcements & Online Reference today. At ATI, you have a community of individuals that would love
Blog Posts to hear what you have to say. Your site profile will include a list of your submitted
articles.
>> Automation Events - Do you know about a cool automated testing meetup,
webinar or conference? Let the rest of us know about it by posting it on the ATI site.
Automation Add the date, time and venue so people will know where to go and when to be there.
Events
Learn more today at http//www.about.automatedtestinginstitute.com
De
A
S
utomated
T
oftware esting
Au
to
m
ve
lo
pe
r
at
September 2010, Volume 2, Issue 3
or

Contents
Ending Developer/Automator Tug of War
If you’re interested in bridging the gap between software developers and software test automators, then you’ve
come to the right place! While both groups may feel that theirs is the only true automation team in Information
Technology, there’s really more that connects the two teams then divides. Delve into this issue of the AST magazine
to learn how automators may participate in software product development, how developers may participate in
software test automation, and how one resource could even function in both capacities at once.

Features
Continuous Integration and Automation 16
Like many of the most popular pairs in history continuous integration and automation have the power to be an unstoppable
force when it comes to ensuring the quality of your software product’s code. By Eric Pugh

Development’s role in software test Automation 24


This article identifies developer tasks necessary for making test automation possible and efficient. In addition, learn how
assisting test automation efforts actually results in developers assisting themselves and the end user By Linda G. Hayes

Testing With Reflection 28


Leap into the reflection pool with this article that provides detailed instructions for developing a tool that automatically
produces information and data for learning about and testing .NET assemblies and web services. By Nick Olivo

Columns & Departments


Editorial 4 I ‘B’log to U 40
Developer/Automator Tug of War Read featured blog posts from the web.
Learn about the ties that bind the two IT groups.
Hot topics in automation 42
Authors and events 6 2nd Annual ATI Automation Honors Is Hot!
Learn about AST authors and upcoming events. The polls are coming to a close. Who will get crowned?!

Sweat the technique 8 The Automated Software Testing (AST) Magazine is an Automated Testing
Institute (ATI) publication.
Dynamic Data Entry For more information regarding the magazine visit
http://www.astmagazine.automatedtestinginstitute.com
Learn how to create a reusable dynamic data entry function.
September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 3
Editorial

Developer/Automator Tug of War


by Dion Johnson
Automation is generally an
ambiguous term, but for many years within
the realm of Information Technology (IT)
there was little ambiguity. The term was
clearly and distinctly used to describe the
activities and products that came out of
the software development team. Once
the test team began to use automation
for testing tasks, however, the waters
began to get a little muddy. Now when IT
project teams use the term ‘automation’,
particularly when testers are involved
in the discussion, they are very often
referring to software test automation
activities led by the test team. Software
developers are often baffled by this
because they still believe ‘automation is
what we do’. Hence a tug of war between
test and development teams began.

In truth, however, there is more that blurred. I’ve been on projects where that software developers may play in the
connects test automators and developers simulators have been developed and used successful creation, implementation and
than divides the two groups. For example, by test teams to aid in testing a software maintenance of software test automation.
I regularly receive software development product, then used by development as a In an article entitled “Development’s
job requisitions from human resource stub for developing related components. Role in Software Test Automation”,
personnel that have located my resume On other projects developers have Hayes addresses crucial developer
from somewhere out on the internet. authored simulators that the test team tasks that are necessary for making test
This is odd given the fact that the bulk was able to repurpose in completing automation both possible and efficient.
of my experience is in quality assurance, their QA tasks. I also know of resources In addition, she relays how assisting
test automation efforts actually results
in developers assisting themselves and
...this issue is dedicated to the celebration of the end user. Next, Mr. Eric Pugh speaks
from the developer side of the fence on
the developer/automator connection. how test automation may aid developer
software builds. His article entitled
“Continuous Integration and Test
Automation Go Together Like Peanut
quality control, requirements analysis that have used skills they acquired as Butter and Jelly” is surprisingly not
and process improvement; a fact that software test automators to springboard really about a sandwich! It reveals the
is clearly reflected in my resume. Why them into a software developer position importance of test automation to ensuring
then would I be receiving software and were able to hit the ground running. code quality through a Continuous
development job requisitions? It is my Inversely, I have a colleague that has Integration cycle. Finally, Mr. Nick
skills and experience in software test staffed her entire automated software test Olivo picks up where our last issue left
automation that piques their interest. team with software developers, that were off with an article dedicated to testing
Maybe they aren’t looking closely able to get going following some QA and with reflection. His aptly titled, “Testing
enough at the resume to notice the automation training. So this relationship with Reflection” article delves much
differences between what I’ve done and should be celebrated as one that is deeper into the reflection pool, however,
what software developers do, but the complimentary as opposed to relegated with instructions for developing a tool
similarities between my test automation as one that is competitive. that automatically produces information
work and the work performed by software and data for learning about and testing
development professionals regularly gets Therefore, this issue is dedicated to .NET assemblies and web services.
their attention. the celebration of the developer/automator This article reveals how a single person
connection. Ms. Linda Hayes provides can simultaneously embody the roles
In many other instances the lines an article that addresses the relationship of software developer and software test
between the two groups are extremely between the groups by expressing the role automator.

4 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


For more information, visit http://www.networking.automatedtestinginstitute.com
September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 5
Authors and Events

Who’s In This Issue?


Linda G. Hayes has over 20 years of
experience in software development and testing,.
A
S
utomated
T
oftware esting
She is the founder of three software companies
including AutoTester, the first PC-based test Managing Editor
automation tool, and Worksoft, Inc., pioneer of Dion Johnson
next generation code-free test solutions. Linda
Contributing Editor
holds degrees in accounting, tax and law and is
Edward Torrie
a frequent industry speaker and award-winning
author on software quality. Named one of Fortune Magazine’s People to Director of Marketing and Events
Watch and one of the Top 40 Under 40 by Dallas Business Journal, she is Christine Johnson
a regular columnist and contributor to numerous publications, including
the Automated Software Testing Magazine, StickyMinds.com and Better
A PUBLICATION OF THE AUTOMATED
Software magazine. She is the author of the Automated Testing Handbook, TESTING INSTITUTE
contributing author to Testing SAP R/3:A Manager’s Guide, and coeditor
of Dare to be Excellent with Alka Jarvis on best practices in the software
industry. Her article “Quality is Everyone’s Business” won a Most Significant
Contribution award from the Quality Assurance Institute and was published
as part of the Auerbach Systems Development Handbook. You can contact
Linda at Lhayes@worksoft.com.
CONTACT US
AST Magazine
Eric Pugh has been heavily involved in the open astmagazine@automatedtestinginstitute.com
source world as a developer, committer, and user for
ATI Online Reference
the past 5 years. In biotech, financial services and contact@automatedtestinginstitute.com
defense IT, he has helped European and American
companies develop coherent strategies for embracing
open source software. As a speaker he has advocated
the advantages of Agile practices in software development. Currently, ATI and Partner Events
Eric is focusing on building his Solr consulting practice, and is coauthor
of the Solr 1.4 Enterprise Search Server book. His blog site is http://www. September 17, 2010
opensourceconnections.com/blog/ ATI Honors Voting Concludes
www.atihonors.automatedtestinginstitute.com

Nick Olivo is a Software Engineer, test automator October 2010


and tech writer all rolled into one. In addition, he
ATI Honors Video Presentation
specializes in exploratory testing, JScript, Python, www.atihonors.automatedtestinginstitute.com
C#, and presenting. He currently works for a leading
tool vendor, and is the lead author of a blog entitled October 28, 2010
Autonomicon. Autonomicon has a loyal following
Mentora/ATI Virtualization
and has been nominated in both the 1st and 2nd Annual ATI Automation Webinar
Honors Awards. Find his blog at http://www.autonomicon.blogspot.com/. http://www.mentora.com/testing_virtualized_sys-
tems/testing_virtualized_systems.html

6 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


2nd Annual

ATI Automation
Honors
Celebrating Excellence in the
Discipline of Software Test
Automation

Nominations: April 1 - June 1


Voting: June 14 - September 17

www.atihonors.automatedtestinginstitute.com

September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 7


Sweat the Technique

Dynamic Data Entry


To the Rescue

Application behavior can be sinisterly may be maintained in an external file, as in Table 1.


dynamic, so it’s time that our automated scripts be
more dynamic to combat the application under The information then may be entered into the
test (AUT). One way to accomplish this is by application with statements that resemble
providing our scripts the ability to stretch those in Figure 1.
a minimal amount of scripting as far as it
will go via dynamically inputting data. Using a data-driven approach
Dynamically inputting data, which allows all three rows of data to
simply refers to the ability to input be entered using the same
or verify a lot of data with group of inputs, but each
minimal scripting, greatly input into a specific object
increases the elasticity of the still requires a separate
scripts. Elasticity is important statement (illustrated in
because time and resources the highlighted statements
often are limited for automated in Figure 1). This can be
script development. Reducing time particularly arduous when
and effort involved in the script there are a lot of fields that
development and maintenance require input on a given page. Not
helps to make more scripting to mention that if it becomes necessary
and analysis possible, which to come back to the page and verify the
ultimately will help to increase data still exists in these fields, another
the coverage achieved by the set of statements must be added to the
scripts. test to verify each field. Not only is this

This technique is similar


Dynamic Data laborious, but also it really clutters the
script, thus decreasing the ability
to data-driven automation
which addresses the ability Man to read and understand what the
script is doing. Dynamic scripting
to input multiple values techniques provide the ability
into a field by parameterizing to reduce the script to the much
the field’s inputs. The approach being introduced now actually simpler set of statements shown in Figure 2.
suggests parameterizing the fields themselves, as well as the
inputs. This level of parameterization provides the ability to This is made possible by creating a reusable function that
input a variable number of data elements with a single statement.leverages the information that is in the data table and the
Making your scripts super scripts! information that exists in the object map(s) for that application.
(Note: This could be accomplished even if you don’t utilize an
This principle should not be confused with keyword object map, but here we’ll focus on how to accomplish it with an
automation. Keyword automation still typically requires each object map.) An object map (also known as an object repository)
and every input to have a separate line in the keyword file. The is a file that stores information about the application objects that
technique being proposed here can be used to reduce the amount may be automated in a script. This file contains the properties
of statements necessary associated with a
for automating a test, particular object
whether using standard and assigns a
scripting or keyword logical name to
automation. that object. For
example, the First
Let’s examine a typical Name text box and
data-driven approach to Table 1: Date Parameters Phone list box in
inputting data. The data an application may

8 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


Sweat the Technique

dataTable - This will indicate to the


function what data source must be opened
for retrieving the data.

currentRow - This is the current row


number in the data source from which
data will be retrieved.

Step 2. Open Data Table

Figure 1: Standard Input Code Structure This step opens the data source and
makes it accessible to the function.
the Parameter in the data table is identical
Screen{
to the logical name in the object map, a Step 3. Get Total Number of Input
FirstName{
function can be created to do the bulk of Columns
Class: textbox
the input logic for you. This function may
Id: fname001
resemble Figure 3. This step gets the total number of input
Name: First
columns in the data source. For example,
}
This function is what would be Table 1 has six columns (excluding the
Phone{
Class: listbox
Id: phn001
Name: Phone
}
)

appear in an object map as illustrated


above.
Figure 2: Dynamic Input Code Structure
Note that the logical name for the
First Name text box is ‘FirstName’, while
called in line 3 of Figure 2. Below is an first column with the numbers in it). The
the associated properties of this object
explanation of this function. syntax for actually performing this will
are the Class, Id, and Name properties.
vary depending on the data source and
(Also note that this article doesn’t use
Step 1. Define Function and Pass the tool used to access the data source.
actual syntax, but rather pseudo-syntax
Appropriate Parameters For example, when accessing Microsoft
simply to convey the essence of what the
Excel via Vbscript, the CreateObject
actual syntax should accomplish. Refer to
In this step, the function is defined function can be used for invoking Excel,
the documentation related to your tool of
with three arguments: followed by Workbooks.Open to open a
choice and the data source to determine
specific table.
what the actual syntax should be.) The
name and properties of the object are Page - Since object maps typically
what allow an automated test to find segment objects by page, it is necessary to Step 4. Create Column Loop
and perform actions on an object in the identify on which page the objects exist.
application. Provided that the name of This loop will be responsible for
accessing each data element in a single
row and inputting the data into the
application.

Step 5. Get Column Header

The parameter name given at the top


of a column must be obtained, because
this name will be used for finding
the appropriate object in the object
repository. As previously mentioned, the
Figure 3: Input_All_Data Function Structure syntax for getting the column header will

September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 9


vary depending on the data source and the a Set statement, while inputting data into determines what syntax will be required
automation tool used for accessing the a list box may require a Select statement. to input data into an object based on that
data source. Once the object’s class is determined, the class. For example, the statement may
function can be instructed on how to input resemble Figure 4.
Step 6. Get Object and Object Class data into that object.
Note that the major input field types are
This step is crucial in determining Step 7. Input Data addressed in the branching construct and
how to input the value into the object. that the statements used for inputting the
There are often different statements or This step represents a branching data associated with the table parameter
keywords for inputting data into different construct (e.g., a select/case branching varies depending on the object class.
types of application objects. For example, construct), that takes the object class
inputting data into a textbox may require captured in the previous statement and
Steps 8 & 9. End Loop and Function

Steps 8 and 9 close out the loop and


the function respectively.

Once this type of function is created,


it is reusable, simple to maintain, and
powerful. A similar function can be
created to verify data values, so now
multiple input fields can have data
entered and verified with only a couple of
Figure 4: Input Branching Construct statements added to a test script.

Join The Movement

10 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


Development’s Role
(Continued from page 27)
when changes are encountered. This is issues do not surface, and by the time they
an inefficient process at best because it understand the impact on automation the

C
requires repeated test/fix/test cycles, and effort to remedy the issues is not permitted
a risky approach at worst because it may
allow errors in new functionality to go
by the schedule.
The key, then, is to engage with
ontribute
undetected. development early to educate and

C
The ideal solution to this challenge indoctrinate them. Explain their role

ontent
is for development to actually supply the and the impact on automation. It’s also
object inventory. For some applications, important to educate management to avoid
this information is already stored in the the perception that testability imposes
metadata or database; for others, the additional overhead without offsetting

T
developers may have to add extra code benefits. Sell them on how automation not
to register or log the objects as they
are defined. But by whatever means,
only helps testers, it accelerates the overall
life cycle for developers and improves oday
if development can produce the object quality for end users. Developers can
inventory for each build, then automation leverage automation for unit, integration
engineers can focus on identifying the and build testing and testers can expand
changes and updating or adding new tests. their test coverage dramatically, both of
Once the object inventory is acquired, which result in a more reliable system.
detecting changes can be fully automated Further, consistency of coding conventions
by simply comparing the previous version makes it easier for developers to transfer
and maintain the software over its useful Community
to the new one and flagging any additions,
deletions or modifications for review. life. Comments Box
It should be noted that not all changes If the basics of automated testability
will be reflected in the object inventory; are communicated and understood from the
functional requirements may introduce start, it is often a straightforward proposition
new business rules or edits that involve to integrate them into the development
existing objects. Thus managing the process. Developers can invest a few
object inventory is not a substitute for a minutes in their coding approach and save Announcements &
robust requirements management process, automation engineers and testers untold Blog Posts
but it can work wonders for improving the hours of work down the line
efficiency of maintaining automated tests.
The next step is to automate impact Summary
analysis, which is relating the object or
requirements changes to all affected tests The way in which an application
so they can be reviewed and updated. That is developed can enable or prevent Automation
is, once changes to the object inventory test automation altogether. If you can
or requirements are identified, then all successfully integrate automated testability Events
tests that reference them are candidates throughout the development process and
for maintenance. Depending on whether application life cycle, you will be able
tests are stored as code or data, or some to redirect enormous effort away from
combination thereof, this process may be struggling with intractable impediments
automated as well. and towards creating better and more As a registered user you
comprehensive test coverage. And that, in
Extending automation into change the end, is what it’s all about. can submit content directly
management assures that all of the effort
invested into test automation is protected to the site, providing you
and enhanced over the life of the
application, instead of compromised or
with content control and
lost by the very changes that make testing the ability to network with
critical.
like minded individuals.
Education and
Indoctrination
As you can see, development’s role Learn more today
in test automation is pervasive, but all too
often testability issues are not uncovered at
until the application has already been
developed and it is too late to resolve
them. In my experience, developers are
not unwilling to fulfill their role, they are http//www.about.automatedtestinginstitute.com
simply uninformed. Most are familiar
with manual test processes in which these

September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 11


Open Sourcery

Repurposery
Open Source Tools Built From Other Open Source Tools

multiple members of the QA lifecycle, this


framework deploys more than 30 open
source projects including JUnit which is
used for writing tests and Ant which is
used as the scenario execution engine.

TestMaker – This automated testing

R epurpose: To reuse for a


different purpose. To use or
framework is built for developers,
software quality engineers and IT
managers to work in concert to effective
convert for use in a format different validate an application. TestMaker
integrates several tools including soapUI,
from the one originally intended.
Selenium and HTMLUnit to allow team
members to develop automated unit test
scripts that get repurposed as automated
functional test scripts. The functional
scripts may then be repurposed as
automated performance test scripts or
In conjunction with the internet, the open It turns out, however, that these new business service monitor.
source paradigm has transformed how challenges are not always specific to the
technology is offered and used. Gone are users that originally repurposed the tool. Typically, open source tools have a single
the days where users are forced to choose Very often, the same challenges are faced purpose, but through repurposing them,
between building a tool from scratch by many other projects around the world, the capabilities are greatly expanded.
and using a proprietary tool that is often making the case for the framework built on Repurposing may very well end up being
fraught with limitations that hopefully get repurposed open source tools to become the vehicle that brings open source closer
fixed in some subsequent release. Open an open source offering itself. And that to being on par with commercial tools.
source allows users to not only get free is exactly has been happening with many Commercial tools often enjoy advantages
tools, but the source code behind those open source tools out in the free market over open source tools in the areas of

Repurposing may very well end up being the vehicle that brings open
source closer to being on par with commercial tools.
tools. Thus, if a user requires specific today. Many of the most popular open support, training and flexibility. While
functionality not currently in the tool, the source tools that are available are built on
repurposing doesn’t directly address
ability exists for the user to create that top of existing, repurposed open source
support and training, it does impact
functionality from an established code automated test tools. Two of these tools
greatly the advantage of flexibility. At
base. In addition, users often integrate are very popular and are finalists in the
times commercial tools are more flexible
various open source tools together into 2nd Annual ATI Automation Honors
with respect to how they may be used
an in-house framework that uses the tool awards: Jsystem and TestMaker.
for a different purpose or in a different and with what technologies they support.
format than it was originally intended. Jsystem – This automated test tool Repurposing takes a good, yet narrowly
This repurposing allows the tool to be functions as a framework for writing and focused tool and expands its reach,
expanded to meet a new and specific managing automated system tests. In order allowing open source tools to be more
challenge. to make it a collaborative environment for competitive in areas beyond just price.

12 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 13
14 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010
September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 15
Continuous
Integration
And
Automation
Go Together li

16 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


The promise of test automation is that by investing in
writing automated tests your code’s functionality can be
quickly verified as changes are introduced. The promise of
Continuous Integration is that at any time you check out
the code, it will compile and run correctly, as proven by the
Continuous Integration loop. This means you’ll never hear
a developer say “but it works on my machine!” But without
great automated testing, a Continuous Integration loop
is nothing more than a glorified compilation tool.

By Eric Pugh

ike Peanut Butter


and Jelly
C ontinuous Integration without automation
may prove your code is syntactically cor-
rect, and can compile, but does nothing to prove
that the intent of the developer was met. Like-
wise, running automated tests on an irregular,
or infrequent schedule means you lose much
of the value of having those tests, since by the
time the tests identify an issue, the developer
has moved on to other tasks.
The promises of both Continuous Integration
and test automation only come to fruition when
they are performed in conjunction with each
other! One complements the other. They go
together like salt and pepper, like Batman and
Robin, like… peanut butter and jelly!
I
n this article we’ll briefly cover how to implement While there are a number of great open
a Continuous Integration system, how to source and commercial packages out
prioritize what tests to automate, and some of there, I recommend you look at the open
source tool Hudson as a good starting
the people factors involved in building out a suite of
point when dipping your toes into the CI
automated tests. waters.
Hudson is available from http://
What is Continuous environment, and testing the build in hudson-ci.org/. You can download it
Integration? multiple environments, is just icing on the either as a Java WAR file that you deploy
CI cake. into the servlet container of your choice
such as Jetty or Tomcat, or you can use the
Continuous Integration (CI) is often The term Continuous Integration built in web server via java -jar hudson.
bandied about by process consultants as was coined by Martin Fowler in 2000, war on the command line. Configuration
a complex process that can ensure code who came up with the definition: a is simply a matter of putting in some
quality metrics are met, new features are fully automated and reproducible build, configuration data about the source of the
magically deployed to production, and including testing, that runs many times code, what build script to run, and who to
nary a bug ever creeps into your code! But a day. You can read his original article notify. The Apache Software Foundation,
at its heart, CI is actually a very simple online at http://martinfowler.com/articles/ home to some of the most popular open
source projects, runs a large Hudson
installation at http://hudson.zones.apache.
org/hudson/ that you can play around
with.
Once you have your build server
up and running, you need to provide an
automated build script. And by automated
build script, I mean something that runs
on the command line! If you are building
a .NET project, and your build process
is “right click on Project menu, choose
Assemble Project” then you do not have
an automated build! It’s time to get
down and dirty with the documentation
for Visual Studio and figure out how to
run the build via command line with no
human intervention. The same goes if
you have any manual steps during an
otherwise automated process. I’ve seen
Figure 1: Conceptual Model people say “First I run this build script
for Project A, then I copy the generated
artifacts to Project B, and then run the
idea: every time the code is changed, it’s continuousIntegration.html. build for Project B”. In that case, attaining
compiled, a battery of tests are run, and a full automation is a fairly straightforward
notification of the test results is sent to the process of scripting the moving of build
Getting Started artifacts between Project A and Project B.
developer.
CI is not a revolutionary idea: the idea Lastly, you need a way of notifying
The actual mechanics of implementing the development team of the results of the
of the daily build with an automated sniff CI are pretty straightforward. You’ll need
test has been around for a long time. What build. Often this is through an email sent
the source code to be stored in a version to the developers mailing list, or through
is new is that the process of integrating control system such as SVN. You’ll need
new code occurs as the code commits a web dashboard.
a build server, preferably a beefy one, that
come fast and furious from a team of has all the tools necessary for compiling One of the challenges with email is
developers throughout the day, ensuring the code and running the automate tests. that it is very intrusive in your workflow.
that they continually adjust to the changes And you will need some software to Imagine that you are in “the zone” working
done by each other. coordinate the process of monitoring for on an interesting problem and ‘ding’,
All the other stuff, like code quality code changes, to initiate the build, and your email alerts you to a notification
reports, promoting the build into a QA notify the interested parties of the results! that the build succeeded! If this happens

18 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


Figure 2: Web Dashboard

once, it’s not a problem; several times a device that provides you with some Keeping it Running
day, however, can be disruptive. A web information without shouting at you. One
dashboard is problematic as well, given of the most popular ambient information
Getting CI setup is one thing, but
that it requires constant visits to a web devices is to hook up two lava lamps - a
keeping it running is another. I talked
page to find out the status of your build. green one, and a red one - using cheap
to a Project Manager at a conference
The solution? An ambient information X10 protocol power modules. When
once about his CI system, and he talked
the build succeeds, the green lava lamp is
about how rock solid it was. His exact
illuminated, and the red one is turned off:
words were, “I never get an email about
But if the build fails, then the green the build failing”. After the conference
one is turned off and the red one turned I followed up with him about his builds,
on. One of the things that I like about and he sheepishly acknowledged that
using lava lamps is that while at first in fact the builds were succeeding only
glance they are a binary system of green/ because the testing phase of the build had
red, they actually are an analog system of been commented out! The challenges
notification when you add in the time it with having a successful CI system
takes for the lava lamp to melt the wax and revolve around cultural (otherwise known
start it bubbling. In some organizations as people!) issues and project structure
breaking the build, and therefore turning
issues.
on the red lava lamp is a big no-no and is
frowned on. However, I like to think of Often developers feel the CI loop is
the CI build as a buddy looking over my like Big Brother from George Orwell’s
shoulder and helping me work better. So novel 1984, watching over them and trying
if the red light is on, thats okay, as long as to catch them in a mistake. Especially
the red lava doesn’t start bubbling! If I when the CI system is used as leverage
break the build, causing the red lava lamp to perform other tasks not directly related
to light up, I know I have 10 minutes or to verifying the code, like measuring
so to fix the build and turn the light off code quality, looking for inappropriate
before that lava starts bubbling! cut and paste of code, and providing

September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 19


a leader board of who has broken the issues must be resolved so the team can manual test suite so that you quickly start
build the most. Indeed, there is even a start gaining the value of investing in gaining the most value from those you
website: YouBrokeTheBuild.com that has the CI system by building the suite of automate.
a humorous poster that suggests breaking automated tests. One of the quickest If there is no unit testing being
the build is the worst thing possible! ways of disillusioning the team on the performed by the developers, then getting
I prefer a more positive perspective value of CI is to have tests that sometimes the developers started on writing them
that by having CI available to me, I have pass and sometimes fail. Having rock is very low hanging fruit. It provides
the guarantee that when the CI loop solid testing will make or break your CI immediate rewards by encouraging them
completes my code is solid. I also have implementation! to think about testing, providing the first
the convenience of running tests that layer of testing, and encouraging them to
focus on my area of code change while So Where Does Test write more modular, testable code. Then
my invisible partner (the CI loop) is Automation Fit In? look at your existing test suite. Do you
responsible for running the lengthy full have functional or UI based tests that
suite of regression tests. As a developer, I This is where test automation is could be more effectively implemented as
can be more productive in my work by not important to the successful implementation unit tests? For example, you may have
constantly second guessing what impact of CI. You may already have a significant a tax calculating test for an e-commerce
on the system every code change or suite of tests, all carefully documented site that requires creating products with
refactoring might have. By committing certain prices, and generating tax
on a frequent basis, we reduce the tables for specific states by clicking
chances that various changes will around an admin web interface. The
conflict with each other, and when tax calculation business logic may
conflicts happen, we can quickly be more easily tested in isolation as
narrow down the code change part of the unit testing layer with
that caused it. I remember a time mocks for the products and locale
when a weekly build on Thursday information. What about sniff or
would be followed by an entire smoke tests that are manually run
Friday (and sometimes Saturday) after each build? Those are a prime
was spent in reconciling conflicting candidate for automating and
code changes because we had 4 days adding to the CI loop. Great sniff
worth of changes from a 12 person tests verify that the full breadth
development team to dig through. of the application, including all
Keeping the CI system running dependencies, appear to work at
also requires structuring the project a macro level, and typically don’t
to work well in an automated CI loop take much time to run. So instead
environment. Setting up CI requires of running the sniff tests on a
an investment in time and energy, periodic basis, start running them
and if your project isn’t structured to after every code change so you
work well in that environment, then can identify when a small code
it’ll seem like a waste of time. For change makes a big error! How
example, the manager I previously many times have you sat down to
mentioned whose project wasn’t perform a session of Exploratory
running any tests was probably partly Testing and clicked the login
upset that the tests were not being run, button and had the application
but also upset that his team invested blow up immediately? Automated
in a CI system that they aren’t gaining in Microsoft Word documents, that you sniff tests will ensure that basic functions
value out of! Probably chief among are performing manually before every such as logging in work properly before
structural issues, after automating the release; but until you fully automate those releasing the code to the QA team for
build process, is having a project that is tests, they don’t have a place in the CI further testing.
difficult to test. Is the system one giant loop! After you’ve identified manual tests
code base that doesn’t allow unit testing So how do we start automating our that would be better suited as unit tests and
or make it easy to mock up dependencies? tests? I typically look at testing as not just automated your sniff tests, look at your
Does it require a fully populated database the responsibility of the QA team, but of historical and current bug list, and see
that takes an hour to build in order to run the entire team, so the effort of embarking how that matches up to specific manual
the functional tests? Do tests sometimes on test automation should be shared tests. You should automate the tests that
produce false positives based on external across all the team members. Begin by cover modules that currently have open
service dependencies? These testability performing a triage operation on your full bugs, and then look at which modules

20 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


have historically had a large number of maintained by the developers, to higher servers. Also assess whether or not these
bugs and look to automate those. This level functional tests that validate the tests should be run every time the code
way you are focusing on what are likely to integration of large modules, to system changes, or if they can be run out of band
be the problem areas in the future. Don’t tests that verify that all the systems in a in their own parallel process? Set up an
have good historical records of which major application work together. Not to independent build loop that includes all
modules had bugs? Another approach mention tests that have a specific focus the longer running tests that is kicked off

Complexity Code Coverage

Figure 3: CruiseControl Treemap Diagrams

is to prioritize your testing by looking at such as load tests and compatibility after the successful short CI build loop is
which modules contain complex code and tests. Within the CI world, the tests that completed.
lack decent unit testing code coverage. give you the biggest bang for your buck Another technique is to run just a
The theory being that complex code start from the most focused tests, the unit subset of the full test suite. Either run
with out unit testing is going to be the tests, and work their way outwards to tests related to the specific modules that
source of more bugs. For the Java world, the system level tests. Because of the have been impacted by the changes, or do
there is a great tool called Panopticode overriding requirement that the build loop a rolling sub sample of the tests on each
(http://www.panopticode.com) that you be as fast as possible, with 10 minutes
run so you still get the full test suite run on
can use to build visual treemaps for being the goal, and 30 minutes being too
a frequent basis. No matter what angle
metrics about your code, including code long, you quickly see to the importance of
you take, make sure that the tests that most
coverage and code complexity. This focusing on which tests work feasibly in
directly test the code are run frequently
information you can use to prioritize those time limits. Thousands of true unit
and quickly. A CI loop that takes 2 hours
your automation efforts. For example, tests will have no problem running in just
to run might as well be a daily build loop
by looking at the treemap diagrams for a couple minutes. Functional tests also
for all the value it adds!
the venerable CI tool CruiseControl you shouldn’t take too long, and can easily
can see the method getLabel() on the class have a full suite run in 20 minutes. But
PropertyFileLabelIncrementer is both if you start seeing your tests take longer Successful Test
complex, with a CCN rating of 10, and then that, say because you are doing web Automation is a People
has only 40% code coverage. browser based testing with tools such as Problem
Selenium or Watir, or running massive
You will quickly discover that there regression tests that generate thousands
are many types of automated tests that of test cases, then you need to embark Building up a great suite of automated
you will need to develop for full coverage on different techniques. Some ways of tests is a task that requires input from
of your application; from finely grained reducing the time that these test take is to many people across many areas. In any
unit tests that are tightly coupled to the start looking at writing tests that can be large software project that is embarking
structure of the code and are primarily run in parallel fashion across a farm of on integrating automated tests into their

September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 21


Successful test automation writers blend
the coding chops of a developer with the
analytical skills of a business analyst and the
rigorous attention to detail of a tester

Automated tests, CI loop, it’s important to seek input from a wide tests from the shoulders of the automated test writer!
variety of folks on what to test. For example, your Successful test automation writers blend the
run frequently
support folks may have the best sense of what areas coding chops of a developer with the analytical
under a CI system, of the application generate the most requests for skills of a business analyst and the rigorous attention
help with bugs. The developers may have a better to detail of a tester, which means this can be a
evolve with the sense of which modules suffer the most technical challenging role to be in! Companies are starting
debt. And your business folks will want to have to develop specific roles for this type of person. At
source code,
the team focus first on testing application areas Microsoft, they are called Software Development
ensuring that prioritized highly by the customer. This means a Engineer in Test (SDET) and are one of the three core
test automation engineer must be able to work with engineering positions, along with program manager
bugs are found, many different types of people, from developers, to and software engineer, and are on an equal footing
business analysts, to business stakeholders, and be as the others! At Google they are called Software
and tests never
comfortable in speaking in their language! Engineer in Test (SET) and looking at a recent job
go broken for Often one of the most challenging relationships posting for a SET show the responsibilities are
is that between the developers and the test automation challenging and exciting:
long!
engineers, because at times the work of one is
impacted by the work of the other. For example, • Work in tandem with other engineers to
if you are developing automated tests using a tool build advanced automated frameworks
like FitNesse, where you need to have custom code to deliver and exercise our world-class
developed to translate your tests into something applications.
that can interact with the Application Under Test • Conduct research on emerging technologies.
(AUT), then you will need to work with a developer
• Train and mentor other team members.
to either implement those hooks, or learn enough
about the code to implement those hooks yourself. • Develop innovative solutions for large
Other times, as you develop the automated tests, you scale automated testing strategies, quality
may find that you need more information about the assurance, data analysis, and monitoring of
interactions of the various modules, especially when our systems.
testing beneath the user interface. This is an area
where collaborating with a developer to come up As with all Google Software Engineers,
with the right test will be critical. responsibilities may also include aspects of project
management, release engineering, task delegation,
Building a solid relationship with the developers team coordination, and production monitoring.
means understanding their concerns about what the
tests are verifying, and how long they take when Test automation is one of the key techniques
the build is being run, but also means that you can embraced by QA teams when the volume of testing
educate them on what it takes to develop successful required to ensure a quality product has grown beyond
automated tests. Get them thinking at the design stage what individuals can perform. Automated tests, run
about how a function would be programmatically frequently under a CI system, evolve with the source
tested, not just at the unit level, but at the system code, ensuring that bugs are found, and tests never go
level as well. And, with a really good relationship broken for long! As the coverage of the automated
and understanding of the goals of the automated tests increases, the quality of the application goes
tests, you can also enlist the developers in updating up, and the value of the CI build goes up. Just like
the automated tests as part of their development PB&J, one without the other doesn’t make much
work when writing new code. This helps lift some sense and is not nearly as satisfying!
of the burden of maintaining the automated suite of I think I’ll have a PB&J for lunch today!

22 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


2nd Annual

ATI Automation
Honors
Celebrating Excellence in the
Discipline of Software Test
Automation

Nominations: April 1 - June 1


Voting: June 14 - September 17

www.atihonors.automatedtestinginstitute.com

September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 23


24 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010
By Linda G. Hayes

Development’s
Role in Software Test Automation

A
side from their obvious principles described in this article can be
responsibility for developing applied to almost any interface, they are The Software
the software to be tested, specifically directed to testing UIs.
developers also have a
crucial role to play in the automation Within the context of UI test automation, Development Team has
of testing. Simply put, they must design there are certain basics that apply to
and develop the software in a way that almost all tools. First, they rely on an several pieces that help
enables and supports automation. This inventory, map or repository of objects
article describes what is required to make that comprise the application. Next, to ultimately complete
automation possible as well as what is these objects are referred to and located
by names or attributes and accessed via
desired to make it efficient. But lest this the Software Test
all be perceived as work in service of their methods and properties. Without
these fundamentals in place, tools are
automation, it is important to realize that
relegated to using text or images to
Automation Puzzle
most of the factors that make automation
practical also result in an application that locate components, and keystrokes or
is easy for end users to learn and use and mouse events to interact with them;
for developers to support and maintain. these techniques are unreliable and
almost impossible to read or maintain.
Therefore, for purposes of this article,
To set context, it should be noted that
there are many types of applications
we will focus on the preferred object- Accessibility
oriented approach.
and levels of automation. Most of us
think of automating through a user With all that being said, the fundamental
interface (UI), but the fact is there are tenets of automated testability are Consistency
many other types of interfaces: Web accessibility and consistency. That
services, APIs, embedded systems and is, the application components must
databases all represent layers that can be accessible to the test automation
be automated. What is interesting is
Change
tool and consistently implemented.
that these interfaces are usually the Although necessary, accessibility and Management
easiest to automate, simply because they consistency alone may not be sufficient.
are either built using well-recognized If the application is large or complex
standards or because they must already enough, it may defy automation through
support automated interaction with other
Indoctrination
its sheer scope or rate of change. In that
applications or devices. Ironically, it is case, automated testability must shift
the user interface that usually represents from simple automation of execution to
the greatest challenge, and so while the automation of change management.

September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 25


Elements of Granted, it may require extra effort on Elements of
Accessibility the part of development to adopt naming Consistency
conventions and apply them consistently
The concept of accessibility applies to to all objects, but this investment will Consistency requires that developers
the application internals such as naming be returned in spades. Not only will adopt and adhere to coding conventions
conventions and class implementation. automated tests be readable and reliable, for the class library, error handling, and
Without accessibility, automation will be but both the application and the test code appearance of the user interface.
difficult or even unworkable. will be transferable and maintainable. The The class library for an application
most common – and often the simplest should be standard across all developers;
Naming conventions are crucial
– naming scheme is the same as the one developers should not be permitted to
because they provide the vocabulary that is
applied to elements in the database. This mix and match their favorite classes just
used to express an automated test. Names
lends an extra dimension of consistency because they are familiar with them. It
that are gibberish, such as “TXT_015”,
throughout all components and layers of also means there should not be different
make it hard if not impossible to read and classes for the same type of control or
the application.
understand the test script. Typically the multiple implementations of the same
development or design tool automatically Class implementation, which is one. The reason is that if the same classes
generates these types of names and the the way in which the class library is are used in the same way throughout the
developer is too lazy or pressed for time employed, also calls for accessibility. application then the automation engineers
to modify them. This in turn places a The reason is that test tools rely on the can develop reusable routines to handle
burden on the automation engineer to object class, methods and properties to each class.

The concept of accessibility applies


to the application internals such
as naming conventions and class
implementation. Without accessibility,
automation will be difficult or even
unworkable.

apply a label that is more useful, such as interact with the application components. For example, there are at least four
“Customer Name”. In a large application Object names should be unique but other different ways to implement a combo
with hundreds or even thousands of properties should not be, and methods box: 1) allows you to type in the selection
objects, this becomes a significant effort. should be exposed and available, not or select it, 2) allows selections only, no
private or hidden. Container objects input, 3) moves the selection list to the
But accessibility also demands that should expose their included objects. For item based on the letters being typed,
the same name be used for the same object example, a frame that contains a grid that moving around the list for each letter, or
across sessions, builds and releases. In contains a cell that contains a combo box 4) selects the items closest to the typed
other words, “TXT_015” cannot suddenly should permit direct access to the methods input. If any one of these can be used for
become “TXT_016” simply because and properties for each layer. Opaque any given object, then the test automation
another object is added to the window or objects that won’t share their contents code will be more complex. And, not
page. This lack of persistence will doom make automation tricky, and in some coincidentally, your end users will
an automation effort, because the test cases impossible. probably be confused.
script cannot reliably locate the same But accessibility alone is not enough. Error handling is another area
object from one run to the next. This risk The application must also be developed in where consistency is important. If each
is especially high when the names are a consistent manner across all components developer handles errors in their own
being generated dynamically. and developers. unique way, then the automation engineer

26 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


Development’s Role

next level of automated testability comes


in: change management.

Why conformance is Change


Management
sometimes good:
Applications are continually
expanding in scope and complexity
Without accessibility, automation will be at the same time as cycle times are
becoming shorter and the rate of change
difficult or even unworkable... Consistency is accelerating. In this new reality, it is not
enough to be able to develop automated
requires that developers adopt and adhere to tests, you must be able to keep them current
by maintaining existing tests or adding
coding conventions for the class library, error new ones. If it takes too long to update
your test library, automation introduces
handling, and appearance of the user interface. delay instead of adding efficiency.

This need for speed may require


direct support from development for
will be forced to develop multiple routines and persistent. Not only will this assist producing the object inventory, detecting
for responding to error conditions both end users in navigating the application, it changes and managing their impact. This
planned and unplanned. Conversely, a makes it easy and efficient to automatically enhanced level of testability, as detailed
consistent approach can be supported by verify context so the script can assure the further below, extends and strengthens
the partnership between development and
a reusable routine within the automation application is responding as expected.
automation. It also enables automation
framework, saving time and effort and Ideally this title is stored within the same
earlier in the life cycle and may entice
enabling robust execution of positive location or property within the display,
developers to integrate automation into
and negative tests. And clearly any error and it should be unique in the sense that it
their own unit, integration and build test
handling should be graceful and well- is not used by other displays, meaningful
processes
behaved in the sense that the application in the sense that it is descriptive enough to
does not terminate abruptly, leaving be recognizable, and persistent from one The object inventory, as described
the system and data in an unstable and build or release to the next. above, contains the testable components
unpredictable state. of the application. In most cases this is
Furthermore, a display should contain acquired by “spying” on the application
Consistency of appearance refers a predictable collection of objects. This using a test tool. Typically this process is
to the way in which the interface is means that if the objects change – for used to identify objects, diagnose issues
presented. For example, there may be example, fields or buttons appear or with names, methods or properties, and
a header area at the top, a footer at the disappear based on data contents or user construct the application inventory or
bottom, a navigation tree on the left, and responses – then that is a new and different map. And while this process may be easy
the work area in the center. Usually this display. A so-called page or window that enough to perform, it may be impractical
is achieved using a design template. This can completely change its contents and in a large application. For example, a
matters because it lends itself to organizing appearance depending on a wide range of system with hundreds or even thousands
automation along similar lines. If each of factors is not a display at all, it is a house of displays – not unheard of for enterprise
these areas is consistently presented, then of mirrors. An automated script has no level applications – may take longer to
automation scripts can reuse any content reliable means of predicting the contents inventory than there is time available.
that refers to common elements and invest or behavior of such a display and so must
new development only in the areas that are invest significant effort in trying to handle This technique becomes especially
unique to each functional area. Without whatever comes its way. problematic for subsequent releases.
a common layout or design, each screen, Without a well-managed change
window or page (“display” for short) – As you can see, consistency is a management process, automation
even though it contains common elements simple idea with far-reaching impact. engineers will not know which areas of
- must be individually implemented. Without it, automation will be thorny at the application have changed. Completely
best. But even if you are successful in reconstructing the object inventory by
But consistency must go even further. institutionalizing consistency throughout spying would take too long, which forces
For example, each display should have a the development process, automation still them to find the changes by executing
unique identifying title that is meaningful may not be practical. This is where the the tests and allowing them to fail
(Continued on page 11)

September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 27


T
by Nick Olivo
a typical waterfall development cycle, developers build a backend
In system that will become the foundation for an application. This can

consist of stored procedures, web services, or dlls; but the thing that these have in common is

that they typically don’t provide an easy way for testers to access them. This prevents testers

from being able to do their job until much later in the development cycle. However, with a
little creativity and access to Visual Studio, you can build your own tool that can learn about

the assemblies and web services that your developers have created, and help you test them

more effectively. This article will show you how to do this via .NET’s reflection capabilities.

P
eople typically use reflection Let’s start out with the basics of learning what
to learn about classes and methods are in a given assembly. To do this, we

4
assemblies. RedGate’s Reflector will:
tool is probably the best known
Build a small utility that will list an assembly’s
example of reflection out in the wild. Reflector
methods, and the parameters those methods
uses reflection to list all the classes, methods and
take.
parameters that live in a given assembly. However,
reflection can go a step further and also be used Write some code that builds input files for
each of those methods.
to automatically generate test cases for use in
white box testing. In this article, you’ll learn how Have our utility call the methods and pass
to use reflection against a .NET assembly to learn them the data stored in the input files.
about its methods, how to autogenerate test data,
and how to automatically generate baseline files
steps
We’ll also have the utility keep a list of all the
methods that the assembly contains, so we
for later comparison, so you can see whether or can keep a tally of all the methods that have
not the methods are behaving properly. been added during the course of the release.

28 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


Build A
Small Utility

Let’s begin with the basics of


identifying the methods that exist in
a given assembly. We’ll build a small
utility that will list an assembly’s
methods and the parameters those
methods take. The first step in
building this utility is creating a new
project in Visual Studio, and choosing
the Visual C# Windows Forms
Application Template. Name the
application MethodTester.

Add the following using


statements to your solution: Figure 1: Assembly Method Utility
• using System.Reflection;
1 namespace methodTester
• using System.Xml;
2 {
• using System.IO; 3 public partial class frmMain : Form
4 {
• using System.Net; 5 private string OutputDirectory = Directory.GetCurrentDirectory().
ToString() + "\\Output";
• using System.Collections; 6 private string InputDirectory = Directory.GetCurrentDirectory().
ToString() + "\\Input";
Rename Form1 to frmMain. 7
8 …
Now, add the global variables
shown in Table 1 to the form’s 9 }
declaration. Next, add the controls
listed in Table 2 to frmMain. Table 1: Global Variables
Control Type Name Caption/Text
ListBox lstMethods N/A
Button btnGetMethodList Get Methods
TextBox txtAssembly N/A
Label lblAssemblylocation Select an assembly
Button btnBrowse …
Button btnTestMethod Test Single Method
Button btnTestAll Test All
Button btnListParams List Parameters
OpenFileDialog openFileDialog1 N/A

Table 2: Utility Control List

September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 29


Write Code To Build Input
Files

Now, we’ll add the code in Table 3 to the Browse button control.
By testing your
1 private void btnBrowse_Click(object sender, EventArgs e)
backend methods
2 {
3 if (openFileDialog1.ShowDialog() == DialogResult.OK) early, you’ll be
4 txtAssembly.Text = openFileDialog1.FileName;
5 }
able By testing your
backend methods
Table 3: Browse Button Code
early, you’ll be

The next control we’ll add code to is the Get Methods button. Double-click able to get more
the Get Methods button, and add the code from Table 4.
testing done sooner
This code first clears the contents of the list box. Then it tries to load the in the release

1 private void btnGetMethodList_Click(object sender, EventArgs e)


cycle. Reflection
2 { also provides a
3 lstMethods.Items.Clear();
4 Assembly targetAssembly = Assembly.LoadFrom(txtAssembly.Text); relatively easy way
5 Type assemblyType = targetAssembly.GetType(); to learn more about
6 MethodInfo[] methodInfoArray = assemblyType.GetMethods();
7 foreach (MethodInfo method in methodInfoArray) the applications
8 { you’re testing,
9 lstMethods.Items.Add(method.Name);
10 } and provides a
11 lstMethods.Sorted = true;
way to keep up
12 }
with the methods
Table 4: Get Method Code & parameters that
desired assembly based on the path the user specified in the txtAssembly make up your back-
box. Line 5 determines what type of object you’re dealing with (a class, an
object, etc). Once you know what type you’re dealing with, you can call the end code.
GetMethods() method, and store a list of all the assembly’s methods in an array
called methodInfoArray. Then we loop over each item in the array and add it to
the lstMethods box. Finally, we sort the methods box.

Now that we know what methods live inside the assembly, the next step is to
have our code loop through each method and learn about its parameters. We’ll
create an xml file for each method, which will contain a list of the necessary
parameters, and we can enter our desired values for those parameters. In the
case of an overloaded method, there will be one xml file created per overload.
30 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010
1 private void GetParameters(MethodInfo method) So if you have a method called Hello
2 { which is overloaded three times, there
3 int counter = 0; will be three xml files created for that
4 string filename; method.
5 XmlDocument xmlDoc = new XmlDocument();
6 filename = InputDirectory + "\\" + method.Name + "Input.xml"; We’ll begin this step by checking
7 while (File.Exists(filename)) to see if an input file for our target
8 { method exists. If it does, we’ll append
9 counter++; a 1 to its name, and check if that exists.
10 filename = InputDirectory + "\\" + method.Name + "Input_" + counter.ToString() + ".xml"; (This is how we handle overloaded
11 } methods as mentioned above.) Once
12 XmlTextWriter xmlWriter = new XmlTextWriter(filename, System.Text.Encoding.UTF8); that’s done, we’ll create elements in
13 xmlWriter.Formatting = Formatting.Indented; our xml file for the methods’ name,
14 xmlWriter.WriteProcessingInstruction("xml", "version='1.0' encoding='UTF-8'"); its return type, and its parameters. If a
15 xmlWriter.WriteStartElement("Method"); method has no parameters, then we’ll
16 xmlWriter.Close(); create a text node that states this.
17 xmlDoc.Load(filename);
18 XmlNode root = xmlDoc.DocumentElement;
Once you’ve created the
19 XmlElement nameTag = xmlDoc.CreateElement("Name");
GetParameters method, you can add it
20 XmlText nameVal = xmlDoc.CreateTextNode(method.Name);
to the btnGetMethodList_Click event;
21 XmlElement returnType = xmlDoc.CreateElement("ReturnType");
place it right after line 9. The call will
22 returnType.InnerText = method.ReturnParameter.ToString();
look like this: GetParameters(method);
23 nameTag.AppendChild(nameVal);
24 root.AppendChild(nameTag);
Now, you’ll notice that our current
25 root.AppendChild(returnType);
implementation is just creating a
26
placeholder for our parameter values
27 ParameterInfo[] parameters = method.GetParameters();
(the node called ReplaceMe). While
28 // Point out if there are no parameters
29 if (parameters.GetLength(0) == 0)
it is possible to automatically create
30 {
input data, there’s a good chance
31 XmlElement data = xmlDoc.CreateElement("NoParams");
that the data generated wouldn’t be
32 XmlText noVal = xmlDoc.CreateTextNode("This method takes no parameters"); valid, or they might not be useful. For
33 data.AppendChild(noVal); example, let’s say you had a method
34 root.AppendChild(data); called createCustomer, which took the
35 } customer’s first, middle and last names
36 // Output details for each parameter as parameters. You could automatically
37 foreach (ParameterInfo param in parameters) generate strings for those values, but
38 { they’d be gibberish. I’ve found that
39 XmlElement par = xmlDoc.CreateElement("Parameter"); it’s better to manually edit the xml files
40 par.SetAttribute("Name", param.Name); once they’ve been created to ensure
41 par.SetAttribute("Type", param.ParameterType.ToString()); you’re testing with good data. You can
42 XmlElement val = xmlDoc.CreateElement("Value"); keep multiple copies of a given input
43 XmlText Val = xmlDoc.CreateTextNode("replaceMe"); file, and each copy can have its own
44 val.AppendChild(Val); unique input data.
45 par.AppendChild(val);
46 root.AppendChild(par); Once you have your input data,
47 } the next step is to be able to run a
48 xmlDoc.Save(filename); given method with the desired input.
49 } To do that, we’ll create a method
called executeMethod, which will
Table 5: Get Parameters Method Code
September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 31
1 /// <summary>
2 /// This method reads a parameter (as a string) from an input file and converts it to
3 /// the appropriate type. For example, if a parameter is type System.Int32, and its
4 /// string value is "23", then this function converts that string to an Int32. You
5 /// can keep tabs of all the parameter types used by the BLL by calling the listParameters()
6 /// method below.
7 /// </summary>
8 /// <param name="paramType">The type of parameter; e.g. System.Int32</param> with a little
9 /// <param name="paramValue">The parameter's value</param>
10 /// <returns></returns>
creativity and
11
12
public object convertParameter(string paramType, string paramValue)
{
access to Visual
13 string[] sArr; Studio, you can
14 ArrayList paramArr;
15 switch (paramType) build your own
16 {
tool that can learn
17 case "System.String":
18 return paramValue; about the assemblies
19 case "System.Guid":
20 return new Guid(paramValue); and web services
21 case "System.Int32":
22 return Convert.ToInt32(paramValue);
that your developers
23 case "System.Int16": have created, and
24 return Convert.ToInt16(paramValue);
25 case "System.Nullable`1[System.Int32]": help you test them
more effectively.
26 if (paramValue == "null")
27 return null;
28 else
This article will
29 return Convert.ToInt32(paramValue);
30 case "System.Nullable`1[System.Guid]": show you how
31 if (paramValue == "null")
32 return null; to do this via
33
34
else
return new Guid(paramValue);
.NET’s reflection
35 case "System.Nullable`1[System.Int64]": capabilities. People
36 if (paramValue == "null")
37 return null; typically use
38 else
reflection to learn
39 return Convert.ToInt64(paramValue);
40 case "System.Nullable`1[System.Double]": about classes and
41 if (paramValue == "null")
42 return null; assemblies.
43 else
44 return Convert.ToDouble(paramValue);
45 case "System.Nullable`1[System.Decimal]":
46 if (paramValue == "null")
47 return null;
48 else
49 return Convert.ToDecimal(paramValue);

Table 6: Convert Parameter Method Code (Part 1)


32 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010
Are You Contributing
Content Yet?
The Automated Testing Institute relies heavily on the automated testing
community in order to deliver up-to-date and relevant content. That’s why
we’ve made it even easier for you to contribute content directly to the ATI
Online Reference! Register and let your voice be heard today!

As a registered user you can submit content directly to the site, providing you with
content control and the ability to network with like minded individuals.
Community
Comments Box >> Community Comments Box - This comments box, available on the home page of
the site, provides an opportunity for users to post micro comments in real time.
>> Announcements & Blog Posts - If you have interesting tool announcements, or
you have a concept that you’d like to blog about, submit a post directly to the ATI
Announcements & Online Reference today. At ATI, you have a community of individuals that would love
Blog Posts to hear what you have to say. Your site profile will include a list of your submitted
articles.
>> Automation Events - Do you know about a cool automated testing meetup,
webinar or conference? Let the rest of us know about it by posting it on the ATI site.
Automation Add the date, time and venue so people will know where to go and when to be there.
Events
Learn more today at http//www.about.automatedtestinginstitute.com
open our xml file and call the target 1 case "System.Nullable`1[System.DateTime]":
method with the parameters provided. 2 if (paramValue == "null")
Since all our information is technically 3 return null;
stored as strings inside the xml files,
4 else
we’ll need another method called 5 return Convert.ToDateTime(paramValue);
convertParameter that will change the 6 case "System.DateTime":
stored parameters into their relevant
7 return Convert.ToDateTime(paramValue);
types. The convertParameter method
8 case "System.Boolean":
reads the type of parameter from the
9 return Convert.ToBoolean(paramValue);
xml node and then calls the relevant
Convert function. I’ve tried to cover 10 case "System.Guid[]":

as many of the basic parameter types 11 sArr = paramValue.Split(',');


as I could think of, but you may need 12 paramArr = new ArrayList();
to add additional clauses to the switch 13 Guid[] guids = new Guid[sArr.Length];
statement illustrated in Table 6 and 14 for (int counter = 0; counter < sArr.Length; counter++)
7 depending on the parameters your 15 {
methods take. 16 guids[counter] = new Guid(sArr[counter]);
17 }
18 return guids;
19 case "System.Int32[]":
Calling The
20 sArr = paramValue.Split(',');
Methods
21 paramArr = new ArrayList();
22 Int32[] ints = new Int32[sArr.Length];
23 for (int counter = 0; counter < sArr.Length; counter++)
Table 8 reveals the 24 {
executeMethod function. It’s going to 25 ints[counter] = Convert.ToInt32(sArr[counter]);
take our xml file, read what method 26 }
we’re testing and then pass in the 27 return ints;
parameters we’ve specified to that 28 case "System.String[]":
method. It then outputs its results to a 29 return paramValue.Split(',');
subfolder in the results directory. This 30 default:
subfolder will have the current date & 31 return "unable to determine type";
time appended to it, so you can easily 32 }
review what your methods returned 33 }
at any time during the course of your
release cycle. Table 7: Convert Parameter Method Code (Part 2)

Reflector uses reflection to list all the classes, methods and


parameters that live in a given assembly. However, reflection
can go a step further and also be used to automatically
generate test cases for use in white box testing.

34 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


1 private void executeMethod(string filename)
2 {
3 string runDate = DateTime.Now.ToString();
4 Directory.CreateDirectory(OutputDirectory + "\\" + runDate);
5 string ResultsDirectory = OutputDirectory + "\\" + runDate;
6 object returnCode;
7
8
object[] methodParams; //this array contains the parameters the method takes
ArrayList alpha = new ArrayList();//this arrayList is also used to hold the method's param-
In a typical
eters
9 XmlDocument xmlDoc = new XmlDocument(); waterfall

development cycle,
10 xmlDoc.Load(filename);
11 XmlNode root = xmlDoc.DocumentElement;
12 string mName = root.SelectSingleNode("Name").InnerText;
13 Assembly targetAssembly = Assembly.LoadFrom(txtAssembly.Text); developers build
14 Type assb = targetAssembly.GetType();
15 MethodInfo mi = assb.GetMethod(mName); a backend system
16 returnCode = mi.ReturnType;//cast the return object to the appropriate return code type, e.g.
Int32 or System.Data.Dataset that will become
17 if (root.SelectSingleNode("NoParams") != null)//if the function has no parameters, just
invoke it the foundation for
18 returnCode = mi.Invoke(targetAssembly, null);
19 else //otherwise, create an array of parameters an application.
20 {
21 XmlNodeList paramList = root.SelectNodes("Parameter"); This can consist of
22 foreach (XmlNode param in paramList)
23 { stored procedures,
24 string val = param.SelectSingleNode("Value").InnerText;
25 alpha.Add(convertParameter(param.Attributes["Type"].Value, val)); web services, or
26 }
27 methodParams = alpha.ToArray(); dlls ; but the thing
28 try
29 { that these have in
30 returnCode = mi.Invoke(targetAssembly, methodParams); //call the method with its
appropriate params; common is that
31 if (returnCode.ToString() == "System.Data.DataSet")
32 { they typically don’t
33 DataSet d = (DataSet)returnCode;
34 d.WriteXml(ResultsDirectory + "\\" + mi.Name + "Output.xml"); provide an easy way
35 }
36 else for testers to access
37 File.WriteAllText(ResultsDirectory + "\\" + mi.Name + "Output.txt", returnCode.
ToString());
them.
38 }
39 catch (Exception ex)
40 {
41 MessageBox.Show(ex.InnerException.ToString(), mi.Name);
42 }
43 }
44 }

Table 8: Exceute Method Code


September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 35
Now, add the code from Table 9 to 1 private void btnTestMethod_Click(object sender, EventArgs e)
your TestMethod button. This is what
2 {
will allow a user to select which input
file they want to test. 3 if (openFileDialog1.ShowDialog() == DialogResult.OK)

4 executeMethod(openFileDialog1.FileName);
What we’ve created so far is great
5 }
for executing one method at a time, but
chances are you’ll want to run all the
methods, one after another. To do that, Table 9: Browse Button Code
we’ll add the code from Table 10 for
the Test All button. It will loop over all
the xml files in the Input directory, and 1 private void btnTestAll_Click(object sender, EventArgs e)

call them one after another. 2 {

3 foreach (string inputFile in Directory.GetFiles(InputDirectory))


Once all your methods have
4 {
been run, their output is stored in the
5 if (inputFile.EndsWith(".xml"))
Output folder. The first time you run
the methods, you’ll need to manually 6 {

verify that they are returning the 7 executeMethod(inputFile.ToString());


expected values. Once that’s done, you 8 }
can copy those results to the Oracle
9 }
directory. Then you can compare the
10 }
files using a tool like WinMerge or
BeyondCompare.
Table 10: Test All Button Code
Keeping
A List of 1 private void btnGetMethodList_Click(object sender, EventArgs e)

Methods 2 {

3 StringBuilder mNames = new StringBuilder();

4 lstMethods.Items.Clear();

Now, there are a handful of 5 Assembly targetAssembly = Assembly.LoadFrom(txtAssembly.Text);

other things we can do to help make 6 Type assemblyType = targetAssembly.GetType();

this tool more useful. The first is to 7 MethodInfo[] methodInfoArray = assemblyType.GetMethods();

check and see if there have been any 8 foreach (MethodInfo method in methodInfoArray)

new methods created since we last 9 {

ran the tool. To do that, modify the 10 lstMethods.Items.Add(method.Name);

btnGetMethodList_Click method 11 GetParameters(method);

by adding the highlighted lines from 12 mNames.Append(method.Name + "\r\n");

Table 11. 13 }

14 lstMethods.Sorted = true;
This code writes out a list of all 15 string currentTime = DateTime.Now.ToString("yyyy-MM-dd hh-mm-ss");
the methods to a text file, and you can 16 File.WriteAllText(OutputDirectory + "\\methodList" + currentTime + ".txt", mNames.
ToString());
compare that list to a baseline list that
was placed in the Oracle directory, 17 }

again using a tool like Winmerge or


BeyondCompare. Table 11: Get Method Code Modifications

36 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


1 private void listParameters()
Another handy troubleshooting
feature is the routine illustrated in Table 12, 2 {

which checks all the parameter types that 3 ArrayList types = new ArrayList();
the various methods take. This will allow 4 foreach (string f in Directory.GetFiles(InputDirectory))
you to easily see if the convertParameter 5 {
routine requires any updates. 6 if (f.EndsWith(".xml"))
7 {
Once this method runs, it writes out a
8 XmlDocument xmlDoc = new XmlDocument();
list of the parameters that all your methods
9 xmlDoc.Load(f);
take. Add a call to this method in code for
10 XmlNode root = xmlDoc.DocumentElement;
the btnListParameters button as illustrated
11 XmlNodeList paramList = root.SelectNodes("Parameter");
in Table 13.
12 foreach (XmlNode p in paramList)
Now you have a tool that allows 13 {
you to call methods directly from .NET 14 string val = p.Attributes["Type"].Value;
assemblies and send any input you see fit. 15 if (!types.Contains(val))
The tool can then automatically compare 16 types.Add(val);
the methods’ return values to stored ones, 17 }
and provide you with a report on which
18 }
methods aren’t behaving as expected.
19 }
You can also use reflection against web
20 string parameterList = "";
services. The only catch here is that
21 foreach (string v in types)
the .NET framework doesn’t have a
mechanism to dynamically load web 22 parameterList += v + "\r\n";

services at runtime. So to reflect against a 23 string count = types.Count.ToString() + " Parameter Types\r\n";

web service, you’d need to add a reference 24 string output = count + parameterList;
to your webservice, and then change the 25 MessageBox.Show(output);
code accordingly. In the Table 14 example, 26 File.WriteAllText(OutputDirectory + "\\ParameterList.txt", output);
I’ve modified the btnGetMethodList_ 27 }
Click methods to retrieve methods from a
web service. In my VS project, I’ve added
a reference to a temperature conversion
Table 12: List Parameters Code
web service, which can be found at
http://www.w3schools.com/webservices/ 1 private void btnListParameters_Click(object sender, EventArgs e)

tempconvert.asmx. 2 {
3 listParameters();
Then recompile your app and you’ll 4 }
pick up a reference to any new methods
that have been added. The app will work
without any other changes. Table 13: List Parameters Button Code
By testing your backend methods
1 private void btnGetMethodList_Click(object sender, EventArgs e)
early, you’ll be able to get more testing
2 {
done sooner in the release cycle.
Reflection also provides a relatively easy 3 lstMethods.Items.Clear();

way to learn more about the applications 4 // Assembly targetAssembly = Assembly.LoadFrom(txtAssembly.Text);

you’re testing, and provides a way to keep 5 tempConvert.TempConvert targetAssembly = new tempConvert.TempConvert();
up with the methods & parameters that 6 Type assemblyType = targetAssembly.GetType();
make up your back-end code. You can 7 MethodInfo[] methodInfoArray = assemblyType.GetMethods();
learn more about reflection on Microsoft’s 8 foreach (MethodInfo method in methodInfoArray)
site, at http://msdn.microsoft.com/en-us/ 9 {
library/f7ykdhsy%28VS.71%29.aspx. 10 lstMethods.Items.Add(method.Name);
11 GetParameters(method);
Wait! There’s More! 12 }
13 lstMethods.Sorted = true;
Want more reflection? Find it at: 14 }

http://www.reflection.
automatedtestinginstitute.com Table 14: Get Method Code (Web Services)
September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 37
38 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010
September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 39
I ‘B’Log To U

Latest From the B


Automation blogs are one of the greatest sources
to-date test automation information, so the Autom
Testing Institute has decided to keep you up-to-d
some of the latest blog posts from around the we
interesting posts, and keep an eye out, because y
your post will be spotlighted.
Blog Name: Dorothy Graham Blog Blog Name: Test Obsessed
Post Date: May 27, 2010 Post Date: June 19, 2010
Post Title: Automated test should find bugs? No! Post Title: Why Test Automation Costs Too Much
Author: Dorothy Graham Author: Elisabeth Hendrickson

I have recently been having what seems Bottom line: the reason test automation
like the same discussion with a number of costs so much is that it’s done in a silo
far removed from the development effort.
different people. “Automated tests should
Buffered from the consequences of design
find bugs” or “find more bugs” is a very decisions that decrease testability, the
common misconception. Basically this developers continue to create software
says that finding bugs is a valid objective that’s nigh onto impossible to automate.
for automation. I don’t agree - I think this And isolated from the technical expertise
of how the software was constructed,
is generally a very poor objective for test
the test automation specialists are in a
automation. The reasons are to do with situation where they cannot help but
the nature of testing and of automation. be both inefficient and ineffective.

Read More at: Read More at:


http://dorothygraham.blogspot.com/2010/05/automated-tests-should- http://testobsessed.com/2010/07/19/why-test-automation-costs-too-
find-bugs-no.html much/

40 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


Blogosphere
s of up-
mated
date with
eb. Read below for some
you never know when

Blog Name: Autonomicon Blog Name: DevelopSense


Post Date: June 14, 2010 Post Date: June 28, 2010
Post Title: MS Office Automation Post Title: Is Unit Testing Automated?
Author: Nick Olivo Author: Michael Bolton

You see, MS Office apps (Word, None of the steps in the process (with the
Powerpoint Excel, Outlook) are not exception of the running of the checks)
record & playback friendly. I learned can be automated, since each step requires
this the hard way a long time ago, and a thinking person, rather than a machine,
I’m hoping to spare others the same
to seek information, to make decisions,
pain. You will not be able to create a
and to control the overall process.
reliable, robust set of MS Office tests
Parts of unit testing can be assisted by
via a record and playback tool. The only
way I’ve found to effectively work with automation, but the automation isn’t
Office apps is via scripting. The Office doing anything particularly on its own;
object model lets you programmatically it remains an extension of the person’s
access any bit of text, cell or slide. ability to execute and to observe.

Read More at: Read More at:


http://autonomicon.blogspot.com/2010/06/ms-office-automation. http://www.developsense.com/blog/2010/06/questions-from-listeners-
html 2-is-unit-testing-automated/

September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 41


Hot Topics in Automation

ATI Automation Honors


2nd Annual Awards Crowns Automation Royalty Again

The industry’s original set of • Best Open Source Unit


awards dedicated to software Automated Test Tool
test automation is again looking
to crown industry leading tools, • Best Open Source Functional
resources and practitioners with Automated Test Tool
automation’s top honor. The ATI
Automation Honors celebrate • Best Open Source Performance
excellence in the discipline of test Automated Test Tool
automation from the corporate
level down to the practitioner level. • Best Commercial Functional
This celebration specifically pays Automated Test Tool
tribute to those that have displayed
leadership in moving and keeping • Best Commercial Performance
test automation in its proper place Automated Test Tool
as a distinct IT discipline, those that
drive innovation within the field, • Best Automated Testing Book
and those that display excellence
in automation implementation, thus • Best Automated Testing Blog
playing a big role in the delivery of Site or Social Networking Site
a high quality product to customers
and/or production. • Best Automated Testing Forum
Site
By honoring such high value discipline in the IT industry; IT wins,
people, products and services, everyone because it will be able to effectively This year, the awards have also been
wins. Individuals win, because they are produce products on pace with a expanded to take into account the
encouraged to continue to innovate; rapidly changing world; the world wins, technologies supported by the tool.
organizations win, because they are because it will be filled with higher As the voting closes, look for Winners
encouraged to continue to invest in quality products that make the world go to be announced in a special video
effective test automation efforts that ‘round! presentation in October. In addition,

By honoring such high value people, products and


services, everyone wins.

will provide them with a positive ROI Several tools, organizations and the winners and nominees will be
in the long run; the discipline of test highlighted in a special November
individuals were nominated in the issue of the Automated Software
automation wins, because it will garner
higher respect and prestige as a distinct following categories: Testing Magazine.

42 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010


http://www.googleautomation.com
September 2010 www.automatedtestinginstitute.com Automated Software Testing Magazine 43
44 Automated Software Testing Magazine www.automatedtestinginstitute.com September 2010

S-ar putea să vă placă și