What Is

Test ng
India’s 1st Software Testing Magazine

April’08, Issue

Reduce number of testing scenarios with this insight on -

Cover Story
Pg
21
Risk based
Testing

Inside this issue

Are you TEAM ready?
Putting the “Test” back in “Test plan” Pg 57
Some things I have learned in Pg 44
Software Testing Pg
Morning Ideas Pg 04
Have I got a deal for you?
Finally Usability Testing
Pg 28
Pg 40
14
With little Preparation & Teamwork
you can pave the way for new Team Memers.
and more...
 Welcome to 
 is the first conference being organized by PureConferences in India.

We intend making
 an experience for all participants by creating a platform for professionals and experts from the testing
community of the Globe

 will revolve around ’Agility in Testing’

The theme will explore and expand on the need for testing teams to adapt to the changing demands of release and quality.

 will cover diverse topics. Keynotes, Tutorials, and Paper presentations by renowned speakers known nationally
and internationally will be one of the highlights of the conference.
Let’s mark the conference program in our calendar today itself.

 will offer opportunities to the partners to associate themselves with this endeavor and showcase their
products in the conference.

We are sure you want to participate and/or present a paper or tutorial.

Do visit us at http://www.test2008.in
 Issue1: Vol 1: April 2008 Contents

Cover Story Is bad testing

pg
eating your
21
Pr fits?
04 FROM THE DESK: Morning Ideas
by Sunil Gupta
Good design is key for a good and reliable product

08 THE PROCESS ROOM: Software Testing: ~ A field par excellence

by Sudhir & Anjan
Software testing as a field has brought in a high level of confidence in customers and users
and has thus far provided far more challenges to the new age engineers and academicians.

12 TEAM MANAGEMENT: Are You Ready?

by Michael Bolton
With a little preparation and teamwork, you can pave the way for a new team member.

21 COVER STORY: Is Bad Testing Eating Your Profits?

by Rex Black
Testing should focus on mitigating specific risks to the quality of the system. Sequence of test
execution should be driven by associated risk...

28 THAT TIME OF YEAR: Have I Got A Deal For You

by Matthew Heusser
The methods and extend of our testing are all choices; you could even say they are trade-offs.

33 LEARN HOW TO TEST: What a Tester Should Know, even After

Midnight
by Hans Schaefer
Testing The Normal Way is Not Enough.... Where you find defects, dig deeper!

40 LEARN HOW TO TEST: Finally Usability Testing?

by Erik van Veenendaal
Will it finally happen!

Other Stories
44 FROM DEV DESK: Some Things I’ve Learned in Software Testing
by Jonathan Kohl
50 CONTROVERSY CORNER: QA is More Challenging than Development
by Mrityunjay
53 Is scripted testing bad?
by Vipul Kocher
57 Putting the “Test” back in Test Plan
by Paul Carvalho

WhatIsTesting.com 1
 Tea testers look at
aspects of dried & brewed
tea leaves and the liquor

Likewise, PureTesting goes into various

aspects & nuances of soware testing

We build innovati ve, end-to-end solutions,

and manage critical testing processes,
and reduce total cost of producing quality soware

• Banking & Finacial Services • Datacom & Telecom

• Pharmaceuticals • Embedded Systems
• eLearning • EAI & ISVs

Test Consulting • Testing Services • Testing Products

India • USA • UK • NZ
www.puretesting.com
+91 (120) 4621010; info@puretesting.com

Global Software Test Consulting & Services company

 Editor’s Note

Release of any software is a moment of mixed emotions, pleasure and

pain, joy and anxiety -- joyousness of seeing the hard work resulting
in a release and the anxiety of the unfound bugs that could have been
found and fixed.
After a lot of effort, this magazine is also being released with similar
emotions. We take great pleasure in introducing Version 1.0 of WIT
magazine. We hope to have many more versions of this magazine, both
major and minor, in years to come and hopefully with minimum number
of patches. We also hope that the future versions of the magazine will
have more features, will be more robust, will have an expanded user
base and establish itself as a dominant player.
Instead of the usual Editorial about the contents of the magazine
which you anyway can find out from the table of content, we take this
opportunity to share with you the reasons for this magazine and who
are the people who have contributed to it.
Many of the technical magazines “grow-up” with time. The articles
become more “philosophical” and more abstract. Readers who are
regular subscribers too grow with the magazine but most of the new
comers start finding the magazine “too abstract” for their liking. WIT
magazine hopes to keep itself relevant to both old readers who grow up
with the magazine and new readers who are either new to the testing
profession or new to the magazine.
Our sincere thanks to the initial group of people who helped
brainstorm the concept of the magazine. They are – Bernard Homès,
Danny R. Faught, Johanna Rothman, Kamesh Pemmaraju, Matthew
Heusser, Michael Kelly, Rex Black, Scott Barber, Stefan Steurs. Without
them the magazine would not have become a living reality. All good
things about the magazine are because of this group, the authors and
other well wishers. All bugs are because of us.
For putting together this version of the WIT magezine, I acknowledge
the efforts of Ashwin Razdan for collaborating with the authors and
collecting all the articles; Parag Sapre for designing the elaborate
graphics; Anushree Tewari for providing editing support, and Satish
Thakur for giving it the final shape in which you see the WIT magezine.
Hope you find this version readable and useful and we hope to have a
better V 1.1.
~ Vipul

WhatIsTesting.com 3
 From Daily LifeIdeas
Morning

the

Morning
Desk

Ideas
Though a good design is key for a good and reliable
product, a robust test and evaluation of the product
only ensures that the product meets all-explicit and
implicit requirements and is ready for deployment and
usage.

O
ne morning when I was engrossed bags had torn almost at the same place.
in my newspaper, my daughter
Since it was not possible to repair the bag
created panic as her school
in ten minutes, thus she carried her stuff in
bag had given way at one of the
another bag but this episode left me thinking.
shoulder belts.
I realized that the bag was made to meet
I was annoyed at her letting
the functional requirements of the student
me know only when I was
such as adequate space to carry stuff,
completely torn while she would
compartments for lunch box, pencil box and
have seen this happening over
water bottle. The bag was also providing a
the last few days but my bigger
good aesthetics but lacked the robustness
reason of annoyance was due
in terms of handling the stress that comes
to the fact this was the 3rd or
on the bag with daily usage....
4th time the different school

4 WhatIsTesting.com
 by Sunil
by Gupta, Head of Testing Practice : Flextronics Software Systems
Composition and
organization of the test team
plays a very crucial role in
the success of the product.
Two things emerged when I extended my thought
process further.
One, product making whether it is a school
bag or a large and complex telecom product
requires good understanding and research in
terms of features, stress areas, usability, etc
without which the product would give numerous
problems to the user as the school bag gave to
my daughter.
Second, testing plays a very crucial role to
ensure that the product meets all requirements
including the ability of the product to handle
normal and abnormal stress conditions.
If we take the school bag example, it is very
important for the product manufacturer to
ensure that the various
stitches in the bag take up
the full bag load else there
are going to be numerous
dissatisfied customers like
me.
Though a good design is key
for a good and reliable product,
a robust test and evaluation
of the product only ensures that the product
meets all-explicit and implicit requirements and
is ready for deployment and usage.
It is seen across the IT and software industry
that the organizations give a lot of focus to hiring
and developing software development teams
but do not focus enough on developing testing
teams. The test team is put in place either very
late on the programs or the test teams are not
organized well enough to provide the vital value
expected from the Test team.
WhatIsTesting.com
For product organizations, the composition and
organization of the test team plays a very crucial
role in the success of the product. Enough time
and energy needs to be spent by senior test
folks to
• Visualize the product (software and
hardware)
• Look at the various field configurations
• Identify the tools required for testing
• Look at the FOUR views:
o User view
o Maintenance View
o Functional/Operational View
o Performance View
In order to achieve a good understanding of
the product before the exhaustive test cycle
commences, a few people from the test team
need to be dedicated to carry out the Test
Engineering tasks (as defined above). This set
of people should have the mandate to carry out
the verification and validation tasks.
In the school bag example, the User View would
be to look at all the ways in which the school
bag would be used by a user. There would be
several user scenarios such as:
• Student lifting the bag from one belt
• Student lifting the bag using the side belt
and loading on its back
• Student fully loading the bag with books but
weight may not be too much
• Student
partially
loading the
books but
the weight
may have
exceeded the
5
 Morning Ideas

Suddenly I realized that it was 9:00 hrs by the

watch and I had a 9:30 am meeting with the
Product testing and Quality Group to review the Post Release Defect
Density of the products that we shipped in the
deployment is a very last quarter and had only 30 minutes to reach
involved and rigorous task office. Thanks to god, for my home is only a 7
minutes drive from office.
recommended weight Bye for now……
Maintenance View would be to look at aspects
such as ease of cleaning and washing the bag;
ease of repairing the bag and how easy it is to
find the spares such as zippers, buckles, etc.
Functional or Operational view is another
important aspect of testing. Tester has to
consider all the functional aspects of the
product. In this example, aspects such as
following would be tested:
• Ease of keeping books in the bag
• Access to all the pockets of the bag
• Ease of closing and opening the bag
• Ease of lifting the bag on shoulder and on
the back
• Enough space to keep books, lunch box,
and water bottle, etc.
Last but not the least, the Performance view of
the product in terms of load handling, ability
to bear stress due to regular use through
load balancing is considered. In the example,
tester has to take following aspects into
consideration:
• Testing the bag for various load and stress
situations
• Testing the bag for full load capacity
• Testing the key joints/stitches under the
stress situations
• Identify the weak areas and strong areas of
the product
Product testing and deployment is a very
involved and rigorous task and involves the best
of people and the best of the practices.

6 WhatIsTesting.com
 Neilsoft is a specialist engineering services & solutions
company focused on helping our clients enhance their
product engineering efficiency.
Our services in the software product engineering domain
span the entire product lifecycle including development,
testing, and localization engineering.

T E S T I N G E X P E R T I S E
Glo
e ba
e rtis Compatibility l de
p Testing liv
ex e
n

ry
ai

In Te

mo
in on
m

st st
st ti
Do

al in
Te gra
g

de
la g
tio
te

l
In

n
rage
Sto S
Partnerships with ind

ec
uri
C AE
Performance

Functionality
ty
Testing

Testing
M
CA

PL
M

C AD
ustr
y le

s
M Tes

sse
in ed
ul ti

st at
ad

ti- ng

ce
g
Te tom
by
ers

te

ro
u
A

p
st

bu
Certification ro
Testing
ven&
Pro

testing@neilsoft.com | www.neilsoft.com/testing.htm

Pune Bangalore Canton Chicago Cambridge Dalian ) +91-20-2605 3003

Focus verticals: Construction Industrial Machinery Software Transportation Energy
 The Software Testing: ~ A field par excellence

Process

Software
Corner

Testing
~ a Field Par Excellence

Apart from giving fillip to the economy, Software test-

ing as a field has brought in a high level of confidence
in customers and users and has thus far provided far
more challenges to the new age engineers and acad-
emicians.
Introduction and ITeS segment known as Software Testing.
Renewed growth of the Software Industry has
Awareness on Software Quality has, over the last
helped testing achieve the position of being
couple of decades, become one of the defining
called the watchdog of quality. Apart from giving
features not only in Indian software sector but
fillip to the economy, Software testing as a field
also from a global perspective. Thanks to the
has brought in a high level of confidence in
dotcom crash which was an eye opener for
customers and users and has thus far provided
everyone in the field to look back and focus
far more challenges to the new age engineers
on enhancing the quality of the products that
and academicians. Looking at some of the
are already in the market for many years. The
astounding figures:
quality initiatives taken through the last couple
of decades, have now become all pervasive and 1. Bangalore has seen a 24% growth in its
thus given rise to a new avenue within the IT recruitment drive in software testing. It is

8 WhatIsTesting.com
 by Sudhir & Anjan, Product Testing & Documentation : Accelrys, ITPL,
Bangalore
Composition and organization
of the test team plays a very
crucial role in the success of
the product.
forecasted that India needs about 18000
test engineers in the year 2005-2006.
2. With India having more than 89 companies
at SEI CMM Level 5 assessment; 275 Indian
software and BPO companies acquired
quality certification; and more and more
IT-ITES companies have dedicated quality
departments responsible for developing and
deploying the quality policies and reviewing
them.
3. NASSCOM survey has indicated that the
road ahead for the Indian software and
services companies within the quality
arena was largely dictated and tuned to
the developments taking place in the global
marketplace.
4. The future would see organizations move
from quality assurance to business
assurance and focus on information security
complying with international legislations,
developing new global delivery capabilities
and enhancing quality processes in new
business areas.
This is testimony to the fact that software
testing is poised to grow at a phenomenal rate
in future. With all these happenings, no one can
deny the conclusion that software testing has
grown as an independent field within IT/ITeS
service segments. And it has given a clarion
call to everyone within the professional arena,
specifically academicians, to closely monitor,
analyze and push forward the new horizons of
the technology movement that software testing
as a field is going to provide. Looking at some
more interesting statistics on testing industry:
WhatIsTesting.com
Facts from Industry
1. $3.0 billion of $4.6 billion in outsourced
testing is sent offshore, throwing up
opportunities for companies in India that
thrive on low-cost knowledge workers. ~
Source Aztech Software - Bangalore
2. Testing could make up to 25-50 percent
of software budgets. Independent testing
is growing at 50-65 percent while the part
of work done offshore is growing at 35-40
percent. ~ Source - Partha Iyengar, vice-
president at industry researcher Gartner
3. Testing service team at Wipro has jumped
four-fold to 2,400 in two years. In the
nine months to December, revenue grew
90 percent to $64 million, three times
the industry average. ~Source - C.P.
Gangadharaiah, VP Testing services, Wipro
4. The global market for software testing is
around $13 billion. In India alone, the
demand for software testing professionals
is expected to touch 20,000 to 30,000
by December 2006. . ~Source – Internet
Search
Recent Trends
With this background, we can ask ourselves
- what conclusion we want to drive from it?
Why suddenly is everyone so statistical about
gathering all these figures? To take a close look
below we see from 50’s onwards till 80’s either
the products were not tested or if they were then
it was out of compulsion.
Quality as a term itself took the real shape
under the leadership of Borris Beizer who gave
the awareness to the industry and the world
about its very existence. The role of quality has
become a necessity than a need in the software
industry.
If we start looking into history, we will find
thousands of such cases where a small mistake
in the software has brought havoc to life and
money. The most recent one is the Columbia
Space Shuttle crash. All such incidents have
9
 Software Testing: ~ A field par excellence

espoused within the IT-geeks eagerness towards
“checking before delivering” approach. On the
contrary, there was time when products were
released without or with minimal testing done.
Of course, many a time, these are dictated by
how the market wants them. Time-to-market
was probably more important than quality
of the product. But, time has changed post
dotcom crash era and software quality has the
maximum thrust in the market today, be it
consumer products, electronics, manufacturing
or software. And quite rightly, software testing
is enjoying its share of this quality conscious
market.
Testing as a field has now been more
strategic and approach-centric
thus adding new dimensions as
well as many challenges to it.
Previously testing used to be an
afterthought activity in the post
release phase when the clients
call and say “we found something wrong in
your delivered software”. But with time, it is no
longer the case. Any small error today is a big
issue from a client perspective. With more and
more third party software testing companies
grooming overtime, the availability of a talented
resource pool, making timely deliveries and the
commitment to meet the quality expectations
of the customer has added many feathers to
India’s software testing market. Needless to
make a note that Software testing as a field
within IT/ITeS has now emerged as a strong
area of business solutions and a source of
major revenue generator and emerging more as
a serious business goal as well an intrinsic part
within the IT/ITeS organization culture.
human life such as Insurance, Healthcare,
Banking, Financial services, Pharmaceuticals,
logistics etc. What this means is that, software
developed in each of these Industry verticals
need testing – some rigorously and some which
can give adequate confidence in the product. At
the end, software testing too is spanning across
all these Industry verticals. When it comes to
domains, the field looks even more lucrative
by the amount of challenge each one of them
provide to the individuals involved in testing.
Therefore, as software testing started to support
the development activities in these domain
areas, it became increasingly necessary for the
test engineers to gain knowledge and specialize
within each of these domain areas. This is
another area where there is a huge potential
for test engineers to grow and make an impact.
This led to different teams being formed, some
heterogeneous and some homogeneous groups
to cater to testing software in these domains.
The requirements of the client keep on
growing and we as organizations need to meet
the demands of the end users to stay in this
business. This translates to the whole gamut
of technology issues starting from software,
hardware to third-party products. Software can
be legacy application running on a mainframe
or the latest technologies such as .NET, J2EE
etc... Technology as we know has advanced from
the days of mainframes to today’s applications
running on a PC based Linux systems. However,
at the same time, applications developed on the
legacy technologies have not gone away or have
not been replaced with new technology; they are
still being used in Banks, Insurance companies

Future Challenges
There is software in almost
every aspect of human life....
Domain and Technology Perspective
software developed... need
The growing demand from within the customer
groups as well the growing demand of the
testing -- some rigourously
organizations to meet and extend their client and some which can give
support has made software testing as a field adequate confident in the
nothing but a paramount knowledge pool.
There is software in almost every aspect of product.

10 WhatIsTesting.com
 by Sudhir & Anjan, Product Testing & Documentation : Accelrys, ITPL,
Bangalore

and many others. This even complicates the Conclusion

scenarios wherein the vendors have to support
Moving a little off track to give a comparative
both old applications as well as new applications.
analysis we see in the past Public Administration
This tells us how complex it can be for testing
has struggled a lot to give itself an individual
to handle such complex set-ups. Besides, the
identity and for a long time shared a part of its
applications run on varied platforms from
glory with either Political science or Sociology.
different vendors including: Windows, Linux,
But today it has gained a public reputation and
Macintosh, IRIX, AIX, unix & Mainframe. We
has become a part of everyone’s life.
are currently looking from a macro level. Since
with the growing numbers of different flavors of So does it hold true for software testing which has
an individual OS (e.g. Linux as such has Linux remained cocooned within Software Engineering
Workstation2.0/3/0, Advanced Server 2.0/3.0, or Software Development Life Cycle. But now
Enterprise server 2.0/3.0 etc.) the coverage from the field has not only gained reputation but also
testing point of view becomes immeasurable. has stood on its own. Software testing as a field
Does that not give a test engineer enough has come of age bearing the pain of “We can do
challenge to work on so many platforms? Of without you” to share the glory of “With you we
course yes, it’s not that developers only code are right through”. A field that is still growing,
for making an application run over different shining and awaiting to flourish still needs a lot
platforms but the test engineers too have their of planning & strategizing from all sections in
own way of reaching end to end. Apart from this the Industry and academicians. As we need to
we can also predict a good deal of exploration remember the following lines:
in areas like Client-Server, Web (2-tier, 3-tier)
“The Road to Excellence is a never ending
applications, Mainframe application and an
Road” ~H. James Harrington, The Improvement
endless list of such applications.
Process
What we discussed as far as technology is
concerned is only the tip of the iceberg. There
seems to be an endless list and mushrooming
application areas to be explored in the time to
come.

WhatIsTesting.com 11
Team Are You Ready?

Mana-
gment

Are
You
Ready?
With a little preparation and
teamwork, you can pave the way
for a new team member.
Use a checklist to make sure you have everything
ready before the new team member arrives.
● Compile all necessary documents into one binder
that can be given to any new arrivals.
● Be sure to provide some context about the company
and the team to help the newbie get acclimated.

12 WhatIsTesting.com
 by Michael Bolton, DevelopSense, a Toronto-based consultancy
and training firm.
I
f your company is typical, it is hiring
more and more temporary workers while
scrambling for every bit of efficiency and
value. Hiring a contractor to help with
temporary problems can be cost-effective, but
wasting his or her time undermines the purpose
of the exercise and costs hundreds of dollars
per day. Moreover, people simply don’t work as
well when they feel uninformed, frustrated, or
stymied. On the other hand, a contractor who
can hit the ground running will make better
relationships, will be more effective, and will
use your time and hers productively.
This article describes ways to prepare
for a new contractor; however, the tips here
could just as easily be applied to a temporary
employee, a new permanent employee, or in
some cases, a transfer into your department. I
don’t assume any particular job description or
title for the person, nor do I assume anything
about the person’s technical skill in any given
category. Neither should you make such
assumptions. A crackerjack developer may be
completely oblivious to network configuration
issues — especially in a new environment
— and a brilliant documentation writer might
be flummoxed by a cryptic voicemail system.
Your contractor’s time is best used to solve the
problem for which you’re hiring her, not her
own infrastructural or contractual issues.
Contracts and Administrivia
Before beginning work, your contractor will
typically sign a contract outlining the scope of
the work to be done, the deliverables expected,
and a schedule detailing when each element
of the work should be completed. The more
specific the contract, the less opportunity there
is for miscommunication and disagreement,
and the better your interests are protected.
Your contractor may have requirements that are
outside the scope of your company’s standard
contract, or she may have to provide certain
kinds of documents relating to work eligibility.
Make sure these kinds of issues are ironed out
before the contractor is slated to begin work.
Most contracts come with a non-disclosure
WhatIsTesting.com
agreement. Make sure this, too, is prepared
in advance, and that all agree on each of its
points before the contractor’s first day. If your
human resources department has policies and
procedures manual appropriate for your hire,
make a copy available to the contractor early
in the game. Human resources will typically
require various kinds of information from
your new hire. Be sure to ask in advance what
information they’ll need so that you can pass
on any questions to the contractor prior to
her arrival. Human resources or security will
typically supply ID badges, pass cards, access
codes, and keys to the office, restrooms, or
other secure facilities. These items should be
available and tested.
If the contractor is arriving from out of the
country, make sure that your human resources
department and the contractor have fulfilled all
requirements related to visas, work permits,
and the like. The accounting people will need
to deal with administrative and taxation issues.
Make their jobs easier by coordinating and
relaying the required information in advance,
both from your staff and from the contractor.
Issues related to eligibility, tax numbers, tax
forms, and withholding information should be
sorted out concurrently with the contract.
Restrooms with locks but
no keys are scourges upon
the land. When I started at
one small outfit, it was a
week before security could
produce a restroom key
for me. It wasn’t only a bother for me
to ask my new colleagues for a key—I
was interrupting their work, too.
At this company, the restrooms were
locked, but the door to the server room
was always open.
13
 Are You Ready?
Elizabeth, a European
contractor of my
acquaintance, was hired
to work in an American
company. She submitted
an invoice, covering salary
and expenses, which was
rejected on the basis of a couple
of questions from the accounting
department about her tax withholding
status. She dropped in to investigate
the problem by speaking with Mark,
the accountant who had sent her the
mail rejecting the invoice. “I don’t have
time,” said Mark. “Not even for just a
couple of questions?” asked Elizabeth.
“No,” Mark replied. “It would be a
different story if you were an employee
rather than a contractor. But,” he said
imperiously, “I have more important
strategic things to do.” This incident
occurred at a company where more
than half the staff was made up of
contractors.
Eventually the problem was sorted
out, but it meant inconvenience and
exasperation for Elizabeth and her
manager, and embarrassment for
Mark’s manager. On the other hand,
Elizabeth’s final report included a
subtle recommendation—which Mark’s
manager followed—to give Mark less
strategic work to do. Whether Mark is
currently doing strategic work or not,
he’s doing it somewhere else.
14 WhatIsTesting.com
There are few things that make a contractor
happier than being paid on time and with
a minimum of hassle. Ask your accounting
department what they’ll need from the contractor
to pay invoices promptly.
Ensure that expense and reimbursement policies
are clear from the get-go. If your company uses
a standard expense reconciliation form, make
sure that a few copies or an electronic template
are available to your contractor. Again, your
most friendly and helpful contact person in
accounting should be available to the contractor
to help sort out confusion or difficulties.
IT Issues
Notify the IT support department that a new hire
is on her way, and request the specific things
you’ll need and the date you’ll need them. Most
new arrivals will need at least one computer set
up, linked
to the network, loaded with tools, attached to
printers, and tested. If you’re hiring a developer
or tester, additional terminals or workstations
may be necessary. Don’t forget to tell the IT
department about this, and above all, don’t
assume that they’ll know what the new arrival
needs. Your new arrival will need access to
various areas of the network, so make sure
she has the appropriate set of access rights. If
your company’s security is very restrictive—for
example, limited
Internet access—notify your contractor
beforehand so that other arrangements can
be made for Net-based research tools and
resources. An analog line might be required
for some purposes. If specific tools—examples
include compilers, configuration management
programs, and testing utilities—are required,
have these loaded and tested on the contractor’s
machine in advance. Be sure to follow up,
verifying that the work has been done and
that resources, configuration settings, network
policies, and default passwords are documented.
Make sure that email accounts have been set
 We challenge you.

At GrapeCity, we challenge you

with assignments on the latest
.NET and Java technologies,
enabling you to emerge a winner
in this game called life.

Lines of Business

Business Solutions

GrapeCity is a software development multinational company Custom Applications

with over 650 employees across Asia and the United States. For Mobile Applications
more than 25 years, we have brought optimized software solutions
CRM
and services to enterprises around the world.
ERP
We rely on four key principles to help our clients achieve their goals:
Technical Services
thoroughly understanding our clients’ business objectives, providing
highly personalized experiences, maintaining a strong emphasis on Mobile Porting
quality, and adhering to the highest ethical standards. Quality Assurance

Technical Support

Send your resume to Component Development

jobs.india@grapecity.com
now! Technical Tools

GrapeCity India Pvt. Ltd.

A-15, Sector 62, Noida 201307 Phone: +91 120 247 0123 Fax: +91 120 247 0124
www.grapecity.com
 Are You Ready?
up and checked by sending a
message to the new arrival’s Note the people who are most likely to be
address, and then check with helpful in answering questions or fixing
the IT department to be sure
the message has been received problems, and provide their contact information.
in the new mailbox. In addition,
obtain a document from the IT
department on how to change network login
and email passwords.
Everyone needs a working telephone and
The new arrival can often
help people to notice
old bumps in the road. I
was brought into a test
organization to help
identify problems in testing
efficiency. There were two types of
network drops in the test lab—one
for the company’s internal network
and one for the test network. These
weren’t labeled, since everyone who
used the lab knew which was which. I
suggested that this shouldn’t be taken
for granted as one of the locals helped
me find the right connection, and I
used a bit of masking tape to label the
drops nearest to my workstation. The
next day, a fairly senior tester reported
a defect that was later found to be
spurious, at a fairly significant waste of
time and effort. The misunderstanding
happened because he had plugged the
test machine into the wrong network
connection. The day after that, labels
magically appeared on all of the
network drops.
16 WhatIsTesting.com
voicemail system. Have the telephone installed,
and a number ready for the contractor when she
arrives. Make sure that this number is listed in
the company directory — and remember that
directories can exist on the network, in the
voicemail system, and with the receptionist.
Despite everyone’s best intentions, a company’s
information infrastructure can be complex
and confusing to a new arrival. Have the IT
department provide the name of a specific
person—a guide or sponsor who is helpful and
knowledgeable about the company—to help the
contractor navigate through network access,
email configuration, and printer problems. In
addition, make sure that your contractor will be
able to find her default printer, email address,
network login name and password, and IP
address. Try to counsel colleagues against
sending email announcements saying things
like “This week’s build can be found in the usual
places.” Instead, make the details explicit.
Providing Context
The contractor has been hired to accomplish
some task or solve some problem. If people
within your company are having difficulty
sorting out an issue, imagine the difficulty that
a stranger will face!
Even people who have worked inside your
company and who know the general environment
need background information when they
arrive in a new position. This is yet another
motivating force for your company to develop
clear specifications, design documents, plans,
schedules, budgets, and so on. Start by writing a
brief description of what the company and your
division do, and prepare an organizational chart
or departmental roster. At the very least, identify
yourself and your direct superiors and reports,
along with the other people with whom you
expect the contractor will be working. As you’re
 by Michael Bolton, DevelopSense, a Toronto-based consultancy
and training firm.

preparing this information,

note the people who are most The processes and documents that
likely to be helpful in answering
questions or fixing problems,
and provide their contact
information, so as to minimize
contractor will be highly recyclable.
the amount of time that the
contractor must wait—unproductively—for
answers. Document the key business processes
and workflows associated with the new arrival’s
task at hand. If you provide answers early, there
will be less time spent wondering and asking
questions.
Don’t get hung up on tools or formatting: A
document composed using a text editor or
even a pen and paper is infinitely superior to a
document that doesn’t exist at all. When offered
a choice between a detailed Notepad file and an
empty Gantt chart, I’ll pick the former every time.
Simply make sure that the information exists,
and that it can be found and read easily. Make it
a mandate from the beginning, and check with
the new arrival often. If the new arrival feels
that she is being forced to do detective work,
encourage her to keep a log of what she needed
to know and what she found. Make sure that
one of her deliverables is a list of the things that
will be useful to her successor, and be sure to
budget time for her to prepare it.
City Guide
If your new arrival is coming in from out of town,
she’ll feel more welcome with a guide and some
idea of where important services can be found.
If your company is providing accommodations,
make sure that the contractor’s lodgings are
quiet, secure, comfortable, and reliable, and
that they’re ready for the contractor by the
end of the first day of work, at the latest. If
arranging accommodations is the contractor’s
responsibility, provide a list of facilities in a
variety of price ranges. While it’s nice to be close
to work, don’t strand the contractor near an
industrial park without food or services nearby.
Prepare a list of restaurants in the area that
cater to a variety of tastes and budgets.
Ask your department for recommendations,
you prepare for the arrival of one
and you’ll quickly have a list that is easily
maintained and printed.
One of the benefits of traveling, from the point
of view of many contractors, is using free time
At one company where I
was a full-timer, we hosted
a number of developers
during an outsourced
project. We created a list
of local restaurants and
tourist attractions by sending out
email to everyone in the department,
soliciting one suggestion from each
person. I compiled the suggestions
into a single document, removed the
duplicates, and posted the results to
an internal discussion board. A few of
the suggestions inspired still more.
Some might suggest that discussion
of restaurants on company time and
company bulletin boards is wasteful.
Our experience was that the one-
suggestion technique cost next to no
time and was a way of getting to know
people—and people who knew each
other better almost always worked
smarter and harder.
We also found that a handy list made it
easy to book company gatherings, to
provide directions, and so on.

WhatIsTesting.com 17
 Are You Ready?
to explore their surroundings. Survey your
staff for a number of interesting— and possibly
undiscovered— places to visit in the area. If
those places are somewhat off the beaten path,
ask the person making the recommendation to
provide a link to one of the several useful Web-
based mapping services. If everyone pitches in a
suggestion or two, compiling them into a helpful
guide shouldn’t be burdensome. When you’re
finished, you should have a complete package
of general information that can be passed to
any new employee or visitor on paper, via email,
or on an internal Web site. Ask the new arrival
frequently for feedback on the checklist, and
refine it accordingly.
On the Job
Make sure that you schedule some time on
the contractor’s first day for introductions
and a tour of the essentials. You should be
prepared to walk around for at least an hour
with the new arrival, introducing her to the
The kitchen or lunchroom
is not only the place
for refueling but also a
good place to meet and
converse with others.
Linger for a while and
encourage chatting, even
if it’s not directly related to the task
at hand. Spend a little petty cash
on group lunches, outside coffee
breaks, or visits to the pub with your
contractor and the rest of your staff.
If these informal approaches seem
of questionable value, consider that
productivity and effectiveness are
often based on personal relationships
and rapport—do what you can to foster
them.
18 WhatIsTesting.com
people she’ll need to know. An introduction
from you is likely to carry more significance
than one from a subordinate, so be prepared
to do this yourself. Even if you are routinely
busy, getting your employees and contractors
to work together is a tremendously worthwhile
investment. A contractor’s effectiveness often
depends upon being able to meet and connect
with other people. Most contractors meet their
new colleagues in business meetings, but more
informal circumstances can lead to friendlier
relationships. On the introductory tour, take
your time. Cover the important places and people,
but be relaxed and try to avoid a hard-and-fast
agenda. Spend a few moments introducing the
new contractor to each staff member. Explain
quite generally what the new arrival is there
to do, and describe your staff member’s role,
and how each may be able to help the other.
Don’t just concentrate on peers and colleagues.
Spend a few moments chatting with other useful
contacts with whom your new arrival will need to
interact—people in network services, reception,
the mail room, and accounts payable. As you’re
walking around, show the new person all of the
useful and necessary places around the office.
An early stop should be the restroom. If keys
or pass cards are required, make sure that the
contractor is given them upon her arrival. The IT
department should have set up a default printer
and informed you which one it is. Include this
on the tour, along with specialty printers and
other equipment that might be useful. Also
on the tour, include visits to the office supply
cupboard, the photocopier, the first aid kit, and
the lunchroom. For the duration of the contract,
the contractor is a member of your company
and your department, and thus should be
accorded at least the same kind of assistance
and respect given to your permanent employees.
If you are the contractor’s supervisor, you must
be prepared to go to bat for her, and the rest of
your organization should be prepared to take
the same approach.
No matter what the task, there’s nothing worse
than having to research and answer the same
questions over and over again. Keep a list of
 by Michael Bolton, DevelopSense, a Toronto-based consultancy
and training firm.

frequently asked questions, their answers, and complete, can be reused and refined, but rarely
where someone can go for more information. In will you have to start it again from scratch.
many cases the FAQ information can be posted You’ll need to review things periodically, but
on an internal Web site, captured in a database the benefits will be immediate: happier, better-
within workgroup software, or collected in a prepared, and—most importantly— more
piece of email. If you don’t have such things set productive workers.
up, consider making the collection of FAQs a
task for the contractor, making such a document
part of the contractor’s package of deliverables,
since your contractor will have plenty of insight
on what’s important for a new arrival to know.

Reuse and Recycle

All this might sound like a lot of work— and the
first time you do it, it will be.
Luckily, the processes and documents that you
prepare for the arrival of one contractor will be
highly recyclable; they can be used again and
again, regardless of whether the new arrival is
a temporary or full-time employee. Since you’ll
be consulting with other departments, you can
exchange ideas and plans, and improve the lot
of new arrivals companywide. The work, once

Checklist for a New Team * Policies and procedures manual

Member * Expense and reimbursement policy
Prepare for Arrival * Expense forms
* Signed contract * Telephone and voicemail instructions
* HR info from contractor * Telephone directory, with key contacts
* Keys, badges, and codes ready and highlighted
tested * Internet policy
* Visa, work permit, and eligibility * Password and login instructions
confirmed * Software resource manuals
* Resolution of tax issues * Names of key contact people in IT, HR,
* Telephone and voicemail set up Accounting, etc.
* Phone number listed in directory * Organizational chart
* Set up computer: * Division responsibilities
networked * Overview of company
all applications installed and tested * Key business processes and workflows
email account related to job
login * Guide to city, if applicable
passwords * List of frequently asked questions (with
answers)
printer connection
Documents to Provide

WhatIsTesting.com 19
CORPORATE OVERVIEW
Hexaware is a leading global provider of IT and BPO services. Over 40 of our clients are Fortune 100 companies. We
enable clients achieve competitive advantage by co-developing innovative IT/Process capabilities delivered through
flexible business models. We focus and have achieved leadership positions in domains such as HR, Banking and
Financial Services, Insurance, Leasing and Transportation. On the technology front we specialize in Business
Analytics, Enterprise Applications, Application Modernization/Management and Independent Testing. Founded in
1990, we are a US$ 153 Million company with over 4000 employees in 16 locations worldwide. We are currently ranked
as No. 11 Software Company in India.
TECHNICAL EXPERTISE
Our Key Offerings ‰ Technology – Legacy
Systems, Client Server, J2EE
BR / FS/SR Specification
Industry
Industry Specific
Specific
Business Requirement
Analysis
Gap Analysis
Functional Support
and .NET.
Solutions
Solutions ‰ Operating Systems –
Airlines Software Testing Existing Methodology Review
Existing Tools Evaluation
Multiple Virtual System
Process Consulting Process Recommendations
Manufacturing & High (MVS),
technology
System Integration Black Box - Functional Testing UNIX variants and Window
Banking & Securities White Box - Interface Testing
Testing (SIT) Grey Box - Regression Testing based.
Insurance

Performance
Load Testing
Stress/Volume Testing
AUTOMATION TOOLS
Technology
Technology Practices
Practices Testing Analysis and Interpretation EXPERTISE
Application Management
Test Management
‰ Win Runner, Load Runner
Enterprise Application Test Automation Functional Automation
Load Test Automation
and Test Director - Mercury
Integration
Interactive.
People Soft
UAT
UAT Preparation & Execution
Roll Out / Production Support ‰ Functional, Performance
Oracle User Training
Siebel
Support
and Test Management Tools
E-Business solutions Fail Over / Availability Multi-tier Availability - Rational Software.
Low Resource
Testing Endurance Tests ‰ Silk Test and Silk Performer
from Segue Software.

HITS(HEXAWARE INDEPENDENT TESTING SERVICES) OVERVIEW

¾ Team of 250+ with varied skill sets
¾ Domain focused, Automation driven E2E testing solutions
¾ Independent testing labs based in Chennai and Mumbai
¾ Showcases projects spanning a diverse spectrum of
¾ Domains – BFS, Insurance, ERP and Airlines
¾ Technologies - Mainframe, J2EE, Siebel, People soft and SAP
¾ Expertise in test automation tools like
Win runner, Quick Test Pro, Load Runner, Test Director, Rational suite.
¾ More than 35% have hands on experience in Mercury tools.
¾ Proven Test Methodologies from end-user perspective to meet the customers expectations under real-world
business scenario.
¾ Offshore – Onsite Delivery model providing significant cost savings
¾ More than 1000 Man years experience

Hexaware Technologies
Hexaware Towers, Bldg. No. 152, Millennium Business Park,
51/3, G.N Chetty Road, TTC Industrial Area,
TN S 3 "A" Bl k M h
 Cover by Rex Black, President, RBCS, Inc., Bulverde, TX

Story
Is bad testing
eating your

Pr fits?
Using Quality Risks to
Guide Testing Effort
Testing should focus on mitigating specific risks to the
quality of the system. Sequence of test execution should
be driven by associated risk...

S
ince the 1980s, when Bill Hetzel Categories of Quality Risks
and Boris Beizer published their
Computer software, hardware, and systems can
influential books on software testing,
fail in the most amazing and various ways. Some
we have known that testing should be
bugs are not functional problems, but fall into
risk-based. Testing should focus on mitigating
other quality risk categories. We’ll take a look at
specific risks to the quality of the system. The
some categories of quality risks to stimulate our
sequence of test execution and the total effort
thinking about what to test.
expended on any given test should be driven by
the level of risk associated with that test. In the
Functionality
1990s, people like Rick Craig, Paul Gerrard, and
I independently created some ways to achieve The most obvious quality risk category is that
systematic risk-based testing, often by adapting of functionality. There’s always a chance that
ideas from other forms of engineering, such as the system does not provide some function it
insurance and medicine. In this article and the should. The function can include a capability,
next one, we’ll talk about ways to perform risk feature, processing, input or output.
based testing. For example, if you’re testing a calculator

21 WhatIsTesting.com
WhatIsTesting.com 21
 Using Quality Risks to Guide Testing Effort

4
Unacceptable performance
load above 500
Transaction Processing Time

Unacceptable performance at
2 any load
Unacceptable performance
degeneration above time

1
Acceptable performance

Transaction Arrival Rate

1000

Figure 1: Performance bugs

program, the lack of an add capability would fall
into this category. So would a situation where
the add capability was implemented, but the “+”
key didn’t cause the function to be activated.
The add capability might be implemented and
accessible, but might work only on integers, not
real numbers. The add operation, when carried
out, might give the wrong result, as in 2+2=5.
Or there could be some strange side effect
where the operation is handled but the
result is not at all as expected, as in a
divide function where 2 divided by 2
returned one, but in Roman numeral
format.
Performance and Reliability
In Figure 1, you see a graphical representation
of three types of quality risks in the area of
performance. The shaded area at the bottom of
the figure represents the required performance,
which is one second or less transaction
processing time at a load of up to 1,000
transactions per minute.
Risks to system quality in the area of performance
include the possibility that the system responds
to input, produces output, or
processes data too slowly under
all levels of load. The straight
dashed line shows this possibility.
The system might perform fine up
to some level of load, but have an
unacceptable non-linearity in the
performance curve, as shown by the
steep curved dotted line. Finally, the system
might perform within specifications during an
initial test run, but subsequent tests—when the

22 WhatIsTesting.com
WhatIsTesting.com 22
 by Rex Black, President, RBCS, Inc., Bulverde, TX
Reliability problems exist
when the system fails to
function normally each and
every time.
system is not rebooted between tests—might
reveal unacceptable performance degradation,
as shown in the family of three gently curved
lines.
The risk of performance degradation—and
indeed, behavioral degradation of any sort—
over time brings us to the reliability category
of risks to system quality. Reliability problems
exist when the system fails to function normally
each and every time. Such a reliability bug can
exhibit itself as an intermittent functional
problem. Sometimes, such bugs
appear after a long period of runtime
or after running under heavy load.
I had such a problem with Windows
2000 while writing this article. Extended
uptime between reboots usually resulted in
characters in some applications taking on the
wrong font or the wrong font size.
You might have seen another type of reliability
bug, too, where the system functions normally
when it functions, but crashes or hangs
unpredictably. Such bugs can occur after some
long period of runtime or after running under
heavy load. Alternatively, such bugs can be
truly random. I recently saw such a bug when
a security patch caused reboots on a system
every third or fourth time the system booted.
Stress, Capacity, and Volume
Risks to system quality in the area of capacity
include seeing functionality, performance, or
reliability problems due to resource depletion.
For example, the performance of operating
systems often starts to degrade once the system
consumes more than 80% of the available hard
drive or memory space.
Risks to system quality in the area of volume
WhatIsTesting.com
include seeing functionality, performance,
or reliability problems due to the rate of
computational, data, and communication flows.
For example, the performance of database
management systems often start to degrade
when the database management system load
exceeds 80% of the rated transactions per
minute or the allowed number of simultaneous
connections.
Installation and Deinstallation
Various things can go wrong when installing
programs. These include situations where the
program won’t install or the installation process
causes damage to the system.
The latter kind of problem once was ubiquitous
on Windows systems, with the
appropriate nickname “DLL hell.”
Here, one program overwrote
libraries used by another program
during its installation process.
Deinstallation can also create
problems. Sometimes, deinstallation
does not completely remove files and
undo changes. Sometimes, deinstallation
causes damage to the system.
Various things can go wrong
during installing & deinstalling
programs.
Operations
We also have those risks associated with
recurring operating activities.
These include activities carried out at the end
of some operational period, such as end-of-
quarter or end-of-month closing out of accounts.
Problems can arise especially with failures to
archive inactive logical records in a safe and
recoverable manner, as well as archiving of
data that should remain active. In addition, the
23
 Using Quality Risks to Guide Testing Effort
The effects of changes, even
small, localized, isolated
changes, are not always
small, localized, or isolated.
timing of archiving and the calculation of when
a period has ended and a new period begun can
fail.
For systems with databases, there can be regular
requirements to compact or repair the databases.
In addition, for such systems, operators
often must back up data and configurations
regularly, along with verifying the restore
process. There’s a strong risk of side effects
like performance problems related to backup or
restore operations, the inaccessibility of some
or all system functionality during backup or
restore operations, and the failure of backup or
restore operations under full system capacity
conditions. Sometimes, operations activities
happen automatically, in the background. For
example, rollback logs for a database often
need to be purged automatically to avoid major
performance and reliability bugs.
Regression
By regression, I mean when a
previously-working feature, function,
capability, or behavior fails following a
change. The problem we have in system
testing is that software/hardware systems,
being digital, tend to be discrete rather than
continuous in their failure modes.
The effects of changes, even small, localized,
isolated changes, are not always small, localized,
or isolated. How many times have you heard a
statement like, “How could that have happened?
I only changed one line of code.”
A small change, even one line of code, can affect
the behavior of the rest of the system. A small
change can cause incompatible data to enter
shared data sets, including dynamic dataflows
and static databases, resulting in bugs that
24 WhatIsTesting.com
exhibit their effects a long way from their origin.
Side-effects of changes can impair or even break
cohabiting, interacting, or underlying software.
The adding of yet-another-feature can lead to
systemic performance and capacity issues.
Usability and User Interface
One special category of non-functional quality
risks relates to the usefulness and usability of
the system by the intended user or operator.
Specific usability and user interface bugs
can vary. In some cases, systems present
cumbersome interfaces that do not follow
normal or expected workflows, leading to
user frustration, confusion, and mistakes. In
other cases, functionality, while present in the
system, is practically inaccessible to the user
Data quality poses a major
category of quality risks for
many systems.
or operator. Systems can be inappropriately
difficult for the users to learn, leading
to abandonment, mistakes, and
inefficiency. Finally, users and
operators can find instructional,
help, and error messages that
are misleading, confusing, or
misspelled.
Data Quality
Many systems exist primarily to do interesting
things with data. Inputs are processed,
transformed, and stored. Databases are
queried, records linked through foreign keys,
and integrity constraints enforced. Outputs are
displayed, printed, and transmitted to other
system.
So, data quality poses a major category of
quality risks for many systems. The system
might corrupt or lose data. The system might
store bad or nonsense data in a database
without integrity constraints. Databases shared
across multiple systems can allow data which is
 by Rex Black, President, RBCS, Inc., Bulverde, TX
valid for one system to be accessed by another
system which does not know how to handle
that data, resulting in failures that are removed
in time and feature space from the source of the
problem.
Further complicating these situations, it is
sometimes difficult to restore the system to
working state after a badly-handled error
condition. In some cases, the system, in the
course of succumbing to the error, damages
configuration files or static data stores in a way
that’s hard to fix later. The data quality bug in the
expense reporting package I mentioned earlier
is an example of this kind of misbehavior.
Date and Time Handling
In addition to having to handle
common errors, many systems must
also handle dates. This has created
significant information-technology
problems, including the infamous
“Y2K bugs” that consumed huge
proportions of company and government
IT budgets in the late 1990s and contributed to
the depression in the high-technology sector in
the early 2000s. In addition to having to handle
once-in-a-hundred-lifetimes type of events like
a new millennium, systems must frequently
handle events like leap years.
Many systems must deal with expiration dates.
Licenses expire. Credit cards expire. Insurance
policies expire. After some period of time, the
right of a bank to disqualify a borrower based on
a derogatory like a bankruptcy can also expire.
Failure to handle such expiry events properly
can expose a company to serious financial and
legal risks.
Localization
Localization refers to the ability of a system to
support local customs, languages, norms, and
laws. There are two broad categories of quality
risks related to localization. One category
relates to the user interface, and the other to
operations.
WhatIsTesting.com
Of course, if the system does not support the
character sets used by the local language, we
have a localization problem. Those languages
that use the Roman alphabet, such as English,
German, and Spanish, have single-byte
character sets. Some languages that use other
alphabets, such as Russian and Hebrew, have
single-byte character sets, too.
Localization also has operational implications.
For example, time zones and time formats vary
based on locale. Does the time change in the
summer? (In the United States, this is called
“daylight savings times,” but it goes by different
names around the world.) If so, when does
adjusted time begin and end? Are
dates written “month-day-year” (as
in the United States) or in the more
logical “day-month-year” format (as
in much of the rest of the world)?
Configuration and
Compatibility
Quality risks related to
Localization may be in
the categories of GUI or
operations.
Another family of risks to system quality lives
in the areas of configuration and compatibility.
Different users configure both the hardware
and software of systems differently. A family
of systems may support various hardware
configurations. Often, the number of potential
combinations of configurations is huge and
testing inappropriate combinations can result
in risk of failure in the field.
Standards and Regulatory
Compliance
More risks to the quality of the system exist
in the areas of standards and regulations. A
system can work properly but be excluded from
the market by failing to meet standards, either
25
 Using Quality Risks to Guide Testing Effort

effectively (no one will buy it) or legally (the

government won’t let anyone buy it or sell it).
Security
With the growth of the Internet, awareness of
the risks associated with security has increased.
Security risks are legion. They include viruses
and worms, criminals breaking into servers,
vandals causing denial of service attacks,
bugging and intercepting e-mail and Internet
communications, and more.
Timing and Coordination
When you have communicating components, or
shared components, timing and coordination
between those components is a concern. For
example, with an e-commerce system, how long
should the system wait between events like
mouse-clicks and submitted screens? At what
point can it safely conclude that the customer
has abandoned the transaction?
As another example, consider the
automated teller machine. What
happens when the central computer
times out or the network goes down?
What if that happens in the midst of a
transaction? If we’re testing an inventory
system, what happens when two salespeople
try to sell the same item at the same time?
When you have
communicating components,
timing and coordination
between them is a concern.
Can You Think of Other Quality
Risks?
Yes, and so could I. But isn’t this list already
too long? What if you tried to cover all the
quality risks discussed so far? Could you test
them all? Would they fit into the budget and
schedule context of the project? Would you
end up wasting time and effort on unimportant
problems? Surely any attempt to test this whole
set of quality risks would be neither effective
nor efficient!
So, what shall we do? In the sequel
to this article, we will discuss some
ideas of how to trim the infinite
number of things you could test
into a finite list of risks you should
mitigate.
About the Author:

Documentation
For many people the quality of the documentation
significantly affects their experience of quality
when using the system. Documentation quality
problems include being technically incorrect, of
course, especially the examples. The user can
find the documentation insulting or offensive,
teeming with grammatical or spelling errors, or
afflicted with distracting cosmetic or formatting With a quarter-century of software and systems
problems. engineering experience, Rex Black is president
and principal consultant of RBCS, Inc., a leader
Documentation refers not only to hard copy, but in software, hardware, and systems testing.
also to electronic documentation. Help screens,
installation instructions, error messages, and
wizards are a form of documentation. This article is based on an excerpt from Rex
Black’s book, Pragmatic Software Testing,
published by Wiley

26 WhatIsTesting.com
 That Have I got a deal for you?
Time
of Year
Have I got
a DEAL
for you …?
A
s this is our first issue, we are in a unique
place—: This issue will set the ground
rules for every issue that follows. Future
issues will build on this one, so we have to lay
the right foundation. I would like to start that
foundation with a simple assertion:
The methods and extend of our testing are
all choices; you could even say they are
trade-offs.
Life is all about trades. We trade
a “work week” of our lives for a
pay check, then we trade that
pay check in for a home, for food,
and utilities or clothing. We trade
some of our freedom for a family.
At each stage of the game, we are
28 WhatIsTesting.com
giving something up – time, money, energy, or
effort, in exchange for something we value more.
If you don’t think it’s like that in the world of
software testing … think again.
Having an independent test group splits
management attention and costs money, but it
also decreases risk and provides an impartial
assessment of the quality of the software.
Using bug-tracking software costs money and
creates the risk of “information overload”, but
also ensures that every incident is logged and
tracked. With a complete list
of incidents, a change control
board (CCB) can determine
which defects should be fixed
in the next release, by priority,
but that invariably slows down
bug fixes and decreases the
personal touch.
 by Matthew Heusser, Socialtext, California, US
Documented test cases are
great, right? We couldn’t
live without documented test
cases, could we? Well, maybe.
Time spent documenting test
cases is time that could be spent testing. There
is also some evidence to suggest that following
the same test scripts again and again leads
to thinking about problems in the same way,
which could cause a miss. Many times, finding
a bug (or a repeatable error condition) is like
playing a game of twenty questions; how likely
are those scripted test cases to get the right
answer if they lack feedback?
Requirements-driven testing requires written
requirements, and “locks in” the feature set at
the beginning of the cycle, when we have the
least information about what the users actually
need and how the software will support them.
Reviews and Inspections of documented test
cases slow down the test process, again taking
time away from test execution. Exploratory
testing requires talent and practice; exploratory
testing in the hands of someone without
experience could just be a waste of time.
How can your team get “better?” Mentoring,
Coaching, and Skill Development are all very
hard to measure and hard to evaluate. Test
Process Improvement is easier to measure, but
also of more questionable value.
Not all trade-offs are equal; the trick is to
carefully weigh the pros and cons before making
a decision – to consciously realize what you are
trading away, what you are trading for, and if
the switch is worth it. The question is, what
do you value more? Which outcome is “better”
for your company, with your customers, selling
your product, at this point in time?
I’m harping on this point for a reason. Not
because it’s so blatantlyzing obvious (it is),
but because so few people are talking about
it. Instead of talking about trade-offs, they
are talking about “Best Practices”, which,
apparently, always solve all of your problems
without introducing any new problems. The
fact that this defies common sense doesn’t seem
WhatIsTesting.com
to enter into the equation, and that worries me
a bit.
Okay, more than a bit. It worries me a lot. It’s
downright irresponsible to suggest that there is
Not all trade-offs are equal;
the trick is to carefully weigh
the pros and cons before
making a decision....
one single way to do it – a single one-dimensional
scale for goodness. The real suggestion that the
“Best Practice” crowd is that you listen to them,
doing things the way they suggest, instead of
thinking for yourself. The trade-off is your brain
for their brain. Keep in mind, the guru never set
foot in your office and doesn’t even know what
product you make.
When pressed on this issue,
most gurus admit that, of
course, “best” doesn’t really
mean “best.” To them, “Best
Practice” is just a short way of
saying “A practice that is good
enough for enough people enough of the time
that it’s probably worth considering.” If that
were really the case, then there are a number
of alternatives; James Bach of Satisfice recently
listed several on his blog, which you can find at
www.satisfice.com:
• “Here’s a practice I find interesting”
• “Here’s what I would recommend for this
situation”
• “Here’s my favorite practice for dealing with
{x}”
• “Person {x} attributes Practice {Y} for his
success. Maybe you’d like to learn about
it”
Notice that all of those examples included
conditionals, and that best practices are free of
all conditionals. When someone consistently
29
 Have I got a deal for you?

7) Only through judgment and skill, exercised

At each stage of the game, cooperatively throughout the entire project,
we are giving something up... are we able to do the right things at the right
times to effectively test our products
in exchange for something
- http://www.context-driven-testing.com/
we value more.
The greek philosopher Socrates is credited with
uses the phrase “best practices” and never saying that the unexamined life is not worth
seems to use any conditionals, my radar goes living. We spend thousands of hours a year
up … then I go to protect my wallet. at work. Examining our work life and striving
Some gurus admit that they don’t know your to constantly improve would seem a natural
situation. They explain when a practice might extension of that, and it is what this magazine
fit and when it might not; they talk about the is all about.
pros and cons of the practice. Some experts and Of course, the choice is yours. Each issue will
consultants admit that all new practices involve cost a little bit of your money, and a little bit
a trade-off, and they are up-front about it. more of your time. Would you like to trade?
At WhatIsTesting, we want to follow that example.
Every issue of WhatIsTesting will contain some
ideas, strategies, techniques, practices, and
perhaps some philosophy. The ideas might fit
into your workplace, they might not, or perhaps About the Author:
you can use them later, but not now.
As I mentioned at the beginning of this essay,
this issue will set the ground rules for everything
that follows, so I want to start it out right. I
consider myself a member of the context-driven
school of testing, that has the following basic
principles:
1) The value of any practice depends on its
context.
2) There are good practices in context, but there
are no best practices.
3) People, working together, are the most
important part of any project’s context. Matt Heusser is a Member of the Technical
Staff at Socialtext, where he is a lead in the QA
4) Projects unfold over time in ways that are
group.
often not predictable.
You can read Matt by email at Matt.Heusser@
5) The product is a solution. If the problem isn’t
gmail.com <mailto:Matt.Heusser@gmail.com>
solved, the product doesn’t work.
or read his blog online at xndev.blogspot.com
6) Good software testing is a challenging <http://xndev.blogspot.com>
intellectual process.

30 WhatIsTesting.com
 Learn by Hans Schaefer, Cairman : Norwegian Software Testing Qualifications Board
How To
Test
What a
Tester
Should
Know, even
After Midnight
Testing The Normal Way is Not
Enough.
Systematic testing of software or systems can
be learned, just like any engineering discipline.
There are tester knowledge certification
schemes like the ISTQB certification, which is
an internationally recognized, independent and
non-profit organization, and there
are standards (ISTQB Glossary,
BS 7925, IEEE 829, 1008, 1012).
At least the books and most of the
standards have been around for a
long time, and many techniques are
widely accepted. This means testing
can actually be studied and then
executed in some systematic way.
This does not mean that the typical
testing methods described in the
WhatIsTesting.com
literature are widely practiced.
As test execution is generally one of the last
activities in a project, it is very often run under
severe budget and time pressure. This means
that the testers have a hidden or open motivation
to compromise on the thoroughness of their
job. For example, as detecting defects always
delays both testing and the delivery
or acceptance of the product, there
is a hidden pressure not to find
any defects, or at least not serious
ones. Testing is just “done”. This
job is then not very motivating,
investment in learning is scarce,
people try to find a different job,
and testing does not improve over
time.
33
 What a Tester Should Know, even After Midnight
Can we Do Testing in a Better
Way?
Glenford Myers, in 1979, defined testing
differently: The aim of Testing is to find defects.
He used this definition because it motivates
people. Having a negative focus, trying to break
the product, leads to finding more and more
serious defects than just checking whether the
product “works”.
Most people do as they are told. If they are told
to find as many defects as possible, they will try
Test in a better way:
- Clearly define the purpose of
testing.
- Continuously try to learn
more.
- A mindset that questions
everything is likely to result in
better testing.
- Report the bug with a view to
“sell” it to responsible person.
to do so. If they are told to get the job done (and
explicitly or inherently passing the message
that defects delay the progress), people will try
not to find defects, or they will overlook many.
Thus the first problem is to clearly define the
purpose of testing, and make the purpose
perfectly clear to the testers.
Second problem relates to change. Software
products are growing more and more complex.
The focus of the requirements is changing,
for example emphasizing more security,
interoperability and usability. This leads to
changes in the requirements on the testing job.
Thus, a tester should continuously try to learn
more.
The next problem is the mindset of testers.
As one of the purposes of testing is finding
34 WhatIsTesting.com
problems and defects, a mindset that questions
everything is likely to result in better testing.
Part of the tester’s task is to report incidents.
This is not easy because it requires describing
the problem in such a way that the other side
accepts it as important enough to do something
about it. It means not only collecting more
information about the problem but also to
think about how to “sell” the bug report to the
responsible person.
There is definitely more to it. A tester should
always be on the outlook for more challenges in
field of testing. This paper is only the beginning
of what you may look for.
The Purpose of Testing
There are a lot of possible goals for testing.
One main, but possibly boring, purpose is to
measure the quality of a product. Testing is
then considered just a usual measuring device.
Just usual measuring is not much fun, but
a necessary job, which must be done well.
However there are questions a tester should
ask, in order to measure optimally. The main
question is which quality characteristics are
most important to know, and how precisely the
measurement must be done.
Another definition of testing is trying to create
confidence in the product. Confidence, however,
is best built by trying to destroy it (and not
succeeding in doing so). It is like scientific work:
Someone proposes a new theory, and all the
educated specialists in the area try all they can
to show that it is wrong.
Defects clump together: they
are social!
Defects often have a common
cause!
Try to find common causes,
and then follow them!
Where you find defects, dig
deeper!
 by Hans Schaefer, Cairman : Norwegian Software Testing Qualifications Board
After trying this for some time, unsuccessfully,
the new theory is slowly adopted. This view
supports Myers’ definition of software testing:
Find defects! The approach is a pessimist
approach. The pessimist believes «this probably
does not work» and tries to show it. Every defect
found is then a success.
People function by motivation. The definition
of testing as actively searching for bugs is
motivating, and people find more bugs this way.
It works in two ways: One is by designing more
destructive or just more test cases. The other
one is by having a closer look at the results,
analyzing details a not-so-motivated tester
would overlook. In the latter case this may
mean to analyze results that are not directly
visible at the screen, but are deep inside some
A tester should try to find
defects!
Defects may appear at places
where you do not see them
easily, i.e. not on screen output!
files, databases, buffers or at other places in
the network.
Defects tend to cluster together. We should take
a deeper look in areas where we find defects.
The trouble is that this could be a self fulfilling
prophecy. We are likely to find more defects
because of our focus in that are.
One more definition of testing is «measuring
risk». This means that testing is a kind of risk
mitigation, part of risk management. Testers
should have an idea about product risk, as well
as risk management.
A very basic method for this is looking at the
usage profile, and at possible damage. A tester
should ask which kind of user would do what
and how often. How will different users use the
system? How will this vary over time? And it
is very important to take into account wrong
WhatIsTesting.com
input and misuse.
There are three kinds of tests: The good, the bad
and the ugly tests: Good means everything is
fine, all inputs are right. Bad means inputs are
wrong. The ugly tests is where all wrong things
happen simultaneously: Someone restarts the
server while you do your reservation and at the
same time sets the machine date wrong, …
Another way to look at risk is to asses the possible
damage of the failure. This may be difficult to
analyze but one can make a beginning by asking
“what is the worst thing that can happen if this
function, feature or request fails?”
Better tests forces developers to do better work,
inform management about risks, and lead to
lower cost of software creation. Testers bring bad
News, but that is their job. Nobody loves speed
checks on the motorway! But speed checks
make our roads safer, and we all benefit.
Continuous Learning
Continuous learning is required in nearly any
job. But for testers it is absolutely essential.
Testing is risk mitigation.
What determines risk?
What happens if some input is
wrong?
In most cases, testing is done somehow
systematically, using some black box approach.
In addition, test design may follow heuristics.
Any black box approach may leave important
areas un-covered. Any heuristic is incomplete,
as it is dependent on personal experience (or on
learnt experience from others. And white box
testing does not uncover errors of omission. It
all comes down to: If there is some aspect the
tester doesn’t know, it is not tested. Hence the
tester should know as much as possible. But
how?
35
 What a Tester Should Know, even After Midnight
A tester needs programming experience. There
are lots of programming bugs, even after unit
testing by programmers. The tester should
have an idea of problems posed by a particular
programming language. Most testers who have
tested software written in C or C++ know that
they should check for memory leaks or memory
overwriting. Those who used early versions of
MFC (Microsoft Foundation Classes) know that
the program would crash for any date greater
than a particular date “We need to check
what that date was and with which version
of MFC”
The tester needs design experience. Much design
is about contracts and communication amongst
various modules. If the tester has no idea about
architectures and their inherent problems, she
will have trouble planning integration tests.
The tester needs domain experience. System
testing is about implementing the right solution,
doing what the domain requires. Can you test
a railway interlocking system? (Eh, what does
interlocking mean, by the way...?). At least some
domain experience is a must to test the system
and communicate with different stakeholders.
The trouble is that this may not be enough.
Testing for today’s stakeholders may definitely
be not enough. There are totally new ways a
system may be used or interfaced with other
systems and a tester should try to anticipate at
least part of this! A tester should always try to
find new ways to look at the object under test,
new approaches, and new viewpoints.
And finally: testers should try to use the newest
approaches and technology. You have to learn
them. Read testing books, look for and learn
tools, study journals, participate in discussion
groups, special interest groups, discuss with
Learn more, about everything!
Programming, architecture,
new domains, users, tools,
anything!
36 WhatIsTesting.com
your colleagues, and go to testing conferences!
A Critical Mindset
Don’t believe anything!
As a tester, don’t assume anything. It
may be wrong! Designer, specifiers, and
programmers assume a lot. It may be
difficult to ask because you may look
stupid asking. Someone who could answer
may be far away or not easily available.
You wouldn’t even realize that there may be
another interpretation. Sometimes people
you ask also don’t know, or you get some
sarcastic answers.
Using the pessimist view, you may as well assume
that any assumption is probably wrong.
There are ways to overcome the trouble that you
may seem stupid. Learn how to deal with people,
learn how to interview, learn how to be self-
confident. Ask someone else. Read, review, sleep
over it, and try to find different interpretations.
You may need a lawyer’s mindset.
If you don’t get an answer, have it on your issues
Don’t assume! Ask!
If nobody else asks the right
question, you might do so!
Think about new possibilities,
unknown problems, and the
stuff you learn.
Think «out of the box»!
list. But don’t just assume anything! Don’t take
things for granted! And especially don’t believe
that the system is probably right
Defects
Nobody loves the messenger with bad News!
As a tester, most of what you report is bad news.
The bad News is the bugs, or issues to call them
in a neutral way. Textbooks generally handle this
area well. There is issue reporting, registration,
 by Hans Schaefer, Cairman : Norwegian Software Testing Qualifications Board
handling, removal, retesting, regression testing.
We know all this. But there is something extra
to it, and that is not there in the books:
1 - An issue is only interesting for a tester if
it is accepted as a defect and repaired.
2 - There are defects, which are the result of
running many test cases in a row in some
very special order.
The first problem is one of salesmanship and
discipline. As a tester, one has a sales job.
Nobody is interested in spending any money on
repairing defects. They will only be repaired if
they are important enough. Thus, as a tester you
have to report an issue in such a way that the
developer understands that it must be repaired.
The damage must look big, the probability of it
occurring must look big, and the issue must be
easy to repeat.
Thus the tester should not just write an issue
on the issues list. The tester should think: Are
there any other related scenarios, which would
make this problem worse? Are there any more
stakeholders interested in this? Is this really
the whole problem or is there more to it? It may
also mean to invent and run some more test
cases. Cem Kaner has presented some excellent
material on this cause which is available at
his website at http://www.kaner.com/pdfs/
BugAdvocacy.pdf
Finally, there are the human issues, about
diplomacy, politeness etc. A tester should
make sure not to hurt anyone personally when
reporting an issue.
For every issue (or bug),
research more about it!
Make sure you report it as a
risk, and as the whole risk!
Defect reporting is a sales job!
Be diplomatic when reporting
issues!
WhatIsTesting.com
The second problem is worse: Sometimes we
experience failures, and we cannot recreate
them. These issues are called intermittent
bugs. They are especially difficult if they
introduce system crashes. Upon restarting the
system, any corrupted data in the memory may
be deleted, destroying the evidence. In many
cases, intermittent bugs are the result of long-
term corruption of some resource or memory.
An example is memory leaks. Some function in
the program does not return unused memory
when finishing. But because there is a lot of
available memory, this can go on for a long time,
until the memory is depleted. Other resources
may also be depleted. As an example, the Mars
Explorer ceased working after 18 days due to
too many files accumulated. In many real time
embedded systems, the tasks are restarted at
certain intervals, in order to cancel out possible
corruption of resources.
The trouble is that ANY shared resource can
be corrupted. It comes down to checking the
not very easily visible outputs such as files,
memory pointers, buffers, databases, messages
to remote systems, registry and many other
resources. It could even be the result of a race
condition, which depends on the exact timing
of some parallel tasks. And intermittent bugs
normally require a whole sequence of test cases,
not just one input and output.
However, if intermittent bugs occur, it is a good
idea to be able to rerun the same sequence of
test cases, maybe even with the same timing,
and do more checking than before. James Bach
has a good guide to investigating such problems
available at http://blackbox.cs.fit.edu/blog/
james/archives/000197.html.
One final problem: You may be wrong yourself.
Humans err. Testers are humans. This means
you overlook problems; you misunderstand
outputs, and some of the problems you think
you have found are actually not problems.
Be self-critical: Sometimes it is your fault.
Mistrust your memory. It is restricted. This
means it is better to take notes, to log
what you did, what the system did, what
37
 What a Tester Should Know, even After Midnight
Analyze even intermittent
problems!
Log everything you do in
testing!
Log everything you see, and
look at more remote details!
Make it possible to rerun your
tests, with more logging and
analysis tools switched on!
is installed etc. You can
trust notes more than your memory.
The Late Night Tester’s Toolbox
How should a tester work? What should a tester
always keep in mind when working?
One main trouble is to test “everything”. This is
far too much. It can never be achieved. But the
tester should have some ideas about what is
tested and what not, or what is tested more or
less. That means test coverage.
There are three very basic concepts of coverage,
and they can be applied to any notation for
You may be wrong – don’t trust
yourself 100%!
Take notes of what you do!
example, a control flow diagram, a data flow
diagram, a state transition diagram, a call
graph, system architecture or a use case.
• Basic coverage is executing every box.
• The next level is testing every connection.
• This should be the minimum when testing. If
there is more time, the next level is combining
things, for example pairs of connections.
Next, a test should follow the usage profile. This
is difficult, especially in module and subsystem
testing. But as a tester, one should at least try
to get some idea about how the object under
38 WhatIsTesting.com
test will be used. If nothing can be said, the
test should be directed at any use, testing
robustness. This means especially test cases
for wrong inputs are interesting.
One technique is the basis of most black box
testing: Equivalence partitioning. It helps to save
A tester must be able to state
what coverage a test has
achieved!
test effort, and it can be applied to derive other
test techniques. As a tester, you should know it,
but also be aware that it has caveats: Black box
testing may leave out important aspects. You
should also be aware that combination testing
is interesting. Lee Copeland (Copeland 2004)
has published a good introduction.
Another big problem is test environment that
has not been validated. The test environment
should be prepared and tested early. Waiting for
the test environment to work can kill any testing
Follow the usage profile if
possible!
If not possible, test for
robustness against wrong
input.
effort. Also remember that the defect may not
be in the object under test, but in the test data
or the output analysis. Be self-critical!
And finally, there is test automation. Running
tests using automated tools helps regression
testing. But test automation is more than that:
Equivalence partitioning is a
good basic technique!
Remember combination testing!
 by Hans Schaefer, Cairman : Norwegian Software Testing Qualifications Board

Tools may read specifications and automatically
generate test cases. Tools may automatically
create or restore test environments. Tools may
be used to manage the testing effort and the
test material. Remember that automaton can
be applied to any part of the software testing
life cycle and not just test execution.
Test the test environment – well
before test execution!
Check you test data!
• ISEB: Information Systems Examinations
Board of British Computer Society.
http://www.bcs.org/BCS/Products/
Qualifications/ISEB/ has run a certification
scheme for software testers since 1999.
• ISTQB: www.istqb.org International Software
Testing Qualifications Board. Develops
and runs an international software tester
certification scheme.
• ISTQB Glossary: www.istqb.org/fileadmin/
media/glossary-current.pdf
• Kaner 99: C. Kaner, J. Falk, H. Q. Nguyen,
“Testing Computer Software (3rd ed.), John
Wiley, 1999.

Selected References • Kaner bugadvoc: A presentation about how

a tester should report issues. http://www.
• Bach 2005: James Bach. A blog note about
kaner.com/pdfs/BugAdvocacy.pdf
possible causes of intermittent bugs: http://
blackbox.cs.fit.edu/blog/james/ • Myers 79: Glenford Myers: The Art of
Software Testing, John Wiley, 1979.
• Beizer 95: Boris Beizer, Black Box Testing,
John Wiley, 1995 • Schaefer 2004; Hans Schaefer, “What
Software People should have known about
• Better Software Magazine, www.
Software Testing 10 years ago - What they
bettersoftware.com. www.stickyminds.com
definitely should know today. Why they still
Very practical!
don’t know it and don’t use it”, EuroSTAR
• BS7925: British Standard: www. 2004
testingstandards.co.uk/bs_7925-1.htm
• Copeland 2004: Lee Copeland, A Practitioner’s
Guide to Software Test Design, Artech
House, 2004.
• Crispin: Lisa Crispin, Tip House, Testing Automate testing tasks!
Extreme Programming, Addison-Wesley,
2002, also http://home.att.net/~lisa. Be aware that there is more
crispin/XPTesterBOR.htm to automation than using test
robots!
• Gilb 2003: “Testers Rights: What Test should
demand from others, and why?”., Keynote
at EuroSTAR 2003
• GTB: German Testing Board: www.german-
testing-board.info The German Testing
Board developed an earlier version of the
nowadays-existing ISTQB certification.
• IEEE Standards: See www.ieee.org

WhatIsTesting.com 39
Take Control Of Your Quality

Virtusa
4 uality Assurance Services

Enterprises know well that quality can make or break customer loyalty and brand reputation. However, the growing complexity,
constantly changing market needs, and time to market pressures make software quality an elusive challenge.

Virtusa QA lifecycle services allow you to take control of your QA processes, projects, people, and priorities, while strengthening
your QA capabilities, minimizing risk and reducing cost.

QA Assessment
QA Offerings
- QA Outsourcing Evaluation
- Tools Evaluation
- Quick Wins Workshops
QA
Ass ess men t
QA Process optimization

- Process Improvement Framework

QA Pro ces s - Health Check Assessment & Scorecard
Opt imiz atio n
Testing & Test Automation
Tes tin g & - Functional
Tes t Au tom ati on Automated Testing
Manual Testing
- Performance
- Certification
QA - Internationalization
Ou tso ur cin g - Usability

QA Outsourcing

- SQA Outsourcing and Management

- Test Team Establishment/Augmentation
- ‘Quick Win’ Test Project Consultancy

Mission-Critical Software Product QA Heritage

Leverage Virtusa’s eight years of experience in Offering Various domain & State Of The Art SQA Labs
Tools Expertise
testing and certifying hundreds of commercial
Shipping, 5%
products and mission-critical solutions to rapidly Retail, 5%
Financial, 30%
Publishing, 15%
improve QA effcienceis. Reduce cost and time to
market by leveraging Virtusa’s global delivery
Health
model, world-class QA professionals, and state- care, 15% Telecom, 15%
Insurance, 15%
of-the-art labs, tools and technologies.

About Virtusa

Virtusa is a global provider of custom IT solutions and outsourcing services to Global 2000 enterprises and software vendors in
the financial services, insurance, media and communications, high technology and other industries. Our deep domain strength,
custom solutions and services, and a unique platforming methodology enable our clients to reduce costs, improve business
performance and accelerate revenue generation. We deliver these results by integrating a consultative approach with our global
delivery model.
2000 West Park Drive, Westborough, MA 01581
Phone: 508 389 7300 Fax: 508 366 9901
© 2006 Virtusa Corporation. All rights reserved. www.virtusa.com Contact: info@virtusa.com
 Learn by Erik van Veenendaal, Improve Quality Services BV : The Netherlands
How To
Test
Finally
Usability
Testing?
will it finally happen?
A
new testing magazine that has chosen
web testing as its central theme for
the first issue! I assume that this also
means attention is provided to one of
the most critical success factors for websites:
usability and thus usability testing. In fact
usability has been identified in various surveys
as one of the main reasons for project success
in any IT project! For a number of years I’ve
been lecturing on this topic at test conferences.
At EuroSTAR, I have received the best tutorial
award for Discount Usability Testing and
our company, Improve Quality Services,
regularly runs courses on Usability Testing.
Yet, I don’t have the impression that all this
pioneering work has had much impact on the
testing community. During the last EuroSTAR
conference, an usability expert from South
Africa stated that “usability (testing) will only
start to be performed and receive the right level
of attention, when testing for
functionality is under control”.
An interesting thought, which
as far as I’m concerned, is very
much true. Perhaps we were
just not ready for it, but are
we ready now…?
It remains strange that when
no less than 40% of the
WhatIsTesting.com
software code that is being developed world-
wide is directly or indirectly related to the user-
interface, we still only dedicate a minimum
percentage of our test effort to usability
testing. Yet, there are many so-called best
practices available. Last week I was involved
in a usability test that very clearly showed that
thorough attention for usability throughout the
development process (requirements, standards,
expert reviews, usability testing) can deliver
great results. The users were enthusiastic
regarding the new system that supported their
tasks in an efficient, effective, and user-friendly
manner. This is also possible in real-life IT-
projects!
As far as I’m concerned every (senior) tester that
operates at system level should, in addition to the
conventional functional test design techniques
(boundary value analysis, equivalence
partitioning, decision tables etc.), also have the
level of knowledge and skills
that allows them to apply a
number of relatively easy to
use usability test techniques.
The first things that come
to my mind in this context
are the Heuristic Evaluation
technique developed by
Jacob Nielsen (www.useit.
41
 Finally Usability Testing? Will it finally happen?

com) and the SUMI questionnaire method (www.
improveqs.nl). Both techniques are relatively
easy to learn, do not require more than one or
two days of testing effort and therefore have a
great cost/benefit ratio. Recently, the British
Computer Society (BCS) Special Interest Group
in Software Testing developed a testing standard
in the area of usability testing that is certainly
worthwhile to get acquainted with (www.
testingstandards.co.uk). Sufficient possibilities
exist, therefore, to make a good, low-cost but
thorough start with usability testing; preferably
in co-operation with developers. Prevention is
always better than cure.
During a recent conference, a number of
enthusiastic former participants of my
usability courses told me they were now using
the techniques mentioned above in their test
projects with good results. Usability testing,
will it finally happen?
For queries or comments you can contact Erik
van Veenendaal (eve@improveqs.nl)
A paper on discount usability testing can
be downloaded from www.improveqs.nl

42 WhatIsTesting.com
 Canarys Automation Pvt. Ltd, # 135, 7th Main, 4th Block Jayanagar, Bangalore – 560011
Ph: +91-80-26539915, +91-80-26642922
marketing@ecanarys.com; info@ecanarys.com

Canarys is glad to inform that we are Partners to Radview Software, providing Sales & Support
Services to customers in India. Please find a brief on the Black Box Testing Solutions offered by
Radview:

Only RadView provides software-testing tools that feature JavaScript test-agenda creation for
performance testing, load testing, and functional testing of website applications. Throughout the
web application development lifecycle, the seamless integration between RadView's performance
testing / load testing products (WebLOAD and WebRM) and functional testing product (WebFT)
provides you with the necessary tools to make certain your web application is scalable and
robust.

Award-winning WebLOAD can help companies deploy high performing e-business applications
by modeling and anticipating the real-life demands on their applications. By accurately
simulating Internet user behavior and predicting capacity requirements, WebLOAD reports
bottlenecks, constraints, and weak links within an application – before they cost you downtime,
lost sales, or more importantly, lost customers.

WebFT is the industry's first Web-centric testing solution that efficiently verifies the functional
accuracy of Web applications and ensures applications perform as expected.

Speed implementation time — use of open standards scripting language reduces the amount of
staff training required
Improve quality — resulting from higher quality Web applications
Reduce test script development costs — leverage single transferable scripts for fast, easy reuse
of test scenarios between development teams
Speed Web application deployment — streamline the testing process through the use of single
transferable scripts.

For more information please contact:

Canarys Automation Pvt. Ltd, # 135, 7th Main, 4th Block Jayanagar, Bangalore – 560011
Ph: +91-80-26539915, +91-80-26642922
 Some Things I’ve Learned in Software Testing

Some Things
I’ve Learned in
Software
Testing
S
everal years ago, when I started out as that any error messages that appeared were
a software tester, my goals for learning due to mistakes I made using the software.
software testing were clear. I needed Then, a senior tester encouraged me to follow
to learn how a software company my instincts: he told me that if something
operated, learn about the software I was bothered me, I should log it as a potential
assigned to test, and learn how to test software. problem instead of letting it go by. As I tested,
After a while, I enjoyed finding and reporting I learned that when I had a nagging feeling
difficult-to-reproduce bugs; I felt pleased that that something was wrong, there usually was
we were finding and fixing issues that could a problem in the software. Sometimes it was
negatively affect customers. a usability issue, or an awkward spot in the
application that needed more investigation, but
How did I go from a nervous student to becoming
whatever the issue, it was almost always worth
a confident tester? At first, I read everything
taking the time to investigate further. Even
I could about testing, the product we were
now, it is tempting not to listen to that voice
working on, and the systems we used. When
inside that tells me something is wrong with
I couldn’t find anything to read,
the software, but every time I don’t pay
I asked coworkers to teach me
attention, I miss at least one bug that I
things. I also learned a lot through
regret later. Occasionally, my instinct is
mentoring from more senior
wrong, but more often than not, that
developers and testers.
sense of feeling uncomfortable with
In the end, I learned the most by the software helps me uncover
doing. (I still do.) When I first started important information.
out in testing, my view of what a bug looked
I also learn a lot from my
like was quite narrow. Like many users, I felt

44 WhatIsTesting.com
 by Jonathan Kohl, Software Consultant, Kohl Concepts Inc : Calgary, Canada
mistakes, especially when I try something, fail
and try something else. Do you learn from your
mistakes and from trying new techniques? Who
can you talk to in the organization that can help
you learn more about testing?
There is More Than One Way to
Test Software
It didn’t take long for my experience to show
me that much of popular software testing
information is really folklore. It was interesting
to read, but I soon discovered that when I tried to
apply popular testing wisdom (frequently called
“Quality Assurance”) on real projects,
my efforts failed. One practice involved
writing all the test cases first (pre-
scripting test cases) based on the
project documentation, and then
testing using these procedural test
scripts later. Trying this practice
caused me a lot of problems. I
found that testers would run out of time
to actually test because they were so
busy writing about what they were
going to test. As things changed over the life
of the project, we got bogged down maintaining
documentation. And, when we started testing,
these pre-scripted test cases really narrowed
our focus, often making us miss entire classes
of tests.
In the software start-up I was working for, it
was important for each member of the team
to provide value to the goals of the business.
A development manager then complained to
me that we spent so much time developing
and maintaining test cases that we didn’t do
nearly enough actual testing. Once I took that
to heart, the business was pleased with the
results I provided by using exploratory testing
to focus on what was important to the business
and to the customer. Previously, I had focused
on writing lots of documentation (test plans
and test cases) before testing. At first it felt a bit
wrong, like we weren’t following “the rules,” but
the development team and our customers were
happier with our results. That carried more
weight with me.
WhatIsTesting.com
I also read statements like: “Quality Assurance is
much more than software testing.” In this line of
thinking, it is common to suggest that QA should
be responsible for the development process, that
quality is only as good as the process it follows.
However, in my testing experience working for
small companies whose survival depended on
satisfying paying customers, the process took a
back seat to the product. We strove for product
excellence over process excellence. If a process
helped us deliver a product that met business
goals, we followed it. If a process got in the
way of those business goals, we scrapped it.
Everything we did needed to be aligned with
the goals of the business if we were to survive.
This key lesson shaped my thinking as
a software tester.
My desire to learn more about software
testing grew because I learned the most
and provided the best business results
from actually using and testing software.
I wanted to find more techniques
compatible with what I was learning about
software testing. So I reread the first book I’d
ever read on software testing: Testing Computer
Software by Cem Kaner, Jack Falk and Hung
Nguyen. Again it resonated with me, and I began
to read everything I could find that was written
by Cem Kaner. Through him, I discovered
James Bach and later Brian Marick. That was
when I found my niche. These were people who
thought about testing the way I did. My own
experiences and ideas were echoed in their
work. Their writing was true to the point and
backed by experience, but most importantly,
when I tried what they recommended, it worked.
These three testing thinkers, along with Bret
Pettichord, went on to found the Context-Driven
Testing School1.
The Context-Driven Testing School has identified
different schools of thought in software testing.
Bret Pettichord has outlined four of them in his
Four Schools of Software Testing2 presentation:
• Analytical School
• Factory School
45
 Some Things I’ve Learned in Software Testing
• Quality School
• Context-Driven School
The Analytical School draws heavily on
mathematics, uses a lot of tools and techniques
for code coverage and uses metrics to try to
measure conformance to a specification. This
school proposes that software reliability can
be calculated. Some testers in this school
use complex mathematical models for quality
prediction.
The Factory School generally expresses that that
testing needs to be predictable, repeatable
and software is validated from written
requirements. This school of
thought is often interpreted so
that testing becomes a clerical
task. (Note: many Agile and Lean
testing ideas are variations of
this school of thought. This
shouldn’t be surprising. Many of
the ideas in this movement come
straight from manufacturing where
automation, repeatable processes and
eliminating waste are important factors.)
The Quality School encourages conformance
to a particular process. Often, the tester’s
job is to police the process, and make sure that
everyone is following it. The belief is that if we
follow the process properly, our product quality
goals will succeed as well.
The Context-Driven School believes testing is a
skilled activity, and the usefulness of a practice
is dependant on the context it is used in. Not all
companies developing software are the same,
and it is irresponsible to try to standardize
processes for all software development. Different
companies have very different needs and those
needs are far too diverse to standardize to one
true way of testing. Testers in this school use
whatever testing tools or techniques are the
most effective to help them reach their testing
goals.
Since Four Schools of Software Testing was
published, Cem Kaner has identified Test-
Driven Development as another school of
46 WhatIsTesting.com
testing thought3. Bret Pettichord identifies this
as the “Agile” school of thought. There may be
other schools as well. What is important as a
tester is to understand there are different ways
of thinking about and communicating testing
ideas, and recognize which ones you identify
with. That will help you get a focus on what
to learn, and will help you understand where
others are coming from.
Some people find the identification of schools
of thought in testing objectionable, and this
work by the contextualists is somewhat
controversial. When I first discovered it I
was reminded of different schools of
thought in other disciplines that I
studied in university. For example,
in philosophy classes, when
I learned of another school of
thought in logic, I wanted to learn as
much as I could to see if it altered
my thinking. If I found ideas in
a particular school of thought that
resonated with me, I added them to my
own repertoire. The schools of thought in
testing idea shouldn’t be something that
bothers us as testers; instead, it can be used
as a model for learning. If I get stuck studying
the same old techniques and processes, I can
use this model to identify other schools of
thought that I am less familiar with, and begin
researching them.
As a tester, what works for you, in a given
situation? Have some techniques worked well
on some projects, but not so well on others? Is
there a particular school of thought that you
haven’t studied before that might help you on
what you are testing right now?
Software Testing is
Multidisciplinary
I once worked on an article on tester roles
with my colleague Michael Kelly, who is also
a software tester. It became clear to us how
many different disciplines software testers
draw from. Different testers described the
following disciplines as influencing their work:
 by Jonathan Kohl, Software Consultant, Kohl Concepts Inc : Calgary, Canada
general systems thinking, history, software
development, construction, law, psychology,
family therapy, adult education theory, writing,
writing and performing music, experimental
science, among others. Older, more established
disciplines have often already dealt the same
kinds of problems we are trying to deal with
in software development. Applying successful
principles from other disciplines can be a
rewarding source of learning.
I often explain software testing ideas by
using an intersection of ideas from different
fields. Three that I think of frequently are
philosophy, business and technology. Testing,
like philosophy, involves asking questions. Not
only do I question the assumptions made in the
product I’m testing, in our development efforts,
and in the process we adhere to, but I constantly
challenge my own assumptions about testing.
Studying philosophy and practicing critical
thinking skills especially help me when I’m
generating test ideas.
Business considerations help me focus the work
I’m doing. Am I adding value on a team? Is
what I’m doing helping or hindering the
goals of the organization? Is my testing
focusing on what is important to the
goals of the company, or am I blindly
following a process? Understanding
how businesses work, how they
market themselves, and how they stay
functioning is important to testing.
Technology and testing are related because we
use tools, and interact with designers, developers
and users in order to test. We rely on computer
technology not only in software development,
but in most businesses situations. As testers,
we must have a certain level of skill to be able
to use technology. Since technology moves so
quickly, it is difficult to keep up and it seems
like there are infinite technological areas to
explore as testers. We might specialize in testing
a particular technology, or we may depend on
certain tools for our testing.
Philosophy, business and technology are areas
WhatIsTesting.com
that influence me, and provide enormous
possibilities for learning. My background in
music and sports, and my education also
influences the way I approach learning and
testing. What are your influences in life, and
how much do you draw on these other influences
when testing software? What other disciplines
remind you of your work in software testing?
Testers Can Learn From Other
Roles
I’ve worked in roles other than software tester,
and each of these taught me something
important about software testing.
Working as a programmer, I learned how to
look at applications differently. Understanding
the application from a source code perspective
helped me develop a catalogue of potential areas
in a particular programming language that are
more susceptible to bugs. It also provides a
better understanding of where to find useful
information for reporting a bug. For example,
web applications usually have a server log
that may have debugging information useful
for reporting problems. As a tester
wearing a developer hat, I was the
recipient of bug reports, rather
than a generator. It helped me
understand how a poor bug report
can frustrate developers.
In a technical writing role, I learned about
design and how design (UI or architectural)
can affect the quality of a product. I noticed a
correlation between complicated designs and
the amount of bugs found. I learned that if I
had trouble writing user documentation for a
particular feature, it would likely be a source
of bugs. Writing down requirements and design
ideas from the point of view of a user exposed
complexity.
Working in sales taught me about the big
picture, about how a software tester can serve
the business goals of an organization. A company
does not necessarily survive because they are
such-and-such CMM level, or follow the 12
practices of Extreme Programming, or because
47
 Some Things I’ve Learned in Software Testing
they use .NET or Java, or other technical or
procedural issues we find so important in
development. When I worked in sales, the harsh
reality was that a company survives by fulfilling
an important equation:
Revenues – Expenses = Profit
This equation must be greater than zero in order
for the company to survive.
I learned customers cared that the product
helped them solve a particular problem. They
didn’t care what technology was used to solve
it. They cared that the product was easy-to-use,
that it helped them in their work, and that it
was affordable. It was interesting to see that
they would put up with application failures,
but that they were not as tolerant of usability
issues. When you deal with customers, you get
a different view of the product. Testers serve
customers, and it is in our best interests to
be sure that they are satisfied and impressed
with our product. I learned that it is much
more important to be testing what is important
to the customer rather than trying to fulfill
some generally-accepted testing practice. It is
absolutely paramount to do software testing
from a user’s perspective. Testers need to
understand the company, what its goals are,
and to figure out how testing work maps to
those goals.
Understanding how an application works from a
technical perspective, and being able to describe
that in a document helps me test more effectively.
It helps me ask questions that people I work
with find useful. Working directly with potential
and existing customers has helped me focus on
their needs in my testing. Understanding the
goals of the business helps me prioritize my
mission in testing on a particular task.
What other jobs have you had other than in
software development and testing? What have
you learned on those jobs that you could apply
to testing? Are there specific skills you developed
in other jobs that give you an advantage as a
tester?
48 WhatIsTesting.com
Practicing Software Testing is
Important
We can also learn by practicing software
testing skills. For example, I regularly practice
my testing skills, and ask my testing friends
to help me develop my skills. One of these
friends is James Bach, who taught me how to
be more consistent with my testing approach.
He found that when he gave me a number of
practice testing problems, my test approach
was inconsistent. James helped me develop
my testing ideas into a consistent system when
solving different kinds of testing problems. Now
when I am faced with a testing problem, I have
a strategy and a game-plan instead of an ad-
hoc approach. Practicing testing with someone
else helped identify an area of testing weakness
for me to address and improve.
I’ve also worked with programmers and
asked for feedback on how to provide better
information when logging bugs. Early in my
career when I was first testing web applications,
I wasn’t investigating beyond the error I saw on
the screen. One of my programmer colleagues
taught me how to find more information in log
files. I practiced developing my web testing
investigation skills by:
• Finding relevant server logs on different web
application architectures
• Reading error messages in the server logs
• Changing the level of logging to get more
information on the server
• Saving relevant information and talking
about possible causes with administrators
and developers
Because I practiced, when I next tested the web
application, gathering relevant information from
server logs to use for a bug report was second
nature. It took much less time to do after I had
practiced those skills.
This is just one very simple example, but it
helped my testing effectiveness. There are many
skills we can develop, and many techniques
we can practice. What specific skills you rely
 by Jonathan Kohl, Software Consultant, Kohl Concepts Inc : Calgary, Canada
on as a tester? Can you name them? Can you
identify technical skills, investigation and
problem-solving skills and softer skills related
to communication, and working with others?
Conclusion
At the end of each section of this article, I have
posed questions for you, the reader. If you would
like to learn more about software testing, I urge
you to answer these questions as best you can.
Take out a pencil and paper, and write down
your answers to the questions. Read them over,
and see if there are any new areas for study for
you to pursue in testing. You should have some
new ideas about software testing you’d like to
learn more about. Do you want to learn more
about testing techniques? How about different
schools of thought or definitions on testing? Are
you interested in developing your technical or
programming skills? Would you like to learn to
use a test automation tool? How about learning
more about the end users of your software?
With software testing, there are many avenues
for learning, and the more you learn, the more
you realize how much more there is to learn.
I constantly learn more about software testing
and I hope I have spurred your thinking on how
you can learn more about software testing too.
Even though I am no longer a beginner I still do
the following:
• Read about software testing and other
disciplines
• Attend and contribute to workshops and
conferences on software testing
• Identify skills and techniques I am
using when testing, and explain them to
colleagues
• Participate in mailing lists, forums and other
communities of practice related to software
testing
• Work with others skilled practitioners who
constantly challenge me to do better
• Practice, practice, practice
WhatIsTesting.com
Learning software testing is a rewarding,
career-long journey. We all learn differently,
and I encourage you to try different ideas and
see what works for you. Don’t be afraid to fail
when you try something new; we often learn
more from our failures than successes. Quality
guru W. Edwards Deming said: “Drive out fear.”
Don’t be afraid to try something new, if you
have an open mind, you are guaranteed to learn
something from it.
About the Author
Jonathan Kohl is the founder and principal
software testing consultant with Kohl Concepts
Inc., based in Calgary, Alberta, Canada. A noted
testing thinker, Jonathan is recognized as a
leader in the exploratory testing community. He
is a popular author and speaker who believes
that testing is a challenging intellectual craft.
Jonathan’s blog on software development and
testing issues is one of the most well-read testing
blogs in the industry. Jonathan is a regular
contributor to Better Software magazine, as an
author and technical editor. Check out other
writings at www.kohl.ca.
Notes
1
Kaner, Bach, Pettichord, Marick. The
Seven Basic Principles of the Context-
Driven School http://www.context-driven-
testing.com/
2
Pettichord, Bret.(2003-2007.) Four Schools
of Software Testing http://www.io.com/
~wazmo/papers/four_schools.pdf
3
Kohl, Jonathan. (2005) TDD - A Fifth
School of Software Testing http://www.
kohl.ca/blog/archives/000087.html
49
Sierra Atlantic Testing Competency Center
Our Testing Competency
With over 12 years of experience, Sierra Atlantic is an established offshoring services company
using CMM Level 5, ISO 9001:2002, and BS7799 certified process. Using our testing competency
center SATIN (Sierra Atlantic Testing Intelligence), we create speedy, low cost and high-quality
software. This in turn is delivered through our well-defined engagement models and process
methodologies and incorporates the industry’s best practices.

State-of-the-Art Facilities
Our state-of-the-art Interoperability Lab provides compatibility / interoperability testing services
to software product companies. The scope of this service includes:
x Hardware / OS platforms
x Enterprise Applications
x Middleware platforms
x Application Servers
x Web Servers
x Databases
x Browsers

With over 250 person years of experience in software testing services, our resources have in-
depth understanding of various testing methodologies and technologies. The following graph
illustrates this:
7% 8% Only manual testing
7%
WinRunner
10% Test Director
28% QTP
LoadRunner
Silk Test
Rational Robot

23%

17%

The following instances help demonstrate our competency.

A leading Enterprise Application product vendor

Sierra Atlantic’s Interoperability Lab enabled the testing of a leading Enterprise Application
vendor’s product on platforms such as Sun Solaris, HP-UX and IBM-AIX. This testing allowed the
customer to remotely log in and execute tests using its staff without having to build its own test
environments, an expensive and time-consuming process.

A leading portal product vendor

This client of ours required its product to be tested and certified across a variety of web servers
and platforms. The product also required to be tested for compliance under Rule 508. Sierra
Atlantic’s Interoperability Lab created multiple functional and compliance test cases and an
automation strategy based on Mercury test automation tools. As a result, the client was able to
cut in-house costs, and the testing rendered its product totally compatible and fully interoperable
with other leading enterprise applications and infrastructure products.

About Sierra Atlantic

Founded in 1993, Sierra Atlantic is a leader in offshoring enterprise applications, helping our
customers optimize their investments in Oracle, PeopleSoft, SAP and Siebel. We offer complete
lifecycle application management solutions -- development, implementation, integration, upgrade
and support -- using our NShore™ methodology. We integrate these point solutions into
Application Networks® that enable seamless business processes within and outside the
enterprise.
 by Mrityunjay :

QA
is More
Challenging
than

Development
I
come from a background in product product functionality and issues first hand, and
development and have been in QA for about delivering a complex product to the market. I am
a year or so, that too in a senior management very convinced that in a real world, it is easier
capacity and not necessarily an engineer. to don a developer hat than a QA hat. In this
It saddens me when I interview some good QA article, I attempt to show in a very simplistic
engineers and they seem almost apologetic way what I mean by this statement.
about the fact that they are into testing, and
Of course I say real world, because in an ideal
more so when they are into manual testing and
world, a Dev will be much more like a QA and
not automation. This probably stems from the
QA will be much more like a Dev, and they have
fact that writing code is considered as the best
joint ownership of quality of the product. The
(and smartest) aspect of software development
company I work in doesn’t claim to be the ideal
and anyone not doing that is considered less
company, but I find it much closer to it than
smart.
many other companies I have seen.
I started thinking about whether this is really
Let me start with an example. We are given
the case, that writing code is the smartest part
a functionality to implement: List all prime
of the software development lifecycle and what
numbers between two given numbers X and Y.
I came out was the reverse; that writing code
Now here is what a developer has to deal with:
is probably the easier part of the development,
it is testing and Quality Assurance (QA) which ● Find/implement an algorithm to check if a
is tougher. This is based on my discussions given number is a prime number.
with the QA engineers and leads, looking at the
● Implement a logic which cycles through

WhatIsTesting.com 51
 QA is More Challenging than Development

Test Entry

Design

Test Entry F

I
Functional
Requirement
G
Test Entry
U
Design
R
Test Entry
E
Product
Requirement -

Test Entry 1

Design

Test Entry

Functional
Requirement

Test Entry

Design

Test Entry

every number between X and Y, and apply
the algorithm in #1 above.
● Unit test this with some combinations of X
and Y and make sure output makes sense.
A QA engineer who relies on black box testing
and hasn’t looked at the implementation will
know that there are infinite inputs to this
implementation and so he/she will try to pick
some intelligent inputs which are more likely to
produce issues (think equivalent partitioning,
boundary values, etc). However, this is decidedly
risky since some of the options are being left
out and hence some issues may still remain.
A QA engineer who is a white box tester may
look at the code and decide there are 232
options only because the developer used a long
variable. This is nowhere better than infinity.
Some choices have to be exercised, probably
based on decision points in the code. Again
a risky affair, as QA is likely to miss many,
including non-function, parts, like dependency
on CPU and memory, impact of running on say

52 WhatIsTesting.com
 by Mrityunjay :
Japanese OS, etc.
Note some of the differences in the roles in a
developer and QA here:
● A developer decides on an algorithm and
unambiguously sticks to it, implements
and delivers to QA. For a QA, life is full of
ambiguities and risk-taking. A week-at-
heart QA is not a good QA.
● A QA has to second-guess Dev to figure
out what are the likely error scenarios if
he/she wants to avoid running an infinite-
input problem. This is analogous to 20-
questions game, where you think of a word
and someone has to guess it by asking only
20 questions. This is tough.
● Think of the implication if a developer
or a QA goofs up. If a Dev goofs up, it is
supposed to be caught by QA, causing bugs,
design changes, etc, a costly affair indeed.
However, if a QA does so, customers hit
those defects, and this is a much higher
cost to the company, in terms of money,
credibility and market share. Which
role is easier?
In a complex product, locating a bug is
like searching for a needle in the haysack;
you can apply heuristics, but it still is
a hard and challenging work, scary too,
when you know that a missed needle can
cause immense customer frustration,
and management backlash.
Let’s look at it from another perspective.
The waterfall model of product development is
a tree structure. A product requirement spawns
many functional requirements, each functional
requirement spawns many design and algorithm
pieces, which in turn spawn many test cases
(see fig 1).
Thus, even in such a simplified scenario
(where next phase only doubles the entities), a
product requirement generates 8 test entities
for verification. No wonder then that the QA
has to play the game of probabilities: trying to
find maximum number of high impact defects
WhatIsTesting.com
in the given time and money through various
heuristics and techniques, none of which are
scientifically proven to discover all the defects.
This is decidedly tougher job than development,
since the choices and the implications of choices
are much better known and understood when a
developer chooses a specific algorithm and its
implementation. Given the fact that a functional
requirement is broadly defined, it is easier for a
developer to meet those requirements and verify
they are met, than for a QA to verify developers’
output, since they are very precisely defined
(algorithms and code) and hence so many more
ways of proving failure exist.
To summarize my arguments above, there are
two basic points I have made:
● It is easier to drop a needle in a haysack (or
bug in a product) than to find it (discover
defect before release). The latter requires
you to be much smarter, customer-focused,
with good problem-solving abilities.
● QA problem is an undefined problem and
hence solving it takes very different skills
and lots of smartness in addition to hard
work.
However, don’t get me wrong: these
arguments take the example of a typical
developer role and a typical QA role and
not of individuals who play them. As I have
mentioned before, a good developer indeed
dons the QA hat many times, verifies the
algorithms he/she has produced and only
then allows it to be given to QA. So do not
use these arguments against any individual;
I have been a developer, and I would not love
to hear these arguments because I thought
I donned QA hats many times to produce
good quality code. Same goes for many of my
developer friends out there.
Now that we know doing QA takes a lot of
smartness and skill, what does it take to be
a great QA? Well, the trick is to think like a
developer sometimes, and that will be the topic
for a forthcoming issue.
53
 Is scripted testing bad?
Is
scripted testing
bad?
W
hen I started my testing career
over a decade ago, my manager
gave me two directives:
● Raise thirty bugs every day
● 80% of your bugs should be coming from
test cases that you would have written prior
to the test execution
While I was able to accomplish the task of
raising thirty bugs every day, I have not been
able to achieve the second target even after
twelve years of trying really hard. To figure out
what was going wrong I analyzed the bugs,
created checklists, got my test cases reviewed,
learnt about various test design techniques and
used them.
Sure, I was able to improve the percentage of
bugs being found but I never could achieve a
figure anywhere close to 80%. This made me
question the assumption that most of the bugs
can be caught by pre-written test cases. Many of
54 WhatIsTesting.com
25
20
15
10
0
the bugs I caught were the result of conditions
that occur in the real world but are impossible
to think about while designing test cases. And
there were many bugs that I did not find rather
the bugs found me. With these findings my
respect for writing test cases took a beating.
Exploratory testing had begun getting talked
about when I had formed my own philosophy
and technique of mixing test cases and ad-
hoc testing and it struck a chord immediately.
Exploratory testing described what I was doing
all my life. Now I had a name for it.
However, there was another trend that I noticed
- scripted testing, which I had been doing all my
life, started getting painted as a bad practice.
To set the context right, let me describe what
is generally meant by scripted and exploratory
testing.
“Exploratory testing is simultaneous
learning, test design, and test execution”
www.satisfice.com/articles/et-article.pdf
 by Vipul Kocher, CEO, PureTesting, NOIDA, India
“Scripted testing means that learning and
test design happens prior to test execution.”
http://en.wikipedia.org/wiki/Software_testing
Before you read on the rest of the article, please
pause for a moment and answer honestly if you
consider scripted testing a bad practice.
If you are careful about the contexts you are
likely to have answered “It depends… scripted
testing might be useful in some situations”
and that is the truth. Unfortunately, contexts
in which scripted testing is useful do not often
get elaborated resulting in a message “scripted
testing is bad.”
Today scripted testing is taking a beating from
exploratory testing and to take a stance against
it is seen as a sign of enlightenment. The stance
may vary from “scripted testing is bad” to
“amount of scripted testing to be done depends
on context” but in all cases the underlying
message that gets conveyed is “scripted testing
is bad.” It is unfortunate that there are very few
advocates of scripted testing who are actively
writing in favor of scripted testing or explaining
the contexts in which scripted testing is good.
Is scripted testing really bad?
Before we can answer the question “is
scripted testing bad” we first need to agree
on the definitions of scripted testing and also
understand the contexts in which scripted
testing is good or bad.
Scripted and Exploratory
Testing: definition is important
Without going into definitions of scripted and
exploratory testing the only questions I would
like to ask are:
● Who has created the definitions of scripted
testing?
● Who has defined exploratory testing?
If the definition of scripted testing comes from
the exploratory camp or vice-versa, the definition
would naturally portray one as more negative
than the other. The definitions put scripted
WhatIsTesting.com
Can there be scripted-
exploratory testing
continuum: meaning thereby,
scripted and exploratory
testing can be mixed in any
percentage ?
testing at one extreme and exploratory testing
at the other extreme. Then, there is this idea of
scripted-exploratory testing continuum which
means that scripted and exploratory testing
can be mixed in any percentage
Unfortunately there exists no definition for
this type of testing. But isn’t this nameless,
definition-less continuum what most testers,
at least the good ones, have been using?
The process being ‘create scripted tests, use
exploratory methods to create more tests, run
them, learn from them, create more tests and
run purely exploratory tests...’
To me it would be more natural if somebody
provided a definition for this mix of “scripted”
and “exploratory” testing. Let me call this
type of testing as Natural testing. With this
definition, I can call myself a “natural tester”
and not necessarily a “scripted” or “exploratory”
tester because in real life I am neither. This is
the position most of the testers I interact with,
take. They are neither pure exploratory nor pure
scripted testers. We start our testing whenever
our organizational goals permit us to which can
be at the requirements stage or late in the cycle
when development is about to be completed. We
write test cases which help us understand the
system and in the process of writing test cases
we uncover bugs, raise uncomfortable questions
and help improve the product. When we execute
tests, we use our pre-written tests, come up
with more test and scenarios to explore the
product; we document some of these discovered
tests and discard the rest. We mix testing using
test cases (scripted testing) with guided ad-hoc
(exploratory) testing. This is “natural testing” to
55
 Is scripted testing bad?
In exploratory testing test
design, execution and
learning happen at the same
time whereas scripted testing
uses pre-designed test cases.
us and we fall in neither the exploratory nor the
scripted camp.
Painting scripted testing as a
bad practice
Some of the arguments against scripted testing
include:
● Scripted testing does not promote thinking
about the problem in different ways. Scripted
test cases mechanize the act of testing,
taking intelligence away from testing.
Statements like above make people believe
that scripted testing promotes stilted
thinking. If you define scripted testing
as “following the pre-written test cases
VERBATIM” then the statement would make
sense. If this is how scripted testing was
to be defined, I would not like to be found
siding with scripted testing. Of course, there
is no need to define scripted testing in this
manner because that is not how most testers
test. Unless there is a client demand, most
testers do not follow the same test script
again and again. They use test script only
as a guide and not the gospel that has to be
followed verbatim. Testers vary their data,
their actions, and the sequence of actions.
● Scripted testing wastes time. Time is wasted
in writing test cases whereas it could be
spent testing. Also requirements change
often, resulting in time wastage when test
cases have to be reworked.
The difference between scripted and exploratory
testing is said to be that in exploratory testing
test design, execution and learning happen at
the same time whereas scripted testing uses
56 WhatIsTesting.com
pre-designed test cases.
There are numerous advantages of “scripted”
testing over “exploratory” testing, some of them
being:
• The act of writing test cases promotes logical
and more complete thinking.
• Written test cases can be reviewed. This is
not a small advantage. The review is likely to
result in better test cases and consequently
better testing.
• When test cases are written along with
the requirements being written, many
requirement bugs can be found and removed
even before any code is written.
• Test cases can be shared with developers
resulting in removal of bugs even before
they can be “created”.
• Written test cases can be used as an aid for
training new testers.
• Even if requirements change, the impact of
requirement change as related to testing can
be addressed in a better way with written
test cases.
In my experience of testing, many complex
products including some which are used by
millions of users, review of written test cases
has resulted in better understanding of testing
and consequently better execution of testing by
all testers involved in the review and not only
the person who reviewed the test cases.
Of course, there are situations where one
does not have the time to write test cases. But
enough people write about these contexts. My
purpose to write this article was to defend what
most people today would not defend. I do not
favor exploratory testing over scripted testing or
vice-versa. I favor using whichever style gives
the most benefit in a given situation, rather
than clinging to one end or the other of the
continuum.
 Putting the “Test” back in Test Plan

Putting the
“Test”
back in
Test Plan

U
ntil fairly recently, many people who What to do when Confusion sets
called themselves Software Testers in
came by their jobs by accident rather
I suddenly found myself in the uncomfortable
than by design. This placed many
position of not really knowing something that
of us in the difficult position of having to learn
I thought I already understood. What do I do
how to do our job while we were trying to do that
now? Where do I start?
job. One of my biggest challenges was trying to
figure out what goes into a Test Plan. I began by asking people I worked with. I
networked a bit and asked other testers for
In the first company that I worked for, the answer
advice, but most of the time no one would say
was straightforward enough. A Test Plan was
much because they thought it was confidential
simply a commonly-grouped set of test cases
to share information like that. I even went to
that was executed by a Tester. For example,
a day-long workshop on “Managing Systems
there was a Print Test Plan, a GUI Test Plan, a
Testing Projects”. While I found the workshop
Math Test Plan, and so on. No problem.
to be very insightful, I didn’t believe everything
Doubt in that definition began to creep into the instructor had said because a lot of it went
my mind when I started to network with other against what I had learned so far on my own.
Testers in town and found out that Test Plans For example, a Test Plan was not supposed to
meant different things to different people. When have any test cases in it. This just increased
I got my first Team Lead role in the next company my doubt and anxiety.
I worked for, I discovered that the Test Plan
Back at my desk, faced with an empty Microsoft
I was being asked to produce was something
Word document in front of me, I searched the
completely different. It was supposed to be a
Internet for Test Plan templates and examples.
planning document of some kind.
I needed more opinions and references than

58 WhatIsTesting.com
 by Paul Carvalho
Acknowledging that prior
experience and knowledge
based upon someone else’s
incomplete knowledge is the
biggest hurdle
just my word against my workshop notes. It
was at that time that I stumbled upon a few
mailing lists focussed on Software Testing and
Quality Assurance. People on those lists were
very friendly and helpful and in no time I had
several examples to help me develop my first
test plans.
The biggest hurdle I had to overcome was
acknowledging that my prior experience and
knowledge were based upon someone else’s
incomplete knowledge. Until that moment, I
thought that I knew what a test plan was. Then
how was it that the definition changed when I
changed companies? I had to unlearn the ways
of the Dark Side.
Test Plan 1.0
Many of the examples and templates that I had
were very similar to each other with some minor
variations between them. In general, I found
that a Test Plan contained elements like the
following:
● Overview and Objectives
● Testing Scope and Strategy (Features To
Be Tested, Not Tested, etc.)
● Approach, Entry and Exit Criteria, etc.
● Responsibilities and Staff
● Schedule and Deliverables (Test Reports,
etc.)
● Resource Requirements (Test
Environment, Tools, etc.)
● Risks and Contingencies
WhatIsTesting.com
I became quite proficient at creating these
Test Plan documents and customising them
for the needs for each project. I even became
familiar with some industry standards for Test
Documentation1.
Then I started to notice an interesting pattern
emerge with the more projects I worked on. For
one thing, no one from any other department
ever read any of these documents, even when
they asked me to produce one. For another
thing, no matter how I modified this document,
I never really found it useful in the day-to-
day management of the test projects I worked
on. Finally, creating and maintaining the
test documentation always seemed to take a
significant amount of time and effort away from
the actual testing activity itself.
Naturally I tried to find ways to improve the
efficiency of creating and maintaining these
documents, but after a while I just started to
question the usefulness of the document entirely.
One day I realised that I had fallen into the trap
of forgetting to question why I even had to create
one. This time when I asked the question the
response disturbed me: “Because it is expected
that you create this documentation.”
Wait a minute! Hold the phone! Who expects me
to create this documentation? Why do I have to
create this documentation? Where’s the value
in that? I’m the one doing the job here and I’m
seeing that I’m wasting my time creating and
maintaining documents and procedures that
don’t actually help me do my job any better.
Where’s the sense in that?
A Brief Glimpse into
Project Management
By sheer coincidence, it was at
that moment that I attended a
Project Management workshop. I
had requested it because I was
struggling to manage several
different test groups and projects
simultaneously. An interesting
thing happened after attending
that workshop: I realised that I
59
 Putting the “Test” back in Test Plan
Nature abhors a vacuum,
and Software Development is
no different
never wanted to create a Test Plan again.
During the course I learned about the Project
Management Institute2, we covered the Project
Management Body of Knowledge (PMBOK),
and worked through several examples in
order to understand how to apply the correct
knowledge, tools and techniques in order to
successfully manage a project to completion. I
highly recommend such a course for any Test
Manager. One of the most significant aspects
of the course for me was learning about the
elements of a Project Work Plan:
● The Project Team
● Project Charter
● Scope Management Plan
● Risk Management Plan
● Communications/Reportin
Plan
● Work Breakdown Structure
● Master Budget
● Task Responsibility
● Critical Path Schedule
Wow. That was it! I finally understood what it
was that a Test Plan must have originally tried
to accomplish. At some time, before Project
Management became a formally recognised
role in a Software Development organisation,
it must have been commonplace for each of
the teams to be required to produce their own
little mini project work plans. Since that need
doesn’t exist anymore, it was time to get with
the program and stop wasting my time creating
test documentation that clearly no one needed
anymore.
Nature abhors a vacuum, and Software
60 WhatIsTesting.com
Development is no different
When you stop creating Test Plans, you run
into all kinds of resistance, especially from
Management. There are some good reasons
to create extra documentation and there are
many bad reasons. To be clear, “Because it’s
expected” is not a good reason.
One valid reason for producing some
documentation is to help you learn from what
you did on a test project. Being faced with
all of the missed bugs now being reported by
Customers through the Support department
after a product ships is a harsh wake-up call. If
you don’t have a record of the who, what, where,
when, how, or why you did or didn’t test certain
things, it is more difficult for you to learn how
to improve and reduce the number of escaped
bugs in the next project.
It is important to note that a Project Plan doesn’t
completely replace a Test Plan though. While
the project plan may outline all the players,
the general work breakdown structure and so
on, there is at least one thing not captured to
any useful detail by a Project Manager. One
of the single most important things that a Test
Team needs to own on any project is the Test
Strategy.
It took me a few years to become reasonably good
at developing and implementing an effective Test
Strategy. I think it is a key skill for any Tester or
Test Manager. My learning path involved “going
back to square one” so that I could really know
what I needed to be effective. Nowadays there
are some good references and workshops that
can help jumpstart you to reasonable success.
Test Plan 2.0 – A Fresh Start
In the end, it comes down to this: you will never
be able to test everything; you will never find
all the bugs; and you will never have as much
time, people, or resources as you really want.
Starting with that in mind, your Test Strategy
needs to help you work with what you have,
in the time you have, to do the best job you
possibly can.
 by Paul Carvalho
Why do certain test practices
or approaches seem to be
successful sometimes but not
always?
For me, the best way to accomplish this kind
of “plan” was to use a Risk-Based Testing
approach3. A few authors in particular had
the most influence on me during my on-the-
job education: James Bach4, Cem Kaner5, and
Ross Collard. It was at that time that I also
came to learn about the Context-Driven School
of Software Testing.6
Context-Driven Testing was an interesting
paradigm shift that helped me to understand the
last piece of the test management puzzle: why
do certain test practices or approaches seem
to be successful sometimes but not always?
Because that’s the way it is. Success depends
on a great number of factors that change with
almost every project. To be a successful Test
Manager requires that you are a keen observer
of these influencing factors and can adapt your
team’s dynamics and approaches to suit the
needs of your current situation. Darwin had it
right when he said that things in nature adapt
or die. We should too.
Armed with this new knowledge, I am now
quite comfortable developing Test Strategies
tailored to suit the needs of each project that
I work on. I take no offence when I am asked
to produce a “Test Plan”. Instead of providing
a document that someone else thinks I need, I
create something that I find useful – a usable,
working Test Strategy. Instead of worrying
about templates, cover pages, and document
control systems, I worry about Quality Criteria,
Technical Risks, and Test Techniques.
A typical Test Strategy document of mine is
about one or two pages in length, and I don’t
worry about formatting or templates anymore.
It sits on a network drive that is accessible to
the whole development team, and we update
WhatIsTesting.com
our strategy as the risks and priorities change
throughout the development phase. This is
Agile Development. Things change and we need
to adapt our testing strategy along with the new
information as it arises, whenever it arises.
The next time you are asked to provide a “Test
Plan”, remember to focus more on the “Test” than
the “Plan”. If you find you are putting too much
effort on the latter, you should take a serious
look at the value that the documentation effort
is providing your organisation. Time is money,
and the last time I checked, I’m getting paid to
test.
References
1. An example of an Industry Standard for
Test Documentation is IEEE Standard 829
- see http://standards.ieee.org/
2. To learn more about the Project Management
Institute, you can find them on the web at
http://www.pmi.org/
3. For an excellent article to get you started
on Risk-Based Testing, read James Bach’s
article “Heuristic Risk-Based Testing”,
available on his web site at http://www.
satisfice.com/articles/hrbt.pdf
4. James Bach: to find out more about his
work and for excellent resources on Test
Methodology, explore his web site at
http://www.satisfice.com/
5. Cem Kaner: author, professor, and
advocate of great software testing practices,
he maintains several useful web sites.
His personal site is http://www.kaner.
com/ but most of his work and Testing
educational materials can be found for free
at http://www.testingeducation.org/
6. To find out more about Context-Driven
School of Software Testing, check out
http://www.context-driven-testing.com/
or pick up the book “Lessons Learned
in Software Testing, A Context-Driven
Approach” by Kaner, Bach and Pettichord.
61
 Accelrys has over twenty years of innovation and technology leadership in

delivering solutions that transform discovery and development research.

The company provides market-leading modeling, simulation and informatics

software, data pipelining and platform technology, and solutions consulting

and support services. A unique combination of expertise in both life and

materials science lets Accelrys serve a diverse range of clients, including some of

San Diego, CA USA the world’s leading pharmaceutical, biotechnology, chemical, nanotechnology,

government and academic research organizations.

Cambridge, UK
Accelrys is a global company. Our office in Bangalore, India undertakes
Bangalore, India
engineering and quality control tasks.
Burlington, MA USA

Tokyo, Japan For more information, visit our website:

Paris, France
 EXPRESS YOURSELF
“WhatIsTesting” magazine is written for software and application development mangers, project
managers, team leaders and Test & QA managers. The magazine is not written for the entry
level tester running test scripts all day, but rather has a larger more strategic view of the entire
application life cycle.
The reader is a decision maker within an IT department. Articles in the magazine should provide
useful information to help him/her understand trends and emerging technologies, come to grip
with new and timeless challenges, adopt best practices and ultimately make better decisions to
improve software quality.

Presentation
Articles should be in the range of 1500 to 4000 words, and be written for practitioners in the field;
with a style that is down to earth, practical, interesting and original. Avoid jargon, speculative
theory and abstract concepts; instead, focus on practical information that can help our readers
today or in the near term.
The purpose of an article is to transfer useful knowledge to the magazine’s readers, rather than to
promote the author’s accomplishments or the products, services and technologies offered by the
author’s employer.
Don’t start the article with long fancy anecdote, a “first-the-earth-cooled” historical timeline or
an academic-style abstract that says, “This article will do such and such”. Just get right into the
subject matter. Also, don’t include footnotes or references. Refer to other sources, such as book,
magazine articles or Web sites, within the main body of the article. Be sure to define terms and
acronyms that may not be familiar to the reader; incase of doubt, define.
Include, annotated source code, only when it helps explain the concepts in the article. Also, add
copious figure, tables, listings, screen captures and other graphics to make the article easier to
understand. Please avoid bulleted points within our prose; it is hard to read such lists. If you feel
that a list can best be expressed using bullets, consider pulling it into a separate tale or textbox.
Do not assume that the reader has specialized experience or knowledge about particular
methodologies, platforms or tools. If the article is written about a specific methodology, platform or
tool, present that dependency in the initial proposal. In the manuscript, include a short introduction
or overview of prerequisite background concepts, if appropriate.

Article Reviews
All proposals and manuscript submissions should be sent to the magazine by the article’s author.
If there are multiple authors, one author will be designated as the lead author. The editors may
send article, proposals and finalized manuscripts for peer review. In some cases, the editors might
ask perform a peer technical review of the manuscript.
Once the necessary review has been completed’ the article will be sent back to the article’s author
for final approval.
Welcome to TEST 2008
Test2008 is the first conference being organized
by PureConferences in India. Our conference
will provide a platform for international
and national test professionals to interact
and participate. Speakers from around 10
countries, such as USA, UK, France, Sweden,
Canada, Italy, Netherlands, including India will
deliverkeynotes, tutorials, and papers during
the conference. We intend to involve academia
and institutions of learning with our conference.
We also intend to make Test2008 a ‘green’
conference as far as possible. Additionally, we
will institute two test scholarships for educating
promising students unable to finance their
studies.
Animated discussions among testers frequently
threw up the requirement of agility in testing
and the challenges associated with it. Creating
a balance between conflicting and changing
demands of quality by customers and timely
releases by organizations is a real challenge.
Agility in Testing is the theme of Test2008--Test
Excellence through Speed and Technology.
Agility represents nimbleness, resourcefulness,
and adaptability. In the world of testing,
agility is synonymous with the testing team’s
resourcefulness and ability to respond quickly
to changing contexts. Organizations need
to catch the ‘window of opportunity’ when
releasing a product to remain profitable. At the
same time, we, as customers only want to use
‘quality’ products.
We are hoping to have intense discussions
around the theme to make it a learning
experience for all participants. The theme is
broad enough and will cover diverse topics.
Keynotes, Tutorials, and Paper presentations
by renowned speakers known internationally
and in our country will be the highlight of the
conference.
Please mark theTest2008 conference program in
your calendar. Our conference will offer partners
theopportunities to associate themselves with
this endeavor and showcase their products and
participate in the conference.
If you want to present a paper or tutorial, please
fill in theSpeaker Submission Form. To attend
the conference, please Register.
We welcome all delegates to our conference.