Functional Testing Software Evaluation Form

ITA – Premium: Selection Advice

Introduction: How to Use This Tool

Scoring vendor solutions based solely on their presentations or marketing material provides only a limited picture of
a software solution. Due diligence must be performed in order to verify vendor claims. Like a resume, a vendor’s
proposal often requires closer evaluation beyond the information provided to the client. Just as job candidates must
be interviewed to uncover potential holes in their resume, so too should vendor responses be scrutinized to validate
the information given in their proposals.
This evaluation form includes two sections:
• Trial software evaluation. A vendor evaluation may be performed by way of a software trial or a
demonstration of a customer installation. The demonstration should be well planned and scripted, and the
scoring of the demonstration should focus on ensuring that both primary and specialized business needs
are met.
• Vendor references. When investigating implementation and training times and overall success of software
deployment vendor references can put some real figures to validate expected results promoted by vendors.
Look for references where:
o The business is relatively similar in size and industry.
o Business needs are reasonably comparable.
o The scope of implementation is also comparable; for example, number of licenses, full suite versus
functional tool only.
Delete all information where text is colored grey (such as this paragraph). Fill in or delete all form fields shaded in
grey. Be sure to change all text to black before printing or sending.

Trial Software Evaluation Form

The vendor demonstration should be well established and planned in advance. Specifically, list the capabilities that
you want to compare across all vendors and notify the vendor demonstration team so that they are prepared to
show exactly what you need to see. All objectives and scenarios should be pre-scripted in a separate document.
Sample evaluative categories and descriptions have been provided for you. Feel free to add, delete or amend as
required.

Scoring Scale:
1 = Very Negative
2 = Negative
3 = Somewhat Negative
4 = Somewhat Positive
5 = Positive
6 = Very Positive
Functional Testing Software Evaluation Form
ITA – Premium: Selection Advice

Basic Operations
Example comments and scoring included – erase grey text before using

Score
Operation Description Comments
[(Out of 6)]
How well does this tool: Some GUI controls needed
• Recognize non-standard GUI controls. additional programming as
• Detect and record timing changes in events. they weren’t recognized by the
• Validate the system under test as the test tool
Execution case progresses.
4
• Selectively run a portion of a test suite.
• Allow users to set the verbosity level of log
files.

How well does this tool: Test error messages hung up

• Run completely unattended and provide a the tool overnight and test did
log of events. not complete.
• Drive the system under test automatically.
• Reboot the system under test when needed
and continue testing.
Unattended
Operation • Recover to a known state from a hung state 2
when required, such as when a model
dialogue is left open.
• Drive multiple test suites to allow complete
unattended operation in a chronological
fashion.

Test Case Development

Example comments and scoring included – erase grey text before using
Score
Test Case Development Description Comments
[(Out of 6)]
How well does the tool playback test Did not test in Safari without
scripts in browsers other than the one plug-ins
used to record the test case without
Test Playback Format 3
significant reworking of the script (e.g.
record in Firefox and test in IE)

Does it support script generation Scripted recorded tests

Script Generation
Ease of Scripting
through GUI action recording? If so, appropriately
6
how flexible is it and how well does it
integrate with manual coded scripts?
Are test scripts easy to create? Recording function works well 5
Functional Testing Software Evaluation Form
ITA – Premium: Selection Advice
Are there tools to assist in creating Yes, but they require further
Scripting Tools test plans or scripts? training for us to learn how to 3
use them
Functional Testing Software Evaluation Form
ITA – Premium: Selection Advice

Tool Integration
Score
Integration Description Comments
[(Out of 6)]
Does the tool integrate with
Project Management project management tools?
Requirement Does the tool integrate with
Management requirements management tools?
Does the tool integrate with a
portfolio of testing tools by the
Suite Integration
vendor? (i.e. Is there an overall
test management suite?)
How well does the tool integrate
Version Management
with versioning tools?
How well does the tool integrate
Third party integration
with any other systems?
What operating systems does the
Operating Systems
tool support?

Reporting
Score
Reporting Description Comments
[(Out of 6)]
Does the product deliver
Reporting Aesthetics
adequate look and feel reports?
Is there a good selection of
Variety of Reporting
reports?
Accuracy of Reports What is the accuracy of reports?

How well do the reports

Communication with
communicate defects to
Developers
developers?

Additional Features
Score
Additional Features Description Comments
[(Out of 6)]
Does the product offer any
Unique Features
features that others do not?
Does the product come with
Security Features
security features?
Application Lifecycle Does the product offer tools for
Management other stages in the SDLC?
Does the tool contain a
Script Library
centralized component library?
Workflow Are there workflow capabilities?
Functional Testing Software Evaluation Form
ITA – Premium: Selection Advice

Vendor/Product References
Checking vendor references should only be done once the vendor candidate has been established as a finalist in
the bidding process. The reference should be someone who had direct experience with the related purchase, and
the company must be relevant in size, industry, and business need. Questions should be prepared in advance and
written in a way that will uncover any areas of uncertainty.

Reference Interviewers
Interviewer James Woods
Interviewer Jack Fox
Interviewer [Insert name]

Evaluation Questions
Sample questions have been provided for you. Feel free to add, delete or amend as required.

Scoring Scale:
1 = Very Negative
2 = Negative
3 = Somewhat Negative
4 = Somewhat Positive
5 = Positive
6 = Very Positive

Score
Category Question Comments
[(Out of 6)]
General What does your development and QA n/a
environment look like ((# of testers, testing
methods, technical and analysis skills, formality
of process)
What was the primary business driver behind n/a
procuring the solution when you did?
What was the primary factor behind picking this n/a
particular solution compared with the
alternatives?
Have you used solutions from this vendor n/a
before?
Financial Did the cost to implement the solution match your
Performance projections at time of purchase?
Have the costs associated with ongoing
maintenance thus far met your projections at time
of purchase?
Have you achieved a return on your investment
yet?
Did you experience any cost issues that came as
a surprise?
Functional Testing Software Evaluation Form
ITA – Premium: Selection Advice

Score
Category Question Comments
[(Out of 6)]
Technical Did the delivered solution include all
Performance required technical elements as presented
to the vendor?
Has the solution met all documented
service-level expectations?
Has the solution met or exceeded your
expectations from a technical
performance perspective?
What is the monthly number of help
tickets received internally in regards to
the solution?
What is the monthly number of calls
made to the vendor for solution support?
How frequently do the same
performance problems have to be re-
addressed with the vendor?
Did you experience any technical issues
that came as a surprise?
Vendor For how long have you worked with this
Relationship particular vendor?
Do you have a dedicated contact person
with whom you interact as required?
Have your interactions with vendor
representatives been professional and
courteous?
Has the vendor responded in a timely
manner to your questions and concerns?
Did the vendor meet all stated
implementation and other project
deadlines?
How effective has the vendor been in
following through on promised
improvement and upgrade plans for the
solution in question?
Did you experience anything in the
vendor relationship that came as a
surprise?
Do you plan to continue a relationship
with the vendor after your current
agreement with them expires? If not,
why?
How would you rate your overall
satisfaction with the solution?
If there is one thing the vendor could
improve, what would it be?
Functional Testing Software Evaluation Form
ITA – Premium: Selection Advice

References to Be Interviewed
Role in Dealing Score
Company Name Contact Comments
with Vendor [(Out of 6)]
Bartholomew P. Primary Claims vendor was on-time, and
ABC Company
Karalewski Purchasing Agent continues to deliver strong service.

_____________________________________________________

Info-Tech Research Group tools and template documents are provided for the free and unrestricted use of
subscribers to Info-Tech Research Group services. These documents are intended to supply general information
only, not specific professional or personal advice, and are not intended to be used as a substitute for any kind of
professional advice. Use this document either in whole or in part as a basis and guide for document creation. To
customize this document with corporate marks and titles, simply replace the Info-Tech Information in the Header
and Footer fields of this document.