Methodoly to evaluate

automated testing
 Steps involved
 Define your test requirements :
 Goals to be achieved should be identified in a format that will help us identify if they have been
met (can be like reducing the processing time of a program).
 When a centralized management method is employed, the statement of goals may include a
budget and a desired completion date.
 The statement of goals should also identify responsibilities.

 Set tool objectives:

 The goals generated in Step 1 should be translated into desired tool features and requirements .
 Constraints on tool cost and availability may also be added at this step.
 For example, a typical tool objective for a program format is to provide header identification,
uniform indentation, and the facility to print listings
 . Only tools that have been in commercial use for at least one year and at no fewer than N sites
should be considered.
 Conduct Selection Activities for Formal Procurement:
 Develop the Acquisition Plan
 Create the Technical Requirements Document
 Review Requirements
 Generate the Request for Proposal
 Solicit Proposals
 Perform the Technical Evaluation

 Procure the Testing Tool:

 In addition to verifying that the cost of the selected tool is within the approved budget, the procurement
process considers the adequacy of licensing and other contractual provisions and compliance with the
fine print associated with all government procurements.
 The vendor must furnish the source program, meet specific test and performance requirements, and
maintain the tool.

 Create the Evaluation Plan:

 The evaluation plan is based on the goals identified in Step 1 and the tool objectives derived in Step 2.
 It describes how the attainment of these objectives should be evaluated for the specific tool selected
 Create the Tool Manager’s Plan:
 The tool manager’s plan describes how the tool manager is selected, the responsibilities for the
adaptation of the tool, and the training that is required.
 The tool manager should be an experienced systems programmer who is familiar with the current
operating system.
 The tool manager’s plan must be approved by software management.

 Create the Training Plan:

 The training plan should first consider the training that is automatically provided with the tool (e.g.,
documentation, test cases, and online diagnostics).
 These features may be supplemented by standard training aids supplied by the vendor for in-house
training (e.g., audio- or videocassettes and lecturers).
 . The personnel to receive formal training should also be specified in the plan, and adequacy of in-house
facilities (e.g., number of terminals and computer time) should be addressed.

 Receive the Tool:

 The tool is turned over by the procuring organization to the software engineer.
 Perform the Acceptance Test:
 The software engineer or staff should test the tool in an as-received condition with only those
modifications made that are essential for bringing the tool up on the host computer
 Once a report on the test has been issued and approved by the software manager, the tool is officially
accepted.

 Conduct Orientation:
 When it has been determined that the tool has been received in a satisfactory condition, software
management should hold an orientation meeting for all personnel involved in the use of the tool and
tool products .
 Personnel should be reassured that allowances will be made for problems encountered during tool
introduction and reminded that the tool’s full benefits may not be realized for some time.

 Implement Modifications:
 This step is carried out by the tool manager in accordance with the approved tool manager plan.
 It includes modifications of the tool, the documentation, and the operating system.
 Typical tool modifications involve deletion of unused options, changes in prompts or diagnostics, and
other adaptations made for efficient use in the current environment.
 Train Tool Users:
 Training is a joint responsibility of the software engineer and the tool users and should help promote tool
use.
 The software engineer is responsible for the content (in accordance with the approved training plan), and
the tool user controls the length and scheduling of sessions.
 . The tool user should be able to terminate training steps that are not helpful and to extend portions that
are helpful but need further explication.

 Use the Tool in the Operating Environment:

 The first use of the tool in the operating environment should involve the most qualified user personnel
and minimal use of options.
 . This first use should not be on a project with tight schedule constraints.
 If the first use is successful, use by additional personnel and use of further options may commence.

 Write the Evaluation Report:

 Using the outline generated in Step 5, the software engineer prepares the evaluation report.
 User comments and toolsmith observations provide important input to this document.
 Tool and host computer modifications are also described in this report.
 Determine Whether Goals Have Been Met
 Funding management receives the evaluation report and should determine whether
the goals that were established in Step 1 have been met. This written determination
should address the following:
 Attainment of technical objectives
 Adherence to budget and other resource constraints
 Timeliness of the effort
 Cooperation from other agencies
 Recommendations for future tool acquisitions