How to Set Up Successful Test Practices in a Large Organization

BY SVEN SAMBAER CTG Test Consultant

Introduction
Staying competitive in a global economy requires the development of high-quality, reliable IT systems that support the business fully and cost-effectively. Because structured testing provides invaluable insight into the risks involved in each stage of a development project, it is an essential element in the successful development and deployment of those systems. In recent years, companies have begun to recognize the crucial importance of a structured test approach and an efficient test organization. Nevertheless, many organizations are still looking for a successful way to set up a company-wide structured testing process. This report presents a proven approach CTG has used to assist many organizations in establishing effective test practices. It also describes valuable lessons weve learned in the process, and suggests practical considerations that need to be taken into account.

A SPECIAL CLIENT REPORT

Copyright 2005 Client Confidential: A special report for clients only. The material covered in this report is for guidance only. Application and implementation guidelines, advice, and consulting are available.

Copyright 2005 Client Confidential: A special report for clients only. The material covered in this report is for guidance only. Application and implementation guidelines, advice, and consulting are available.

A SPECIAL CLIENT REPORT

Confidential Information: Special Report and Analysis for CTG Client Use Only

The Challenge
The basic requirements for a company-wide test approach are extensive and nonnegotiable. A successful approach must be: Simple and easily understandable, yet complete and consistent Readily adaptable to widely different kinds of projects (e.g., new development or maintenance; small or large) Effective in different environments and for different application types (e.g., mainframe applications, web applications, or packages) Capable of being seamlessly integrated with existing methodologies and frameworks Able to accommodate the growing complexity of current applications, tightening development and deployment schedules, shorter release cycles, and limited resources Acceptable from an organizational perspective, both for IT and for the business The most critical success factor in the development of an organizational test program is the formulation of an efficient strategy for implementing the new practices across the enterprise. The right strategy is one that can gain the necessary management support, create awareness of the importance of testing sufficient to overcome resistance, aid in training the project teams, and address the company-wide adoption of the new procedures. Designing and implementing a new software test process in a small or medium-sized organization is relatively manageable. The challenge becomes more complex in large companies with several hundreds or thousands of employees working in different application groups, business lines, and project teams, who have most likely developed their own methods and views on testing and how it should be carried out.

Designing and implementing a new software test process in a small or mediumsized organization is relatively manageable. The challenge becomes more complex in large companies with several hundreds or thousands of employees working in different application groups, business lines, and project teams, who have most likely developed their own methods and views on testing and how it should be carried out.

The Right Approach

Assess Current Practices
The essential first step is to perform a comprehensive and unbiased inventory of current test practices. Begin by gathering information on the organizational structure, the technical environments, and the different projects, to name a few areas. Learn about existing development and maintenance processes and how testing fits into those processes. Interview project managers, developers, and test staff to gain a clear understanding of their concerns, methods, and expectations regarding a new test approach. Analyze test documentation (including procedures, templates, and deliverables from different projects) and evaluate the use of test tools and techniques. This inventory identifies strengths and weaknesses in the current test process. The strengths represent the practices that have already proven their usefulness, and which should be incorporated into the new methodology wherever possible to help overcome resistance to the new processes being developed.

A SPECIAL CLIENT REPORT

Confidential Information: Special Report and Analysis for CTG Client Use Only

The weaknesses deserve equal consideration when designing the new test approach. Pay special attention to any low-hanging fruit: weaknesses that are particularly obvious and annoying and that can be easily resolved (quick wins) by a new approach, which in turn can attract strong allies in the effort to gain support for the new test process.

Design the Methodology

In recent years, companies have begun to recognize the crucial importance of a structured test approach and an efficient test organization. Nevertheless, many organizations are still looking for a successful way to set up a company-wide structured testing process.

The first design consideration is whether to adopt an existing test methodology or formulate a new one. CTGs experience suggests that the best strategy aims for an efficient mix between a standard, by the book methodology and the best practices already in place. The same approach is true of any test terminology already standardized across the organization. While some excellent glossaries are available (e.g., BS 7925-1 Vocabulary of Terms in Software Testing), it may be more advisable to integrate the organizations existing test vocabulary with the the industry standard. The new test methodology is set forth in a test handbook that describes the what, when, how, by what, and by whom of testing in your organization. The handbook contents should include: A description of the organizations test process, levels, and types Procedures and guidelines on the test activities throughout the process Templates and examples of all test deliverables A description of the test organization that includes responsible parties, tasks, responsibilities, and reporting lines Techniques in support of the test activities A glossary with the uniform test terminology decided upon Where test automation is concerned, test tools may offer support for test activities in all phases of the test process. The assessment phase will identify the current use of test tools, if any. The following questions must be asked: Within the new test approach, what is the potential for test automation? What opportunities exist for test tools to maximize efficiency and deliver significant ROI? During the design of the test handbook, define the documents target audience. The potential audience includes all those involveddirectly or indirectlywith test activities. They might include: Test managers, test coordinators, and test engineers Developers, analysts, and integrators Project managers, program managers, and unit managers Users System administrators and database administrators The test handbook should be structured to allow these different profiles to access the information they need without requiring them to read the complete methodology

A SPECIAL CLIENT REPORT

Confidential Information: Special Report and Analysis for CTG Client Use Only

which may run over 100 pages. For that reason, we advise keeping the handbook simple and practical through the use of process workflows. A test process workflow typically consists of: Actors (those involved in the test process) Activities (the test activities they perform) Output (the deliverables of a particular activity) Input (information necessary as input for a particular activity) The advantage of the workflow approach is its clarity and comprehensibility to the reader. A few workflows can explain the foundations of the test process. More detailed information can then be found behind the workflows.

Input

Set Up the Test Organization

Activity Activity Actor Activity Activity Activity Activity

An important element in the test handbook is the definition of the organization supporting the test process. One of the most crucial prerequisites for the successful implementation of a company-wide structured test process is a clear description of the various roles of those engaged in the test activities, including their tasks and their responsibilities. Typical roles are the test manager, test coordinator, test engineer, and test tool administrator. Here, as with practices, the starting point should be the strengths of the current test organization.

Output

Output

One of the key questions that will probably surface is the choice between a centralized and a distributed test organization. Each has its advantages. A centralized test department promotes focus and specialization, offers better career paths and profesWorkflow key element
Actor

Detailed information
Access to information regarding the tasks and responsibilities of this actor (e.g., the test coordinator), as well as a list of the deliverables this individual is responsible for producing throughout the process Detailed guidelines on the purpose, steps, tools, and techniques of this activity Templates and examples of the test deliverables The relevance of documents from other processes (e.g., requirements) and how they act as input for the test process

Activity

Output Input

A SPECIAL CLIENT REPORT

Confidential Information: Special Report and Analysis for CTG Client Use Only

sional growth for testers, allows more resource flexibility, and guarantees a consistent use of the test process across the organization. Distributed test teams, on the other hand, have the major advantage of better business knowledge and closer working relationships in projects. We advise blending both models in a hybrid organization. In this approach, a test staff with extensive business expertise is still found in the application groups, and a centralized test division is responsible for the test methodology; training programs; test tool expertise; test environments; and the consolidation of test expertise and experience, data, and test coordination.

One of the most crucial prerequisites for the successful implementation of a company-wide structured test process is a clear description of the various roles of those engaged in the test activities, including their tasks and their responsibilities.

A concept that has proven its worth in many large organizations is the test coordinator function. Typically, a test coordinator is assigned to one or more projects and is responsible for defining test strategy, creating test plans, performing test planning, appointing test resources, following up on test activities, and reporting to the project management. The test coordinator also acts as a mentor and proponent of the test methodology who helps existing test teams relate the new test methods to their jobs; trains them; and coaches them in their day-to-day test activities.

Define the Implementation Plan

Once the test handbook has been prepared and the test organization has been designed, the next step is to define a strategy for implementing the new test practices across the organization. This strategy is consolidated in the implementation plan, which describes the change process for converting the current test practices to a new test approach that can be generally accepted and applied across the organization. To facilitate conducting the change process gradually and in a controlled manner, the plan should be organized in phases and activities. Definition of the implementation activities should take the following points into account: Presentations, mailings, and publications in company magazines can increase awareness of the importance of a consistent test approach and the changes that are being proposed. The organizations intranet can be an excellent medium for publishing the test handbook, especially if the workflow approach described earlier is used. It is possible to create a test process website that enables clicking through the workflows in search of the relevant information. Furthermore, the intranet can feature training schedules, contact lists, manuals, white papers, and links to other useful information. Training sessions are useful in promoting the concepts of the new test approach and helping employees understand how it applies to their jobs. Training should be provided not only for the test profiles, but for everyone who will be confronteddirectly or indirectlywith the changes. One sensible approach is a modular training program that incorporates separate training modules for the various profiles. This allows a close focus on the particular concerns and deliverables of a specific profile (e.g., test engineer, project manager, developer). Its best to avoid a big-bang implementation of the new test methodology. A better approach is to select a few pilot projects to evaluate the proposed

A SPECIAL CLIENT REPORT

Confidential Information: Special Report and Analysis for CTG Client Use Only

Training sessions are useful in promoting the concepts of the new test approach and helping employees understand how it applies to their jobs. Training should be provided not only for the test profiles, but for everyone who will be confronted directly or indirectly with the changes.

test methods in practice and deliver a proof of concept in the pursuit of winning over resistance. Pilot projects can be new projects or projects in an early stage, low-risk and staffed by enthusiastic people who are eager to learn from a project leader who supports the change process. Fine-tune the test handbook based on the feedback from the pilot projects. For this purpose, its a good idea to convene a forum as a mechanism for receiving suggestions, elaborating on possible enhancements, and discussing implementation issues. The new test approach might accommodate customization of the current usage of test tools or the selection and implementation of new tools. Its advisable to define these activities as separate projects with clear objectives in terms of results and timing.

Manage the Change Process

We recommend appointing a change manager to head the change process, with the implementation plan as the blueprint. This individual acts as the project leader for the transition period. He or she performs planning and follow-up, monitors implementation activities, and takes the lead in the various phases of the implementation. The change manager needs excellent social, negotiating, and management skills, as well as in-depth knowledge of the organization and solid experience in change processes. Experience has taught us that a change manager may spend up to 30% of his or her time explaining, persuading, and influencing. These are crucial activities, since the success of the implementation depends largely on the support of all layers of the organization.

COMPANY: MAJOR GLOBAL AUTOMOTIVE MANUFACTURER

S T U D Y
A SPECIAL CLIENT REPORT

BACKGROUND: In pursuit of optimizing the efficiency of its processes, the company decided to implement a comprehensive Quality Management System. Recognizing that its test processes routinely failed in most of its projects, the company asked CTG to design and implement a more effective test program. SOLUTION: CTG defined generic procedures, templates, and guidelines for the companys testing process, along with additional guidelines for specific situations such as data warehouse testing, testing in maintenance projects, and web testing. CTG also investigated the manufacturers test automation options and proposed a short list of vendors and tools relevant to its environment. CTG test specialists defined an implementation plan describing the approach and the planning for the implementation of the new test approach, and a CTG test coordinator managed the set-up of the new test practices and coordinated the implementation process.

C A S E
7

Confidential Information: Special Report and Analysis for CTG Client Use Only

COMPANY: EUROPEAN TELECOMMUNICATIONS FIRM

S T U D Y
Experience has taught us that a change manager may spend up to 30% of his or her time explaining, persuading, and influencing. These are crucial activities, since the success of the implementation depends largely on the support of all layers of the organization.

BACKGROUND: The Mobile Number Portability (MNP) regulation was enacted by the European Commission to allow customers of Belgian telecom operators to switch to a new operator without changing their current numbers. For CTGs client, MNP necessitated the creation of complex new systems and drastic changes to existing legacy applications. In view of the strategic importance and complexity of this project, the company asked CTG to develop a test strategy and then to set up and manage the MNP test practices. SOLUTION: After defining a master test strategy for the MNP project, CTG implemented a hybrid test organization with a test manager, test coordinators, and a wide variety of test staff in the various project groups. As a result of CTGs structured test approach and professional test management, our client successfully met the strict MNP deadline without major difficulties or reductions in service quality. The test approach for MNP was defined in such a way that our client is able to use it as a template for setting up test activities for new projects.

Conclusion
Many IT projects fail due to the lack of a structured and well-managed test process. Project schedule delays, extra resource costs, poor quality, user frustration, and company image loss all too frequently lead to an ROI below expectations. An effective way to address those issues is to implement a standardized test approach carried out by a well-managed test organization. Because specific test processes are unique to every organization, its important to start with an assessment of current test process and identification of the practices that have already proven useful. These should be incorporated into a test handbook that provides a set of procedures, guidelines, and templates that describe how testing ought to be done in your organization, in an easy-to-use and practical format that organizes your companys test processes by workflows. The best way to set up the test organization may be to implement a hybrid organization that encompasses distributed test groups with a high level of business expertise supported by a centralized test department in charge of test coordination, test tools, and test training. A central role in such a hybrid test organization is that of the test coordinator, responsible for test planning, test management, and training of project teams.

A SPECIAL CLIENT REPORT

Confidential Information: Special Report and Analysis for CTG Client Use Only

C A S E

It is crucial to formulate a strategy for a transition from the current test practices to a consistent and efficient test process accepted and used across the organization. This strategy is described in the implementation plan, which features awareness creation, communication, training, pilot projects, fine-tuning the methodology, and test automation activities. A change manager should be appointed to lead the implementation activities and mobilize support for the new process across the organization. The key to the successful creation of test practices involves taking a pragmatic approach. Such an approach builds on current best practices, does not overregulate, implements in small steps, and welcomes and incorporates feedback and suggestions from the field.

Many IT projects fail due to the lack of a structured and wellmanaged test process. Project schedule delays, extra resource costs, poor quality, user frustration, and company image loss all too frequently lead to an ROI below expectations. An effective way to address those issues is to implement a standardized test approach carried out by a well-managed test organization.

A SPECIAL CLIENT REPORT

Confidential Information: Special Report and Analysis for CTG Client Use Only

For more information on CTGs Testing Services, please contact us: CTG 800 Delaware Avenue Buffalo, NY 14209 Phone: 716/882-8000 Fax: 716/887-7464
Backed by 39 years experience, CTG provides IT staffing, application management outsourcing, consulting, and software development and integration solutions to help Global 2000 clients focus on their core businesses and use IT as a competitive advantage to excel in their markets. CTG combines in-depth understanding of our clients businesses with a full range of integrated services and proprietary ISO 9001:2000-certified service methodologies. Our 3,500 IT professionals based in an international network of offices in North America and Europe have a proven track record of delivering solutions that work. More information about CTG is available on the Web at www.ctg.com.

A SPECIAL CLIENT REPORT

Confidential Information: Special Report and Analysis for CTG Client Use Only

10