Documente Academic
Documente Profesional
Documente Cultură
Software Testing
Community
EuroSTAR
Paul Herzlich Research Analyst, Creative Intellect Consulting
Creative Intellect Consulting UK is constantly monitoring adoption of software testing tools and best practices. In this paper we examine the benefits and challenges of early lifecycle testing for a cluster of technical test types, commonly known as performance testing. The aim is to bring managers up to date on the latest thinking on the subject and practice in the field, with a view to cutting away the myths and removing obstacles blocking adoption.
Summary Messages
Early Testing Reserved for Functional Testing
Early testing delivers business benefits by enabling IT organizations to deliver more at lower cost. However, organizations have concentrated on introducing early functional testing and have largely ignored the potential of early performance testing.
However, to most testers and QA managers, there are two aspects of performance testing that render it unsuitable for early testing. The beliefs are: You can only test performance on a fully built system which doesnt exist early on. You dont have suitable component level performance requirements. The solutions to these objections require a finer look at what performance testing consists of and a pragmatic approach to achieving your goals without necessarily achieving textbook perfection.
1 PAGE
Practitioners believe there are two unsolvable challenges to early performance testing
There are a whole host of potential difficulties to introducing early performance testing. Most of them are the same as those that impede the adoption of early functional testing. They are largely soft, people issues, centered around the roles and psychology of software coders. There are fairly well understood change management techniques for dealing with them.
2 PAGE
Both of these are extremely important in a commercial environment where business success depends on IT to minimize costs and meet changing market demands quickly. Businesses cannot afford cost overruns and delayed product releases. The days when an estimated 60% of all projects get canned usually for not delivering the right systems at the right time are over. Project budgets must be controlled, timescales managed and uncertainty kept to a minimum.
art 1:
3 PAGE
Chart 1: Relative cost of fixing a coding bug at later lifecycle stages For bugs in the initial requirements the picture is equally as dramatic. This is worthy of attention since most bugs emanate from badly captured or ill-defined requirements. If the cost of finding a requirements bug during the requirements capture stage is x, it costs 5x in coding and unit test stages, 10x in later test stages, 15x in pre-production and 30x in the released system. If youre skeptical, think about the processes required to locate and fix a bug. When a bug is found in a stage later than coding and unit test, it has to be documented and returned to a developer. A developer has to establish an environment to reproduce the bug. This may entail re-creating test data and restoring an earlier version of source code. Locating the bug the error that is the source of a fault is complicated by the fact youre now dealing with a complex assembly of components rather than simple units. Attending to a bug report comes with an immediate disruption to the developers other work, which in turn has a cost to activities wholly unrelated to this project. When the bug is fixed, code must be promoted and retested at later stages, with all the headaches that entails for source code and build management.
4 PAGE
Early testing contributes to greater predictability by providing better information about the true state of a project in the early stages. As work is passed from coding to integration or system testing (or iteration to iteration), there is a great deal more certainty about the codes readiness when it has been unit tested by developers (real unit testing, not the mythical or ad hoc testing developers have historically performed.) A manager has a great deal more confidence that coding is truly complete for 10 programs handed over with a full suite of passed unit tests than 10 programs without documented tests. This confidence works not only at handovers within a project, but also outwards from the project. Without completion evidenced by passed tests, all but the most experienced managers fall into the 95% complete syndrome, and pass that misinformation to their managers, the sponsor and end users. Such self-delusion means that everyone is in for an unwelcome surprise not just a surprise about the software quality but also about the project progress.
5 PAGE
Performance testing is commonly used as a term for several related test types whose goals are to establish that an end user will get a functionally correct response from an application and that the timing of the response will be acceptable. Some of the constituent test types are load, volume, stress, concurrency and soak. We need to look more closely at the individual test types to understand which are most suitable for early testing. Straddling the functional and performance domains is concurrency testing. Running more than one instance of a process concurrently can raise functional problems like inconsistencies from simultaneous database updates and performance issues like badly formed locks on resources, which slow things down. In the narrow sense, performance testing is a measurement of response time or latency and the resources consumed to achieve it. To
6 PAGE
Chart 2: Relationships among various performance-family test types to create a background of infrastructure activity disk access, CPU, network traffic that represents a particular normal loading of the production environment. Then user transactions are driven manually and performance is measured of the manually entered workload. When we look at how to do early performance testing, we need to keep in mind all the various test types available and not get locked into the image of the set-piece, largescale performance test. test and QA managers plus specialist test service vendors? History and tradition in software testing practice, and more widely in development, have a lot to answer for. The Waterfall model for development and, to a lesser extent, the V model widely known among testers, both convey a picture in which technical tests such as performance are relegated to late in the lifecycle. That is, at least, the most common interpretation, even if it was not strictly the intention. Testing tool vendors have also contributed. They have bought into the late cycle testing vision for commercial reasons and, in turn, the available tools have shaped (or misshaped) current practice. We alluded earlier to the soft issues surrounding people and process that are used to undermine early functional testing. Those issues are still present for performance testing: developers dont like testing and are generally not accountable for it. Testing is
7 PAGE
However, adoption of Agile does not totally solve the problem. Early performance testing poses two key intellectual challenges that experienced developers on both Agile and Waterfall projects have trouble seeing their way through. Challenge 1: You can only test performance on a fully built system in a production-like environment. Challenge 2: It is impossible to set and you almost never have meaningful performance requirements at the level of individual components. It takes a highly motivated and creative test or QA practitioner to see how these challenges can be met.
8 PAGE
Chart 3: Test types by suitability for early performance testing Stress tests are unlikely candidates for early performance testing. Load (as opposed to background load) and soak are possible primarily as part of continuous integration. The others are good candidates for developers alongside functional unit testing. components is impossible when they dont have performance requirements at the individual component, method or unit level. The obvious solution is to analyse the highlevel performance requirements and create requirements for components. If no one in a team not even an architect or designer is capable of doing that, you have to conclude there is no certainty that the built application will perform adequately when the components are assembled into a whole application. In which case, the need for more granular performance requirements and early testing is even greater.
9 PAGE
10 PAGE
You need to convey two distinct messages: that the benefits of early cycle performance testing are worth it and that it is feasible to do. The starting point has to be a clear explanation of how finding defects early has both a financial benefit to the business and improves project manageability and predictability. It must be made clear that project delivery schedules will be adjusted to take into account the additional up front work, and that any learning curve is built-in. Whats in it for those who will adopt the practice? It depends on your organisations culture. The certain benefit is that for an additional effort in development, there will be far fewer awkward disruptions later for debugging and rework. Beyond the benefits, developers need to be reassured of the feasibility. For this you will need to ensure that the change management project includes a skilled champion or evangelist someone who
We recommend that your process include risk-based analysis for deciding whether to do early performance testing on a component, how much and what type (based on the types we outlined earlier.) There are several existing processes that are likely to be touched by early performance testing:
Tools
Performance testing of all types requires tools. For early performance testing, you need tools that fit in with coding and integration environments and that dont require all the scaffolding and preparation of the set piece late cycle performance test. They need to support middleware and backend components as easily as they support client-end scripts. One commonly held idea is that functional unit tests can form the basis of performance tests. More often than not, this proves not to be true, so dont rely on it. It is worth pointing out that for background load testing where you are just trying to create a background of machine loading against which you will run manual transactions you dont need as fine a control of the timing and ramp up of automated scripts as you need for full load testing. Virtual user scripts and orchestration by a virtual user controller can be relatively simple. You are not concerned about the performance of the individual virtual user transactions; you are interested in the load they place overall on resources. A key factor in the cost-benefit equation of early performance testing is the cost of tools. When they are available in developers IDEs the marginal cost is low. However, the cost of virtual users for driving load tests can become a significant expense. It is important to make a reasonable stab at estimating how many are really required. On one hand, there is a tendency to over-
11 PAGE
Specialists
You will need the buy-in of those who are usually involved in late cycle performance testing, including any specialist performance testers, architects, infrastructure experts (network, DB, service orchestration). They will need to change their own processes to include support for the early performance testing. This is sometimes viewed as an opportunity because they have been at the sharp end of the problems inherent in leaving all the testing towards the end. However, as
Conclusion
Early performance testing may not be the first test process improvement a development organization chooses to make, but there is no good reason why it should be as overlooked as it is. Agile processes and powerful development environments with integrated testing capabilities are conducive to a pragmatic approach. Determination to obtain the benefits of lower rework costs and more manageable projects provides the motivation.
12 PAGE
Microsofts ALM Assessment has been designed by expert ALM practitioners to help you understand how well your organisation currently approaches ALM and where and how you can make improvements. To find out more, click here.
EuroSTAR
Software Testing
C o n fe re n c e
Amsterdam from the 5 8 November. Its our 20th anniversary and we are going to
EuroSTAR
celebrate in style!
Software Testing
Community
EuroSTAR
Become a fan of EuroSTAR on Facebook Join our LinkedIn Group Contribute to the EuroSTAR Blog Check out our free Webinar Archive Download our latest ebooks
w w w. e u r o s t a r c o n f e r e n c e s . c o m