Flipping The Triangle

Flipping the Triangle
Paths to Best, Least-Cost Automated Testing

Patrick Wilson-Welsh Lisa Crispin
Thursday, August 7, 2008
Part 1; Part 2.
Part I
A Triangle of Bricks, Sticks, and Straw
brief history of manual regression testing
Our experiences of relying solely on manual regression testing to detect and then x defects have not been good. Thats why were all in this room together. There are several bad things that happen when most of your testing e!ort is manual, and especially when all of your regression testing is manual. But we can sum it up in the last chapter of any release:
were deploying to prod this afternoon

Thursday, August 7, 2008 4
The hidden phase that comes after the nal delivery phase, after deployment: the emergency debug and x phase, and blamestorming phase. We are here to talk with you about how a transition to a healthy automated testing design can prevent exploding deployments and related headaches.
It takes 4 days to run the smoke tests; Ill get back to you in 4 days.
And from the testers perspective, manual regression testing is Sisyphian. There are never enough resources for perfect repeatability, total coverage, and it gobbles all the time there is. There is then no time for important activities like Exploratory Testing.
3 kinds of automated tests

So we are here to talk test automation, and three little kinds of it. Brick tests, Stick tests, and Straw tests. And we are here to talk about the best mix to have, and how it takes time to get to that best mix.
brief survey: who here has tested what? who wants to learn what?
But rst of all, what testing experiences to people here have? Who in the room is a professional tester? What kinds of testing, manual and automated, have you been trying? Who here is a programmer? What kinds of manual and automated testing have you been trying? How much time do you spend in the debugger?
Cohns ideal triangle

Here is our goal state, though the heights are not to scale! Mike Cohn, smart agilist, devised this notion of a testing pyramid or triangle that places emphasis, and allocates resources, in certain ways.
TCO
% of automated testing work
At the top of the triangle, we have the tests that are most expensive to maintain. At the bottom, we have the tests that are least expensive to maintain. The width corresponds to how much of our total testing resources budget we want to dedicate to the di!erent kinds of tests. So what are they, exactly?
GUI
end-2-end, integration, story, example, acceptance unit/micro/isolation
10
At the top, our straw tests are those that test the system as a black box, through its GUI. In this presentation, well mainly be talking about automated testing solutions for web applications, not rich client apps, BTW. At the bottom, we have programmer tests: unit tests that isolate bits of behavior. And in the middle, our stick tests do not go through the GUI, but typically test much more of the system than our unit tests do.
good: least investment
good: most investment
11
In our ideal triangle, again, we want most of our time and money for automated tests going into the unit tests. And we want the least of it going into the through-the-GUI tests.
least rework & waste; lowest TCO

In theory, this gives us the lowest Total Cost of Ownership, because it gives us the least amount of rework and waste. It detects the most bugs the cheapest, it drives the best software design, it veries features most e"ciently, and it is least expensive to extend and maintain. So lets explore these contentions.
tales of straw
13
Lets talk a bit about through-the-GUI tests
Watir, Selenium, Canoo... commercial products...

There are several open-source and commercial products for recording through-the-GUI tests. Here, well be talking about the ones for web applications, not rich client applications (though such tools obviously do exist).
Exercise: break into discussion groups, come up with one good story about GUI-style tests.
Lets have folks break into groups of 4 or 5, and spend 8 minutes nding the most interesting, illustrative, or compelling story about GUI tests in the group. Who in each group has the best story about success or failure, joy or pain? Each group pick one person to tell the best story from that group. Then write a 17-syllable headline for your story. Make it compelling. Notice, BTW, that a haiku has 17 syllables!
story time
16
Lets now have each groups elected member read us their storys headline. Well pick the best one or two to dig into.
Lisa: tales of Canoo tests.
17
Lisa Crispin also has a story to tell about settling on Canoo, after some experimentation, to test a web app...
black box, real-world, functional, slow
18
So here are the kinds of characteristics that Patrick and Lisa nd inherent to GUI tests. They tend to use the entire system as a black box, mimicking real-world behavior, talking to all of the real code and external resources and dependencies of the system. They tend to focus on large chunks of behavior. And they tend to be slow
big. easy to learn to write. also brittle and slow. also brittle.
These tests also tend to be big. They tend to be fairly easy to learn to write, at rst. They tend to be slow, again. And they break a lot. It can consume ungodly amounts of time keeping them running, since, in a web app for example, the tiniest tweak to the GUI can break a test.
brief Selenium IDE demo
20
Writing a test on the y for a survey on the http://nerdtests.com/ft_nt2.php site. Selenium is a fairly mature open source app for recording tests for web apps, and organizing them into suites. They can also be exported to a number of di!erent languages, and run in at least three di!erent ways. By far the best introduction to Selenium is the Firefox Selenium IDE plugin.
tales of sticks
Now lets focus on the middle layer in our triangle; the tests that are larger than unit tests, but do NOT go through the applications GUI. What are they like?
Fit, xUnit, FitNesse, ZiBreve, Concordian...

So there is a growing handful of open source and commercial tools that t this small but growing automated testing niche.
Exercise: break into discussion groups, come up with one good story about non-GUI, e2e tests.
So again, lets nd some good stories. First of all, who here as done any of this kind of testing, whether through jUnit, Fit, nUnit, or whatever? Lets break into groups again and pick the best stories about this kind of testing, and write 17-syllable headlines. Haiku optional, but encouraged.
story time
24
Lets here a few headlines...
bypass GUI, less brittle
25
So what Lisa and Patrick have found to be true about this layer of tests is that partly since they bypass the GUI, then tend to be less brittle. They are certainly more brittle than really good unit tests, however.
functional, acceptance, real-worldish, slow, biggish
26
They do tend to be about chunks of behavior as large as a feature, more than about smaller isolated behavior. They again tend to use real external resources, real-ish data. They again tend to be large, and to run slowly. By this we mean many minutes, as opposed to a few seconds.
about building the right thing, more than building the thing right
27
One oft-repeated organizing principle about these kinds of tests is that they are best at verifying that we are building the right thing, much more than they are good at verifying that we are building the thing right.
brief FitNesse demo
28
The original standard web app IDE for automated acceptance testing is FitNesse, built on top of the Fit framework. Lets briey explore a couple of t test pages.
tales of brick
Ah, the good stu!.
xUnit, TestNG, MbUnit...
30
These are a few of the usual suspects for true unit testing, isolation testing, programmer testing, micro testing.
Exercise: break into discussion groups, come up with one good story about xUnit tests.
Once more, lets hear some stories of pain and joy, failure and success with unit tests. Break into groups again, and write some haiku.
story time
32
fast, small, pinpoint bugs, less brittle, hard to learn, single-most valuable software artifact per system...
So Patrick and Lisa have found that well-written unit tests, let me repeat that, WELL-WRITTEN unit tests, tend to run really fast (on the order of 10 to 100 per second) because they run entirely in memory. They tend to pinpoint bugs really well. Then tend to be hard to break. Then are hard to learn to write truly well. And they are the single most important thing you can learn to automate. Without a solid suite of wellwritten unit tests, its hard to nd a software team is that is not basically screwed.
brief jUnit demo
34
Lets look at a few unit tests for a small project, and talk a bit about what it means for them to be well written.
GUI
so we want this
OK. So this is where we want to be. This, again, is our ideal state.
GUI
but we dont start here

But virtually no teams that are new to agility, new to unit testing, new to automated GUI testing, new to Refactoring -- virtually none of these teams get to start here.
were deploying to prod this afternoon

Most of us started out, basically, here, without much test automation.
bad: most investment
bad: least investment
we often start here

And our initial automate testing triangle actually looks more like this. We are throwing too much time and money into through-the-GUI automated tests. And we are throwing far too little time, money, skill, and discipline into really good unit tests.
good: easy to learn
good: hard to learn
for good reason!

And we should forgive ourselves for this, because the ideal triangle is hard to achieve.
big. easy to learn to write. also brittle and slow. also brittle.
We often start focusing on through-the-GUI automated tests because they are easier, at rst, to learn to write, because our programmers dont do testing, because we have dedicated manual testers who at rst seem like the logical test automators. We tend to fall naturally into this pattern: starting with the through-the-GUI tests.
fast, small, pinpoint bugs, less brittle, hard to learn...
41
We tend not to start with unit tests, because they are hard to learn to do well, because programmers are afraid of them, or because programmers feel that they do not have the time or permission to write them. I still sometimes hear software professionals claim that these tests are not worth the e!ort, that they are not valuable enough.
SRP, small modules, mocking/faking, dependency injection, Refactoring, legacy code rescue, TDD, OO, BDD, CI, Design Patterns...
To be fair, here are some of the things a programmer must learn to do well, in order to produce high-value, low-maintenance-cost unit test suites. This is quite a bit to learn.
Kent Beck, Martin Fowler, Bob Martin, Gerard Meszaros, Dave Astels, Michael Feathers, Lasse Koskela, GoF, etc, etc
There are several books, written by several very smart guys, that one must have read in order to claim to be conversant with the state of the unit testing art. Then on top of that, one must write several thousands of such tests before truly becoming skillful at it.
hard
This is a hell of a lot of work. Unit testing is really, really, hard for most teams to learn. This is especially true when they are already being slogged about by enormous, nasty, untested legacy codebases.
might not yield earliest win

So, starting out with a massive management initiative dictating that all code shall be written using TDD, and we shall achieve and maintain 95% test coverage with our unit tests? That might not work the way you would hope.
So, what automated testing choice should the team make today?
46
So, given these patterns and trends, how, for any team, in their particular context, in their place along the path toward the ideal triangle, do they decide to do on a particular day, iteration, release.
Q/A
Questions on material so far?
30-Minute Break
48
So, given these patterns and trends, how, for any team, in their particular context, in their place along the path toward the ideal triangle, do they decide to do on a particular day, iteration, release.
Part II
Flipping the Triangle
49
First, an administrative note. We had originally planned to make this a three-hour session, but discovered during rehearsals that that was an absolutely cruel plan. So instead this will be a 90-minute session, repeated again starting two hours from now, which will be 4:00 PM. Please, if you enjoy this session, Twitter and IM and text all of your friends who are not here, and tell them they MUST attend the second session.
How about some principles?
50
Lets start with a few principles for this journey from upside down triangle to right-side-up triangle.
1. Follow the pain.
51
This is true for most agile transitions. Is it really production defects blowing up badly that you want to focus on? And are you constrained in the unit tests you can create just yet? And are all or most of your tests currently manual? Then it might, in fact, make sense to start with GUI test suites.
2. Early wins; low-hanging fruit.
52
For such teams, GUI tests are often the lowest hanging fruit, a logical starting place (though a dreadful ending place).
3. The Whole Team owns the transition.
53
Let there not be at testing initiatives from above. No THou Shalt Do TDD or Else directives, nor coverage rate directives. At least, not without the buy-in and input from the entire team. Let the team self-organize -- testers, programmers, BAs, managers -- around where the biggest pain points are, where the low hanging fruit is, and which steps to take when, while still nding a way to meet ridiculous production schedules.
4. Pair programmers with testers/QA persons.

The parable of the overloaded tester.
54
There should be no such thing as a group of programmers blocked by testers who cannot test code fast enough. Thats just a version of thats not my job. Here is what you get when any role or responsibility on the team says thats not my job.
Moving sticks? Not my job, man.

5. Pair programmers with testers/QA persons.

Testing is everybodys job.
56
6. Three initiatives: straw, sticks, and bricks.
57
The team needs to look at these three initiatives as separate. We need separate plans for enough bricks, enough sticks, and just enough straw. We need plans for how to increase the bricks as we scale back the straw.
7. Learn to make bricks, no matter what.
58
And no matter what happens, if you dont end up with an absolutely great suite of unit tests -- on the typical non-trivial project, this means thousands of tests than run in a few seconds -- then you wont ever get your automated test costs down to least TCO levels. And your code maintain costs, turnover costs, and customer satisfaction costs will all be higher as well.
8. Earn testers freedom to do more Exploratory Testing.
59
What are the best testers really good at? They are good at sni"ng out bugs where others would never think to nd them. This is instinctual, blink skill (from Malcolm Gladwells book on deep expertise). Use a great automated testing strategy to buy your testers time to do more and more and more exploratory testing (ET) per iteration.
9. There is such a thing as too little courage, skill, trust, respect
60
how do teams learn new skills and tools?

We are talking about culture change here, which involves a group of people unlearning a bunch of habits and skills, and learning a bunch of new habits and skills.
so, not like this

All you guys are gonna to TDD from now on, and youre all gonna sustain test coverage rates of 85% or higher! How does that work for you guys?
The fatal pedagogical error is to throw answers like stones at the heads of those who have not yet asked the questions. -- Paul Tillich
63
If I am not asking the question you are answering, then your answer sounds to me like a Bowex commercial in the middle of my late night movie. I wish you would just shut up and go away, and let me go back to my code in my cubicle.
so, who enables the questions?

So where do the right questions come from? How do they arise?
natural leaders
[denition]
How can I enable the right questions?

continuous learning courageous experimentation socializing over evangelizing community-building encouragement, inspiration
leading by trust & example

what if we started the GUI
tests with small smoke tests?
what if we started with a Fit

test for 1 critical feature per iteration? no longer than 10 lines?
what if all new methods were which parts of the codebase
most deserve legacy rescue, and how much?

Exercise: break into discussion groups, come up with one good story about objections team members have to trying new initiatives in the straw, stick, or brick areas.
So again, lets nd some good stories.
story time: how do we meet the objections

How can the team work together, how can natural leaders work, to en-courage people, to get folks to try small courageous experiments? Lets get some ideas out.
time permitting: micro Open Space
71
This crowd has a lot of natural wisdom and hard-won experience around automated testing, I bet. Are there a half-dozen topics we could have 30-minute Open Space talks around? [Briey go over Open Space rules/guidelines...]
Open Space rules
72
Whoever shows up is the right group.
73
Whatever unfolds is the only thing that could have.
74
Whenever it starts is the right time.
75
When its over, its over.
76
The Law of Two Feet.
77
* If you feel that you are not contributing or beneting from a presentation, please feel free to move on to something else. * Allow the discussion convener to steer his or her topic.
Getting Started
78
* Come up with an interesting topic and title for your discussion, related to what we have covered. * Write it on a sticky, announce it, and put it over here. * If you see topics that have something in common, consider combining them into a single time slot. * just drop in to the one that most resonates with you. * Use the law of two feet.
bad: most investment
bad: least investment
get from here

GUI
to here
GUI
hard; worthwhile
Agile Testing:
a Practical Guide for Agile Testers and Teams
-- Lisa Crispin and Janet Gregory
82
Agile Testing:
a Practical Guide for Agile Testers and Teams
--Lisa Crispin & Janet Gregory
Fit for Software Development

xUnit Test Patterns
--Rick Mugridge & Ward Cunningham

--Gerard Meszaros
Test-Driven Development: By Example Refactoring
--Kent Beck
--Martin Fowler
Working Effectively With Legacy Code Clean Code: a Handbook of Agile Software Craftsmanship
-- Michael Feathers
83
Q/A
Questions?

Flipping The Triangle

Încărcat de

Informații document

Titlu original

Drepturi de autor

Formate disponibile

Partajați acest document

Partajați sau inserați document

Opțiuni de partajare

Vi se pare util acest document?

Este necorespunzător acest conținut?

Drepturi de autor:

Formate disponibile

Flipping The Triangle

Încărcat de

Drepturi de autor:

Formate disponibile

Flipping the Triangle

Paths to Best, Least-Cost Automated Testing

Thursday, August 7, 2008

Thursday, August 7, 2008

brief history of manual regression testing

Thursday, August 7, 2008

were deploying to prod this afternoon

3 kinds of automated tests

Thursday, August 7, 2008

Cohns ideal triangle

% of automated testing work

Thursday, August 7, 2008

end-2-end, integration, story, example, acceptance unit/micro/isolation

Thursday, August 7, 2008

good: least investment

good: most investment

Thursday, August 7, 2008

least rework & waste; lowest TCO

Thursday, August 7, 2008

Lets talk a bit about through-the-GUI tests

Watir, Selenium, Canoo... commercial products...

Thursday, August 7, 2008

Lisa: tales of Canoo tests.

Thursday, August 7, 2008

black box, real-world, functional, slow

Thursday, August 7, 2008

brief Selenium IDE demo

Thursday, August 7, 2008

Fit, xUnit, FitNesse, ZiBreve, Concordian...

Thursday, August 7, 2008

Lets here a few headlines...

bypass GUI, less brittle

Thursday, August 7, 2008

functional, acceptance, real-worldish, slow, biggish

Thursday, August 7, 2008

Thursday, August 7, 2008

brief FitNesse demo

Thursday, August 7, 2008

Ah, the good stu!.

xUnit, TestNG, MbUnit...

Thursday, August 7, 2008

Thursday, August 7, 2008

brief jUnit demo

Thursday, August 7, 2008

end-2-end, integration, story, example, acceptance unit/micro/isolation

end-2-end, integration, story, example, acceptance unit/micro/isolation

but we dont start here

were deploying to prod this afternoon

Most of us started out, basically, here, without much test automation.

bad: most investment

bad: least investment

we often start here

good: easy to learn

good: hard to learn

for good reason!

fast, small, pinpoint bugs, less brittle, hard to learn...

Thursday, August 7, 2008

might not yield earliest win

Thursday, August 7, 2008

Questions on material so far?

Thursday, August 7, 2008

Thursday, August 7, 2008

How about some principles?