James A.

Whittaker +docjamesw (Google+) @docjamesw (Twitter)

It's a book! It's a blog! It's a way of life! Follow me on

Google+ (+docjamesw) Twitter (@docjamesw)

A few words about Google

Engineering culture

Fail fast
20% time Project mobility

The Google testing philosophy

Quality Test Google is an engineering/computer science centric company Quality must be owned by engineering Test must be part of engineering

Testing won't get done unless it is unavoidable Test is everyone's job Actual testers are rare, scarcity creates clarity
The result The SET role Engineering Productivity not Testing

Engineering Productivity
A separate organization beholden to Google, not to Product Areas
"To accelerate Google" Tools from IDEs, compilers, build, test Testing, release engineering, engineering education

Product area leads negotiate with Eng Prod leads for test engineering Testers are assigned to product areas but report through Eng Prod Testers responsible for their own fate: hiring, promotion, reviews

Testing roles
SWE Feature developers code reviews, TDD, unit tests, acceptance tests own quality for every piece of code they touch

SET Test infrastructure developers code reviews, test frameworks, automation TE

Test scenario developers risk analysis, end-to-end test scripts, quality management

Testing roles

Crawl, walk, run

"Minimum useful product" Don't build a large product, start small and grow it Release the moment you have something useful

Release and iterate Make gathering feedback part of the process Don't be afraid to take chances Fail fast, correct as you go

Release channels Canary (daily build) Developer (weekly build) Test (best weekly build) --> Dogfood Beta (latest, most stable test channel build) --> Release

Types of tests
Google is a melting pot of testers from various companies, all using a different term for a different type of test We standardized:

Small tests Medium tests Large tests Enormous tests

Types of tests
Small tests

Almost always automated Run in a mocked/faked environment Exercise code within a single function or module Focus on functional issues such as data validity and error conditions Execute within seconds Mostly written by SWEs
Does this code do what it is supposed to do?

Types of tests
Medium tests

Usually automated Run in a fake or real environment Test the interaction between functions that call each other directly Focus on functional issues such as data validity and error conditions Execute within a few seconds Written by SETs and SWEs
Do a set of near neighbor functions interoperate the way they are supposed to?

Types of tests
Large tests

Can be automated or manual Run in an actual production environment Test end-to-end functionality Can take minutes to hours to run Written by TEs and SETs
Does the product operate the way a user would expect and produce the desired results. Given all this, an enormous test should be self explanatory!

Automation vs manual
Automate when

Results can be checked by a machine No human judgement or cleverness is required Repetitive So important it can't be left to chance
Manual otherwise

Shared code
The secret?
"Committers" Similar to the open source idea of a person with "commit" status on a project At Google committers have language-specific "readabilities"

Code reviews All code must be reviewed and approved by a person with readability You earn your readability by a track record of writing good code

Platform dependencies
Don't deal with them, minimize them All engineers (dev and test) use the same OS on their desktop as we use in our data centers Local = Test = Staging = Production

Platform libraries are baked into OS images

Common build files for C++, Java and Python

Test Certified
Level 1 Set up test coverage bundles Set up a continuous build Classify your tests as Small, Medium, and Large Identify nondeterministic tests Create a smoke test suite Level 2 No releases with red tests Require smoke test suite to pass before a submit Incremental coverage by all tests >= 50% Incremental coverage by small tests >= 10% At least one feature tested by an integration test Level 3 Require tests for all nontrivial changes Incremental coverage by small tests >= 50% New significant features are tested by integration tests

Level 4 Automate running of smoke tests before submitting new code Smoke tests should take less than 30 minutes to run No nondeterministic tests Total test coverage should be at least 40% Test coverage from small tests alone should be at least 25% All significant features are tested by integration tests Level 5 Add a test for each non-trivial bug fix Actively use available analysis tools Total test coverage should be at least 60% Test coverage from small tests alone should be at least 40%

A user facing test role

SWE - feature developer coding on behalf of the product

SET - test developer coding on behalf of the developers TE - user developer coding on behalf of the user TEs must be engineers to remain first class citizens

TEs progress up the job ladder, some become product managers, some become UX designers, some SETs, some SWEs ...

The 10 minute test plan

Put 3 engineers in a room, hand them a spec and give them 10 minutes to build a test plan Interrupt them after ten minutes, tell them time is up and give them a new problem and a fresh 10 minutes Rinse and repeat until they figure it out

The 10 minute test plan

What we learned Avoid prose in favor of bulleted lists Don't bother selling No fluff, this is not a high school term paper If it isn't actionable, leave it out Make it flow, one section leads to another Guide a tester's thinking The outcome should be test cases

A test plan that doesn't lead to test cases is a waste of time

ACC

Attributes Components Capabilities

Simple and powerful model for analyzing web apps Think of the product in terms of

Adjectives and adverbs (attributes) Nouns (components)

Verbs (capabilities)

ACC: Attributes

Adjectives and adverbs Ask yourself o Why are we building this? o Why is this interesting to users? Examples from Chrome o Fast o Secure o Stable

o Elegant

Sources for attributes: requirements, advertising, packaging

ACC: Attributes are easy to discover

ACC: Attributes are easy to discover

ACC: Components

Nouns Ask yourself

What are the major parts of the product?

Examples from Chrome URL bar Tabs Chrome Webkit

Sources for components: design docs, areas in bug databases, area

owners, common sense

ACC: Capabilities

Verbs Ask yourself

What does this product do?

Examples from Chrome Load webpages Sync preferences Run web apps

Key for capabilities: intersection of attributes and components

If the product does something that does not match an

attribute/components intersection ignore it or take it out

BITE Record & Playback

Tool Walk Through

BITE Bug Filing

Tool Walk Through

BITE Bug Visualization

Tool Walk Through

Quality Bots