Agile Methodology: The Complete Guide to

Understanding Agile Testing

Over the past several years, a new way of creating software has taken the
software development and testing world by storm: Agile.
In fact, according to VersionOne’s State of Agile Report, as of 2018, 97% of
organizations practice agile in some form. However, respondents report that
this adoption is not always widespread within their organizations, which
means there’s still a long way to go in terms of adoption and maturity.

So what exactly is agile and why has it become so popular so

quickly? Let’s explore exactly what agile methodologies entail and how to
introduce it in your organization in more detail. Specifically, we’ll cover:
 How does testing fit into agile methodologies?
 What are the different ways to test on an agile team?
 What’s next for the agile movement?

About Agile Methodology

Agile methodology has taken the software development world by storm and
rapidly cemented its place as “the gold standard.” Agile methodologies all
started based on four core principles as outlined in the Agile Manifesto. These
methodologies are rooted in adaptive planning, early delivery and continuous
improvement, all with an eye toward being able to respond to change quickly
and easily. As a result, it’s no surprise that 88% of respondents in
VersionOne’s 2017 State of Agile Report ranked “ability to adapt to change”
as the number one benefit of embracing agile.
However, as more and more development teams adopt an agile philosophy,
testers have struggled to keep pace. That’s because the widespread adoption
of Agile has led teams to issue releases and totally undocumented software
on a more frequent basis. This frequency has forced testers to shift when they
conduct testing, how they work with developers and BAs and even what tests
they conduct, all while maintaining quality standards.
What Does it Mean to Test on an Agile
Team?
Agile principles are all about being collaborative, flexible and adaptive. It’s
built on the premise that the world now changes regularly, and that means
software teams no longer have years to bring new products to market. In that
time, competitor offerings or customer expectations can change, and the team
risks irrelevance. Agile minimizes this risk by helping teams collaborate
together more by adapting to what the team needs to be successful. It does
this by encouraging teams to regularly show off their work and gather
feedback so that they can adapt to change quickly.
Jump-start your agile collaboration: Read our
article, “The Secret to Alignment Between
Developers and Testers.”
Narrowing in on testing, the fast pace of agile development has led to
several imperatives for testers:
1. Prioritizing requirements based on risk since it’s not possible to test
everything
2. Automating tests to increase efficiency
3. Increasing the use of exploratory testing to accelerate the time from code
delivery to test completion and to emphasize the need to create code
that works
4. Adapting to changes from sprint to sprint
The fourth imperative — adaptability — is particularly important because it
requires testers to have broader, cross-functional testing skills, which
represents a departure from the narrower testing skills often needed in a
waterfall environment. Additionally, unlike in a waterfall environment, testers
following an agile methodology need to stay in close contact with developers
to collaborate on testing throughout the entire software development lifecycle.
In waterfall methodologies, there is often a large requirements document that
testers test against. That document doesn’t change frequently, so testers
have been able to exist fairly independently of developers. However, most
Agile methodologies are light on documentation and the requirements for a
new feature may only be in a ticket in a requirements tracking system without
all the edge cases listed. The testers in these scenarios need to be highly
communicative with the development and business teams, as tests they wrote
a few weeks ago may become obsolete quickly. To succeed, testers need to
be flexible and able to adapt to moving targets.

To succeed, testers need to be flexible

and able to adapt to moving targets.
In general, there are four central tenets of the Agile Manifesto that are
important for testers to remember:
1. Individuals and interactions over processes and tools
2. Working software over comprehensive documentation
3. Responding to change over following a plan
4. Collaborating with customers over contract negotiation
The bottom line in all of this is that everyone — testers, developers and
beyond — must evolve in order to embrace an agile style of working.
Agile is Not One-Size-Fits-All
Every organization is unique and faces different internal factors (i.e.
organization size and stakeholders) and external factors (i.e. customers and
regulations). To help meet the varying needs of different organizations, there
are various agile methodologies and several different types of testing you can
do while working within one of those agile methodologies. Which mix is right
for your team will depend on your internal and external factors, needs and
goals. Let’s take a look at what some of the most popular agile
methodologies and testing methods entail, including:
 Agile Methodologies
o Scrum
o Kanban
 Testing Methods
o Behavior Driven Development (BDD)
o Acceptance Test Driven Development (ATDD)
o Exploratory Testing
o Session Based Testing
2 Agile Methodology Types
1) Scrum

What is it? One of the most popular software testing methodologies (used
by 58% of organizations that have embraced agile according to VersionOne),
Scrum takes a highly iterative approach that focuses on defining key features
and objectives prior to each sprint. It is designed to reduce risk while providing
value quickly.
Scrum starts with a requirement or user story that outlines how features
should perform and be tested. The team then cycles through a series of
sprints to provide small bursts of value quickly. To help the team work in this
flexible way and avoid shifting priorities, Scrum requires that questions be
answered from the very start.
How is it different from Waterfall? Whereas Waterfall includes several
testing and bug fixing cycles before releasing a product, Scrum is far more
collaborative and iterative. One of the biggest differences is that Waterfall
calls for heavy documentation early on. This documentation makes it harder to
change features as the process goes on, which may be a negative in some
environments (such as consumer-grade software) and a positive in others
(such as those where the team is trying to launch a rocket, since no one
wants requirements for something dangerous shifting frequently). That said,
you might think of Scrum like many “mini waterfalls,” since requirements are
well defined at the start of each sprint and shouldn’t shift within it. The
difference is that detailed requirements for the next sprint are not set months
in advance.
Diving deeper, Scrum calls for more regular collaboration between testers,
developers and BAs, typically in the form of daily standups and sprint
retrospectives, to ensure proper communication and alignment. Additionally,
there is a Scrum Master who helps keep the project on task by removing
blockers from the team to make sure they can be most effective. The Scrum
Master can be anyone on the team, such as a developer or a tester.
What does adoption entail? Scrum offers one of the easiest transitions for
teams coming from a Waterfall environment because it’s time-based with
sprints and releases can still be planned out in advance. That said, it does call
for faster iterations and stronger collaboration.
Who is it for? Because of its fast iterations, Scrum is best suited for teams
whose customers and stakeholders want to be actively involved by regularly
seeing working products at showcase meetings. This collaboration allows the
team to make changes for upcoming showcases. Key team members who
should be involved when taking a Scrum approach include:
 Product Owner
 Scrum Master
 Developers
 Automation Engineers
 Testers
 Stakeholders
What are best practices? In addition to strong communication, collaboration
and adaptability, other best practices for testers following a Scrum
methodology include:
 Determining acceptance criteria based on communication (typically in the
form of a user story) from a sales representative or customer (note: this
direct connection should help reduce miscommunications)
 Using the acceptance criteria to develop code and ensuring team
approval of that code
 Testing the code in sandbox-like environments as well as production-like
environments prior to deploying it to production
2) Kanban

What is it? Kanban is a very simple Agile based methodology rooted

in manufacturing (it was developed by Toyota to help increase productivity in
factories). At it’s core, Kanban can be thought of as a large, prioritized to-do
list. Like with Scrum, requirements in Kanban are tracked by their current
stage in the process (to-do, in development, in test, done).
Unlike scrum, Kanban is not time-based. Rather, it is based solely on priority.
When a developer is ready for the next task, he/she pulls it from the to-do list.
Since there are fewer planning meetings, this approach means the team
needs to be extremely close. In this type of environment, if developers work
much faster than the testers, bottlenecks will crop up. In these situations,
anyone on the team should jump in and help in different areas. Of course
meeting this need requires a great deal of flexibility and adaptability.
How is it different from Waterfall? Kanban still has requirements like
Waterfall, but the requirements can change as the testing team doesn’t start
to think about testing each requirement until the developer selects it from the
top of the backlog. In contrast, Waterfall is heavily time-based with a lot of
overhead in planning. The heavy planning that comes in a Waterfall
environment is great in some cases, like when building expensive things, but
it’s not always necessary. With Kanban, releases still get planned, but teams
usually don’t promise anyone features by certain dates unless the item in
question is near the top of the backlog.
What does adoption entail? Kanban offers a simple transition for the right
teams. To make a smooth transition to Kanban, business analysts,
developers, testers and stakeholders should sit close together and
communicate regularly. When transitioning to Kanban, it’s important to
remember that this methodology offers the quickest way to bring code to
production, but the code is likely to have some technical debt. That’s because
developing without always knowing what’s next doesn’t necessarily lend itself
to producing the most reusable code.
Who is it for? Kanban is best suited for small teams or teams that don’t
produce features for the public and/or promise certain dates for releases.
Additionally, it is a top methodology of choice for any products or teams
focused primarily on maintenance work since bugs are not always
straightforward and often require research to resolve, which makes time
management challenging. Teams that cannot minimize the amount of
planning for issues are likely better off following a Scrum or Waterfall
methodology.
Key team members who should be involved in a Kanban environment include:
 Product Owner
 Project Manager
 Developers
 Automation Engineers
 Testers
What are best practices? In addition to maintaining visibility and prioritizing
collaboration, best practices for testers following a Kanban methodology
include:
 Keeping very open lines of communication between the business
owners, developers and testers
 Ensuring the team has flexibility to take on other roles outside of their
core responsibilities in order to help clear bottlenecks
 Making everyone an owner of the product so that they care fully about
the result
4 Agile Testing Methods
1) Behavior Driven Development (BDD)

What is it? Many people have heard of or used Test Driven Development
(TDD). For example, developers use TDD to write unit tests to fail before code
has been written. BDD is based on the same principles as TDD, but instead of
unit tests, it calls for higher level tests at the business level. Instead of starting
with a technical-facing unit test as TDD does, BDD starts with an initial
requirement based on end user behavior and calls for tests that are “human
readable” and can even replace some requirements documentation. This
requirement is based on behaviors that the product should exhibit, creating an
airtight guide for engineers to use as they develop tests.
For more information, read our article: “Why BDD
is the Secret Sauce for Testing in DevOps.”
Specifically, BDD starts with a functional specification using the Gherkin
Given/When/Then syntax. This specification then guides developers, testers
and product owners who move across features. As they do, they use
automated test functions to determine completeness, refining code until it
passes the test, much like in the TDD approach except at the team level. To
ensure the test passes (and it typically requires multiple tries), the developer
should only refactor code, not add any new functionality.
In sum, BDD requires a “smart” automation strategy that drives a high level of
efficiency. This strategy sets BDD apart from other Agile methodologies.
How is it different from standard Waterfall testing? BDD is extremely
different from standard Waterfall testing because the former requires test
cases to be written early against requirements and calls for those tests to be
executed toward the end of the development cycle. With BDD in an Agile
environment though, tests are not based on requirements and testing
happens in conjunction with development of the features.
Additionally, within a Waterfall methodology, testers are the ones writing the
test cases. A BDD approach, on the other hand, lends itself to the business
owners writing the tests. This switch reduces communication (or
miscommunication) between the business analysts, developers and testers.
What does adoption entail? Changing to a BDD methodology can be
challenging when the team is used to a traditional style of testing. It requires a
BA or tester to write tests up front and for the developers to write the test
specification in code to match. It’s a new type of coordination within the team,
but it’s extremely positive in that the team works together as one unit,
business users included.
Who is it for? The BDD methodology is ideal for teams that work on feature-
focused software and/or teams that put the user experience first. Key team
members who should be involved in a BDD environment include:
 Product Owner/Business Analyst
 Project Manager
 Developers
 Automation Engineer/Testers
What are best practices? Best practices for testers following a BDD
methodology include:
 Streamlining documentation in order to keep the entire process lean
 Embracing a “three amigos” model where the product owner, developer
and tester form a cohesive team
 Using a test framework like Cucumber to define criteria
  Building automated tests in a way that make them easy to reuse as much
as possible
 Having business analysts learn the Gherkin syntax and write test cases
directly
2) Acceptance Test Driven Development (ATDD)

What is it? ATDD is like BDD in that it requires tests to be created first and
calls for the code to be written to pass those tests. However, unlike in TDD
where the tests are typically technical-facing unit tests, in ATDD the tests are
typically customer-facing acceptance tests.
The idea behind ATDD is that user perception of the product is just as
important as functionality, so this perception should drive product performance
in order to help increase adoption. To bring this idea to life, ATDD collects
input from customers, uses that input to develop acceptance criteria,
translates that criteria into manual or automated acceptance tests and then
develops code against those tests. Like TDD and BDD, ATDD is a test-first
methodology, not a requirements driven process.
Also like the TDD and BDD methodologies, ATDD helps eliminate potential
areas for misunderstanding by removing the need for developers to interpret
how the product will be used. ATDD goes one step further than TDD and BDD
though because it goes directly to the source (aka the customer) to
understand how the product will be used. Ideally, this direct connection should
help minimize the need to re-design features in new releases.
How is it different from standard Waterfall testing? ATDD is different from
standard Waterfall testing because it is a test-first methodology. Standard
Waterfall testing calls for test cases to be written upfront based on
requirements, whereas ATDD is not a requirements driven testing process.
What does adoption entail? Because ATDD represents such a departure
from traditional methods, going from one to the other isn’t easy for teams to
do. To be in the best position to adopt an ATDD methodology, teams need to
get stakeholder buy-in, which can prove challenging at times.
Who is it for? Due to its emphasis on user perception, ATDD is best suited
for teams that are focused on the user experience, have goals around high
rates of adoption and want to minimize the number of feature changes in
future releases. Key team members who should be involved in an ATDD
environment include:
 Customer/Customer Advocate
 Developer
 Product Owner/Business Analyst
 Automation Engineer/Testers
 Project Manager
What are best practices? Best practices for testers following an ATDD Agile
methodology include:
 Interacting closely with customers, for example through focus groups, in
order to determine expectations
 Leaning on customer-facing team members, such as sales
representative, customer service agents and account managers, to
understand customer expectations
 Developing acceptance criteria based on customer expectations
 Prioritizing two questions:
o Will customers use the system if it does X?
o How can we validate if the system does X?
3) Exploratory Testing

What is it? Next we have exploratory testing, which is actually a type of

functional testing but is important in Agile environments. Exploratory testing
gives testers ownership over the code to test it in an organized, chaotic way.
In this case, testers are not following test steps, but rather using the software
in standard or clever ways to try to break it. Testers will document defects as
usual, but detailed documentation of what and how the application was tested
is not always provided.
Exploratory testing is not scripted. Rather, it’s about developing the best tests
based on each unique piece of software. Because of its unscripted approach,
exploratory testing often mimics how users will interact with the software in
real life.
Across the board, exploratory testing follows four key principles:
1. Parallel test planning, test design and test execution
2. Specific yet flexible
3. Aligned toward investigation of potential opportunities
4. Knowledge sharing
How is it different from standard Waterfall testing? Exploratory testing can
actually be done in both Waterfall and Agile environments, but the tight
integration between testers and developers in an Agile environment helps
ease any bottlenecks that might surface while running exploratory testing in a
Waterfall environment.
Furthermore, in order to run exploratory testing in a Waterfall environment,
documentation on test results is a must and this documentation should be
easy to trace back to requirements. Of course, this type of documentation is
always useful in any environment.
What does adoption entail? Embracing exploratory testing is relatively easy,
as it is quick to launch (and scale), simple to learn and provides benefits for
the entire team. That said, it’s important to keep in mind that it should not be
the only form of testing (rather, it should inform what type of testing happens
next). Additionally, even though it’s unscripted, exploratory testing should not
be unstructured (testers still need to set a goal, log your activities and take on
the mindset of a specific user persona).
Who is it for? Exploratory testing can help reduce time spent testing, find
more defects and improve code coverage. As a result, exploratory testing is
best suited for teams that are under time constraints, teams that need help
identifying the best types of tests to run (especially in cases where there are
no specifications from developers) and teams that want to make sure they
didn’t miss anything in previous tests. Key team members who should be
involved in exploratory testing include:
 Testers (although everyone on the team should participate in some way)
What are best practices? Best practices for testers using exploratory testing
include:
 Organizing the functionality in the application using something like a
Mindmap or a spreadsheet
 Focusing on certain areas or certain scenarios
 Tracking what gets tested to help reproduce any errors
 Documenting results in a tool like qTest Explorer so there is some
accountability for what got tested
4) Session Based Testing

What is it? Finally, let’s review session based testing. Session based testing
builds on exploratory testing by providing more structure. Because exploratory
testing is completely unscripted, it makes accountability difficult and relies
heavily on the skills and experience of the tester(s) involved. Session based
testing aims to ease some of these shortcomings by bringing more structure
to exploratory testing without taking away from the benefits that exploratory
testing provides, such as the ability to better mimic the user experience and
get creative with testing.
Session based testing provides this structure by conducting testing during
time-boxed, uninterrupted sessions, testing against a charter and requiring
testers to report on the testing that took place during each session.
Additionally, session based testing should be capped off with a “debrief”
between the tester(s) and the manager that covers the five PROOF points:
What happened (Past), what was achieved (Results), what got in the way
(Obstacles), what still needs to be done (Outlook) and how does the tester
feel about it (Feelings).
How is it different from standard Waterfall testing? The same as
exploratory testing, session based testing can be run in both Agile and
Waterfall environments, but it is more conducive to the tight collaboration
between testers and developers that is typically found in Agile environments.
What does adoption entail? Much like with exploratory testing, adopting
session based testing proves relatively easy because it is easy to pick up and
launch quickly. For testers already accustomed to exploratory testing, the
biggest hurdle is embracing the additional structure for which session based
testing calls. Also like with exploratory testing, teams running session based
testing should remember that it is not a final stop, but rather a method to help
determine the best type of testing to conduct next.
Who is it for? Session based testing can help reduce testing time while
increasing defect discovery and code coverage, making it ideal for teams that
face time constraints and need more guidance to determine what types of
tests to run. It is also ideal for teams that have found benefits in exploratory
testing but need to improve accountability throughout the process. Key team
members who should be involved in session based testing include:
 Testers
 Managers
What are best practices? Best practices for testers using session based
testing include:
 Outlining a mission so that testers are clear about the software they’re
testing
 Developing a clear charter that indicates the mission, the areas of the
software to test, which tester(s) will run the session, when the session
will take place and for how long, the tests designed and executed, the
bugs found and overall notes (like with exploratory testing, a
documentation tool like qTest Explorer can help here)
 Running the testing session without any interruptions
 Clearly documenting activities and notes during the session in a session
report
 Holding a debrief between the tester(s) and the manager to review
findings from the session and discuss next steps for testing
How to Align Testing with an Agile
Delivery Process
Once you determine which testing methodology is right for your organization,
you’re not quite done yet. You still need to align testing with the delivery. To
achieve this goal we recommend a three-pronged approach:
1) Get involved in the development process
as early as possible
The sooner testers can get involved, the better. Ideally, testers should be
present from day one. That’s because giving testers a seat at the table every
step of the way provides a higher level of insight into requirements and goals,
encourages collaboration and helps hammer home the need to conduct
frequent (if not continuous) testing.
2) Test frequently, but thoughtfully
As more and more teams adopt Agile methodologies, efficiency is
everything. This need for speed has led teams to embrace DevOps and
continuous integration as well in order to keep things moving, and that
requires testing more frequently. But in the midst of an efficiency and
frequency focused shakeup, testers need to remain thoughtful so as not to
create more overhead and run unnecessary tests that actually slow down the
process.
3) Hit the ground running with test creation
Keeping in mind the need for speed in today’s Agile, DevOps driven world,
testers need to hit the ground running when it comes to getting tests created.
Specifically, the more testers can reduce the time from requirements
gathering to test creation, the better. Having a seat at the table for all
conversations from the very beginning should help in this regard.
What’s Next for Agile Testing?
Although Agile has already made significant inroads into the software
development lifecycle, there’s still a long way to go, especially among testing
teams.
Going forward, more widespread adoption and greater maturity of Agile
methodologies will require testers to go beyond test creation and execution
and begin to focus on code delivery and integration too. At the same time,
testers will need to hone their automation skills, become more involved in the
entire software development process and continue to develop a collaborative
relationship with developers. Ultimately, these changes will also require
testers to become experts at development and product usage in order to
provide more holistic testing strategies and take on the role of “quality
champions.”
In the future, three key tenets will become particularly important for testers
working in Agile environments:
1) Communication
Agile requires a tight collaboration between testers and developers, and that
collaboration makes communication a top priority for testers. Additionally, in a
world where quality becomes everyone’s responsibility, testers will become
“quality champions” that serve as internal experts, which will put their ability to
clearly communicate testing needs and reasoning under the spotlight.
2) Skill Diversity
In an Agile environment, everything can change on a dime, and that requires
testers to be adaptable. Part of this adaptability is having a diverse skillset so
that testers can change course as needed. For instance, functional testers
need to expand their skills beyond manual scripted execution. This diverse
skillset will be a must as different sprints require different types of testing to be
executed in a short timeframe.
3) Business Mindset
Finally, Agile takes on a very customer-centric approach in order to ensure
customers receive as much value as possible as quickly and as early on as
possible. Testers have a big role to play in delivering this value, but it requires
them to take on a business mindset so that they can understand customer
expectations, desires and concerns and develop their testing strategies
accordingly.