agile testing whitepaper

8/8/2019 Agile Testing Whitepaper

1/15

How to ReallyPrevent and Manage Bugs

in Agile Projects

Michael DubakovTargetProcess, Founder


2/15

Table of content

Zero Defects Mentality 4

Bug Source #1: Unclear User Stories 5

Clarification meeting

Stories should be ready before iteration start 6

Bug Source #2: Manual Testing in Short Iterations 7

Bug Source #3: Automated Testing in Short Iterations

Bug Source #4: Separate Testing Team 8

Bug Source #5: Bug Clusters

Agile Bug Management 9

Bugs found in iteration

Bugs found in production 10

Bug estimation 12

Agile Bug Management Anti-Patterns 13

Bug fixing phase inside iteration

Bug fixing iteration

Skip Zero open bugs rule in user storys Definition of Done

Defect Tracking Systems 14

Co-located

Distributed

References 15

2


3/15

Foreword

Today, Agile is a widely adopted software development approach. Most developershave either heard of or successfully applied popular agile practices like TDD, CI, PP, etc.

But what about testers? It seems that many teams are having difficulties trying to

integrate quality control teams into an agile development process. Moreover, it also

seems that many teams are having difficulties with overall bug management. How to

treat bugs? Does a bug have a business value? Should we estimate bugs in points or in

hours? Should we treat bugs as user stories and add to velocity?

This article not only answers the questions above and explains how to manage bugs in

agile projects, but it also shows how not to manage bugs.

3


4/15

Zero Defects MentalityAre you familiar with a zero defects mentality? It looks very good from the first sight: Id like

to have zero defects in my projects.

So, what is a zero defects mentality?

Here is the quote from the book Lean Software Development: an Agile Toolkit:

One of the fastest ways to kill motivation is what is called in the US Army a zero defects

mentality. A zero defects mentality is an atmosphere that tolerates absolutely no

mistakes; perfection is required down to the smallest detail. The army considers a zero

defects mentality to be a serious leadership problem, because it kills the initiative

necessary for success on a battlefield

Mary & Tom Poppendieck

Obviously, a zero defects mentality is not something that HOUSE M.D. likes ;) Moreover, I think

Dr. House hates it, because it has several unpleasant effects:

Not enough courage to refactor a complex, messy, buggy, but important piece of code.

Inability to make an important right decision. The result is a less risky, but wrong

decision.

Doing everything to avoid responsibility leads to coward and stupid behavior.

Zero defects may sound good. But in reality you still have errors after production. Even in

predictable and (quite) easily testable industries (hardware, automobile, etc.) there are

problems with 100,000 power adapters that should be replaced (or hard drive problems, or

accumulators problems, I bet you can continue the list).

How can we expect zero defects in software development? It is harder to test, harder to define

in details, harder to predict. Software development is a complex adaptive system, and we

can't predict all effects.

So bugs in production are a normal thing, and by normal I mean that we can't bring them to

zero. We can (and should) minimize them using all possible ways. Bugs are a form of waste.

We should do our best to reduce bugs in software projects and eradicate waste.

There are several strategies that will help:

Test Driven Development. A nice side effect of TDD is a unit tests suite. You have tests

for all new code, and you have unit tests for all old code. If you have good tests - you

have less bugs.

4


5/15

Continuous integration. Instant feedback helps to identify problems early and fix them

early. It saves money and you have less bugs in production.

Automated regression functional testing suite. Unit tests are good, but you need

something else. Functional tests emulate user behavior and identify user interface

errors, integration errors, etc. Needles to say you should have continuous integration in

place to succeed with automated functional tests.

Root cause analysis. There are several ways to fix the bug. You may just hack the code

and apply a patch (please, dont do it at home!). You may think deeper and fix the bug

nicely. Also you may look into the root of the problem and resolve it. Yes, it may take

time, but you will prevent future bugs from this domain.

High development skills. Ironically, high skills do not always reduce a bug rate in

production. Stars may be strong in some areas (business layer), but may not polish

things. It may lead to many small bugs, especially on UI.

Stop worrying about the defect database and start worrying about why you are still

creating code where defects are discovered a significant amount of time after the code

has been written.

Mary Poppendieck

Bug Source #1: Unclear User StoriesI think unclear specifications are a #1 source of bugs. If you deliver something that was not

intended, there will be many complaints from customers and in their opinion almost all of

these complaints are bugs. Also, without instant communication it is very easy to miss somedetails and release an unpolished user story with many small glitches.

Agile development is very rapid. In many cases the path from an idea to user story in progress

is extremely short; it may be even 1 day. In this environment it is very easy to make mistakes

in a user story description. Product Owner may miss some important details and as a result

the story does something wrong, which was not expected.

This is the first question you must ask... is it a bug, or is it a change? I've seen a lot of

bugs that have come up that were "We asked you for x, never thinking about y, so couldyou please change the system so that y is covered?" It's a business scenario, so there's

little reason to expect a dev or tester to anticipate/test it. I know where we've struggled is

in injecting items into the product backlog, we tend to classify it as a bug and so we end

up with a lot of bugs, but not a lot of change in the back log. That's exactly what I think

we're supposed to be avoiding in Scrum or any other agile methodology. As we see

change, we should be taking it on as a feature and prioritizing it accordingly. Only when

it's something broken should it be called a bug.

Jim Leonardo

The best solution to this problem is communication.

5


6/15

Clarification meetingThe important thing in software development is to put everyone on the same page.

Developer, Tester and Product Owner should conduct a small meeting about a user story, thus

forming a mini-team. If the user story has several developers and several testers, all should

participate.

The goals of the meeting are:

To ensure that everyone understands what should be implemented.

To brainstorm acceptance tests/test cases and create a check list.

To reveal and eliminate all discrepancies in a story description/specification.

We have these meetings at TargetProcess and they areveryefficient and helpful. The side

effect of the meeting is that Product Owner receives less questions about functionality later,

thus having less interruptions (which is good).

Stories should be ready before iteration startIt is a good idea to have all user stories described in minute detail before iteration start. Thus,

testers and developers will have time to review them and prepare questions for the iteration

meeting. It may drive a much better discussion and reveal some interesting problems thatwere unclear initially.

Discuss User

Stories,

brainstorm

acceptance tests

Product

Owner

TesterDeveloper

6


7/15

There is a danger to document more user stories than is required, and it will be a form of

waste. Product Owner should maintain good balance here.

Bug Source #2: Manual Testing in Short IterationsMost testers are having hard time switching from waterfall to iterative development. Short

iterations put pressure on a testing activity. It is required to write test cases fast, executethem fast and provide feedback fast. It is far away from a huge testing phase in waterfall.

Under pressure testers make more mistakes, think less and miss some test cases in a hurry.

Testers tend to write brief test cases and skip unusual or not obvious cases. Naturally, it is a

straight way to bugs in production.

The most common error is a testing phase in every iteration. It is a kind of mini-waterfall and

should be avoided. Each feature should have a separate flow with its own testing phase. It is

required to test by feature. Tester should start testing immediately after a feature is developed(or even earlier if possible).

Bug Source #3: Automated Testing in Short IterationsTest Driven Development is a great bug catcher. If you apply it right, you will improve product

quality significantly and bugs in production will be quite rare. However, it is not as easy as it

sounds.

There are two levels of TDD: Unit Tests and Functional Tests.

It is relatively easy to adopt unit tests and make it a mandatory practice in the development

team. Developers write unit tests, then write code, then refactor.

Functional tests are harder to create and maintain. And it is even harder to create them

before feature implementation. And it is especially hard to create automated functional tests

upfront on the User Interface level.

Tools like Fitnesse simplify functional testing of business logic, but still you want to have

complete UI tests using Selenium, WinRunner or any other GUI testing tool.

Here are several pieces of advice that will help you catch more bugs during iterations using

automated tests:

Adopt TDD fully on the unit tests level.

Hire highly experienced automated testers (almost like developers in skills) who can

write automated tests very fast. Speed is important here. An alternative is to force

developers to write automated functional tests.

Create automated functional tests for the previous iteration at the beginning of the

next iteration. It will be regression testing, but it is definitely better than nothing.

7


8/15

Bug Source #4: Separate Testing TeamIt is common for a waterfall process to have separate functional teams: Designers, Developers,

Testers, DBA, etc. Functional teams set additional communication barriers, and poor

communication leads to many problems like:

Missing test cases.

Different understanding of a user story by Developer and Tester.

Delay between story implementation and testing phases.

Obviously, these problems lead to more bugs found in production. The best thing to do here is

to create a cross-functional team and put testers and developers into one shared space. It

should be one team, not several sub-teams.

Quite a common situation with functional departments looks like this. You have several

testers assigned to your project. Then suddenly two of them get relocated to another project

and replaced by two other testers. That is a real danger to your project.

So, your goal is a single cross-functional team. If it is not possible, youll have to play political

games. At least you should try to extract a dedicated testing team for the project and

minimize the power of Functional Testing Department Manager. In this case you have a

chance to join two functional teams into one.

Bug Source #5: Bug ClustersBugs like to live in communes in a single nest. Most likely you have modules with perfect

conditions for bug survival and reproduction. Bugs prefer the following places:

Code written by junior developers.

Code written under time pressure.

Legacy (initially beautiful) code that was passed to the offshore team, then back to the

main development team, then back offshore and then shared between several teams.Code that somehow does not have unit tests (what a shame!).

UI (yes, most user interfaces are buggy, since developers do not unit test UI).

Quite a large and complex piece of code (there is a high chance that complex logic will

have some missed test cases as it is hard not to forget something).

Code written in JavaScript.

Integration with a third-party system.

So if you have a complex UI component written by Junior Developer in JavaScript with no unit

tests and under pressure - I bet you have several bugs in this code.

8


9/15

Anyway, bugs are nesting and so a good strategy is to find the nest and destroy it (refactor,

rewrite, add unit tests, etc). Lisa Crispin has something to say about it.

If you already have a defect database, consider mining it for information about where

defects cluster, and focus on cleaning up the related code. - Lisa Crispin

http://home.att.net/~lisa.crispin/CrispinFinal.pdf

If you have a defect tracking system, you may use tags, thus categorizing defects. It will be

quite easy to find problematic areas in your software. For example, you may find out that most

bugs live in the Reports or Email Integration areas.

Agile Bug ManagementAgile development books often skip a bug management practice. Many agile people assumethat the problem is obvious, unfortunately it is not the case.

In general, there are two types of bugs: bugs found during iteration development and bugsfound in production. Strategies are quite different as you may expect.

Bugs found in iteration

There is a huge difference between a bug found during story development and a bug found bythe customer in production. A bug found during story development is much easier to fix, sinceyou have focus on the story, you know what you did 1-3 days ago and everything is fresh in

Software

Authentication

module

Content

module

Reporting

module

Bug

Bug

Bug

Bug

Bug

Bug

Bug

Bug

Bug

Bug

BugBug

Bug

Bug

9
http://home.att.net/~lisa.crispin/CrispinFinal.pdfhttp://home.att.net/~lisa.crispin/CrispinFinal.pdfhttp://home.att.net/~lisa.crispin/CrispinFinal.pdf


10/15

mind. You may even recall some places in the code where the bug can live right away. If a bugis found a month from now, it will take time to switch to it, fix, include into the new build anddeliver.

So, how should we handle such bugs? First, do not estimate bugs during story development atall. Second, do not prioritize them. Bugs are a usual part of the development process and on

average in each iteration you will have quite similar time spent on bug fixing. So why botherwith estimation and prioritization? One simple rule replaces them: all bugs found in theiteration should be fixed. No exceptions. No excuses.

If you do not estimate bugs, they do not affect iteration velocity. If you use the same estimatetechnique during several iterations, most likely you will have similar bug rates. It means thatthe bug fixing deviation does not affect velocity.

Iteration #1

User Story User Story User Story User Story

Bug

Bug

Bug

Bug

Bug

Bug Bug Bug

User Story User Story

Bug

Bug

Bug

BugBug

Bug

Bug Bug Bug 0 open bugs

Time

Alternatively you may estimate bugs as tasks (in hours), but I dont see much value it thiseffort.

Bugs found in productionUnfortunately, we have to deal with bugs in production. No software is bug free. There are

bugs in business software, medical software and even space shuttle software.Sure medical software has less bugs, but they still exist (http://www.cs.tau.ac.il/~nachumd/verify/horror.html).

Bugs as User StoriesOne strategy is to put bugs into the product backlog and treat them as user stories. From thecustomers perspective there is no much difference between bug in production and a newfeature request. The customer wants to have a problem solved, and the problem may beexpressed as a bug or may be expressed as a user story.

10
http://www.cs.tau.ac.il/~nachumd/verify/horror.htmlhttp://www.cs.tau.ac.il/~nachumd/verify/horror.htmlhttp://www.cs.tau.ac.il/~nachumd/verify/horror.htmlhttp://www.cs.tau.ac.il/~nachumd/verify/horror.htmlhttp://www.cs.tau.ac.il/~nachumd/verify/horror.htmlhttp://www.cs.tau.ac.il/~nachumd/verify/horror.html


11/15

Iteration #1

User Story Production Bug User Story User Story

Bug

Bug

Bug

Bug

BugDevelopment Team

Production Bug Time

Bug

Bug

Is this bug with login screen more important than the Advanced Search user story? It is a

subject for Product Owner to decide. He should maintain a single backlog with bugs and user

stories prioritized and select bugs and stories for the next iteration.

In such a scenario you will have to estimate bugs in points or in hours to have a fair

prediction how many bugs and stories you may take and implement in the next iteration. It

sounds somewhat counterintuitive, since bugs are a form of waste. But still prioritization and

estimation are important in this scenario.

However there is a danger in this approach:

If defects are viewed as features with negative value, they become managed as if they

were features. Development groups store up prioritized repositories of bugs, treat bugs

as user stories, outsource the fixing of bugs, and so on. While each of those can be a useful

technique or perspective in dealing with a project in transition or crisis, it is not a long-

term view that should be encouraged. After all, as the "Manifesto for Agile Software

Development" says, "Working software is the primary measure of progress." It is a little

disingenuous to pass off a feature with known defects as complete and working "Yes,

it's done... but there are a few bugs." - Kevlin Henney

People dedicated to bug fixingAnother strategy is to have dedicated people for bug fixing. For example, you may dedicate

about 10-20% of a development teams effort to production bug fixes. Alternatively, you mayhave one developer each day on production bug fixing, and developers may rotate daily.

11


12/15

Iteration #1

User Story User Story User Story User Story

Bug Bug

Time

Development Team

Production Bug Production Bug Production Bug Support Developer

In this case, 25% of the development team is solely dedicatedto fixing bugs found in production (1 of 4 developers)

Free time. Support Developer drinks bear or

joins the development team (and enjoys userstories). But if a new bug is found inproduction, she switches to it with no delays.

Bug Bug

Bug

BugBug

Bug

Using this approach, you have two separate flows: one for user stories and another for bugsfound in production. Obviously, each flow should be managed separately. While for userstories you have usual metrics like an iteration burn down chart, for a bug flow you need toinvent something different.

If you do not estimate bugs, all you have is the bugs fixed/day metric. If you do estimate them,you may create a bug burn up chart or points fixed/day chart.

If you have few bugs, good advice will be to not track the bug fixing flow at all. But if youhave many bugs that will take several weeks to fix, you need some metrics to forecast whenyou will be able to fix all Critical bugs, all Normal bugs, all Small bugs, etc. Also you shouldknow how many new bugs are coming from production each week. It will definitely influenceforecasts.

Weeks to a bug free release = Total bugs / (Weekly bug fixing rate - Weekly new bugs rate)For example, you have 40 bugs to fix, weekly bug fixing rate is 10 bugs/week, and each week2new bugs are found. In this caseWeeks to a bug free release = 40 / (10 - 2) = 5 weeks.

Bug EstimationYou may not estimate bugs. There are several reasons not to estimate:

Often bugs are hard to estimate. Even a small problem may take a day to fix(unexpectedly revealing dirty roots that should be cleared up).Bugs should be fixed with no exceptions.If you have few bugs from production and use dedicated people for bug fixing,estimation does not bring any additional value.

Bug estimation is required when:

You have many bugs in production and want to provide some forecasting.You treat bugs as user stories and plan iteration accordingly.

12


13/15

Agile Bug Management Anti-PatternsThere are several dangerous misunderstandings (anti-patterns) related to bug managementthat are quite common. It is better to recognize the problem early and fight to win. Here arethree common bug management anti-patterns in a development process.

Bug fixing phase inside iterationSymptom: You have several days at the end of an iteration fully dedicated to bug fixing.

Iteration #1

User Story

Implement User Stories

User Story

User Story

Bug

Bug

Bug

TimeBug Fixing

Bug

Bug

BugUser Story

If you have a bug fixing phase inside an iteration, you have mini-waterfall in each iteration,which is not a very good idea (in agile we blame waterfall, dont we?). A story should betested as soon as it is completed.

Bug fixing iterationSymptom: You run several iterations with almost no or little bug fixing and then have one (oreven two) iterations that are fully dedicated to bug fixing.

Release #1

User Story Bug

Bug

Bug

Time

Bug

Bug

Bug

Iteration #1Implementing stories

Iteration #2Implementing stories

Iteration #3Bug fixing

User Story



A bug fixing iteration kills your iterative development. It is even worse than mini-waterfall.

Such iterations increase your work in progress dramatically. You have nothing done duringseveral iterations before a bug fixing iteration. You have nothing to ship. Moreover, bug fixingiterations reduce motivation, people do not like to fix bugs during 2 weeks or a full month.

Skip zero open bugs rule in user storys Definition of Done

Symptom: You have open bugs in a completed iteration.You may have decided that some bugs discovered during story development are unimportantand may be postponed to future releases/iteration. That is a very dangerous practice that

leads to bug accumulation. Moreover, it may be a reason for a bug fixing iteration (see above).The best strategy is to have a zero open bugs rule in DoD for a user story.

13


14/15

Defect Tracking SystemsShould we use a defect tracking system? The answer is it depends. Here are all possiblealternatives:

Use a defect tracking system (excel, web based, desktop).Use Task Board. Write bugs on sticky notes and stick them to Task Board.Use index cards. Write bugs on a card and give it to Developer.Personal communication (use voice and memory). Just tell Developer about theproblem.

Lets see under what conditions you may use these approaches.

Co-located Distributed

Small teamTask Board, index cards,personal communication

Defect tracking system

Medium TeamTask Board, defect trackingsystem

Defect tracking system

Large Team Defect tracking system Defect tracking system

Obviously, a defect tracking system is a must for a distributed team. A distributed teamdemands more formal communication and a single place to store all bugs.

A small co-located team may use any approach that fits their process. Medium and large co-located teams cant rely on personal communication and simple story cards. Task Board maywork for a medium team, but a large team most likely needs a defect tracking system.

I was surprised that our programmers find the defect-tracking system quite useful. The

ability to read clear, detailed steps to reproduce the problem saves them time. - Lisa

Crispin

14


15/15

References

Coping with Bugs on an Agile/Scrum Project

http://www.infoq.com/news/2009/07/coping-with-bugs

Software testing is an activity, not a phase or a departmenthttp://searchsoftwarequality.techtarget.com/news/article/

0,289142,sid92_gci1337882,00.html

Fix defects right away

http://www.cyphersec.com/2009/05/21/fix-bugs-now/

15
http://www.infoq.com/news/2009/07/coping-with-bugshttp://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1337882,00.htmlhttp://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1337882,00.htmlhttp://www.cyphersec.com/2009/05/21/fix-bugs-now/http://www.cyphersec.com/2009/05/21/fix-bugs-now/http://www.cyphersec.com/2009/05/21/fix-bugs-now/http://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1337882,00.htmlhttp://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1337882,00.htmlhttp://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1337882,00.htmlhttp://searchsoftwarequality.techtarget.com/news/article/0,289142,sid92_gci1337882,00.htmlhttp://www.infoq.com/news/2009/07/coping-with-bugshttp://www.infoq.com/news/2009/07/coping-with-bugs

agile testing whitepaper

Documents