agile - bridging the gap

6
by MICHELE SLIGER Agile Projects in the Waterfall Enterprise 26 BETTER SOFTWARE JULY/AUGUST 2006 www.StickyMinds.com

Upload: sankofakanian

Post on 14-Apr-2015

33 views

Category:

Documents


0 download

DESCRIPTION

Agile - Bridging the Gap

TRANSCRIPT

Page 1: Agile - Bridging the Gap

by MICHELE SLIGER

Agile Projects in the Waterfall Enterprise

26 BETTER SOFTWARE JULY/AUGUST 2006 www.StickyMinds.com

Page 2: Agile - Bridging the Gap

Although agile software development methodologies havebeen around for almost a decade, interest in agile approaches hasexploded recently as companies seek better ways to competewith their global counterparts by getting their software products tomarket more quickly. The economy’s downturn and subsequentbudget cuts have led to greater interest in how to do things fasterand how to do more with less. As a result, agile methodologies—Scrum, Extreme Programming, Lean Software Development,Crystal, Dynamic Systems Development Method, Adaptive Software Development, and others—have attracted many CIOswho are looking for approaches to make product development faster, more reliable, and more satisfying to the end-user.

www.StickyMinds.com JULY/AUGUST 2006 BETTER SOFTWARE 27

Page 3: Agile - Bridging the Gap

While these promises are enticing, mostagile methodologies are designed forsmall, collocated, collaborative teams.Knowing these constraints—yet stillfeeling a need to respond rapidly tomarket pressures—companies withlarge and geographically dispersed ITdivisions have chosen to forge ahead,making changes when necessary, to adoptagile methodologies within their large,historically waterfall-oriented organiza-tions. Because it’s simply impractical to“flip a switch” and have a 1,000-personIT department start doing agile all at once,these organizations must wade through asometimes murky transition period whenagile and waterfall are forced to coexist.

During such a transition, what’s an ITenterprise to do? Agile and waterfall areso utterly different—from the way projectsstart (diving right into coding vs. spendinglong weeks in analysis and design) to thetypes of meetings held (quick, daily planning meetings vs. long, weekly statusmeetings), to what we expect of the team(self-organized vs. directed), and even the

The Agile-WaterfallCooperative

Large companies have clear, valid reasons for requiring certain waterfall activities on otherwise agile projects. Project approval processes designed toweigh benefits, costs, and strategic alignment with corporate objectives areone example of a standard waterfall-up-front requirement. Independent Validation and Verification (IV&V) is awaterfall-at-end requirement for thosecompanies in industries that must complywith external regulations. And waterfall-in-tandem occurs when the system underdevelopment is so large and complex thatmultiple teams, often working on differentplatforms, must work together to preparea release.

None of these requirements will ceaseto exist simply because the teams are employing agile practices. Instead the teamsmust learn to factor these corporate needs into their agile procedures, and management must begin the investigativework of determining how to streamlinethese requirements and activities so theydon’t hamper the project.

Waterfall-up-frontI counseled one team to use Alistair

Cockburn’s “barely sufficient” philosophy(see the Sticky Notes for more informa-tion), in order to avoid doing more thanwas absolutely necessary when facedwith waterfall-associated deliverables.The team members had been newlychristened as an agile pilot team whenthey found that they still had to gothrough the standard waterfall projectapproval process to obtain the necessaryresources and funding.

Using the “barely sufficient” guideline,the team asked, “Is this something thatwe really must do? And if it is, then whatis the simplest thing we can do to satisfythe approval process?” Project approvalwas a must for the team, because with-out approval the project team would be disbanded and the monies and staffwould go to other, previously approvedprojects. But conversations with the financial controllers revealed that thedocumentation required for the approval process could be streamlinedand simplified, thus saving the team

expected deliverables (chunks of workingcode early and often vs. documentationearly with code at the end). These twopractices have different ways of measuringprogress, determining success, managingteams, organizing, and communicating.How can they be managed as part of acohesive project portfolio? Can agile andwaterfall methodologies coexist and stillmake the company successful?

The answer is yes. Not only can theycoexist for the interim but they can coexist for the long term, since not allcompanies will choose to move everysoftware development project to an agileparadigm. The trick is in how they cancoexist peacefully and not detract fromoperational stability and continued projectsuccess. Every transitional environment,agile or otherwise, has to deal with dualityuntil the transition is complete. Doingthis with the least amount of pain anddisruption means tightly embracing one ofthe key agile tenets: “inspect and adapt.”Process reviews and the progress the teamshave made at the end of every iteration(typically one to four weeks in length)guide decisions on how best to proceedin the next iteration. This allows teams toadjust some of the agile practices to workbetter in the current environment, knowingthat these agile practices may change andbecome more “pure” as the environmentand culture change over time.

Some agile purists will say that bystretching the process it’s no longer truly “agile,” and in many cases theymay be right. Semantics aside, it’s stillabout improving the overall softwaredelivery system, and the label we applyto the methodology is secondary. However,teams must be careful not to stretch agilepractices so much that the teams revertto their more comfortable, waterfallhabits. When agile is viewed as an à lacarte menu of practices that can beadopted as the teams see fit, it’s critical to adhere to a diet of key principles—continuous improvementthrough time-boxed iterative deliveriesand reviews, implementation of themost important items first, and constantcollaborative communication. As theybegin their transition, teams followingthese principles will find ways to integrate agile and waterfall.

28 BETTER SOFTWARE JULY/AUGUST 2006 www.StickyMinds.com

Using the “barely sufficient” guideline,the team asked,

“Is this something

that we really must

do? And if

it is, then what

is the simplest thing

we can do

to satisfy the

approval process?”

Page 4: Agile - Bridging the Gap

from having to do the big up-front designthat goes against the agile philosophy.

Based on the agreement reached withthe financial managers, team membersdid the “barely sufficient” analysis and design work required to produce atechnical specification document, whichthey submitted with their business casefor approval. The design specificationwas a high-level overview of the systemwith high-level functional specifications,but it avoided the detailed requirementsthat would have taken the team weeks tocomplete. The documents were consideredjust good enough for the review process,

and the project was approved.Once they had approval, team members

held their first iteration planning meetingand brought along the specification thatthey had created. From the thirteen-pagedocument they extracted a total of twelvesentences that represented the first few features they wanted to develop and thenused this information as their initial product backlog.

They set aside the specification andrarely referenced it during the course of therelease. The creation of this specificationwas not, however, viewed as a waste oftime. Rather, team members felt they hadbenefited from the shared product visioncreated from the exercise of compiling thespecification. And of course, the financialmanagers on the project approval boardgot the information they needed to determine capitalization (some companiesrequire a technical specification before deciding whether a project’s budget can belabeled as capital or expense) and thus

waterfall at the end of each project in orderto prepare the software for production.

Passing through the required phase-gate of the team’s production departmentmeans that all documentation, meetings,end-to-end system testing and compatibilitytesting, and sign-offs must be planned andcarried out. The team decided to use story cards to represent each item on theproduction checklist and allocated an entire iteration to production readiness.Team members did not implement anynew features during this iteration; insteadthey produced documentation for production, customer support, the architecture review committee, and systems test. (Note: This company’s systemtest was concerned primarily with thesubmitted application’s compatibilitywith other existing applications on thenetwork—the agile team still tested itscode in each iteration.)

Figure 2 illustrates this example. Ifteam members had any time left in thisiteration, they spent it refactoring code,installing development and test system upgrades, investigating new technologies, and training staff.

An iteration dedicated to preparingthe passage of potentially shippable codebase to another entity is useful in a widevariety of situations: Sarbanes-Oxley auditing, FDA approvals, and IV&V.Even if your product does not need outsideapproval, you can use this hardening iteration to capture final screen shots formarketing and training materials, finalizethe release notes, conduct performanceand load testing, and other organizationalactivities. Just be sure that your team isn’t using this last iteration solely as a

approved the project.Other teams with which I’ve worked

received provisional funding prior to officialapproval, so they included the project approval requirements in their first twoiterations as deliverables along with requested features. You can see an exampleof both approaches in Figure 1.

Don’t be afraid to put non-productitems in the backlog, especially if youwant to take advantage of the high visibility it affords. Ken Schwaber, authorof Agile Project Management with Scrum,calls the list of project initiation workitems the “Scrum Startup Backlog.”

Waterfall-at-endAnother agile team I know of in a

large telecommunications firm discoveredthat having to deal with waterfall-at-end requirements meant adding an extra iteration to prepare the required deliverables. This team produces application software for internal end-users, and it must switch from agile to

www.StickyMinds.com JULY/AUGUST 2006 BETTER SOFTWARE 29

1 noitaretI

esaeleR

golkcaB

lavorppA tcejorP •1 erutaeF •2 erutaeF •

lavorppA .hcrA •3erutaeF •

tcejorP •lavorppA1 erutaeF •

2 erutaeF •.hcrA •

lavorppA

3 erutaeF •4 erutaeF •5 erutaeF •

6 erutaeF •7 erutaeF •

tcejorP

lavorppA

ssecorPoT

remotsuC

4 noitaretI3 noitaretI2 noitaretI

Figure 1:Waterfall-up-front

e s a e l e R

1 e r u t a e F • 2 e r u t a e F •

3 e r u t a e F • 4 e r u t a e F •

5 e r u t a e F • 6 e r u t a e F • 7 e r u t a e F •

n o i t c u d o r P • s t n e m u c o d w e i v e r . h c r A •

p l e H n i a r T • k s e D

V&VI/noitcudorP

o T

r e m o t s u C

3 n o i t a r e t I 2 n o i t a r e t I 1 n o i t a r e t I 4 n o i t a r e t I

Figure 2:Waterfall-at-end

Page 5: Agile - Bridging the Gap

bug-fixing opportunity; that’s a signalthat the team is committing to more featuresin each iteration than it has time to test.

Waterfall-in-tandemA healthcare management company

with which I’m currently working hasmultiple complex systems—systems solarge they must be broken down intocomponents, with each component havingits own project teams. Data flows fromlegacy minis and mainframes to newer,browser-based applications and thenback again—and sometimes even on to something else—resulting in huge coordination efforts to ensure successfulsystem integration.

One of the component teams was piloting agile and realized that in order tocoordinate milestone deliverables withthe other teams they would have to

include the waterfall project managers inthe planning sessions. This realizationcame after attempts to stay synced up viaemails failed horribly. The agile teammembers realized they had left the “collaborative” out of the “constant collaborative communication” principle(sorry, constant emailing doesn’t count!),and so began inviting the waterfall projectmanagers to all their planning meetings—release planning, iteration planning, anddaily stand-ups. Initially the waterfallproject managers grumbled that all theseplanning sessions were wreaking havocon their calendars. However, once theyhad attended a few sessions, the managersbegan to realize the value of the sharedinformation and the improved ease andcoordination of the work.

Although this approach was more

company-sanctioned style sheets. Insteadthe agile team members created a simpleUI that was functional and scalable, albeit not particularly attractive. At theend of the first release, the product wasdeployed internally with the agile team’sUI. By the next release the waterfall teamhad delivered, and the official screens became part of that second release. In themeantime, the end-users had access tothe needed functionality, and the agileteam was praised for its efforts.

The Anomaly—TwoSeparate Entities

I’m aware of one organization thathas decided to keep the agile and waterfallgroups completely separate, with no overlap at all. This particular companyset up its agile teams as a separate organizational unit, and its internalclients pay the department for the use ofits agile teams for a fixed period of time(not for a set project).

These agile teams function as a typeof “skunk works” operation (see theStickyNotes for more information).Copyrighted by Lockheed Martin, the“skunk works” phrase was created by a1943 Burbank, California, team thathad been tasked with creating the P-80 jet fighter from scratch. Clarence“Kelly” Johnson, who headed the team,described a skunk works operation as “asmall group of experts who drop out ofthe mainstream company operations inorder to develop some experimentaltechnology or new application in secrecyor at speed, unhampered by bureaucracyor the strict application of regulations.”These stealth agile teams evade waterfallcontrols thanks to the power of their ex-ecutive sponsors and their paying clients,who work together to protect and incu-bate the teams so they can focus on theirassignments. And by keeping the twoseparate, management avoids dealingwith the issues surrounding changesneeded in organizational structure, project

successful than email, overseeing multipleteams was still difficult for the waterfallmanagers. So, after another iteration review, the process was modified again toinclude other key members of the waterfallteams in addition to the waterfall projectmanagers. All the teams eventually cametogether for release planning meetings,with a smaller set of waterfall team representatives attending iteration planningmeetings when needed (see Figure 3). Asecond tier of daily stand-ups was addedfor the team leaders, as in the KenSchwaber “Scrum-of-Scrums” model.For more information on how thesemeetings work, see the StickyNotes for alink to a nicely articulated definition andgraphic by Mike Cohn. These teams, waterfall and agile, used the iteration reviews to inspect and adapt and thuscreated their own natural transition plan.

Don’t be surprised, however, to findyour agile teams moving ahead of thosewaterfall teams that aren’t inclined to co-operate or are unable to deliver. Agileteams usually figure out a way to keepmaking progress, even without the expected deliverables on which they rely.The agile team will stub out that missingsubsystem and create work-arounds, or theywill simply build the module themselves.

In one instance, agile team membersdecided not to wait for the waterfall userinterface (UI) design team to provide the

30 BETTER SOFTWARE JULY/AUGUST 2006 www.StickyMinds.com

e s a e l e R

m a e T

e l i g A

t s e T … e d o C … n g i s e D … s i s y l a n A m a e T

l l a f r e t a W

s t n i o P c n y S d n a s e n o t s e l i M

o T

r e m o t s u C

4 n o i t a r e t I 3 n o i t a r e t I 2 n o i t a r e t I 1 n o i t a r e t I

Figure 3:Waterfall-in-tandem

Don’t be surprised, however, to find your agile teamsmoving ahead of those waterfall teams who aren’tinclined to cooperate or are unable to deliver.

Page 6: Agile - Bridging the Gap

approval procedures, ongoing portfoliometrics and management, and overallculture that must be addressed in an ag-ile- waterfall cooperative.

Making changes—especially of thismagnitude—is never easy. But most teamshave told me that the hardest part is justgetting started. Once teams beginadopting some of the practices, they findthe rewards surprising and well worth thecontinued effort. Organizations find thatbeing able to better respond to the marketplace isn’t the only reason toswitch to agile. The cooperative and collaborative nature of the approachprovides a more supportive, meaningful,and exciting work environment for themembers of the team. Agile is a commonsense approach to software development, and the success of a waterfall-agile enterprise lies in creatinga transition plan to excellence throughthe cycle of inspection, adaptation, andexecution. {end}

Michele Sliger has worked in softwaredevelopment for more than fifteenyears. She currently works as an agile consultant for Rally Software Development, where she trains softwaredevelopment teams in agile methodologies.Previously a project manager at QwestCommunications and a consultant forFortune 500 companies, Michele was afounding member of the engineeringteams at two biotech start-ups. She is acertified Project Management Professionaland a Certified Scrum Master. Michele isalso an adjunct faculty member of theUniversity of Colorado, where sheteaches software project management tograduate engineering students. EmailMichele at [email protected].

www.StickyMinds.com JULY/AUGUST 2006 BETTER SOFTWARE 31

StickyNotes

For more on the following topics, go towww.StickyMinds.com/bettersoftware

� Alistair Cockburn’s Barely Sufficientmethodology

� Mike Cohn on daily stand-upmeetings

� More on “skunk works”� Further reading

The KEY TO SUCCESS in each of these examples is the teams’ approach to transition problems—solving problemscollectively, collaboratively, and with thefreedom to make decisions and changeswithin the discipline of regular reviews.Here are several recommendations to helpyour organization develop ways for agileand waterfall to effectively coexist:

� Talk with waterfall stakeholders about how you’d like to work together.Answer their questions about agile, and ask for their help in making therelationship as pain-free as possible. The executive sponsor should bepart of the initial discussion, to share her commitment to the process,her dedication to removing roadblocks, and the importance of eachteam’s involvement.

� The waterfall requirements of the project should become stories in the agile team’s backlog. Include waterfall stakeholders in the agile planning meetings, so that everyone understands what qualifies as barely sufficient deliverables, what assumptions the teams are making,and what dependencies exist.

� In the review and retrospective held at the end of each iteration, analyze the benefits and challenges you’ve just experienced and makerecommendations on how to improve the experience in the next itera-tion. Again, be sure to include the waterfall stakeholders.

� Executive and middle management should create their own prioritizedbacklog of transition issues they need to focus on, implementing corporateprocess and organizational change iteratively and incrementally.

� Don’t try to fix everything before you start agile adoption. Just dive rightin—you’ll find a way to work within the current constraints. Use the iteration reviews and retrospectives to make change recommendationsand implement incremental improvements as you go. Be reasonableabout this by focusing on only the top two or three things for the next iteration. It’s a backlog of recommendations and, like the backlog ofproduct features, they all can’t be implemented at once.

� Pay attention to behaviors and avoid reverting to old habits. For example,if you notice that you’re depending on the specification that you had tocreate in the waterfall-up-front iteration instead of having discussionswith the product owner, set the document aside.

� Invite all stakeholders to participate in a project retrospective at the end. These meetings are crucial to help identify and implement broader transition changes that affect the entire enterprise.