agile is the new waterfall

Licensed Under Creative Commons by Naresh Jain

Agile (as practiced today) is the new Waterfall

Embrace Uncertainty

Naresh Jainnaresh@agilefaqs.com

1Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Some examples of heavy weight process

Lean Thinking

Origin of Agile

Agile Values and Principles

Summary of how Agile helps

Where do things go Wrong?

Naked Agile

For me to make sense lets look at...

2Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Software Engineering?

3Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Software Engineering?

Creating Software is a Craft.

Converting source code to executable is the engineering bit.

3Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

“Software Engineering is the application of

a systematic, disciplined, quantifiable

approach to development, operation and

maintenance of software: that is, the

application of engineering to software.”IEEE Standard Computer Dictionary,

ISBN 1-55937-079-3, 1990

IEEE defines Software Engineering as...

4Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Who used Software Engineering?

5Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Who used Software Engineering?

5Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

For the space shuttle’s operating system

6Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Some StatisticsNASA’s Defect Density

7Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

The last 11 versions of the space shuttle’s 420,000 line systems

had a total of 17 defects.

Some StatisticsNASA’s Defect Density

7Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

The last 11 versions of the space shuttle’s 420,000 line systems

had a total of 17 defects.

Some StatisticsNASA’s Defect Density

7Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

One More Data Point

8Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

One More Data Point

8Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Another real software engineering project

9Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Safeguard - Ballistic Missile Defense System

Another real software engineering project

9Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

1969-1975, 5407 person years

Hardware designed at the same time as software specs being written

Late changes in requirements not an option

42

1820

20

reqmts20 %

design20 %

code &unit test

18 %

integrationtesting42 %

Safeguard - Ballistic Missile Defense System

Another real software engineering project

9Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

1969-1975, 5407 person years

Hardware designed at the same time as software specs being written

Late changes in requirements not an option

42

1820

20

reqmts20 %

design20 %

code &unit test

18 %

integrationtesting42 %

Safeguard - Ballistic Missile Defense System

Did it Succeed?

Another real software engineering project

9Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

SafeguardBallistic Missile Defense System…

10Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Revised Project Statistics

SafeguardBallistic Missile Defense System…

10Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Revised Project StatisticsThe project was delivered according to specifications

SafeguardBallistic Missile Defense System…

10Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Revised Project StatisticsThe project was delivered according to specifications

Cost: $25 Billion (not adjusted)

SafeguardBallistic Missile Defense System…

10Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Revised Project StatisticsThe project was delivered according to specifications

Cost: $25 Billion (not adjusted)

1969-1975, 5407 person years

SafeguardBallistic Missile Defense System…

10Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Revised Project StatisticsThe project was delivered according to specifications

Cost: $25 Billion (not adjusted)

1969-1975, 5407 person years

Operational for 133 days - Project terminated in 1978

SafeguardBallistic Missile Defense System…

10Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Revised Project StatisticsThe project was delivered according to specifications

Cost: $25 Billion (not adjusted)

1969-1975, 5407 person years

‘By the time the 6-year anti-missile system project was completed, the new missiles were faster than the anti-

missile missiles’

Operational for 133 days - Project terminated in 1978

SafeguardBallistic Missile Defense System…

10Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Where did things go wrong?

11Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Software Engineering is a heavy weight methodology and such heavy weight methodologies characteristically are most successful when:

Requirements are stable

Technology is well known and mature

Everything happens as one would expect

We are not taking on anything new or unknown

We have done this many times before

Where did things go wrong?

11Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Projects with these characteristics are few and far between.

Software Engineering is a heavy weight methodology and such heavy weight methodologies characteristically are most successful when:

Requirements are stable

Technology is well known and mature

Everything happens as one would expect

We are not taking on anything new or unknown

We have done this many times before

Where did things go wrong?

11Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

SEI/IEEE Project Standards and

DefinitionsWaterfall

Requirements ManagementRUP

Heavy Weight

Other Heavy Weight Methodologies

12Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Heavy weight methodologies work in some instances, but there are high costs, and the risk in using them in

dynamic environments is high.

SEI/IEEE Project Standards and

DefinitionsWaterfall

Requirements ManagementRUP

Heavy Weight

Other Heavy Weight Methodologies

12Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So, heavy weight methodologies don’t seem to meet our needs

Is there an alternative?

13Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agenda for this session

Some examples of heavy weight process

Lean Thinking

Origin of Agile

Agile Values and Principles

Summary of how Agile helps

Where do things go wrong?

Naked Agile

14Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Traditional cost profile

Lower cost of change through higher quality software

15Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Traditional cost profile

Time spent:• Finding defects• Fixing Defects• Regression Testing• Deploying

Lower cost of change through higher quality software

15Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agile system cost profile

Traditional cost profile

Time spent:• Finding defects• Fixing Defects• Regression Testing• Deploying

More defects found

& corrected – causes lower cost of defect

correction

Lower cost of change through higher quality software

15Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

The Toyota Production System, 1988 (1978), Taichii Ohno

Pull Scheduling - Just-in-Time Flow

Expose Problems - Stop-the-Line Culture

Study Of ‘Toyota’ Production System, 1981, Shigeo Shingo

Non-Stock Production - Single Minute Setup

Zero Inspection – Automatic Error Detection at Every Step

Lean Thinking – Eliminate Waste

16Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Building Quality into the Process

Toyoda Loom

17Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh JainSource: Beyond Agile Software Development Becoming Lean, Mary Poppendieck, Poppendieck.llc

Utilization (%)

Lessons from Queuing theory

18Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Traditional Process

19Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Traditional Process

19Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Applying Lean Principles to Software Development

20Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

End-to-End small slices of

work

Applying Lean Principles to Software Development

20Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

End-to-End small slices of

work 20 % done = 100 % usable

Applying Lean Principles to Software Development

20Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Fix / Integrate $

Test

Code

DesignSpecifications

Use Cases / Functional Specs

Requirements Gathering

Project Plan/Estimation

$

Inception

$

$

$

Lean Principles applied to Software Development

21Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agenda for this session

Some examples of heavy weight process

Lean Thinking

Origin of Agile

Agile Values and Principles

Summary of how Agile helps

Where do things go wrong?

Naked Agile

22Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

XP

ScrumFDD

DSDMCrystal Family

New Methodologies Emerged

23Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

2000

24Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

FDD | Feature Driven Development (Jeff DeLuca)

DSDM | Dynamic System Development Method (Dane Faulkner)

Adaptive Software Development (Jim Highsmith)

Crystal (Alistair Cockburn)

SCRUM (Ken Schwaber)

XP | Extreme Programming (Kent Beck)

Lean Software Development (Mary Poppendieck)

2000

24Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain25Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

2001Agile

manifesto

25Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

2001 “We are uncovering better ways of developing software by doing

it and helping others do it. Through this work we have come to value:

Agile manifesto

25Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

2001 “We are uncovering better ways of developing software by doing

it and helping others do it. Through this work we have come to value:

– Individuals and interactions OVER processes and tools. Agile manifesto

25Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

2001 “We are uncovering better ways of developing software by doing

it and helping others do it. Through this work we have come to value:

– Individuals and interactions OVER processes and tools. – Working software OVER comprehensive documentation.

Agile manifesto

25Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

2001 “We are uncovering better ways of developing software by doing

it and helping others do it. Through this work we have come to value:

– Individuals and interactions OVER processes and tools. – Working software OVER comprehensive documentation. – Customer collaboration OVER contract negotiation.

Agile manifesto

25Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

2001 “We are uncovering better ways of developing software by doing

it and helping others do it. Through this work we have come to value:

– Individuals and interactions OVER processes and tools. – Working software OVER comprehensive documentation. – Customer collaboration OVER contract negotiation. – Responding to change OVER following a plan.

Agile manifesto

25Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

2001 “We are uncovering better ways of developing software by doing

it and helping others do it. Through this work we have come to value:

– Individuals and interactions OVER processes and tools. – Working software OVER comprehensive documentation. – Customer collaboration OVER contract negotiation. – Responding to change OVER following a plan.

That is, while there is value in the items on the right, we value the items on the left more.”

© 2001 Agile Alliance. http://www.agilemanifesto.org

Agile manifesto

25Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agenda for this session

Some examples of heavy weight process

Lean Thinking

Origin of Agile

Agile Values and Principles

Summary of how Agile helps

Where do things go wrong?

Naked Agile

26Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Principles behind the Manifesto

Our highest priority is to satisfy the customer through early and continuous delivery of valuable software.

Welcome changing requirements, even late in development. Agile processes harness change for the customer's competitive advantage.

Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale.

Business people and developers must work together daily throughout the project.

27Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Principles behind the Manifesto...

Build projects around motivated individuals. Give them the environment and support they need, and trust them to get the job done.

The most efficient and effective method of conveying information to and within a development team is face-to-face conversation.

Working software is the primary measure of progress.

Agile processes promote sustainable development. The sponsors, developers, and users should be able to maintain a constant pace indefinitely.

28Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Principles behind the Manifesto...

Simplicity--the art of maximizing the amount of work not done--is essential.

Continuous attention to technical excellence and good design enhances agility.

The best architectures, requirements, and designs emerge from self-organizing teams.

At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.

29Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agile is Designed to deal with

30Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agile is Designed to deal with

Ziv's law - specifications will never be fully understood.

30Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agile is Designed to deal with

Ziv's law - specifications will never be fully understood.

Humphrey's law - the user will never know what they want until after the system is in production (maybe not even then)

30Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agile is Designed to deal with

Ziv's law - specifications will never be fully understood.

Humphrey's law - the user will never know what they want until after the system is in production (maybe not even then)

Wegner's lemma - an interactive system can never be fully specified nor can it ever be fully tested.

30Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agile is Designed to deal with

Ziv's law - specifications will never be fully understood.

Humphrey's law - the user will never know what they want until after the system is in production (maybe not even then)

Wegner's lemma - an interactive system can never be fully specified nor can it ever be fully tested.

Langdon's lemma - software evolves more rapidly as it approaches chaotic regions (taking care not to spill over into chaos)

30Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agile is Designed to deal with

Ziv's law - specifications will never be fully understood.

Humphrey's law - the user will never know what they want until after the system is in production (maybe not even then)

Wegner's lemma - an interactive system can never be fully specified nor can it ever be fully tested.

Langdon's lemma - software evolves more rapidly as it approaches chaotic regions (taking care not to spill over into chaos)

Any association of predictive or defined processes with Agile is an exercise in futility. - Jeff

30Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agenda for this session

Some examples of heavy weight process

Lean Thinking

Origin of Agile

Agile Values and Principles

Summary of how Agile helps

Where do things go wrong?

Naked Agile

31Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Helps handle changing requirements & priorities

Lowers cost of change

Provides better visibility into project progress

Reduces risk

Maximizes Return on Investment (business value prioritized)

Encourages higher quality, simpler code

Delivers business value early & often

Use of Agile Methods

32Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

But...with....

33Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Courage!!

But...with....

33Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Courage!!

Constant Business involvement

But...with....

33Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Courage!!

Constant Business involvement

A need for more Discipline

But...with....

33Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Courage!!

Constant Business involvement

A need for more Discipline

Greater emphasis on Testing

But...with....

33Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Courage!!

Constant Business involvement

A need for more Discipline

Greater emphasis on Testing

Whole Organization Involvement

But...with....

33Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Courage!!

Constant Business involvement

A need for more Discipline

Greater emphasis on Testing

Whole Organization Involvement

Last but not the least, with an OPEN MIND

But...with....

33Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agenda for this session

Some examples of heavy weight process

Lean Thinking

Origin of Agile

Agile Values and Principles

Summary of how Agile helps

Where do things go wrong?

Naked Agile

34Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?Do you really think this is how companies are adopting Agile?

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?Do you really think this is how companies are adopting Agile?

Mass adoption of Scrum/XP across the organization

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?Do you really think this is how companies are adopting Agile?

Mass adoption of Scrum/XP across the organization

What about Organizational Transformation & Cultural Changes?

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?Do you really think this is how companies are adopting Agile?

Mass adoption of Scrum/XP across the organization

What about Organizational Transformation & Cultural Changes?

Unreasonable, false expectations

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?Do you really think this is how companies are adopting Agile?

Mass adoption of Scrum/XP across the organization

What about Organizational Transformation & Cultural Changes?

Unreasonable, false expectations

Making Agile a very prescriptive, consistent PROCESS

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?Do you really think this is how companies are adopting Agile?

Mass adoption of Scrum/XP across the organization

What about Organizational Transformation & Cultural Changes?

Unreasonable, false expectations

Making Agile a very prescriptive, consistent PROCESS

Process and Tools OVER Teams and their needs

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?Do you really think this is how companies are adopting Agile?

Mass adoption of Scrum/XP across the organization

What about Organizational Transformation & Cultural Changes?

Unreasonable, false expectations

Making Agile a very prescriptive, consistent PROCESS

Process and Tools OVER Teams and their needs

Scrum Masters and Coaches are the new Process Police (Silver Bullet)

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?Do you really think this is how companies are adopting Agile?

Mass adoption of Scrum/XP across the organization

What about Organizational Transformation & Cultural Changes?

Unreasonable, false expectations

Making Agile a very prescriptive, consistent PROCESS

Process and Tools OVER Teams and their needs

Scrum Masters and Coaches are the new Process Police (Silver Bullet)

Business model built around Agile

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?Do you really think this is how companies are adopting Agile?

Mass adoption of Scrum/XP across the organization

What about Organizational Transformation & Cultural Changes?

Unreasonable, false expectations

Making Agile a very prescriptive, consistent PROCESS

Process and Tools OVER Teams and their needs

Scrum Masters and Coaches are the new Process Police (Silver Bullet)

Business model built around Agile

Certification

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?Do you really think this is how companies are adopting Agile?

Mass adoption of Scrum/XP across the organization

What about Organizational Transformation & Cultural Changes?

Unreasonable, false expectations

Making Agile a very prescriptive, consistent PROCESS

Process and Tools OVER Teams and their needs

Scrum Masters and Coaches are the new Process Police (Silver Bullet)

Business model built around Agile

Certification

Tools vendors

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?Do you really think this is how companies are adopting Agile?

Mass adoption of Scrum/XP across the organization

What about Organizational Transformation & Cultural Changes?

Unreasonable, false expectations

Making Agile a very prescriptive, consistent PROCESS

Process and Tools OVER Teams and their needs

Scrum Masters and Coaches are the new Process Police (Silver Bullet)

Business model built around Agile

Certification

Tools vendors

Training and Coaching by non-practitioners

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

So what makes Agile the new Waterfall?Do you really think this is how companies are adopting Agile?

Mass adoption of Scrum/XP across the organization

What about Organizational Transformation & Cultural Changes?

Unreasonable, false expectations

Making Agile a very prescriptive, consistent PROCESS

Process and Tools OVER Teams and their needs

Scrum Masters and Coaches are the new Process Police (Silver Bullet)

Business model built around Agile

Certification

Tools vendors

Training and Coaching by non-practitioners

Today Agile seems to be full of Ceremony and Dogmatism

35Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Where do things go wrong?Lack of understanding of software development

Software development is not

a Manufacturing process

an Engineering or Construction process

Ziv, Humphrey, Wegner's laws and Langdon's lemma

Mythical man month

Gold rush

Lack of theoretical understanding of the underlying principles

Myths and Fallacies

36Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Theory to understand why Agile works

Theory of Constraints and Lean Thinking

Complex adaptive systems: the science of uncertainty

Cognitive science: the nature of human decision making

Evolutionary psychology & anthropology: the origins of social interaction & its nature

37Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Cynefin Framework

Dave Snowden, Cognitive Edge

38Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Project Design based on Cynefin

Dave Snowden, Cognitive Edge

39Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Agenda for this session

Some examples of heavy weight process

Lean Thinking

Origin of Agile

Agile Values and Principles

Summary of how Agile helps

Where do things go wrong?

Naked Agile

40Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Introducing “Naked Agile”Naked Agile: True essence of Agile without the Process Crap (ceremony and dogmatism)

The Bloat Effect

Iterations and Time-boxes are high ceremony

Cease Inspection; Who needs a separate QA Team

Requirements Considered Harmful

Estimations Considered Harmful

Projecting Velocity is Useless

Simple Design; Yes Simple not Simply Design & Coding is NOT Programming

TDD is not about Testing & TDD is not limited to Unit-tests

Agile != Rapid Software Development; Embrace Simplicity (Art of maximizing work not done)

41Thursday, April 30, 2009

Licensed Under Creative Commons by Naresh Jain

Questions? Thank You

naresh@agilefaqs.com

42Thursday, April 30, 2009