The Impact of Methods and

Techniques on Outcomes

from Agile

Software Development

Projects

David Parsons, Hokyoung

Ryu, Ramesh Lal

Agenda

Analysis of Scott Ambler’s 2006 survey

Deeper statistical analysis

• Do agile methods have a positive effect?

• Do agile methods work better in combinations?

• What are the most effective methods, individually

and in combination

• What are the most effective techniques?

Ambler’s Survey

March 2006

Used mailing lists from both Dr. Dobb’s Journal and Software Development magazine

Hosted on line using SurveyMonkey.

Summary article published in Dr. Dobbs Journal (September 2006)

Raw data available for download• www.ambysoft.com/surveys/

4,232 respondents (good sample size!)

Survey Questions

Please rate your skills at the following IT activities…

Number of IT people in your organization

How would you rate your knowledge of agile methodologies?

Which agile methodologies has your organization adopted?

Which agile techniques has your organization adopted? (check all that apply)

How have agile approaches affected your productivity?

How have agile approaches affected the quality of the systems produced?

How have agile approaches affected the cost of development?

How have agile approaches affected the satisfaction of your business stakeholders in the work produced?

Ambler’s conclusions

‘adoption of agile processes has clearly

been a resounding success’

• Productivity, satisfaction and quality all improved

• (Ambler did not mention that cost also increased)

Is this true?

Can we ‘prove’ it?

Adopting and Adapting Agile

Methods

Adoption and adaptation of agile methods

is a complex process

Social, organizational, and technical factors

that may influence why an agile technique

may or may not be used by a particular

software development team.

Our Observations

Many organisations are using more than

one method

• Is there any point?

Techniques and methods don’t seem to be

used as one might expect

• What is the underlying relationship?

WARNING!

Approach the rest of this presentation with

caution

‘There are three kinds of lies: lies, damned

lies, and statistics’

• Disraeli

Do Different Methods Have

Different Outcomes?

Outcomes from Agile Methods

0.00

0.50

1.00

1.50

2.00

2.50

3.00

3.50

4.00

4.50

Productivity Quality Cost Satisfaction

Outcomes

Va

lue

Agile MSF

Agile UP

Crystal

DSDM

FDD

Scrum

XP

Other Method

No Agile Method

Very murky data!

Does the Use of Techniques

Vary Between Methods?

Percentage usage of techniques within methodologies

0

10

20

30

40

50

60

70

80

Active

stak

eholde

r par

ticipat

ion

Agile M

odel D

riven

Dev

elop

men

t (AM

DD)

Cod

e re

factor

ing

Cod

e re

gres

sion

testing

Co-

loca

tion

Com

mon

cod

ing

guidelines

Con

tinuo

us in

tegr

ation

Dat

abas

e re

factor

ing

Dat

abas

e re

gres

sion

testing

Pair p

rogr

amm

ing

Single

sour

cing

info

rmat

ion

Test D

riven

Des

ign

(TDD)

No

Agile T

echn

ique

s

Agile Techniques

Ag

ile M

eth

od

olo

gie

s

Agile UP

AgileMSF

Crystal

DSDM

FDD

Scrum

XP

None

Other

Very murky data!

The Data is Deceptive Because

Lots of ‘don’t knows’

Large variations in sample sizes• DSDM used by only a handful, Crystal Clear also

not a large sample, unwise to draw too many conclusions

Lots of overlap between methods

Lack of clarity about the relationship between techniques and methods• For example ‘we are doing XP’…

How Many Methods?

No agile methods: 2,541

One agile method: 1,019

Two agile methods: 500

Three or more methods: 175

Does this make any difference?

Our Approach

Do a ‘proper’ statistical analysis• Non parametric analysis (Tukey test) to identify

significantly different groupings

Remove all the ‘don’t knows’

Separate out those using zero, one, two three (four, five, seven!) methods• We only want to count them once!

Analyse techniques separately from methods

Some Things to be Aware of

Respondents almost certainly

misunderstood Feature Driven

Development and over-responded

Respondents may well have

misrepresented their use of Test Driven

Development and over-responded

Result 1: Number of Methods

Cost• no difference

Productivity• significant difference between using no method or using

one

Quality and Satisfaction• significant difference between using no method or using

one AND between using one method and using two!

So:• Use at least one, preferably two

• No point using three or more

Result 2: If I Use One, Which

One?

Productivity:• There is a significant different between both Crystal Clear and

Scrum, (0.02) and Crystal Clear and XP (0.02)

Quality• There is a significant difference between Crystal Clear and XP

(0.03)

Cost• There is a significant difference in cost between Crystal Clear

and Scrum (0.02)

Satisfaction• There is a significant difference between Crystal Clear and

Scrum (0.023)

So: Maybe use XP or Scrum?

Which Pairs are Used Most?Agile Method Pairs

0 20 40 60 80 100 120 140 160

Crystal Clear and FDD

Agile MSF and FDD

XP and Other

Crystal Clear and XP

Agile UP and Scrum

Agile UP and FDD

FDD and Scrum

Agile MSF and XP

Agile UP and XP

XP and Scrum

XP and FDD

Agile Method Pairs

Result 3: If I Use Two, Which

Two

Quality• There is a significant difference between the XP/Scrum

combination and all other pairs of methods.

• Only 6 pairs were considered in the sample because the sample sizes were too small for other combinations.

Productivity• There is a significant difference between the top pair (XP

and Scrum) and the lowest three pairs

There was no significant difference in cost between any pairs of methods

So: Mix XP and Scrum together?

Agile Techniques

Techniques Number of

respondents

Percentage of

respondents

Active stakeholder participation 938 22.15%

Agile model driven development 260 6.14%

Code refactoring 1,467 34.64%

Code regression testing 1,383 32.66%

Co-location 447 10.55%

Common coding guidelines 1,595 37.66%

Continuous integration 1,113 26.28%

Database refactoring 416 9.82%

Database regression testing 407 9.61%

Pair programming 587 13.86%

Single sourcing 241 5.69%

Test driven design 959 22.64%

What About Techniques?

Not using any agile techniques is significantly worse in terms of productivity, satisfaction and quality

Only DB regression testing is significantly more expensive

Best techniques for productivity

• Co-location, continuous integration, pair programming, test driven design

Best techniques for Quality

• Test driven design, pair programming, continuous integration,

Best techniques for Satisfaction

• Active stakeholder participation, agile model driven development, test driven design

Are We Doing XP?

Agile Technique used with XP Number Percentage of Sample

Active stakeholder participation 114 27.14%

Code refactoring 269 64.05%

Code regression testing 210 50.00%

Co-location 66 15.71%

Continuous integration 176 41.90%

Pair programming 183 43.57%

Test Driven Design (TDD) 180 42.86%

Does it matter?

Productivity Quality Cost Satisfaction

‘Full’ XP 4.25 4.39 2.70 4.07

Partial XP3.62 3.77 2.63 3.55

Best Practices?

In the XP context, the most effective

techniques appear to be

• Test driven design

• Code refactoring

• Collaborative working

• Stakeholder involvement

• Co-location

How Many XP Techniques?

1

2

3

4

5

0 1 2 3 4 5 6 7

Number of XP Techniques

Productivity

Quality

Cost

Satisfaction

Tentative Conclusions

Using agile methods and techniques does appear to be beneficial

It may be useful to combine aspects of XP and Scrum

These techniques appear to consistently contribute to better results• Co-location, pair programming, test driven

design

More techniques = better?

Caveat Emptor

This was not our data set and we cannot

guarantee its validity

However its conclusions can help us to

direct our future research to refute or

validate them