![Page 1: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/1.jpg)
Putnam’s Effort-Duration Trade-Off Law: Is theSoftware Estimation Problem Really Solved?
Han SuelmannOctober 7th, 2014
![Page 2: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/2.jpg)
2
Putnam’s study – reference
L.H. Putnam,
“A generic empirical solution to the macro software sizing and estimating problem,”
IEEE Transactions on Software Engineering,
vol. 4, pages 345 ─ 361,
July 1978.
![Page 3: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/3.jpg)
3
![Page 4: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/4.jpg)
5
Agenda
• Putnam’s study: results and influence• Putnam’s approach• Intermezzo – A statistical pitfall• Critical evaluation:
o dataset is very limitedo model and assumptions are unclearo analysis is incorrect
• Other studies provide no corroboration• Simulation study demonstrate incorrectness
![Page 5: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/5.jpg)
6
![Page 6: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/6.jpg)
7
Putnam’s study: results and influence
Claims:• Generic empirical equations that describe size – effort –
duration relationships.• Method will produce accurate estimates.• Only a few quick reference tables and a pocket calculator
needed.• Trade-off law: K ~ 1 / T4.
Proble
m s
olved!
![Page 7: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/7.jpg)
8
Putnam’s study is very influential
Influence:• incorporated in estimation software• many references• sometimes cited as authoritative
![Page 8: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/8.jpg)
9
Putnam’s approach
1) Gather data on effort (K), duration (T) and size (S).
2) Define difficulty: D = K / T 2
3) Define productivity: P = S / K
4) Find relationship between D and P.Result: P ~ D -0.67
5) Perform basic algebraic manipulations to find relationships between S, T, and K.Result: S = C ∙ K1/3 ∙ T4/3,
and therefore: .4
3
T
SK
![Page 9: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/9.jpg)
10
The crucial relationship…
2T
KD
K
SP
difficulty
prod
uctiv
ity
![Page 10: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/10.jpg)
11
Putnam’s approach
1) Gather data on effort (K), duration (T) and size (S).
2) Define difficulty: D = K / T 2
3) Define productivity: P = S / K
4) Find relationship between D and P.Result: P ~ D -0.67
5) Perform basic algebraic manipulations to find relationships between S, T, and K.Result: S = C ∙ K1/3 ∙ T4/3,
and therefore: .4
3
T
SK
![Page 11: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/11.jpg)
12
Intermezzo – A statistical pitfall
•Two researchers examine relationship between S and K.•Both assume linear relationship.
•Researcher 1 writes K = aS + b
•Researcher 2 writes S = a’K + b’
S
K
S
K
![Page 12: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/12.jpg)
Researcher 2’s does his linear fit
![Page 13: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/13.jpg)
14
Intermezzo – The results are quite different
•Researcher 1 writes K = aS + b and finds (1) K = 1.01 S −0.02.
•Researcher 2 writes S = a’K + b’ and finds (2) S = 0.50 K + 3.0.
•Researcher 2 then derives
(3) K = 2.02 S – 6.2.
![Page 14: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/14.jpg)
Researcher 1: does her fit
![Page 15: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/15.jpg)
16
Intermezzo – A statistical pitfall
•Researcher 1 writes K = aS + b and finds (1) K = 1.01 S −0.02.
•Researcher 2 writes S = a’K + b’ and finds (2) S = 0.50 K + 3.0.
•Researcher 2 then derives
(3) K = 2.02 S – 6.2.
^
^
^
![Page 16: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/16.jpg)
Linear fit: minimising least squares
S
K
![Page 17: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/17.jpg)
18
Critical evaluation (1) – dataset is very limited
• only 13 projects• all US Military• 4 are left out => 9 projects remaining
![Page 18: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/18.jpg)
19
Critical evaluation (2) – model is unclear
size
duration
effort
Putnam does not make clear and consistent choices regarding model structure.
Only one parameter to capture effort-duration interaction
![Page 19: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/19.jpg)
20
Critical evaluation (3) – analysis
CS = K1/3
T /33 3 4
![Page 20: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/20.jpg)
21
Critical evaluation (3) – analysis
C S=K T4 33
![Page 21: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/21.jpg)
22
Critical evaluation (3) – analysis
C= K1/3
T /33 3 4S
^
![Page 22: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/22.jpg)
23
Critical evaluation (3) – analysis
C =K T43 3
S^
![Page 23: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/23.jpg)
24
Critical evaluation (3) – analysis
C=K
T 43
3S^
![Page 24: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/24.jpg)
25
Critical evaluation (4): Difficulty – Productivity relationship
Putnam’s reasoning:
3/2 DP
3/43/1 TKS
3/2ˆ DP
More precisely notated:
??ˆˆ 3/43/2 TKKS
3/2
2
T
K
K
S
![Page 25: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/25.jpg)
26
Other studies − Corroboration by Putnam et al.
Putnam & Putnam, “A data verification of the software fourth power trade-off law,” (Proc. of the Int. Soc. of Parametric Analysts – 6th Annu. Conf., vol. III(I), pp. 443–471, 1984.)
Putnam & Myers, “Measures for excellence – Reliable software on time, within budget”, (Englewood Cliffs: Yourdon, 1992.)
Confirmed that K ~ 1 / T4, but…
Found (Dunsmore et al., 1986) and admitted (Putnam & Myers, n.d.) to be based on circular reasoning.
![Page 26: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/26.jpg)
27
Other studies – No corroboration from Jeffery
Result:•P ~ K−0.47T −0.05
•essentially no productivity – duration relationship•comparison with Putnam’s P ~ K−0.67T 1.33
•no confirmation•strictly speaking: no refutation either
Jeffery (1987):•47 MIS in 4 large organisations•Find P as a function of K and T.
![Page 27: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/27.jpg)
28
Other studies – No corroboration fromBarry, Mukhopadhyay, and Slaughter
Barry, Mukhopadhyay, and Slaughter (2002):
Ansatz: ln K = … + β1 T
Result: β1 = 0.000677 ± 0.000103, p = .031.
So – larger duration predicts larger effort.
![Page 28: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/28.jpg)
29
Other studies – Team size affects effort, so…?Putnam & Myers (n.d.): larger team size predicts larger effort:
Teams of 5 or less have better productivity than teams of 20 or more.
Supported by other studies. Example (Rodríguez et al.):
PDR ~ (average team size)^0.57
But…•translation to effort-duration trade-off unclear•interpretation in terms of causation dubious
![Page 29: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/29.jpg)
30
Several interpretations are possible…
larger team size
more effort
?
?
![Page 30: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/30.jpg)
31
Simulation (1)
Goal: check whether the analysis issues really lead to incorrect results.
Method:•generate simulated data with known structure•analyze simulated data, following Putnam’s approach•check whether results are consistent with assumptions
![Page 31: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/31.jpg)
32
Simulation (2)
Model assumptions:
•Size, effort, and duration are unrelated random numbers.•Log-normal distributions.•1000 projects.
![Page 32: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/32.jpg)
33
Simulation (3) – analysis
![Page 33: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/33.jpg)
34
2T
KD
K
SP
![Page 34: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/34.jpg)
35
Simulation (4) – result
Fit yields:
After transformation:
After some manipulations (same as Putnam’s):
02.067.0 DP
constantln67.0ln DP
4.01.4
1
TK Yet, no
relationship actually exists!
![Page 35: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/35.jpg)
36
Simulation (5) – coincidence?
For convenience, write s = ln S, k = ln K, and t = ln T.
Difficulty and productivity:
•ln D = k – 2t
•ln P = s – k
Derive the slope of P against D:
Follow Putnam closely, finding K ~ T u , with
.4)var(ln
)ln,cov(ln)ln|(ln 22
2
tk
k
D
PDDPB
2
2
21
t
ku
which yields u = − 4 if 8t
k
![Page 36: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/36.jpg)
37
Simulation (6) – resultSK
![Page 37: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/37.jpg)
38
Conclusions
Claims:• Generic equations that
describe size – effort – duration relationships.
• Method will produce accurate estimates.
• Trade-off law: K ~ 1 / T4.
Limited dataset,no corroboration
Not addressed
Faulty analysis,no corroboration
![Page 38: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/38.jpg)
39
Conclusion
No credibility forPutnam’s result
No corroboration
Putnam’s original study
was wrong
![Page 39: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/39.jpg)
40
The bad news
• Handling statistical relationships as if exact.
• Interpreting statistical relationship as causal relationships without sufficient support.
Both issues are rather common in the estimation /
metrics literature.
![Page 40: Iwsm2014 putnam revisited (han suelmann) for publication](https://reader035.vdocuments.us/reader035/viewer/2022062514/558e3b2f1a28aba9228b478d/html5/thumbnails/40.jpg)
Question time