copyright qsm associates, inc. 1 michael c. mah managing partner qsm associates, inc. 75 south...
Post on 22-Dec-2015
215 views
TRANSCRIPT
1Copyright QSM Associates, Inc.
Michael C. MahManaging Partner
QSM Associates, Inc.75 South Church Street
Pittsfield, MA 01201413-499-0988
Fax 413-447-7322e-mail: [email protected]
Presentation for Chicago SPIN
January, 2002
Deadline-Driven Software Project EstimationNegotiating Trade-offs and Risks
Web Site: www.qsma.com
2Copyright QSM Associates, Inc.
Background:QSM Associates
“The Metrics Company”
QSM Software Lifecycle Management Tools (SLIM Suite) Used Worldwide by Fortune 100 Clients to Measure, Estimate, and Control Software Development
“Management By The Numbers” - “Numbers into Pictures”
3Copyright QSM Associates, Inc.
Partial List of QSM Clients
British Telecom EDS
Rockwell Intel BellSouth IBM Global Services
Sprint
Honeywell Computer Sciences Corp
United Technologies
GTE
Compaq
Keane
Boeing
Royal Bank of Canada
Lockheed Martin
Alcatel
4Copyright QSM Associates, Inc.
Worldwide Software Productivity Trend
The Good News: Over the last decade, sw dev’t productivity has increased fairly consistently. 10% faster speed, 25% less cost, about every 2.5 years.
The Bad News: It’s Not Enough. Demand continues to outstrip capacity.
Companies are reporting growing backlogs ranging from 7 months to over 2 years.
The Badder News: Continued labor shortages. The More Badder News: Don’t expect any
schedule relief in an “Internet Speed” economy.
5Copyright QSM Associates, Inc.
Industry Overrun Statistics
$250 Billion + Spent on IT Application Development
31% of Projects Will Be Cancelled, Representing $81 Billion in Losses
52.7% of Projects Will Overrun by >189%
Only 16.2% On-time,Under Budget
But with only 42% Original Functionality!
* Source: Standish Group, Dennis MA,
7Copyright QSM Associates, Inc.
Reality Bites
“We don’t have the luxury of determining our schedules. They’re told to us. Then, given the time frame, we try to tell the client what we can build. In the end, we usually wind up working lots of overtime, because they want everything.”
- Manager of Development Wall St. Financial Firm
8Copyright QSM Associates, Inc.
Size
Probabilityon
Time
Scenario: Size Growth,or “Feature Creep”
50%
75%
25%
A B C
9Copyright QSM Associates, Inc.
Good to Know...
"As impressive as growth of the software industry has been, it is outpaced by growth of software-related litigation. It is not unusual for a large software development organization today to have upwards of 50 active cases on its hands."
Tom DeMarco, Cutter IT Journal
10Copyright QSM Associates, Inc.
“Most litigation ends up focused on [lack of] measurement, management, requirements practice, or some combination thereof.”
“Organizations that can’t or don’t measure themselves in a fairly systematic way are at a huge disadvantage in litigation. If you are deficient at measurement and the other side is on top of it, then the jig is up for you.”
Tim ListerCutter IT Journal
Good to Know...
11Copyright QSM Associates, Inc.
QSM Mixed Application Data Base
Effective Source Lines of Code
Calandar
Months
0.1
1
10
100
1000
100 1000 10000 100000 1000000 10000000
Real Time
Engineering
Info Systems
Impossible Zone
Are Deadlines/Plans in the “Impossible Zone”?
12Copyright QSM Associates, Inc.
Building Your Own Benchmarks
Main Build Time vs. Size
New and Modified Size10 100 1000 10000
Months
0.1
1
10
100 Main Build Effort vs. Size
New and Modified Size10 100 1000 10000
Person-M
onths
0.1
1
10
100
1000
10000
MTTD - 60 Days of Production vs Size
New and Modified Size1 10 100 1000 10000
MT
TD
- 60 Days
0.01
0.1
1
10
100
1000 PI Average and Stand. Dev.
PI
1 3 5 7 9 11 13 15 17 1921 23 25 27 29 31 33 35 37 39
Num
ber of Projects
0
2
4
6
8
All Projects in Sample Avg. 1 Sigma
13Copyright QSM Associates, Inc.
Main Build Time vs. Size
New and Modified Size10 100 1000 10000
Months
0.1
1
10
1002001 BASELINE SCHEDULE TREND
Main Build Effort vs. Size
New and Modified Size10 100 1000 10000
Person-M
onths
0.1
1
10
100
1000
100002001 BASELINE EFFORT TREND
MTTD - 60 Days of Production vs Size
New and Modified Size1 10 100 1000 10000
MT
TD
- 60 Days
0.01
0.1
1
10
100
10002001 BASELINE RELIABILITY TREND
PI Average and Stand. Dev.
PI
1 3 5 7 9 11 13 15 17 1921 23 25 27 29 31 33 35 37 39
Num
ber of Projects
0
2
4
6
8 2001 BASELINE PI
All Projects in Sample Avg. 1 Sigma
Building Your Own Benchmarks
14Copyright QSM Associates, Inc.
Ed Yourdon on “Sizing”..
“Studies by the Carnegie Mellon SEI indicate that the most common failing ofLevel 1 (Ad-hoc) software organizations is an inability to make size estimates accurately.”
15Copyright QSM Associates, Inc.
Ed Yourdon on “Sizing”..
“If you underestimate the size of your next project, common sense says that it doesn’t matter which methodology you use, what tools you buy, or even what programmers you assign to the job.”
16Copyright QSM Associates, Inc.
Many Functional Metrics Can be used to Represent S/W Size
Number of subsystems
Number of entities
Number of function points
Number of modules
Number of objects
Number of programs
Number of SLOC
Number of object instructions
17Copyright QSM Associates, Inc.
Breaking things Down to Size
Existing Code,Database 4GL, Forms, PL/SQL,Pro*C, Reports,Utilities, etc.(Item C)
New Code(Item A)
Modified Code(Item B)
Existing,or “Base Code”
18Copyright QSM Associates, Inc.
When Time & Effort are “Fixed” Something’s Got to Give...
SizeTime
Effort
Defects
19Copyright QSM Associates, Inc.
“Feature Creep” - Impact on Quality with Fixed Schedule
Time Profile
11
12
13
14
15
16
17
18
Mo
nth
s
1 2 3 4 5Solutions
FOC MTTD Profile
1.5
2.0
2.5
3.0
3.5
4.0
4.5
Day
s
1 2 3 4 5Solutions
Size Profile
60
70
80
90
100
110
ES
LOC
(thou
san
ds)
1 2 3 4 5Solutions
Solution 4
TimeEffortUinf CstPk StaffMTTDSize
14.06116.40
106712.922.15
87000
MonthsPM$ 1000PeopleDaysESLOC
100% Prob
38% Prob100% Prob 1% Prob
PI 17.0
Schedule Fixed at 14 Months
Size Increases in 5KSLOCIncrements from 72K to 92K
Quality Levels Cut in Half - MTTD 3.5 Days to 1.75 Days
20Copyright QSM Associates, Inc.
Overall Project Risk:Green (Minimal Risk)
Schedule, Cost, andQuality Targets all at 80% Probability orBetter
Staffing Profile
0
5
10
15
200 1 2 3 4 5 6 7 8
Sta
ff
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 *Jan'97
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'98
Feb Mar Apr May
Feas
FD
MB
Maint
0 = FSR
1 = PDR
2 = CDR
3 = FCC
4 = SIT
5 = UOST
6 = IOC
7 = FOC
8 = 99R
RISKTimeEffortUinf CstMin Pk StaffMax Pk StaffFOC MTTD
% 0 10 20 30 40 50 60 70 80 90 100
TimeEffortUinf CstPk StaffMTTDStart
MonthsPM$ 1000PeopleDaysDate
MB7.49
72.14661
13.001.46
7/6/97
Life Cycle15.50
115.21105613.006.32
1/1/97
Size39473
ESLOC
MBI 4.2PI 16.0
Risk Analysis - Determinethe Probability of Success
Deadline
21Copyright QSM Associates, Inc.
When Schedules/Resources are
Fixed - Assess FunctionalityTime Sensitivity to Size
22.5
25.0
27.5
30.0
32.5
35.0
37.5
40.0
Mo
nth
s
30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60ESLOC (thousands)
FOC MTTD Sensitivity To Size
8
9
10
11
12
13
Ho
urs
30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60ESLOC (thousands)
Uninflated Cost Sensitivity to Size
1.8
2.0
2.2
2.4
2.6
2.8
3.0
3.2
$ (m
illion
s)
30 32 34 36 38 40 42 44 46 48 50 52 54 56 58 60ESLOC (thousands)
Current Solution Alternative Solutions Acceptable Solution RegionLife Cycle includes R&D, C&T, I&P
TimeEffortUinf CstPk StaffMTTDSize
Life Cycle
31.03259.80
238215.0010.4442000
MonthsPM$ 1000PeopleHoursESLOC
MBI 2.0PI 10.5
TargetSchedule
TargetCost
TargetQuality
Size Range to Test
22Copyright QSM Associates, Inc.
Staffing Profile
0.0
2.5
5.0
7.5
10.0
12.5
15.0
17.50 1 2 3 4 5 6 7 8
Sta
ff
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 *Jan'97
Feb Mar Apr May Jun Jul Aug Sep Oct Nov Dec Jan'98
Feb Mar Apr May Jun Jul Aug
Feas
FD
MB
Maint
0 = FSR
1 = PDR
2 = CDR
3 = FCC
4 = SIT
5 = UOST
6 = IOC
7 = FOC
8 = 99R
RISKTimeEffortUinf CstMin Pk StaffMax Pk StaffFOC MTTD
% 0 10 20 30 40 50 60 70 80 90 100
TimeEffortUinf CstPk StaffMTTDStart
MonthsPM$ 1000PeopleDaysDate
MB8.82
65.33599
10.002.05
8/9/97
Life Cycle18.29
104.34956
10.008.88
1/1/97
Size46473
ESLOC
MBI 3.3PI 16.0
Risk Analysis - Determinethe Probability of Success
Overall Project Risk:Red (High Risk)
Only 45% Probabilityof Meeting Target Schedule
Deadline
Cost and Quality - High Probability
Copyright QSM Associates, Inc.
0
10
20
30
40
50
60
70
80
J an Apr J ul Oct J an Apr J ul Oct
Defects Discovered Each Month High defect rate
Low Mean Time to DefectPoor Quality
Low defect rate High Mean Time to Defect
Good Quality
Reliability ModelingMean Time to Defect
24Copyright QSM Associates, Inc.
How DevelopmentLifecycles Behave
A Pop Quiz:
“If you tried to shorten the schedule on an application development project by adding staffto say, double (20 people versus 10), how muchwill you able to compress it? Will defects go upor down, and by how much?”
25Copyright QSM Associates, Inc.
How SoftwareLifecycles Behave
Answer:
Schedules will only compress (nominally) byabout 20 percent.
Defects typically rise by about 6 fold.*
Rule of Thumb: “20/200/6x”
*Source: QSM Industry Database Statistics
Copyright QSM Associates, Inc.
Reliability ModelingDefect Severity Categories
Defect Discovery Plan by Category(Expected 50%)
0
20
40
60
80
100
1201 2 3 4 5 6 7 8 9
Defects
1 4 7 10 13 16 19 22 25 28 31 34 *Jan'93
Apr Jul Oct Jan'94
Apr Jul Oct Jan'95
Apr Jul Oct
* Months from beginning of project start month
Critical
Serious
Moderate
Tolerable
Cosmetic
1 = PDR
2 = CDR
3 = FCC
4 = SIT
5 = UOST
6 = IOC
7 = FOC
8 = 99R
9 = 99.9R
5 Severity Categories
Copyright QSM Associates, Inc.
Total Defect Rate
0
100
200
300
400
500S 754321
Defects
2 5 8 11 14 17 20 23 26 29 32 35 *Jan'95
Apr Jul Oct Jan'96
Apr Jul Oct Jan'97
Apr Jul Oct Jan'98
Actual
Interpolated
Plan
Green CB
Yellow CB
S = Start
1 = PDR
2 = Bld_1
3 = CDR
4 = Bld_2
5 = TRR
7 = Bld_3
Reliability ModelingReal Data - Actual Vs.
PlannedDefects Starting from
Design
Copyright QSM Associates, Inc.
Reliability ModelingReal Data - Actual Vs.
PlannedDefects Starting from Code
Total Defect Rate
0
2
4
6
8
10
12
14
S 75421S 421
Defects
1 5 9 13 17 21 25 29 33 37 *2/4'95
3/4 4/1 4/29 5/27 6/24 7/22 8/19 9/16 10/14
Actual
Interpolated
Plan
Green CB
Yellow CB
S = Start
1 = RB
2 = DD
4 = SIT
5 = UOST
7 = FOC
Copyright QSM Associates, Inc.
These reliability drivers are able to be controlled! Reliability and availability is something that we can INFLUENCE!
Reliability ModelingThe Moral of the Story
SizeStaffing
Productivity
Variables Defects
30Copyright QSM Associates, Inc.
Core Metrics Provide an “Early Warning Indicator”
Gantt Chart
T&E
C&T
R&D
S 1 2 3 4 5 6 7 8S 1 2 3
3 6 9 12 15 18 21 24 27 *Oct'92
Jan'93
Apr Jul Oct Jan'94
Apr Jul Oct Jan'95
Size
0
10
20
30
40
50
60
70S 1 2 3 4 5 6 7 8S 1 2 3
ES
LO
C (th
ou
sa
nd
s)
3 6 9 12 15 18 21 24 27 *Oct'92
Jan'93
Apr Jul Oct Jan'94
Apr Jul Oct Jan'95
Total Cum Cost
0
500
1000
1500
2000
2500
3000
3500S 1 2 3 4 5 6 7 8S 1 2 3
$ (th
ou
sa
nd
s)
3 6 9 12 15 18 21 24 27 *Oct'92
Jan'93
Apr Jul Oct Jan'94
Apr Jul Oct Jan'95
Total Defect Rate
0
20
40
60
80
100S 1 2 3 4 5 6 7 8S 1 2 3
De
fec
ts
3 6 9 12 15 18 21 24 27 *Oct'92
Jan'93
Apr Jul Oct Jan'94
Apr Jul Oct Jan'95
Current Plan Actual Interpolated Green Control Bound Yellow Control Bound Life Cycle includes R&D, C&T, T&ES = Start, 1 = PDR, 2 = INC_1, 3 = INC_2, 4 = SIT, 5 = ST, 6 = TRR, 7 = FOC, 8 = 99R
Schedule/MilestonesAppear on Target
Cost is Closeto the Plan
Product is BeingConstructed at a SlowerRate than Planned
Defect Rates areHigher than Planned
Example
Yellow
YellowGreen
31Copyright QSM Associates, Inc.
Traffic Lights - A VisualTrigger for Course
CorrectionSize
0
10
20
30
40
50
60
70
S 1 2 3 4 5 6 7 8S 1 2 3
ES
LOC
(thousands)
Oct'92
Jan'93
Apr Jul Oct Jan'94
Apr Jul Oct Jan'95
Apr Jul
Actual
Interpolated
Plan
Green CB
Yellow CB
S = Start
1 = HLD
2 = LLD
3 = UIT
4 = SIT
5 = SVT
6 = BT
7 = GA
8 = 99R
Size (ESLOC(K))PI 10.8 9.1 -1.7MBI 1.2 0.6 -0.6
Date 1/2/94 (14.1 mos)
Plan Actual Diff37.12 32.30 -4.82
Data are Consistentlyin the Amber Region
Example
32Copyright QSM Associates, Inc.
I f W e’re O ff C ourse, W here are W e H eaded?
1 3 5 7 9 11 13 15 17 19 21 23
0
5 0 0 0
1 0 0 0 0
1 5 0 0 0
2 0 0 0 0
2 5 0 0 0
3 0 0 0 0
3 5 0 0 0
4 0 0 0 0
4 5 0 0 0
5 0 0 0 0
E S L O C
M o n th s fro m S ta r t
O rig in a l P la n
7 5 % o f P la n9 0 % o f P la n
1 1 0 % o f P la n1 2 5 % o f P la n
W hich of T hese Ideal C urves has the Best Fit to the A ctual D ata?
E x p e c te d S iz e
4 8 ,0 0 0 E S L O C
33Copyright QSM Associates, Inc.
Gantt Chart
T&E
C&T
R&D
S 1 2 3 4 5 6 7 8S 1 2 3 4 5 6 7 8
3 6 9 12 15 18 21 24 27 30 33 *Oct'92
Jan'93
Apr Jul Oct Jan'94
Apr Jul Oct Jan'95
Apr Jul
Size
0
10
20
30
40
50
60
70S 1 2 3 4 5 6 7 8S 1 2 3 4 5 6 7 8
ES
LO
C (th
ou
sa
nd
s)
3 6 9 12 15 18 21 24 27 30 33 *Oct'92
Jan'93
Apr Jul Oct Jan'94
Apr Jul Oct Jan'95
Apr Jul
Total Cum Cost
0
500
1000
1500
2000
2500
3000
3500S 1 2 3 4 5 6 7 8S 1 2 3 4 5 6 7 8
$ (th
ou
sa
nd
s)
3 6 9 12 15 18 21 24 27 30 33 *Oct'92
Jan'93
Apr Jul Oct Jan'94
Apr Jul Oct Jan'95
Apr Jul
Total Cum Defects Remaining
0
200
400
600
800
1000
1200S 1 2 3 4 5 6 7 8S 1 2 3 4 5 6 7 8
De
fec
ts
3 6 9 12 15 18 21 24 27 30 33 *Oct'92
Jan'93
Apr Jul Oct Jan'94
Apr Jul Oct Jan'95
Apr Jul
Current Plan Actual Interpolated Current Forecast Green Control Bound Yellow Control Bound Life Cycle includes R&D, C&T, T&ES = Start, 1 = PDR, 2 = INC_1, 3 = INC_2, 4 = SIT, 5 = ST, 6 = TRR, 7 = FOC, 8 = 99R
Adaptive Forecasting: What’s the Remaining “Trajectory”?
Forecasted Schedule:+5.5 Months
Forecasted Cost:+$620K
Forecasted Code Production:Planned PI 10.8, Actual PI 9.1
40 Total Defects WillBe Remaining at theInitial Production Date
34Copyright QSM Associates, Inc.
Case Study:Case Study:Original Plan vs. Actual DataOriginal Plan vs. Actual Data
Gantt Chart
C&T
S 7654321S 7654321
1 9 17 25 33 41 *2/17'01
4/14 6/9 8/4 9/29 11/24
Size
0
20
40
60
80
100
S 7654321S 7654321
ES
LOC
(thousands)
1 9 17 25 33 41 *2/17'01
4/14 6/9 8/4 9/29 11/24
Aggregate Staffing Rate
0
5
10
15
20
25
S 7654321S 7654321
People
1 9 17 25 33 41 *2/17'01
4/14 6/9 8/4 9/29 11/24
Total Defect Rate
0
10
20
30
40
50
S 7654321S 7654321
Defects
1 9 17 25 33 41 *2/17'01
4/14 6/9 8/4 9/29 11/24
Total Cum Effort
0
20
40
60
80
100
120
140
S 7654321S 7654321
SM
1 9 17 25 33 41 *2/17'01
4/14 6/9 8/4 9/29 11/24
Elapsed WeeksSize (ESLOC(K))Agg. StaffTotal Defect RateTotal Cum Effort (SM)PI 19.6 16.5 -3.0MBI 4.8 3.8 -1.0
Date 7/25/2001 (23.53 weeks)
PlanActual/
Forecast Diff23.00 23.00 0.0068.25 58.00 -10.2516.13 15.00 -1.13
6 31 2558.18 59.56 1.37
Current Plan Actual Interpolated Current Forecast Green Control Bound Yellow Control Bound Life Cycle includes C&TS = Start, 1 = IDDC, 2 = FDDC, 3 = SIT, 4 = CCUT, 5 = CSIT, 6 = SUOST, 7 = FOC
Actual Data (black squares)
Plan Data Bounds
Forecast Data (white squares)
Example
35Copyright QSM Associates, Inc.
Gantt Chart
C&T
S 7654321S 7654321
1 9 17 25 33 41 *2/17'01
4/14 6/9 8/4 9/29 11/24
Size
0
20
40
60
80
100
S 7654321S 7654321
ES
LOC
(thousan
ds)
1 9 17 25 33 41 *2/17'01
4/14 6/9 8/4 9/29 11/24
Aggregate Staffing Rate
0
5
10
15
20
25
S 7654321S 7654321
People
1 9 17 25 33 41 *2/17'01
4/14 6/9 8/4 9/29 11/24
Total Defect Rate
0
10
20
30
40
50
S 7654321S 7654321
Defe
cts
1 9 17 25 33 41 *2/17'01
4/14 6/9 8/4 9/29 11/24
Total Cum Effort
0
25
50
75
100
125
150
S 7654321S 7654321
SM
1 9 17 25 33 41 *2/17'01
4/14 6/9 8/4 9/29 11/24
Elapsed WeeksSize (ESLOC(K))Agg. StaffTotal Defect RateTotal Cum Effort (SM)PI 16.7 16.6 -0.1MBI 3.8 3.7 -0.0
Date 7/29/2001 (24.14 weeks)
PlanActual/
Forecast Diff23.57 23.57 0.0063.12 58.32 -4.8015.00 15.05 0.05
14 29 1561.66 61.66 0.00
Current Plan Actual Interpolated Current Forecast Green Control Bound Yellow Control Bound Life Cycle includes C&TS = Start, 1 = IDDC, 2 = FDDC, 3 = SIT, 4 = CCUT, 5 = CSIT, 6 = SUOST, 7 = FOC
Case Study:Case Study:Re-Plan vs. Actual DataRe-Plan vs. Actual Data
Defects currently tracking in high range; probably due to early aggressive schedule
Forecast is on target with plan
Example
36Copyright QSM Associates, Inc.
Project Office Warboard(Currently in Use by QSM Clients)
Project Sched Cost Size Qlty Ovrll Plan Forecast
InvRptg Jun97 May97
RealFin May97 Apr97
Frcstng Jan97 Aug97
Regtn Jul97 Jun97
PropMgt Aug97 Aug97
*
* underrun ** no data
**
**
37Copyright QSM Associates, Inc.
In Summary...
It Comes Down to Promises, Commitments, and Expectations!
Understand that SW Dev’t is R&D, and It Behaves That Way! Non-linear interdependencies.
Apply “Negotiation on the Merits” Generate Multiple Options
Test Each Option for Legitimacy, Reasonableness, & Risk
“Know Your Capability” – Estimates Based on History
38Copyright QSM Associates, Inc.
The Role of Measurement, and Commitments
Estimation& Planning
Control&
Forecasting
Support FutureCommitments
ManageCommitmentCommitment
Analyze Performance on Commitment
HistoryRepository
Assess Viable Strategies
Monitor Status & ReplanPost Project Analysis
Make Commitment
39Copyright QSM Associates, Inc.
Info Sources on the Web
Software Measurement, Estimation, ControlQSM Associates - www.qsma.com
Information Technology Research PubsCutter Consortium - www.cutter.com/consortium
NegotiationProgram on Negotiation at Harvard Law - www.pon.harvard.edu
Workshops from QSM Associates/Triad Consulting - www.qsma.com/education.html
40Copyright QSM Associates, Inc.
RecommendedReading - Negotiation
Fisher, Roger and Alan Sharp, “Getting It Done, How to Lead When You’re Not in Charge” HarperCollins 1998.
Fisher, Roger, William Ury and Bruce Patton, “Getting to YES, Negotiating Agreement Without Giving In” Penguin 1981.
Heen, Sheila, Doug Stone and Bruce Patton “DifficultConversations - How to Discuss What Matters Most” Viking/Penguin 1999.
41Copyright QSM Associates, Inc.
RecommendedReading - Metrics
Carleton, Anita, Park, Robert, and Goethert, Wolfhart , “The SEI Core Measures: Background Information and Recommendations for Use and Implementation” © 1994 The Journal of the Quality Assurance Institute.
Mah, Michael C., “Software Estimation Tricks of the Trade;Secrets They Never Told Me” IT Metrics Strategies© June 2000 Cutter Information Corp.
Putnam, Lawrence H., and Myers, Ware, “Executive Briefing: Controlling Software Development” © 1996 IEEE Computer Society Press.
Tufte, Edward, “Visual Explanations, Images and Quantities,Evidence and Narrative” © 1997 Graphics Press.