a structured approach to estimating software intensive projects - rené berghuijs and shaun irvin

A Structured Approach to Estimating

Software Intensive Projects

René Berghuijs and Shaun Irvin

10 March 2015

10/03/2015 NATO UNCLASSIFIED 1

• NATO is a military, political Alliance

headquartered in Brussels, Belgium

• Core business: peace and security

• The NATO Communications & Information

Agency is NATO’s acquisition organization, with

entities in Brussels, The Hague and Mons (BEL)

• In today’s context NATO is a user organization


Introduction

http://www.google.be/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://www.stanleyfoundation.org/courier/1998spring3.html&ei=l4jwVJeOMpfXaqXdgsAK&bvm=bv.87269000,d.d2s&psig=AFQjCNGSuDI2-uSUTlcufi--UVAqdSS5ng&ust=1425136130373506

http://www.google.be/url?sa=i&rct=j&q=&esrc=s&frm=1&source=images&cd=&cad=rja&uact=8&ved=0CAcQjRw&url=http://www.stanleyfoundation.org/courier/1998spring3.html&ei=l4jwVJeOMpfXaqXdgsAK&bvm=bv.87269000,d.d2s&psig=AFQjCNGSuDI2-uSUTlcufi--UVAqdSS5ng&ust=1425136130373506


Software Estimating Progression

Rene’s NATO

Experience

Early Days of NATO Software

Estimating

NATO Reform & Software Intensive

Task Force

NATO Software

Estimating Today


Rene’s NATO estimating experience

• Hired in 2005 to do independent SW cost estimating for changes and block upgrades to NATO’s AIR C2 program– Previous experience as HW cost estimator for production

plants of (petro) chemicals

• Familiarization with SW cost estimating through:– Literature (Barry Boehm, Capers Jones)

– Training on COTS products

– Joining cost society (ISPA)

• NATO Theatre Missile Defense program, based on the ACCS program (nowadays integrated Air & Missile Defense)

• Ingredients of the cost estimate:– Requirements driven

– Function Point Analysis

– COTS estimating program (calibrated)

– Validation

• Outcome: an investment cost estimate used in the funding request to the nations– Screened by WGNTEs, no comments on cost

– Authorized by the then Investment Committee


First big estimating project

Lack of Analytical Expertise

• Budget already set

• Type-B Cost Estimate (TBCE) based on technical expert level of effort (LOE) input

• Speculation at lower levels led to large error margins

• Estimates not defendable or repeatable

• “Catch-22” with NATO Office of Resources (NOR) and requirements

Cost/Schedule Overruns

• Expert opinion ALWAYS led to underestimation

• No BOEs or baselines to track contractor progress or support during negotiations


NATO Early Days

Software Estimating


LOE Estimating

MAIN GROUP ACTIVITY WORK Labor Duration Total Unit Cost Subtotal Total Cost Total Labor Group Cost

Project Manager 1.0 24.0 24.0 20,000€ 480,000€

Technical Management 1.0 24.0 24.0 20,000€ 480,000€

Configuration Management 0.2 24.0 4.8 18,000€ 86,400€

Quality Management 0.2 24.0 4.8 18,000€ 86,400€

Test Management 0.2 24.0 4.8 18,000€ 86,400€

Contract Management 0.2 24.0 4.8 18,000€ 86,400€

Finance Management 0.2 24.0 4.8 18,000€ 86,400€

Project Admin Office 0.2 24.0 4.8 15,000€ 72,000€

IT System Admin 0.2 24.0 4.8 18,000€ 86,400€

Other Staff Contribution 0.2 24.0 4.8 10,000€ 48,000€

System Analysis (SME) 4.0 2.0 8.0 20,000€ 160,000€

SSS 2.0 1.0 2.0 16,000€ 32,000€

System Requirements Review (SRR) 4.0 0.2 0.8 18,000€ 14,400€

System Design 2.0 1.0 2.0 20,000€ 40,000€

TA, ICD 4.0 1.0 4.0 18,000€ 72,000€

SSDD 2.0 1.0 2.0 18,000€ 36,000€

Critical Design Review (CDR) 6.0 0.2 1.2 18,000€ 21,600€

Software Requirements Analysis 8.0 4.0 32.0 20,000€ 640,000€

SRS, IRS 4.0 2.0 8.0 16,000€ 128,000€

Software Design 8.0 4.0 32.0 18,000€ 576,000€

SDD, IDD 4.0 2.0 8.0 16,000€ 128,000€

Software Implementation (coding) 18.0 20.0 360.0 16,000€ 5,760,000€ 360.0 5,760,000€

Software Unit Testing 10.0 6.0 60.0 16,000€ 960,000€ 60.0 960,000€

Specification, Integration and Test 2.0 1.0 2.0 18,000€ 36,000€

Procurement 2.0 0.5 1.0 18,000€ 18,000€

Test Plan Preparation 2.0 0.5 1.0 20,000€ 20,000€

Test Procedures 4.0 2.0 8.0 16,000€ 128,000€

Test Readiness Review (TRR) 10.0 0.1 1.0 18,000€ 18,000€

Pilot System Installation and Test 6.0 2.0 12.0 16,000€ 192,000€

Factory Acceptance Test - FAT 15.0 1.0 15.0 18,000€ 270,000€

System Acceptance Test - SAT 10.0 1.0 10.0 18,000€ 180,000€

808,000€

Development Systems Engineering

376,000€ 20.0 376,000€

Software DevelopmentDetailed in Sheet:

WP1 Software Dev.8,192,000€

80.0 1,472,000€

Hardware Development54,000€ 3.0 54,000€

System Integration

and Testing

808,000€ 47.0

1,392,000€

Support Functions

14.4 206,400€

WP 1: XXXX Implementation

Management Project Management

1,598,400€

72.0

Lack of Analytical Expertise

• Budget already set

• Type-B Cost Estimate (TBCE) based on technical expert level of effort (LOE) input

• Speculation at lower levels led to large error margins

• Estimates not defendable or repeatable

• “Catch-22” with NATO Office of Resources (NOR) and requirements


• Expert opinion ALWAYS led to underestimation

• No BOEs or baselines to track contractor progress or support during negotiations


NATO Early Days

Software Estimating

• NCI Agency was established 2012 as the outcome of NATO’s Agency Reform – merger of NC3A, NACMA etc.

• Founding Principles:

• Customer Funding

• Full Project Lifecycle

• Best Practices

• Service Lines and Program Offices

• Centralized Cost Capability:

• Four Cost Professionals co-located in Brussels

• Common tools (SEER suite)

• Using shared historical data

• Setting up new policies and processes


Birth of NCI Agency

• Established in Dec 2013 to analyze cost and

schedule overruns in NATO software projects

• Goal: To identify specific process improvements

for better project management


Software Intensive Project

Task Force



• Initial analysis of 27 recently implemented SIPs

• 5 cancelled or failed with little or no capability

• 10 completed projects with severe cost/schedule impacts (>4 years

late or >30% over budget

• 10 projects still being implemented (average 3 years late)

• Only 2 completed less than 2 years late and less than 15% over

budget

• Independent, data-driven, lifecycle cost estimating is truly in NATO’s DNA:– Software Intensive Projects are now estimated using

Function Point Analysis and SEER tools

– SIPs are recognized as being more risky than other projects; contingencies are now project specific

– Full Life-cycle analysis supports O&M Phase

• Estimate baselines are used in the analysis of contractor ECPs through:– Audited Labour rates (basic rates, overhead and general /

administrative)

– Tracking contractor’s performance across life-cycle (productivity, quality, schedule)


Fast Forward to Today…


Function Point Counting






administrative)





SEER Outputs

1.11 Program - CTE Missions Application

Activity Proj Mgr Analyst Design Program Data Test/QA CM/RM QC Lead Total

Sys Reqs 10 44 12 0 5 10 2 2 85

SW Reqs 30 115 35 15 15 30 5 5 249

Pre Design 169 153 629 184 123 215 31 31 1,533

Det Design 93 84 346 101 68 118 17 17 844

Code 273 117 234 2,148 234 586 156 156 3,905

Int & Test 47 12 23 227 47 169 29 29 583

Prog Test 31 8 16 153 31 114 20 20 393

Sys I&T 148 37 74 352 19 1,092 93 37 1,851

Development Total 801 570 1,369 3,180 541 2,334 352 296 9,443

Maint 857 214 429 4,072 107 4,286 536 214 10,716

Life Cycle Total 1,658 784 1,797 7,252 648 6,620 888 511 20,159






administrative)





Risk Analysis & Uncertainty

Risk #01 JTS interim system continues to evolve, making it a moving target and endangering the user acceptance of the N-JTS.

Probability High

Description As N-JTS delivery is only possible after a lengthy NSIP procurement (approximately 3-4 years after 1st stage authorization), JTS interim system might be asked to implement emerging NATO requirements, making it a “moving target” for the N-JTS. As a result, the user community may not be motivated to embrace the N-JTS, as it will eventually be delivered with less functionality than the interim system. On the other hand, incorporating these charges can delay the project and add additional costs.

Impact Medium

Scope Impact

Yes

Schedule Impact

Up to 6 months

Cost Impact Up to € 500,000

Mitigation 1 Limit changes to JTS to MMR, governed by the IPMT. During Execution

2 Foreseen additional budget/time to incorporate necessary changes.

Not included in current proposal

3 Introducing early releases will allow to re-prioritise some of the solution features to ensure that those features that are mandatory are included

Included in current proposal

• Schedule and Cost impact

factored into Contingency plan

• Higher confidence level used in

estimates






administrative)




0

2

4

6

8

10

12

2018 2019 2020 2021 2022 2023 Fiscal Year

Effort Months

Corrective Adaptive Perfective Enhancements

TMD 1 (Step 4) (Effort)


Software Maintenance






administrative)




• Better negotiation techniques through SEER

• Investment Committee has more confidence in our estimates

• Challenges:– Better defined initial requirements are necessary

– Closing the feedback loop (actual vs. planned)

– To be at maturity level 2 by 2016 and level 3 by 2017


Today…

Hours Price Travel Hours Price Travel Hours Price Travel Hours Price Travel

PM 3,672 501,230€ 125,580€ 3,030 413,597€ 19,000€ 3,030 413,597€ 19,000€ 3,030 413,597€ 19,000€

SE 14,263 1,946,840€ 683,622€ 5,233 714,275€ 97,500€ 6,047 825,434€ 97,500€ 8,443 1,152,514€ 97,500€

SW Dev 23,074 3,149,615€ 323,054€ 10,720 1,463,236€ 125,000€ 12,581 1,717,248€ 125,000€ 18,056 2,464,668€ 125,000€

QA 1,720 234,781€ -€ 414 56,459€ -€ 509 69,458€ -€ 789 107,710€ -€

CM/DM 664 90,636€ -€ 491 66,960€ -€ 604 82,379€ -€ 936 127,747€ -€

ILS 1,390 189,736€ -€ 550 75,075€ -€ 550 75,075€ -€ 550 75,075€ -€

44,783 6,112,838€ 1,132,256€ 20,437 2,789,603€ 241,500€ 23,320 3,183,192€ 241,500€ 31,804 4,341,311€ 241,500€

AreaSEER Low

3,031,103€

3,679,535€

Agreed Price: €4,100,000 (57% savings) !!!

Proposed SEER HighSEER Med

TOTAL: 4,582,811€ 3,424,692€ 7,245,094€


Repeat Customer







Today…


PM 3,672 501,230€ 125,580€ 3,030 413,597€ 19,000€ 3,030 413,597€ 19,000€ 3,030 413,597€ 19,000€

SE 14,263 1,946,840€ 683,622€ 5,233 714,275€ 97,500€ 6,047 825,434€ 97,500€ 8,443 1,152,514€ 97,500€

SW Dev 23,074 3,149,615€ 323,054€ 10,720 1,463,236€ 125,000€ 12,581 1,717,248€ 125,000€ 18,056 2,464,668€ 125,000€

QA 1,720 234,781€ -€ 414 56,459€ -€ 509 69,458€ -€ 789 107,710€ -€

CM/DM 664 90,636€ -€ 491 66,960€ -€ 604 82,379€ -€ 936 127,747€ -€

ILS 1,390 189,736€ -€ 550 75,075€ -€ 550 75,075€ -€ 550 75,075€ -€

44,783 6,112,838€ 1,132,256€ 20,437 2,789,603€ 241,500€ 23,320 3,183,192€ 241,500€ 31,804 4,341,311€ 241,500€

AreaSEER Low

3,031,103€

3,679,535€



TOTAL: 4,582,811€ 3,424,692€ 7,245,094€


Mature Requirements







Today…


PM 3,672 501,230€ 125,580€ 3,030 413,597€ 19,000€ 3,030 413,597€ 19,000€ 3,030 413,597€ 19,000€

SE 14,263 1,946,840€ 683,622€ 5,233 714,275€ 97,500€ 6,047 825,434€ 97,500€ 8,443 1,152,514€ 97,500€

SW Dev 23,074 3,149,615€ 323,054€ 10,720 1,463,236€ 125,000€ 12,581 1,717,248€ 125,000€ 18,056 2,464,668€ 125,000€

QA 1,720 234,781€ -€ 414 56,459€ -€ 509 69,458€ -€ 789 107,710€ -€

CM/DM 664 90,636€ -€ 491 66,960€ -€ 604 82,379€ -€ 936 127,747€ -€

ILS 1,390 189,736€ -€ 550 75,075€ -€ 550 75,075€ -€ 550 75,075€ -€

44,783 6,112,838€ 1,132,256€ 20,437 2,789,603€ 241,500€ 23,320 3,183,192€ 241,500€ 31,804 4,341,311€ 241,500€

AreaSEER Low

3,031,103€

3,679,535€



TOTAL: 4,582,811€ 3,424,692€ 7,245,094€


Questions?