value of systems engineering; summary report 1/04 1 value of systems engineering summary report...

Value of Systems Engineering; Summary Report 1/04 1

Value of Systems Engineering

Summary report SECOE and related projects

Eric HonourINCOSE Director, Sponsored Research


Agenda

Background Heuristic Claims of SE Value

Gathered results on Value of SE NASA Tracking 1980s “Boundary Management” study “Large Engineering Projects” MIT study “Impact of SE at NASA” (SECOE 02-02)

“Impact of SE on Quality & Schedule” Boeing “SE Effectiveness” IBM study “Value of SE” research (SECOE 01-03)


Heuristic Claim of SE

Better systems engineering leads to Better system quality/value Lower cost Shorter schedule

SYSTEMDESIGN

DETAILDESIGN

PRODUCTIONINTEGRATION TEST

Traditional Design

Time

Risk

SavedTime/Cost

“System Thinking” Design Time

Risk


NASA Tracking 1980s

Source Werner GruhlNASA Comptroller’s Office

Total Program Overrun32 NASA Programs

R2 = 0.5206

0

20

40

60

80

100

120

140

160

180

200

0 5 10 15 20

Definition Percent of Total Estimate

Pro

gra

m O

ve

rru

n

Definition $Definition Percent = ---------------------------------- Target + Definition$

Actual + Definition$Program Overrun = ---------------------------------- Target + Definition$

GRO76OMV

GALL

IRAS

TDRSS

HST

TETH

LAND76

MARS

MAG

GOES I-M

CENACT

CHA.REC.

SEASAT

DE

UARS

SMM

EDO

ERB77

STS

LAND78

COBE

GRO82

ERB88VOY

EUVE/EP

ULYS

PIONVEN IUE ISEE

HEAO


“Boundary Management” Study

Study of 45 high-tech new product development teams

...Ancona and Caldwell, Research Technology Management, 1990

Significant portion of time is spent at team boundaries

Individual Time Spent Individual Time Spent Outside TeamOutside Team 14%*14%*

* Typically limited to few individuals* Typically limited to few individuals

Within TeamWithin Team 38% 38%

AloneAlone 48%48%


Boundary Management in Technical Teams

Boundary management occurs in four different roles.

Level of effort in each role changes with time.

• AmbassadorAmbassadorBuffering, building support, Buffering, building support, reporting, strategyreporting, strategy

• Task CoordinatorTask CoordinatorLateral group coordination, info Lateral group coordination, info transfer, planning, negotiatingtransfer, planning, negotiating

• ScoutScoutObtain possibilities from outside - Obtain possibilities from outside - interface with marketinginterface with marketing

• GuardGuardWithhold information, prevent Withhold information, prevent disclosuredisclosure

CREATIONCREATIONDEVELOPMENTDEVELOPMENT

DIFFUSIONDIFFUSION


“Boundary Management” Study

Significant Findings:

High-performing teams did more external interaction than low-performing teams

System technical leadershipcreates greater success

Internal team dynamics (goals, processes, individual satisfaction) did not correlate with performance

Process definition is important but not sufficient.


Cost Targets

Schedule Targets

Objective Targets

82%

Percent of Projects Meeting:

“Large Engineering Projects”

Study of 60 LEPs (power generation, transportation, oil production, technology)

The Strategic Management of Large Engineering Projects, MIT Press 2000

Evaluation by interviews and by objective and subjective quality measures.

72%

45% 18% 37%

Failed!


“Large Engineering Projects”

Significant Findings:

Most important determinant was a coherent, well-developed organizational/team structure

A structure of leadership creates greater success

Technical difficulties, social disturbance, size were not statistically linked to performance All projects had turbulent events

Technical excellence could not save a socially unacceptable project

Process definition is important but not sufficient.


Impact of SE at NASA (SECOE 02-02)

Survey research within NASA Form with 38 questions, answers on graded scale Typical questions:

• On your most recent project, what percent of your total project cost was spent on Systems Engineering tasks?

• On your most recent project, did systems engineering shorten or lengthen the planned schedule?

Aggressive survey pursuit with management push NASA: 250 sent, 54% valid response INCOSE: 650 sent, 38% valid response

Engineering of Complex Systems – The Impact of Systems Engineering at NASA, A.K.P.Kludze, Jr. doctoral dissertation George Washington Univ. 2003


Response Demographics

Participating Organization

NASA 136INCOSE 243Total 379

Work Experience

0-10 41%11-20 37%21+ 22%

Age Groups

20-39 29%40-59 61%60+ 10%

Education Level

Bachelor 34%Master 55%Doctor 11%

Job Titles

SE 56%PM 17%Other 27%

...significant differences by organization


Key Survey Results - Cost

Percent Spent on SE

0-5% 6-10% 11-15% 16% +

• Respondents marked bracket to show percent of total cost spent on SE on last project.

• Mode at 6-10% of project• Few projects spent 11-15%• Unexplained bimodal response >16%

(perhaps interpretation of “project”)

Cost Benefit of SE

Very Poor Poor Fair Good Excellent

• Respondents believe strongly in cost benefit of SE

• In secondary question, few respondents could quantify


Key Survey Results - Schedule

At What Stage is SE Most Effective?

Very Early Midway End No Matter No Need

• Vast majority of respondents believe that SE is most effective very early in a project.

Impact of SE on Schedule

Shorten Lengthen Don’t Know

• INCOSE respondents believe SE shortened schedule on most recent project

• NASA respondents uncertain• Secondary questions uncertain

in the quantification.

...other results available in dissertation


Impact of Systems Engineering on Quality and Schedule

Empirical evidence obtained from three parallel (same time) projects Each developed a complex, robotic Universal

Holding Fixture (UHF) Each used a different level of SE Results are compared

Trait UHF1 UHF2 UHF3

Size 10’ x 40’ 8’ x 50’ 6’ x 14’

Accuracy ±0.005” ±0.003” ±0.003”

Contact Sensors None 57 108

Vacuum Sensors 1 70 108

Real-time checks No Yes Yes

Probe contours No Yes Yes

NC interface No Yes Yes

Impact of Systems Engineering on Quality and Schedule – Empirical Evidence, W. Forrest Frantz, Boeing Corp. 1995


Project Differences

Project Trait UHF1 UHF2 UHF3

Sys mgmt experience Low Low-Medium Low-Medium

Subcontract approach Design Reviews Full-time SE on site

Access to SE support Low High, but not used

High, used

Requirements approach Token req’s Complete, detailed, integrated req’s

Design approach HW/SW specs, multi-org approach

Functional specs fully address HW/SW processes and interfaces

Functional adherence Design docs took precedence; specs updated per design

Specs followed, CCB control

Design reviews Weekly team reviews

Formal internal; little external

Formal internal and external

Integration approach Patterned after design

Drive by functional specs; defined early in life cycle

Acceptance testing Defined in high-level plan

Formal tests based on Req’s and Functional specs


Impacts

Use of better SE reduced Overall cycle time Time to create req’s Time to design/produce Time to test

...even in the face of more complex, higher quality systems!

0 50 100

UHF3

UHF2

UHF1

Overall Development Time (weeks)

0 10 20 30

UHF3

UHF2

UHF1

Requirements to RFP (weeks)

0 20 40 60

UHF3

UHF2

UHF1

Design to Production (weeks)


Systems Engineering Effectiveness

Study of 8 software product development projects during upgrade of SE processes Determining Systems Engineering Effectiveness, Bruce Barker, IBM Commercial Products, Conference on Systems Integration,

Stevens Institute 2003

Evaluation by cost and schedule against a standard estimating method.

Identifyaffected

components

EvaluateImpact,

Complexity

Convertto

“points”

EstimateCost,

Schedule

Historical Database, Cost per “Point”

NewProductConcept

ProductLine Architecture

Costing method applies only to project management, business management, systems engineering, system integration, and delivery into production. Application development costs are not included.

© Copyright IBM Corp 2003Used With Permission


Project Data

$K/Point Averages

Without SE

With SE

$1,350/pt

$944/pt

200020012002

$1,454/pt$1,142/pt

$818/pt

2000

2000

2001

2001

2001

2002

2002

2002

Project 1

Project 2

Project 3

Project 4

Project 5

Project 6

Project 7

Project 8

12,934

1,223

10,209

8,707

4,678

5,743

14,417

929

18,191

2,400

11,596

10,266

5,099

5,626

10,026

1,600

0

0

9.2

0

10.7

14.4

10.2

16.0

1,406

1,962

1,136

1,179

1,090

980

695

1,739

Year Project “Points” Cost ($K)

SE Costs (%)

$K/Point



Timeline of Projects

1999 2001 2002

500

1000

1500

2000

2500

2000

SE organization created

SE Process documented

SE Formal Training Started

Project w/o SE

Project With SE

As the Systems Engineering process has been enabled and integrated through the organization, productivity has increased

Yearly Avg

Cost per “Point”



Systems Engineering Effectiveness

Significant Findings: Impact and complexity provide an effective

method to perform parametric costing.Early parametric costing works.


Preliminary data indicates that the use of Systems Engineering will improve project productivity when effectively combined with the Project Management and Test Processes.

Systems engineering improves productivity.

$K/Point Averages

Without SE

With SE

$1,350/pt

$944/pt

200020012002

$1,454/pt$1,142/pt

$818/pt


“Value of SE” (SECOE 01-03)

Multi-year effort to obtain statistical data Correlate amount/quality of SE with project

quality/success

SE Effort

Parameterized by• Technical “Size”• Technical Complexity• Risk Level

Development Quality

(function ofTechnical Value,

Cost, Schedule,

Risk)6-10% ?


Respondent Data

43 respondents 1 project not completed Values $1.1M - $5.6B SE Cost 0.3% - 26%

0

2

4

6

8

10

0 5 10 15 20 25

SE Cost %

Nu

mb

er

of

Pro

jec

ts

Cost, schedule, quality correlate better with “Systems Engineering Effort”:

SEE = SE Qual * (SE Cost %)

0

2

4

6

8

10

0 5 10 15 20 25

SE Effort = SE Quality * (SE Cost %)

Nu

mb

er

of

Pro

jec

ts


0.6

1.0

1.4

1.8

2.2

2.6

3.0

0% 4% 8% 12% 16% 20% 24% 28%

SE Effort = SE Quality * SE Cost/Actual Cost

Act

ual

/Pla

nn

ed C

ost

Cost Overrun vs. SE Effort

Source: SECOE 01-03INCOSE 2003

Average Cost Overrun

90% Assurance (1.6)


0.6

1.0

1.4

1.8

2.2

2.6

3.0

0% 4% 8% 12% 16% 20% 24%

SE Effort = SE Quality * SE Cost/Actual Cost

Act

ual

/Pla

nn

ed S

ched

ule

Schedule Overrun vs. SE Effort


Average Schedule Overrun

90% Assurance (1.6)


0.4

0.6

0.8

1.0

1.2

0% 4% 8% 12% 16% 20% 24%

SE Effort

Dev

elo

pm

ent

Qu

alit

y(C

ost

/Sch

edu

le B

ased

)

Test Hypothesis: Quality


SE Effort

Qu

ality Hypothesis

0.0

1.0

2.0

3.0

4.0

5.0

6.0

7.0

8.0

9.0

10.0

0% 4% 8% 12% 16% 20% 24%

SE Effort

Co

mp

arat

ive

Su

cces

s


Conclusions “Value of SE”

SE effort improves development quality Cost, schedule, subjective Hypothesis is supported by the data

Optimum SE effort is 10-15% or more Cost, schedule overruns are minimized However, note wide dispersion of data Also note few data points at this level; most

projects spent far less Quality of the SE effort matters

Lower quality SE reduces effectiveness


Company Participation

Project benchmarking service funded by participants Aggregated data shared among participants Raw data protected by data blinding

SECOEBlinded

Raw DataProject

Benchmarkingdata

Statistical dataValue of SE

Reportedresults

ParticipatingCompanies

Public

1 month

ParticipatingCompanies(all data)

Quarterly

Every two years


Company Participation

Data gathering Select 4-6 programs One day session per participating company

every 4 months 1½-hour sessions with PM+SE of each program Data gathered by two SECOE researchers Forms & notes do not identify programs

Reports Benchmark report within 30 days of each

quarterly session, compares to all prior data 4-month reports to all participants with

aggregated results from all data, all sources Participation price $??K for each year


Questions?

Eric Honour

INCOSE Director for Sponsored ResearchPensacola, FL, USA+1 (850) 479-1985 [email protected]

value of systems engineering; summary report 1/04 1 value of systems engineering summary report...

Documents