p1314-005216-3 generic the business case for systems ... · • reduced by as much as 75% through...

48
P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018 Page 1 of 48 THE BUSINESS CASE FOR SYSTEMS ENGINEERING 10 March 2018 Robert J Halligan, FIE Aust CPEng

Upload: others

Post on 19-Mar-2020

0 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 1 of 48

THE BUSINESS CASE FOR SYSTEMS ENGINEERING

10 March 2018

Robert J Halligan, FIE Aust CPEng

Page 2: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 2 of 48

Robert J Halligan, FIE Aust CPEng

Managing DirectorProject Performance International

Content Contributorto EIA/IS-632, EIA 632, IEEE 1220,

ISO/IEC 15288 SE standards

Past INCOSE Head Of Delegationto ISO/IEC SC7 on Software and

Systems Engineering

Past Member of the INCOSE Board of Directors

Past PresidentSystems Engineering Society of Australia

Consultant/Trainerto BAE Systems, Mitsubishi, Airbus, Thales, Raytheon, General Electric, Boeing, Lockheed, General Dynamics, OHB, Nokia, AREVA, BHP Billiton, Rio Tinto, Embraer, Halliburton and many other leading enterprises on six continents

[email protected]

Page 3: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 3 of 48

SEI/AESS/NDIA 2012 STUDY RESULTS

http://resources.sei.cmu.edu/asset_files/specialreport/2012_003_001_34067.pdf

DriverRelationshiptoPerformance(Gamma)AllProjects Lowerchallenge Higherchallenge

SEC-Total– totaldeployedSE +0.49 +0.34 +0.62

SEC-PP– projectplanning +0.46 +0.16 +0.65

SEC-REQ– reqts.developt.&mgmt. +0.44 +0.36 +0.50

SEC-VER– verification +0.43 +0.27 +0.60

SEC-ARCH– productarchitecture +0.41 +0.31 +0.49

SEC-CM– configurationmanagement +0.38 +0.22 +0.53

SEC-TRD – tradestudies +0.38 +0.29 +0.43

SEC-PMC– projectmonitor&control +0.38 +0.27 +0.53

SEC-VAL– validation +0.33 +0.23 +0.48

SEC-PI– productintegration +0.33 +0.23 +0.42

SEC-RSKM– riskmanagement +0.21 +0.18 +0.24

SEC-IPT– integratedproductteams +0.18 -0.12 +0.40

Gamma Relationship-0.2<|Gamma|≤0 Weaknegative

0≤|Gamma|<0.2 Weakpositive

0.2≤|Gamma|<0.3 Moderate

0.3≤|Gamma|<0.4 Strong

0.4≤|Gamma| Verystrong

Source:“TheBusinessCaseforSystemsEngineeringStudy:ResultsoftheSystemsEngineeringEffectivenessSurvey”,CMU/SEI-2012-SR-009,November2012

Page 4: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 4 of 48

OUR JOURNEY:

• What are our challenges?

• What is systems engineering?

• Why systems engineering?

• Studies on the value of systems engineering

• ROI for one facet: Requirements Analysis

Page 5: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 5 of 48

KICK-OFF EXERCISE:

In groups of 3-4 people, consider the question “what are the greatest challenges that we (you) face in your engineering”? List as many challenges as you can, in the time designated.

Page 6: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 6 of 48

WHERE WE HAVE COME FROM– OOPS GOT THAT WRONG!

Page 7: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 7 of 48

TODAY, AND NOT JUST IN KABUL!

Page 8: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 8 of 48

THE PROBLEM IN GENERAL:

Standish Group study of 8380 IT-based projectsSee also Morris and Hough, “The Anatomy of Major Projects”

Page 9: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 9 of 48

Average cost overrun: 89%

For “challenged” and cancelled projects:

% of Projects

Average cost overrun: 89%

Standish Group study of 8380 IT-based projects

For “challenged” and cancelled projects:

% of Projects

% of Cost Overrun

THE PROBLEM – COST:

Page 10: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 10 of 48

% of Projects

Standish Group study of 8380 IT-based projects

For “challenged” and cancelled projects:

% of Projects

% Schedule Overrun

Average scheduleoverrun: 122%

THE PROBLEM – SCHEDULE:

Page 11: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 11 of 48

THE PROBLEM – QUALITY:

Average missingfeatures: 39%

For “challenged” projects:

% of Projects

Average missingfeatures: 39%

For “challenged” projects:

% of Projects

% of Missing Features

Standish Group study of 8380 IT-based projects

Page 12: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 12 of 48

SYSTEMS THINKING – A FOUNDATION

leads to

leads to

Outward and Upward Looking View(part of one or more bigger systems)

System Views

Object View(object with required and desired characteristics)

Inward Looking View(seeing the system destined to become a set of interacting objects, the properties of the whole coming from the objects and their interactions)PPI-005918-4

Requirements+

Goals

Page 13: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 13 of 48

WHAT IS SYSTEMS ENGINEERING?

Page 14: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 14 of 48

INTEGRATE VERIFICATION & VALIDATION

WEDGE MODEL

e.g. testRSA

S

SE

SE

SE

S

SE

SE

SE

(e.g., Aircraft, Air Traffic ControlSystem)

(e.g., Propulsion System, Airframe)

(e.g., Engine, Fuel Pump)

S R A

timetrend

DESIGN BUILDVerification:Is the work product correct-meets requirements? Validation:Does the work product satisfy the need for the work product?

O T & E

S R A , S R R , A D R , D D R

Legend :A Build, increment, etc.

ADR Architectural Design ReviewDDR Detailed Design ReviewHWITLS Hardware in the Loop SimulationOT&E Operational Test & EvaluationPCA Physical Configuration AuditPITLS People in the Loop SimulationRSA (FCA) Requirements Satisfaction AuditS Top-Level SystemSE System ElementSRA System Requirements AnalysisSRR System Requirements ReviewSWITLS Software in the Loop Simulation

PPI-006003-9© Copyright Project Performance (Australia) Pty Ltd 2007 - 2017

PCA

PCA

PCAS R RA D R A D R

D D R D D R

trendtime

HWITLSPITLS

SWITLS

time

NeedsInformation

PROJECT PERFORMANCEINTERNATIONAL

www.ppi-int.com

Page 15: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 15 of 48

THE ESSENCE OF SE:

• ensure adequate problem definition• define possible solution alternatives• qualify solution alternatives for feasibility & effectiveness• develop qualified alternatives• use logical design as an aid in developing physical

design (model-based design)• design through levels of abstraction – architecture and

detail• maintain a clear distinction between problem and solution

Page 16: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 16 of 48

AND MORE …

• conduct trade-off studies and optimization to maximizeoverall effectiveness

• specify solution elements to objective adequacy• integrate engineering specialties with technology expertise• verify work products (correct – the product right)• validate work products (needed – the right product)• employ configuration management• do only work that adds value• manage the engineering – plan, organize, inspire, assess,

control.

Page 17: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 17 of 48

WHEN IS SYSTEMS ENGINEERING APPLIED?

• Solution development phase

• New Systems/Products

• Families of Products

• Solution build/production phase

• To correct design deficiencies

• Sustainment/operations and support phase

• Modifications to track changing need

• Incremental/competitive improvements for business reasons

• Response to obsolescence

Page 18: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 18 of 48

WHAT IS SYSTEMS ENGINEERING APPLIED TO?

• The enterprise

• Capability/business/enterprise systems

• End-use products

• Production systems

• Maintenance systems

• Training systems

• Project systems

• Engineering systems

• Anything else for which a solution does not already exist, and is sought!

Page 19: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 19 of 48PPI-005332-6

© Copyright Project Performance (Australia) Pty Ltd 2007-2014

0

1000

Benefit to Company - e.g. ∆ NPV

Benefit to Customer (external or internal)

Optimumsolution for the

company

1:1 Customer/Contractor Business Model

Internet ScamCompanyBusiness Model

Product-OrientedFree Market PlaceBusiness Model

Trade Off: Interests of Secondary Stakeholder (Customer) versus Primary Stakeholder (company)

Optimumsolution for the company

0

+

-

origin on x axis represents a threshold of acceptability.

Internal Project

(Net Present Value)

IN WHAT BUSINESS MODELS?

Page 20: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 20 of 48

INDICATORS OF EFFECTIVE SE – PRODUCT-ORIENTED ENTERPRISE:

• On, under, or close to development budget

• On, ahead of, or close to development schedule

• High Return on Sales

• Market leadership

• Low warranty costs

• Repeat business is the norm

• High staff satisfaction and retention

Page 21: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 21 of 48

INDICATORS OF EFFECTIVE SE – CONTRACT-ORIENTED ENTERPRISE:

• On, under, or close to development budget

• On, ahead of, or close to development schedule

• High contract gross margin

• High customer satisfaction

• Low warranty costs

• Repeat business is the norm

• High staff satisfaction and retention

Page 22: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 22 of 48

INDICATORS OF EFFECTIVE SE – INTERNAL PROJECTS:

• On, under, or close to development budget

• On, ahead of, or close to development schedule

• High internal customer satisfaction

• No desire to outsource

• High staff satisfaction and retention

Page 23: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 23 of 48

INDICATORS OF EFFECTIVE SYSTEMS ENGINEERING MANAGEMENT:

• Effective systems engineering

• Harnessing of creativity

• A learning environment

• Growing intellectual capital within the enterprise

• High staff satisfaction and retention

• Shared vision of the outcome and a related focus on quality, cost, time

Page 24: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 24 of 48

INDICATORS OF NO SE OR INEFFECTIVE SE:

• Milestones missed

• Significant dispute with stakeholders over requirements

• Many problems and delays occuring during system integration

• Significant dispute with customers over testing

• Significant problems occuring in released or fielded systems/products

• Engineering effort tends to be back-end loaded during development

Page 25: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 25 of 48

WHERE DOES THE MONEY GO?

Cost component Ideal % Actual %What proportion of development cost is spent due to genuine system requirements changes?

There is no ideal. ?

What proportion of development cost is spent due to defective system requirements? 0% ?

What proportion of development cost is spent due to system design errors undetected in design reviews?

0% ?

What proportion of development cost is spent due to system design errors undetected in system testing?

0% ?

What proportion of cost in a system integration phase is spent on system integration as opposed to rework?

Close to100% ?

Page 26: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 26 of 48

McKINSEY STUDY (1)

Page 27: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 27 of 48

McKINSEY STUDY (2)

5 C SOPTIMISING U E

S R4 T V

MEASURED O I3 M C

DEFINED E ER

3 CDEFINED >20 <4.0 <3.0 O

2 E RMANAGED R R

R E1 O C

PERFORMED <20 >4.0 >3.0 R T0 I

INITIAL ON

<0.8 <0.5

<2.0 <2.0

>35

>25

Service QualityMaturity Level

Design QualityMarket Share %

Process QualityScrap% Rework%

Quality Benefits

Equivalent to SMI

SOURCE: Excellence in Quality Management, McKinsey & Company, Inc.

CMMI

Page 28: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 28 of 48

OTHER OLD CLAIMS

1. Improved Quality of Designs• Resulted in reduced Change Orders (> 50%)

2. Product Development Cycle• Reduced as much as 40-60% by concurrent rather than sequential

design of products and processes

3. Manufacturing Costs• Reduced by as much as 30-40% by having integrated product teams

integrate product and process designs

4. Scrap & Rework• Reduced by as much as 75% through product and process design

optimization

Data based on a study of 14 companies that had applied concurrent engineering - Institute for Defense Analysis (IDA), 'The Role of Concurrent Engineering in Weapons System Acquisition’, December 1988

Page 29: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 29 of 48

NASA AND THE VALUE OF SE

Page 30: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 30 of 48

INCOSE STUDY - COST

SE Effort = SE Quality * (SE Cost/Actual Cost)

Page 31: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 31 of 48

INCOSE STUDY - SCHEDULE

SE Effort = SE Quality * (SE Cost/Actual Cost)

Page 32: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 32 of 48

MCPM – MATURITY BY PROJECT CATEGORY MODEL, BRAZIL

Archibald & Prado, “PM Maturity 2006 Research –Maturity and Success in IT”, March, 2007

Page 33: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 33 of 48

CMU/NDIA 2007 STUDY RESULTS

Source: “A Survey of Systems Engineering Effectiveness”, CMU/SEI-2008-SR-034, December 2008

Page 34: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 34 of 48

PROJECT ENGINEERING MATURITY MATRIX

Feedback: Process Continuously Improved

Quantitative:Process MeasuredFocus on metrics

Qualitative:Process defined andinstitutionalized

Focus on process org.

Intuitive:Process depends onindividuals

Ad hoc/chaotic:Unpredictable

System problem preventionTechnology innovationProcess management

Process mapping/variationProcess improvement databaseQuantitative quality plans

Enterprise process definitionEducation and trainingReview and testingInterdisciplinary teamworkLife cycle engineeringIntegrated systems management

System requirements mgmtProject planning and trackingSystem configuration mgmtQuality managementSystem risk management

IncreasedCustomerandProducerSatisfaction

IncreasedRisk

5 OPTIMIZING

4 MANAGED

3 DEFINED

2 REPEATABLE

1

Maturity Level Characteristics Key Process Areas

Page 35: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 35 of 48

SEI/AESS/NDIA 2012 STUDY RESULTS (1)

Source: “The Business Case for Systems Engineering Study: Results of the Systems Engineering Effectiveness Survey”, CMU/SEI-2012-SR-009, November 2012

CMU/SEI-2012-SR-009 | x i

Executive Summary

The National Defense Industrial Association Systems Engineering Division (NDIA-SED) collabo-rated with the Institute of Electrical and Electronic Engineers Aerospace and Electronic Systems Society (IEEE-AESS) and the Software Engineering Institute (SEI) of Carnegie Mellon® to obtain quantitative evidence of the benefit of systems engineering (SE) best practices on project perfor-mance. The team developed and executed this survey of system developers to identify SE best practices used on projects, collect performance data on these projects, and identify relationships between the application of these SE best practices and project performance.

The study found clear and significant relationships between the application of SE best practices to projects and the performance of those projects, as seen in the mosaic chart in Figure 1 and as ex-plained below.

Figure 1: Project Performance vs. Total SE Capability

The left column represents projects deploying lower levels of SE, as measured by assessing the quantity and quality of specific SE work products. Among these projects, only 15% delivered higher levels of project performance, as measured by satisfaction of budget, schedule, and tech-nical requirements. Within this group, 52% delivered lower levels of project performance.

The second column represents those projects deploying moderate levels of SE. Among these pro-jects, 24% delivered higher levels of project performance and 29% delivered lower levels of per-formance.

® Carnegie Mellon is registered in the U.S. Patent and Trademark Office by Carnegie Mellon University.

52%29% 20%

33%47%

24%

15% 24%57%

0%10%20%30%40%50%60%70%80%90%

100%

Lower SEC (n=48) Middle SEC (n=49) Higher SEC (n=51)

Proj

ect P

erfo

rman

ce (P

erf)

Total Systems Engineering Capability (SEC-Total)

Gamma = 0.49 p-value < 0.001

All Projects

Higher Perf

Middle Perf

Lower Perf

Page 36: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 36 of 48

SEI/AESS/NDIA 2012 STUDY RESULTS (2)

Legend: PC Project Challenge

Source: “The Business Case for Systems Engineering Study: Results of the Systems Engineering Effectiveness Survey”, CMU/SEI-2012-SR-009, November 2012

CMU/SEI-2012-SR-009 | x iv

Figure 3: Project Performance vs. Total SE Capability controlled by Project Challenge

The chart on the right side of Figure 3 shows a very strong relationship between SEC-Total and Perf for those projects with higher PC. The percentage of projects delivering higher performance increased from 8% to 26% to 62% as SEC-Total increased from lower to middle to higher. Addi-tionally, the percentage of projects delivering lower performance decreased from 69% to 39% to 27% as SEC-Total increased.

Thus, for these higher challenge projects, the likelihood of delivering higher performance in-creased more than sevenfold and that of delivering lower performance decreased by almost two-thirds with improved SEC-Total. This relationship is characterized by a Gamma value of +0.62 and a very low p-value less than 0.001.

The meaning of this information is clear:

Projects that properly apply systems engineering best practices perform better than projects that do not.

This report identifies the SE process groups that have the strongest relationships to project per-formance. It also shows that more challenging projects tend to perform worse than less challeng-ing projects. However projects that face less challenge still tend to benefit from implementing systems engineering best practices. Moreover the impact of employing systems engineering best practices is even greater for more challenging projects.

With this knowledge, system acquirers and system developers can inform their judgments regard-ing the application of SE to their projects and improve their SE practices to further enhance pro-ject outcomes.

32%19% 12%

45%58%

36%

23% 23%

52%

0%10%20%30%40%50%60%70%80%90%

100%

Lower SEC(n=22)

Middle SEC(n=26)

Higher SEC(n=25)

Low PC

Gamma = 0.34 p-value = 0.029

69%

39%27%

23%

35%

12%

8%26%

62%

0%10%20%30%40%50%60%70%80%90%

100%

Lower SEC(n=26)

Middle SEC(n=23)

Higher SEC(n=26)

High PC

Gamma = 0.62 p-value < 0.001

Page 37: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 37 of 48

SEI/AESS/NDIA 2012 STUDY RESULTS (3)

http://resources.sei.cmu.edu/asset_files/specialreport/2012_003_001_34067.pdf

DriverRelationshiptoPerformance(Gamma)AllProjects Lowerchallenge Higherchallenge

SEC-Total– totaldeployedSE +0.49 +0.34 +0.62

SEC-PP– projectplanning +0.46 +0.16 +0.65

SEC-REQ– reqts.developt.&mgmt. +0.44 +0.36 +0.50

SEC-VER– verification +0.43 +0.27 +0.60

SEC-ARCH– productarchitecture +0.41 +0.31 +0.49

SEC-CM– configurationmanagement +0.38 +0.22 +0.53

SEC-TRD – tradestudies +0.38 +0.29 +0.43

SEC-PMC– projectmonitor&control +0.38 +0.27 +0.53

SEC-VAL– validation +0.33 +0.23 +0.48

SEC-PI– productintegration +0.33 +0.23 +0.42

SEC-RSKM– riskmanagement +0.21 +0.18 +0.24

SEC-IPT– integratedproductteams +0.18 -0.12 +0.40

Gamma Relationship-0.2<|Gamma|≤0 Weaknegative

0≤|Gamma|<0.2 Weakpositive

0.2≤|Gamma|<0.3 Moderate

0.3≤|Gamma|<0.4 Strong

0.4≤|Gamma| Verystrong

Source:“TheBusinessCaseforSystemsEngineeringStudy:ResultsoftheSystemsEngineeringEffectivenessSurvey”,CMU/SEI-2012-SR-009,November2012

Page 38: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 38 of 48

SEI/AESS/NDIA 2012STUDY RESULTS (4)

Source: “The Business Case for Systems Engineering Study: Results of the Systems Engineering Effectiveness Survey”, CMU/SEI-2012-SR-009, November 2012

CMU/SEI-2012-SR-009 | 76

Table 9: Summary of Relationships Between SE Deployment and Project Performance

Driver Relationship to Performance (Gamma)

All projects Lower challenge projects

Higher challenge projects

SEC-Total – total deployed SE +0.49 � Very strong positive

+0.34� Strong positive

+0.62 � Very strongpositive

SEC-PP – project planning +0.46 � Very strong positive

+0.16 � Weak positive

+0.65 � Very strongpositive

SEC-REQ – requirements development and management

+0.44 � Very strong positive

+0.36 � Strong positive

+0.50 � Very strongpositive

SEC-VER ࣓ verification +0.43 � Very strong positive

+0.27 � Moderate positive

+0.60 � Very strongpositive

SEC-ARCH – product architec-ture

+0.41 � Very strong positive

+0.31 � Moderate positive

+0.49 � Very strongpositive

SEC-CM – configuration management

+0.38 � Strong positive

+0.22 � Moderate positive

+0.53 � Very strongpositive

SEC-TRD – trade studies +0.38 � Strong positive

+0.29 � Moderate positive

+0.43 � Very strong positive

SEC-PMC – project monitoring and control

+0.38 � Strong positive

+0.27 � Moderate positive

+0.53 � Very Strongpositive

SEC-VAL ࣓ validation +0.33 � Strong positive

+0.23 � Moderate positive

+0.48 � Very strongpositive

SEC-PI ࣓ product integration +0.33 � Strong positive

+0.23 � Moderate positive

+0.42 � Very strong positive

SEC-RSKM ࣓ risk management +0.21 � Moderate positive

+0.18 � Weak positive

+0.24 � Moderate positive

SEC-IPT ࣓ integrated product team utilization

+0.18 � Weak positive

-0.12 � Weak negative

+0.40 � Very strong positive

Table 10: Summary of Relationships Between Other Factors and Project Performance

Driver Relationship to Performance

All projects Lower challenge projects

Higher challenge projects

PC – Project challenge -0.26 � Moderate negative

-0.26 � Moderate negative

-0.23 � Moderate negative

EXP – Prior experience +0.36 � Strong positive

+0.51 � Very strongpositive

+0.19� Weak positive

The data in Tables 8 and 9 are illustrated in Figures 70 through 72.

Page 39: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 39 of 48

A Look at Return on Investmentfor One Facet of Systems Engineering:

Requirements Analysis

Page 40: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 40 of 48

Initial (Originating)Stakeholder

Requirements (if any) - e.g. user.

(SyRS, e.g. URS)

Other Info

AND

1

2

3

4

AND

5

AND

6

Ref.Ref. AND

4.2.1

4.2.2

4.2.3

4.2.4

OR OR LPLP

35

SyRS-Refined

VRS

OCD

VMAnalytical work products

OCD: operational concept description (CONUSE)URS: SyRS of user requirementsSyRS: system requirements specificationVM: value (or system/software effectiveness) modelVRS: verification requirements specificationPPI-005499-4

is a restatement oftraces to/from

Page 41: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 41 of 48

Legend:

IDENTIFYSTAKEHOLDERS

STATES &MODES

ANALYSIS

READ &ASSESSINPUTS

PLANTHE SRA

CONTEXTANALYSIS

FUNCTIONALANALYSIS

REST OFSCENARIOANALYSIS

PARSINGANALYSIS

OUT-OF-RANGEANALYSIS

ERAANALYSIS

CLEAN-UP

MEASUREREQUIREMENTS

QUALITY

DESIGNREQUIREMENTS

ANALYSIS

OTHERCONSTRAINTS

SEARCH

VERIFICATIONREQUIREMENTS

DEV.OCD

DEVELOPMENTS/H

VALUEANALYSIS

*

*

*

PPI-006248-2

REQUIREMENTS ANALYSIS(CAPTURE AND VALIDATION)METHODOLOGY

SRA System Requirements AnalysisS/H StakeholderDEV DevelopmentOCD Operational Concept Description (CONUSE)ERA Entity Relationship Attribute Perform only if there are initial specified

requirements as an input to the analysis.*

Page 42: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 42 of 48

REQUIREMENTS QUALITY AND REQUIREMENTS ANALYSIS EFFORT

HaveNeedNumber of RequirementsSkillsTech-EnvironmentAccess & Cooperation)

0 0

.3

1 1

Risk M

Risk H

Have0.5

WORK = f(

0.85-0.98

Need0.9

WORK!(SRA)

Requirements Quality and Requirements Analysis Effort

P007-004138-4

Risk L

Page 43: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 43 of 48

Organization/Project Overruns Attributed to Requirements Problems

NASA over two decades (Werner Gruhl)

70% of overruns

U.S. Census Bureau project 2009 80% cost overrun locked in solely due to poor requirements

Marine One Helicopter Program 83% cost overrun attributed by Lockheed to requirements problems

Schwaber, 2006; Weinberg, 1997; Nelson et al, 1999

“Requirements errors are the single greatest source of defects and quality problems”

Hofmann and Lehner, 2001 “Deficient requirements are the single biggest cause of software project failure”

Standish Group, The Chaos Report on 8300 IT projects

60.9% of an average 89% cost overrun

IMPACT OF REQUIREMENTS DEFECTS

Page 44: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 44 of 48

Parameter Value Contract value $4B Requirements on the Ship 27,000, only fair in quality Consequence if uncorrected At least 20% loss of capability, costing at

least $800M; or Rework costs exceeding 20%

Cost of fixing the requirements $8M (0.2% of contract value) Return on Investment Approximately 100:1

REQUIREMENTS ANALYSIS ROI TO CUSTOMER

Page 45: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 45 of 48

Paramater Value % Sales spent on marketing 12.5% % Sales spent on bidding 9-10% Win ratio for the more successful companies

1 in 2 to 1 in 4

Typical cost/bid, % Total Contract Value 2-3% TCV Cost of winning business from a new customer vis-à-vis a satisfied existing customer

5:1

Cost of preserving customer satisfaction through requirements analysis

0.2% TCV

REQUIREMENTS ANALYSIS ROI FOR A CONTRACTOR

TCV: Total Contract Value

Page 46: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 46 of 48

CONCLUSIONS:

1. The practice of engineering can be immature• Sometimes ad hoc and chaotic – that is destructive to success via

satisfaction of users and other stakeholders.

2. The evidence is now compelling that the practice of systems engineering contributes to enterprise success in terms of: • reduced costs• shorter timeframes• increased value achieved in using the system.

Page 47: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 47 of 48

CHALLENGES – HOW DID WE GO?

We will review the list of challenges from earlier, and the contributions made by "a systems approach to the engineering of systems".

Page 48: P1314-005216-3 GENERIC The Business Case for Systems ... · • Reduced by as much as 75% through product and process design optimization Data based on a study of 14 companies that

P1314-005216-3 © Copyright Project Performance (Australia) Pty Ltd 1998-2018

Page 48 of 48

THE QUESTION IS NO LONGER, “SHOULD WE BE PRACTICING SYSTEMS ENGINEERING?”

"YES" IS BEYOND DOUBT.

TODAY'S QUESTION IS,“HOW BEST DO WE DO IT?”

Robert J. HalliganEmail: [email protected]