dod systems engineering and management implications for evolutionary acquisition of major defense...

DoD Systems Engineering and Management Implications for Evolutionary Acquisition of

Major Defense Systems

DoD Systems Engineering and Management Implications for Evolutionary Acquisition of

Major Defense Systems

A DoD SERC Quick-Look Studyand CSER 2010 Invited Presentation Barry Boehm, Jo Ann Lane, USC-CSSE

March 17, 2010Charts with notes

Summary: Evolutionary Acquisition (EvA) Summary: Evolutionary Acquisition (EvA)

• EvA is the preferred DoD strategy for MDAPs (DoDI 5000.02)– It is more responsive to DoD’s current and future challenges

• There are several forms of EvA – With different strengths and shortfalls

• EvA requires significant advances in current practices– In systems engineering (SE) and development processes, particularly to

avoid unscalable, easiest-first, sunny-day architectural commitments– In rapid, adaptive integration of SE and acquisition management (AM)– In financial and human resource allocation– In workforce capability and empowerment– In research and technology priorities and transition speedup

• Initiatives are needed to provide these advances in time for EvA to succeed

23/1/2010

What is “Evolutionary Acquisition”?What is “Evolutionary Acquisition”?• An acquisition approach that involves

– Delivering mission capability in increments– Continuing reassessment of future-increment priorities

• It departs from the outdated assumptions underlying traditional acquisition– System requirements can be specified in advance– System requirements are largely stable– Full operational capability cost estimation is feasible– Full-development, up-front Integrated Master Plans, Integrated

Master Schedules, and Earned Value Management Plans are feasible

• Its successful application requires rethinking traditional systems engineering (SE) and acquisition practices

33/1/2010

4

Future DoD Challenges: Asymmetric Conflict and OODA Loops

Future DoD Challenges: Asymmetric Conflict and OODA Loops

Development Commitment Milestone for Cycle

Decide on next-cycle capabilities, architecture upgrades, plans

Act on plans, specifications

Orient with respect to stakeholders priorities, feasibility, risks

Observe new/updated objectives, constraints, alternatives

•Adversary

•Picks time and place

•Little to lose

•Lightweight, simple systems and processes

•Can reuse anything

•Defender

•Ready for anything

•Much to lose

•More heavy, complex systems and processes

•Reuse requires trust

3/1/2010

Average Change Processing Time: Two Complex Systems of SystemsAverage Change Processing Time: Two Complex Systems of Systems

0

20

40

60

80

100

120

140

160

WithinGroups

AcrossGroups

ContractMods

5

Average workdays to process

changes

Incompatible with turning within adversary’s OODA loop

3/1/2010







63/1/2010

Time phasing terms: Scoping; Architecting; Developing; Producing; Operating (SADPO) Prespecified Sequential: SA; DPO1; DPO2; DPO3; …Evolutionary Sequential: SADPO1; SADPO2; SADPO3; …Evolutionary Overlapped: SADPO1; SADPO2; SADPO3; …Evolutionary Concurrent: SA; D1 ; PO1… SA2; D2 ; PO2… SA3; D3; PO3 …

There is No One-Size-Fits-All EvA ModelThere is No One-Size-Fits-All EvA Model

7

Type Examples Pros ConsPrespecified Sequential

Platform base plus PPPIs

Prespecifiable full-capability requirements, scalability when stable

Emergent requirements or rapid change,

architecture breakersEvolutionary

SequentialSmall: Agile

Larger: Rapid fieldingAdaptability to change, need for usage feedback

Easiest-first; late, costly fixes; SysE time gaps

Slow for large systemsEvolutionary Overlapped

Stable development; Maturing technology

Mature technology upgrades

Emergent requirements or rapid change; SysE

time gapsEvolutionary Concurrent

Rapid, emergent development

Systems of systems

Emergent requirements or rapid change, SysE

continuity

Overkill on small or highly stable systems

3/1/2010

Evolutionary Acquisition (EvA) Decision TableEvolutionary Acquisition (EvA) Decision Table

8

Type Stable, prespecifiable requirements?

OK to wait for full system to be developed?

Need to wait for next-increment

priorities?

Need to wait for next-increment

enablers*?Single Step Yes YesPrespecified Sequential

Yes No

Evolutionary Sequential

No No Yes

Evolutionary Overlapped

No No No Yes

Evolutionary Concurrent

No No No No

* Example enablers: Technology maturity; External-system capabilities; Needed resources

3/1/2010

Evolutionary Overlapped Form per DoDI 5000.02 Defers the start of a new increment until its technology is mature

Evolutionary Overlapped Form per DoDI 5000.02 Defers the start of a new increment until its technology is mature

93/1/2010

Figure 2. Requirements and Acquisition Process Flow.

Evolutionary Concurrent: Rapid Change with High Assurance

Evolutionary Concurrent: Rapid Change with High Assurance

10

Agile Rebaselining for Future Increments

Short, StabilizedDevelopmentof Increment N

Verification and Validation (V&V)of Increment N

Deferrals

Artifacts Concerns

Rapid Change

HighAssurance

Future Increment Baselines

Increment N Transition/

Operations and Maintenance

Future V&V

Resources

Increment N Baseline

Current V&V

Resources

Unforeseeable Change (Adapt)

ShortDevelopmentIncrements

ForeseeableChange

(Plan)

Stable DevelopmentIncrements

Continuous V&V

3/1/2010

“Evolution” and “Maintenance”“Evolution” and “Maintenance”• EvA often includes maintenance of fielded increments

– Changes to fielded increments must be coordinated with development of new increments

• Even if a separate organization handles maintenance

• As above, there is no one-size-fits-all process for this– Can involve an evolutionary next-increment with a Milestone B gate

• As in USD(AT&L) Designation of MDAP Subprograms memo [Carter, 2009]

– Can also involve pools of maintainers performing low levels of as-needed maintenance not requiring Milestone Bs

• Can’t have predetermined process for change traffic– Best available process shown in next chart

• Situation even more complex with systems of systems– May need to integrate all types of evolution processes

113/1/2010

Agile Change Processing and RebaseliningAgile Change Processing and Rebaselining

12

Assess Changes,Propose Handling

StabilizedIncrement-NDevelopment Team

Change Proposers

Future Increment Managers

Agile Future-IncrementRebaselining Team

Negotiate change disposition

Formulate, analyze options in context of other changes

HandleAcceptedIncrement-Nchanges Discuss, resolve deferrals to

future increments

ProposeChanges

Discuss, revise,defer, or drop

Rebaselinefuture-incrementFoundations packages

Prepare for rebaselinedfuture-incrementdevelopment

Defer some Increment-N capabilities

Recommend handlingin current increment

Accept changes

Handle in current rebaseline

Proposed changes

Recommend no action,provide rationale

Recommend deferrals to future increments

3/1/2010

O&S PD EMD

● ● ●

Constituent System n

(pre-existing)

TDMSANew System A

Constituent System B

(pre-existing)

MS A MS B MS C

Current and Future DoD Challenges: Systems of Systems

Current and Future DoD Challenges: Systems of Systems

13

Increment m

Increment n-1

SoS SE Level*

* DoD Systems Engineering Guide for SoS, 2008

Increment m+1

Increment n Increment n+1







143/1/2010

EvA Differences: SE and Development Processes

EvA Differences: SE and Development Processes

• EvA not a “one-size-fits-all” process– As shown in charts above

• No clear boundary between “development,” “operations and maintenance”– Don’t release your key SE people at PDR

• Short prioritized increments; Schedule as independent variable (SAIV)– Architect to add or drop borderline-priority features to meet schedule

– Exceptions for non-subsettable deliverables

• Earlier testing, verification, and validation (V&V)– Architecture, infrastructure V&V to avoid unscalable easiest-first increments– Earlier/continuous test, V&V

• Rapid, concurrent, pro-active SE and acquisition management– Of product and process; requirements and solutions; development and evolution;

hardware/software/human factors– Proactive vs. reactive with respect to technology, NDI, external interfaces

• Evidence/risk as decision criterion, first class deliverable– Process for evidence preparation– Risk/opportunity-driven: avoid overkill, underkill (balance risk, opportunity)– Competitive prototyping as a way of buying information to reduce risk

153/1/2010

*Schedule As Independent Variable; Feature set as dependent variable– Also works for cost, schedule/cost/quality as independent variable – Also known as timeboxing, time-determined development, time-certain development

The SAIV* Process Model—Cross Talk, January 2002 (http://www.stsc.hill.af.mil/crosstalk)

The SAIV* Process Model—Cross Talk, January 2002 (http://www.stsc.hill.af.mil/crosstalk)

1. Shared vision and expectations management2. Feature prioritization3. Schedule range estimation and core-capability

determination- Top-priority features achievable within fixed schedule with 90% confidence

4. Architecting for ease of adding or dropping borderline-priority features - And for accommodating post-IOC directions of growth

5. Incremental development- Core capability as increment 1

6. Change and progress monitoring and control- Add or drop borderline-priority features to meet schedule

163/1/2010

Rapid, Adaptive Acquisition ManagementRapid, Adaptive Acquisition Management

• Need to rework traditional guidance documents– No longer prespecified, total-package, sequential IMP, IMS , WBS, EVMS – Concurrently engineered with evidence-based decision milestones

• No “one-size-fits-all” contracting– E.g., for build-to-spec, agile rebaselining, V&V teams– SE budgets a function of system size, criticality, requirements volatility– Award fees rewarding collaborative, effective change adaptation

• Continuous monitoring: INCOSE leading indicators; SERC effectiveness measures, or equivalent– Evidence preparation; schedule; exit criteria– Success-critical stakeholders participation; validation by experts

• Competitive prototyping critical success factors• Acquisition corps empowerment

– Next generation of tools, education, and training; career paths

173/1/2010

Types of Milestone ReviewsTypes of Milestone Reviews

• Schedule-based reviews (contract-driven)– We’ll hold the PDR on April 1 whether we have a design or not– High probability of proceeding into a Death March

• Event-based reviews (artifact-driven)– The design will be done by June 1, so we’ll have the review then– Large “Death by PowerPoint and UML” event

• Hard to avoid proceeding with many unresolved risks and interfaces

• Evidence-based commitment reviews (risk-driven)– Evidence provided in Feasibility Evidence Description (FED)

• A first-class deliverable• Based on concurrently engineered ConOps, specs, and plans

– Shortfalls in evidence are uncertainties and risks– Should be covered by risk mitigation plans– Stakeholders decide to commit based on risks of going forward

183/1/2010

Content of Evidence-Based ReviewsContent of Evidence-Based Reviews• Evidence provided by developer and validated by independent

experts that:If the system is built to the specified architecture, it will– Satisfy the specified operational concept and requirements

• Capability, interfaces, level of service, and evolution– Be buildable within the budgets and schedules in the plan– Generate a viable return on investment– Generate satisfactory outcomes for all of the success-critical stakeholders

• Shortfalls in evidence are uncertainties and risks – Should be resolved or covered by risk management plans

• Assessed in increasing detail at major anchor point milestones– Serves as basis for stakeholders’ commitment to proceed– Serves to synchronize and stabilize concurrently engineered elements

Can be used to strengthen current schedule- or event-based reviews193/1/2010

Scalable remotely controlled operationsScalable remotely controlled operations

203/1/2010

Total vs. Incremental Commitment – 4:1 RPVTotal vs. Incremental Commitment – 4:1 RPV

• Total commitment– Agent technology demo and PR: Can do 4:1 for $1B– Winning bidder: $800M; PDR in 120 days; 4:1 capability in 40 months– PDR: many outstanding risks, undefined interfaces– $800M, 40 months: “halfway” through integration and test– 1:1 IOC after $3B, 80 months

• Incremental commitment [number of competing teams]– $25M, 6 mo. to MDD [4]: may beat 1:2 with agent technology, but not 4:1– $75M, 8 mo. to Milestone A [3]: agent technology may do 1:1; some risks– $225M, 10 mo. to Milestone B [2]: validated architecture, high-risk elements– $675M, 18 mo. to IOC [1]: viable 1:1 capability– 1:1 IOC after $1B, 42 months

213/1/2010

Finance, Human Resource AllocationFinance, Human Resource Allocation

• Need balance of short-term, long-term budget commitments– Budget for long-term continuity but build in short-term off-ramp options– E.g., for technology maturity as in the DoDI 5000.02 example of EvA in chart 9– Prespecified total-package capabilities, budgets, and schedules generally

infeasible• Higher up-front SE schedules, costs for larger, more critical programs

– Earlier infrastructure and test plans and preparations– Heavier penalties for late rework; generally high payoffs for competitive

prototyping – Program-specific budgets, schedules, staffing established via mix of parametric

models and expert judgment• Need to address total cost of ownership, lifecycle ROI (DOTMLPF)

– Lifecycle success-critical stakeholder involvement– Mission effectiveness analysis with respect to alternatives; – Use to help prioritize increments

• Use as basis for monitoring progress vs. plans– Continuing update of resources required

223/1/2010

Risk of Delaying System-Specific Knowledge Blanchard-Fabrycky, 1998, pg. 37

Risk of Delaying System-Specific Knowledge Blanchard-Fabrycky, 1998, pg. 37

Detail Designand

Development

100

25

50

75

Conceptual-Preliminary

Design

Constructionand/or

Production

System Use, Phaseout,and Disposal

NEED

% Commitment to Technology,Configuration, Performance, Cost, etc.

Cost Incurred

System-Specific Knowledge

Ease of Change

233/1/2010

Improvement Opportunities

Effect on Size on SE Investment Payoff Effect on Size on SE Investment Payoff

243/1/2010

Effect of Volatility and Criticality on SE Investment Payoff

Effect of Volatility and Criticality on SE Investment Payoff

253/1/2010

COSYSMO

SizeDrivers

EffortMultipliers

Effort

Calibration

# Requirements# Interfaces# Scenarios# Algorithms+Volatility FactorReuse Factor

- Application factors-8 factors

- Team factors-6 factors

- Schedule driver WBS guided by ISO/IEC 15288

COSYSMO Operational ConceptCOSYSMO Operational Concept

263/1/2010

Workforce, Capabilities, And EmpowermentWorkforce, Capabilities, And Empowerment

• Increasing gaps in DoD SE Knowledge, Skills, and Abilities (KSAs)– Workforce gaps: domain experts retiring– Needed balance of specialists and systems thinkers

• Hardware/software /human factors combined expertise needs• Needed to avoid commitments to unscalable , easiest-first EvA

– Needs for life-long learning; learning how to learn– Applies to acquisition corps as well as SEs

• Need to anticipate trends in workforce capabilities• Tool empowerment

– Life cycle coverage, interoperability, domain tailorability– Tailorability to new and wider ranges of warfighter skills

273/1/2010

EvA Research and Technology NeedsEvA Research and Technology Needs• Incremental vs. start-over tools and methods (e.g., for formal methods)• Ambiguity-tolerant methods, processes, and tools• Powerful, flexible, composable models and simulations• Rapid change-impact analysis• Scalable methods, processes, and tools• Concurrent engineering support

– Collaboration technology; quality attribute tradeoff analysis tools– Value-based architecting and design methods, processes, and tools

• Continuous V&V; early, continuous testing; reliable autonomy• Use of multi-core computing processors for rapid multiple-option analysis• Integration of hardware, software , and human factors solution

approaches and architectures • Next-generation process maturity models• Evolutionary acquisition contract and incentive structures• Rapid technology maturation and adoption

283/1/2010

System/Software Architecture Mismatches- Maier, 2006

System/Software Architecture Mismatches- Maier, 2006

• System Hierarchy– Part-of relationships; no

shared parts– Function-centric; single data

dictionary– Interface data flows

– Static functional-physical allocation

29

• Software Hierarchy– Served-by relationships;

layered multi-access– Data-centric; class-object

data relations– Interface protocols;

concurrency challenges– Dynamic functional-

physical migration

3/1/2010

Examples of Architecture MismatchesExamples of Architecture Mismatches

• Fractionated, incompatible sensor data management

• “Touch Football” interface definition earned value– Full earned value taken for defining interface dataflow– No earned value left for defining interface dynamics

• Joining/leaving network, publish-subscribe, interrupt handling, security protocols, exception handling, mode transitions

– Result: all green EVMS turns red in integration

30

…

Sensor 1

SDMS1

Sensor 2

SDMS2

Sensor 3

SDMS3

Sensor n

SDMSn

……

3/1/2010







313/1/2010

References - I References - I Baldwin, C. and Clark, K., Design Rules: The Power of Modularity, MIT Press, 2000.Blanchard, B. and Fabrycky, W., Systems Engineering and Analysis. Upper Saddle River: Prentice

Hall, 1998.Boehm, B., Ingold, D., Dangle, K., Turner, R., P. Componation et al., “Early Identification of SE-

Related Program Risks,” SERC Technical Report and TR USC-CSSE-2009-518, Sept. 2009.Boehm, B., Port, D., Huang, L., and Brown, A.W., “Using the Spiral Model and MBASE to

Generate New Acquisition Process Models: SAIV, CAIV, and SCQAIV,” Cross Talk, January 2002, pp. 20-25.

Boehm, B., Brown, A. W., Basili, V, and Turner, R. “Spiral Acquisition of Software-Intensive Systems of Systems,” CrossTalk, May 2004, pp. 4-9.

Boehm, B., Valerdi, R., and Honour, E., "The ROI of Systems Engineering: Some Quantitative Results for Software-Intensive Systems," Systems Engineering, Fall 2008, pp. 221-234.

Boehm, B., and Lane, J., "Guide for Using the Incremental Commitment Model (ICM) for Systems Engineering of DoD Projects, v0.5,“ USC-CSSE-TR-2009-500.

Carter, A. , “Designation of Subprograms for Major Defense Acquisition Programs, USD(AT&L) Memorandum, June 23, 2009.

Cusumano, M., and Selby, R., Microsoft Secrets, Free Press, 1995.Department of Defense (DoD), Defense Acquisition Guidebook, version 1.6,

http://akss.dau.mil/dag/, 2006.Department of Defense (DoD), Systems Engineering Guide for System of Systems, June 2008.Department of Defense (DoD), Instruction 5000.02, Operation of the Defense Acquisition

System, December 2008.Fortune, J., “Estimating systems engineering reuse with the constructive systems engineering

cost model (COSYSMO),” USC Ph.D. Dissertation, December 2009.323/1/2010

http://akss.dau.mil/dag/

References - IIReferences - II

Gelosh, D., “Systems Engineering Workforce Development Update,” Proceedings, NDIA SE Conference , October 2009.

Hall, E.T., Beyond Culture, Anchor Books/Doubleday, 1976.INCOSE Systems Engineering Handbook v.3.1 INCOSE-TP-2003-002-03.1, 2007.Johnson , J., My Life Is Failure, Standish Group, 2006.Maier, M., “System and Software Architecture Reconciliation,” Systems Engineering 9 (2),

2006, pp. 146-159.Maranzano, J., et al. 2005. Architecture Reviews: Practice and Experience. IEEE Software,

Mar./Apr. 2005.Parnas, D., “Designing Software for Ease of Extension and Contraction,” IEEE Trans. SW

Engr., March 1979, pp. 128-137.Pew, R. W., and Mavor, A. S., Human-System Integration in the System Development

Process: A New Look, National Academy Press, 2007.Schroeder, T., “Integrating Systems and Software Engineering: Observations in Practice,

22nd International Annual Forum on COCOMO and Systems/Software Cost Modeling, 2007.

U.S. Congress, “Weapon System Acquisition Reform Act of 2009,” January 6, 2009.U.S. General Accountability Office, “Defense Acquisitions: Assessments of Selected

Major Weapon Programs,” March 2008.Valerdi, R, Systems Engineering Cost Estimation with COSYSMO, Wiley, 2010 (to appear).Young, J., “Prototyping and Competition,” USD(AT&L) Memorandum, September 19,

2007

3/1/2010 33

dod systems engineering and management implications for evolutionary acquisition of major defense...

Documents

complex systems of systems

dod systems engineering

forms of eva

acquisition practices

simple systems

acquisition approach

future dod challenges

future challenges