systems engineering = best practices nicholas m. torelli deputy director, human capital and...

Systems Engineering = Best Practices

Nicholas M. TorelliDeputy Director, Human Capital and Specialty Engineering

Systems and Software Engineering (SSE)Office of the Deputy Under Secretary of Defense

for Acquisition and Technology

Presented to2009 Systems Engineering Summit:

Harnessing Best Practices

Sponsored by the Huntsville Regional Chapter of INCOSE,

US Army, RDECOM, AMRDEC, DAU, and NASA

March 3, 2009

SYSTEMS & SOFTWARE ENGINEERING DIRECTORATE, DUSD(A&T) 2

DoD Vision for Systems Engineering

Systems engineering principles and disciplines are fully accepted and assimilated into the DoD acquisition workforce positioning the DoD for acquisition excellence leading to a stronger national defense.


SSE’s 2009 Priorities

• Acquisition work force development• Renewed focus on early application of systems

engineering to affect affordability and total ownership cost• Increased level of visibility for our role in developmental

test and evaluation• Systems of Systems engineering tools and techniques• Integration of program protection activities into acquisition

oversight to address cyber threat• Measuring results of our efforts


DoD SE Best Practice Continuum

Policy &Guidance

Teamwork &Collaboration

Developingthe Workforce& AdvancingSE Practice

Program Support& Assessment


CBAJoint Concepts

MS CMS B

Strategic Guidance

MS A

ICD TechnologyDevelopment

Engineering and Manufacturing

DevelopmentO&S

MDD

Materiel Solution Analysis

CDRPDR

Full Rate ProductionDecision Review

* DoDI 5000.02, 8 December 2008

JCIDS Process

Production and Deployment

CPDCDD

or

Overview of Acquisition Policy Changes*

• Mandatory Materiel Development Decision (MDD)• Mandatory competing prototypes before MS B• Mandatory PDR and a report to the MDA (“the sliding PDR”)

[PDR Report to MDA if before MS B; formal PDR Assessment by MDA if after MS B]

• Configuration Steering Boards at Component level to review all requirements changes

• Renewed emphasis on manufacturing during system development: – Re-titles SDD phase to EMD with two sub phases: Integrated System Design and

System Capability and Manufacturing Process Demonstration– Establishes consideration of manufacturing maturity at key decision points

• Mandatory system-level CDR with an initial product baseline and followed by a Post-CDR Report to the MDA

• Post-CDR Assessment by the MDA between EMD sub phases


Acquisition Policy Opportunities for SE

• Early SE engagement with programs• Program Support Reviews (PSRs) Pre-MS A/B/C• Risk Reduction activities (e.g., Technical Risk

assessment in AoAs, Competitive Prototyping)• SE Technical Reviews - Informed Trades for Feasible

Solutions • Developmental Test and Evaluation

– Integrated DT/OT– Updated T&E Strategy at MS A


New Systems Engineering Enclosure

• Codifies several previous SE policy memoranda• Codifies a number of SE-related policies and statutes

since 2003:– Environmental Safety and Occupational Health– Modular Open Systems Approach– Data Management and Technical Data Rights– Item Unique Identification– Spectrum Supportability– Corrosion Prevention and Control

• Introduces new policy on Configuration Management


DoD Guidance

• Defense Acquisition• Systems Engineering• Developmental Test and Evaluation (DTE) • Modeling and Simulation (M&S) • Safety• System Assurance

http://www.acq.osd.mil/sse/pg/guidance.html



Policy &Guidance





OSD Systems Engineering Strategy

• Achievable Acquisition Strategy• Enhanced Gate Review Process• Enhanced Staff Capabilities

• Milestones A/B Critical• Correct SE Staffing• Component Development Planning• Pre-MS A analysis

NR

C S

tud

y

Re

co

mm

en

da

tion

sS

ys

tem

ic A

na

lys

is

Re

co

mm

en

da

tion

s*

*Based on 3,700 Program Assessment findings from 40 Programs Support Reviews

ReliabilityWorkforce

Development

DT&E Revitalization

• Integrated DT/OT• Reliability Improvement• Early SE• Access to Relevant Data for

Evaluation & Decision Making

DS

B D

T&

E R

ep

ort

& 2

31

Re

po

rt R

ec

om

me

nd

atio

ns

OS

D P

olic

y &

Gu

ida

nc

eEnhanced SE

Pre-MS B


The Defense Acquisition Community126,033 Government and Military Certified Professionals

500,000+ Defense Industry PersonnelSSE Functional Leader for SE, T&E, and PQM workforce

Workforce Development


SE Human Capital Strategy

• FY08 NDAA Section 852: DoD Acquisition Workforce Development Fund: >$300M per year across DoD– SE, PQM and T&E initiatives to Recruit, Train, and Retain the

workforce– DoD Human Capital Initiative - Published Annex for SPRDE,

PQM, and T&E Career Fields

• SE “core competency” assessment effort; completion - Summer 2009

• “Program Systems Engineer” career path• Partnership with INCOSE SE Certification Program

– “CSEP-Acq” aligned with Defense Acquisition Guidebook• Equivalency granted for DAU Courses SYS101 and SYS202

– Expanding potential for Industry “certifications” • what the future could hold


Human Capital Initiatives (Defense Acquisition Workforce Development

Fund 1)

1 Based on NDAA Section 852, Defense Acquisition Workforce Development Act

Defense Acquisition Workforce

Development Fund

Recruiting & Hiring

Training & Development

Retention & Recognition

Intern Programs

Recruiting Incentives

Outreach Programs

Journeyman Hiring

Highly Qualified Experts

Training Enhancement & Capacity Expansion

Acquisition WorkforceManagement & Student

Information System

Competency Management &

Assessments

Workforce Planning

Retention & Recognition

Career Broadening & Academic Programs


Notional DoD Systems Engineering Population

Wo

rkfo

rce

Siz

e

25-35 35-45 45-55 >55Workforce Age


Notional DoD Systems Engineering Workforce Strategy

Wo

rkfo

rce

Siz

e

25-35 35-45 45-55 >55Workforce Age

Train:Mentor

Retain

Recruit:Journeymen

Recruit: Highly

Qualified Experts

Rec

ruit:

In

tern

sRetain:

Rotational Training AssignmentsEducation with Industry

Qualification CardsIncentives

Hazardous Duty Pay for PSEs??


NDIA Systems Engineering Division E&T Committee

Mission: Strengthen Systems

Engineering capabilities through education, training and experience across government,

industry, and academia.

2009 Task:To address the issue of

“self-proclaimed” Systems Engineers, the

E&T Committee is attempting to determine the core essential set of

competencies that distinguish systems

engineers from domain engineers.

Contact Govt Committee Chair:Dr. Don Gelosh, CSEP-Acq ODUSD (A&T) SSE / HC [email protected]


Co

mp

. L

evel

SE Competency Model

Domain

Knowledge

Systems ThinkingUnit of

CompetenceCompetence Element(s)

Essence of

Systems

Problem Solving

Unit of


Stakeholder

Requirements Analysis

Requirements Analysis

Unit of


Decision Management

Risk Management

SE Technical

SE Technical Mgmt

Systems Engineer

Composite Competency Map

Architecture Design

Implementation

Lvl

Lvl

Lvl

Configuration

Management

Analytical

Analytical

Analytical

Analytical

Technical

Management

Technical

Management

Technical

Management

Technical

Management

Technical Planning

Behavior

Structure

Goal

Problem

Solving

Method

Analytical

Analytical

Time

Minimum

Essential

SE Criteria

Growth Professional / InterpersonalUnit of

CompetenceCompetence Element(s) Lvl

Life Cycle ViewUnit of

CompetenceCompetence Element(s) Lvl

Material

Solution Analysis

Technology

Development

EMD

Production &

Deployment

Operations &

Support

Factors that Affect

the SE Competency Profile


Research to Practice

SERC Governance & Collaborators

• Co-Sponsors• Government PMO• Tasking Activities• Industry• Associations• Academia

SERC Governance & Collaborators

• Co-Sponsors• Government PMO• Tasking Activities• Industry• Associations• Academia

Ongoing Developmen

t Acquisition Community (DoD and Industry)

SERC

Provide Lessons Learned and Challenges

Deliver New/Improved SE Methods,

Processes & Tools (MPTs)

PolicyGuidanceEducation & Training


Need for SE Research

• State-of-the-practice must keep up with application needs of DoD acquisition– Methods, processes, and tools to enable effective acquisition and

sustainment of systems– Leveraging Modeling & Simulation for SE

• Size and complexity of modern systems drive need for commensurate extensions of SE, – To System of Systems (SoS), to complex systems, to

“architecting,” to net-centric sets of services– Validation & Verification challenges

• SE “theory and practice” should be inclusive of, and establish linkages to, challenged sub-specialty areas– Software engineering, reliability engineering, system safety,

costing, etc.


Stevens-SERC Team

• Auburn University• Air Force Institute of Technology• Carnegie Mellon University• Fraunhoffer Center at UMD• Massachusetts Institute of

Technology• Missouri University of Science and

Technology (S&T)• Pennsylvania State University• Southern Methodist University

Stevens Institute of TechnologyLead University

• Texas A&M University • Texas Tech University• University of Alabama in Huntsville • University of California at San Diego • University of Maryland• University of Massachusetts• University of Southern California• University of Virginia • Wayne State University

University Partners

The DoD Systems Engineering Research Center will be responsible for systems engineering research that supports the development, integration, testing and sustainability of complex defense systems, enterprises and services. Stevens has MOUs to develop enhanced SE courseware for competency development within DoD with AFIT, NPS and DAU. Further, SERC members are located in 11 states, near many DoD facilities and all DAU campuses.


Initial SERC Research Tasks

• “Assessing Systems Engineering Effectiveness in Major Defense Acquisition Programs (MDAPs)”– USC task lead, with support from Stevens, Fraunhofer Center, MIT,

University of Alabama at Huntsville– Characterize SE effectiveness within the context of DoD acquisition and

identify methods of measurement suitable for application to project execution organizations (e.g., defense contractors), program management organizations (e.g., project managers and program executive offices) and oversight organizations (e.g., OUSD(AT&L)) (OSD)

• “Evaluation of Systems Engineering Methods, Processes, and Tools (MPT) on Department of Defense and Intelligence Community Programs”– Stevens task lead, with support from USC, University of Alabama at

Huntsville, and Fraunhofer– Examine and recommend areas for advancing current SE methods,

processes, and tools (MPTs) as they are applied across the DoD acquisition life cycle focusing on three different development environments: individual weapons systems, SoS, and network-centric systems (NSA)


Ways Your Organizations Can Become Involved…

• Identify SE research challenges• Provide funding to sponsor research as a Tasking Activity• Collaborate on DoD-sponsored research by identifying

pilot programs to participate, provide relevant acquisition information, identify subject matter experts

• Make use of research findings to improve systems engineering on acquisition programs



Policy &Guidance





Our World of Stakeholders

• DoD– Components and Agencies– Product Centers– Program Executive Offices– Acquisition Programs (PMs

and CSEs)– Practicing PMs and SEs– Other DAWIA Career Fields

• Industry– First tier contractors (e.g.,

Lockheed Martin, Boeing, Northrop-Grumman, Raytheon)

– Second tier contractors– Practicing SEs– SE tool vendors

• Education & Training Institutions– DoD organizations: DAU,

AFIT, NPS– US universities with SE

programs (~60)– SE short course providers

• Professional and Industrial Associations– DAU Alumni Association– INCOSE– NDIA SE Division, other assoc

Div– IEEE Standards Committee;

Systems Council– AIAA SE Committee– TechAmerica– ISO


DoD Challenge: “A”cquisition Process

Requirements Budgeting

Acquisition

• ICD, CDD, CPD• Future Threat and Op

Tempo

• Planning, Programming, and Budgeting Process

• FYDP• POM Process• Appropriation and

Authorization of Funds

• Acquisition Policy, Guidance, and Oversight.

• MDAP Decision Authority

“Sweet Spot”


Requirements Budgeting

Acquisition

“Sweet Spot”

DoD Goal: Increase the Overlap


Systems & Software EngineeringOrganization Overview


Congress

Sec Def

RequirementsDevelopers

Service AcquisitionExecutives

PEOs

Program Offices

Prime ContractorsSecond Tier Contractors

Education & Collaboration InfrastructureProfessional / Industry Associations

(NDIA, INCOSE, AIA, ITEA, TechAmerica, etc.) DAU, Academic Institutions, SERC

Opportunities for SSE Engagement

SSEPolicy & Guidance

• Systems Engineering

• DT&E

Program Support

• Program Support Reviews

• OIPT, T&E and SE WIPTs

• AOTR, Post-CDR Review & Assessment

Workforce Planning

• Competency Models

• Certification Reqts

• Education & Training

Emerging Concepts

• Systems of Systems

• SE Research

Outreach

• SE Forum

• Engagement Strategy

Statutory Direction

ICD, CDD, CPD

DAB, ITAR, DSAB, OIPT

PSR, SEP, TEMP,Technical Reviews

CSEP-Acq, Research, Industry-University SE Workforce Initiatives,

Industry-Gov’t Projects

AT&L Direction


DoD Human Systems Integration (HSI)

Problem• HSI is not being consistently

integrated within DoD acquisition resulting in increased ownership costs and lower system effectiveness.

Approach• Assign DUSD(A&T) Executive

Authority to define policy, as necessary, to implement changes.

• Maximize the good work that has been done (policies, plans, methods, tools and standards).

• Engage DoD stakeholders and strengthen industry affiliations including NDIA, TechAmerica and INCOSE.

• Identify gaps and apply necessary resources to resolve gaps or barriers to successful HSI implementation.

Cross-DoD and Technical Community Collaboration


Collaboration with INCOSE

• International Panel: “What Defines a Systems Engineer? Comparing and Contrasting Global Perspectives on Systems Engineering Competency”

– Moderator: Dr. Don Gelosh, Human Capital Strategy and Planning, OSD Directorate of Systems and Software Engineering

– Dr. Arthur Pyster, Distinguished Research Professor, School of Systems and Enterprises, Stevens Institute of Technology; Deputy Executive Director, DoD Systems Engineering Research Center; and member of the INCOSE Board of Directors

– Dr. John Snoderly, Program Director, Systems Engineering, Defense Acquisition University

– Samantha Brown, President-Elect of INCOSE and Systems Engineering Innovation Centre (SEIC), Loughborough, UK

– Mark Kupeski, Director, Complex Systems Integration, IBM Global Business Services

– Dr. U. Dinesh Kumar, Professor in Quantitative Methods and Information Systems, Indian Institute of Management Bangalore

– Professor Stephen Cook, Director, Defence and Systems Institute, University of South Australia

July 19 – 23, 2009


INCOSE Multi-LevelProfessional Certification

Foundation Level

ESEP Expert Systems Engineering ProfessionalCSEP-Acq CSEP w/ US DoD AcquisitionCSEP Certified Systems Engineering ProfessionalASEP Associate Systems Engineering Professional

Entry Level

DisciplineExtensions

Senior Level


CSEP with US DoD Acquisition Extensions

• Targeted towards Systems Engineers who support or work in a US Department of the Defense acquisition environment

• Same core CSEPs experience, education, and knowledge requirements

• Additional acquisition knowledge items tested• Available since July 2008


DoD Best Practices Clearinghouse (BPCh)

https://bpch.dau.mil/

The DoD Acquisition Best Practices Clearinghouse (BPCh) facilitates the selection and implementation of systems engineering and software acquisition practices appropriate to the needs of individual acquisition programs.

The BPCh uses an evidence-based approach, linking to existing resources that describe how to implement various best practices.


We Need You!

• Browse the Clearinghouse for applicable best practices• Provide feedback on practices• Submit additional evidence for an existing practice• Submit a lead or a possible best practice• Tell others about the Clearinghouse• Consider becoming an editor to help vet practices

submitted in your area of expertise



Policy &Guidance





OSD’s Program Support Reviews…“The Source of Systemic Root Cause Analysis

Data”

Defense Acquisition ProgramDefense Acquisition ProgramSupport Methodology (DAPS) vSupport Methodology (DAPS) v1.11.1

Defense Acquisition ProgramDefense Acquisition ProgramSupport Methodology (DAPS) vSupport Methodology (DAPS) v1.11.1DAPS

1.0 Mission Capabilities/Requirements Assessment AreaSub-Area 1.1 – Mission Capabilities/Requirements

2.0 Resources Assessment AreaSub-Area 2.1 – Technology Development Planning and AllocationSub-Area 2.2 – PersonnelSub-Area 2.3 – FacilitiesSub-Area 2.4 – Engineering Tools

3.0 Management Assessment AreaSub-Area 3.1 – Technology Development StrategySub-Area 3.2 – Technology Development PlanningSub-Area 3.3 – Program and Project ManagementSub-Area 3.4 – Contracting and SubcontractingSub-Area 3.5 – Communication

4.0 Technical Process Assessment AreaSub-Area 4.1 – Technology Selection AssessmentSub-Area 4.2 – Requirements DevelopmentSub-Area 4.3 – Functional Analysis & AllocationSub-Area 4.4 – Design SynthesisSub-Area 4.5 – System Integration, Test and VerificationSub-Area 4.6 – Transition to SDDSub-Area 4.7 – Process Improvement

5.0 Technical Product Assessment AreaSub-Area 5.1 – System DescriptionSub-Area 5.2 – System PerformanceSub-Area 5.3 – System Attributes

6.0 Environment Assessment AreaSub-Area 6.1 – Statutory and Regulatory Environment

DAPS1.0 Mission Capabilities/Requirements Assessment Area

Sub-Area 1.1 – Mission Capabilities/Requirements2.0 Resources Assessment Area

Sub-Area 2.1 – Technology Development Planning and AllocationSub-Area 2.2 – PersonnelSub-Area 2.3 – FacilitiesSub-Area 2.4 – Engineering Tools

3.0 Management Assessment AreaSub-Area 3.1 – Technology Development StrategySub-Area 3.2 – Technology Development PlanningSub-Area 3.3 – Program and Project ManagementSub-Area 3.4 – Contracting and SubcontractingSub-Area 3.5 – Communication

4.0 Technical Process Assessment AreaSub-Area 4.1 – Technology Selection AssessmentSub-Area 4.2 – Requirements DevelopmentSub-Area 4.3 – Functional Analysis & AllocationSub-Area 4.4 – Design SynthesisSub-Area 4.5 – System Integration, Test and VerificationSub-Area 4.6 – Transition to SDDSub-Area 4.7 – Process Improvement

5.0 Technical Product Assessment AreaSub-Area 5.1 – System DescriptionSub-Area 5.2 – System PerformanceSub-Area 5.3 – System Attributes

6.0 Environment Assessment AreaSub-Area 6.1 – Statutory and Regulatory Environment

Plan

Conduct

Analyze

Report

Tools & Materials• Tailorable, Scalable

Methodology (DAPS v1.1)• Templates• Criteria• Questions• Training Materials• Execution Guidance• Subject Matter Expertise

Rigor in process …

…ensures integrity


Program Support Review Data Set (since March 2004)

PSRs/NARs completed: 57AOTRs completed: 13Nunn-McCurdy Certification: 13Participation on Service-led IRTs: 3Technical Assessments: 13Reviews planned for CY08:

PSRs/NARs: 10AOTRs: 1NM: 1

Decision Support Reviews

DAE Review5%

OTRR9%

Other25%

Pre-MS C19%

Pre-MS A3%

Pre-MS B26%

Nunn-McCurdy

13%

Service-Managed Acquisitions

Marine Corps 10%

Army 27%

Navy 21%

Air Force 33%

Agencies 9%

Programs by Domain Area

Unmanned 4%

Land 13%

C2-ISR 13%

Ships 8%Munitions 3%

Rotary Wing 21%

Comms 3%

Space 4%

Business 2%Missiles 9%

Fixed Wing 18%

Other 2%

Data derived from

diverse & broad program set


• Policy/Guidance• Education & Training• Best Practices

• Policy/Guidance• Education & Training• Best Practices

• Other Processes (JCIDS, etc)• Oversight (DAB/ITAB)• Execution (staffing)

• Other Processes (JCIDS, etc)• Oversight (DAB/ITAB)• Execution (staffing)

DoD AcquisitionCommunity

Program Review

Findings

ProgramUnique

Solutions

ProgramEffects &

Root Cause

Systemic Issues

SystemicRoot

Causes

Corrective Actions

Tactical, Program and Portfolio ManagementTactical, Program and Portfolio Management

Strategic ManagementStrategic Management

OSD Systemic Analysis: Data Model

Use collective set of program findings…

… to identify systemic issues at the root cause level…

… and develop recommendations that mitigate

problems at their source!


• Use “Core” and “Systemic” Root Cause tagging to cull data into focused subsets• Analyze and trend using Pareto charting and Pivot Tables

PSRfindings

PSRfindings

“Negative”Findings

“Negative”Findings

AnalysisPareto &

PivotTables)

AnalysisPareto &

PivotTables)

FindingsFindingsPrelim.

AnalysisResults

Prelim.AnalysisResults

Deriving Systemic Issues…

Hypothesis testing: Re-analyze, review…44 Programs

Systemic Issues

Systemic Issues

Occurred on 50% or more of the programs

Occurred on 50% or more of the programs


SRCA Milestones & Results

OUSD/SSE root cause analysis yields systemic issues and preliminary recommendations

OUSD/SSE sponsored NDIA Task Group to develop executable acquisition recommendations

NDIA Task Group concludes effort and proposes actions for both government and industry

Sep 07 – Feb 08Sep 07 – Feb 08

Mar 08 – Oct 08Mar 08 – Oct 08

Oct 08 – Dec 08Oct 08 – Dec 08

3 Recommendation

Areas

3 Recommendation

Areas

Acquisition Strategy & Planning

(ASP)

Acquisition Strategy & Planning

(ASP)

DecisionGate

Review(DGR)

DecisionGate

Review(DGR)

Enhanced Staff

Capability(ESC)

Enhanced Staff

Capability(ESC)Early & sufficient

planningEarly & sufficient

planningClear, enforceable execution criteriaClear, enforceable execution criteria

People: the right skills and numbersPeople: the right skills and numbers


Full RateProductionDecisionReview

CBAJoint Concepts

MS CMS B

Strategic Guidance

MS A

ICD TechnologyDevelopment

CDDEngineering and Manufacturing

Development and Demonstration

CPDProduction and

Deployment O&S

New Opportunitiesfor Independent Reviews

MDD

Materiel Solution

Analysis

* PDR – Preliminary Design Review * CDR – Critical Design Review * OTRR – Operational Test Readiness Review

What’s relevant: Mandatory Milestone A for all “major weapon systems” MS B after system-level PDR* and a PDR Report to the MDA EMDD with Post-CDR* Report and MDA Assessment PSR and AOTR in policy

OTRRPotential Independent Technical Reviews - PSRs and AOTRs

Program Support Reviews (PSRs) All ACAT ID and IAM To inform the MDA on technical planning and management processes thru risk identification and mitigation recommendations To support OIPT program reviews and others as requested by the MDA

Assessments of Operational Test Readiness (AOTRs) All ACAT ID and special interest programs To inform the MDA, DOTE, & CAE of risk of a system failing to meet operational suitability and effectiveness goals To support CAE determination of materiel readiness for IOT&E



Policy &Guidance





The Mission:Delivering Timely and Affordable Capabilities to the Warfighter

The Defense Acquisition Community126,033 Government and Military Certified Professionals

500,000+ Defense Industry Personnel

Always Our Focus

For additional information: http://www.acq.osd.mil/sse


DoD Guidance (1/2)• Defense Acquisition

– Defense Acquisition Guidebook (DAG)– Integrated Defense Acquisition, Technology, & Logistics Life Cycle Management Framework, Version 5.3.2:

December 8, 2008 • Systems Engineering

– DAG Chapter 4, Systems Engineering– Systems Engineering Plan (SEP) Preparation Guide, Version 2.01, April 2008– SEP Frequently Asked Questions (FAQs), February 8, 2008– SE WIPT Brief: An Overview of Technical Planning and Systems Engineering Plan (SEP) Development,

Version 1.0, March 20, 2008– SE Working Integrated Product Team (WIPT) Generic Charter Template – Systems Engineering Guide for Systems of Systems, Version 1.0, August 2008– Systems Engineering Assessment Methodology, Defense Acquisition Program Support (DAPS), Version 2.0,

October 21, 2008– Guide for Integrating Systems Engineering into DoD Acquisition Contracts, Version 1.0, December 11, 2006 – Risk Management Guide for DoD Acquisition, 6th Edition, Version 1, August 2006– Integrated Master Plan / Integrated Master Schedule Preparation and Use Guide, Version 0.9, October 21,

2005– Program Manager's Guide: A Modular Open Systems Approach (MOSA) to Acquisition, Version 2.0,

September 2004 – Modular Open Systems Approach (MOSA) Program Assessment and Rating Tool (PART), Version 1.02 – DoD Guide for Achieving Reliability, Availability and Maintainability, August 1, 2005– Program Reliability and Maintainability Review Template, Version 1.0, August 15, 2008 – Designing and Assessing Supportability in DOD Weapon Systems: A Guide to Increased Reliability and

Reduced Logistics Footprint, October 24, 2003 – Technical Review Checklists



DoD Guidance (2/2)

• Developmental Test and Evaluation (DTE) – DAG Chapter 9, Integrated Test and Evaluation– Guide on Incorporating Test and Evaluation into DoD Acquisition Contracts, FINAL DRAFT

FOR FINAL APPROVAL VERSION, September 2008 • Modeling and Simulation (M&S)

– DoD 5000.59-M, DoD Modeling and Simulation (M&S) Glossary, January 15, 1998– Recommended Practices Guide for Verification, Validation, and Accreditation (VV&A), Build

3.0, September 2006– Guide for Modeling and Simulation for the Acquisition Workforce, Version 1.01, October 2008

• Safety– System Safety — ESOH Management Evaluation Criteria for DoD Acquisition, Version 1.1,

January 2007 – Joint Systems Safety Review Guide for USSOCOM Programs, Version 1.1, October 12, 2007– Unmanned System Safety Guide for DoD Acquisition, Version 0.92, June 27, 2007– Joint Weapons and Laser Safety Review Guide (Draft), Version 0.92, September 4, 2007– ESOH in Acquisition: Integrating ESOH into SE, Version 3.0, January 2008

• System Assurance – DAG Chapter 7, Acquiring Information Technology and National Security Systems – DAG Chapter 8, Intelligence, Counterintelligence, and Security Support – DoD 5200.1-M, Acquisition Systems Protection Program, March 16, 1994 – Engineering for System Assurance, Version 1.0, October 2008



DoD Best Practices Clearinghouse (BPCh) Program Goals

A Single Source For answers about

practices, how to apply them, when they are good to use; lessons learned; and risks to

avoid

Useful Information Help finding,

selecting, and implementing

practices appropriate to User’s situation; fill the gap between “what” and “how”

Validated practices Consistent, verifiable

information

Repository

Active Knowledge BaseNot just another practice

list; experience data updated, expanded refined; encourages

organic growth

Living KnowledgeIntegration with

DoD communities of practice (ACC)

and experts

systems engineering = best practices nicholas m. torelli deputy director, human capital and...

Documents

specialty engineering

systems engineering

se early se engagement

early application of

se access

number of se

system development

previous se policy memoranda