acquisition modeling & simulation working group (amswg
TRANSCRIPT
ITEA LVC Conference
24-27 Jan 2011 Page-1
Defense Modeling & Simulation
Verification, Validation & Accreditation
Campaign Plan
John Diem, Associate Director (Services)
OSD/AT&L Modeling & Simulation Coordination Office
ITEA LVC Conference: January 24 – 27 , 2011
DISTRIBUTION STATEMENT A -- Cleared for public release by OSR (case numbers 10-S-3029 & 10-S-3232 apply).
ITEA LVC Conference
24 - 27 Jan 2011 Page-2
VV&A Campaign Plan Summary
• Established May 2010 to be broadly focused – policy,
standards, guides, tools, methodologies
• Driven largely by work led by Acquisition M&S Working
Group
• On-going effort – work in 2011 includes several efforts to
gain feedback from users and sponsors
• Feedback will influence re-look of key policy and standards
components in 2012, including:
– DOD 5000.61 (“DoD M&S VV&A”)
– MIL-STD-3022 (“Documentation of VV&A for M&S”)
– DOD VV&A Guidance (“The VV&A Recommended Practices Guide (RPG)”)
ITEA LVC Conference
24 - 27 Jan 2011 Page-3
Key VV&A Documents
• DoD VV&A Policy: DoDI 5000.61 ―DoD M&S VV&A‖
• DoD VV&A Guidance: The VV&A Recommended
Practices Guide (RPG)
• MIL-STD-3022 ―Documentation of VV&A for M&S‖
• IEEE 1516.4-2007 ―IEEE Recommended Practice
for VV&A of a HLA Federation‖
• IEEE 1278.4-1997(R2002) ―IEEE Recommended
Practice for Distributed Interactive Simulation (DIS)
VV&A)‖
3
ITEA LVC Conference
24 - 27 Jan 2011 Page-4
• Implements policy, assigns responsibilities, prescribes
procedures, and establishes common terminology
• Current version dated December 9, 2009
• Modified to reflect updated format requirements as defined
in DoDI 5025.01 (Oct 2007)
• Streamlined document that establishes responsibilities,
procedures, information requirements, and key definitions
• Incorporated reference to MIL-STD 3022
• Scheduled for review in 2014 – work will begin in 2012
DoD Instruction 5000.61
―DoD M&S VV&A‖
DoDI 5000.61 dated 9 December 2009
available at http://www.dtic.mil/whs/directives/corres/pdf/500061p.pdf
ITEA LVC Conference
24 - 27 Jan 2011 Page-5
5
DoD VV&A Recommended Practices Guide
• RPG promotes the effective application of VV&A
methodology
– Defines philosophy, principles and methodologies
– Developed by subject matter experts
– Built to address diverse audiences, situation and
applications
• Web-based format intended to:
– Provide the foundation for multi-tiered, multi-
dimensional information at various levels of depth
and detail
– Be extensible to allow for the addition of
information over time
– Allow the reader to build guidance tailored to a
situation or application
Available at
vva.msco.mil
ITEA LVC Conference
24 - 27 Jan 2011 Page-6
6
MIL-STD-3022
• 2005-2006: DoD-sponsored Tri-Service VV&A
Templates Tiger Team developed templates for
four core VV&A documents
– Accreditation Plan, V&V Plan
– V&V Report, Accreditation Report
• 2007: Processed through DoD M&S
management system and Defense
Standardization Program
• 2008: Approved as a DoD Standard Practice
• May be cited as a solicitation requirement and
Data Item Descriptions (DIDs) included on
Contract Data Requirements List (CDRL)
– DI-MSSM-81750 DoD M&S Accreditation
Plan
– DI-MSSM-81751 DoD M&S V&V Plan
– DI-MSSM-81752 DoD M&S V&V Report
– DI-MSSM-81753 DoD M&S Accreditation
Report
• 2009: DoD Instruction 5000.61, DoD M&S
VV&A, references MIL-STD-3022 Free: Available from the Acquisition Streamlining
and Standardization Information System (ASSIST)
ITEA LVC Conference
24 - 27 Jan 2011 Page-7
Information Shared across the Templates
Accreditation Plan V&V Plan V&V Report Accreditation Report
Executive Summary Executive Summary Executive Summary Executive Summary
1 Problem Statement 1 Problem Statement 1 Problem Statement 1 Problem Statement
2 M&S Requirements and Acceptability
Criteria
2 M&S Requirements and
Acceptability Criteria
2 M&S Requirements and
Acceptability Criteria
2 M&S Requirements and
Acceptability Criteria
3 M&S Assumptions, Capabilities,
Limitations & Risks/Impacts
3 M&S Assumptions,
Capabilities, Limitations &
Risks/Impacts
3 M&S Assumptions, Capabilities,
Limitations & Risks/Impacts
3 M&S Assumptions, Capabilities,
Limitations & Risks/Impacts
4 Accreditation Methodology 4 V&V Methodology 4 V&V Task Analysis 4 Accreditation Assessment
5 Accreditation Issues 5 V&V Issues 5 V&V Recommendations 5 Accreditation Recommendations
6 Key Participants 6 Key Participants 6 Key Participants 6 Key Participants
7 Planned Accreditation Resources 7 Planned V&V Resources 7 Actual V&V Resources
Expended
7 Actual Accreditation Resources
Expended
8 V&V Lessons Learned 8 Accreditation Lessons Learned
Suggested Appendices
A M&S Description
B M&S Requirements Traceability
Matrix
C Basis of Comparison
D References
E Acronyms
F Glossary
G Accreditation Programmatics
H Distribution List
Suggested Appendices
A M&S Description
B M&S Requirements
Traceability Matrix
C Basis of Comparison
D References
E Acronyms
F Glossary
G V&V Programmatics
H Distribution List
I Accreditation Plan
Suggested Appendices
A M&S Description
B M&S Requirements Traceability
Matrix
C Basis of Comparison
D References
E Acronyms
F Glossary
G V&V Programmatics
H Distribution List
I V&V Plan
J Test Information
Suggested Appendices
A M&S Description
B M&S Requirements Traceability
Matrix
C Basis of Comparison
D References
E Acronyms
F Glossary
G Accreditation Programmatics
H Distribution List
I Accreditation Plan
J V&V Report
Information that is common and shared across the documents appears in italicized font in the table above
ITEA LVC Conference
24 - 27 Jan 2011 Page-8
DoD VV&A Documentation Tool
http://msco.mil
Implements MIL-STD 3022 and Data Item Descriptions
DI-MSSM-81750 - DI-MSSM-81751 - DI-MSSM-81752 - DI-MSSM-81753
ITEA LVC Conference
24 - 27 Jan 2011 Page-9
DoD VV&A Documentation tool
Producers register here or
log in if already registered.
There is online help for
using the tool, as well as
help for VV&A in general.
The DVDT automates the
MIL-STD-3022
documentation templates.
A benefit of using the tool
is easier management of
VV&A information. The
tool is designed to walk
the Producer through
VV&A planning,
implementation, and
reporting.
Register, log in, and create a project to download a set of XML-
enabled document templates for local use. Compliant with DoD M&S
Community of Interest Discovery Metadata Specification (MSC-DMS).
https://dvdt.nmso.navy.mil
ITEA LVC Conference
24 - 27 Jan 2011 Page-10
IEEE Std 1516.4
• Defines processes and
procedures to implement VV&A
for federations being developed
using the HLA Federation
Development and Execution
Process (FEDEP) (IEEE Std
1516.3™-2003)
• Focuses on unique aspects of
VV&A of federations as a higher-
level framework into which
VV&A practices can be
integrated and tailored for
specific uses.
• Provides implementation-level
guidance to VV&A practitioners.
ITEA LVC Conference
24 - 27 Jan 2011 Page-11
IEEE Std. 1278.4
• Guidelines established for the VV&A
of distributed interactive simulation
(DIS) exercises.
• Provides ―how-to‖ procedures for
planning and conducting DIS
exercise VV&A
• Intended for use in conjunction with
IEEE Recommended Practice for DIS-
Exercise Management and Feedback
(IEEE Std 1278.3-1996)
• IEEE-SA approved reaffirmation of
IEEE Std 1278.3-1996 and IEEE Std
1278.4-1997 in June 2010
• Presents data flow and connectivity
for all proposed verification and
validation activities and provides
rationale and justification for each
step.
• Provides VV&A guidance to exercise
users/sponsors and developers.
ITEA LVC Conference
24 - 27 Jan 2011 Page-12
Where to Obtain VV&A Standards
• MIL-STD-3022, Documentation of Verification, Validation, and
Accreditation for Models and Simulations, 28 Jan 2008
– https://assist.daps.dla.mil/quicksearch/
• IEEE Std 1516.4™-2007, IEEE Recommended Practice for
Verification, Validation, and Accreditation of a Federation—An
Overlay to the High Level Architecture Federation Development
and Execution Process, 20 December 2007
– https://sbwsweb.ieee.org/
• IEEE Std 1278.4-1997, IEEE Trial-Use Recommended Practice for
Distributed Interactive Simulation - Verification, Validation, and
Accreditation, 20 July 1998 (reaffirmed June 2010)
– https://sbwsweb.ieee.org/
IEEE Standards are free to SISO Members
ITEA LVC Conference
24 - 27 Jan 2011 Page-13
Acquisition Community Led VV&A Efforts - key inputs to Campaign Plan
VV&A
Strategy
ITEA LVC Conference
24 - 27 Jan 2011 Page-14
Acquisition Community-led
VV&A High Level Task (HLT) Summaries
V-AQ-2: ―Risk Based Methodology for Verification, Validation and
Accreditation (VV&A)‖ The degree of VV&A required is explicitly tied to both
M&S use and the user risk incurred if the M&S does not provide accurate results.
A methodology that tailors VV&A planning and implementation based on known
risk factors will provide a framework in which VV&A implementation trade-offs
can be made, information/fidelity requirements can be assessed, and a VV&A
cost model can be developed.
V-C-2: ―Improving VV&A Implementation‖ Increase VV&A implementation
and enhance M&S credibility by transforming VV&A practices from current
subjective methods into objective examples or use cases. Explore emerging
technologies, standards, and applicable methods that could be applied to reduce
costs, schedule, and improve reuse.
ITEA LVC Conference
24 - 27 Jan 2011 Page-15
Risk-Based VV&A (M&S PE Funded V-AQ-2)
The purpose of this effort is to:
Improve Department of Defense
(DoD) Modeling and Simulation
(M&S) Verification, Validation,
Accreditation (VV&A) by
establishing an effective risk-based
methodology for VV&A.
Since the cost of verifying and
validating simulations is often high,
V&V investments should be
weighed against the risk of making
a bad decision because of
unreliable M&S results.
Risk Resources
VV&A
Balancing the cost of knowing
against the risk of assuming.
ITEA LVC Conference
24 - 27 Jan 2011 Page-16
Systems Engineering Research Center VV&A Topic
(M&S PE Funded V-C-2)
Proposed research topic provided SERC:
Explore & promote a deeper
understanding of emerging & current
approaches, technologies, or practices
underway within the academic
community & gain a broader perspective
on their potential applicability to the M&S
community V&V needs
The 19 SERC university partners
generated six responses, with selection
of two proposals for implementation
lasting 6 to 12 months
Two performers:
Georgia Institute of Technology
University of Alabama Huntsville
ITEA LVC Conference
24 - 27 Jan 2011 Page-17
Georgia Institute of Technology VV&A Study
(M&S PE Funded V-C-2)
The purpose of this SERC effort :
Demonstrate how to address many of
the gaps identified in this VV&A
Research Topic by applying SysML and
MBE/MBSE technology
Engage practitioners in the DoD
ecosystem to prioritize needs and to
test resulting concepts
Apply concepts in one or more key
DoD test beds
ITEA LVC Conference
24 - 27 Jan 2011 Page-18
University of Alabama at Huntsville VV&A Study
(M&S PE Funded V-C-2)
The purpose of this SERC effort :
Focus on verification & validation
(V&V) of complex systems
A survey & evaluations of useful tools
Leverages experience working the
government customers in the Army Aviation
Directorate, Software Engineering
Directorate, Redstone Test Center, and PEO-
Aviation.
Will benefit from insights resulting from
Systems Architecture Virtual Integration
(SAVI), an international industry consortium
involving Boeing, Airbus, Lockheed Martin,
BAE, FAA, SEI, etc
ITEA LVC Conference
24 - 27 Jan 2011 Page-20
Domain-relevant “Best of Breed” M&S Tool Suite VV&A NAVAIR Use Case
(M&S PE Funded V-C-2)
Multiple communities within the
Department of Defense are interested in
accelerating schedules, improving
credibility, & reducing costs of technical
execution of M&S support to decision-
makers through tailored accreditation
support
The scope of this effort was to:
Bring knowledge of existing VV&A
―best practices‖ to funded
implementation of credibility
assessments of high-priority DoD
M&S applications;
Examine the practicality of
leveraging Service-specific VV&A for
domain-relevant Department-wide
tool suite applications; and
Identify Service-specific needs &
estimate both cost-avoidance &
incremental costs for plan to address.
ITEA LVC Conference
24 - 27 Jan 2011 Page-21
VV&A Program Management & Integration
(M&S PE Funded V-C-2)
The VV&A Subcommittee of the
Acquisition M&S Working Group
(AMSWG) is a coalition of the willing.
Help evolve the practice of VV&A by:
Reviewing and commenting on
VV&A studies
Providing input to the review and
update of VV&A policy and
standards
Supported by:
User feedback
―Gray Hair‖ Support to Key
Activities
ITEA LVC Conference
24 - 27 Jan 2011 Page-22
• Expand leverage of DoD, government
agency and Industry standards for M&S
VV&A
• Mature existing products with documented
and archived lessons learned
• Emerging concepts (e.g., Risk-based VV&A)
will result in a standard/guideline
downstream
DoD M&S VV&A Campaign Plan Way Forward
ITEA LVC Conference
24 - 27 Jan 2011 Page-23
Key M&S VV&A Contact Information
DoD Modeling & Simulation Coordination Office John Diem, Associate Director (Services)
(703) 681-6631, [email protected]
Acquisition Modeling & Simulation Working Group
Chair Phil Zimmerman, DDRE/SE/SA
(703) 681-6544; [email protected]
Support Team Ralph Gibson, BAH Support to M&S CO (Services – VV&A)
(703) 681-6643; [email protected]
Crash Konwin, BAH Support to DDRE/SE/SA
(703) 681-6576; [email protected]
ITEA LVC Conference
24 - 27 Jan 2011 Page-25
Acquisition Community Led VV&A Efforts - key inputs to Campaign Plan
V-C-2
Improving VV&A Implementation
Johns Hopkins APL
Improving VV&A
Implementation
Systems Engineering
Research Center
VV&A Shortfalls & Needs Complex Systems Modeling
Pilot Implementations
(SIMAF & NAVAIR)
VV&A/Distributed Joint
VV&A/Vehicle Vulnerability
Practitioners Forum
AMSWG VV&A
Sub-Committee
ITEA LVC Sub-committee
“Gray Hairs” Study Support
V-AQ-2
Risk Based Accreditation
FY: 09-10 FY: 10-11
ITEA LVC Conference
24 - 27 Jan 2011 Page-26
Risk-Based VV&A (FY09/10 M&S PE Funded V-AQ-2)
Since the cost of verifying and
validating simulations is often high,
V&V investments should be
weighed against the risk of making
a bad decision because of
unreliable M&S results.
The purpose of this work effort is to
improve Department of Defense
(DoD) Modeling and Simulation
(M&S) Verification, Validation,
Accreditation (VV&A) by
establishing an effective risk-based
methodology for VV&A.
ITEA LVC Conference
24 - 27 Jan 2011 Page-27
JHU/APL VV&A Implementation: Defining & Addressing Gaps
(M&S PE Funded V-C-2)
The purpose of the effort is to:
Support the identification of VV&A gap
areas that inhibit effective
implementation of VV&A processes;
Develop supplemental capability
required to implement the Risk-based
Accreditation (RBA) methodology; and
Evolve technology, in the form of
automated tools and metadata
specifications, that increase VV&A
implementation efficiencies.
ITEA LVC Conference
24 - 27 Jan 2011 Page-28
Standard Intended Use & Tailoring
• Intended to support VV&A of DoD models, simulations,
federations, and other types of distributed simulations by
providing a common framework for documenting
information produced during the VV&A processes.
• Specifies procedures on documenting information obtained
through implementing the VV&A processes when the
outputs will be used to supplement decisionmaking in the
DoD.
• These types of information, at least some of the time, are
obtained via contract from commercial firms.
• Tailoring guidance for contractual applications. Invitation
for bids, request for proposals, and contractual statements
should tailor Appendices A through D in accordance with
the scope of the contractual effort.
ITEA LVC Conference
24 - 27 Jan 2011 Page-29
29
RPG Assessment Overview
• Purpose of the Recommended Practices Guide (RPG)
Assessment
– Assess the viability of the web-based RPG
– Identify immediate problems that inhibit the RPG’s effective use
– Identify potential improvements that could enhance the RPG’s use
– Identify advanced technology features to increase the RPG’s
reach
• Assess Primary RPG characteristics
– Content: The information contained in the website
– Presentation and Structure: Organization and navigation
– Technology: Web-based features
• RPG Update Status
– Phase I updates completed (e.g., fixing hyperlinks, updating
special topics documents)
– Elements of Phase II updates in process (e.g., developing new
special topics, updating examples)