applying modeling and simulation verification, validation and accreditation (vv&a) techniques to...
TRANSCRIPT
Applying Modeling and Simulation Verification, Validation and Accreditation
(VV&A) Techniques to Test and Laboratory Facilities
Dr. James Elele, Jeremy Smith NAWCAD BSMVV Branch
David Hall, Charles PedrianiSURVICE Engineering Company
ASME V&V Conference3 May 2012
2
Introduction
• Tasking to support accreditation of Test and Evaluation Facilities in support of IFF Program– Develop an accreditation case for T&E facilities for operational
testing
• Applied risk-based M&S VV&A approach to facilities– Approach applied successfully to M&S for over 20 years
• Test case for future T&E facility accreditation efforts– Successful application can support efforts to institutionalize
process for T&E as well as M&S
IFF = Identification Friend or FoeT&E = Test and Evaluation
M&S = Model and Simulation
3
• Verification: The process of determining that a model implementation and its associated data accurately represent the developer's conceptual description and specifications.
– Does the model do what the originator intended, and is it relatively error free?
• Validation: The process of determining the degree to which a model and its associated data are an accurate representation of the real world from the perspective of the intended uses of the model.
– How well do model results match real world data, in the context of your needs?
• Accreditation: The official certification [determination] that a model, simulation, or federation of models and simulations and its associated data are acceptable for use for a specific purpose
– Does the accreditation authority have adequate evidence to be confident that a model is fit for purpose?
M&S VV&A Definitions
Definitions from DODI 5000.61 dated 13 May 2003
Did you build the model right?
Did you build the right model?
Did your customer accept it?
4
Underlying Principles
• The ultimate goal of VV&A efforts is to form a foundation for making good decisions
• Nature and extent of information required to support accreditation decision is at the discretion of the accreditation authority and is generally based on assessment of risk
1. Role of M&S results in decision making process
2. Importance of decision that M&S is supporting
3. Severity of the consequences of making incorrect decisions because M&S were wrong
4. Probability that analysis results based upon M&S will be challenged
Bestof
ShowGood
Enough“Better is the Enemy of Good Enough!”
5
Steps to an Accreditation Decision
Make Accreditation Decision
Review Accreditation Case
Develop Accreditation Case
Develop/Execute Accreditation Plan
Develop M&S Reqts and Accreditation Info Reqts
Analyze Intended Use Intended Use
Statement
V&V
A
Accreditation Plan
Fit for Intended
Use
6
How Much Credibility Is “Enough”?
M&S UserIt Depends on Risk
BUT
A Makeshift Bridge is Good Enough If You Need To Cross a
Meandering Shallow Stream
M&S
7
Greater Risks...
PROBLEM CREDIBLESOLUTION
Indicate the Need for Evidence of Greater Credibility
M&S
Supporting Evidence
8
Output Accuracy
Simulation meets design requirements, operates as designed
and is free of errors in software
Software (S/W) Accuracy
Simulation input data, validation data and data manipulations are
appropriate and accurate
Data Accuracy
Simulation outputs match the real world “well enough” to be of use
in a particular problem
ValidationVerification
V&V: The Central Pillars of Simulation Credibility
S/W Accuracy Output AccuracyData Accuracy
But, V&V are Just the Middle of the Bridge!
9
Simulation possesses all required functionality and fidelity for the
problem being solved
Anchors the M&S to the Problem
Simulation has adequate user support to facilitate correct operation and
interpretation of its outputs
Ties the M&S to a Useful Solution
AccuracyAccuracyM&S
Requirements
CapabilityCapability
UserCapabilities
UsabilityUsability
The “Other Pillars” of Simulation Credibility
Problem CredibleSolution
Capability Usability
Can I Be Sure I’m Not Mis-
Using the M&S?
M&S
Does the M&S Do What I Need
It To Do?
Accuracy of:•Software•Data•Outputs
10
The Essence of Accreditation
TO PROVE THE M&S IS SUITABLE FOR THE NEED REQUIRES AN OBJECTIVE COMPARISON
OF M&S INFORMATION WITH M&S REQUIREMENTS WITHIN THE CONTEXT OF THE PROBLEM
M&SREQUIREMENTS
M&SINFORMATION
IDENTIFYWORK-AROUNDS,
USAGE CONSTRAINTS,REQUIRED IMPROVEMENTS
AND RISKS
IDENTIFY M&S DEFICIENCIES
ACCREDITATIONDECISION
• Capability• Accuracy• Usability
• Data Quality• M&S Documentation• Design Documentation• Configuration Mgt• V&V Results• Etc.
Defined by the User(Formally or Implied)
Provided by the Model DeveloperProvided by the Model Developer or Model Proponentor Model Proponent
PROBLEM CONTEXT
11
How Much V&V is Enough?
It Depends on Risk– Risk means something “bad”
might happen because you believed an incorrect simulation result
• Decisions based on M&S results are at risk
• VV&A reduces that risk
RISK =
PROBABILITY x IMPACT
HIGH
LOWPROBABILITY
RISK
IMPACT
MODERATE
12
Quantifying Risk Level
RISK LEVEL VALUES ARE:• Subjective• Consistent with MIL-STD-882• Tailorable to each application
RISK LEVEL VALUES ARE:• Subjective• Consistent with MIL-STD-882• Tailorable to each application
HIGHER RISK MEANS MORE CREDIBILITY EVIDENCE IS NEEDED TO ACHIEVE ACCREDITATION
Probability of Incorrect
Simulation Result
Level of Impact on the Program
Negligible Marginal Serious Critical Catastrophic
Frequent Low Moderate High High High
Probable Low Moderate Moderate High High
Occasional Low Low Moderate Moderate High
Remote Low Low Low Moderate Moderate
Impossible Low Low Low Low Moderate
13
Low Moderate High
1
2
3
4
5
1 2 3 4 5
Impact
Lik
elih
oo
d
Reduced reliance on M&S results
Imp
rove
d M
&S
Cre
dib
ility
Risk = Likelihood x Impact
Risk Reduction Strategies
14
Application to Test and Laboratory Facilities
• This was a trial application of the M&S VV&A approach to test facilities– Identification Friend-or-Foe (IFF) system testing to evaluate
new IFF system performance
• Accreditation of test facilities required by Commander, Operational Test and Evaluation Force (COMOPTEVFOR)
• Facilities used for system assessment include:– All-up ship radar and related system representations– Simple stimulators– Engineering Test Equipment (ETE) facility specific to IFF
system testing
15
ETE Facility
• Generates waveform signals to stimulate a production IFF transponder in the laboratory
• To evaluate system requirement for resistance to signals from transmitters other than the desired transmitter– Critical technical parameter “susceptibility of the IFF
system to false interrogations”– Metric: Probability of resistance to false signals = Pr
16
Initial ETE Risk Assessment
M&S Characteristic Criterion Risk
CAPABILITY
Intended Use The specific intended use(s) of the facility, model or simulation is/are clearly stated.
LOW
Design The facility and analysis process (framework, algorithms, data sources, and assumptions) produces credible results.
MODERATE
ACCURACY
Input Data For each facility, model or simulation, input data are credible and subject to review and revision.
LOW
System Verification
The facility, model or simulation has been formally tested or reviewed and has been demonstrated to accurately represent the specific intended use(s) and requirements.
MODERATE
Results Validation The facility’s, model’s or simulation’s responses have been compared with known or expected behavior from the subject it represents and has been demonstrated to be sufficiently accurate for the specific intended use(s).
MODERATE
USABILITY
Configuration Management
For each facility, model or simulation, modeled components are supported by a sound written Configuration Management (CM) Plan.
MODERATE
User Community For each facility, model or simulation, the capability is designed and developed for the level of competency for its intended purpose. The capability is supported by documents such as user’s manual, technical manual, and/or reference guide.
MODERATE
17
Initial ETE Assessment and Recommendations*
CAPABILITY:• The intended use is clearly stated: to evaluate the probability of
responding to a false signal (Pr)• No formal design documents exist: recommend laboratory design be
adequately documented ACCURACY:
• Input data are provided by actual hardware • Recommend documenting a complete set of test cases and results and
any previous verification activities• Recommend independent subject matter expert (SME) review of
laboratory resultsUSABILITY:
• Recommend facility develop and implement an overall configuration management plan
• The test approach appears to have been successfully used over a span of many years to support a variety of identification programs for DOD and the FAA: recommend the facility provide documented results of previous uses
* Incorporated into ETE Facility Accreditation Plan
18
Observations
• Application to test facility similar to M&S– Biggest issue for both: getting good documentation– Lack of configuration management plans– Poor documentation of prior V&V results
• They did V&V, they just didn’t write it down
– Accreditation Support Package (ASP) document format works equally well for both
• Differences:– Developing intended use statement more natural for
test facilities than for M&S• Use of T&E facilities seems to have more focused objectives
initially than use of M&S
19
Accreditation Support Package
• EXECUTIVE SUMMARY
• ASP OVERVIEW
• CAPABILITY– M&S Description– Functional Capabilities– Development History– Assumptions and Limitations– Implications for M&S Use
• USABILITY– Documentation Assessment– User Support– Usage History– Implications for M&S Use
• ACCURACY– Software Accuracy
• Software Verification Results– Design Verification– Implementation Verification
• Software Development and Management Environment
– Software Development Environment
– Configuration Management
• Software Quality Assessment• Implications for M&S Use
– Data Accuracy• Input Data• Data Transformations• Implications for M&S Use
– Output Accuracy• Sensitivity Analysis• Benchmarking• Face Validation• Results Validation• Implications for M&S Use
20
Thoughts on Broader Application
• Risk-based approach seems as applicable to test facilities as to M&S
• Risk assessment can also help prioritize which facilities justify spending more VV&A resources
• Suggest standardizing and institutionalizing risk-based VV&A process for both M&S and T&E– No consistent application across DOD for either– Risk-based VV&A promotes cost-effective VV&A for
both M&S and T&E facilities