1 simone youngblood (703) 824-3436 [email protected] vpmm evolved: an update s.y. harmon (541)...
TRANSCRIPT
1
Simone Youngblood
(703) 824-3436
VPMM Evolved: An Update
S.Y. Harmon
(541) 863-4639
September 2007
2
Credibility, The Beginning of the VPMM
• Credibility is nearly synonymous with trustworthiness.
• Credibility is a property of the information being presented.
• However, credibility is a property that involves the belief of the observer.
• Therefore, the perception of credibility is inherently subjective.
• Credibility is only loosely coupled to the process for deriving the information.
• Therefore, the integrity of the process can only contribute to credibility if the observer understands that process and appreciates its limitations (e.g. mathematical logic).
• Further, the observer must also trust that the people who applied that process did so correctly.
1. The quality or power of inspiring belief2. Capacity for belief
Merriam-Webster Online Dictionary
1. The quality of being trusted and believed in2. The quality of being convincing or believable
Apple Dictionary, Version 1.01
The believability of a statement, action, or source, and the propensity of the observer to believe that statement
Wikipedia
1. The quality, capability, or power to elicit belief2. A capacity for belief
The Free Dictionary
Observations on Credibility
• The motivation for the VPMM coalesced in 1999 when the ITOP Committee on V&V Procedures tried to develop levels of credibility & asked for help.
• In response, Ms. Youngblood asked her team to explore simulation credibility & its relationship to V&V.
• We started with the definitions of credibility:
3
Proposed Levels of Validation Process Maturity(as of May 2006)
Level Validation Criteria
Referent Conceptual Model
Development Products
Simulation Results
0 none exist none chosen none constructed verified only enough to support development
not validated at all
1 represented by SME opinion
represented by SME opinion
if one exists, validated by SME review
verified only enough to support development
validated by SMEs observing simulation results
2 developed from user input in terms of entities represented, their attributes & the dependencies between them
represented solely by SME opinion
validated by SME against the validation criteria
verified against the conceptual model inventory
validated by SME against the validation criteria
3 developed from user input in terms required for Level 2 plus the attribute ranges, domains & maximum errors
derived from a single source with estimates of errors
validated against the validation criteria & referent
verified against the conceptual model
validated against the validation criteria & referent
4
Proposed Levels of Validation Process Maturity(as of May 2006)
Level Validation Criteria
Referent Conceptual Model
Development Products
Simulation Results
4 developed from user input in terms required for Level 3 plus maximum tolerable risk
sampled from multiple independent sources & correlated statistically with estimates of confidence levels
validated against the validation criteria & referent; analyzed to suggest results sampling space & estimate the confidence associated with that sampling
verified against the conceptual model; provides information to guide results sampling & estimate the confidence associated with that sampling
sampled from guidance developed from CM & verification results analysis; validated against the validation criteria & referent
5 mathematically derived from user statements using causality arguments & stated in terms required for Level 4
mathematically derived from multiple independent sources & characterized statistically with estimates of confidence levels
formally stated & validated through proof against the validation criteria & referent; analyzed to define results validation sample space & confidence associated with that sampling
verified against the conceptual model & used to define results validation sample space & the confidence associated with that sampling
sampled from guidance defined from CM & verification results analysis; validated through proof against the validation criteria & referent
5
VPMM Application to Real Simulation Validation Efforts
Simulation Validation Criteria
Referent Conceptual Model
Verification Products
Simulation Results
Integrated Maturity
Level
Trial 1 1 2 2 1 1 1
JMRM 2 2 2 4 2 2
JTLS 2 2 3 4 2 2
JCATS 2 3 1 1 2 1
Fed 1 2 2 2 1 1 1
Fed 2 2 4 3 4 2 2
JLVC 1 1 - 1 1 1
STORM 1 1 - 1 1 1
JEM 3 3 - 1 3 3
JOEF 2 2 2 2 2 2
SERPENT 3 3 - 2 2 2
• Constructed an evaluation questionnaire that elicits information characterizing validation processes (36 questions)
• Applied this questionnaire, through interviews, to 11 simulations
• Analyzed the questionnaire responses to obtain the results in the table below
The numbers in the leftmost cells are the maturity levels of each of the components of the validation process.
The numbers in the leftmost cells are the maturity levels of each of the components of the validation process.
The VPMM was also applied in formulating the content of the proposed IEEE Standard for the VV&A Overlay to the FEDEP.
The VPMM was also applied in formulating the content of the proposed IEEE Standard for the VV&A Overlay to the FEDEP.
6
VPMM Development & Application History
Date EventOctober 2002 Presented initial VPMM concepts
March 2003 Detailed the VPMM concept & presented it at Spring 2003 SIW
March 2004 Revised the VPMM to account for levels of simulation validation & presented it at Spring 2004 SIW
October 2004 Revised the VPMM to account for uncertainty, applied it to evolving a VV&A technology roadmap & presented these at Foundations ‘04 Workshop
May 2005 Initiated a proof of concept study with JWFC (LTC Emily Andrew) to use the VPMM to improve their V&V processes for training simulations; developed the maturity level questionnaire
November 2005 Applied the VPMM to JEM V&V processes
January 2006 Prepared & submitted a draft DoD VPMM standard for review
March 2006 Revised the maturity level questionnaire & began applying it to additional ongoing simulation V&V efforts
April 2006 Proposed a SISO study group for an IEEE VPMM standard
May 2006 Began developing a maturity model for VV&A processes (not just validation) that incorporates simulation use risk management as a keystone element & accounts for additional information sources (e.g., use & V&V histories)
June 2006 Completed the JWFC proof of concept & its final report; revised the maturity level questionnaire
November 2006 Applied the VPMM to V&V of the JOEF simulation component
December 2006 Applied the VPMM to the SERPENT V&V processes
7
Lessons Learned from the VPMM’s Applications
• Despite performing these evaluations in an interview setting, several important terms seemed to be misunderstood even in the VV&A community (e.g., validation criteria, referent, conceptual model & subject matter expert).
• This version of the VPMM does not handle evaluation of efforts that do not use the information from a simulation conceptual model.
• Most VV&A practitioners feel that verification improves the validation process but they do not verify against a conceptual model; they do not use verification products to guide results validation; and the verification products do not contribute to confidence estimation. What’s left?
• Some simulation programs build sophisticated referents but appear not to use them to estimate error. Why?
• Face validation can be improved with information from conceptual model validation and verification but this VPMM version does not resolve that improvement.
• Use & V&V histories can contribute valuable information to the validation process but this VPMM version does not apply that information.
• This version of the VPMM does not directly associate risk reduction with higher levels of maturity & account for improved ability to estimate validation effort costs.
8
Approach to Building the Next Generation VPMM
• Construct a model of simulation use using decision theory, simulation-based decision making
• Characterize the uncertainties, and therefore the risks, that a user of simulation-based decision making encounters
• Develop a model of the VV&A processes needed to minimize, or at least identify, the risks that a user of simulation-based decision making encounters
• Develop individual models of validation, verification and accreditation processes from the global VV&A model that take into account all of the information available to VV&A practitioners
• Identify the functions associated with each of the components of risk-based VV&A
• Organize a maturity matrix for risk-based VV&A using the same principles as the original VPMM that structures the risk-based VV&A functionality
9
Simplified Models of Simulation-Based Decision Making
User Situation
observed situation
state
effects predictions
user actions
actual situation
state
Simulation
situation model
User Situation
observed situation state
effects predictions
user actions
actual situation
state
Simulation
situation model
VV&A Processes
user needs accreditation recommendations
A decision maker uses a simulation to change decision making under uncertainty to decision making under either risk or certainty but depends upon the simulation to provide complete and correct information to make that change.
VV&A processes can provide that decision maker with information about the conditions under which the simulation’s results are complete and correct enough for their intended use and thus reduce the risk that they take by incorporating the simulation into their decision making.
10
More About Use Risk & Decision Errors
• Users can make one of two types of error when using simulation results in their decision making:
– Type I error > The user chooses to ignore simulation results even though they are sufficiently correct and
complete enough for their use.
> The user incorrectly believes the simulation results to be incomplete or incorrect, may rely upon less reliable sources for the information to support their decisions, and make poor decisions as a result.
> Example: Crater simulation used for Columbia damage assessment decision
– Type II error> The user chooses to use simulation results that are insufficiently complete or correct
enough for their use.
> The user incorrectly believes the simulation results to be complete and correct enough to support their decisions and may make poor decisions as a result.
> Example: Patriot control rule testing
• Experiencing Type II errors from using a simulation may lead the user to make Type I errors.
In risk-based VV&A, the accreditation recommendations define the conditions under which the user can depend upon a simulation’s results while incurring risks below tolerable levels.In risk-based VV&A, the accreditation recommendations define the conditions under which the user can depend upon a simulation’s results while incurring risks below tolerable levels.
11
Top Level Model of Risk-Based VV&A
Simulation Validation
(V&V agent)
Simulation Verification(developer or V&V
agent)
Use RiskModeling
(accreditation agent)
Accreditation Analysis
(accreditation agent)
use characterization & prioritized validation criteria
verified conceptual model & integrated
verification evidence
V&V evidence
accreditation recommendations
user needs
conceptual modeluse & V&V histories
simulation results
accreditation history
validated conceptual
modelsimulation referent
In order to support risk-based VV&A, all of the components of the VV&A processes must focus upon contributing to defining the accreditation recommendations (i.e., the conditions of recommended use). These recommendations will enable the user to avoid making Type II errors and the credibility gained through that success will help them avoid making Type I errors as well.
12
Model of Use Risk Modeling
Need & Risk Elicitation
Requirements Modeling
Use Risk Modeling
User
need & risk elicitation discourse
needs information
risk information
use information
needs
risk information
needs
requirements model
use characterization
prioritized acceptability
criteria
Use risk modeling replaces the front end components of the accreditation assessment. The process now focuses upon eliciting user needs and what they know about their risks then elaborates upon that input using requirements and use risk modeling techniques. This risk modeling permits prioritizing acceptability criteria based upon risks.
13
Model of Risk-Based Verification
Conceptual Model
Verification
Design Verification
Implementation Verification
Verification Analysis
system design products
system implementation products
verified conceptual model
system conceptual model
validated conceptual model
verified design products
verified implementation products
integrated verification evidenceCM verification evidencedesign verification evidenceimplementation verification evidence
Verification ensures the consistency of the conceptual model and contributes evidence to the validation process. This evidence reduces the coverage needed in results testing and increases the confidence that the simulation’s results are correct and complete enough for the intended use.
14
Model of Risk-Based Validation
All available information, not just the conceptual model and verification products, contribute to focusing the results validation (through design of experiments) and, thus, building the picture of the simulation’s representational capabilities in the context of the intended use.
ResultsValidation
Conceptual Model Validation
History Evaluation
Results Validation Planning
use & V&V histories conceptual model
simulation results
integrated verification
evidence
validated conceptual model
results validation plan
V&V evidence
simulation referent
relevant historical validity information
prioritized acceptability criteria use characterization
verified conceptual
model
15
Model of Risk-Based Accreditation Analysis
accreditation history
V&V evidence
accreditation recommendations
use risk assessment
V&V Evidence Integration
Use RiskAnalysis
Accreditation Recommendation
Derivation
use characterizationprioritized
acceptability criteria
integrated suitability picture
The back end of the accreditation process builds an integrated picture of the simulation’s suitability for the intended use through evidence theory and estimates the confidence in that picture. This information contributes to both the use risk assessment and accreditation recommendations.
16
Maturity Characteristics of Validation-Related Products
Maturity Level
Use Characterization Validation Referent Validation Evidence
0 None None None
1 SME judgments SME opinion of referent behavior
SME judgment of completeness & correctness
2 representational inventory + primary use risks for inventory groups
SME opinion of referent behavior
inventory completeness + SME judgment of correctness
3 Level 2 + tolerable error characteristics & use risk estimates for errors & bounds
quantitative estimate of simuland behavior
completeness & correctness evaluation results
4 Level 3 + desired confidences, tolerable confidences & sensitivities
Level 3 + confidence in behavior estimates
Level 3 + confidence evaluation results
5 Level 4 + mathematical derivation description
Level 4 + mathematical derivation description
Level 4 + mathematical proof
increasing use risk
17
Salient Properties of Validation Process Maturity Levels
Judge of Validity
Validation Criteria Uncertainty Estimates
mathematical proof
explicit specification of needed entities, behaviors, ranges of errors, errors & tolerable uncertainty levels
yes
independent observer
explicit specification of needed entities, behaviors, ranges of errors, errors & tolerable uncertainty levels
yes
independent observer
explicit specification of needed entities, behaviors, ranges of errors & errors
no
SME explicit specification of needed entities & behaviors and SME knowledge of user tolerances of errors
no
SME SME knowledge of user needs no
none none no
Subjective(1)
Provable(5)
Confident(4)
Accurate(3)
Complete(2)
Initial(0)
18
Summary & Conclusions
• The VPMM has gradually matured since 2002 through generous feedback from the technical community. The VPMM has been vetted by the
– US DoD M&S community through the DoD VV&A TWG & DoD standards process,– VV&A community through Foundations ‘04), and – M&S community in SISO through SIWs & in SCS through JDAMS article.
• Proof of concept studies are complete and the results from these studies will further mature the VPMM concepts.
• Activities aimed toward creating a SISO standard have been started. These activities may integrate with a larger study group exploring the fundamentals of V&V.
• Activities aimed toward creating an interim DoD standard have been started and draft documentation will be completed in July 2007.
• The VPMM concepts are being expanded to encompass risk-based VV&A.