vartan – validation reporting templates jürgen teutsch, nlr caats workshop, 16-feb-2006,...
TRANSCRIPT
![Page 1: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/1.jpg)
VARTAN – Validation Reporting Templates
Jürgen Teutsch, NLRCAATS Workshop, 16-Feb-2006, Lanzarote
![Page 2: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/2.jpg)
VARTAN – Project Objectives
VARTAN is concerned with...
Definition of Validation Reporting Templates in accordance with European Operational Concept Validation Methodology (E-OCVM)
Analysis of E-OCVM
Refinement of Life Cycle Model and Case-based View
Context of templates in Life Cycle Model
Study for automated transfer of template contents into the Validation Data Repository (VDR)
![Page 3: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/3.jpg)
VARTAN – Analysis Issues
Performance expressed in structured way?– Key Performance Areas (PRC/ICAO)– Key Performance Indicators
Evidence with right level of quality?– Validity and accuracy of results (and objectives)
Right levels of detail at different decision-making levels?– Strategy, programme, project, and experiment level
Information needs of all decision makers addressed?– Airlines, service providers etc.
E-OCVM comprehensible for less proficient user?– Simplify methodology for different E-OCVM views
![Page 4: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/4.jpg)
VARTAN – Results Overview
Suggested (or similar) refinements are envisaged for implementation in the updated methodology:
Improve comprehensibility of Life Cycle Phase Model Refine case-building process description Describe and give examples of possible stakeholder needs and
associated stakeholder analysis issues Consider confidence in results
Activities for preparation of document standards with automation:
Build templates at different planning levels and determine triggers for reporting
Provide automation example for experiment-level validation reporting template
Describe process for working with result templates
![Page 5: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/5.jpg)
VARTAN – Suggested Life Cycle Refinements
Plan
Do
Check
Act
Define Goals and Cases
Decide onWay to Proceed
Carry OutExperiments
Study Results of Experiment
Deming Cycle for ATM Concept Validation
I mplementedProcedure or ProcessI nitial I dea
Scope Operability Performance I ndustrialisation I mplementation
V1 V2 V3 V4 V5
Concept Principles
Techniques•High-level Modelling•Fast-time Simulations•Expert J udgement
KPA/ Cases•Acceptability•Access and Equity•Capacity and Delay•Cost Efficiency•Efficiency•Environment•Flexibility•Interoperability•Safety•Security
Expected Results•Stakeholder I ssues•Time Planning•Scenario Definitions•Benefit Mechanisms•Potential Benefits•Showstoppers•Risks
Feasibility
Techniques•Requirement Capturing•Rapid Prototyping•Operational Simulations•Procedure Analysis•Scenario Evaluation
KPA/ Cases•Acceptability•Efficiency•Human Factors•Interoperability•Predictability•Safety
Expected Results•System Operability•User Operability•Procedure Description•Non-nominal Cases•HMI Prototype•Workload Estimates•Concept Baseline•Risks and Mitigation
Concept I ntegration
Techniques•Real-time Simulations•Live Trials•Shadow-mode Trials•Cost-Benefit Analyses•Safety Analysis
KPA/ Cases•Acceptability•Capacity and Delay•Cost Efficiency•Efficiency•Environment•Safety
Expected Results•Performance Data•Stakeholder Approval•Acquisition Process•Detailed Procedures•Detailed Concept
I ndustrialisation
•Procedure Approval•Acquisition•Factory Test
I mplementation
•In-Service Test•In-Service
![Page 6: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/6.jpg)
VARTAN – Suggested Case-building Refinements
From KPA and KPI to case definition: which cases are necessary for decision making?
Interrelationships between defined areas (cases)
Differences in focus from V1 to V3; from identification of showstoppers (V1) to implementation decision (V3)
Discern Verification and Validation (especially in V2)
Different levels of detail in cases: what level of detail is necessary for decision making per lifecycle phase?
![Page 7: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/7.jpg)
VARTAN – Proposals for Stakeholder Needs
Conflicting stakeholder objectives/goals or interests– Goal balancing– Harmonisation of concepts and ideas
Different stakeholder information needs– Different levels of reporting
Different levels of quality (validity/accuracy) of results – Use of standard methods to increase (or find a
common level of) validity
![Page 8: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/8.jpg)
VARTAN – General Template Considerations
• Pre-defined introductory part (background, context, scope)
• Summary of planning document
• Differences between planning and actual achievements
• Report confidence in results
• Objective should be related to a certain KPA
• Correlation between results / conclusions / recommendations and objectives (from high-level to low-level to hypotheses)
![Page 9: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/9.jpg)
VARTAN – Planning and Reporting LevelsPlanning Levels
Life Cycle Phase
Strategic Level (Decision Makers)Set overall and intermediate goalsDetermine timeframeDetermine Key Performance Areas (KPA)Solve stakeholder issuesPrioritise ATM needs
Programme Level (Programme Managers)Address particular intermediate goal(s)Focus on particular KPAIdentify interrelations between KPAsDetermine KPIs inherent to KPAsDefine cases necessary to support goals
Project Level (Project Managers)Define validation strategyImplement case(s)Identify hypotheses based on goals/objectivesProduce scenarios and experiment planProduce analysis plan
Experiment Level (Experiment Leader)Prepare platform or facilityPerform pre-exercise testing and trainingConduct validation experiment(s)Perform analysisStart reporting cycle on lowest level
Experiment Level Results Report
Summary of planningExperiment executionExperiment resultsExperiment analysisConclusionsRecommendations
Programme Level Results Report
Goals and ObjectivesKPA, KPI and casesProgramme resultsProgramme analysisConclusionsRecommendations
Project Level Results Report
Project ObjectivesValidation strategyImplemented case(s)Project resultsProject analysisConclusionsRecommendations
Continue ?
Continue ?
Continue ?
YES
YES
NO
NO
NO YES
Strategic Level Planning Report
KPAATM ProblemsATM NeedsPrioritisationOp. ImprovementsExpected BenefitsProgramme Outline
Programme Level Planning Report
ATM NeedsOperational ConceptOI ClustersProgr. ObjectivesProgr. StakeholdersProgr. DependenciesFactors of InfluenceTools and TechniquesProgramme OverviewResourcesDeliverablesRisks
Project Level Planning Report
Project ContextProj. ObjectivesProj. StakeholdersProj. DependenciesIdentified CasesFactors of InfluenceTools and PlatformsProject Overview ResourcesDeliverablesRisks
Experiment Level Planning Report
N/A
Considered to betoo prescriptive
Experiment Level
Strategy Level
Programme Level
Project Level
![Page 10: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/10.jpg)
VARTAN – Planning and Reporting Levels
Planning Levels
Life Cycle Phase
Strategic Level (Decision Makers)Set overall and intermediate goalsDetermine timeframeDetermine Key Performance Areas (KPA)Solve stakeholder issuesPrioritise ATM needs
Programme Level (Programme Managers)Address particular intermediate goal(s)Focus on particular KPAIdentify interrelations between KPAsDetermine KPIs inherent to KPAsDefine cases necessary to support goals
Programme Level Results Report
Goals and ObjectivesKPA, KPI and casesProgramme resultsProgramme analysisConclusionsRecommendations
Continue ?
Continue ?
YES
NO
NO YES
Strategic Level Planning Report
KPAATM ProblemsATM NeedsPrioritisationOp. ImprovementsExpected BenefitsProgramme Outline
Programme Level Planning Report
ATM NeedsOperational ConceptOI ClustersProgr. ObjectivesProgr. StakeholdersProgr. DependenciesFactors of InfluenceTools and TechniquesProgramme OverviewResourcesDeliverablesRisks
![Page 11: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/11.jpg)
VARTAN – Planning and Reporting Levels
Project Level (Project Managers)Define validation strategyImplement case(s)Identify hypotheses based on goals/objectivesProduce scenarios and experiment planProduce analysis plan
Experiment Level (Experiment Leader)Prepare platform or facilityPerform pre-exercise testing and trainingConduct validation experiment(s)Perform analysisStart reporting cycle on lowest level
Experiment Level Results Report
Summary of planningExperiment executionExperiment resultsExperiment analysisConclusionsRecommendations
Project Level Results Report
Project ObjectivesValidation strategyImplemented case(s)Project resultsProject analysisConclusionsRecommendations
Continue ?
YES
NO
Project Level Planning Report
Project ContextProj. ObjectivesProj. StakeholdersProj. DependenciesIdentified CasesFactors of InfluenceTools and PlatformsProject Overview ResourcesDeliverablesRisks
Experiment Level Planning Report
N/A
Considered to betoo prescriptive
relevantVDR information
![Page 12: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/12.jpg)
VARTAN – Unexpected Results
Validation Reporting Triggers (Exceptional triggers)
Exceptional situations with unexpected results or events might lead to a re-evaluation of the current activity
Need to report to higher levels immediately and therefore trigger a special report
Important: there must be a way to stop the validation cycle and reconsider the complete approach
Generally, such reporting is part of a good management approach (not part of E-OCVM guideline structure)
![Page 13: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/13.jpg)
VARTAN – Validation Reporting Process (1/3)
Computer with
MS-Word Client
I nternet
Validation ReportingTemplates
VDR Website
![Page 14: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/14.jpg)
VARTAN – Template Life Cycle
Empty Template
Accepted Template
Prepared Template
Empty Template
Accepted Template
Prepared Template
Filled-in Template
Received Template
Approved Template
Filled-in Template
Received Template
Approved Template
Author accepts data
VDR Data Collectoradds data
Author does notaccept data
VDR Administratorapproves template
VDR Administratorrejects template Author
transmits template
Authorfills in template
Authorupdates template
![Page 15: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/15.jpg)
VARTAN – Validation Reporting Process (2/3)
VDR(Oracle Database)
Computer with
MS-Word Client
I nternet
Filled-in Templates
![Page 16: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/16.jpg)
VARTAN – Template Life Cycle
Empty Template
Accepted Template
Prepared Template
Empty Template
Accepted Template
Prepared Template
Filled-in Template
Received Template
Approved Template
Filled-in Template
Received Template
Approved Template
Author accepts data
VDR Data Collectoradds data
Author does notaccept data
VDR Administratorapproves template
VDR Administratorrejects template Author
transmits template
Authorfills in template
Authorupdates template
![Page 17: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/17.jpg)
VARTAN – Validation Reporting Process (3/3)
Validation Expert
VDR Administrator
TCP/IP
HTTP
Web Services – SOAP (XML)
Apache Axis
Java Servlet
VDR (Oracle)
VDR Interface
SQLStatements
![Page 18: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/18.jpg)
VARTAN – Template Life Cycle
Empty Template
Accepted Template
Prepared Template
Empty Template
Accepted Template
Prepared Template
Filled-in Template
Received Template
Approved Template
Filled-in Template
Received Template
Approved Template
Author accepts data
VDR Data Collectoradds data
Author does notaccept data
VDR Administratorapproves template
VDR Administratorrejects template Author
transmits template
Authorfills in template
Authorupdates template
![Page 19: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/19.jpg)
VARTAN – Template Contents:Help Text
![Page 20: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/20.jpg)
VARTAN – Template Contents:Introduction
1. Introduction
1.1 Project/Experiment Background and Context
1.2 Concept Overview
1.3 Purpose of Document
1.4 Scope and Structure of Document
1.5 References
1.6 Acronyms
![Page 21: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/21.jpg)
VARTAN – Template Contents:Project Level Strategy and Planning
2. Summary of Project Strategy and Planning
2.1Stakeholder Requirements
2.2Supporting Cases
2.3Expected Project Outcomes and Objectives
2.3.1 Description of Expected Project Outcomes2.3.2 Summary of Expected Project Outcomes2.3.3 Description of Project Objectives2.3.4 Summary of Project Objectives
2.4Decision Processes in Experiment Definition
2.5Experiment Context and Planning
2.6Analysis Strategy
![Page 22: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/22.jpg)
VARTAN – Template Contents:Project Level Conduct and Results
3. Conduct of Validation Project3.1Executed Experiments3.2Deviations from the Planning
4. Results of Experiments4.1Results of Different Experiments per Key Performance Area4.2Confidence in Results of Different Experiments
4.2.1 Quality of Results of Different Experiments
4.2.2 Significance of Results of Different Experiments
4.3Unexpected Project Results
5. Consolidation of Project Results5.1Consequences of Confidence in Experiment Results5.2Consequences of Experiment Results for Project Objectives
5.2.1 Interpretation of Results with respect to High-level Objectives
5.2.2 Interpretation of Results with respect to Key Performance Areas
6. Conclusions and Recommendations for Programme Level6.1Conclusions6.2 Recommendations
![Page 23: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/23.jpg)
VARTAN – Template Contents:Experiment Level Strategy and Planning
2. Summary of Experiment and Strategy Planning
2.1 Expected Experiment Outcomes, Objectives and Hypotheses
2.1.1 Description of Expected Experiment Outcomes
2.1.2 Summary of Expected Experiment Outcomes
2.1.3 Description of Experiment Objectives
2.1.4 Summary of Experiment Objectives
2.1.5 Description of Experiment Hypotheses
2.1.6 Summary of Experiment Hypotheses
2.2 Choice of Metrics and Measurements
2.3 Choice of Methods and Techniques
2.4 Scenario Specifications
2.5 Experimental Variables and Design
![Page 24: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/24.jpg)
VARTAN – Template Contents:Experiment Level Conduct and Results
3. Conduct of Validation Exercise Runs3.1Experiment Preparation3.2Executed Experiment Schedule3.3Deviations from the Planning
4. Experiment Results4.1Measured Experiment Results4.2Confidence in Experiment Results
4.2.1 Quality of Results of Experiment
4.2.2 Significance of Results of Experiment
4.3Unexpected Experiment Results
5. Analysis of Experiment Outcomes5.1Analysis of Outcomes on the Basis of Determined Hypotheses5.2Analysis of Consequences of Outcomes for Experiment Objectives
6. Conclusions and Recommendations for Project Level6.1Conclusions6.2 Recommendations
![Page 25: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/25.jpg)
VARTAN - Template Automation: Basic Information
![Page 26: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/26.jpg)
VARTAN - Template Automation: Standard Structure
![Page 27: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/27.jpg)
VARTAN - Template Automation: Entry Fields
![Page 28: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/28.jpg)
VARTAN – Outlook into the Future
Data definition and requirements for (static) document data and pre-formatted data
Data structures for VDR entry including mutual relations
Encryption standards for usernames and passwords
Data protection modes and protected sections
List of all property fields, bookmarks and macros in use by the automated solution to avoid conflicts
Define rollback mechanisms for all automated changes
Templates available at VALUG website: http://www.eurocontrol.int/valug/
Website offers contact information and is available for feedback on methodology and standard methods
![Page 29: VARTAN – Validation Reporting Templates Jürgen Teutsch, NLR CAATS Workshop, 16-Feb-2006, Lanzarote](https://reader035.vdocuments.us/reader035/viewer/2022062417/55154d79550346c77d8b642f/html5/thumbnails/29.jpg)
Questions