measurements & validation
DESCRIPTION
Measurements & Validation. R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK. What is Validation?. EATMS Validation Strategy (EVAS) - PowerPoint PPT PresentationTRANSCRIPT
11
Measurements & ValidationMeasurements & Validation
R G StampR G Stamp
Head of Department of ATM ResearchHead of Department of ATM Research
National Air Traffic Services Ltd, UKNational Air Traffic Services Ltd, UK
22
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
33
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a demonstrated against a pre-defined levelpre-defined level of of functionality, operability and performance”functionality, operability and performance”
44
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a The process through which a desired level of desired level of confidenceconfidence in the ability of a deliverable to in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
55
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a operate in a real-life environmentreal-life environment may be may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
66
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
Validation builds confidence that the system is Validation builds confidence that the system is fit for purposefit for purpose
77
How to ValidateHow to Validate
Operations
Operationaltrial
Shadow-modetrial
Field test
Large-scalereal-time
simulation
Small-scalereal-time
simulation
Fast-timesimulation
Analyticmodelling
88
How to ValidateHow to Validate
Operations
Operationaltrial
Shadow-modetrial
Field test
Large-scalereal-time
simulation
Small-scalereal-time
simulation
Fast-timesimulation
Analyticmodelling
99
Where to ValidateWhere to Validate
Small-scalereal-time
simulation
1010
Where to ValidateWhere to Validate
Small-scalereal-time
simulation
NATSNATS
DLRDLR
CENACENA
NLRNLR
EECEEC
real-life environmentreal-life environment
1111
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
1212
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
1313
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
1414
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
1515
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
1616
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
pre-defined level of functionality, operability and performanceof functionality, operability and performance
1717
Measures of MeritMeasures of Merit
MoM’s Chosen to Test HypothesesMoM’s Chosen to Test Hypotheses Controller WorkloadController Workload Airspace CapacityAirspace Capacity Quality of ServiceQuality of Service UsabilityUsability SafetySafety
Measurements Taken:Measurements Taken: e.g. Workload: ISA, TLX, R/T, …..e.g. Workload: ISA, TLX, R/T, ….. Objective; Subjective; ObservedObjective; Subjective; Observed
1818
AnalysisAnalysis
Descriptive StatisticsDescriptive Statistics
ORG 0 ORG 1 ORG 21.5
1.6
1.7
1.8
1.9
2
ORG 0 ORG 1 ORG 2
PC
TC
1919
AnalysisAnalysis
Descriptive StatisticsDescriptive Statistics
Inferential StatisticsInferential Statistics significant at 5% confidence levelsignificant at 5% confidence level
ORG 0 ORG 1 ORG 21.5
1.6
1.7
1.8
1.9
2
ORG 0 ORG 1 ORG 2
PC
TC
2020
AnalysisAnalysis
Descriptive StatisticsDescriptive Statistics
Inferential StatisticsInferential Statistics significant at 5% confidence levelsignificant at 5% confidence level
Backed up by Controller Subjective CommentsBacked up by Controller Subjective Comments ““I can still handle the aircraft while doing the I can still handle the aircraft while doing the
crossword!”crossword!”
ORG 0 ORG 1 ORG 21.5
1.6
1.7
1.8
1.9
2
ORG 0 ORG 1 ORG 2
PC
TC
2121
AnalysisAnalysis
Descriptive StatisticsDescriptive Statistics
Inferential StatisticsInferential Statistics significant at 5% confidence levelsignificant at 5% confidence level
Backed up by Controller Subjective CommentsBacked up by Controller Subjective Comments ““I can still handle the aircraft while doing the crossword!”I can still handle the aircraft while doing the crossword!”
desired level of confidencedesired level of confidence
ORG 0 ORG 1 ORG 21.5
1.6
1.7
1.8
1.9
2
ORG 0 ORG 1 ORG 2
PC
TC
2222
Effectiveness of PHARE ValidationEffectiveness of PHARE Validation
Core Validation Group with Representation from all Core Validation Group with Representation from all Participating SitesParticipating Sites
Common Methodology CrucialCommon Methodology Crucial Methodology Improved with ExperienceMethodology Improved with Experience Validation Conducted by Individual SitesValidation Conducted by Individual Sites
Lessons Re-learnt &/or LostLessons Re-learnt &/or Lost Effectiveness DiminishedEffectiveness Diminished
Validation Should be Applied Throughout Project - Validation Should be Applied Throughout Project - Not Just at the EndNot Just at the End
2323
SummarySummary
Validation Methodology Validation Methodology Major Contributor to Major Contributor to Credibility of PHARECredibility of PHARE
Forms the Backbone for Forms the Backbone for much of the Validation much of the Validation Activities within Activities within European ATM R&DEuropean ATM R&D
2424
SummarySummary
Validation Methodology Validation Methodology Major Contributor to Major Contributor to Credibility of PHARECredibility of PHARE
Forms the Backbone for Forms the Backbone for much of the Validation much of the Validation Activities within Activities within European ATM R&DEuropean ATM R&D