1 measurements & validation r g stamp head of department of atm research national air traffic...
TRANSCRIPT
![Page 1: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/1.jpg)
11
Measurements & ValidationMeasurements & Validation
R G StampR G Stamp
Head of Department of ATM ResearchHead of Department of ATM Research
National Air Traffic Services Ltd, UKNational Air Traffic Services Ltd, UK
![Page 2: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/2.jpg)
22
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
![Page 3: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/3.jpg)
33
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a demonstrated against a pre-defined levelpre-defined level of of functionality, operability and performance”functionality, operability and performance”
![Page 4: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/4.jpg)
44
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a The process through which a desired level of desired level of confidenceconfidence in the ability of a deliverable to in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
![Page 5: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/5.jpg)
55
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a operate in a real-life environmentreal-life environment may be may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
![Page 6: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/6.jpg)
66
What is Validation?What is Validation?
EATMS Validation Strategy (EVAS)EATMS Validation Strategy (EVAS)
““The process through which a desired level of The process through which a desired level of confidence in the ability of a deliverable to confidence in the ability of a deliverable to operate in a real-life environment may be operate in a real-life environment may be
demonstrated against a pre-defined level of demonstrated against a pre-defined level of functionality, operability and performance”functionality, operability and performance”
Validation builds confidence that the system is Validation builds confidence that the system is fit for purposefit for purpose
![Page 7: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/7.jpg)
77
How to ValidateHow to Validate
Operations
Operationaltrial
Shadow-modetrial
Field test
Large-scalereal-time
simulation
Small-scalereal-time
simulation
Fast-timesimulation
Analyticmodelling
![Page 8: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/8.jpg)
88
How to ValidateHow to Validate
Operations
Operationaltrial
Shadow-modetrial
Field test
Large-scalereal-time
simulation
Small-scalereal-time
simulation
Fast-timesimulation
Analyticmodelling
![Page 9: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/9.jpg)
99
Where to ValidateWhere to Validate
Small-scalereal-time
simulation
![Page 10: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/10.jpg)
1010
Where to ValidateWhere to Validate
Small-scalereal-time
simulation
NATSNATS
DLRDLR
CENACENA
NLRNLR
EECEEC
real-life environmentreal-life environment
![Page 11: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/11.jpg)
1111
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
![Page 12: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/12.jpg)
1212
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
![Page 13: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/13.jpg)
1313
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
![Page 14: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/14.jpg)
1414
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
![Page 15: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/15.jpg)
1515
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
![Page 16: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/16.jpg)
1616
Validation MethodologyValidation Methodology
Consistent Application of Validation Methodology Consistent Application of Validation Methodology Across All SitesAcross All Sites
H ig h L eve l O b jec tive
L ow L eve l O b jec tive
N u ll H yp o th es is A lte rn a te H yp o th es is
L ow L eve l O b jec tive L ow L eve l O b jec tive
H ig h L eve l O b jec tive H ig h L eve l O b jec tive
TR IA L A IM
pre-defined level of functionality, operability and performanceof functionality, operability and performance
![Page 17: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/17.jpg)
1717
Measures of MeritMeasures of Merit
MoM’s Chosen to Test HypothesesMoM’s Chosen to Test Hypotheses Controller WorkloadController Workload Airspace CapacityAirspace Capacity Quality of ServiceQuality of Service UsabilityUsability SafetySafety
Measurements Taken:Measurements Taken: e.g. Workload: ISA, TLX, R/T, …..e.g. Workload: ISA, TLX, R/T, ….. Objective; Subjective; ObservedObjective; Subjective; Observed
![Page 18: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/18.jpg)
1818
AnalysisAnalysis
Descriptive StatisticsDescriptive Statistics
ORG 0 ORG 1 ORG 21.5
1.6
1.7
1.8
1.9
2
ORG 0 ORG 1 ORG 2
PC
TC
![Page 19: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/19.jpg)
1919
AnalysisAnalysis
Descriptive StatisticsDescriptive Statistics
Inferential StatisticsInferential Statistics significant at 5% confidence levelsignificant at 5% confidence level
ORG 0 ORG 1 ORG 21.5
1.6
1.7
1.8
1.9
2
ORG 0 ORG 1 ORG 2
PC
TC
![Page 20: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/20.jpg)
2020
AnalysisAnalysis
Descriptive StatisticsDescriptive Statistics
Inferential StatisticsInferential Statistics significant at 5% confidence levelsignificant at 5% confidence level
Backed up by Controller Subjective CommentsBacked up by Controller Subjective Comments ““I can still handle the aircraft while doing the I can still handle the aircraft while doing the
crossword!”crossword!”
ORG 0 ORG 1 ORG 21.5
1.6
1.7
1.8
1.9
2
ORG 0 ORG 1 ORG 2
PC
TC
![Page 21: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/21.jpg)
2121
AnalysisAnalysis
Descriptive StatisticsDescriptive Statistics
Inferential StatisticsInferential Statistics significant at 5% confidence levelsignificant at 5% confidence level
Backed up by Controller Subjective CommentsBacked up by Controller Subjective Comments ““I can still handle the aircraft while doing the crossword!”I can still handle the aircraft while doing the crossword!”
desired level of confidencedesired level of confidence
ORG 0 ORG 1 ORG 21.5
1.6
1.7
1.8
1.9
2
ORG 0 ORG 1 ORG 2
PC
TC
![Page 22: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/22.jpg)
2222
Effectiveness of PHARE ValidationEffectiveness of PHARE Validation
Core Validation Group with Representation from all Core Validation Group with Representation from all Participating SitesParticipating Sites
Common Methodology CrucialCommon Methodology Crucial Methodology Improved with ExperienceMethodology Improved with Experience Validation Conducted by Individual SitesValidation Conducted by Individual Sites
Lessons Re-learnt &/or LostLessons Re-learnt &/or Lost Effectiveness DiminishedEffectiveness Diminished
Validation Should be Applied Throughout Project - Validation Should be Applied Throughout Project - Not Just at the EndNot Just at the End
![Page 23: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/23.jpg)
2323
SummarySummary
Validation Methodology Validation Methodology Major Contributor to Major Contributor to Credibility of PHARECredibility of PHARE
Forms the Backbone for Forms the Backbone for much of the Validation much of the Validation Activities within Activities within European ATM R&DEuropean ATM R&D
![Page 24: 1 Measurements & Validation R G Stamp Head of Department of ATM Research National Air Traffic Services Ltd, UK](https://reader036.vdocuments.us/reader036/viewer/2022070305/55154d8d550346c77d8b6435/html5/thumbnails/24.jpg)
2424
SummarySummary
Validation Methodology Validation Methodology Major Contributor to Major Contributor to Credibility of PHARECredibility of PHARE
Forms the Backbone for Forms the Backbone for much of the Validation much of the Validation Activities within Activities within European ATM R&DEuropean ATM R&D