single european sky implementation support through validation · the validation lessons learned...

Episode 3

D2.3-06 - Lessons Learnt for the application of E-OCVM to integrated Validation processes

Version : 1.01

of 24

Issued by the Episode 3 consortium for the Episode 3 project co-funded by the European Commission and Episode 3 consortium.

EPISODE 3 Single European Sky Implementation support through Validation

Document information Programme Sixth framework programme Priority 1.4 Aeronautics and Space

Project title Episode 3

Project N° 037106

Project Coordinator EUROCONTROL Experimental Centre

Deliverable Name Lessons Learnt for the application of E-OCVM to integrated Validation processes

Deliverable ID D2.3-06

Version 1.01

Owner Alistair Jackson EUROCONTROL

Contributing partners ISDEFE

Episode 3


Version : 1.01

of 24


- This page is intentionally blank -

Episode 3


Version : 1.01

of 24


DOCUMENT CONTROL Approval

Role Organisation Name Document owner EUROCONTROL Alistair Jackson

Technical approver EUROCONTROL Giuseppe Murgese

Quality approver EUROCONTROL Frédérique Sénéchal

Project coordinator EUROCONTROL Philippe Leplae

Version history

Version Date Status Author(s) Justification - Could be a

reference to a review form or a comment sheet

1.00 25/09/2009 Approved Alistair Jackson Alexandra Dorbes Martijn Koolloos

Approval of the document by the Episode 3 Consortium.

1.01 25/09/2009 Approved Catherine Palazo Minor format changes

Episode 3


Version : 1.01

of 24


TABLE OF CONTENTS EXECUTIVE SUMMARY ...........................................................................................................5 1 INTRODUCTION................................................................................................................6

1.1 PURPOSE OF THE DOCUMENT ........................................................................................6 1.2 INTENDED AUDIENCE ....................................................................................................6 1.3 BACKGROUND ..............................................................................................................6 1.4 SCOPE .........................................................................................................................6 1.5 CONTEXT .....................................................................................................................7 1.6 DOCUMENT STRUCTURE ...............................................................................................8 1.7 GLOSSARY OF TERMS ...................................................................................................9

2 KEY LESSONS LEARNT FROM THE EXPERIENCE OF ORGANISING VALIDATION PROCESSES ...........................................................................................................................10

2.1 PROJECT MANAGEMENT..............................................................................................10 2.2 VALIDATION PROCESS MANAGEMENT ..........................................................................11

3 VALIDATION LESSONS LEARNT WORKSHOP...........................................................13 3.1 VALIDATION WORKSHOP PARTICIPATION AND ORGANISATION .......................................13 3.2 MAIN AREAS OF RESULTS ............................................................................................13

4 LESSONS FROM OTHER WP2 TRANSVERSAL ACTIVITIES .....................................16 4.1 TRANSVERSAL COORDINATION CELL............................................................................16 4.2 CONCEPT DEVELOPMENT............................................................................................16 4.3 PERFORMANCE FRAMEWORK ......................................................................................17

5 CONSOLIDATION: RECOMMENDATIONS & ACTIONS ..............................................19 5.1 PRINCIPAL RECOMMENDATIONS TO THE EPISODE 3 PROJECT .......................................19 5.2 ACTIONS TAKEN ON RECOMMENDATIONS TO THE EPISODE 3 PROJECT ..........................19 5.3 PRINCIPAL RECOMMENDATIONS TO THE VALIDATION COMMUNITY & E-OCVM...............20 5.4 ACTIONS TAKEN ON RECOMMENDATIONS TO THE VALIDATION COMMUNITY AND E-OCVM 21 5.5 PRINCIPAL RECOMMENDATIONS TO SESAR.................................................................21 5.6 ACTIONS TAKEN ON RECOMMENDATIONS SESAR ........................................................22

6 REFERENCES.................................................................................................................23

Episode 3


Version : 1.01

of 24


EXECUTIVE SUMMARY This document presents a synthesis of the lessons learnt in the Episode 3 project in the application of the European Operational Concept Validation Methodology to an initial validation of a large scale, complex operational concept, namely the SESAR CONOPS. The lessons derive from the experience of the project staff involved in the organisation and coordination of validation practice throughout the project, and from the experience of those who had to apply this practice to the definition and conduct of validation exercises. The experience of the latter group was collected through a dedicated workshop, managed by an ‘external’ facilitator. The resulting ‘lessons’ are expressed as ‘recommendations’ and are presented in terms of the communities who can most readily benefit from them: the Episode 3 project itself; the wider validation community; and the SESAR Joint Undertaking which faces the challenge of managing the SESAR CONOPS through validation and verification to implementation.

Episode 3


Version : 1.01

of 24


1 INTRODUCTION

1.1 PURPOSE OF THE DOCUMENT The purpose of this document is to capture and communicate the experience gained within the Episode 3 project in the application of the European Operational Concept Validation Methodology (E-OCVM [1]) to validate an operational concept of the scale and complexity of the SESAR CONOPS [2]. The objective is to improve the capability and effectiveness of future concept clarification and validation activities.

1.2 INTENDED AUDIENCE The intended audience consists of all those who will be involved in the planning, management, resourcing or conduct of activities to develop ATM Operational concepts through the concept development lifecycle towards implementation and operation. Consequently, it has potential significance to the programme, project and exercise levels of SESAR JU.

1.3 BACKGROUND Current changes in European aviation and ATM research, and in the development environment will place a focus on the rapid transfer of promising validated concepts into real operation.

The Episode 3 project, co-ordinated by EUROCONTROL, is an FP6 project co-funded by the European Commission, and bringing together leading experts in ATM operational validation from 21 different organisations, including Air Navigation Service Providers (ANSPs), research centres, Subject Matter Experts (SMEs), and universities. The purpose of Episode 3 was initially to set up a performance-based concept validation approach. This approach was adapted to prepare for the SESAR concept validation.

Episode 3 is charged with beginning the validation of the operational concept expressed by SESAR (Single European Sky ATM Research) Task 2.2 and consolidated in SESAR D3 (The ATM Target Concept) [2]. The emphasis is on:

• Clarification of the concept;

• Expanding the repertoire of cost-effective validation techniques;

• Consolidating our learning on the application of the E-OCVM to SESAR-scale Concept of Operations.

The validation process applied in Episode 3 is based on version 2 of the E-OCVM [1]. The E-OCVM is an agreed methodology for establishing fitness for purpose of proposed concepts. It aims to provide a common approach for all projects contributing to the validation of operational ATM concepts from early issues identification to full pre-operational validation. The E-OCVM focuses on the R&D activities involved in ATM concept development and covers – in terms of the concept lifecycle model – the first four phases prior to industrialisation and operational introduction: V0 to V3.

1.4 SCOPE The current document focuses specifically on lessons learnt with respect to validation management and the application of the E-OCVM. Its content is drawn principally from three sources:

Episode 3


Version : 1.01

of 24


• From the experience of project staff1 directly involved in the work developing and managing validation across the project.

• From the many staff involved in applying E-OCVM at the Work Package (WP) and at the Exercise level across the project. This was managed through a dedicated, externally facilitated workshop which took place in Madrid in March 2009. A separate report [3] provides a detailed description of the process and conduct of that workshop. Only the results are reported here.

• From the work of other transversal Episode 3 activities, e.g. Coordination Cell, Performance Framework, where experience indicated the need for additional support to validation.

The Validation Lessons Learned task, and the associated Madrid workshop, was included in the EPISODE 3 Project Description of Work [4] following the contract revision in July 2008. A second ‘Lessons Learnt’ Workshop will take place in September 2009 to consolidate experience from all aspects of the project. If there are significant additional inputs relating to validation, the present document will be updated as a second edition.

1.5 CONTEXT Much of the interest in considering validation lessons learnt from Episode 3 derives from its historical circumstances and the challenges which it had to face as a consequence.

As a result, a number of major issues were identifiable from a very early stage in the project and will appear as recurrent themes throughout this document:

• The existence of a DOW, which was detailed to the exercise level, prior to the development of a validation strategy.

• A significant increase in scale and complexity of the target operational concept effectively making the notion of a ‘complete’ validation unachievable. There were also timing issues linked to the availability of a stable concept description at project outset.

• A mature performance framework was not available prior to the definition of the exercises in the DOW. This was particularly important given the pressure to produce an integrated approach to validation within performance targets recently identified for SESAR.

• Relatively little experience with the E-OCVM in projects of this scale and complexity and the lack of suitable training and case guidance material for E-OCVM V2.0.

These factors played an important role in establishing the initial approach to validation in the project. This was based on the recognition that an integrated validation approach could not be readily achieved within the initial exercises already defined. Instead, the emphasis was placed on:

• Establishing a performance framework which would address performance issues at both a system wide and local levels;

• Providing guidance material and templates to assist in the understanding of the E-OCVM and its application to WP and exercise planning;

• Achieving understanding and alignment with the newly presented SESAR CONOPS;

with the intention of using the progress made and the experience gained to be prepared to identify exercises and measures suited to the test of a more top-down and integrated

1 Mainly in the Project Management (WP0) and Validation Process Management (WP2.3) work packages.

Episode 3


Version : 1.01

of 24


approach to performance based validation in the later stages of the project. However, following an audit of Episode 3 in the first half of 2008, the duration of the project was decreased and the priorities and focus of the project shifted to place more emphasis on concept clarification and the exploration of validation techniques [5]. Consequently, some of the recommendations that follow from the lessons learnt within this document represent Episode 3’s best understanding and the practice that would now be proposed, based on experience. They may not yet have been validated through successful application.

1.6 Document Structure

The document is structured in the following manner:

• Section 1 (this section) provides a brief introduction to the Report;

• Section 2 identifies key lessons learnt from the experience of organising validation processes within Episode 3;

• Section 3 presents lessons based on how validation process was experienced by the partners in Episode 3 and captured through the Madrid Lesson Learnt Workshop and the subsequent consolidation process;

• Section 4 complements the previous sections by identifying lessons which emerged from other WP2 transversal activities;

• Section 5 consolidates the lessons as recommendations and closes with a description of the actions and initiatives, which are being taken to ensure that these experiences bring benefit to future validation activities.

Episode 3


Version : 1.01

of 24


1.7 GLOSSARY OF TERMS

Term Definition ATM Air Traffic Management

CAATS II Cooperative Approach to Air Traffic Services, a 6th Framework Project of the European Commission

CLM Concept Lifecycle Model, as described in E-OCVM V2.0 and extended by SESAR

CONOPS Concept of Operations

DOD Detailed Operational Description of the operational concept

DOW Description of Work

E-OCVM European Operational Concept Validation Methodology

FP6 6th Framework Programme (of the European Commission)

PMB Project Management Board

R&D Research and Development

RMC Requirements Management Cell

SARD Strategic Analysis of Research and Development

SESAR Single European Sky ATM Research

SJU SESAR Joint Undertaking

SME Subject Matter Expert

TMA Terminal Manoeuvring Area

VMC Validation Management Cell

WP Work Package

Episode 3


Version : 1.01

of 24


2 KEY LESSONS LEARNT FROM THE EXPERIENCE OF ORGANISING VALIDATION PROCESSES

This section summarises the key lessons learnt from the experience of project staff involved in Project Management (WP0); System Consistency (WP2) and the principal validation workpackages, organising, conducting and managing validation across the three operational segments, Collaborative Planning Processes (WP3); En-Route (WP4) and TMA & Airport (WP5).

2.1 PROJECT MANAGEMENT The following lessons are derived from the experience of managing a project in which large scale concept validation was the principal activity. They were elicited directly from the project management staff.

• Managing validation requires flexibility and the ability to focus and refocus priorities as validation needs are identified. Consequently, there is a need to be able to re-allocate resources (especially between partners) and to change work priorities. This is only possible if this need has been anticipated and written into the consortium agreement beforehand, which was not the case for Episode 3. Similarly, there was an important lack of mechanisms within the project to direct and steer technical content and resources.

• Resource planning for validation activity is difficult and requires that the planning managers already understand the nature of validation activities and the needs for special skills. Those skills also have to be available, directly or through training. Planning can be improved by:

o Having a clearer view of the skills required by staff performing validation activities (e.g. designing exercise plans, understanding measurement);

o Having better information on what skills are available (e.g. CVs of staff assigned);

o Budgeting and planning for the provision of any necessary training;

o Having better guidance material on relevant validation techniques and their usefulness, applicability, and likely scale and cost, for exercises at different lifecycle stages and concept maturity levels;

o Having a better understanding of the effort required for the review of experimental plans, etc. which became key tasks in Episode 3 for ensuring consistency but placed unanticipated demands on resources.

• There is a need for a better approach to tracking validation progress. The Episode 3 approach was based on the production and delivery of documents, plans, etc. Perhaps a better approach would have been to base it on some of the objectives of identified in the E-OCVM, e.g. identification of validation objectives, identification of stakeholder needs, etc.

• The roles and responsibilities of a number of key actors, e.g. WP leaders, exercise leaders, etc, could have been more clearly defined with respect to validation.

• Project participants achieved considerable benefit from common mechanisms and tools, EPISODE 3 Website, review tool, templates and guidelines, the Information Navigator, DODs, etc. This is especially important for large, multi-partner, distributed projects.

Episode 3


Version : 1.01

of 24


2.2 VALIDATION PROCESS MANAGEMENT Lessons and issues in this section are drawn from the experience of project staff in WP2.3, Validation Process Management. The WP2.3 activity was responsible for the overall project validation strategy and approach, ensuring the consistent application of the E-OCVM throughout the project, identifying validation requirements for the performance framework and establishing common references and baselines. This included the discussion on how to achieve representativeness through the exercises, e.g. traffic, airspace, controller participation, etc.:

• When applied to a large and diverse concept of the scale and complexity of SESAR, the already difficult task of initial validation maturity assessment becomes much more difficult and more critical. With hindsight, resources should have been specifically identified and focused on this area much earlier. Similarly, when a maturity assessment has been made, significant effort must be foreseen for the identification of the appropriate validation techniques.

• To ensure consistency of the validation results, it is important that a consistent validation framework (i.e. performance framework, targets, key assumptions, etc) is made available at the outset of the project (see also sections 3.2 & 4.3).

• In order to be effective, a clear document review process has to be put in place and the supporting tools should be ready and available at the outset of the project.

• In a validation process, there is a need for guidance material and templates tailored to the type of validation exercise. However, as ‘one size doesn’t fit all’, it is equally important to allow flexibility in the use of templates and to be clear on how the flexibility should be used.

• Any guidance material should be provided in a timely manner: too soon may mean that the required understanding is not there, too late may mean that more material has to be re-worked.

• In Episode 3, much of the guidance and reference documentation that was provided by the validation team was not used or was ignored, sometimes probably due to ineffective communication. Guidance information must be made accessible by being put in the right place and promoted to ensure that it is used.

• Written validation support material alone is not enough. Experience in Episode 3 has shown that having direct and immediate access to ‘live’ advice and coaching on validation issues and matters is highly beneficial to the validation practitioners.

• At the start of the Episode 3 project, there was a general lack of understanding of a number of important validation issues, i.e. how to express assumptions, how to design exercises, how to express the exercise hypotheses and to how to make effective measurement. This lack stresses the importance of having the right skill profiles and appropriate training materials to ensure that the right competences are made available to perform effective validation (See also section 2.1).

• Episode 3 undertakes validation activities covering a range of operational segments (e.g. Collaborative Planning, En-route, etc), thus validation exercises have very different needs. These needs may be difficult to reconcile when we also wish to keep the results within a common reference framework (ECAC wide traffic sample, common assumptions on routing, etc). In Episode 3, considerable effort was devoted to building the right traffic. En-route exercises may need representative ‘on the day traffic’ whereas the network planning may need traffic ‘6 months prior to take-off’. Despite the effort, it was difficult to establish a suitable, all embracing,

Episode 3


Version : 1.01

of 24


traffic sample, i.e. only 50% of exercises used the provided reference ‘demand’2 and many of these had to make qualifying assumptions. The remaining exercises not only did not use the reference demand but were unable to establish the relationship between their traffic and the reference demand. On this basis, there is not yet a clear understanding of how to build suitable traffic and what assumptions are necessary to make it representative. A similar situation applies to identifying representative airspace, and ‘typical’ controller working methods.

2 ‘Demand’ is more than ‘traffic’: as well as aircraft and type mix, traffic needs to include context information e.g. routes and military areas, etc.

Episode 3


Version : 1.01

of 24


3 VALIDATION LESSONS LEARNT WORKSHOP

3.1 VALIDATION WORKSHOP PARTICIPATION AND ORGANISATION The lessons presented in this section were derived from a specially organised facilitated workshop which took place in Madrid over 2½ days from 10th – 12th March 2009.

It involved 20 participants chosen to represent the different work packages, exercise types and validation roles across the project. In contrast to the validation expert perspective presented in section 2, the intention of the workshop was to capture how the project members experienced validation from the user perspective.

Based on preparatory questionnaires, the workshop focused on four main areas of the Episode 3 validation activity:

• Validation Strategy;

• Concept Validation;

• Concept and Performance breakdown and consolidation;

• Validation Techniques.

The material gathered during the workshop was consolidated into a database of over 350 individual comments before each was classified according to a custom taxonomy. The comments were then analysed using a hierarchical cluster analysis method in order to give structure and priority to the findings.

The details of the workshop, organisation, analysis techniques and results are fully described in the dedicated report [3].

3.2 MAIN AREAS OF RESULTS The outputs of the cluster analysis highlighted seven topics for further discussion. The outcome of the discussion is as follows.

Topic 1: Validation Strategy and project Description of Work

According to the workshop participants, the Validation Strategy was too dependent on the Project DoW, with additional constraints due to project timescales. They considered that Validation Strategy should aim to:

• Provide a top down approach to validation and planning;

• Account for concept maturity;

• Allow the proper selection and prioritisation of exercises;

• Provide suitable coverage of the various aspects of the operational concept;

• Promote coherency across all exercises and WP;

• Allow performance to be broken down and integrated again.

The participants felt that the processes actually used in Episode 3 were heavily reliant on project deliverables and pre-defined constraints. They considered that this could be improved if the processes were more task driven and strategically driven, with greater consideration given to the areas of concept and performance breakdown and integration.

Topic 2: Concept clarification and suitable techniques

Participants felt that one of the major successes in the Episode 3 project was the use of a number of techniques, which are relatively new to ATM concept validation, and the

Episode 3


Version : 1.01

of 24


assessment of their suitability for clarification of the SESAR concept. Examples include gaming and facilitated expert groups.

Topic 3: E-OCVM guidance

Participants stated that one area they need more guidance on was on the Concept Lifecycle Model, in relation to:

• Characteristics of each phase;

• How to correctly and consistently identify concept level of maturity;

• Transition criteria between the phases;

• Typical validation techniques for each phase.

Additional guidance may also be required for new validation techniques (e.g. Expert Groups, Gaming), but for the E-OCVM to remain usable, it must remain simple and concise. Therefore additional validation support may be required from (local) validation community representatives.

Topic 4: Use of common exercise participants

Participants stressed the importance of continuity of experts/controllers across exercises when using certain techniques, particularly Expert Groups and in Iterative Prototyping sessions. Variability in exercise participants implies:

• Repetition of basic tasks, which restricts the ability to build on previous exercises; and

• Increased variability in exercise results, meaning less scope for comparison across exercises.

In addition, the choice of the right experts regarding the topic, with a good level of knowledge/expertise is equally as important as their continuity.

However, it is important to note that in other kinds of exercise it may be important to have as wide a sample of experts as possible in order to achieve a broad or representative input.

Topic 5: Concept and performance integration

Participants recognised the difficulty of integrating and consolidating results and findings from different environments, different exercises, different techniques, different concepts and different Key Performance Areas. The Episode 3 performance framework was seen to give some benefit. For example, the use of common baselines, time horizons, scenario characteristics and geographic area would increase the effectiveness of the exercise measurements and the comparability of Key Performance Indicators.

Topic 6: Project Management

Project planning:

• More consideration should be given to the effort taken for certain tasks such as document reviews.

WP management:

• Attention should be paid to coordination and communication across WP, and particularly between the operational and the technical WP.

Topic 7: Assumptions

Participants stressed the need for common and consistent assumptions used to set the scope of the contextual information, related to the concept and to the validation scenarios. To avoid

Episode 3


Version : 1.01

of 24


stand-alone results which cannot be compared with other exercise results and to facilitate the performance of a cross-WP result analysis and compilation, it is important that there is consistency in assumptions across all exercises. To enable this consistency requires assumptions management from the very beginning of the project.

Episode 3


Version : 1.01

of 24


4 LESSONS FROM OTHER WP2 TRANSVERSAL ACTIVITIES

This section brings together some additional lessons derived from other transversal activities within System Consistency (WP2) that have been closely associated with validation.

4.1 TRANSVERSAL COORDINATION CELL One of the key mechanisms for ensuring consistency of working method and content in Episode 3 was through transversal coordination meetings (across work packages) organized by WP2. Initially there were two of these; the Requirements Management Cell (RMC) dealing with the operational requirements and concept and the Validation Management Cell (VMC) dealing with validation issues. These were complemented by the Project Management Board (PMB) made up of WP leaders and chaired by WP0 (Project Management) which addressed project management issues. Through overlap in content and participation, the RMC and VMC merged into a single body concerned with the technical quality and content of Episode 3 processes and exercises – the Coordination Cell. The lessons in this section relate to the role and functioning of this transversal Coordination Cell.

• The Coordination Cell played an important and useful role in addressing a diverse range of transversal issues. These included providing a coordination forum on the definition and use of traffic samples, initiating the work on assumptions management, developing validation reporting templates, etc. It provided an important resource for identifying and reactively managing practical issues across the project.

• The review of exercise plans as documents and through presentations at Coordination Cell meetings was not as effective as it should have been because there was not sufficient time between plan delivery and exercise execution to effect change. This situation arose because of too much focus on the exercise plan as a deliverable and not enough on it as a tool to facilitate exercise design and execution.

• It is important that the Coordination Cell should have access to some skilled resources, beyond that needed to attend meetings, so that they can act based on the validation issues identified at the meetings. It is also important that it has a clearly defined link to the PMB because there is potential overlap between the two groups. – It is sometimes impossible to maintain a clear distinction between management and purely technical issues. – The links allow actions to be unambiguously passed to the correct body and ensure that there is no duplication of effort.

• There was benefit in having a plan for the contents of the series of coordination meetings already identifying what would be covered and at what meeting.

4.2 CONCEPT DEVELOPMENT3 The lessons in this section are drawn from the experience of project staff in WP2.2, Clarification and Refinement of the SESAR CONOPS. As it became clear, at an early stage in the project, that not all of the SESAR CONOPS was detailed to the same level of precision, it was also recognised that documenting and clarifying the concept would be an important aspect of the Episode 3 contribution. The principal components of this work were the development of Detailed Operational Description (DOD) [6] for documenting different

3 The term concept development is used to describe the development of a concept from the original recognition of need for change, through the concept lifecycle to implementation and operation. It is not intended to imply only the initial concept definition activity.

Episode 3


Version : 1.01

of 24


operational segments of the concept, illustrative scenarios, and the SESAR-Episode 3 Information Navigator [7], which allowed association of concept and validation information within a powerful browsing tool. The key lessons from the concept clarification activity were:

• Problems were experienced in comprehending and supporting the concept elements and maintaining a common interpretation and understanding. The SESAR CONOPS was initially proposed by a core group, who were no longer accessible as Episode 3 gained momentum. The project team was then left with the task of explaining design decisions in which the project had not been involved. Related to this, it is important to note:

o The need to maintain continuity in the concept expert team; and

o The difficulty when there is no ‘authority’ to speak on behalf of the concept. (Efforts to mitigate this by creating a User Group could not be realised).

• The description of the operational segments could have been presented more effectively to the validation exercises by presenting the DODs top down with the overview (G DOD) being introduced first [6].

• The process model was useful to those detailing the concept as it ensured a comprehensive and coherent treatment of the concept both with initial specification of the DODs and the subsequent detailing.

4.3 PERFORMANCE FRAMEWORK The lessons in this section are derived from two sources: from the experience of the WP2.4.1 team in establishing a project wide performance framework, and from a specific lessons learnt exercise conducted by the team responsible for the development of the influence diagram based, performance model [8].

• To establish a common basis for performance oriented validation and ensure consistency of all the exercise results, a performance framework describing the targeted performance areas, indicators and metrics, has to be made available from the start of the project (see also section 2.2).

• Influence modelling and the identification of uncertainties can be very useful in the prioritisation of which validation activities are important and which potential benefits are the most worth pursuing. More specifically:

o Performance modelling and influence diagrams are appropriate tools to make clear performance requirements and performance mechanisms. They can be shared and discussed and are a good tool for constructing consensus. However, especially at the start of the project, analysis should be kept at a low level of detail (‘don’t model too deep, too soon’). The level of detail can be adjusted when necessary.

o Influence and performance modelling are also useful to show gaps in both understanding and information. The lack of information becomes particularly evident when there is a need to construct the equations of a numerical model.

• It is important in the performance modelling to have continuity of the work team and to have someone responsible and dedicated to the process of integration and maintenance of the models. In addition to a strong modelling expertise, the development team needs to include the right balance of wide ATM operational design, safety and environmental expertise.

• There is still a need to do basic work to generate taxonomies allowing the linking of performance at global and local levels in En-route and TMA, comparable with the taxonomies which have been developed for airports.

Episode 3


Version : 1.01

of 24


• It is important to be able to identify when a target can really not be achieved by the selected concept in order to either revise the concept or recognise when the target is unachievable and has to be revised.

• More emphasis should be placed on the representativeness of validation data. Often when validated at a local level (airport, portion of airspace), its validity at the global European level is uncertain or unknown. In order to estimate how other areas could also benefit from the results to be able to build a picture at a more global level, key representative characteristics of the required data should be part of the definition of validation exercises.

Episode 3


Version : 1.01

of 24


5 CONSOLIDATION: RECOMMENDATIONS & ACTIONS

The findings in the preceding sections have been analysed, and expressed in terms of recommendations with respect to three main stakeholder groups: the Episode 3 project, the E-OCVM and validation community, and the SESAR project (including the SESAR JU). Where the Episode 3 project can undertake some influencing activity to further these recommendations, actions are identified.

5.1 PRINCIPAL RECOMMENDATIONS TO THE EPISODE 3 PROJECT With regard to the Episode 3 project, it was felt that the main action on the project was to report and disseminate the lessons learnt to a wider audience as quickly as possible and to continue the lessons learnt process until the end of the project. The success of the Episode 3 project in many areas was also noted, especially as it was the first project to use the E-OCVM and to apply newer validation techniques on such a wide validation programme scale.

In summary, Episode 3 should:

a) Present the findings of the Episode 3 Validation Lessons Learned exercises in a manner designed to be clearly understandable by the key stakeholder groups;

b) Disseminate the findings as quickly as possible and to as wide an audience as possible;

c) Ensure that any potentially useful examples and materials that might be of benefit to the wider validation community and the SJU are suitably packaged, available and publicised;

d) Use the findings from the workshop themselves in the remaining project tasks;

e) Conduct a 2nd workshop on the wider project issues and disseminate the findings;

f) Carefully manage and document the assumptions management process, and define the lessons learnt in that respect;

g) Recognise the successes that the project has achieved.

5.2 ACTIONS TAKEN ON RECOMMENDATIONS TO THE EPISODE 3 PROJECT The following actions have been initiated or are planned to support realisation of the recommendations proposed in section 5.1:

• A report specifically describing the Lessons Learnt Workshop is already available [3]. Both it and the present document will be publicly available through the Episode 3 website (http://www.episode3.aero/). {R/E3 a,b}4

• Lessons learnt will be reported as part of the Episode 3 final reporting and will form part of the extended dissemination planning. {R/E3 a,b}4

• Episode 3 has established a close relationship to the CAATS II project and in their joint final dissemination forum in October, they will explain and promote their improved understanding on concept validation. {R/E3 a,b,c,g}4

4 The bracketed list identifies the recommendations that a particular action addresses or supports. R/E3, R/VC & R/JU identify the recommendations directed to Episode 3, the validation community and E-OCVM, and the SJU respectively. The following lower case letters identify the individual recommendations.

Episode 3


Version : 1.01

of 24


• Internally the full results of the lessons learnt workshop were communicated as soon as they became available through a series of presentations at significant Episode 3 meetings; e.g. the Coordination Cell, the individual Work Package Coordination meetings and the performance modelling team. Initial results were also presented to the workshop participants in the final session of the lessons learnt workshop itself. {R/E3 a,b,d}4

• Externally, the Episode 3 Project Manager has presented the results of the workshop directly to the E-OCVM Validation Forum, the steering group responsible for the management of E-OCVM and the development of E-OCVM V3.0. The results have also been presented to the EC and the SJU as part of the Episode 3 mid term review, which took place in June 2009. {R/E3 a,b}4

• A second Episode 3 Lessons Learnt workshop, with a scope beyond validation, is organised for September 2009. Assumptions management has been identified on the agenda of that meeting where the work of a Coordination Cell taskforce on assumptions management will be presented. {R/E3 c,e,f}4

• Additionally, Episode 3 is undertaking an additional activity to better document experience with new validation techniques. {R/E3 c, R/VC f}4

5.3 PRINCIPAL RECOMMENDATIONS TO THE VALIDATION COMMUNITY & E-OCVM

For the E-OCVM and wider validation community the main recommendation is for the provision of more guidance related to the Concept Lifecycle Model (CLM), elaborating on the definition, characteristics and transition criteria of the concept lifecycle phases, concept and performance breakdown and integration, and more practical guidance on applying the methodology. The E-OCVM should also provide guidance and reference to the newer validation techniques such as those used in Episode 3.

To be more specific, the E-OCVM should:

a) Provide guidance on how to incorporate a better approach to project definition using validation strategy to ensure that the planning of validation activities and exercises, reflects ATM performance and strategic priorities, whilst recognising the practical constraints of programmes and projects;

b) Provide more guidance or illustration on how the methodology applies to larger, more complex projects and concepts: emphasising the need to define the roles and responsibilities of the actors involved in concept clarification and validation activities in order to avoid duplication of effort and conflicting decisions;

c) Provide more guidance on the Concept Lifecycle Model, how to define concept maturity and how to recognise transition from one level of maturity to the next;

d) Explain the need to anticipate possible changes in validation priorities and ensure appropriate resource re-allocation mechanisms;

e) Provide guidance on how concept and/or performance results from various projects and exercises can be consolidated together into more global results and cases;

f) Provide guidance on or include reference to the newer validation techniques such as those used in Episode 3.

In addition, the validation community should:

g) Gain a better understanding of the Expert Group techniques, including roles, uses and best practices;

h) Begin to understand how the validation community can provide support and guidance to practitioners where the E-OCVM does not.

Episode 3


Version : 1.01

of 24


5.4 ACTIONS TAKEN ON RECOMMENDATIONS TO THE VALIDATION COMMUNITY

AND E-OCVM The following actions have been initiated or are planned to support realisation of the recommendations proposed in section 5.3.

Both CAATS II and Episode 3 are making direct contributions to the update of the E-OCVM to Version 3.0 through the E-OCVM Validation Forum thus ensuring that their lessons have an appropriate impact. It is intended that E-OCVM Version 3.0 will be available by the end of the 2009. Planned changes to the E-OCVM include:

• Based on Episode 3 experience, the distinction between initiation of a single validation process and the organisation of a complex programme with many exercises or even projects will be clarified. In particular, the importance of driving the planning of validation exercises based on strategic priorities and validation objectives will be emphasised. {R/VC a,b,d}

• Increasing the emphasis on the Concept Lifecycle Model and reinforcing the importance of maturity assessment and management of transitions and progress through the lifecycle. Guidance on maturity assessment will be provided based on SARD [9] as special initiative by CAATS II. {R/VC c, d}

• The validation process will be put more clearly in context, emphasizing the importance of the performance framework and shared assumptions from an early stage to allow integration of results and support for decision making. {R/VC e}

While additional guidance is planned, based on the work of CAATS II, to support application of the case based approach, and in particular, its extension to cover the environmental, business human performance and standardisation and regulation cases, Version 3.0 will integrate only general guidance material on the application of innovative validation techniques. Consequently, the E-OCVM update will not cover point R/VC f, completely. This will be addressed further by an additional Episode 3 deliverable summarising the project experience with newer techniques [10].

5.5 PRINCIPAL RECOMMENDATIONS TO SESAR For SESAR, the main consideration is to ensure that the lessons learnt in Episode 3 are expressed clearly and in a manner relevant to the activities of SESAR so that they can be taken on board. The objective is to capitalise on the successes of Episode 3 within SESAR while ensuring that the issues identified by Episode 3 can be addressed effectively.

At programme level, SESAR JU and the rest of the SESAR partners should:

a) Take on board the lessons learnt during Episode 3 and act upon them;

b) Ensure that the programme is driven by a strategic approach, not a partner/resource approach;

c) Make use of new and emerging validation techniques, such as those used in Episode 3, where appropriate;

d) Clearly define the roles and responsibilities of the different bodies involved in concept clarification and validation activities in order to avoid duplication of effort and conflicting decisions;

e) Designate a ‘concept authority’ who is entitled to take decisions with regard to the concept interpretation and evolution;

f) Provide common validation support materials, communication and information/data sharing processes and supporting tools;

g) Contribute to the definition and approval of E-OCVM V3.0;

Episode 3


Version : 1.01

of 24


h) Promote the principles of the E-OCVM (consistency, coherency, transparency) across all aspects of the project management process in general and across exercises, assumptions, techniques and concepts in particular.

At a project level, there is benefit to be gained by:

i) Ensuring resource planning fully takes into account key validation tasks and required skills;

j) Anticipating the possibility of changes in validation priorities and ensuring appropriate resource re-allocation mechanisms.

5.6 ACTIONS TAKEN ON RECOMMENDATIONS SESAR Actions, already identified in 5.2 and 5.4, are considered as supporting realisation of the recommendations proposed in section 5.5, in particular:

• A report specifically describing the Lessons Learnt Workshop is already available [3]. Both it and the present document will be publicly available through the Episode 3 website (http://www.episode3.aero/). {R/JU a}

• Lessons learnt will be reported as part of the Episode 3 final reporting and will form part of the extended dissemination planning. {R/JU a}

• The proposed updates to the E-OCVM V3.0 are intended to ensure that it provides a better support to validation emphasising key points in the recommendations. {R/JU b, d, h}

The principal action Episode 3 can take is to ensure that all the results and conclusions of the project, with respect to validation, are made widely available in an easily accessible and comprehensible form.

Episode 3


Version : 1.01

of 24


6 REFERENCES

[1] E-OCVM European Operational Concept Validation Methodology E-OCVM, 2nd Edition, March 2007

[2] SESAR D3: The ATM Target Concept, SESAR Definition Phase, DLM-0612-001-02-00a, September 2007

[3] Episode 3 Validation Lessons Learned Report, TRS 08-222764-T-D2.2- v1.0, Think Research, June 2009

[4] Episode 3 Description of Work, DOW rev 3.1/Contract amendment N°3 /FINAL, May 2009

[5] Episode 3 Consolidated Validation Strategy, D2.0-01, v1.0, December 2008

[6] Episode 3 Interim DODs, D2.2: http://www.episode3.aero/project/key-deliverables/public-deliverables/

[7] Episode 3 SESAR-Episode 3 Information Navigator: http://www.episode3.aero/project/navigator/sesar-navigator

[8] Episode 3 Influence model Lessons Learned, EP3-WP2-IM20090507-WORK-V0.1-IM lessons learned (EP3 working document)

[9] SARD Strategic Assessment of ATM R&D Results (SARD), Assessment Process and Criteria, v1.0, EUROCONTROL, 2008

[10] Episode 3 Guidelines on use of validation techniques in Episode 3. Planned deliverable

Episode 3


Version : 1.01

of 24


END OF DOCUMENT

single european sky implementation support through validation · the validation lessons learned...

Documents