output of engagement with esa climate change initiative ... · auth, csic, tu/e, s&t, ucl, jrc,...

Output of Engagement with ESA Climate Change Initiative Participants

QA4ECV CGI

Technical Note

Document Reference : CGI-QA4ECV-TN_CCISEWG

Clive Farquhar

24th August 2014

Output of Engagement of ESA CCI Participants

2 of 28

Table of contents

1 Introduction _____________________________________ 3

1.1 QA4ECV Objective _________________________________________ 3

1.2 Purpose & Scope of this document ___________________________ 3

1.3 Potential ESA-EC Synergies _________________________________ 3

1.4 Acronyms and Abbreviations ________________________________ 4

2 ENGAGEMENT ___________________________________ 6

3 CLOUD ECV _____________________________________ 7

4 GHG ECV _______________________________________ 9

5 Ocean Colour ECV _______________________________ 12

6 Sea Level ECV __________________________________ 14

7 Soil Moisture ECV _______________________________ 17

8 SST ECV _______________________________________ 20

9 CM SAF ________________________________________ 22

10 Findings _______________________________________ 24

10.1 System Engineering _______________________________________ 24

10.2 Uncertainty and Error Propagation __________________________ 25

10.3 Some common conclusions ________________________________ 25

Appendix A Survey Questionnaire ____________________________ 26


3 of 28

1 Introduction

1.1 QA4ECV Objective

Quality Assurance for Essential Climate Variables (QA4ECV) is a four-year collaborative project funded

by the European Commission under the Seventh Framework Programme for Research (FP7). The project

is coordinated by KNMI and operated by 17 European partners (KNMI, IASB–BIRA, IUP–UB, MPG, ULB,

AUTH, CSIC, TU/e, S&T, UCL, JRC, EUMETSAT, Brockmann Consult, FASTOPT, Govaerts Consulting,

NPL and CGI).

The objective of QA4ECV is to develop an internationally accepted Quality Assurance framework geared

to ECVs - a common framework, generic to ECV producer organisation, whether ESA CCI, EUMETSAT

or European Commission ECVs. The framework will be applied to new multi-decadal Climate Data

Records for 6 previously un-tackled GCOS ECVs to be investigated and generated as part of the project.

1.2 Purpose & Scope of this document

CGI are tasked in the QA4ECV User Requirements work package (WP1, 1.2) to “create formal links

between this project and CCI system engineering endeavours, towards collaboratively exploring and

exploiting appropriate synergies between both”. [Description of Work].

The main objectives of the SEWG engagement was to explore the specific SE approaches to error and

uncertainty propagation and monitoring in the various CCI‟s and to determine what aspects of a proposed

uncertainty simulation / propagation tool would be most useful to each CCI team.

The main ways such information can be used in WP2 include:

1. Allows for an understanding of the approach to uncertainty through ECVs focusing on all three ECV domains. This will ensure that any proposed QA4ECV system is robust enough to be applied to any ECV

2. Can help identify commonalities in ECVs in all three domains, which could be utilised within a QA4ECV system

3. Identify particular approaches to quality assessment that can be used for uncertainty for individual processing steps, i.e. we are not reinventing the wheel.

4. Identify any potential issues in modelling uncertainty caused by particular processing steps

5. Help shape the nature of the Uncertainty Propagation tool, specifically whether it‟s more beneficial to be an abstract tool for modelling uncertainty through a processing chain or it need to be imbedded within an actual processing chain to review / exact QI from the input data and processing chain steps.

To that end, this Technical Note serves two purposes

1. Transparency to all participants

2. Feed into the WP1 of the QA4ECV User requirements. Absorbed into user requirements baseline, on record for drawing from by others – this will secure commonality of the QA4ECV framework across ECVs.

1.3 Potential ESA-EC Synergies

An appetite for potential synergy exists on EC & ESA ECVs given that quality assurance is an importance

consideration for the take up of ECV products by users. There is a potential ESA-EC synergy perhaps


4 of 28

worthy of exploration, between (i) the need for sustained take up of CCI ECV data products beyond the

end of the CCI Programme, (ii) the need for sustainability of the operational systems generating these CCI

ECVs beyond the end of the CCI programme, and (iii) the availability of a QA4ECV tool which may

support that CCI sustainability.

1.4 Acronyms and Abbreviations

The following acronyms and abbreviations have been used

Acronym Details

AIECAR Algorithm Inter-comparison and Error Characterization & Analysis Report

ATBD Algorithm Baseline Document

AUTH Aristotle University of Thessaloniki

AWST Angewandte Wissenschaft Software und Technologie GmbH (Applied Science, Software and Technology)

CCB Configuration Control Board

CCI Climate Change Initiative

CCI Climate Change Initiative

CDR Climate Data Records

CECR Comprehensive Error Characterisation Report

CGI (Ltd) Conseillers en Gestion et Informatique/Consultants to Government and Industry

CLS Collecte Localisation Satellites

CM SAF Climate Monitoring – Satellite Application Facility

CSIC Consejo Superior de Investigaciones Científicas

DLR Deutsches Zentrum für Luft- und Raumfahrt ( German Space Agency )

DOI Document Object Identifier

DSWG Data Standards Working Group

DUACS Data Unification and Altimeter Combination System

DWD Deutscher Wetterdienst (German Meteorological Service )

EC European Commission

ECSS European Cooperation for Space Standardization

ECV Essential Climate Variable

ESA European Space Agency

EUMETSAT European Organisation for the Exploitation of Meteorological Satellites

FASTOPT FastOpt GmbH

FP7 Seventh Framework Programme

GCOS Global Climate Observing System

GEOSS Global Earth Observation System of Systems

GHG Green House Gas

GRACE Gravity Recovery and Climate Experiment

GUI Graphical User Interface

http://www.awst.at/en/index.html

http://www.esa-cci.org/

http://www.esa-cci.org/

http://www.cgi.com/en/governments/space

http://www.cls.fr/en/

http://www.cmsaf.eu/EN/Home/home_node.html

http://www.dlr.de/dlr/en/desktopdefault.aspx/tabid-10002/#/DLR/Start/About

http://globwave.ifremer.fr/activities/existing-activities/item/460-ssalto---duacs

http://www.dwd.de/

http://www.esa.int/TEC/Software_engineering_and_standardisation/TECMKDUXBQE_0.html

http://www.eohandbook.com/eohb2011/climate_variables.html

http://www.esa.int/ESA

http://www.eumetsat.int/website/home/index.html

http://www.fastopt.com/

http://cordis.europa.eu/fp7/home_en.html

http://gosic.org/gcos

http://www.geoportal.org/web/guest/geo_home_stp

http://www.csr.utexas.edu/grace/


5 of 28

Acronym Details

HR-DDS High Resolution Diagnostic Data Sets

IASB–BIRA Belgian Institute for Space Aeronomy

INSPIRE Infrastructure for Spatial Information in Europe

IOCCG International Ocean Colour Coordinating Group

IUP–UB IUP Universitat Bremen

JRC Joint Research Centre

KNMI Royal Netherlands Meteorological Institute

LTDP Long Term Data Preservation

LUT Look Up Table

MAE Mean Absolute Error

NETCDF Network Common Data Format

NPL National Physical Laboratory

PCB Product Control Board

PECVPS Prototype ECV Production System

PML Plymouth Marine Laboratory

QA Quality Assurance

QA4ECV Quality Assurance for Essential Climate Variables

QA4EO Quality Assurance Framework for Earth Observation

RMSE Root Mean Square Error

RTM Radiative Transfer Model

S&T Science & Technology Corporation (Netherlands)

SE Systems Engineering

SEWG Systems Engineering Working Group

SRD System Requirements Document

SSALTO Segment Sol multi-missions dALTimetrie, d'orbitographie et de localisation précise

SSD System Specification Document

SSM Surface Soil Moisture

SSMV Surface Soil Moisture Volumetric

SST Sea Surface Temperature

TCCON Total Carbon Cloumn Observing Network

TM Technical Memo

UCD Uncertainty Characterisation Document

UCL University College London

UCR Uncertainty Characterisation Report

ULB

URD User Requirements Document

http://www.aeronomie.be/en/

http://inspire.ec.europa.eu/

http://www.ioccg.org/

http://www.iup.uni-bremen.de/deu/

https://ec.europa.eu/jrc/

http://www.knmi.nl/

http://earth.esa.int/gscb/ltdp/

http://www.unidata.ucar.edu/software/netcdf/

http://www.npl.co.uk/

http://www.pml.ac.uk/

http://www.qa4ecv.eu/

http://qa4eo.org/

http://www.stcorp.nl/

http://globwave.ifremer.fr/activities/existing-activities/item/460-ssalto---duacs

http://www.tccon.caltech.edu/

http://www.ucl.ac.uk/


6 of 28

2 ENGAGEMENT A survey questionnaire was distributed to the System Engineering Leads of the ESA CCI projects via the

CCI System Engineering Working Group (SEWG); the survey questionnaire in full is in Appendix A.

Follow-up interviews were conducted with some of the responding ECV projects thereafter. Please note

that in addition to the responses from the System Engineering Leads in 6 CCIs, DWD also provided a

response with respects to ECVs provided by the Satellite Application Facility on Climate Monitoring (CM

SAF). This is of interest as it will provide a complementary view to that from the ESA CCI team.

The engaged ECVs comprise of the following, with their feedback in the corresponding appendices.

Cloud (DWD) ; §3

GHG (DLR) ; §4

Ocean Colour (PML) ; §5

Sea Level (CLS & CGI) ; §6

Soil Moisture (AWST) ; §7

SST (Brockmann) ; §8

CM SAF (DWD); §9


7 of 28

3 CLOUD ECV

ECV : Cloud

DWD

A. System Engineering

C_1 What are your specific User Requirements on Quality Assurance?

For CCI products we do not have user requirements on quality assurance by definition, as they concern categories like temporal and spatial resolution, accuracy, precision, stability or observing cycle. These user requirements have to be translated into system requirements internally taking into account quality assurance aspects as well.

C_2 What is the System Engineering approach for your Operational System to specifically meet these User Requirements on Quality Assurance?

Within CCI a phased development process is applied. Each phase ends with a defined output and a quality check. This process comprises the following phases: (1) Requirements analysis and product specification, (2) Algorithm development, inter-comparison and selection of algorithm, (3) System Prototyping, (4) System specification, (5) System design, (6) System implementation, (7) Product generation, (8) Product validation and assessment.

C_3 Have there been useful System Engineering standards or other standards adopted for the Operational System in defining & implementing Quality Assurance?

(1) GCOS principles for ECV generation, (2) ECSS-E-ST-40C (software engineering), (3) Common standards on metadata, data models and network services as described by INSPIRE and the GEOSS Architecture and Data Committee, (4) NetCDF Climate Forecast convention used as standard for ESA Cloud CCI end product, (5) Application of DOI for identification of ECV (for the long-term data record at end of CCI phase 2)

C_4 What have been the major System Engineering challenges in this defining & implementing of Quality Assurance?

(1) Integration of third-party-software, (2) Handling and processing large amounts of data, (3) System optimisation, (4) QA in case of hardware change, (5) Considerable effort for review process and documentation

C_5 Has there been any System Engineering reuse of existing Quality Assurance assets for the Operational System definition & implementation?

Reuse of parts of the CM SAF processing system (e.g. input interfaces).

C_6 Is any part of your Operational System traceability chain shared with another ECV or existing operational system outside of the CCI Programme?

Partly shared with the CM SAF system.

B. Error & Uncertainty Propagation

C_7 What error & uncertainty computing types are currently being conducted on your processing chains, e.g neural networks, statistical models, etc.

Optimal estimation retrieval scheme for the derivation of the properties of clouds enabling rigorous error propagation and inclusion of a priori knowledge.


8 of 28

C_8 Are your ECV project members or users using an existing error & uncertainty propagation tool to simulate the propagation of error & uncertainty across a processing chain, or do you envisage such a tool being useful?

(1) A general error & uncertainty propagation tool is not used and is not envisaged to use, because this depends strongly on the applied accuracy concept, (2) Is very specific from ECV to ECV, (3) A white book on best practices in terms of error & uncertainty propagation would be preferable

C_9 Which of the following would be most useful to your ECV – (a) simulation tool, (b) tool hooked to the real processing system

(a) Not needed, is done in specific studies, (b) Not needed.

C_10 Would such a tool be more amenable to your L1->L2 processing or L2->L3 or higher?

n/a

C_11 What type of human involvement is involved in your error & uncertainty calculation?

Studies in retrospect and not part of the processing.


9 of 28

4 GHG ECV

ECV : GHG

DLR


GHG_1 What are your specific User Requirements on Quality Assurance?

The GHG-CCI products are Level 2 products. Users require validated main products (XCO2 and XCH4) with reliable error estimates per single ground pixel plus averaging kernel and a priori information. The most important requirement is that systematic errors (biases) should be as small as possible. This needs to be demonstrated by comparison with all available reference data plus additional means such as simulations. These are the main requirements. In addition to this, there are no specific user requirements related to QA. See GHG-CCI User Requirements Document

1(URDv1). During GHG-

CCI Phase 1 (2010-2013) a first data set has been generated (Climate Research Data Package 1 (CRDP#1)). An important component of QA in the future will be to compare new versions (CRDP#n+1) with previous version (CRDP#n) by direct comparison and comparisons with reference data.

GHG _2 What is the System Engineering approach for your Operational System to specifically meet these User Requirements on Quality Assurance?

The GHG-CCI approach is a scientific approach carried out by scientific experts (retrieval team) and is based on intensive comparisons with all available relevant reference data (observations and models) and, if required, additional simulated retrievals. GHG-CCI does not have a specific system engineering approach, however, a mathematical description for the implemented algorithms is available as ATBD (Algorithm Baseline Documents).

GHG _3 Have there been useful System Engineering standards or other standards adopted for the Operational System in defining & implementing Quality Assurance?

GHG-CCI aims at pushing the scientific state of the art. Methods used are scientifically state-of-the-art but specific SE QA standards have not been adopted. Naturally the algorithms are checked by their respective developers after code changes, but there is no formal procedure.

However, the source codes used in the project have been put under version control by the respective developers of the algorithms.

Also, generally standard programming languages and libraries are used as far as possible, which ensures a certain degree of code quality

The input data for the project (Level 1) are generated by the operational system of the respective mission. Thus the quality and traceability of these products is guaranteed by the operational processing system.

GHG _4 What have been the major System Engineering challenges in this defining & implementing of Quality

None.

1http://www.esa-ghg-cci.org/sites/default/files/documents/public/documents/URDv1_GHG-CCI_final.pdf

http://www.esa-ghg-cci.org/sites/default/files/documents/public/documents/URDv1_GHG-CCI_final.pdf


10 of 28

Assurance?

GHG _5 Has there been any System Engineering reuse of existing Quality Assurance assets for the Operational System definition & implementation?

The GHG-CCI system is a distributed system which in large parts already existed, when the GHG-CCI project started. This system has been improved during the project. A new definition and implementation was neither required nor recommended.

GHG _6 Is any part of your Operational System traceability chain shared with another ECV or existing operational system outside of the CCI Programme?

No.


GHG _7 What error & uncertainty computing types are currently being conducted on your processing chains, e.g neural networks, statistical models, etc.

Statistical models (computing mean differences, standard deviations, correlations, etc.).

GHG _8 Are your ECV project members or users using an existing error & uncertainty propagation tool to simulate the propagation of error & uncertainty across a processing chain, or do you envisage such a tool being useful?

Uncertainties are estimated by mapping Level 1 (spectral) errors onto Level 2 (XCO2 and XCH4) errors via the Optimal Estimation retrieval procedure. This is an integral part of the retrieval algorithms and therefore an external tool is not required.

GHG _9 Which of the following would be most useful to your ECV – (a) simulation tool, (b) tool hooked to the real processing system

The GHG-CCI algorithms are highly complex. Computing errors reliably, in particular residual systematic errors, is a challenging task which is addressed via various approaches including simulations and inter-comparisons using various reference data and methods (per single product but also for an ensemble of products, etc.). Therefore, a single tool (e.g., any of the ones listed above) is not suitable. The users would also like to get information on error correlations and reliable methods to provide this information still need to be developed.

GHG _10

Would such a tool be more amenable to your L1->L2 processing or L2->L3 or higher?

GHG-CCI focusses on L1->L2 and as mentioned above, standard tools would not be useful in any case.

GHG _11

What type of human involvement is involved in your error & uncertainty calculation?

The errors are computed automatically via the retrieval algorithm. The human involvement is needed to establish the quality of the error estimates. This includes the collection of reference data needed for comparison, preparation of all data (reference and satellite) to be able to perform various types of comparisons, and to perform the comparisons. Standard comparisons (satellite versus ground-based (TCCON), satellite versus models) but also


11 of 28

specific comparisons tailored to the scientific use of the data are carried out by scientific experts to investigate various aspects.


12 of 28

5 Ocean Colour ECV

ECV : Ocean Colour

PML


OC_1 What are your specific User Requirements on Quality Assurance?

There are two main types:

The scientific requirements tend to be taken from the GCOS requirements which usually include the percentage accuracies of the measurements.

Additionally user requirements specific to Ocean Colour were gathered as part of the project and collated into the project‟s User Requirement Document. These are typically more focused on the nature of the data in that there should be complete records, have good integrity and metadata, and have particular resolution. The users engaged were primarily to be academics and research institutes, some affiliated with GCOS, with little commercial aspects.

OC_2 What is the System Engineering approach for your Operational System to specifically meet these User Requirements on Quality Assurance?

The approach taken was to develop this during the project. With regards to the climate data products, the overall approach was to validate the processing chain ECV outputs against in situ data. WP4 was focused on this activity.

In terms of tools used, we developed some tools, using Python scripts, to undertake some of the automated validation. Other tools used included the ESA commissioned CalVaus system developed by Brockmann and the Felyx system initially developed for HR-DDS of SST.

OC_3 Have there been useful System Engineering standards or other standards adopted for the Operational System in defining & implementing Quality Assurance?

The closest thing to standards was following the IOCCG procedures for the validation of ocean products. We also followed industry standards regarding metadata layout, CF standards and for code used version control and a code review process. There was an awareness of QA4EO, but not used. LTDP was not something focused on in this project.

OC_4 What have been the major System Engineering challenges in this defining & implementing of Quality Assurance?

The main challenge was the sheer volume of the data; with the need to examine each insitu-satellite matchup pixel, as well as checks such as variables being within a range of values, overall trends plausible, etc. Significant effort was spent to optimize this search process. To do so periodically would likely be a challenge for the upcoming operational system, which might involve human intervention.

There is also some concern regarding validation as the in situ data tends to available only a few years after collection, due to needing to have physical samples of water taken to lab for testing. Some water systems (e.g. Pacific) are not that well sampled due to their remoteness and inaccessibility. Due to it's utility in operational weather forecasting, SST's validation systems are more systematic and widespread and they can measure in situ quite straightforwardly and accurately.


13 of 28

OC_5 Has there been any System Engineering reuse of existing Quality Assurance assets for the Operational System definition & implementation?

From the 4 – 5 systems built internally there has been some reuse or the code itself.

OC_6 Is any part of your Operational System traceability chain shared with another ECV or existing operational system outside of the CCI Programme?

Following DSWG (unique IDs). Full code & data versioning. There is provenance and traceability of data back to the inputs into the CCI system, but from thereon the data (EO and in situ) is outside the CCI domain.


OC_7 What error & uncertainty computing types are currently being conducted on your processing chains, e.g neural networks, statistical models, etc.

A statistical approach is used, firstly to classify each pixel as being a certain ocean type, such as open ocean, coastal, and then use a Look Up Table (LUT) to predict its uncertainty. This is a procedural process, 10s per file, with the most challenging part being to figure out the contents of the LUT.

No use of neural networks, although this could be the case in the future if particular NN-based algorithms are used.

OC_8 Are your ECV project members or users using an existing error & uncertainty propagation tool to simulate the propagation of error & uncertainty across a processing chain, or do you envisage such a tool being useful?

No we do not use a propagation tool, and would be very interested in using such a tool.

On a related note we use certain criteria to determine the feasibility of algorithm usage. See Brewin, RJW et al (2013) The Ocean Colour Climate Change Initiative: A round-robin comparison on in-water bio-optical algorithms, Remote Sensing of Environment.

OC_9 Which of the following would be most useful to your ECV – (a) simulation tool, (b) tool hooked to the real processing system

a. This would be of interest, it‟s a useful tool for overview / first step. But it might be more effort to use the simulator using real data than just using the actual processing chain.

b. We would be most interested in this. Useful to look at traceability back to the processing chain and highlight particular areas etc. use for algorithm selection and Quality Control. If integrating a real change, such as use of new algorithms, can test the overall quality and impact of the processing. Have 12 steps of their chain and each point have ability to dump out data for analysis.

OC_10 Would such a tool be more amenable to your L1->L2 processing or L2->L3 or higher?

We do produce a lot of data at L3 and follow industry standard procedures (IOCCG) for the binning of the data. Therefore it would be good if the tool can be applied to both processes.

OC_11 What type of human involvement is involved in your error & uncertainty calculation?

Very little, most effort is in deciding the number of water classes, Ensuring good representation of the uncertainty and good interpretation of the inputs used to compute the LUT.


14 of 28

6 Sea Level ECV

ECV : Sea Level

CLS & CGI


SL_1 What are your specific User Requirements on Quality Assurance?

In the context of the production of altimeter products, the user requirements are to be sure that the products delivered are of good quality, that they reach the specified level of error. They should be described with enough details for the users.

Also, the User Requirements deliverable, as well as the SoW, refer to the aimed for GCOS requirements.

SL _2

What is the System Engineering approach for your Operational System to specifically meet these User Requirements on Quality Assurance?

Phase 1 defined the Sea Level CCI operational system, via a System Requirements Document (SRD, D5.1) and System Specification Documented (SSD, D5.2, D5.3), defined by CGI & CLS. The technical approach strongly aims at system reuse and sustainability, through engineered reuse of DUACS and its use in MyOcean.

The Sea Level CCI benefits from the principles, methodology, and rules described in CLS Product Assurance Plan. The overview is the following:

System approach:

Feasibility/specification requirement engineering, Design engineering and Verification engineering

Software specification and design

Verification engineering: integration and qualification

Maintenance

Project management

Meetings (KO, PM, technical), Key points and acceptance

Deliverables (template for the project documentation)

Report (Actions and risks management)

In practice for the SL-CCI project, a list of items is checked, in relationship with the technical development, the documentation, software, tests/validation of the products. And the achievements of all workpackages (scientific developments, production of the datasets and validation) are controlled all along the project with strong interactions between the project manager, the earth observation leader and the leader of the Climate Modelling Group.

SL _3

Have there been useful System Engineering standards or other standards adopted for the Operational System in defining & implementing Quality Assurance?

The system of production of the SL-CCI products is based on the SSALTO/DUACS altimeter processing chain (supported by CNES) and has benefited from more than 10 years of previous system engineering developments.

The production system had to be adapted to provide data format homogeneous with some standards (Netcdf CF).

Phase 1 definition of the operational system used a variety of ECSS standards towards system quality. ISO 42010 was used for system specification.

SL _4

What have been the major System Engineering challenges

Specification of the parameters to be addressed for science quality assurance in the quality assurance was requested by the client. Our solution was a means for the science staff to


15 of 28

in this defining & implementing of Quality Assurance?

dynamically define parameters and their thresholds themselves. This directly links to the monitoring alert system (re-used of the MyOcean usage of DUACS).

SL _5

Has there been any System Engineering reuse of existing Quality Assurance assets for the Operational System definition & implementation?

Definition of the operational system through SRD & SSD in Phase 1 fully defines reuse of DUACS and its furnishing in context to MyOcean. A fully requirements and use case mapping, following a gap and overlap analysis of MyOcean DUACS to SLCCI, was conducted.

SL _6

Is any part of your Operational System traceability chain shared with another ECV or existing operational system outside of the CCI Programme?

Cf (3)


SL _7

What error & uncertainty computing types are currently being conducted on your processing chains, e.g neural networks, statistical models, etc.

Two different types of errors affect altimeter processing chains: (i) the sampling errors which are related with the fact that no measurement is available between two altimeter tracks (interpolation required) and (ii) the computing errors related with the processing of the data.

Indeed, the inputs of the production system are level 2 along-track altimeter measurements. They are validated, subsampled, filtered and a reduction of the inter missions biases (at global and regional scales) are performed. Orbit errors reduction and Long Wavelength Errors reduction contribute to make measurements of different altimeter more homogeneous. Data are then mapped (L4 products) via optimal interpolation, which requires the knowledge of the a priori variance of the data. The error of the mapped L4 products is thus directly related with the distance to the original altimeter track (sampling error). It is traduced by maps of formal error adjustment related with the optimal interpolation of the input data.

Uncertainty characterisation report contains more information.

Sea Level altimeter measurements are more or less linearly processed –processing of input data, editing of raw data with flagging errors, inter-annual variability processes, then main step for mapping of altimeter measurements – optimal interpolation. Some studies aim at improving this, with the same approach as numerical models which assimilate external data. Computationally, there is no feedback loop with output data.

SL _8

Are your ECV project members or users using an existing error & uncertainty propagation tool to simulate the propagation of error & uncertainty across a processing chain, or do you envisage such a tool being useful?

In order to estimate / simulate the propagation of error and uncertainty, we can use simulated measurements (model outputs) to provide pseudo observations which are associated with virtual satellites. Mapped L4 products are then computed with these pseudo observations with our processing chain and they are compared with the real original model output which is considered as the truth. The difference provides an estimation of the error propagation through the processing chain. It tells us in which extent our processing chain “modifies” the input data.


16 of 28

This is punctually performed as a “single study”. This can help for instance, to estimate the impact of the number evolution of instruments in a satellite constellation.

A dedicated error propagation simulation tool would be useful.

SL _9

Which of the following would be most useful to your ECV – (a) simulation tool, (b) tool hooked to the real processing system

Solution a) preferred. Solution b would be useful but the challenge is to remain very general and to adapt to the constraints of every ECV.

The solution which is currently used is to simulate an error (modify input parameter and/or, modify the input data), and then estimate how it affects output data with the real processing chain. Then see the difference. This is commonly used, in particular when data are interpolated, we always want to know how to reproduce the truth at best.

Such tool would be useful when working on the sea level budget approach: basically, the altimeter global mean sea level is compared with steric (Argo) plus mass (GRACE) measurements (zero is expected as output). Several dataset are used for each of these three quantities and there are some differences between them. Climate modellers are very interested in knowing which one is the best. In the case of altimeter measurements, the global mean sea level can be computed with different approaches (along-track or box-averaging) and an error and uncertainty propagation (simulation) tool would be useful to determine which method should be used. At last climate modellers would like to estimate/simulate how the choice of a given dataset affects the sea level budget (differences of the data after processing).

In terms of CPU time, it takes only a few hours to assess the impact of modifying an input parameter on a a single map of altimeter sea level anomalies but if the impact is to be estimated on the global mean sea level trend over 20 years (for instance), hundreds of weekly/monthly maps have to be computed and it thus requires a few weeks of computation.

SL _10


L2 -> L3 and even more L3 -> L4 processing

SL _11


None


17 of 28

7 Soil Moisture ECV

ECV : Soil Moisture

AWST


SM_1 What are your specific User Requirements on Quality Assurance?

URD v2.0 specifies requirements for product accuracy, precision, stability, characteristics and quality flags (cf. Table 11).

Requirements for QA are specified in the SRD v1.0 in the following sections:

Section 6.1.3: Input Data QC

Section 6.1.5: EDC Data Products Verification

Section 6.7.1: System Verification

Section 6.7.2: Document Reviews

Section 6.8.1: Quality Control Manuals, Standards and Procedures

Section 6.8.5: Operations and Maintenance Manuals

Section 6.11: Quality Requirements

Section 6.12: Reliability Requirements

Section 6.15.1: Configuration Management

Section 6.15.2: Operations and Maintenance

Section 6.15.3: Operational Quality Control

Section 6.15.5: Product Development

SM _2 What is the System Engineering approach for your Operational System to specifically meet these User Requirements on Quality Assurance?

SSD v2.0 defines the following system functions which have responsibility for implementing QA requirements (cf. Section 5.1; for allocation of requirements to system functions cf. the traceability matrix in Section 6):

F1.5 Documentation

F2.4 Algorithm Publication

F3.3 Software Testing

F3.6 Software Maintenance

F4.1 System Administration

F4.4 Platform Maintenance

F6.3 Data QC

F7.2 Product Verification

F7.4 Scientific Publishing

F7.5 Product Validation (ext.)

F8.4 Quality Assurance

As part of the human organization, SSD defines two steering and controlling boards: Product Control Board (PCB) and Configuration Control Board (CCB) (cf. Section 5.2). Their QA responsibilities are described by their participation in the main system processes (cf. Section 5.4).

SSD defines main system processes which strongly emphasize QA activities and processes (cf. Section 5.4).

SM _3 Have there been useful System

A standard for system requirements specification was adopted in SRD Section 3. It applies to the system‟s design and construction


18 of 28

Engineering standards or other standards adopted for the Operational System in defining & implementing Quality Assurance?

phases and is likely to carry over to the system‟s operational use phase.

Technical Memo TM-001 adopts a standard for the system specification (cf. SSD Section 4.1, and the introduction to Section 5). It applies to the system‟s design phase and is likely to carry over to the subsequent live-cycle phases.

TM-006 provides a discussion and decision about the adoption of standards for the system‟s operational use phase (cf. SSD Section 4.1).

SM _4 What have been the major System Engineering challenges in this defining & implementing of Quality Assurance?

The system is currently in the design live-cycle phase. Its design was aligned to QA aspects as outlined above. At this stage of development no attempt was made to implement the QA regime systematically.

Preparation of QA manuals was deferred to a later phase due to non-availability of input documents and human resources for this task (cf. TM-006).

QA functions related to data, algorithms and products were implemented on an informal basis to achieve the product generation requested by the CCI phase 1 project.

SM _5 Has there been any System Engineering reuse of existing Quality Assurance assets for the Operational System definition & implementation?

Long-term experience and the QA manuals and procedures available to the system engineers heavily influenced the system requirements and system design for QA.

SM _6 Is any part of your Operational System traceability chain shared with another ECV or existing operational system outside of the CCI Programme?

We don‟t understand what it means to share a traceability chain. The traceability matrix is part of the SSD and shared with the public. We have not received feedback from entities outside the CCISM consortium, though.

There is no dependency on other ECV projects. This was discussed in TM-005 (cf. SSD Section 4.1).

The Prototype ECV Production System (PECVPS) of CCI Phase 1 was implemented and hosted by TU Wien; it makes use of software and hardware infrastructure available at their site.


SM _7 What error & uncertainty computing types are currently being conducted on your processing chains, e.g neural networks, statistical models, etc.

Error propagation is performed up to the L2 SSM products. The current merging methods for the L3 SSMV products do not comprise error & uncertainty propagation. The Soil Moisture Prototype ECVPS covers the L2->L3 processing chain. At the current stage of development it does not foresee processes or tools for error & uncertainty propagation, and there are no pertaining requirements specified in the Soil Moisture SRD. An issue was recorded with SR-0560-FUN-PROC. For more information on error & uncertainty propagation please confer with the scientist (TU Wien).

SM _8 Are your ECV project members or users using an existing error & uncertainty propagation tool to simulate the propagation of error

n/a


19 of 28

& uncertainty across a processing chain, or do you envisage such a tool being useful?

SM _9 Which of the following would be most useful to your ECV – (a) simulation tool, (b) tool hooked to the real processing system

n/a

SM _10


n/a

SM _11


n/a


20 of 28

8 SST ECV

ECV : SST

Brockmann Consulting


SST_1 What are your specific User Requirements on Quality Assurance?

Requirements on quality in URD and no requirements on quality assurance. Error characterisation report deliverable has information on error & uncertainty addressed on processing chain.

SST _2


See (1).

SST _3


See (1).

SST _4

What have been the major System Engineering challenges in this defining & implementing of Quality Assurance?

See (1).

SST _5


See (1).

SST _6

Is any part of your Operational System traceability chain shared with another ECV or existing operational system outside of the CCI Programme?

See (1).


SST _7 What error & uncertainty computing types are currently being conducted on your processing chains, e.g neural networks, statistical models, etc.

Statistical models; Physics-based models

SST _8 Are your ECV project members or users using an existing error & uncertainty propagation tool to simulate the propagation of error & uncertainty across a

No tool is used. We have developed specific coding steps ourselves


21 of 28

processing chain, or do you envisage such a tool being useful?

SST _9 Which of the following would be most useful to your ECV – (a) simulation tool, (b) tool hooked to the real processing system

a. This sounds potentially of use ; Discussed. Both (a) and (b) have GUI allowing user-interaction to move nodes and relationship between those nodes. Difference is that with (a) the error & uncertainty logic in each node is entirely self-contained & separate from the processing chain, whereas with (b) the error & uncertainty calculation in each node is hooked to the corresponding logic / code in the actual processing chain. In discussion, (b) was deemed not useful as (i) the actual processing chain would be used anyway rather than a separate tool, (ii) no need for graphical interface, and (iii) the error & uncertainty logic in the processing chain is an integral part of the processing chain and so should not be decoupled from the error & uncertainty logic.

SST _10


Would be amendable to both, L1->L2 and L2->L3 or higher

SST _11


Parameterising the uncertainty models. Validation of uncertainty values.


22 of 28

9 CM SAF

ECVs provided by Satellite Application Facility for Climate Monitoring

DWD


SAF_1 What are your specific User Requirements on Quality Assurance?

For CM SAF products we do not have user requirements on quality assurance by definition. Product requirements are characterised by e.g. the application area, record length, temporal and spatial resolution, accuracy, precision or stability. These product requirements have to be translated into system requirements internally taking into account quality assurance aspects as well.

SAF _2


Within CM SAF scenario 4 (set-up of chains for generation of Data Records) of the EUMETSAT SAF Life-Cycle Process Model and Engineering Guidelines are applied. These guidelines are based on the V-model scaled in single phases which end with a defined output and a quality check. This process comprises the following phases:

Requirements analysis and product specification

System specification

System design

System implementation

System testing

Product generation

Product validation and assessment

Note: CM SAF is part of the operational services of the EUMETSAT Ground segment.

SAF _3


GCOS principles for ECV generation

IDEF Methodologies

ECSS-E-ST-40C (software engineering)

Common standards on metadata, data models and network services as described by INSPIRE and the GEOSS Architecture and Data Committee

NetCDF Climate Forecast convention used as standard for ESA Cloud CCI end product

Application of DOI for identification of all CM SAF CDR‟s (www.cmsaf.eu/doi)

SAF _4

What have been the major System Engineering challenges in this defining & implementing of Quality Assurance?

Integration of third-party-software

Handling and processing large amounts of data

System optimisation

QA in case of hardware change

Considerable administrative effort for review process and documentation

SAF _5


Use of parts of the operational environment together with ESA-CCI clouds

SAF Is any part of your Operational Partly shared with ESA-CCI Clouds system


23 of 28

_6 System traceability chain shared with another ECV or existing operational system outside of the CCI Programme?


SAF _7

What error & uncertainty computing types are currently being conducted on your processing chains, e.g neural networks, statistical models, etc.

Usually error propagation is applied. This is done as part of the effort to release a data set to the users.

CM SAF covers more than 10 different CDRs of several ECV‟s. The error & uncertainty characterisation is very ECV specific (e.g. for some there is a reference (standard) for others not.

For some ECVs optimal estimation retrieval schemes are applied for the derivation of the properties of clouds enabling rigorous error propagation and inclusion of a priori knowledge

SAF _8

Are your ECV project members or users using an existing error & uncertainty propagation tool to simulate the propagation of error & uncertainty across a processing chain, or do you envisage such a tool being useful?

A general error & uncertainty propagation tool is not used from our project members and is not envisaged to use, because this depends strongly on the applied accuracy concept

Is very specific from ECV to ECV

A white book on best practices in terms of error & uncertainty propagation would be preferable

SAF _9

Which of the following would be most useful to your ECV – (a) simulation tool, (b) tool hooked to the real processing system

(a). not needed, is done in specific studies ,

(b) not needed

SAF _10


n/a

SAF _11


Studies in retrospect and not part of processing


24 of 28

10 Findings

10.1 System Engineering

Regarding Q1 („What are your specific User Requirements on Quality Assurance?‟), responses comprised

– (i) GCOS requirements on data accuracy, (ii) Quality nature of the ECV products to be produced,

including – Product accuracy ; Product completeness ; Product integrity ; Product metadata ; Product

resolution ; Product Stability, (iii) Input data quality, and (iv) Quality of the process by which the ECV

products are generated. The key underlying requirements are those defined by GCOS, which are broad

requirements with some QA considerations or components. Additionally some ECVs have specific User

Requirements focused on data or product quality, which include some focused on data integrity including

Quality Assurance.

Regarding Q2 („What is the System Engineering approach for your Operational System to specifically

meet these User Requirements on Quality Assurance?‟), responses comprised – (i) Phase 1 Task 5

defines CCI operational systems via a System Requirements Document (SRD, D5.1) and System

Specification Documented (SSD, D5.2, D5.3), (ii) a preceding activity, Task 4, aims to validate the data

produced by a prototype non-operational system (Task 3) through in-situ data. During Phase II it may be

the case that for development of the operational system, each cycle of ECV product generation will

require in-situ comparison as was the case with validating the prototype data sets within Phase I Task 4,

(iii) Similarly, an operational system process will require algorithm selection akin to Task 2 in Phase I; a

mathematical description for the implemented algorithms is available as ATBDs, Algorithm Baseline

Documents.

Regarding Q3 („Have there been useful System Engineering standards or other standards adopted for the

Operational System in defining & implementing Quality Assurance?‟), responses comprised (1) GCOS

principles for ECV generation, (2) ECSS-E-ST-40C (software engineering), (3) Common standards on

metadata, data models and network services as described by INSPIRE and the GEOSS Architecture and

Data Committee, (4) NetCDF Climate Forecast convention, (5) IOCCG procedures for the validation of

ocean products (C_3), (6) ISO 42010 for description of system architecture (SL_3). (7) Awareness of

QA4EO and LTDP commonly exists, but neither explicitly adopted, with the exception of Sea Level CCI

which although not adopting the LTDP, proposes a systematic impact analysis for the long term by

adoption of the European LTDP Guidelines. (8) IDEF Methodologies. (9) Application of DOI for

identification of all CM SAF CDRs.

With regards to Q4 („What have been the major System Engineering challenges in this defining &

implementing of Quality Assurance?‟), (1) integration of third-party-software, (2) handling and processing

large amounts of data, (3) system optimisation, (4) QA in case of hardware change, (5) considerable effort

for review process and documentation, (6) sheer volume of observation data in processing comparison

with in situ data, (7) in one case in situ data is available only a few years after collection, (8) periodic

Human intervention needed for checks on range values in variables and plausibility of trends. There is

limited evidence across the ECVs that a systematic regime is in place for quality assurance, with some

teams deferring systematic implementation of a QA regime to a later project phase.

With regards to Q5 on reuse, CM-SAF is used for the Cloud ECV; similarly, a SAF is used for Sea Ice

ECV. With GHG, large parts of the current system already existed with the GHG ECV project started. The

Ocean Colour ECV has applied some reuse. Sea Level CCI has made heavy reuse of the DUACS system

used operationally as part of the MyOcean operational system. Long-term experience and the QA

manuals and procedures available to the system engineers heavily influenced the system requirements

and system design for QA of the Soil Moisture ECV. Generally, reuse is applied across all ECVs,

nevertheless there is limited evidence of systematic reuse supported by ECSS standards.


25 of 28

On Q6 („Is any part of your Operational System traceability chain shared with another ECV or existing

operational system outside of the CCI Programme?‟), traceability chains exist on all ECVs, but there is no

evidence that parts of a traceability chain are shared with other ECVs or another system with the

exception of the Cloud ECV (sharing with CM-SAF).

10.2 Uncertainty and Error Propagation

With regards to Q7 („What error & uncertainty computing types are currently being conducted on your

processing chains, e.g neural networks, statistical models, etc.‟), no particular examples were raised of

non-linear computational procedures, nor computational approaches towards emergent solutions or

behaviours such as neural-networks. Nevertheless, computational expense is generally very large.

On Q8 („Are your ECV project members or users using an existing error & uncertainty propagation tool to

simulate the propagation of error & uncertainty across a processing chain, or do you envisage such a tool

being useful?‟), two ECVs expressed usefulness in such a tool, others use the existing processing chain

and/or the error and uncertainty propagation is tied to the processing chain itself.

With regards to Q9, on the usefulness between an error / uncertainty propagation tool or simulator, there

were a wide variety of answers. Three ECVs deemed a tool or simulator not especially useful because of

its falling outside requirements, unsuitability compared to existing methods being too complex to simulate.

Of the other responses, a simulation tool was seen as the more useful in contrast to a tool hooked into the

real processing chain. A simulation tool could be used as a useful first step, and quickly ascertaining the

impact of new data on the final product. One ECV expressed the tool would be the more useful compared

to the simulator, nevertheless the simulator would be interesting also.

Regarding Q10 („Would such a tool be more amenable to your L1->L2 processing or L2->L3 or higher?‟),

those ECVs which expressed a usefulness to the tool, did not foresee significant differences between

different level processing as to the use of the tool. One ECV highlighted that modelling uncertainty when

data is binned into higher products would be of interest.

With regards to Q11 on human involvement, ECVs varied widely between not requiring any human

intervention at all to requiring considerable human involvement.

10.3 Some common conclusions

The diverse nature of the input data, ECVs generated and the methods used are reflected in the range of

answers received for each questions. As such it is difficult to draw definitive answers to each of the

questions, but some common themes do emerge.

One thing that can be noted is that those ECVs reliant on Radiative Transfer Models, for example Ocean

Colour, for determining atmospheric parameters use the inherent uncertainty information contained within

the retrieval process (i.e. Jacobian‟s) to acquire error / uncertainty information. Whilst this is useful for that

particular part of the processing, it does not address the full end-to-end processing, such as errors in the

input data or during pre-processing. As such, consideration should be given to the feasibility of the

uncertainty propagator being able to handle Jacobian or covariance matrices.

It can also be seen that there is a clear consensus that it would not be worthwhile to include actual pixel-

level processing within any developed uncertainty propagation tool; rather a higher-level view of error /

uncertainty propagation would be preferred.

Finally, there is some appetite for developing a reference guide or guidance on best practices in terms of

error and uncertainty propagation. Such a guide could also help the ECV team to use the developed tool.


26 of 28

Appendix A Survey Questionnaire Dear SEWG Colleagues,

QA4ECV („Quality Assurance for Essential Climate Variables) is a four-year collaborative project funded

by the European Commission under the Seventh Framework Programme for Research (FP7). The project

is coordinated by KNMI and operated by 17 European partners (KNMI, IASB–BIRA, IUP–UB, MPG, ULB,

AUTH, CSIC, TU/e, S&T, UCL, JRC, EUMETSAT, Brockmann Consult, FASTOPT, Govaerts Consulting,

NPL and CGI).

The aim of QA4ECV is to develop an internationally accepted Quality Assurance framework geared to

ECVs - a common framework, generic to ECV producer organisation, whether ESA CCI, EUMETSAT or

European Commission ECVs. The framework will be applied to new multi-decadal Climate Data Records

for 6 previously un-tackled GCOS ECVs to be investigated and generated as part of the project.

Given that quality assurance is an importance consideration for the take up of ECV products, there is a

potential ESA-EC synergy perhaps worthy of exploration, between (i) the need for sustained take up of

CCI ECVs beyond the end of the CCI Programme, (ii) the need for sustainability of the operational

systems generating these CCI ECVs beyond the end of the CCI programme, and (iii) the availability of a

QA4ECV tool which may support that CCI sustainability.

Your input would be much appreciated in the form of an interview, supported by the series of questions

set out below, and indeed also including any further information you may like to provide. This System

Engineering survey complements the QA4ECV DSWG survey conducted from 20th Feb 2014.

These questions include exploration towards development of a QA4ECV uncertainty simulation /

propagation tool, geared to understanding the overall quality / fitness for purpose of ECVs by modelling

the theoretical uncertainty of each processing step.

System Engineering

1. What are your specific User Requirements on Quality Assurance?

2. What is the System Engineering approach for your Operational System to specifically meet these

User Requirements on Quality Assurance?

3. Have there been useful System Engineering standards or other standards adopted for the

Operational System in defining & implementing Quality Assurance?

4. What have been the major System Engineering challenges in this defining & implementing of

Quality Assurance?

5. Has there been any System Engineering reuse of existing Quality Assurance assets for the

Operational System definition & implementation?

6. Is any part of your Operational System traceability chain shared with another ECV or existing

operational system outside of the CCI Programme?

Error & Uncertainty Propagation

7. What error & uncertainty computing types are currently being conducted on your processing

chains, e.g neural networks, statistical models, etc.


27 of 28

8. Are your ECV project members or users using an existing error & uncertainty propagation tool to

simulate the propagation of error & uncertainty across a processing chain, or do you envisage

such a tool being useful?

9. Which of the following would be most useful to your ECV

a. An error & uncertainty propagation simulation tool, represented as a series of „nodes‟. At

each node the user specifies an algorithm for the error & uncertainty calculation, and how

it must be projected to the next „node‟ of the processing chain. Each node could hook into

an external process, outside of the simulator.

b. An error & uncertainty propagation tool hooked to the real processing chain. The tool

would allow the user to observe error & uncertainty propagation across the processing

chain. The real processing chain would need to have new functionality code „injected‟ to

allow the tool to interrogate it.

10. Would such a tool be more amenable to your L1->L2 processing or L2->L3 or higher?

11. What type of human involvement is involved in your error & uncertainty calculation?