Yosuke Fujii (JMA), Elisabeth Rémy (Mercator Ocean), and TT members

Observing System Evaluation Task Team

Current status and Overview

GOVST VIII, Bergen, Norway, Nov. 6th, 2017

Outline

OSEval-TT overview and goals

Achievements and progress over the last 8 years

Short-term future and challenges in near term

Vision for the next 10 years

1. Overview and Goals

Change of TT co-chairs and Updating the membership

Old Co-chairs(2007-2017)

Gilles Larnicol(CLS, France)

Peter Oke(CSIRO, Australia)

New Co-chairs

Yosuke Fujii(JMA/MRI, Japan)

Elisabeth Rémy(Mercator Ocean,

France)

Thank you for your leading the OSEval TT for a long time, Gilles and Peter.

We are also currently updating the membership of the TT.

Although some members may drop off from the TT due to their promotion,change of studying areas, etc., some participants of the DA & OSEval TT meetingin La Spezia last month have agreed to join the TT as new members.

Oct. 2017

OSEval-TT Missions

To provide consistent and scientifically justified requirements and feedbacks toagencies in charge of Global and Regional Ocean Observing Systems

Evaluation studiesCollect, synthetize and perform impact studies of GOOS and ROOS on GODAE systems (monitoring, forecasts and reanalysis) Methodologies: OSE, OSSE and alternative methods

Provision and dissemination of Observation

Impact Statements (OIS) based on OSEval

evidence

Aims of OSEval TT

A place to:

• Discuss methods and results of Observing System Evaluation activities

• Share experiences on developments for better use of observations.

• Transfer new OSEval methods from research to operation

• Benefit from the discussions and outcomes from the other TTs

• Give recommendations on observing system evolutions from GOV systems point of view

Outcomes from the OSEval TT:

• Support to operational agencies to adopt new methods for OSEval and better use of observation

• Guidance for observational agencies: show the benefit of observations from “users” point of view, help designing the future network / their evolution

Be identified as a partner by observation agencies and operational centers.

GOV OSEval TT is essential because it is a unique international group which aims to assess the role of existing and future observing systems in ocean monitoring and forecasting systems.

2. Achievements and Progress over the last 8 years

OSEval activities

OSEval activities have always been conducted in GOV forecasting centres.

In situ• Argo: present, deep, x2 in

WBC• Tropical moorings • TPOS• HF radars, …

Methods• OSE and OSSE in

delayed time and NRT• 4DVar Observation

Impacts and sensitivities• Design: Array Mode• DFS, FSOI

Time scales: from one month to decades (reanalysis)

Satellite• SSS• Nadir and large swath• Nb of altimeters (and

MDT)

Regions of interestGlobal, Atlantic, California, Australia,…

Observation networks

Mark Heckman

Planned

SMOS-Nino2015 ESA project

Many efforts are currently ongoing to assess the impact of SSS data on ocean and coupled forecasts and analysis.

The SMOS-Nino2015 ESA project is ongoing. The objectives are:

• Design, implement and perform OSEs of the SSS with different operational ocean forecasting systems (MetOfffice and Mercator Ocean) and different SSS products (SMOS, SMAP and Aquarius) during the strong El Niño 2015/16 event in the Tropical Pacific Ocean.

• Provide feedbacks to the operational forecasting community (GODAE Ocean View) and ocean and climate science community

- Through a dedicated web site and Observation Impact Statement Report- Organization of workshop to promote and enhance the use the SSS SMOS products.

To learn more and follow the progress:

https://www.godae-oceanview.org/projects/smos-nino15/).

Optimizing and enhancing the integrated Atlantic Ocean Observing Systems

OMOC/OSSE System Configuration

• Nature Run (NR) – free running• HYbrid Coordinate Ocean Model (HYCOM) run at high resolution (3-4 km)• Multi-year unconstrained simulations

• Forecast Model (FM) – with Data Assimilation (fraternal twin system)• The FM has to be substantially differently from the NR

• Ocean Data Assimilation (DA) procedure• In-house statistical interpolation system designed specifically for HYCOM

• Synthetic Observation Simulation Toolbox• Realistic errors added on synthetic observations (uncertainties in actual observations

vs FM eg., instrument noise, representation of different scales in FM/obs.)

Halliwell et al, 2014; Oke et al, 2015; Kourafalou et al, 2016

System fully validated (Kourafalou et al., 2016; Androulidakis et al., 2016; Halliwell et al., 2017)

Evaluation of future satellite altimetry constellation

SSH Error variance. Variance of the nature run explained by the error in percentage. (M. Benkiranand A. Bonaduce)

46 % 26 % 20 %

19 % 25 % 16 %

10 cm2 6 cm2 5 cm2

5 cm2 6 cm2 4 cm2

Assessment of the impact of adding large swath altimeter(s) with OSSE (ESA project)

Impact of the number of the nadir looking altimetry⇒ Increasing the number

generally has positive impacts.

Impact of wide-swath altimetry⇒ Substantial positive impacts

will be expected.

Still important work on treating complex correlated observation errors.

Currently supported OSEval studies

Presently in Europe, some “community” impact studies are integrated in funded projects:

• Evolution of the Atlantic in situ networks

- European AtlantOS H2020 project: includes coordinated OSSEs between Copernicus MFC in Italy, UK, France + OSEs (https://www.atlantos-h2020.eu)

- Previously: E-AIMS (European FP7 project): Evolution of the Argo network including OSE and OSSE (Italy, UK, France)

• SMOS Nino 2015 project (https://www.godae-oceanview.org/smos-nino15/)

- ESA fundings to support GODAE: SSS OSEs (France and UK)

• Large swath altimetry

- SWOT project (NASA US / CNES France): include OSSE (US/France)

- ESA large swath constellation concept: OSSE (Mercator)

Some OSEval TT members are also involved in TPOS2020 (http://tpos2020.org)

Evaluation of Regional Observing Systems

Several groups conducted Evaluation of regional observing systems

ROMS-4DVAR system (California Current System) • Contribution of the analysis increments to the cost function• Forecast System Observation Impact (FSOI)• Array-Modes

Univ. of Miami-RSMAS/NOAA-AOML joint Ocean Modeling and OSSE Center• Evaluation of glider data• Impacts on Tropical Cyclone Prediction

Univ. of New South Wales (East Australian Current)• Evaluation of HF radar, EAC array, and glider data

Array modes• Estimation of the forecast error mode structures (Californian Current,

Bay of Biscay).

La Jolla Workshop (Jan. 2014)

Evaluation of the impacts of Tropical Pacific Observing System (TPOS)

on the ocean Data Assimilation systems (Fujii et al. 2015, QJRMS)

⇒ Partly contributed by GODAE Ocean View

NOAA promised to recover TAO buoys to the level before the crisis and

followed through. However this may just be a temporary solution.

TPOS2020 project was proposed for reassessing TPOS / recommending

design changes towards a more sustainable observing system

Real-Time Multi-ORA intercomparison project is proposed for real-time

assessment of TPOS.

TAO array crisis and the La Jolla workshop

TAO array crisis (2012-2014)(Reduction of the number of data from TAO)

Apr2012

Real-Time Multi-ORA Intercomparison project

Apr2014An example figure on the web page

Led by NCEP (temperature) and ABOM (salinity)

Results of comparison are opened in near-real time via internet and NCEP Ocean

Briefing. (Updated every month.) ⇒ Some of GODAE centers participate.

Temp.: http://www.cpc.ncep.noaa.gov/products/GODAS/multiora_body.html

Sal.: http://poama.bom.gov.au/project/salt_19812010

Large spread and anti-correlation with the nb of data indicates that model

TS fields are not enough constrained by observation data.

For T. Averaged in 0-300m

Launched in 2015 according to the recommendation in La Jolla workshop.

Purpose: Propose an efficient and sustainable design of TPOS for its reorganization

First report was published in 2016 (see http://tpos2020.org)

New design of TPOS is proposed (evaluation is requested).

Moorings : Mainly 2ºN-2ºS. Enhanced sampling of the mixed later.

Argo floats: Doubling in 10ºN-10ºS

Extension of Real-Time MultiORA/sharing of information on observational data and their influence (QC flags, increments, innovation, etc.) and forcing data is suggested.

Conducted initially in ECMWF and later in JMA/MRI and NOAA/NCEP. Results were exploit in the pre-TPOS2020 meeting in La Jolla.

It includes Evaluation of analysis/reanalysis and that for seasonal predictions by coupled models in which OSE assimilation runs are used for initial conditions.

The common setting for the OSEs was discussed in TT meetings and partly referred in following OSEs.

We also seek the possibility to conduct multi-system OSEs (comparison of results among OSEs by different systems) in a similar setting but it has not been conducted.

Delayed-mode OSEs for Seasonal forecasting

TAO/TRITON Argo

REGULAR ALL ALL

TTeq 2˚S-2˚N alone ALL

noTT None ALL

noArgo ALL None

Fig. Example of delayed-mode OSE for seasonal forecasting in JMA. It shows normalized increase of RMSE from REGULAR.

Observing System Evaluation Experiments for the PIRATA mooring array in the Atlantic

RMSD S (psu) wrt PIRATA data (5 buoys) Jan-Dec 2012

OSE with the REMO ocean data assimilation system (RODAS) and

HYCOM w/without PIRATA HYCOM 1/12+RODAS

• HYCOM+RODAS tool was developed.

It assimilates SST, SLA, Argo T/S, and

only recently XBT and PIRATA data.

• OSE with PIRATA needs longer

assimilation run to better show the

impact of PIRATA data, but preliminary

results show that locally the impact is

substantial.

Temperature differencesfor July 2011

An example of near- real Time OSEImpact of ARGO

30 m

100 m

-2°C +2°C

Proposed in the TT meeting in 2011

An OSE run is conducted in near-real-time parallel to the operation assimilation run. A specific type of data which is changed every month, are withheld in the OSE run.

OSEval TT explored a possibility of conducting Near-Real-Time OSEs in multi centers mainly in first 5 years.

Near-Real-Time was conducted in UKMO (see Lea et al. 2013, QJRMS).

Although feasibility of near-real-time OSE is confirmed, the activity does not extend to other centers due to the difficulty to create incentive to use large human and computer resources.

Near Real Time OSEs

Proposed in the TT meeting in 2014 Perform one study per year; duration: 6 months One study includes 3 OSEs

- ALL: OSE that assimilates “all” observations- NONE: OSE that assimilates no observations- COSE: OSE that with-holds an agreed set of observations

Annual community OSEs

Initial time End time Data with-held Due date

7/2013 12/2013 Degraded Argo array (-20%) 4/2015

7/2014 12/2014 X altimeter(s) of the constellation (TBD) 4/2016

7/2015 12/2015 Tropical moorings 4/2017

7/2016 12/2016 X% of Argo (TBD) 4/20187/2017 12/2017 X altimeter(s) of the constellation (TBD) 4/2019

Proposed Schedule

Community OSE has not been conducted due to following issues: How can it attract international people with limited financial support?How can it communicate among international people?Who lead the processing of the results?

Lessons learned from the past

From NRT OSEs (Met Office 2011)Significant benefits from running themSignificant overhead to run them (human, supercomputer, storage)

funding cheaper (alternative ?) techniques community experiment: Shall be relevant if more than one system to do it

One month experiment probably not long enough to see full impact

From annual community OSEsIt’s hard to motivate members to work for free …

Prepare funding is favourable. Plan for publishing OIS gives members a clear target.

Although community activities, i.e., applying multiple systems to equivalent experiments, is a way of establishing system-independent, robust results, we have experienced difficulty of it this 8 years.

Review papers

Review Papers

Oke et al. 2009: Observing System Evaluations Using GODAE Systems, Oceanography Special Issue, Oceanogr. (For GODAE final)

Oke et al. 2010: Observational Requirements of GODAE Systems, Proceedings of OceanObs’09, Community White Paper

Oke et al. 2015: Assessing the impact of observations on ocean forecasts and reanalyses: Part 1, Global studies and Part 2, Regional applications, J. Oper. Oceanogr. (From GOV Symposium 2013)

Fujii et al., 2015: Evaluation of the Tropical Pacific Observing System from the ocean data assimilation perspective, QJRMS (For Pre-TPOS2020 meeting in La Jolla)

3. Short-term future and challenges in near term

Strengthen communication among TT members via internet

Update webpages (including bibliography, adding recent documents)

Use the mailing list, wiki, SNS, etc.

Use Observing Impact Statements (OIS) more effectively

Seek a better format and a more effective strategy of publishing

Use as a tool of creating motivation for TT collaboration

Collaborate with observational project (Publish OIS)

Collaborate with TPOS2020 by publishing OIS for the current status and proposed design of TPOS

Seek possibility to extent the activity of SMOS-Nino2015 (publishing OIS) to international collaboration

Seek possibility to collaborate with the international Argo project, Altimetry satellite missions, etc.

Plan the next TT meeting (Spring of 2019?)

Share information on the ongoing OSEval related activities in each centers

Discuss how to collaborate for publishing OISs

Actions of OSEval TT in next 2 years

Observing Impact Statement (OIS) published before

Change the method of publishing in a more internationally collaborated manner

Include contributions from many TT members

Open to the public from GOV web pages (?)

How to communicate on OSE/OSSE, their use and limitations:

Strong hypothesis for OSSEs

Not always well posed problem in a current version of a system -> development are required on the DA system (H, R…)

Dependence on the system -> Multi-System Evaluation is essential

-> Future sophisticated systems may change the situation

Use of alternative approaches “less costly” to assess the impact of current networks and design the future ones:

DFS/FSOI, array mode: transfer from R&D to GOV Monitoring and Forecasting Systems.

How to monitor the status of current observing system in the real-time production:

Examples: NRT OSE, Real-Time Multi-ORA, Routine monitoring of DFS, FSOI, etc.

Extend the activity to international community (currently activity is mainly in EU countries)

How to create a incentive for corporation (budget is always very limited!)

Challenges in OSEval TT

4. Vision for the next 10 years

Future Challenges

Toward ocean models resolving higher frequency and smaller scale processes:

- model including tides, HF forcings, … (link with the COSS TT)

larger spectrum of signals to observe and control

Toward coupled ocean-atmosphere-ice – BGC – ecosystem models:

- New platforms/sensors, variables and complex observation operators

Complex observation errors (non gaussian / correlated)…(link with the DA, CP, and MEAP TT)

From the observation point of view:

- High resolution satellite observations (large swath altimetry,…)

- High resolution images

- Complementarities with in situ networks

Increasing demands on impact/design studies:

- demand to increase reliability and robustness of evaluation

⇒Multi-system evaluation, New methodology

- many demands to evaluate new-type observations (without fundings)

- demand to evaluate in real-time (obs. Agencies)

Future Vision

Aim to construct a positive feedback cycle b/w observational agencies and operational centers.

Observational Agency

Operational Centers

Contribute to complete the total GODAE System

OSEval TT activitiesEffective use of Observing Impact Statement (OIS)Built up a system for Real-Time OSEval Support for better use of observations

Be identified as a partner by both the agency and centers

Data

Evaluation

Thank you

Short description and objectives of the activities started /planned for this year:

0. Coordination of the TT

1. Near-real-Time (NRT) Observing System Experiments (OSEs) with the primary objective to generate "up to date" results in order to be able to respond to observing system events. We plan to set up both standard NRT OSES and alternative methods

2. Community Observing System Experiments (OSEs).

Accomplishments of the TT this year:0. The following activities have been performed:

• Main activities has consisted to seek and find new TT co-chairs.Yosuke Fujii and Elisabeth Remy have been nominated.

• ESA SMOS studies ongoing (CLS, MetO, Mercator)• H2020 AtlantOS project ongoing. Numerous european partners are involved. •TPOS2020 project ongoing. (Yosuke Fujii is involved.)

1. No request from observational agencies to provide feedbacks relative to Observing System Events

Future plans to continue/ improve current activities:

1. Update the membership2. Preparation of DA-TT/Oseval-TT workshop with old and new

co-chairs3. Participiation of DA-TT/Oseval-TT workshop and GOV ST-VIII

meeting4. Issuing observing Impact Statement (OIS) for ESA SMOS, and

TPOS2020. Collaboration with H2020 AtlantOS5. Seek the way to collaborate with international Argo project6. Prepare for next TT meeting (spring of 2019?)

Issues/ problems:

• How to strengthen the communication with observational agency

• How to communicate on OSE/OSSE, their use and limitations

• How to evaluate the status of current observing system in real-time (NRT-OSE is expensive and can be implemented in few institute)

• How to Extend the TT activity to international (Currently activity is mostly limited in EU countries)

Name of TT: Observing System Evaluation Date: Oct. 13th, 2017

Quad chart describing the progress of annual activity in the GOV TTs

TT meetings

Nov. 2007 (Paris) : OSEval TT established;

Jun. 2009 (Toulouse) : Towards routine monitoring (promotion of routine DA-derived metrics for routine observing system assessment);

Jun. 2011 (Santa Cruz) : Joint mtg IV (promotion of class-4 inter-comparisons with an intention to be used for routine OSEval … but the final step hasn’t been accomplished)

Dec. 2014 (Toulouse) : Joint mtg with CLIVAR/GSOP – provided a forum for exchange of ideas etc in OSEval area (as community OSE/OSSEs)

Oct. 2017 (La Spezia) – Joint mtg with DA-TT. Co-chairs are changed. Potential requests of OSEval studies are confirmed. How members can collaborate for the TT is discussed.

TT meetings