1

Response from the NEMO Consortium to the NCAR RFI for new ocean modelling capability

The NEMO Consortium organizes and ensures the development and distribution of the NEMO platform. Six major European institutions are members of this Consortium: CMCC (Centro Euro-Mediterraneo sui Cambiamenti Climatici, Italy), CNRS (Centre National de la Recherche Scientifique, France), INGV (Insitituto Nazionale di Geofisica e Vulcanologia, Italy), Mercator-océan (France), the Met Office (UK) and NERC-NOC (UK). As stated in the Consortium Agreement (see http://www.nemo-ocean.eu/):

“ The purpose of this Agreement is to set up appropriate arrangements for the successful and sustainable development of the NEMO System as a well-organised, state-of-the-art ocean model code system suitable for both research and operational work.”

The NEMO Steering Committee, i.e. the executive board of NEMO Consortium, sends the following response to the CESM Request for Information on new ocean modelling capability. The first section describes NEMO as it is today and answers the questions on the State of the Art of NEMO regarding NCAR’s scientific and technical needs. The second section describes the organization of NEMO’s development and distribution, and how the overall strategy ensures appropriate answers to the consortium’s institutions needs. If NEMO was to be “short listed”, further discussions should take place to define more precisely the future organisation and resources (see “Contact persons” at the end of this document). NEMO as a possible candidate regarding the needs of ocean modelling in CESM Summary of NEMO components today NEMO (Nucleus for European Modelling of the Ocean) is a state-of-the-art modelling framework for oceanographic research, operational oceanography, seasonal to decadal forecasting and climate studies. It is a shared, reliable, and evolving system, developed by a “NEMO System Team” of experts. NEMO includes: • 5 major components:

o The blue ocean (ocean dynamics, NEMO-OPA); o The white ocean (sea-ice, NEMO-LIM); o The green ocean (biogeochemistry, NEMO-TOP); o The adaptive mesh refinement software (AGRIF); o The assimilation component NEMO-OBS, NEMO-ASM and NEMO-TAM, including observational operators and

the linear tangent and adjoint models; • An OASIS-MCT interface for coupling; • Some “reference configurations” allowing the user to easily set-up and validate the model and its applications (see

http://www.nemo-ocean.eu/Using-NEMO/Configurations); • A set of scripts and tools (including pre- and post-processing). The complete descriptions of NEMO’s components are available in the Reference Manuals on line: http://www.nemo-ocean.eu/About-NEMO/Reference-manuals. NEMO is a shared reliable and evolving system. These objectives rely on the work of the NEMO System Team. Answers from NEMO to CESM’s requests

The functionalities and options available in NEMO are described in the reference manual: “NEMO: the ocean engine”, http://www.nemo-ocean.eu/About-NEMO/Reference-manuals. As a starting point for further possible discussions, the requests from CESM are listed below, with the current status within NEMO today, and also the likely future developments in these areas.

2

3

CE

SM request

NE

MO

today N

EM

O perspectives (see D

evelopment

Strategy below)

Advanced dynam

ical core technical capabilities, including flexible vertical coordinates and resolution, advanced tracer advection schem

es, natural boundary conditions on freshw

ater and tracers, and support for non-B

oussinesq configurations;

The NEM

O num

erical core today is based on: a discretization on C grid; a m

odified leapfrog - Asselin filter tim

e stepping scheme, and

flux or vector invariant form for ocean dynam

ics. The vertical coordinates currently running are: z (w

ith partial steps), s and ztilda coordinates, or a combination of them

. The ztilda coordinate (isopycnal to high frequency m

otion) is not currently used operationally because of robustness problems.

Currently, there is no autom

atic tool available for users to configure these vertical coordinates, or for Arbitrary-Lagrangian-Eulerian

coordinates.. A

wide range of resolutions are already in use w

ithin the NEM

O com

munity, from

global paleoclimatology studies at low

spatial resolution, to coastal 1 km

effective resolution. A

dvanced tracers advection schemes: are available:

• 2

nd and 4th order

• FC

T 2nd and 4

th order (including 4th order com

pact in the vertical), •

Quickest, 3

rd order UB

S (also called UP3)

Most param

eterizations are scale aware:

• G

LS or KE are not resolution dependent

• B

iharmonic scales as x

3 but the coefficient at the equator must be user defined

Natural boundary conditions on fresh w

ater and tracers are fully implem

ented when running w

ith variable volume layers option.

Non B

oussinesq: some m

otivation has been expressed w

ithin the NEM

O consortium

, but this is not seen as high priority for now

. Some

identified work w

ould have to be done by a group m

otivated for this functionality. Future for vertical coordinates: m

ake ztilda m

ore robust. Scheduled developm

ents for dynamical core:

compensated space tim

e schemes from

2nd to

5th order

Model infrastructure and a developm

ent environm

ent that provides strong support for collaborative m

odel development w

ith the university-based C

ESM com

munity. This

includes both structured programs such as

Clim

ate Process Teams and sm

all group entrepreneurial projects;

NEM

O is distributed through a free licence.

Technically, the code is developed under strict coding rules to ensure its readability and orthogonality of the modules in order to facilitate

new developm

ents within the w

hole comm

unity. N

EMO

is part of structured programs at national and European levels.

NEM

O has a public yearly w

ork plan of developments. To prepare for the near future, the N

EMO

consortium has also set up W

orking G

roups to think out and design future development plans.

Strong support for both regional and climate

modeling applications;

NEM

O is indeed used today in a w

ide diversity of applications from regional to clim

ate modelling (see N

EMO

comm

unity below). It is

also used in a number of operational applications both w

ithin the European Copernicus M

arine Services and seasonal forecasting (eg. EC

MW

F, Met-O

ffice, Météo-France, etc).

The support is organised within the relevant groups of interest, since the N

EMO

System team

itself is a group of developers, and not a support team

(see below).

Support for a wide range of resolutions and

grids, and accompanying scale aw

are param

eterizations;

A w

ide range of applications successfully uses NEM

O, from

effective kilometric scale to global clim

ate configuration at different resolutions from

2° to 1/12°). The following processes or options are available: open boundaries, sea ice, tides, coupling to atm

osphere and grid refinem

ent with tw

o way nesting using A

GR

IF. M

ost of the available parameterizations are scale aw

are.

Global 1/36° configuration is under

development.

State-of-the-science parameterizations

(comparable to those in the present PO

P) suitable for use in coarse resolution clim

ate m

odels;

Within C

MIP6, N

EMO

is used by a number of clim

ate centres (see below) w

ith the coarse global 1° resolution, the ¼° resolution and also

the 1/12°. Prelim

inary results are seen as satisfactory (see Minutes of w

orkshop on NEM

O in C

MIP6, G

renoble, Jan 2016). Expertise on this configuration is shared w

ithin a collaborative forge project (https://forge.ipsl.jussieu.fr/shaconemo)

Science parameterizations available w

ithin NEM

O are com

parable to those in POP, except the m

ethod to compute coefficients for G

ent and M

ac William

s.

Ability to configure and run sim

pler idealized configurations for process m

odeling and educational applications;

The so-called GY

RE configuration is a built-in idealized configuration of N

EMO

used for process modelling. Several options are available

for simplified vertical air-sea interaction physics.

A N

EMO

working G

roup on “tests cases and idealized configurations was set-up 2 years ago to ensure flexible idealized configuration

building for the users. This group is now w

orking with the configuration M

anager and the AG

RIF w

orking groups.

Templates for easy set up of new

configurations w

ill be available by end 2016. W

orking group on test cases.

Com

patibility with the C

ESM sea-ice

model;

NEM

O is fully com

patible with the C

ESM sea-ice m

odel since the Met-O

ffice uses CIC

E in NEM

O for operational forecasting and

climate applications. The N

EMO

-CIC

E interface is part of the NEM

O official shared reference.

Moreover the coupling has also been done betw

een NEM

O and the C

ESM and is running at C

MC

C.

4

Ability to interface w

ith CESM

coupled data assim

ilation system;

NEM

O-O

BS contains built-in observation operator allow

ing data assimilation and m

odel/data coherent intercomparison.

NEM

O-A

SM can apply any increm

ents to the NEM

O initial state (follow

ing Incremental A

nalysis Update or by direct initialization).

NEM

O also includes its linear-tangent and adjoint m

odels for ocean dynamics, w

hich are used by the NEM

OV

AR

consortium at the M

et-O

ffice and ECM

WF.

Some centres also use the Ensem

ble Kalm

an filter and its derivatives in their operational procedures (Mercator-océan and IN

GV

).

Familiar post-processing and analysis

capabilities (akin to CESM

workflow

tools). A

ll input and outputs of NEM

O are in N

ETCD

F format, allow

ing application of all suitable existing post-processing and analysis tools. A

n IDL- based library is available (SA

XO

). W

ithin the comm

unity a few clim

ate/ocean-specific packages are also available such as CLiM

AF (at IPSL and C

NR

M in France), or

OceanA

ssess (a python based library) at the Met O

ffice

Published solutions showing a m

odel version that is com

parable to POP in the

context of Coordinated O

cean-ice Reference

Experiments (C

OR

E)

NEM

O is indeed included in all m

ajor papers on CO

RE intercom

parison results, including: •

Griffies, S. M

., et al. : Coordinated O

cean-ice Reference Experim

ents (CO

REs), O

cean Modelling, 26, 1-2, 1-46,

doi:10.1016/j.ocemod.2008.08.007

• G

okhan Danabasoglu et al. N

orth Atlantic sim

ulations in Coordinated O

cean-ice Reference Experim

ents phase II (CO

RE-II). O

cean M

odellingVolum

e 73, January 2014

Scalability and computational perform

ance on m

odern architectures N

EMO

is running with full M

PI capability, allowing efficient scalability to at least ~10 000 processes for the larger configurations. R

ecent scalability im

provements include the use of the X

IOS library for IO

and the complete suppression of global com

munications. Today,

NEM

O is used on a large variety of existing architectures.

A specific effort on H

PC has been decided for

the coming years. A

NEM

O H

PC w

orking group has been set up w

hich organises the w

ork between the System

Team and com

puter vendors to prepare for the future: both for perform

ance on new architectures (hybrid

MPI/O

penMP), and to im

prove single node perform

ance.

5

NEMO collaborative development: partnership and sustainability

NEMO community

The six institutions of the NEMO Consortium today are not only located in three different countries (France, UK, Italy), they also have different objectives and agendas: three institutions are focused on oceanic research (CNRS, NOC and CMCC), whereas the three others are focused on operational applications (Mercator, Met-Office and INGV). They all share and use a unique NEMO reference code, the development of which is driven to ensure the needs and evolution of all six partners. Indeed, whereas everyone wants the “best possible model”, this does not have exactly the same meaning for each institution. Through a “NEMO Development Strategy” document defining the perspectives for the next ten years, and through a yearly work plan of developments, NEMO’s evolution is built on consensus and driven closely to meet all needs of the Consortium members.

Aside from the six institutions directly involved in NEMO development, the NEMO community is active in a number of countries and continents:

Figure 1: Over past 12 months (1 May 2015 – 1 May 2016):

a) Number of sessions on NEMO web site by countries b) Top ten countries

Moreover, NEMO is indeed used in a wide diversity of applications covering a very large range of space and time scales, from global to coastal and from paleo-climate studies to high-resolution forecasts, including Copernicus (marine) services. A summary of projects using NEMO is available on the NEMO web site: http://www.nemo-ocean.eu/About-NEMO/Projects. Concerning CMIP6, NEMO is the ocean component of the following Earth System/climate models: CNRS (IPSL), CNRM, CERFACS, CMCC, UK Met-Office (Hadley Centre), EC-EARTH, Kiel, Hawaii, Canada, … for one or more of its components (ocean dynamics, biogeochemistry and sea-ice). Coupling is implemented through OASIS-MCT with an interface available in the NEMO reference. The CMCC-CESM model runs CESM coupled with NEMO (3.4 and 3.6 releases) in place of POP. The model is already in use, documented and results are published, for example, in the COREII intercomparison. NEMO sustainable development: organisation and results The NEMO Consortium ensures the sustainable development of NEMO. Indeed, an institution joining the Consortium commits to dedicate a minimum of one man-year per year into the NEMO System Team, which is in charge of the development and distribution of NEMO.

6

Figure 2: Organizational chart: NEMO Consortium and community

The NEMO System Team is composed of high-level experts in model development (natural and computer sciences). Their main task is to sustainably develop NEMO (i.e. develop the code appropriate for all applications, the associated documentation, validate the results, and distribute the NEMO shared reference code including all coherently merged developments). For an institution of the consortium, adding one-man year per year into the System Team brings back ~ 10 man-years per year from the other members of the NEMO System Team.

Figure 3: Organizational chart of NEMO System Team in 2016 NEMO Development Strategy

The mid- to long-term development strategy is clearly established through a written document, updated once a year if needed (see http://www.nemo-ocean.eu/About-NEMO/News/Development-Strategy-2014 , available once logged onto the NEMO web site). The NEMO Steering Committee (executive group with one representative per institution of the Consortium) endorses this document.

Some areas of model development demand a best possible answer for some open scientific questions, in order to define

an appropriate development action. The NEMO working groups are dealing with these aspects of the work. In 2016, there are Working Groups for: Wave coupling, HPC, Robustness and test cases, Configuration Manager, AGRIF(nested grids) and Configuration Manager (see http://forge.ipsl.jussieu.fr/nemo/wiki/WorkingGroups). A yearly work plan is elaborated each year, and approved by the NEMO Steering Committee. It identifies the development tasks and their PI within the NEMO System Team (for 2016: https://forge.ipsl.jussieu.fr/nemo/wiki/2016WP).

EuropeanCopernicusMarineServices

NEMOConsor3um

NEMOSystemTeam:

responsiblefordevelopmentand

distribu3onofthesharedNEMOreferencecode

NEMOConsor3umandCommunityNEMOcommunity:researchandopera3onnalapplica3ons

Morethan1000registeredusersand~200projectsworldwide:hDp://www.nemo-ocean.eu/About-NEMO/Projects

NEMODevelopersCommiDee

(advisoryboard)

NEMOSteeringCommiDee(execu3veboard)

NEMOWorkingGroups

NEMO System Team Project manager

Claire Lévy ( CNRS LOCEAN-IPSL)

Scientific Leader Gurvan Madec (CNRS LOCEAN-IPSL)

CMCC NEMO officer Dorotea iovino

Members

Silvia Mocavero Tomas Lovato

Stefania Ciliberti

INSU-CNRS NEMO officer

Simona Flavoni

Members Pierre-Antoine Bouttier

Christian Ethé Sébastien Masson

Nicolas Martin Clément Rousset

Martin Vancoppenolle

INGV NEMO officer

Emanuela Clementi

Members Damiano Del Rosso

Gelsomina Mattia

Mercator NEMO officer

Clément Bricaud

Members Jérôme Chanut

Julien Paul Romain Bourdalle-Badie

Mondher Chekki

Met Office NEMO officer Tim Graham

Members

Enda O’Dea Dave Storkey Pierre Mathiot

NERC-NOC NEMO officer

Andrew Coward

Members James Harle

George Nurser

Nom % Laboratoire Institution StatutPierre-Antoine Bouttier 50 LGGE -OSUG, Grenoble CNRS IR PermanentChristian Ethé 50 LOCEAN-IPSL, Paris CNRS IR PermanentSimona Flavoni 50 LOCEAN-IPSL, Paris CNRS IE PermanentClaire Lévy 100 LOCEAN-IPSL, Paris CNRS IRHC PermanentGurvan Madec 30 LOCEAN-IPSL, Paris CNRS DR PermanentNicolas Martin 50 LOCEAN-OSU Ecce Terra UPMC IE PermanentSébastien Masson 50 LOCEAN-IPSL, Paris CNAP PA PermanentClément Rousset 50 LOCEAN-IPSL CNRS IR PermanentMartin Vancoppenolle 30 LOCEAN-IPSL CNRS CR Permanent

L’Equipe Système NEMO de l’INSU-CNRS comprend les personnels listés ci dessous selon les pourcentages indiqués.

French Science Coordinator Julien Le Sommer (CNRS LGGE)

Autres experts (0% équipe NEMO): Laurent Debreu (INRIA Grenoble), Marie-Alice Foujols( IPSL), Rachid Benshila (LEGOS)

7

Experience in many modelling centres has shown that to develop model code suitable for a range of configurations requires substantially more effort than to develop code to be used in “research mode” by a single researcher or team. The development of NEMO follows a carefully defined process, which has been steadily refined over the 10 years since the NEMO consortium was initiated. Developments are supported by the NEMO System Team funded by the consortium members, whose job is to incorporate new scientific developments into the NEMO reference code and ensure that the code and its documentation are of robust quality. The NEMO software development process consists of: • Initial code design, reviewed by science and software experts; • Definition of specific test cases; • Development of code in a separate code branch, including validation and documentation by the developer; • Run and evaluate test case(s) including the new development, write a report to demonstrate success of development; • Full expert review of development and testing; • The NEMO Systems Team merges all code developments (on a yearly cycle) into the main NEMO reference code. This process has been tested and refined over a number of development cycles. The developments of the year, once completed, validated, and (scientifically and technically) reviewed are merged together to build the new version of NEMO during a week of work of the NEMO System Team who meet at one of the Consortium’s centres (the so-called “Merge Party”). Throughout the rest of the year, collaborative work is discussed and synchronized through videoconferences every 3 weeks. The collaborative development work within the NEMO System Team uses the up-to-date tools and processes of AGILE software development, including: • A preview phase before implementation (https://forge.ipsl.jussieu.fr/nemo/wiki/Developers/WorkingPractices), • A source versioning system (Subversion) allowing developments in parallel, each development action having its own branch

(https://forge.ipsl.jussieu.fr/nemo/browser?order=name#branches/2015), • A forge project and its associated wiki to document and share the on going work (http://forge.ipsl.jussieu.fr/nemo/wiki) Results of NEMO development process

NEMO is available to the community of projects through a unique shared reference version, from the web portal (http://www.nemo-ocean.eu/). The official release is fully validated (natural and computer sciences) by the NEMO System Team through the so-called reference configurations. Indeed, the code is distributed with a fixed list of reference configurations (http://www.nemo-ocean.eu/Using-NEMO/Configurations) for which all input and validation files are made available to the community. These reference configurations play a key role in the process. They are the simple way for new projects to set up and gain experience with NEMO. They also represent the key applications of NEMO (e.g. configurations for a global domain, a regional domain with open boundaries and tides, a benchmark…). These reference configurations are run for each new release of NEMO to ensure full validation of each new version. A new version of NEMO is released approximately every 2 years and includes all the new validated developments, together with an updated reference manual, on-line documentation, web pages and release notes. The documentation is organised as follows: • The code is documented on-line in each module/routine to describe the variable names and the computing steps • Web pages are also available on the NEMO web site, representing the User’s manual (How-to, quick start guide, IO system,

etc.) • A complete reference manual is available as a PDF file from the web site, describing all the functionalities and options

available in NEMO.

In 2015, the NEMO reference was downloaded and updated as follows, showing the activity of NEMO community: • Number of downloads of NEMO: 2568 ( 546 for 3_6_STABLE released June 2015) • Number of updates of NEMO: 40737 (6067 for 3_6_STABLE released June 2015) Publications utilising or developing NEMO are more than 200 per year (including those related to IPCC), as seen in Research Gate, Google Scholar, etc. Regarding NEMO development itself, the Geoscientific Model Development Journal has - since 2014 - a permanently opened Special Issue on NEMO to ensure the developments are well documented and visible to the community: http://www.geosci-model-dev.net/special_issue40.html. The published papers are directly related to recent developments of NEMO which are available in the shared reference (16 publications since mid 2014). Partnership and collaboration Partnership and collaboration are natural when working with a free licence software. Two levels of partnership are available:

For a scientific project taking NEMO “off the shelf” and using/developing it towards the user’s specific needs, a usual free licence partnership is suitable. The code and its development are available and feedbacks from projects are welcome, as is participation in the NEMO users meetings, etc.

8

However, NEMO is indeed a consortium model (rather than a “community” model): a unique shared reference developed following the consensus defined within the institutions of the Consortium. For an institution looking for a long-term usage and development of NEMO, entering the NEMO consortium is the best path to guarantee its present and future needs can be fulfilled. Conclusions The NEMO platform today already meets most of CESM’s requests. Moreover, its environment, development processes and agenda, allowing sustainability of long-term projects and applications, demonstrate its capabilities for the future.

However, considering the importance of NCAR’s needs as well as the extent of its scientific community, the entrance of NCAR into the NEMO consortium appears to the NEMO consortium as leading to a full partnership between all parties and the best path towards success. Contact persons:

• Sébastien Masson (Sebastien.Masson@locean-ipsl.upmc.fr) is an expert of NEMO System Team for more than 10 years now. He is more specifically an expert of the ocean in coupled models and of HPC. He is currently on sabbatical at UCLA.

• Claire Lévy (Claire.Levy@locean-ipsl.upmc.fr) is the NEMO Project Manager. She led the preparation of this response and is available to organize any further discussions between NCAR and the NEMO Consortium.