slide 1 nemovar-lefe workshop 22/3 2007. slide 1 current status of nemovar kristian mogensen
DESCRIPTION
Slide 3 NEMOVAR-LEFE Workshop 22/ Slide 3 Why do we need NEMOVAR? Anthony Weaver et al from CERFACS has been developing a variational data assimilation system for OPA version 8.2 (OPAVAR). -Incremental approach -Supports 3D-VAR (FGAT) and 4D-VAR -Has been used with the ORCA2 grid and the TDH tropical pacific area grid -Written mostly in the OPA 8.2 coding style (Fortran-77) with a few extensions Dynamic memory in a few routines -No distributed memory (MPI) parallelization, only shared memory (OpenMP) parallelization The OpenMP scaling is not spectacular Scaling to higher resolution than ORCA2 problematic due to memory constraints (>26 GB needed for ORCA1 needed) OPA 8.2 is not actively developed anymore All work within the OPA developers team are focussed on the new NEMO version of the OPA modelTRANSCRIPT
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 1
Slide 1
Current status of NEMOVAR
Kristian Mogensen
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 2
Slide 2
Outline of talk:Why do we need NEMOVAR?
- Isn’t OPAVAR good enough?Goals of the NEMOVAR projectImplementation planWhere are we nowOutstanding issues
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 3
Slide 3
Why do we need NEMOVAR? Anthony Weaver et al from CERFACS has been developing a
variational data assimilation system for OPA version 8.2 (OPAVAR).- Incremental approach
- Supports 3D-VAR (FGAT) and 4D-VAR
- Has been used with the ORCA2 grid and the TDH tropical pacific area grid
- Written mostly in the OPA 8.2 coding style (Fortran-77) with a few extensions
Dynamic memory in a few routines- No distributed memory (MPI) parallelization, only shared memory
(OpenMP) parallelization The OpenMP scaling is not spectacular Scaling to higher resolution than ORCA2 problematic due to
memory constraints (>26 GB needed for ORCA1 needed) OPA 8.2 is not actively developed anymore All work within the OPA developers team are focussed on the new
NEMO version of the OPA model
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 4
Slide 4
ECMWF/CERFACS goals for the NEMOVAR project
Short term (in ~2 years) goal- To have a 3D-VAR system based on NEMO- Support distributed memory parallelization
Possible also support shared memory parallelization- Support for different ORCA configurations
We do not worry about limited area versions of NEMO- Support for profiles and altimeter observations and SST products
It should easy to add a new observation type- Multi-incremental with different resolution in the inner loops
compared to the outer loops Not a trivial task
Long term goal- A full 4D-VAR system with all of the above properties
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 5
Slide 5
Splitting the variational problem into outer and inner loops:1. In the initial outer loop the following is done:
1. Compute misfit of observations relative to background2. Store initial trajectory for the inner loop
2. In the inner loops the following is done:1. Minimize the incremental cost function using an iterative
procedure to produce an increment
3. In the subsequent outer loop the following is done:1. Update trajectory with the increment2. Update misfit of observations
3. Update misfit of background
4. IF ( iloop < noutloop ) GOTO 2
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 6
Slide 6
Implementation plan: overviewWe have defined the following plan:
- Phase 1: Split the existing OPAVAR Fortran code into separate executables for inner and outer loops
- Phase 2: Develop an MPP implementation of the observation operators in the outer loop using NEMO
- Phase 3: Develop a hybrid system with NEMO in the outer loop and OPAVAR in the inner loop
- Phase 4: Develop an MPP implementation of the 3D-VAR with NEMO in both inner and outer loops
- Phase 5: Develop an MPP implementation of the full 4D-VAR with NEMO in both inner and outer loops
Phase 1 and 2 can done in parallelBy phase 4 we will have archived our short term
goalBy phase 5 we will have archived our long term goal
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 7
Slide 7
Implementation plan: phase 1 Split the existing OPAVAR Fortran code into
separate executables for inner and outer loops- This is needed for phase 3- Will allow scientific developments to use OPAVAR to continue
work in parallel with the NEMOVAR work We don’t want this to stop while we do the technical work Not all options of OPAVAR will be migrated to NEMOVAR
- Supports both 3D-VAR and 4D-VAR and combinations of both for multiple inner loops
- Will verify the approach of using different executables in the inner and the outer loop Similar to what is done for the IFS 4D-VAR system
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 8
Slide 8
Implementation plan: phase 2Adding observation operators to NEMO
- Import interpolation routines from OPAVAR- Distribute the observations according to NEMO domain
decomposition initially Load imbalance in the observation operators?
- Initially we will focus on the following observations: Profiles (XBT’s, Argos etc.) SLA data SST data
- Easy extendable to other observationsCan be used for other assimilation schemesThe observation operators can be used for diagnostics to
compare a model run with observations
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 9
Slide 9
Implementation plan: phase 3Develop a hybrid system with NEMO in the outer loop
and OPAVAR in the inner loop- Useful to verify the implementation of the NEMO outer loop- Since the NEMO inner loop is the “hard” part of the migration to
NEMO this system will be useful for scientific developments for some time
Writing of model trajectories in NEMO for input to the OPA inner loop will have to be coded
Reading of increments for applying the them to the non-linear states will have to be coded as well
The later 2 items are relevant for phase 4We will probably not worry too much about MPP for
phase 3Can in principle run both 3D-VAR and 4D-VAR
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 10
Slide 10
Implementation plan: phase 4Develop an MPP implementation of the 3D-VAR with
NEMO in both inner and outer loops- Parallelization of the control vector will be done based on
ECMWF IFS experiences A Fortran-90 derived type (control_vectors) is used to
defined the distributed memory layout of the control vector Fortran-90 operations like assignment and dot product are
overloaded for this type and all message parsing is done in the overloaded functions
Flexible code for non MPP developers- The minimization of the cost function is going to be based on
work done at CERFACS- Initially we will assume the same resolution in the outer and inner
loop Later we will consider different resolution in the outer and
the inner loop
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 11
Slide 11
Implementation plan: phase 5
The full 4D-VAR system is dependent on the existence of the tangent linear and adjoint of the NEMO model
We will get the status for this in the next talkWe aim to have the code of phase 4 flexible enough
that adding 4D-VAR as an option is easy once the TL/AD is available
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 12
Slide 12
Where are we nowPhase 1 (the splitting of OPAVAR) has been
completed.Phase 2 (observation operators in NEMO) is well
under way- It has been done for profile observations- Work on including altimeter is in progress- Work SST data will start shortly
Work on phase 3 (hybrid system) is just about to start
Work on phase 4 (3D-VAR NEMO) is being discussed and is like to start Q2 2007
NEMOVAR-LEFE Workshop 22/3 2007.
Slide 13
Slide 13
Some outstanding issues for discussionWe have not yet considered quality control of the
observations:- Some checks could be done between the first outer loop and the
first minimization- Other checks could be done before the first outer loop
How to best implement the change of resolutions between outer and inner loops:- Straight interpolation?- Something more advanced which better preserves the physical
quantities of the ocean fields?