© crown copyright met office atmosphere-chemistry model coupling in the uk earth system model...
TRANSCRIPT
© Crown copyright Met Office
Atmosphere-Chemistry Model Coupling in the
UK Earth System Model (UKESM)Richard Hill, April 2015
Credits: Marc Stringer, Colin Johnson, Mohit Dalvi and many more including several in this room.
© Crown copyright Met Office
Contents
1. UKESM Overview
2. Background on performance issues
3. 3D coupling and OASIS3-MCT
4. Possible approaches
5. Progress to date
Questions
© Crown copyright Met Office
UKESM – UK Earth System Model• A Joint Met Office/NERC Collaboration
• Pooling expertise and resources from various centres
• The UK community contribution to CMIP6
• Aims to include various component models:
• UM: atmosphere
• JULES: land surface
• NEMO: ocean
• CICE: sea-ice
• BICICLES: Ice shelf
• MEDUSA: ocean biogeochemistry
• UKCA: UK community atmospheric chemistry-aerosol
• Wave model – one day e.g. WAVEWATCH III?
JULESLand surf UM Atmosphere
NEMOOcean
CICESea Ice
OASIS3or
OASIS3-MCT
Current Met Office climate models
JULESLand surf UM Atmosphere
UKCAChemistry
NEMOOcean
Ice Shelf
Wave?
CICESea Ice
MEDUSAOcean Biogeochem
OASIS3-MCT
UKESM potential component models
© Crown copyright Met Office
But: Chemistry is a BIG cost!UM+UKCA profiling: UKCA increases cost by ~400%
© Crown copyright Met Office
Chemistry Costs! UKCA features heavy dynamic load imbalance
• Typically ~25%-30% of cost is dead time as work follows the sun!
– Cray XC40 task positioning might help load balance but won't get us to where we need to be.
• Cost is prohibitive for intended global resolution (N216 = 432x324x85)
• Optimisation work on UKCA code over the years has not delivered more than a few percent improvement
• We need more like ~80%
• Tinkering with the code has no chance of delivering that.
We need another way!We need another way!.
.
© Crown copyright Met Office
How about this?
• Run UKCA at a REDUCED resolution (somehow) and couple to higher resolution atmosphere?
• Maybe.... but...
– UKCA is embedded in UM atmosphere
– UKCA is tied to UM atmosphere resolution
– UKCA is not currently a viable stand-alone code!
– We'd need to exchange (potentially) lots of 3D fields somehow
Can we even do that?Can we even do that?
© Crown copyright Met Office© Crown copyright Met Office
Is 3D coupling viable? OASIS3-MCT offers hope with 3D fields.
• Two possible approaches:
• One allows us to deal with each 3D field as a single entity
• The other allows us to bundle multiple 2D fields together in order to act as a 3D field.
• Both components have the same number of vertical levels (typically 85). This appears not to be an issue as far as the science is concerned.
• i.e. We only need HORIZONTAL regridding.
© Crown copyright Met Office
3D Field Coupling
Option 1): define 3D fields as a true 3D record.
• Requires special 3D remapping weights files which repeat the same weights 85 times in the same record! (assuming no vertical interpolation)
• Requires special program and operations to generate such a file
• Such files are huge.
• Separate files required for fields of differing numbers of vertical levels
• i.e. You can't just re-use weights files in the same way that you can with 2D equivalents.
• Generation, management and debugging of such files is much more difficult than the 2D case.
• But you CAN define and put an entire 3D field in one go!
© Crown copyright Met Office
Level 8
Level 7
Level 6
Level 5
Level 4
Level 3
Level 2
Level 1
1
2
3
4
5
6
7
8
1
2
3
4
5
6
7
8
Coupling using full 3D regridding weights files
Source 3D fildSource 3D fieldTarget 3D field
Full set of 3D weights in 1 file
© Crown copyright Met Office
3D Field Coupling
Option 2): Treat each vertical level as a 2D field
• Use functional features of OASIS3-MCT to allow definition of multiple 2D fields in a single namcouple record – thereby defining a 3D entity.
• OASIS3-MCT effectively bundles multiple fields together. So the “put” operation only actually takes place once the final level is supplied to the “put” call.
• Allows use of a single 2D remapping weights file by each “2D” field at each vertical level.
• 2D weights files are much smaller and easier to create, manage and debug than anything which might constitute a 3D weights file.
• But we have to define and put lots of separate 2D fields.
© Crown copyright Met Office
Coupling using 2D regridding weights files
Source 3D fildSource 3D fieldTarget 3D field
Single level 2D weights file
© Crown copyright Met Office© Crown copyright Met Office
3D coupling is viable!OASIS3-MCT offers hope with 3D fields.
• Tests with toy model systems and iteration with OASIS3-MCT developers:
• We can successfully exchange ~200 3D fields (85 vertical levels) between N216 and N96 toy models –
• i.e. Equivalent to 17,000 2D fields!!! (atmos-ocean uses ~53 2D fields in total !)
• Cost appears acceptable – 5-10% of projected model run times, massively offset by potential reduction in elapsed time and overall resources.
© Crown copyright Met Office
3D Field Coupling
So 3D field coupling looks viable
Option 2) “multi-2D” approach looks preferable
Even if we end up having to define and exchange large numbers of fields... (~85 x option 1)... long term maintenance and usability is overriding factor.
OASIS namcouple file generation and field naming is an issue.
– Potentially 1000's of different field names
– OASIS EXPOUT gets “interesting”...but we can probably cope
So what about the component models?.....
UKCA Component Options Create a stand alone UKCA component?
But remember....
UKCA is embedded in UM atmosphere
UKCA is tied to UM atmosphere resolution
UKCA is not currently a viable stand-alone code!
© Crown copyright Met Office
UKCA Component Options Unraveling UKCA from UM is like....
Unraveling
two of these
UKCA Component Options Create a stand alone UKCA component?
Very involved
Requires restart and diagnostic handling system, MPP decomposition, launching mechanism, etc, etc. In fact a whole new independent supporting infrastructure.
ALL BEFORE YOU START WORRYING ABOUT THE ALL BEFORE YOU START WORRYING ABOUT THE COUPLINGCOUPLING
• Also requires coupling of huge numbers of 3D, 2D, 4D (and 1D) fields, mostly chemical tracers, some integers and even some logicals!
• Some good news: Tests with toy models and OASIS3-MCT Some good news: Tests with toy models and OASIS3-MCT suggest volume of coupling data would suggest volume of coupling data would notnot be a limiting factor. be a limiting factor.
JULESLand surf(High res)
UM Atmosphere(High res)
UKCAChemistry(Any res)
OASIS3-MCT
So we can’t do this… (at least not easily)
JULESLand surf(High res)
UM Atmosphere(High res)
UKCAChemistry(Low res)
OASIS3-MCT
…or this… (at least not easily)
© Crown copyright Met Office
UKCA Component Options
Alternative:
Simply run an entire “junior” UM+UKCA at the lower res coupled to a “senior” UM at the higher res.
● Pass key coupling fields from higher resolution senior component, to keep junior UM evolution in line.
● Nudging/assimilation or complete overwriting of fields.
● Simpler than creating a brand new stand-alone UKCA component.
● Requires fewer coupling fields than a true stand-alone UKCA - 3D fields only and no awkward integers, etc.
Senior – Junior coupled models proposed set-up
© Crown copyright Met Office
Senior UM
N216
Junior UM
N96
UKCA
NEMOOcean
CICESea-ice
MEDUSA
OBGC
OASIS3-MCT Coupling
© Crown copyright Met Office
UKCA Component Options
Running a “junior” UM-UKCA needs...
● Modifications to user interface to allow TWO UM components to be specified and submitted (along with existing NEMO-CICE and XIOS capability)
● Modify system control to allow two UM components to run concurrently... naming conventions of each component.
● Archiving support.
● Ability for both UM components to start from “the same” initial conditions... the UM reconfiguration and coupling exchanges at timestep zero can facilitate this.
© Crown copyright Met Office
UKCA Component Options
Running a “junior” UM-UKCA features...
●Tracer advection is performed in junior.
●Extra cost in terms of duplicated operations in junior is only about 20% of total junior cost (remember earlier performance chart.)
●Ready-made IO, decomposition, diagnostic, etc system.
●Generation of namcouple files on-the-fly is desirable due to flexible scientific configuration needs.
● Which in turn demands more code and data handling developments to enable this.
Current position
• Routinely running a prototype system coupling UM N216 with UM N96 + Chemistry.
• Currently exchanging ~10 3D fields using “option 2)” mentioned previously in the N216 -> N96 direction.
The main purpose of these is to ensure the “junior” model evolves in the same way as the “senior”.
e.g. U, V, W, temperature, moisture fields
• Currently setting up the N96 -> N216 coupling to pass back critical fields from chemistry which will affect atmos evolution.
– e.g. Aerosols, dust etc. © Crown copyright Met Office
JULESLand surf
UM Atmosphere(High res)
UKCAChemistry(Low res)
NEMOOcean
CICESea Ice
MEDUSAOcean Biogeochem
OASIS3-MCT
UKESM current prototype
JULESLand surf(Low res)
UM Atmosphere(Low res)
Current position
• Investigations concentrated on:
– Most effective interpolation methods
– Nudging/assimilation of “junior” with coupled fields versus complete overwriting?
– Likely resource requirements
• Bottom line(s)...
– Technical viability of 3D coupling is not a major concern!
– Performance of 3D coupling is not a major concern!
At least for now!.....At least for now!.....© Crown copyright Met Office
Summary
● UKESM aims to include chemistry in climate model
● Chemistry is very expensive – BUT SCALABLE!
● Aim to run chemistry at reduced resolution cf. atmos
● OASIS3-MCT facilitates this by allowing 3D field coupling
● Toy model tests indicate technical and speed viability
● Actual model developments and tests to date still indicate viability..... still relatively early stages.
● Detailed scientific tests yet to be completed – potential issues with time stepping and position of coupling exchanges.
● Only tested on IBM power7.... Cray XC40 porting soon!© Crown copyright Met Office