extension of the esmf for space weather cecelia deluca [email protected] [email protected] sww april 7,...
TRANSCRIPT
Extension of the ESMFfor Space Weather
Cecelia DeLuca [email protected] SWW
April 7, 2005
http://www.esmf.ucar.edu
NSIPP Seasonal Forecast
NCAR/LANL CCSM NCEP ForecastGFDL FMS Suite
MITgcmNASA GSFC GSI
Outline
• Overview• Application Integration Projects• ESMF Status• Extension to Space Weather
ESMF Goals
1. Increase scientific productivity by making model components much easier to build, combine, and exchange, and by enabling modelers to take full advantage of high-end computers.
2. Promote new scientific opportunities and services through community building and increased interoperability of codes (impacts in collaboration, code validation and tuning, teaching, migration from research to operations).
What is ESMF?1. ESMF provides tools for turning model codes into components with standard
interfaces and standard drivers2. ESMF provides data structures and common utilities that components use for
routine services such as data communications, regridding, time management and message logging
ESMF InfrastructureData Classes: Bundle, Field, Grid, Array
Utility Classes: Clock, LogErr, DELayout, Machine
ESMF SuperstructureAppDriver
Component Classes: GridComp, CplComp, State
User Code
GEOS-5
surface fvcore gravity_wave_drag
history agcm
dynamics physics
chemistry moist_processes radiation turbulence
infrared solar lake land_ice data_ocean land
vegetation catchment
coupler
coupler coupler
coupler
coupler
coupler
coupler
• Each box is an ESMF component• Every component has standard methods and interfaces to facilitate exchanges• New components can be added to the application systematically• Data in and out of components are state types that contain fields, bundles of fields • Coupling tools include regridding and redistribution methods
Application Example: GEOS-5 AGCM
ESMF Community◦ NSF National Center for Atmospheric Research ◦ NOAA Geophysical Fluid Dynamics Laboratory ◦ NOAA National Centers for Environmental Prediction◦ NASA Goddard Global Modeling and Assimilation Office◦ NASA Goddard Institute for Space Studies ◦ NASA Jet Propulsion Laboratory◦ NASA Goddard Land Information Systems project ◦ DOD Naval Research Laboratory ◦ DOD Air Force Weather Agency ◦ DOD Army Engineer Research and Development Center ◦ DOE Los Alamos National Laboratory ◦ DOE Argonne National Laboratory ◦ University of Michigan◦ Princeton University◦ Massachusetts Institute of Technology ◦ UCLA◦ Center for Ocean-Land-Atmosphere Studies ◦ Programme for Integrated Earth System Modeling (PRISM) ◦ Common Component Architecture (CCA)
ESMF Impacts
ESMF impacts a very broad set of research and operational areas that require high performance, multi-component modeling and data assimilation systems, including: • Climate prediction• Weather forecasting• Seasonal prediction• Basic Earth and planetary system research at various time and spatial scales• Emergency response• Ecosystem modeling• Battlespace simulation and integrated Earth/space forecasting• Space weather (through coordination with related space weather frameworks)• Other HPC domains, through migration of non-domain specific capabilities from ESMF – facilitated by ESMF interoperability with generic frameworks such as CCA
Infrastructure Timeline: ESMF Path
• ESMF common modeling infrastructure (~5 year)Technical foundation that allows for organized and exchangeable codes
• Set of modeling codes that are based on ESMF (~5 year)Community pool of interoperable science components with which to assemble applications
• Multi-agency and international organization (~5 year) Organizational foundation on which to establish collaborations and set priorities
• Standards for model and data description (~10 year)Prerequisite for an advanced modeling and collaboration environment that includes knowledge management
• ESMF part of an established end-to-end community-based modeling environment (~20 year)An Earth System Modeling Environment (ESME) that combines models, data, experiments, collaborative tools and information resources in a way that fosters knowledge sharing and accelerates scientific workflow
ESMF Project
Develops and manages the ESMF Product
DoD BEI Project
Generates ESMF-compliant science applications
NASA Modeling, Analysis, and Prediction Environment
Generates ESMF-compliant science applications NOAA ESMF Adoption
Project
Generates ESMF-compliant science applications
Application Integration Projects and Development Plug-Ins from the DoD, NASA, NOAA, and elsewhere interact with the ESMF Project. Areas of overlap are contributions of funds and code to the core development effort, and participation in ESMF development and management.
Other Application
Integration Projects and Development Plug-Ins
CCSM, WRF, university applications, more ….
ESMF and Application Integration Projects
NASA CAN ESMF Project DescriptionPRODUCTS: • Core framework: Software for coupling geophysical components and utilities for building
components • Applications: Deployment of the ESMF in 15 of the nation’s leading climate and weather
models, assembly of 8 new science-motivated applicationsMETRICS:
RESOURCES and TIMELINE: $9.8M from NASA over 3 years, starting FY02
Reuse Interoperability Ease of Adoption Performance
15 applications use ESMF component coupling services and 3+ utilities
8 new applications comprised of never-before coupled components
2 codes adopt ESMF with < 2% lines of code changed, or within 120 FTE-hours
No more than 10% overhead in time to solution, no degradation in scaling
NASA CAN Adoption Status0 1 2 3 4 5 6 7 8
EVA Test Atmosphere
EVA Test Coupled Atm/Ocn
GFDL B-Grid Atm
GFDL Spectral Atm
GFDL MOM4 Ocean
MITgcm Coupled Atm/Ocn
MITgcm Regional Ocn
NCAR CAM / fvCAM
NCAR CLM
LANL POP
LANL CICE
CCSM
GEOS-5 Coupled Atm/Ocn
NCEP GFS
NCEP SSI
NCEP/NCAR WRF
ODAS w ith NSIPP Ocn
MITgcm Adjoint
GSI Analysis
GEOS-5 Atmosphere
Superstructure (4 = running asESMF component)
Infrastructure (number of ESMFutilities used)
adoptiondue summer 2005
DoD Battlespace Environments Institute
To develop a DoD-wide whole-earth environment which interoperates with that from other agencies:◦ Migrate core DoD models to ESMF
• Navy (e.g., NCOM, HYCOM, SWAN, COAMPS™)• Air Force (Kinematic Solar Wind and GAIM )• Army (e.g., ADCIRC, WASH123)
◦ Development of tools and applications (e.g., extend ESMF to support unstructured grids and nesting)
◦ Coupled applications: Air/ocean, air/ocean/ice, air/ocean/groundwater, air/space-weather
◦ Thorough testing, prototyping, documentation of all components◦ $11M over six years of funding, starting FY05
NASA Modeling Analysis and Prediction Program
• ESMF-based computational environment for researchers investigating climate variability and change
• New set of climate, chemistry, and related applications coming in to ESMF• NASA plan includes support for Independent Test Team• Awards to be announced April 2005
ESMF Development Status• Overall architecture is well-defined• Components and low-level communications stable• Concurrent or sequential execution, single executable• Serial or parallel execution (mpiuni stub library)• Logically rectangular grids with regular and arbitrary distributions can be represented
and regridded• On-line parallel regridding (bilinear, 1st order conservative) completed and optimized• Other parallel methods, e.g. halo, redistribution, low-level comms implemented• Utilities such as time manager, logging, and configuration manager usable and
adding features• Virtual machine with hooks for load balancing implemented• Fortran interfaces and complete documentation, some C++ interfaces
ESMF Platform Support• IBM AIX (32 and 64 bit addressing) • SGI IRIX64 (32 and 64 bit addressing) • Compaq OSF1 (64 bit addressing) • Linux Intel (32 and 64 bit addressing, with mpich and lam) • Linux PGI (32 bit addressing, with mpich) • Linux NAG (32 bit addressing, with mpich) • Linux Absoft (32 bit addressing, with mpich) • Linux Lahey (32 bit addressing, with mpich) • Mac XLF (32 bit addressing, with lam) • Mac Absoft (32 bit addressing, with lam) • SGI Altix (64 bit addressing) • Cray X1 (64 bit addressing)
These are the platforms and options that we test on nightly. Other options are provided in the ESMF build system.
Some Metrics …• Core Implementation Team at NCAR currently has
◦ 2 FTE testers◦ 1/2 FTE performance analyst ◦ 5 FTE developers◦ 1 FTE admin/web support◦ 1 managerAnticipate sustained level of staff for next 5-6 years
• Test suite currently consists of◦ ~1250 unit tests◦ ~15 system tests◦ ~40 examples
runs every night on ~12 platforms, ~ 40 configurations• ~284 ESMF interfaces implemented, ~266 fully or partially tested, ~93% fully or
partially tested – list of untested methods on ESMF website• ~150,000 SLOC• ~1000 downloads
Open Source Development• Open source license (GPL) • Open source environment (SourceForge)• Open repositories: web-browsable CVS repositories accessible from the
ESMF website http://www.esmf.ucar.edu◦ for source code◦ for contributions (currently porting contributions and performance testing)
• Open development priorities and schedule: priorities set by users (previously during meetings and telecons, transition to new Change Review Board), web-browsable task lists
• Open communication: frequent telecons and mailing list discussions • Open testing: all tests are bundled with the ESMF distribution and can be run
by users• Open port status: results of nightly tests on many platforms are web-browsable• Open metrics: test coverage, lines of code, requirements status are updated
regularly and are web-browsable
ESMF Component Overhead
• Measures overhead of ESMF superstructure in NCEP Spectral Statistical Analysis (SSI), ~1% overall
• Run on NCAR IBM
• Runs done by JPL staff, confirmed by NCEP developers
ESMF Current Challenges• Process and techniques for software requirements analysis,
design and interface review with a VERY large distributed community (installation of commercial software (DOORS) for requirements archival and tracking, ties to testing and tasking)
• Transition to multi-agency organizational structure• Design strategy for advanced grids and regridding; science and
math hires to implement new grids and regridding methods• Continued improvement of development and test processes• Clear, complete, carefully edited documentation and training
program materials• New collaborations for technical and domain extension
Planned ESMF Extensions1. Looser couplings: support for multiple executable and Grid-enabled versions of
ESMF2. Support for representing, partitioning, communicating with, and regridding
unstructured grids and semi-structured grids3. Support for advanced I/O, including I/O support for both logically rectangular,
semi-structured, and unstructured grids, asynchronous I/O, checkpoint/restart, and multiple archival mechanisms (e.g. NetCDF, HDF5, binary, etc.)
4. Advanced support for data assimilation systems, including data structures for observational data and adjoints for ESMF methods
5. Support for nested, moving grids and adaptive grids 6. Support for regridding in three dimensions and between different coordinate
systems7. Advanced optimization and load balancing
Extension Process
• Meaningful agreement for Earth/space frameworks to collaborate and interoperate (i.e., includes resources, staged goals, timeline)
• Collective analysis and archival of requirements for space weather applications
• Collective and ongoing design, review, and testing• Participation in joint meetings and events (e.g. GEM, SWW,
ESMF Annual Meeting (July 20-22, Cambridge, MA)• Shared tools?