PARAMESH: A PARALLEL, ADAPTIVE GRID TOOL FOR THE

SPACE SCIENCES

Kevin Olson(NASA/GSFC and GEST/Univ. of MD,

Baltimore)

Presented, AISRP PI MeetingApril, 2005

NASA/AMES Research Center

COLLABORATORSPeter MacNeice (NASA/GSFC and Drexel U.)

Joan Centrella (NASA/GSFC)Don Lamb (U. of Chicago)

other developers include:C. Mobarry, R. DeFainchtein, M. Gehmeyer, M. Bhat,

C. Packer, M. Rilee, J. VanMetre, D. Choi (NASA/GSFC)

R. Devore (NRL), M. Zingale, J. Dursi, K. Riley, A. Siegel, T. Linde,

D. Sheeler (U. Chicago)

Initial funding provided by NASA/ESTO-CT

TALK OUTLINE

I. AN OVERVIEW OF PARAMESH

II. AISRP GOALS

III. APPLICATIONS OF PARAMESH

IV. PROGRESS TOWARD GOALS

An Overview of PARAMESH

PARAMESH: what is it ?• A Package designed to ease the task of adding

parallelization and dynamic, adaptive mesh refinement (AMR) to an already existing uniform mesh, serial code

• A library of subroutines and accessible data structures • PARAMESH is the basic parallelization and AMR tool for

several important space science applications• Written in Fortran90 (NAG, Lahey, Intel, Portland Group,

HP-Compaq, IBM, SGI)• Interprocessor communication using MPI• Version 3.3 (beta) released January 2005, version 3.3

(stable) and 3.4 (beta) to be released June of 2005.• PARAMESH WEB site:

http://ct.gsfc.nasa.gov/paramesh/Users_manual/amr.html

• A subset of Berger-Oliger, block-adaptive scheme

• Computational Volume is recursively bisected into ‘blocks’, forming a tree data structure.

• Blocks are ordered and distributed to processors using a space filling curve as in particle tree codes.

• Each Block is a logically cartesian, uniform mesh of cells.

• Each cell in a block can store user specified data at cell centers, corners, edges or faces

A 2-D Block of Cells A Single Mesh Cell in 3-D

• Support for consistent fluxes and ensuring conservation for finite volume schemes

• Support for averaging data at cell edges to ensure consistent circulation integrals around cell faces

AISRP GOALS

AISRP Goals

● Extend and improve PARAMESH– Parallel I/O (HDF5, NetCDF), I/O formats for graphics

packages such as ChomboVis and Techplot.

– C Interface

– Improved support for multigrid and linear systems solvers

– Improve divergence of B control

– Improve user interface for interpolation

– Improve support for non-cartesian coordinate systems

AISRP Goals

● Effective Open Source Development– Develop coding standards– Improve automatic testing procedure– Self-documenting comments using ‘Robodoc’– Developers’ Guide– System for bug and feature request tracking– More formal release and patching strategy

● Integrate new versions of PARAMESH into actual, working, space science applications

SOME SPACE SCIENCE APPLICATIONS

USING PARAMESH

NRL AMRMHD3DR. DeVore (NRL)

P. MacNeice, K. Olson (NASA/GSFC)

• Solves the equations of MHD (DeVore, 1991)• Code for which PARAMESH was developed• Used for solar physics applications• Numerical schemes: FCT with constrained

transport for MHD and multigrid for implicit formulation of non-linear thermal diffusion

CORONAL MASS EJECTIONS(P. MacNeice et al., Drexel U. and NASA/GSFC)

QuickTime™ and aBMP decompressor

are needed to see this picture.

General Relativity, HAHNDOLJ. Centrella, D. Choi, B. Imbiriba, J. Baker, D. Fiske, &

J. Van Meter (NASA/GSFC), D. Brown, L. Lowe (N.C. State) • Solves Einstein Equations• Goal: To simulate gravitational waves resulting

from the collision of super-massive black holes in order to help interpret data from LISA mission (to be launched 2011).

• Numerical Schemes: Multigrid, Finite Difference• Major user of Columbia System

Gravitational Wave Propagation(J. Van Metre NASA/GSFC, movie by C. Henze, NASA/AMES)

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

FLASH ASTROPHYSICS CODEFLASH code team

• Fryxell et al., 2000, ApJS, 131, 273, www.flash.uchicago.edu

• Implements various CFD Schemes, MHD, Nuclear Reactions, Stellar Equations of State, and self-gravity using multigrid.

• Designed to model Astrophysical thermonuclear ‘flashes’ (X-ray bursts, Novae, and Type 1a Supernovae).

• Awarded 2000 Gordon-Bell Prize (0.25 Tflops on 6,420 processors, ASCI ‘Red’)

Simulation of an X-ray Burst due to a detonation in He atmosphere on a neutron star.

(Mike Zingale, U.C. Santa Cruz)

QuickTime™ and aVideo decompressor

are needed to see this picture.

Type Ia Supernova due to an off-center explosion in a white dwarf. (T. Plewa, A. Calder,

and D. Lamb, U. of Chicago)

QuickTime™ and aYUV420 codec decompressor

are needed to see this picture.

Other Space Science Applications● CASIM (M. Benna and P. Mahaffy at GSFC)

– MHD application for modeling comet-solar wind interaction● YDFCT (D. Odstrcil at NOAA)

– MHD application for modeling multiple interacting CME’s, integrated into CCMC

● ZeusAMR (W. Abbett et al. at U.C. Berkeley)

– Combination of Zeus MHD code and PARAMESH for modeling magnetic flux emergence from the sun

● IBEAM (D. Swesty et al. at UIUC/NCSA and SUNY-SB)

– Modern Astrophysics framework, radiation hydrodynamics for modeling gamma ray burst fireballs

● Plus others, the list continues to grow!

PROGRESS TOWARD GOALS

Progress Toward Goals

● Parallel I/O checkpointing capability added to PARAMESH using HDF5 (version 3.3)

● Capability added to write files which can be viewed using ChomboVis graphics and analysis package (version 3.3)

● C interface work begun● Divergence of B control working and in currently

released version, user interface being improved● Non-cartesian coordinates are working in tests now, will

be released in a beta version this coming Fall

Progress Toward Goals

● PARAMESH being developed as an open source project through sourceforge.net web site. We are actively seeking developers!

● Sourceforge.net used for bug and feature request tracking.

● Rododoc documentation partially complete and will form a portion of the developers’ guide

● Release policy established for patches and major releases● Developers’ guide under development

Progress Toward Goals

● HAHNDOL is using latest version of PARAMESH

● FLASH 3.0 development begun and will incorporate the latest version of PARAMESH

CONCLUSIONS

• Parallel, Adaptive Mesh Refinement has wide applicability in the space sciences

• PARAMESH provides a useful and flexible tool for adding parallel AMR to a wide variety of applications, allowing the efficient solution of ‘real’ problems

• We are making good progress toward the goals we promised for the AISRP project.