computational support for parallel/distributed amr

Computational Support for Parallel/Distributed AMR

Manish ParasharThe Applied Software Systems Laboratory

ECE/CAIP, Rutgers Universitywww.caip.rutgers.edu/~parashar/TASSL

30 September, 1999 Manish Parashar 2

Roadmap

Introduction to Berger-Oliger AMR Hierarchical Linked Lists (L. Wild) Overview of the GrACE Infrastructure GrACE Programming Model and API GrACE Design & Implementation Current Research & Future Direction

30 September, 1999 Manish Parashar 3

Cactus and GrACE

Cactus + GrACE– Transparent access to AMR via Cactus

» GrACE Infrastructure Thorn

» AMR Driver Thorn

– Status » Unigrid driver in place

» AMR driver under development

Berger-Oliger Adaptive Mesh Refinement

30 September, 1999 Manish Parashar 5

The AMR Concept

Problem: How to maximize the solution accuracy for a given problem size with limited computational resources ?

Solution: Use dynamically adaptive grids (instead of uniform grids) where the grid resolution is defined locally based on application features and solution quality.

Method: Adaptive Mesh Refinement (AMR)

30 September, 1999 Manish Parashar 6

Adaptively Griding the Application Domain

Marsha Berger et al. (http://cs.nyu.edu/faculty/berger/)

30 September, 1999 Manish Parashar 7

Adaptive Grid Structure

30 September, 1999 Manish Parashar 8

Berger-Oliger AMR: Algorithm Define adaptive grid structure Define grid functions Initialize grid functions Repeat NumTimeSteps

– if (RegridTime) Regrid at Level– Integrate at Level– if (Level+1 exists)

Integrate at Level+1Update Level from Level+1

End Repeat

30 September, 1999 Manish Parashar 9

Berger-Oliger AMR: Grid Hierarchy

Hierarchical Linked Lists (HLL)

30 September, 1999 Manish Parashar 11

HLL

AMR system devised by Lee Wild in 1996

Grid points split into nodes of size refinement-factor in each direction

Refine on nodes– Avoids clustering problems needed by

box based AMR schemes

30 September, 1999 Manish Parashar 12

Status of HLL

Lee wrote a shared memory version which was tested on various problems and showed excellent scaling properties.

It is currently being re-implemented as a standalone library with shared memory and MPI parallelism. This library will be used by a Cactus thorn to provide an AMR driver layer.

GrACE:An Framework for Distributed AMR

30 September, 1999 Manish Parashar 14

GrACE: An Overview

30 September, 1999 Manish Parashar 15

Programming Interface

Coarse grained SPMD data parallelism C++ driver

– declares and defines computational domain and application variables in terms of GrACE programming abstractions

– defines overall structure of the AMR algorithms FORTRAN/FORTRAN 90/C computational

kernels– defined on regular arrays

30 September, 1999 Manish Parashar 16

Programming Abstractions

Grid Hierarchy Abstraction– Template for the distributed adaptive grid

hierarchy Grid Function Abstraction

– Application fields defined on the adaptive grid hierarchy

Grid Geometry Abstraction– High-level tools for addressing regions in the

computational domain

30 September, 1999 Manish Parashar 17

Grid Geometry Abstractions

Coords– rank, x, y, z, ...

BBox– lb, ub, stride

BBoxList Operations

– union, intersection, cluster, refine/coarsen, difference, ...

(lbx, lby)

(ubx, uby)

dx

dy

30 September, 1999 Manish Parashar 18

GridHierarchy Abstraction

Attributes:– number of dimensions

– maximum number of levels

– specification of the computational domain

– distribution type

– refinement factor

– boundary type/width

GridHierarchy GH(Dim,GridType,MaxLevs)

30 September, 1999 Manish Parashar 19

GridFunction Abstraction

GridFunction(DIM)<T> GF(“gf”, Stencils,GH,…)

Attributes:– dimension and type

– vector?

– spatial/temporal stencils

– associated GridHierarchy

– prolongation/restriction functions

– “shadow” specification

– alignments

– ghost cells

– boundary types/updates

– interaction types

– flux registers?

– parent storage?

30 September, 1999 Manish Parashar 20

GridFunction Operations GridFunction storage for a particular time, level,

and component (and hierarchy) is managed as a Fortran 90 array object.

GF(t, l, c, Main/Shadow) <op> Scalar

GF(t, l, c, Main/Shadow) <op> GF2(….)

RedOp(GF, t, l, Main/Shadow)– <op> : =,+=,-=,/+,*=,…– RepOp: Max, Min, Sum, Product, Norm,….

30 September, 1999 Manish Parashar 21

Ghost Communications

Ghost region communications based on GridFunction stencil attribute at the specified grid level

Sync (GF, Time,Level,Main/Shadow)Sync (GF, Time, Level,Axis,Dir,Main/Shadow)Sync (GH, Time,Level,Main/Shadow)

30 September, 1999 Manish Parashar 22

Arbitrary copy (add, subtract) from Region1 to Region 2 the specified grid level.

Region-based Communications

Copy (GF, Time, Level, Reg1, Reg2, Main/Shadow)

R1

R2

30 September, 1999 Manish Parashar 23

Data-parallel forall operator

forall (gf, time, level, component) Call FORTRAN Subroutine…...end_forall

Parallel operation for all grid components at a particular time step and level.

30 September, 1999 Manish Parashar 24

Refinement & Regriding

Refine(GH, Level, BBoxList)RecomposeHierarchy(GH)

Encapsulates:– Generation of refined grids

– Redistribution

– Load-balancing

– Data-transfers

– Interaction schedules

30 September, 1999 Manish Parashar 25

Prolongation/Restriction Functions Set prolong/restrict function for each GridFunction

foreachGF(GH, GF, DIM, GFType)

SetProlongFunction(GF, Pfunc);

SetRestrictFunction(GF, Rfunc);

end_forallGF

Prolong/RestrictProlong(GF, TimeFrom, LevelFrom, TimeTo, LevelTo, Region, ….,

Main/Shadow);

Restrict(GF, TimeFrom, LevelFrom, TimeTo, LevelTo, Region, …., Main/Shadow);

30 September, 1999 Manish Parashar 26

Checkpoint/Restart/Rollback Checkpoint

Checkpoint(GH,ChkPtFile);» Each GridFunction can be individually selected or

deselected for checkpointing» Checkpoint files independent of # of processors

Restart

ComposeHierarchy(GH,ChkPtFile); Rollback

RecomposeHierarchy(GH,ChkPtFile);

30 September, 1999 Manish Parashar 27

IO Interface Initialize IO

ACEIOInit(); Select IO Type

ACEIOType(GH, IOType);» IOType := ACEIO_HDF, ACEIO_IEEEIO,..

BEGIN_COMPUTE/END_COMPUTE mark region not executed by a dedicated IO node

Do IOWrite(GF, Time, Level, Main, Double);

End IOACEIOEnd(GH);

30 September, 1999 Manish Parashar 28

Multigrid Interface Determine the number of multigrid levels available

MultiGridLevels(GH, Level, Main/Shadow); Setup the multigrid hierarchy for a GridFunction

SetUpMultiGrid(GF, Time, Level, MGLf, MGlc, Main/Shadow);SetUpMultiGrid(GF, Time, Level, Axis, MGlf, MGlc,

Main/Shadow); Do Multigrid

GF(Time, Level, Comp, MGl, Main/Shadow)….; Release multigrid hierachy

ReleaseMultiGrid(GF, Time, Level, Main/Shadow);

GrACE: Design & Implementation

30 September, 1999 Manish Parashar 30

Software Engineering in the Small: Design Principles

Separation of Concerns» policy from mechanisms» data management from solution methods» storage semantics from addressing and access» computer science from computational science from

engineering Hierarchical Abstractions

» application specific programming abstractions» semantically specialized DSM» distributed shared objects» hierarchical, extendible index space + distributed

dynamic storage

31Manish Parashar30 September, 1999

Separation of Concerns => Hierarchical Abstractions

Application

Application Components

Programming Abstractions

Dynamic Data-Management

Grid GeometryGrid Function Grid Structure

Cell Centered

Vertex Centered

Face Centered

Region

Point

Multigrid Hierarchy

Main Hierarchy

Shadow Hierarchy

Modules Kernels

Solver

Clusterer

Interpolator

Error Estimator

App. Objects

Grid

Tree

Mesh

Access

Storage

HDDA

Index Space

Method SpecificApplication Specific Adaptive Data-Mgmt

30 September, 1999 Manish Parashar 32

Hierarchical Distributed Dynamic Array (HDDA) Distributed Array

– Preserve array semantics over distribution» Reuse FORTRAN/C computational components

– Communications are transparent– Automatic partitioning & load-balancing

Hierarchical array– Each element can be a HDDA

Dynamic Array– HDDA can grow and shrink dynamically

Efficient data-management for adaptivity

33Manish Parashar30 September, 1999

Separation of Concerns => Hierarchical Abstractions

HDDA

Index Space

Name Resolution

Partitioning Expansion & Contraction

Storage

Display Objects

Data Objects Interaction Objects

Access

Consistency Communication

30 September, 1999 Manish Parashar 34

Distributed Dynamic Storage

Application Locality

Index Locality

Storage Locality

30 September, 1999 Manish Parashar 35

Partitioning Issues

Locality Parallelism Load-balance Cost

30 September, 1999 Manish Parashar 36

Composite Distribution

Inter-grid communications are local Data and task parallelism exploited Efficient load redistribution and clustering Overhead of generating & maintaining composite structure

30 September, 1999 Manish Parashar 37

IO & Visualization

30 September, 1999 Manish Parashar 38

Integrated Visualization & IO Grid Hierarchy

» Views: Multi-level, multi-resolution grid structure and connectivity, hierarchical and composite grid/mesh views, ….

» Commands: Refine, coarsen, re-distribute, read, write, checkpoint, rollback, ….

Grid Function» Views: Multi/single-resolution plots, feature extraction and

reduced models, isosurfaces, streamlines, etc….» Commands: Read, write, interpolate, checkpoint, rollback, ….

Grid Geometry» Views: Wire-frames with resolution and ownership information» Commands: Read, write, refine coarsen, merge, ….