pdf (1.05 mb) - iopscience

Journal of Physics Conference Series

OPEN ACCESS

Concurrent parallel multiphysics coupling in theFACETS projectTo cite this article J R Cary et al 2009 J Phys Conf Ser 180 012056

View the article online for updates and enhancements

You may also likePlasma wakefield accelerationexperiments at FACET IIC Joshi E Adli W An et al

-

LOFAR FACET CALIBRATIONR J van Weeren W L Williams M JHardcastle et al

-

Rethinking Electrochemical Science andEngineering EducationFernando H Garzon Lok-kun TsuiVanessa Svihla et al

-

Recent citationsECOM A fast and accurate solver fortoroidal axisymmetric MHD equilibriaJungpyo Lee and Antoine Cerfon

-

Multi-fluid transport code modeling of time-dependent recycling in ELMy H-modeA Yu Pigarov et al

-

Summary of the ARIES Town MeetinglsquoEdge Plasma Physics and PlasmaMaterial Interactions in the Fusion PowerPlant RegimersquoMS Tillack et al

-

This content was downloaded from IP address 21010815447 on 29122021 at 0208

Concurrent Parallel Multiphysics Coupling in the

FACETS Project

J R Cary1 J Candy2 J Cobb3 R H Cohen4 T Epperly4 D J Estep5

S Krasheninnikov6 A D Malony7 D C McCune8 L McInnes9

A Pankin10 S Balay9 J A Carlsson1 M R Fahey3 R J Groebner2

A H Hakim1 S E Kruger1 M Miah1 A Pletzer1 S Shasharina1

S Vadlamani1 D Wade-Stein1 T D Rognlien4 A Morris7 S Shende7

G W Hammett8 K Indireshkumar7 A Yu Pigarov6 H Zhang9

1 Tech-X Corporation 5621 Arapahoe Avenue Suite A Boulder CO 803032 General Atomics3 Oak Ridge National Laboratory4 Lawrence Livermore National Laboratory5 Colorado State University6 University of California at San Diego7 ParaTools Inc8 Princeton Plasma Physics Laboratory9 Argonne National Laboratory10 Lehigh University

E-mail carytxcorpcom

Abstract FACETS (Framework Application for Core-Edge Transport Simulations) is now inits third year The FACETS team has developed a framework for concurrent coupling of parallelcomputational physics for use on Leadership Class Facilities (LCFs) In the course of the lastyear FACETS has tackled many of the difficult problems of moving to parallel integratedmodeling by developing algorithms for coupled systems extracting legacy applications ascomponents modifying them to run on LCFs and improving the performance of all componentsThe development of FACETS abides by rigorous engineering standards including cross platformbuild and test systems with the latter covering regression performance and visualization Inaddition FACETS has demonstrated the ability to incorporate full turbulence computations forthe highest fidelity transport computations Early indications are that the framework using suchcomputations scales to multiple tens of thousands of processors These accomplishments werea result of an interdisciplinary collaboration among computational physics computer scientistsand applied mathematicians on the team

1 Introduction

The FACETS (Framework Application for Core-Edge Transport Simulations) project [1] has thegoal of providing whole-tokamak modeling through coupling separate components for each ofthe core region edge region and wall with fully realistic sources This is a complex problem aseach component is parallel in its own right and each can be parallelized in a distinct manner

Direct simulation of the entire system is not possible due to the range of scales The spatialscales vary from the electron gyroradius (asymp 001 mm in the edge to asymp 01 mm in the core) to

SciDAC 2009 IOP PublishingJournal of Physics Conference Series 180 (2009) 012056 doi1010881742-65961801012056

ccopy 2009 IOP Publishing Ltd 1

the system size (of order several meters) ie by a factor 3times 105 The time scales vary from theelectron gyroperiod (20 ps) to the discharge duration (asymp 1000 s) ie by a factor of 6 times 1013Thus a full simulation would require the integration of 3 times 1016 (spatial resolution lengths)3

for 6times 1013 temporal resolution periods for a product of 2times 1030 With the need for of 106minus12

degrees of freedom per spatial resolution volume (100 per length for a modest fluid model easilylarger by 100 to resolve velocity space as well) and 102 floating point operations per updateof a degree of freedom for on temporal resolution period such a fundamental simulation willrequire 2times 1038minus44 floating point operations which even on petascale platforms would require2times 1023minus29 s exceeding the age of the universe by a factor of 106minus12

Given the large disparity between what is possible and what is needed progress can onlybe achieved by separating physics into different parts such that for each part valid reducedapproximations exist For example in the core of the plasma the rapid transport along fieldlines assures that the plasma parameters such as density and temperature are over long timescales constant on toroidally nested flux surfaces This reduces the transport equation to onedimension for evolution on the discharge time scale As another in the plasma edge thoughsimulations must be global they are nevertheless over a narrow region and one can use averagingmethods to reduce the time scale disparity

The above naturally translates to a software component approach which is the approachFACETS is taking It is bringing together successively more accurate and hencecomputationally demanding components to model the complete plasma device It is beingconstructed to run on Leadership Class Facilities (LCFs) to be able to use the mostcomputationally demanding components while at the same time usable on laptops for lessdemanding models To do this FACETS is constructing a C++ framework for incorporating thebest software packages and physics components In this paper we discuss the FACETS progressof the last year

2 Converting legacy applications to components suitable for Leadership Class

Facilities

Transforming legacy fusion applicationslibraries to FACETS components suitable forLeadership Class Facilities (LCFs) requires glue code to connect the legacy application tothe FACETS interface and a cross-compile build environment to produce a statically linkedexecutable We chose to target a static executable to make FACETS portable to the widestpossible collection of current and future LCFs Glue code translates FACETS interface callsto legacy application calls and performs tasks like language interoperability unit conversionsand calculating aggregate quantities from mesh data In the case of UEDGE our fluid edgecomponent we had to replace Python-based glue code with a new approach because Pythontypically uses dynamically loadable libraries mdash not a static executable

Our approach to replace UEDGErsquos Python-based glue code involved making an extensionto Forthon [2] and rewriting functions implemented in Python UEDGE uses Forthon a toolfor generating Python wrappers for Fortran parameters subroutines and functions to create aPython interface to hundreds of variables and tens of subroutines and functions We extendedForthon to generate Babel SIDL [3] files and implementation files to provide a C++ interfaceto UEDGErsquos Fortran code This approach leveraged all the preexisting Forthon interfacedescription files (v files) although we had to insert additional information in the v files inthe form of comments Thus one set of v files supports traditional UEDGE and LCF UEDGEWe wrote C++ glue code to replace code previously implemented in Python The largest partof this work involved writing the routines to input and output UEDGE data structures tofromHDF5 files

For a few of the legacy applications including UEDGE we provided new autoconf-basedconfiguration and build systems These systems were designed to support multiple types of


2

builds for example serial parallel and LCF These systems required some adjustments toperform correctly on LCF machines where the front-end nodes are different than the computenodes Although mechnical in nature making these kinds of changes is time consuming

3 Improving performance of components through algorithmic modifications

Recent work has focused on incorporating robust and scalable parallel nonlinear solvers fromthe PETSc library into UEDGE to solve the nonlinear system f(u) = 0 where u representsthe vector of unknowns We implemented complete functionality for fully implicit parallelmatrix-free Newton-Krylov solvers The use of PETSc has allowed us to overcome a majorbottleneck in the parallel implementation of UEDGE This multispecies code evolves densityand temperature profiles of hydrogen and impurity plasma ions and neutral components of thesame Strong nonlinearities exist owing to ionization and recombination and the equation setmust be well preconditioned to use the efficient Newton-Krylov solver technique Previously wecould not simultaneously advance plasma and neutrals in parallel because of a very limited blockJacobi algorithm for the parallel preconditioner Implementation of PETSc has opened an arrayof possible preconditioners ranging from Jacobi additive Schwarz and finally full LU whichnow allows us to advance both ions and neutrals together Furthermore the PETSc rdquocoloringrdquoscheme for efficiently computing the full finite-difference preconditioning Jacobian in parallelgives further substantial savings

4 Embedding parallel turbulence computations in FACETS

Figure 1 Weak scaling on the In-trepid LCF of embedded turbulentflux calculation using GYRO Thereis a 10 drop in efficiency at highprocessor counts due to decomposi-tionnetwork toplogy mismatch

Turbulence contributes a significant fraction of total trans-port in the core plasma Our recent efforts have focused onextending the FACETS core solver to incorporate fluxesfrom the five-dimensional gyrokinetic continuum turbu-lence code GYRO Embedding turbulence calculationscreates special software and algorithmic challenges Inparticular as these calculations are very expensive spe-cial domain decomposition strategies need to be devel-oped Further efficient algorithms need to be developedto minimize the time it takes to achieve statistical steady-state of the computed fluxes and ensure that the Jacobianneeded to advance the transport equations are computedwith minimum function calls

For performing the domain decomposition we havedeveloped a set of C++ classes to split a given set ofprocessors into different communicators each running aninstance of GYRO in parallel For example given 5120processors and ten flux surfaces we create ten workercommunicators each running GYRO on 512 processorsFor data transfer with the core solver an additionalcommunicator of all rank 0 processors of the worker communicators is created A distributedarray is created on this messaging communicator and is used to transfer gradients and valuesto the GYRO instances and fluxes back to the core solver This infrastructure was tested onthe Intrepid LCF for parallel scaling efficiency by running GYRO on 512 processors per fluxsurface with increasing the number of flux surfaces from 4 to 64 Timing studies show that ourinfrastructure adds a 3 second overhead per 2 hours of embedded GYRO calculations Furtherthe infrastructure scales almost linearly showing a 10 loss in efficiencies going from 32 fluxsurfaces (16384 PEs) to 62 flux surfaces (32768 PEs) This loss is attributed for failing to take


3

into account the network topology of the supercomputer We are exploring ways to improve thisby instrumenting FACETS with the TAU profiling tool

Our current effort is focused on incorporating the fluxes from the GYRO calculations intothe core solver so that we can advance the transport equations in time using turbulent fluxesFor this we are developing algorithms to compute running statistical averages of fluxes and theability to stop a GYRO instance once fluxes are in statistical steady-state Strategies to restartGYRO from perturbed previous solutions as initial conditions are being developed These twoalgorithmic advances will allow us to couple the turbulence fluxes into FACET in the core solverand create a dynamic embedded-turbulence transport solver

5 Coupling realistic particle and energy sources

The parallelized PPPL Monte Carlo package (NUBEAM) is used for computing the core sourcesfrom neutral beam injection Progress in incorporating NUBEAM into FACETS has been madeon two fronts (i) design and development of a Plasma State physics component interface inFortran and a mechanism to access this interface from a CC++ driver and (ii) development ofa parallelized driver for the NUBEAM component

The strategy for coupling NUBEAM to a CC++ driver leverages the Plasma StateComponent interface to NUBEAM that was already under development for the SWIM SCIDACThe Plasma State (PS) is a fortran-90 object incorporating a time slice representation ofa tokamak equilibrium plasma profiles such as temperatures densities and flow velocitiesand core plasma particle momentum and energy sources It also incorporates a machinedescription section containing time invariant quantities such as neutral beam and RF antennageometries The PS implementing code along with the CC++ wrapper is generated from astate specification file using a Python script

A PPPLTech-X collaboration [8] developed a reliable portable method for control ofinstantiation of fortran-90 Plasma State objects from C++ code and setget access to all stateelements This ldquoopaque handlerdquo method identifies state instances using a short array of integersthe CC++ routines thus access fortran-90 routines through an interface involving fortran-77primitive data types only The majority of inputs to NUBEAM-including all physical quantitiesthat would be required for any neutral beam model-are handled through the Plasma Stateusing this method The remainder of inputs-NUBEAM specific numerical controls such as thesizes of Monte Carlo particle lists to use in a given simulation-are handled through two CC++compatible sequenced ldquoshallowrdquo fortran-90 data types (one for initialization and a smaller onecontaining parameters that are adjustable in time) These control structures containing noallocatable elements are directly mapped to C structs

A new parallelized driver for the NUBEAM has been developed This has allowed testingscaling studies and served as a template for the C++ driver in FACETS NUBEAM is the firstcomponent in FACETS using volumetric coupling with core transport equations

6 Plasma-wall interaction module

The edge plasma and material wall are strongly coupled primarily via particle recycling impurityproduction and plasma power exhaust The handling of transient and static peak power loadscore plasma contamination with impurities plasma-facing components lifetime and hydrogenretention in walls are the critical issues affecting the design of next-step fusion devices (like ITERCTF DEMO) To address these issues and to model self-consistently the plasma-wall couplingin FACETS the Wall and Plasma-Surface Interactions (WALLPSI) module was developed [4]

The work on WALLPSI verification and validation against vast experimental data is inprogress Several laboratory experiments had showed clear saturation of retained deuteriumin beryllium and in graphite (eg pyrolitic graphite) In [4] the results of simulations ofstatic deuterium retention in graphite and beryllium at room temperatures with WALLPSI


4

were presented showing good agreement with experimental data (see Fig 2) Here deuteriumis retained via collisional production of broken bond traps followed by population of thesetraps by deuterium atoms The modeled saturated dose for deuterium in graphite is consistentwith [D][C]=04 the measured concentration of statically retained deuterium averaged overthe projectile range WALLPSI verification includes solving of simple diffusive 1-D transportproblems for hydrogen in wall material

Figure 2 Example ofWALLPSI validation againstexperimental retention data

The coupled plasma-wall modeling scheme with WALLPSIwas tested by (i) calculating the inventory build-up ofmobile chemically bonded adsorbed and trapped hydrogenin the wall as well as the nonlinear variation of hydrogenrecycling coefficient and impurity production rate in responseto the incident plasma particle and energy fluxes and (ii)simulating the spatiotemporal evolution of plasma parametersand hydrogen inventory in the wall with the coupled WALLPSIand plasma transport code EDGE1D for a range of plasmaconditions [4]

7 Development

of multicomponent visualization capabilities

Visualization is extremely valuable in providing better under-standing of scientific data generated by fusion simulations Var-ious models need to be compared with each other and validatedagainst experiments That is why FACETS developed a set of standards and tools facilitatingFACETS data visualization All FACETS components use HDF5 data format for the outputand comply to a particular standard in organizing HDF5 so that the data that is supposed to bevisualized is easily found and interpreted We call this standard VizSchema [5] [6] VizSchemais a self-contained vocabulary for describing gridded data and so allows third party applicationsto visualize and post-process FACETS data For example the following pseudo-code snippetshows the metadata indicating thet the dataset (electron temperature in the scape-off-layer)needs to be visualized needs a mesh called solmesh should be interpolated to a zone and is apart of a composite variable called tes

Dataset tesSol

Att vsType = variable

Att vsMesh = solmesh

Att vsCentering = zonal

Att vsMD = tes

Meshrsquos metadata describes its kind and provides appropriate for the kind information Forexample this snippet describes a structured mesh which is a part of a bigger mesh called sol

Group solmesh

Att vsType = mesh

Att vsKind = structured

Att vsMD = sol

The vsMD attributes shown above indicate that all tes variables will be combined into onevariable which will live on a mesh combined from the sol meshes

Based on the VizSchema standard we developed a plugin for the VisIt [7] visualizationtool This plugin allows visualization of all FACETS data An example of multicomponentvisualization (core and three edge resions) is shown in Fig 3


5

8 First physics studies

The turbulence in the core region has been the focus of intense research for over 10years resulting in a reasonable understanding of the involved processes As discussedabove FACETS leverages these efforts by integrating embedded gyrokinetic codes forcalculating core transport fluxes A key part of the goal for FACETS in the comingyear is to extend the solver techniques to enable full time-dependent simulations usingembedded gyrokinetic coupled to Monte Carlo calculation of neutral beam sources

Figure 3 Three-dimensionalvisualization of an integratedcore and edge simulation

The embedded turbulence technique relies on the turbulencecorrelation lengths being much smaller than the system scalelengths This not true for the plasma near the edge of thetokamak and as a result the a complete understanding ofthe transport in the edge region is more primitive (althoughprogress is being made in separately-funded projects) Despitethe limitations of the fluid edge model used in the inital phase ofFACETS the model provides accurate models for the paralleltransport and sources from the wall For the first core-edgephysics study we exploit these strengths to study the pedestalbuild-up of tokamak plasmas

The highest-performing tokamak plasmas are characterizedby steep edge plasma parameters known as the pedestalEventually these steep profiles become so steep that they driveinstabilities which destroy them The details of how the pedestalforms has many unknowns because it is known to depend notonly on the edge turbulence but also the amount of power leaving the core and the way in whichthe plasma interacts with the wall [Maingi09] We have begun simulations of pedestal buildupof the DIII-D experimental discharges utilizing an interpretive mode for UEDGE where plasmaprofiles are taken from experimental measurements and given sources of particles and energythe transport coefficients are computed On the closed magnetic field lines the interpretiveprocedure assumes plasma profiles are approximately constant on flux surfaces (verifiable bydirect 2D UEDGE simulation for spatially varying transport coefficients) The resulting 1Dflux-surface averaged transport equation then treats the plasma fluxes as the unknowns giventhe experimental profiles UEDGE-computed neutral particle source and input edge power andparticle flux from neutral beam fueling From the computed radial plasmas fluxes (density andionelectron energy) transport diffusivities can be determined (as gradients are known)

What is unknown experimentally is the magnitude and detailed 2D distribution of the neutralgas source from gas puffing and walldivertor recycling of ions into neutrals The physics questionto be answered by this procedure is how the pedestal region is fueled from such neutral sourcesin a setting where the transport coefficients are constrained and if this source is consistent withcore density build-up using theory-based transport there

9 Workflow for core-edge analysis

Computational workflow (in this context) refers to the process by which one starts with theinitial set of data runs the simulations to produce a final set of data analyzes the data andproduces scientific results Even for standalone single-purpose codes computational workflowissues can be cumbersome and difficult for new users who have to learn how to modify the inputparameters how to run the code on remote systems and how to analyze the data As FACETSencapsulates more codes into its framework these workflow issues increase in difficulty rapidlyand a constant re-evaluation of workflow issues is necessary to ensure that users are able to useFACETS


6

As described in the previous section we are utilizing experimental data to help constrain thesimulations By constraining what we know we are able to use simulations to help understandthe role of other difficult to measure quantities The type of analysis in this simulation is uniqueand has required development of new tools The general procedure is one of using a combinationof python and bash scripts to setup the simulation and python scripts to analyze the outputThe python setup scripts are able to handle the disparate data sources from the experimentalanalysis (multiple because most of the data is generally written to be easily used by core-onlycodes) The output scripts must collect the output from each component and then present aunified visualization for analysis This is done using either matplotlib for routine visualization oras seen in Fig 3 visit for higher-end visualization and analysis Initial studies are underway andthe workflow is continually improved as the result of feedback from users and as the simulationsprogress

10 Summary

The FACETS project has made steady progress towards providing whole device modeling forthe fusion community Physics components have been refurbished a framework for parallelcomponents is undergoing continuing development data analysis and visualization tools havebeen adapted and the project is now embarking on its first physics studies The framework hasmade tens of packages available on platforms from laptops to LCFs

Acknowledgments

Work supported by the US Department of Energy (DOE) Office of Science under grantsand contracts including DE-FC02-07ER54907 at Tech-X DE-AC02-06CH11357 at ANL DE-FC02-07ER54909 at CSU DE-AC52-07NA27344 at LLNL DE-AC05-00OR22725 at ORNLDE-FC02-07ER54910 at ParaTools DE-AC02-76CH03073 at PPPL DE-FC0207-ER54908 andDE-FG0204-ER54739 at UCSD This work used the resources of the National Energy ResearchScientific Computing Center which is supported by the DOE Office of Science under ContractNo DE-AC02-05CH11231and of the National Center for Computational Sciences at ORNLwhich is supported by the DOE Office of Science under Contract No DE-AC05-00OR22725and of the Argonne Leadership Computing Facility at Argonne National Laboratory which issupported by the Office of Science of the US Department of Energy under contract DE-AC02-06CH11357

References[1] J R Cary J Candy R H Cohen S Krasheninnikov D C McCune D J Estep J Larson A D Malony A Pankin

et al 2008 First results from core-edge parallel composition in the FACETS project J Physics Conf Series

125 012040[2] D P Grote 2009 Forthon Lawrence Berkeley National Laboratory httphifweblblgovForthon (last viewed

June 4 2009)[3] Tamara Dahlgren Thomas Epperly Gary Kumfert and James Leek Babel Userrsquos Guide CASC Lawrence

Livermore National Laboratory Livermore CA 2004[4] AYu Pigarov and SI Krasheninnikov Coupled plasma-wall modeling Journal Nuclear Materials 390-391

(2009) 192[5] VizSchema httpsicetxcorpcomtracvizschemawikiWikiStart[6] S Shasharina J R Cary S Veitzer P Hamill S Kruger M Durant and D Alexander Vizschema - visualization

interface for scientific data to be published in proceedings of Computer Graphics Visualization Computer

Vision and Image Processing 2009 Algarve Portugal June 20-22 2009[7] H Childs E S Brugger K S Bonnell J S Meredith M Miller B J Whitlock and N Max A Contract-Based

System for Large Data Visualization Proceedings of IEEE Visualization 2005 pp 190-198 MinneapolisMinnesota October 23ndash25 2005

[8] A Pletzer D McCune S Muszala S Vadlamani and S Kruger Exposing Fortran Derived Types to C andOther Languages Computing in Science and Engineering 10 (2008) 86


7

Concurrent Parallel Multiphysics Coupling in the

FACETS Project

J R Cary1 J Candy2 J Cobb3 R H Cohen4 T Epperly4 D J Estep5

S Krasheninnikov6 A D Malony7 D C McCune8 L McInnes9

A Pankin10 S Balay9 J A Carlsson1 M R Fahey3 R J Groebner2

A H Hakim1 S E Kruger1 M Miah1 A Pletzer1 S Shasharina1

S Vadlamani1 D Wade-Stein1 T D Rognlien4 A Morris7 S Shende7

G W Hammett8 K Indireshkumar7 A Yu Pigarov6 H Zhang9

1 Tech-X Corporation 5621 Arapahoe Avenue Suite A Boulder CO 803032 General Atomics3 Oak Ridge National Laboratory4 Lawrence Livermore National Laboratory5 Colorado State University6 University of California at San Diego7 ParaTools Inc8 Princeton Plasma Physics Laboratory9 Argonne National Laboratory10 Lehigh University

E-mail carytxcorpcom

Abstract FACETS (Framework Application for Core-Edge Transport Simulations) is now inits third year The FACETS team has developed a framework for concurrent coupling of parallelcomputational physics for use on Leadership Class Facilities (LCFs) In the course of the lastyear FACETS has tackled many of the difficult problems of moving to parallel integratedmodeling by developing algorithms for coupled systems extracting legacy applications ascomponents modifying them to run on LCFs and improving the performance of all componentsThe development of FACETS abides by rigorous engineering standards including cross platformbuild and test systems with the latter covering regression performance and visualization Inaddition FACETS has demonstrated the ability to incorporate full turbulence computations forthe highest fidelity transport computations Early indications are that the framework using suchcomputations scales to multiple tens of thousands of processors These accomplishments werea result of an interdisciplinary collaboration among computational physics computer scientistsand applied mathematicians on the team

1 Introduction

The FACETS (Framework Application for Core-Edge Transport Simulations) project [1] has thegoal of providing whole-tokamak modeling through coupling separate components for each ofthe core region edge region and wall with fully realistic sources This is a complex problem aseach component is parallel in its own right and each can be parallelized in a distinct manner

Direct simulation of the entire system is not possible due to the range of scales The spatialscales vary from the electron gyroradius (asymp 001 mm in the edge to asymp 01 mm in the core) to


ccopy 2009 IOP Publishing Ltd 1







Facilities





2









3












4




7 Development



Dataset tesSol




Att vsMD = tes


Group solmesh

Att vsType = mesh


Att vsMD = sol




5










6


10 Summary


Acknowledgments













7







Facilities





2









3












4




7 Development



Dataset tesSol




Att vsMD = tes


Group solmesh

Att vsType = mesh


Att vsMD = sol




5










6


10 Summary


Acknowledgments













7









3












4




7 Development



Dataset tesSol




Att vsMD = tes


Group solmesh

Att vsType = mesh


Att vsMD = sol




5










6


10 Summary


Acknowledgments













7












4




7 Development



Dataset tesSol




Att vsMD = tes


Group solmesh

Att vsType = mesh


Att vsMD = sol




5










6


10 Summary


Acknowledgments













7




7 Development



Dataset tesSol




Att vsMD = tes


Group solmesh

Att vsType = mesh


Att vsMD = sol




5










6


10 Summary


Acknowledgments













7










6


10 Summary


Acknowledgments













7


10 Summary


Acknowledgments













7

pdf (1.05 mb) - iopscience

Documents