flash workshop hamburger sternwarte , university of hamburg, feb 15 – feb 16, 2012
DESCRIPTION
The Flash Center for Computational Science. A Solution Accurate, Efficient and Stable Unsplit Staggered Mesh MHD Solver in FLASH. Dongwook Lee University of Chicago. FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012. Outline. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/1.jpg)
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
A Solution Accurate, Efficient and Stable Unsplit Staggered Mesh MHD Solver in FLASH
Dongwook Lee
University of Chicago
The Flash Center for Computational Science
![Page 2: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/2.jpg)
Outline
Split vs. unsplit formulations
Unsplit solvers in FLASH (UHD & USM) CFL stability (reduced or full?)
Reduced/Full corner-transport-upwind (CTU) for 3D
Divergence-free magnetic fields for USM-MHD constrained-transport (CT)
Verifications, convergence, performance Runtime parameters
Summary
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 3: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/3.jpg)
Part 1
Dimensionally Split vs. Unsplit???
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 4: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/4.jpg)
Part 1
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Single-mode Rayleigh-Taylor Instability Top figures:
Dimensionally split using PLM, PPM+old limiter, PPM+new limiter
high-wavenumber instabilities grow Bottom figures:
Dimensionally unsplit using PLM, PPM+old limiter, PPM+new limiter
high-wavenumber instabilities suppressed the split solvers experience high compressions and expansions in subsequent directional sweeps where there is a local high strain rate Almgren et al, ApJ, 715, 2010
![Page 5: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/5.jpg)
Part 1
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Weakly magnetized 2D field loop Gardiner and Stone 2005 (JCP); Lee and Deane 2009 (JCP)
![Page 6: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/6.jpg)
Part 1
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
8-wave split MHD scheme (Powell et al. 1999) at t=2.0
Unsplit staggered mesh MHD scheme (Lee and Deane, 2009) at t=2.0
![Page 7: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/7.jpg)
Part 1
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
What is wrong with the split formulation for MHD? In the split formulation, you cannot correctly include terms proportional to
Gardiner and Stone (2005) Dynamics of in-plane magnetic fields in x and y directions are ruined from erroneous growth of magnetic field in z direction:
![Page 8: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/8.jpg)
Part 2
Unsplit Hydro/MHD Solvers & Algorithms
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 9: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/9.jpg)
Hydro Unit in FLASH
Hydro_Unsplit
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 10: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/10.jpg)
Unsplit Staggered Mesh (USM) MHD Solver
Shock-capturing high-order Godunov Riemann solver (Lee & Deane, JCP, 2009; Lee 2012, to be submitted)
Finite volume method New data reconstruction-evolution algorithm for high-order accuracy Adaptive mesh refinement, uniform grid 1st order Godunov, 2nd order MUSCL-Hancock, 3rd order PPM, 5th Order WENO Approximate Riemann solvers: Roe, HLL, HLLC, HLLD, Marquina, modified
Marquina, Local Lax-Friedrichs Monotonicity preserving upwind PPM slope limiter for MHD (Lee, 2010,
Astronum) Divergence of magnetic fields is numerically controlled on a staggered grid,
using a constrained transport (CT) method (Evans & Hawley, 1998) Wide ranges of plasma flows Full Courant stability limit (CFL ~ 1 for 3D)
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 11: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/11.jpg)
Unsplit Formulations
Take a deep breath!
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 12: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/12.jpg)
MHD Governing Equations
MHD system of equations:
This can be written in a simple matrix form:
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 13: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/13.jpg)
MHD Governing Equations
Conservative variables and fluxes:
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 14: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/14.jpg)
A primitive form:
where the coefficient matrix is
Linearized System
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 15: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/15.jpg)
Corner Transport Upwind (CTU)
Linear systemin 3D
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 16: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/16.jpg)
Corner Transport Upwind (CTU)
Linear systemin 3D
Normal predictor Transverse corrector
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 17: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/17.jpg)
Corner Transport Upwind (CTU)
Linear systemin 3D
Normal predictor Transverse corrector
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Traditional approach (Colella 1990; Saltzman 1994)
Characteristic tracing for the normal predictor
Subsequent calls to Riemann solvers for transverse corrector
![Page 18: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/18.jpg)
Corner Transport Upwind (CTU)
Linear systemin 3D
Normal predictor Transverse corrector
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Traditional approach (Colella 1990; Saltzman 1994)
Characteristic tracing for the normal predictor
Subsequent calls to Riemann solvers for transverse corrector
New approach (Lee and Deane 2009):
Characteristic tracing for BOTH normal predictor and transverse corrector!
![Page 19: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/19.jpg)
A primitive form:
where the coefficient matrix is
First consider the evolution in the x-normal direction and treat the normal magnetic field separately from the other variables:
Linearized System, cont’d
Normal predictor
MHD source termFLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 20: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/20.jpg)
Single-step data Reconstruction-evolution in USM
Normal Predictor
Characteristic Tracing
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 21: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/21.jpg)
Characteristic tracing for Transverse corrector
A jump relationship:
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 22: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/22.jpg)
Reduced 3D CTU in USM
Characteristic Tracing for
Normal Predictor
Transverse Corrector
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 23: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/23.jpg)
Full 3D CTU in USM
Characteristic Tracing for
Normal Predictor
Transverse Corrector
F u l l C T U d i a g o n a l c o u p l i n g FLASH Workshop
Hamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 24: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/24.jpg)
Summary of Part 1
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
New approach of using characteristic tracing for BOTH normal predictor and transverse corrector
Reduced 3D CTU A direct extension of 2D CTU to 3D Requires 3 Riemann solves for 3D (6-ctu needs 6 Riemann solves) Only including second cross derivatives CFL limit ~ 0.5
Full 3D CTU Full considerations of accounting for third cross derivatives Requires 3 Riemann solves for 3D (12-ctu needs 12 Riemann solves) CFL limit ~ 1.0 20% relative performance gain compared to reduced 3D CTU
![Page 25: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/25.jpg)
Part 2
Divergence-Free fields:Constrained Transport (CT) MHD
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 26: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/26.jpg)
Part 2
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
CT scheme by Balsara and Spicer, 1998:
![Page 27: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/27.jpg)
Part 2: recall…
Conservative variables and fluxes:
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 28: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/28.jpg)
Part 2
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
New upwind biased modified electric field construction(upwind-MEC), Lee 2012:
![Page 29: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/29.jpg)
Part 2
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Small angle advection of the 2D field loop:
![Page 30: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/30.jpg)
Part 2
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Small angle advection of the 3D field loop:
![Page 31: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/31.jpg)
Summary of Part 2
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Three CT schemes were discussed: Standard CT scheme by Balsara and Spicer, 1998:
Takes a simple arithmetic averaging Lacks numerical diffusion for magnetic fields advection
Modified electric field construction (MEC) scheme by Lee and Deane, 2009: 3rd order accurate in space Not enough numerical diffusion for field advection
Upwind biased MEC (upwind-MEC) scheme by Lee, 2012 (to be submitted) Upwind scheme of MEC Added numerical diffusion to stabilize field advection
![Page 32: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/32.jpg)
Part 3
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Verification, convergence, and performance
![Page 33: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/33.jpg)
Part 3
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 34: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/34.jpg)
Part 3
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 35: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/35.jpg)
Part 3
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 36: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/36.jpg)
Part 3
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 37: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/37.jpg)
Part 3
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 38: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/38.jpg)
Part 3
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 39: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/39.jpg)
Summary of Part 3
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Verification tests for the reduced/full 3D CTU schemes:
CFL=0.95 for all 3D simulations using the full CTU scheme
CFL=0.475 for the reduced CTU scheme
They both converge in 2nd order
20% performance gain in using the full CTU scheme:
Various choices in runtime parameters
![Page 40: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/40.jpg)
Conclusion
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Directionally split vs. unsplit formulations for hydro and MHD
Unsplit hydro/MHD solvers in FLASH4 (also FLASH3 in part) The reduced and full 3D CTU algorithms Upwind-MEC scheme for MHD Stable solutions with 2nd order convergence with CFL=0.95 20% performance gain in the full CTU scheme over the reduced CTU
scheme
Work in progress: Fully implicit Jacobian-Free Newton-Krylov implicit solver for the unsplit
solvers More HEDP capabilities for the USM solver
![Page 41: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/41.jpg)
Thank You
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
Questions?
![Page 42: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/42.jpg)
New Upwind PPM for Slowly Moving Shock
Upwind PPM 5th order WENO
Standard PPM Standard PPM with increasing By
larger By
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 43: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/43.jpg)
New Upwind PPM for Slowly Moving Shock
Upwind PPM 5th order WENO
Standard PPM Standard PPM with increasing By
Lee, 2010, 5th Astronum Proceeding;
Lee, 2011, in preparation
larger By
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012
![Page 44: FLASH Workshop Hamburger Sternwarte , University of Hamburg, Feb 15 – Feb 16, 2012](https://reader036.vdocuments.us/reader036/viewer/2022062521/56816977550346895de164e0/html5/thumbnails/44.jpg)
Block and Mesh Packages
Uniform Grid AMR with variable patch size - CHOMBO
q Mesh package can be selected at configuration time
q The basic abstraction is a block of interior cells surrounded by guard cells
q Grid unit makes sure that blocks are self contained before being given to the solvers
Oct tree based AMR - PARAMESH
FLASH WorkshopHamburger Sternwarte, University of Hamburg, Feb 15 – Feb 16, 2012