local adaptive mesh refinement on the gpumartinsa.at.ifi.uio.no/files/siam_gs13_amr_on_gpu.pdf ·...
TRANSCRIPT
Technology for a better society 1
Martin Lilleeng Sætra
University of Oslo
18-06-2013
Local Adaptive Mesh Refinement
on the GPU
Technology for a better society
Outline
2
• The starting point: Our GPU-based shallow water simulator + Original Local Adaptive Mesh Refinement-paper
• Novel work: Adaptive mesh refinement fully implemented on the GPU
• Performance results
• Video: Malpasset dam break case with adaptive mesh refinement
Technology for a better society
The Shallow Water Equations (SWE)
3
Vector ofConserved variables Flux Functions
Bed slopesource term
Bed frictionsource term
Numerical Simulation of the SWE:• Hyperbolic partial differential equation
– Enables explicit schemes.• Solutions form discontinuities / shocks
– Require high accuracy in smooth parts without oscillations near discontinuities.
• Solutions include dry areas– Avoid negative water depths in
simulations.• Accuracy
– 2nd order spatial/temporal discretization.
Scheme of choice: A. Kurganov and G. Petrova, A Second-Order Well-Balanced Positivity Preserving Central-Upwind Scheme for the Saint-Venant System Communications in Mathematical Sciences, 5 (2007), 133-160
Technology for a better society
Spatial Discretization
4
Write on vector form:
Impose finite-volume grid with discrete fluxes:
Continuous equation
Discrete spatial grid
Discrete flux calculation
Technology for a better society
Calculate Fluxes
5
Continuous variables Discrete variables
Dry states fix
Slope reconstruction
Evaluate integration pointsFlux calculation
Vector ofConservedvariables
Flux FunctionsBed slope
source termBed frictionsource term
Technology for a better society 6
Evolve in Time
Technology for a better society
One Full Time Step Using the Kurganov-Petrova-scheme on the GPU
7
3. Halfstep
1. Calculate fluxes
4. Calculate fluxes5. Evolve in time
6. Apply boundaryconditions
2. Calculate Δt
Technology for a better society
Domain Decomposition
8
• “Traditional” CUDA block decomposition.• Each Streaming Multiprocessor of the GPU computes on a small 2D block.• Neighboring blocks use overlap to exchange information.• Global ghost cells for boundary conditions (wall, open, fixed depth etc.).
Technology for a better society
Local Adaptive Mesh Refinement (AMR)
9
• Goal: Increase accuracy, minimize cost.
Cost
Accuracy
Grid re
finement
Technology for a better society
Capture Local Features in the Solution
10
Technology for a better society
“AMR Grid” Hierarchy
11
• “Traditional” CUDA block decomposition within each grid.• Initialize new subgrids at level ℓ +1, with twice the resolution, using
reconstructed cell values from current grids at level ℓ. • The boundary interfaces of grid ℓ +1 must be aligned with grid cell
boundaries in grid ℓ.• Each new subgrid can be viewed as a stand-alone simulator.
(ℓ=0)
(ℓ=1)
(ℓ=1)
(ℓ=2)
Grid 0
Grid 1 Grid 2
Grid 3
Technology for a better society
Keep Data on the GPU
12
Technology for a better society
AMR: Time Integration
13
AMR: Refine
Technology for a better society 14
Extending the Simulation Cycle
BC
Compute boundaries
Reset AMR data(Not a CUDA kernel)
Flux correction
Coarsen / Average
• Pre-step • “Regular” time stepping
• Post-step
Technology for a better society
Boundary Values – Space
15
• Save the solution at the beginning and at the end of each time step from the level ℓ grid at all boundaries to level ℓ+1 subgrids.
• Each subgrid has data structures for saving these values.• The solution is reconstructed in space at both the beginning and the
end of the time step.
ℓ
ℓ+1
Technology for a better society
Boundary Values – Time
16
• The saved boundary values are then linearly interpolated in time, by a special boundary condition kernel used on all grids except the root grid.
t+Δt0t
Grid 0 (ℓ)
Grid 1 (ℓ+1)
Δt0
Δt1 Δt2
Technology for a better society
Time Step Size
17
• Since the time step size, Δt, is dependent on the eigenvalues of the solution, each grid has a different Δt.
• After each time step on the level ℓ grid, the time step size is given to all level ℓ+1 grids.
• The last time step in Grid 1 and 2 (Δt2 and Δt4) is reduced so that all grids reach t+Δt0 after one full simulation cycle.
t+Δt0t
Grid 0 (ℓ)
Grid 1 (ℓ+1)
Grid 2 (ℓ+1)
Δt0
Δt1 Δt2
Δt3 Δt4
Technology for a better society
Averaging From Level ℓ +1 to ℓ
18
• After all level ℓ+1 subgrids are advanced to the same time as the level ℓ grid, the distribution of the solution will be more accurate in the level ℓ+1 subgrids.
• The solution is “moved up” in the AMR hierarchy by replacing the values in level ℓ by the values from a overlaying subgrid at level ℓ+1.
• A simple average is used.
ℓ+1 ℓ
Technology for a better society
Flux Correction Step – Fine-coarse
19
• Fluxes across all fine-coarse (ℓ+1 to ℓ) interfaces are accumulated for each time step in the level ℓ+1 subgrids.
• After all the level ℓ+1 subgrids are at the same advanced time as the level ℓ grid, a flux correction step is performed on the level ℓ grid.
• The correction step uses the difference between the flux from the level ℓ grid and the accumulated fluxes from the level ℓ+1 subgrids, and corrects the values in the level ℓ grid.
t0
t0
t1
t2
t1
t2
ℓ+1 ℓ
(Δt1 + Δt2 = Δt0)
Technology for a better society
Flux Correction Step – Fine-fine
20
• Fine grid cells adjacent to other fine grids also needs to be corrected.• Again, we use the difference between the flux from the level ℓ grid and
the accumulated fluxes from the level ℓ+1 subgrids, but now we write the correction to the adjacent ℓ+1 grid, instead of the level ℓ grid.
• This, together with the coarse-fine flux correction, maintains conservation of mass.
t2
t2
t0
t1
t0
t1
ℓ+1
(Δt0 + Δt1 = Δt2 + Δt3)
ℓ+1
t3
t3
Technology for a better society 21
Extending the Simulation Cycle
BC
Compute boundaries
Reset AMR data(Not a CUDA kernel)
Flux correction
Coarsen / Average
• Pre-step • “Regular” time stepping
• Accumulate fine fluxes
• Reduce last time step
• Special BC
• Post-step
Technology for a better society
AMR: Refine
22
AMR: Time Integration
Technology for a better society
“Correct” Refinement Criteria
23
Technology for a better society
Refinement Test
24
• After a given number of time steps, each cell in every grid is checked against a given refinement criterion, and potentially flagged for refinement. This test is performed by a dedicated kernel.
• The same kernel does a reduction to tiles, which is a collection of cells, and writes the number of cells marked for refinement per tile to a map.
• This map is used to generate new proposed grids.
• The new proposed grids are checked for overlaps with existing grids. In case of overlap, the proposed grids are broken down into smaller grids.
• The two last points are the only work done by the CPU.
Technology for a better society
Initializing New Subgrids
25
• The bathymetry of new subgrids are bilinearly interpolated from the parent grid using effective dedicated GPU hardware (texture memory).
• The initialization of the variables is done by reconstructing the solution from the last time step on the parent grid, and evaluating it in the cell centers of the new grid.
Technology for a better society
Performance Results – N Subgrids (2D)
26
Technology for a better society
Performance Results – 1 Subgrid
27
Technology for a better society 28
Technology for a better society
Video: Malpasset Dam Break Case w/AMR
29
./malpasset_amr.avi
Technology for a better society 30
Thank you for your attention
Contact:Martin Lilleeng Sæ[email protected]: http://martinsa.at.ifi.uio.no/
SINTEF homepage: http://www.sintef.no/heterocomp
Acknowledgements: Mustafa Altinakar, André R. Brodtkorb, Christopher Dyken, Trond R. Hagen, Knut-Andreas Lie, and Jostein R. Natvig.
Technology for a better society
• A. R. Brodtkorb, M. L. Sætra, and M. Altinakar, Efficient Shallow Water Simulations on GPUs: Implementation, Visualization, Verification, and Validation, Computers & Fluids 55(0):1–12, 2012.
33
References