gpus to mars - nvidiaon-demand.gputechconf.com/gtc/2015/presentation/s5398... · 2015-03-27 ·...

GPUs TO MARS Full Scale Simulation of SpaceX’s Mars Rocket Engine

Adam Lichtl, Stephen Jones, GTC 2015

Background

Independent space launch company

Build and operate our own rocket & capsule

Operations in CA, TX & FL

Founded in 2002, first launch in 2008

18 successful missions to date

Design

Launch

Mars

Length of day 24 hours, 40 minutes

Length of year 687 days

Gravity 0.375gs

Distance from Sun 1.53 AU

Mars Air CO2 (96%), N2 (2%)

Earth Air O2 (21%), N2 (78%)

Atmosphere 1% density / pressure of Earth

Temperature -284 / +86 / -81 F (low / high / avg)

Human Mars Mission Classes

Credit: CollectSpace

Mass to Mars

Apollo Command &

Service Module

Mass: 46 tons

Terrestrial assembly

NASA Mars Architecture

Mass: 300 tons

3x heavy-lift

launches

In-orbit assembly

International Space Station

Mass: 450 tons

36x Space Shuttle,

5x Proton launches

Image Credit: Mark Benson & Kerbal Space Program Image Credit: NASA Image Credit: NASA

Methane Fuel

O H H

O

O

C

O H

H

O O C

Water from below ground

Carbon dioxide from atmosphere

Synthesize return-journey fuel on Mars

2H2O + CO2 -> CH4 + 2O2

O

O O

O

H

H C

H

H

Rocket Engine Design

Why Simulation?

1. Investigate what cannot be measured

2. Reduce need for testing

3. Design optimisation: narrow design space

4. Proactive instead of reactionary design

seconds 10-11 10-9 10-8 10-6 10-10 10-7 10-5 10-3 10-4

Diffusion

Reaction

Advection Acoustics

Chamber Residence

Time scales vary by

8 orders of magnitude

Total points: (103)6 = 1018 = 1,000,000,000,000,000,000 points @1kB per point -> Yottabytes of data

kmax kmin

log E(k)

Injection

Cascade

Viscous dissipation

Kolmogorov Scale: ~1µm

Combustion Chamber: ~1m

Length scales vary by

6 orders of magnitude

Eddy Wavenumber

Edd

y En

ergy

Structure in Turbulence

Credit: SpaceWeather

Simulation Meshing

Structured Grid Adaptive Grid Unstructured Grid

Fourier Transform: Spectral Compression

Wavelets: Local Fractal Basis

Wavelet Compression

Wavelet Compression

δ

δ > ε (ε is some tolerance)

δ < ε

δ

Wavelet Compression

Transform Output

Decompression

Transform Output

Decompression

+δ

Transform Output

Decompression

Transform Output

Regular Grid: Direct Indexing

Index (x, y, z) = p Index (x, y+1, z) = p + Nx

(x, y, z)

(x, y+1, z)

Sparse Grid: Difficult Indexing

Index (x, y, z) = p Index (x, y+1, z) = ???

(x, y, z)

(x, y+1, z)

Sparse Grid Point Lookup

Linear search

Tree-based search

Hybrid tree search + dense indexing

Grid Point Contains Reference to Data

Location (x,y,z)

Data Reference

(x, y, z)

Grid Point 14


Location (x,y,z)

Data Reference

(x, y, z)

Grid Index

Grid Point 14

Grid Data

14 15 16

(11,12,5)

8

(11,11,5)

3

(Location)

(Reference)


ρu4

ρ4

ρw4

ρv4

E4

ρu7

ρ7

ρw7

ρv7

E7

ρu9

ρ9

ρw9

ρv9

E9

ρu0

ρ0

ρw0

ρv0

E0

ρu5

ρ5

ρw5

ρv5

E5

ρu1

ρ1

ρw1

ρv1

E1

ρu8

ρ8

ρw8

ρv8

E8

ρu3

ρ3

ρw3

ρv3

E3

ρu6

ρ6

ρw6

ρv6

E6

ρu2

ρ2

ρw2

ρv2

E2

Simulation Data

density

X-momentum

Y-momentum

Z-Momentum

Energy

Location (x,y,z)

Data Reference

(x, y, z)

Grid Point 14

Grid Data

Grid Index 14 15 16

(11,12,5)

8

(11,11,5)

3

…

…


ρu4

ρ4

ρw4

ρv4

E4

ρu7

ρ7

ρw7

ρv7

E7

ρu9

ρ9

ρw9

ρv9

E9

ρu0

ρ0

ρw0

ρv0

E0

ρu5

ρ5

ρw5

ρv5

E5

ρu1

ρ1

ρw1

ρv1

E1

ρu8

ρ8

ρw8

ρv8

E8

ρu3

ρ3

ρw3

ρv3

E3

ρu6

ρ6

ρw6

ρv6

E6

ρu2

ρ2

ρw2

ρv2

E2

Simulation Data

density

X-momentum

Y-momentum

Z-Momentum

Energy

Location (x,y,z)

Data Reference

(x, y, z)

Grid Point 14

(11,12,5)

8

(11,11,5)

3

…

… Grid Data

14 15 16 Grid Index

P0 P1 P2 P3 P4 P5 P6 P7 P8 P9

Grid Adaptation

0 1 2 3 4 5 6 7 8

x0y0 x1y1 x2y2 x3y3 x4y4 x5y5 x6y6 x7y7 x8y8 x9y9

9

Location

Reference

Grid Index

P0 P2 P4 P6 P8

Grid Adaptation

0 2 4 6 8

x0y0 x2y2 x4y4 x6y6 x8y8 Location

Reference

Grid Index

P0 P1 P2 P3 P4

Grid Adaptation

0 2 4 6 8

x0y0 x2y2 x4y4 x6y6 x8y8 Location

Reference

Grid Index

P0 P1 P2 P3 P4 P5 P6 P7 P8 P9

Grid Transformations Track Data

0 1 2 3 4 5 6 7 8


9

Location

Reference

Grid Index

P0 P1 P2 P3 P4 P5 P6 P7 P8 P9


0 1 2 3 4 5 6 7 8


9

Location

Reference

Delete Add

Sort

Transpose

Grid Index

P0 P1 P2 P3 P4 P5 P6 P7 P8 P9

P0 P1 P2 P3 P4 P5 P6 P7 P8 P9


0 1 2 3 4 5 6 7 8


9

Location

Reference

Grid Index

Location

Delete Add

Sort

Transpose

Grid Index

4 7 9 0 5 1 8 3 6 2 Reference


P0 P1 P2 P3 P4 P5 P6 P7 P8 P9

Losing Coalesced Memory Access

4 7 9 0 5 1 8 3 6 2

ρu4

ρ4

ρw4

ρv4

E4

ρu7

ρ7

ρw7

ρv7

E7

ρu9

ρ9

ρw9

ρv9

E9

ρu0

ρ0

ρw0

ρv0

E0

ρu5

ρ5

ρw5

ρv5

E5

ρu1

ρ1

ρw1

ρv1

E1

ρu8

ρ8

ρw8

ρv8

E8

ρu3

ρ3

ρw3

ρv3

E3

ρu6

ρ6

ρw6

ρv6

E6

ρu2

ρ2

ρw2

ρv2

E2

Coalesced Simulation Data

Grid Index

Reference

P0 P1 P2 P3 P4 P5 P6 P7 P8 P9

Losing Coalesced Memory Access

4 7 9 0 5 1 8 3 6 2

ρu4

ρ4

ρw4

ρv4

E4

ρu7

ρ7

ρw7

ρv7

E7

ρu9

ρ9

ρw9

ρv9

E9

ρu0

ρ0

ρw0

ρv0

E0

ρu5

ρ5

ρw5

ρv5

E5

ρu1

ρ1

ρw1

ρv1

E1

ρu8

ρ8

ρw8

ρv8

E8

ρu3

ρ3

ρw3

ρv3

E3

ρu6

ρ6

ρw6

ρv6

E6

ρu2

ρ2

ρw2

ρv2

E2

Uncoalesced Simulation Data

Grid Index

Reference

P0 P1 P2 P3 P4 P5 P6 P7 P8 P9

4 7 9 0 5 1 8 3 6 2


Restoring Coalesced Memory Access

ρu4

ρ4

ρw4

ρv4

E4

ρu7

ρ7

ρw7

ρv7

E7

ρu9

ρ9

ρw9

ρv9

E9

ρu0

ρ0

ρw0

ρv0

E0

ρu5

ρ5

ρw5

ρv5

E5

ρu1

ρ1

ρw1

ρv1

E1

ρu8

ρ8

ρw8

ρv8

E8

ρu3

ρ3

ρw3

ρv3

E3

ρu6

ρ6

ρw6

ρv6

E6

ρu2

ρ2

ρw2

ρv2

E2


Location

Reference

Grid Index

P0 P1 P2 P3 P4 P5 P6 P7 P8 P9

4 7 9 0 5 1 8 3 6 2


P0 P1 P2 P3 P4 P5 P6 P7 P8 P9

Restoring Coalesced Memory Access

ρu4

ρ4

ρw4

ρv4

E4

ρu7

ρ7

ρw7

ρv7

E7

ρu9

ρ9

ρw9

ρv9

E9

ρu0

ρ0

ρw0

ρv0

E0

ρu5

ρ5

ρw5

ρv5

E5

ρu1

ρ1

ρw1

ρv1

E1

ρu8

ρ8

ρw8

ρv8

E8

ρu3

ρ3

ρw3

ρv3

E3

ρu6

ρ6

ρw6

ρv6

E6

ρu2

ρ2

ρw2

ρv2

E2

0 1 2 3 4 5 6 7 8 9

Reorder Simulation

Data


ρu4

ρ4

ρw4

ρv4

E4

ρu7

ρ7

ρw7

ρv7

E7

ρu9

ρ9

ρw9

ρv9

E9

ρu0

ρ0

ρw0

ρv0

E0

ρu5

ρ5

ρw5

ρv5

E5

ρu1

ρ1

ρw1

ρv1

E1

ρu8

ρ8

ρw8

ρv8

E8

ρu3

ρ3

ρw3

ρv3

E3

ρu6

ρ6

ρw6

ρv6

E6

ρu2

ρ2

ρw2

ρv2

E2

Coalesced Simulation Data

Reset Indices

x4y4 x7y7 x9y9 x0y0 x5y5 x1y1 x8y8 x3y3 x6y6 x2y2 Location

Reference

Grid Index

Coupled Multi-Physics

Chemistry

Diffusion

Advection

Thermodynamics

Real Gas Properties

Extreme pressure requires real-gas properties, evaluated at each grid node

Viscosity

Conductivity

Diffusion constants

Per-species diffusion rates

Independent Calculations

Massively Parallelizable O2 phase diagram, Z-axis is density (colour: liquid is blue, gas is white)

Pressure

Temperature

Strong Temperature Dependence

Arrhenius equation includes temperature in exponent

Chemical Kinetic Model

H H

O

H H

O

O

O H H

H H +

H2 O2 H2O


O H

OH

O

O

O H

O

HO2

H

O H

O

H2O2

H

H

H H

O

H H

O

O

O H H

H H +

H2 O2 H2O

X


O H

OH

O

O

O H

O

HO2

H

O H

O

H2O2

H

H

H H

O

H H

O

O

O H H

H H +

H2 O2 H2O

Methane Chemistry

O

O

O

O H

C

H H

H +

O2 CH4

H H

O

H H

O C

O

O

+

H2O CO2

Methane Chemistry

O

O

O

O H

C

H H

H +

O2 CH4

H H

O

H H

O C

O

O

+

H2O CO2

53 Species, 325 Reactions

(Actual simulation data, rendered with a ray tracer)

Acknowledgements

Oleg Vasilyev, University of Colorado Boulder Department of Mechanical Engineering

Jonathan Regele, Iowa State University Department of Aerospace Engineering

Don Lamb The Flash Center for Computational Science

Joseph Oefelein, Sandia National Laboratories Combustion Research Facility

Praveen Ramaprabhu University of North Carolina at Charlotte

Marc Massot Ecole Centrale, Paris

Want To Help?

If you have a software or physics background and

you’re interested in working with us, please contact

[email protected]

gpus to mars - nvidiaon-demand.gputechconf.com/gtc/2015/presentation/s5398... · 2015-03-27 ·...

Documents