![Page 1: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/1.jpg)
Toward Exascale Seismic Imaging & Inversion
Jeroen TrompDepartment of Geosciences
Program in Applied & Computational MathematicsPrinceton Institute for Computational Science & Engineering
Ebru Bozdağ, Dimitri Komatitsch, Lion Krischer, Matthieu Lefebvre, Wenjie Lei, Daniel Peter & James Smith
ORNL: Judy Hill, Norbert Podhorszki & David Pugmire
![Page 2: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/2.jpg)
SPECFEM3D_GLOBE
SPECFEM3D_Cartesian
Sof tware
![Page 3: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/3.jpg)
Spect ra l -E lement M
Hexahedral finite-elements
Gauss-Lobatto-Legrendre quadrature
Diagonal mass matrix
Explicit time-marching scheme
![Page 4: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/4.jpg)
SPECFEM3D_Cartesian
![Page 5: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/5.jpg)
Open-Source
www.geodynamics.org
SPECFEM3D
Unstructured meshes
Load-balanced mesh partitioning
Fluid-solid coupling
Anisotropy
Attenuation
Adjoint capabilities
![Page 6: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/6.jpg)
SPECFEM3D
Daniel Peter
![Page 7: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/7.jpg)
SPECFEM3D
Yang Luo
Seismic Imaging
![Page 8: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/8.jpg)
SPECFEM3D_GLOBE
![Page 9: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/9.jpg)
Open-Source 3D crust & mantle models
Topography & bathymetry
Rotation
Ellipticity
Gravitation
Anisotropy
Attenuation
Adjoint capabilities
www.geodynamics.org
SPECFEM3D_GLOBE
![Page 10: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/10.jpg)
Automatically triggered by global earthquakes
1D & 3D synthetics
Public outreach movies
Provided by Incorporated Research Institutions for Seismology (IRIS)
ShakeMov iehttp://global.shakemovie.princeton.edu
![Page 11: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/11.jpg)
Globa l Se ismic Tomography
Ebru Bozdag
Assimilation of ~100 million data
![Page 12: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/12.jpg)
High-performance computing
![Page 13: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/13.jpg)
Top500.org - June 2015
GPU C
TitanPiz Daint
![Page 14: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/14.jpg)
SPECFEM3D_GLOBE ~100,000 lines of code Fortran90 / C / Cuda / OpenCL
SPECFEM3D ~70,000 lines of code Fortran90 / C / Cuda
GPU C
![Page 15: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/15.jpg)
• Initial implementation: CUDA
• In collaboration with NVIDIA (Peter Messmer & Cyril Zeller)
• Current implementation:
• BOAST : Bringing Optimization through Automatic Source-to-Source Transformations
• Kernels written in Ruby
• Generates CUDA and OpenCL
• Calls to kernels in C
• Tuned for Fermi and Kepler architectures
GPU Por tab i l i t y
![Page 16: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/16.jpg)
Daniel Peter
![Page 17: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/17.jpg)
Per formance
July 18, 2012 Researchers Squeeze GPU Performance from 11 Big Science Apps
February 05, 2013 Four Applications Sustain One Petaflop on Blue Waters
●
●
●
●
●
●
0.00
50.
010
0.02
00.
050
0.10
0
108 216 300 432 600 864 1350 2400 5400
Strong scaling for the PREM model, N=480
Number of GPUs
Mea
n tim
e (s
) per
tim
e st
ep
● 1 MPI process per GPU2 MPI processes per GPUPerfect scaling
Strong Scaling
0.001
0.01
0.1
100 1000
Mea
n tim
e (s
) per
tim
e st
ep
Number of MPI processes
1 MPI process per GPU2 MPI processes per GPUPerfect scaling
Weak Scaling
![Page 18: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/18.jpg)
SPECFEM3D_GLOBE - Mi lestones
2013 Blue Waters XE6 simulation > 1 PFlops (shortest period < 2 s)on 21,675 XE nodes, 693,600 MPI ranks, 693,600 cores
2003 Earth Simulator simulation > 5 TFlops (shortest period ~5 s) on 243 nodes, 1,944 MPI ranks, 1,944 cores
2008 Kraken XT5 simulation > 160 TFlops (shortest period ~1.72 s)on 149,784 cores
2015 K computer simulation > 1.24 PFlops (shortest period ~ 1.2 s)on 82,134 nodes, 82,134 MPI ranks w/ 8 OpenMP threads, 657,072 cores
![Page 19: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/19.jpg)
T i tan
2013 - 2015 SPECFEM3D_GLOBE a l locat ion: 250M core hours
![Page 20: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/20.jpg)
Next Generat ion Mach ine: Summit
ORNL’s Center for Accelerated Application Readiness (CAAR)
OLCF partnership with IBM, NVIDIA & Mellanox
![Page 21: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/21.jpg)
Exascale Challenges
![Page 22: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/22.jpg)
N observed ADIOS data
No
Iterate
Finish
Request observed data
Extract SEED files
Convert to ADIOS
Pre-processing(embarrassingly parallel)
Post-processing(parallel)
N selected earthquakes
Process data, select windows, make measurements & compute adjoint sources
N ADIOS adjoint source files
Run N adjoint simulations
N ADIOS kernel files
Sum kernels: 1 ADIOS gradient file
Pre-condition & smooth the gradient
Determine step length
Update model: 1 ADIOS model file
N synthetic ADIOS data
Convergence?
Run mesher: 1 ADIOS mesh file
Run N forward simulations
Taming Workflow Issues
![Page 23: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/23.jpg)
N observed ADIOS data
No
Iterate
Finish
Request observed data
Extract SEED files
Convert to ADIOS
Pre-processing(embarrassingly parallel)
Post-processing(parallel)
N selected earthquakes
Process data, select windows, make measurements & compute adjoint sources
N ADIOS adjoint source files
Run N adjoint simulations
N ADIOS kernel files
Sum kernels: 1 ADIOS gradient file
Pre-condition & smooth the gradient
Determine step length
Update model: 1 ADIOS model file
N synthetic ADIOS data
Convergence?
Run mesher: 1 ADIOS mesh file
Run N forward simulations
ASDFASDF
ASDF
ASDF
Seismic Tomography Workflow
1. Current data formats are inadequate for fast, parallel I/O; ASDF: an
Adaptable Seismic Data Format
2. Storage & visualization of Earth models:ADIOS with VisIt
3. Workflow stabilization:Kepler, Swift & Pegasus
L-BFGSQuasi-Newton
![Page 24: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/24.jpg)
Seismic Imaging & Invers ion Cha l lenges
• Cheap, abundant sensors
• Massive amounts of data
- Industry data sets
- Regional & global seismology data sets
- Cross-correlation data sets for seismic interferometry
• On HPC systems, I/O is the bottleneck
• Adopt new data formats for fast parallel I/O (e.g., NetCDF, HDF5 & ADIOS)
• Data culling tools to reduce preprocessing time
• A standard for the exchange of Earth models
• Adopt workflow management tools (e.g., Kepler, Pegasus & Swift)
• Tools for data mining, feature extraction, visualization & virtualization (e.g., ParaView, VisIt)
![Page 25: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/25.jpg)
Taming I/O Issues:Adaptable Seismic Data Format
![Page 26: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/26.jpg)
Data in Regional & Global Seismology
[www.iris.edu] [web.mst.edu]
[drh.edm.bosai.go.jp]
[www.geo.uib.no]
[data.earthquake.cn]
[Simons et al, 2006]
MERMAID/MariScope
![Page 27: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/27.jpg)
Data in Exp lorat ion Se ismology
• Petabytes of data• SEG-Y is the current standard• Variable SEG-Y file structure• SEG-Y programs do not always
follow specifications
3D marine survey can involve 5,000 shots and 50,000 recorders
![Page 28: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/28.jpg)
ASDF: Adaptab le Se ismic Data Format
• Collaboration involving Princeton University, Munich University (ObsPy) and Oak Ridge National Laboratory
• Increase I/O performance by combining all the time series for a single shot or earthquake into one file
• Take advantage of parallel processing
• Use modern file format as container (HDF5)
• Store provenance inside the file for reproducibility
• Use existing standards when possible (e.g., XML)
• Open wiki for development
![Page 29: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/29.jpg)
Deve lopment TeamMichael Afanasiev, Jean-Paul (Pablo) Ampuero, Kangchen Bai, Piero Basini, Céline Blitz, Alexis Bottero, Ebru Bozdag, Emanuele Casarotti, Joseph Charles, Min Chen, Paul Cristini, Clément Durochat, Percy Galvez, Dominik Göddeke, Vala Hjörleifsdóttir, Sue Kientz, Dimitri Komatitsch, Jesús Labarta, Nicolas Le Goff, Pieyre Le Loher, Matthieu Lefebvre, Qinya Liu, David Luet, Yang Luo, Alessia Maggi, Federica Magnoni, Roland Martin, René Matzen, Dennis McRitchie, Matthias Meschede, Peter Messmer, David Michéa, Vadim Monteiller, Surendra Nadh Somala, Tarje Nissen-Meyer, Daniel Peter, Kevin Pouget, Max Rietmann, Elliott Sales de Andrade, Brian Savage, Bernhard Schuberth, Anne Sieminski, James Smith, Leif Strand, Carl Tape, Jeroen Tromp, Brice Videau, Jean-Pierre Vilotte, Zhinan Xie, Chang- Hua Zhang, Hejun Zhu
![Page 30: Toward Exascale Seismic Imaging & Inversionimages.nvidia.com/events/sc15/pdfs/SC15-NVIDIA-Towards... · 2015-11-24 · Data in Exploration Seismology • Petabytes of data • SEG-Y](https://reader034.vdocuments.us/reader034/viewer/2022043001/5f79c8553f23c60d497ba211/html5/thumbnails/30.jpg)
SPECFEM - Awards
2010 BULL Joseph Fourier Prize winner for the partial GPU port of SPECFEM3D_GLOBE
2003 ACM Gordon Bell Award for Best Performancefor SPECFEM3D_GLOBE simulation on the Earth Simulator
2008 ACM Gordon Bell Finalist for SPECFEM3D_GLOBE simulation reaching resolution of 1.72s shortest period