high performance computing for neutron tomography … · 2017-08-09 · high performance computing...
TRANSCRIPT
![Page 1: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/1.jpg)
High Performance Computing for NeutronTomography Reconstruction
A Parallel Approach to Filtered Backprojection (FBP)
Zongpu Li1 Cain Gantt2 Rick Archibald∗
1Department of Physics and Materials ScienceCity University of Hong Kong
2Department of Chemistry, Astronomy, and PhysicsGeorgia College and State University
∗Mentor, Oak Ridge National Laboratory (ORNL)
August 3, 2017
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 1 / 26
![Page 2: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/2.jpg)
Outline
1 BackgroundWhat is Laminography?Filtered Backprojection AlgorithmMATLAB Data
2 Objectives
3 MethodsFilterSerial ProgramParallel Program
4 Future Work
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 2 / 26
![Page 3: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/3.jpg)
Outline
1 BackgroundWhat is Laminography?Filtered Backprojection AlgorithmMATLAB Data
2 Objectives
3 MethodsFilterSerial ProgramParallel Program
4 Future Work
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 3 / 26
![Page 4: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/4.jpg)
Laminography & Tomography
Image Processing
Reconstruction of 3D volume from 2D projections (sinograms)
Fourier/harmonic analysis (specifically Radon transform)
Tomography is special case of laminography (tilt angle = 0◦)
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 4 / 26
![Page 5: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/5.jpg)
2D Tomography
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 5 / 26
![Page 6: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/6.jpg)
3D Laminography
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 6 / 26
![Page 7: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/7.jpg)
Outline
1 BackgroundWhat is Laminography?Filtered Backprojection AlgorithmMATLAB Data
2 Objectives
3 MethodsFilterSerial ProgramParallel Program
4 Future Work
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 7 / 26
![Page 8: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/8.jpg)
Filtered Backprojection (FBP) Algorithm
Inverse Radon Transform
f(x, y) =
∫ π
0pf (x · cos θ + y · sin θ)dθ
projections & orientation information → volume
For each projection. . .
clean it up (filter)
“smear” it through the volume (backproject & interpolate)
. . . then sum all smeared projectionsResult: reconstructed 3D volume
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 8 / 26
![Page 9: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/9.jpg)
Filtered Backprojection (FBP) Algorithm
initialize
filter projections
back-projection
finalize
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 9 / 26
![Page 10: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/10.jpg)
Filtered Backprojection (FBP) Algorithm
initialize
filter projections
back-projection
finalize
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 9 / 26
![Page 11: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/11.jpg)
Filtered Backprojection (FBP) Algorithm
initialize
filter projections
back-projection
finalize
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 9 / 26
![Page 12: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/12.jpg)
Filtered Backprojection (FBP) Algorithm
initialize
filter projections
back-projection
finalize
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 9 / 26
![Page 13: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/13.jpg)
Outline
1 BackgroundWhat is Laminography?Filtered Backprojection AlgorithmMATLAB Data
2 Objectives
3 MethodsFilterSerial ProgramParallel Program
4 Future Work
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 10 / 26
![Page 14: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/14.jpg)
Simulation
Volume: 65× 65× 65
![Page 15: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/15.jpg)
Simulation
Volume: 129× 129× 129
![Page 16: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/16.jpg)
Simulation
Volume: 257× 257× 257
![Page 17: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/17.jpg)
ORNL Spallation Neutron Source (SNS) Data
“Volume”: 1501× 1501× 1
![Page 18: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/18.jpg)
Outline
1 BackgroundWhat is Laminography?Filtered Backprojection AlgorithmMATLAB Data
2 Objectives
3 MethodsFilterSerial ProgramParallel Program
4 Future Work
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 12 / 26
![Page 19: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/19.jpg)
Objectives
Overall reduction in code execution time and memory requirements
Implement laminography filter
Perform FBP serially in C
Structure portions in parallel. . .
Message Passing Interface (MPI)
→ backprojection
Graphics Processing Unit (GPU)
→ FFT and filtering
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 13 / 26
![Page 20: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/20.jpg)
Objectives
Overall reduction in code execution time and memory requirements
Implement laminography filter
Perform FBP serially in C
Structure portions in parallel. . .
Message Passing Interface (MPI)
→ backprojection
Graphics Processing Unit (GPU)
→ FFT and filtering
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 13 / 26
![Page 21: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/21.jpg)
Objectives
Overall reduction in code execution time and memory requirements
Implement laminography filter
Perform FBP serially in C
Structure portions in parallel. . .
Message Passing Interface (MPI)
→ backprojection
Graphics Processing Unit (GPU)
→ FFT and filtering
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 13 / 26
![Page 22: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/22.jpg)
Objectives
Overall reduction in code execution time and memory requirements
Implement laminography filter
Perform FBP serially in C
Structure portions in parallel. . .
Message Passing Interface (MPI)
→ backprojectionGraphics Processing Unit (GPU)
→ FFT and filtering
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 13 / 26
![Page 23: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/23.jpg)
Objectives
Overall reduction in code execution time and memory requirements
Implement laminography filter
Perform FBP serially in C
Structure portions in parallel. . .
Message Passing Interface (MPI) → backprojection
Graphics Processing Unit (GPU)
→ FFT and filtering
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 13 / 26
![Page 24: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/24.jpg)
Objectives
Overall reduction in code execution time and memory requirements
Implement laminography filter
Perform FBP serially in C
Structure portions in parallel. . .
Message Passing Interface (MPI) → backprojectionGraphics Processing Unit (GPU)
→ FFT and filtering
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 13 / 26
![Page 25: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/25.jpg)
Objectives
Overall reduction in code execution time and memory requirements
Implement laminography filter
Perform FBP serially in C
Structure portions in parallel. . .
Message Passing Interface (MPI) → backprojectionGraphics Processing Unit (GPU) → FFT and filtering
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 13 / 26
![Page 26: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/26.jpg)
Outline
1 BackgroundWhat is Laminography?Filtered Backprojection AlgorithmMATLAB Data
2 Objectives
3 MethodsFilterSerial ProgramParallel Program
4 Future Work
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 14 / 26
![Page 27: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/27.jpg)
Filter Selection
Filter necessary to reduce blurring from backprojection
basic high-pass ramp filter (standard in tomography)
laminographic ramp filter (scaled depending laminography angle)
sinc filter (slightly varying frequency response)
cutoff frequency (dependent upon experiment geometry)
Using laminographic ramp filter with adjustable cutoff frequency
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 15 / 26
![Page 28: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/28.jpg)
Filter Selection
Filter necessary to reduce blurring from backprojection
basic high-pass ramp filter (standard in tomography)
laminographic ramp filter (scaled depending laminography angle)
sinc filter (slightly varying frequency response)
cutoff frequency (dependent upon experiment geometry)
Using laminographic ramp filter with adjustable cutoff frequency
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 15 / 26
![Page 29: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/29.jpg)
Filter Selection
Filter necessary to reduce blurring from backprojection
basic high-pass ramp filter (standard in tomography)laminographic ramp filter (scaled depending laminography angle)sinc filter (slightly varying frequency response)cutoff frequency (dependent upon experiment geometry)
Using laminographic ramp filter with adjustable cutoff frequency
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 15 / 26
![Page 30: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/30.jpg)
Outline
1 BackgroundWhat is Laminography?Filtered Backprojection AlgorithmMATLAB Data
2 Objectives
3 MethodsFilterSerial ProgramParallel Program
4 Future Work
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 16 / 26
![Page 31: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/31.jpg)
Serial ProgramChallenges
Reading data formatted for MATLAB:
write values to binary file
dimensions & parameters, data arrays
Projection filtering:
FFTW (Fastest Fourier Transform in the West)
zero-padding on projection arrays
forward transform, multiply filter, reverse transform
Backprojection:
coordinates before & after rotation
meshgrid?
greatly reduced memory requirements!
Pointers and integers...
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 17 / 26
![Page 32: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/32.jpg)
Serial ProgramChallenges
Reading data formatted for MATLAB:
write values to binary file
dimensions & parameters, data arrays
Projection filtering:
FFTW (Fastest Fourier Transform in the West)
zero-padding on projection arrays
forward transform, multiply filter, reverse transform
Backprojection:
coordinates before & after rotation
meshgrid?
greatly reduced memory requirements!
Pointers and integers...
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 17 / 26
![Page 33: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/33.jpg)
Serial ProgramChallenges
Reading data formatted for MATLAB:
write values to binary file
dimensions & parameters, data arrays
Projection filtering:
FFTW (Fastest Fourier Transform in the West)
zero-padding on projection arrays
forward transform, multiply filter, reverse transform
Backprojection:
coordinates before & after rotation
meshgrid?
greatly reduced memory requirements!
Pointers and integers...
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 17 / 26
![Page 34: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/34.jpg)
Serial ProgramChallenges
Reading data formatted for MATLAB:
write values to binary file
dimensions & parameters, data arrays
Projection filtering:
FFTW (Fastest Fourier Transform in the West)
zero-padding on projection arrays
forward transform, multiply filter, reverse transform
Backprojection:
coordinates before & after rotation
meshgrid?
greatly reduced memory requirements!
Pointers and integers...
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 17 / 26
![Page 35: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/35.jpg)
Serial ProgramChallenges
Reading data formatted for MATLAB:
write values to binary file
dimensions & parameters, data arrays
Projection filtering:
FFTW (Fastest Fourier Transform in the West)
zero-padding on projection arrays
forward transform, multiply filter, reverse transform
Backprojection:
coordinates before & after rotation
single coordinate value per loop
greatly reduced memory requirements!
Pointers and integers...
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 17 / 26
![Page 36: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/36.jpg)
Serial ProgramChallenges
Reading data formatted for MATLAB:
write values to binary file
dimensions & parameters, data arrays
Projection filtering:
FFTW (Fastest Fourier Transform in the West)
zero-padding on projection arrays
forward transform, multiply filter, reverse transform
Backprojection:
coordinates before & after rotation
single coordinate value per loop
greatly reduced memory requirements!
Pointers and integers...
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 17 / 26
![Page 37: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/37.jpg)
Outline
1 BackgroundWhat is Laminography?Filtered Backprojection AlgorithmMATLAB Data
2 Objectives
3 MethodsFilterSerial ProgramParallel Program
4 Future Work
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 18 / 26
![Page 38: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/38.jpg)
Parallel ProgramDifferent Approaches
MPI (Myagotin, et al.):
distribute projections (many independent projections)
decompose reconstructed volume (single volume between all nodes)
GPU acceleration:
FFT (many 1D transforms)
applying filter (many multiplication operations)
interpolation (texture mapping for hardware acceleration)
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 19 / 26
![Page 39: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/39.jpg)
MPIParallel decomposition
Data decomposition by projections
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 20 / 26
![Page 40: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/40.jpg)
MPIFindings
Intel MPI LibraryBridges at Pittsburg Supercomputing Center
Advantages:
shorter program execution time
Drawbacks:
large memory consumption (each node has full memory in volume)
MPI Reduce operation (more operations to perform)
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 21 / 26
![Page 41: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/41.jpg)
MPIFindings
Intel MPI LibraryBridges at Pittsburg Supercomputing Center
Advantages:
shorter program execution time
Drawbacks:
large memory consumption (each node has full memory in volume)
MPI Reduce operation (more operations to perform)
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 21 / 26
![Page 42: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/42.jpg)
MPIFindings
Intel MPI LibraryBridges at Pittsburg Supercomputing Center
Advantages:
shorter program execution time
Drawbacks:
large memory consumption (each node has full memory in volume)
MPI Reduce operation (more operations to perform)
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 21 / 26
![Page 43: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/43.jpg)
GPUHardware configuration
Node
CPU GPU
Node
CPU GPU
Node
CPU GPU
Node
CPU GPU
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 22 / 26
![Page 44: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/44.jpg)
GPUFindings
CUDA FFT Library (cuFFT)CUDA Basic Linear Algebra Subroutine Library (cuBLAS)Bridges (PSC): NVIDIA Tesla P100
Advantages:
functions are ready-to-go
no need to write kernel, manually manage device memory, etc.
Drawbacks:
cuBLAS issues
memcopy and device memory
Execution time improved over serial codeTesting data sets could be too small to see major improvement?
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 23 / 26
![Page 45: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/45.jpg)
GPUFindings
CUDA FFT Library (cuFFT)CUDA Basic Linear Algebra Subroutine Library (cuBLAS)Bridges (PSC): NVIDIA Tesla P100
Advantages:
functions are ready-to-go
no need to write kernel, manually manage device memory, etc.
Drawbacks:
cuBLAS issues
memcopy and device memory
Execution time improved over serial codeTesting data sets could be too small to see major improvement?
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 23 / 26
![Page 46: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/46.jpg)
GPUFindings
CUDA FFT Library (cuFFT)CUDA Basic Linear Algebra Subroutine Library (cuBLAS)Bridges (PSC): NVIDIA Tesla P100
Advantages:
functions are ready-to-go
no need to write kernel, manually manage device memory, etc.
Drawbacks:
cuBLAS issues
memcopy and device memory
Execution time improved over serial codeTesting data sets could be too small to see major improvement?
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 23 / 26
![Page 47: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/47.jpg)
GPUFindings
CUDA FFT Library (cuFFT)CUDA Basic Linear Algebra Subroutine Library (cuBLAS)Bridges (PSC): NVIDIA Tesla P100
Advantages:
functions are ready-to-go
no need to write kernel, manually manage device memory, etc.
Drawbacks:
cuBLAS issues
memcopy and device memory
Execution time improved over serial codeTesting data sets could be too small to see major improvement?
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 23 / 26
![Page 48: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/48.jpg)
Future Work
Much to still be done!
bottleneck: size of volume in memory
FFT: real-valued transforms
MPI: communication improvement via “ring” method
GPU: hardware interpolation
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 24 / 26
![Page 49: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/49.jpg)
Future Work
Much to still be done!
bottleneck: size of volume in memory
FFT: real-valued transforms
MPI: communication improvement via “ring” method
GPU: hardware interpolation
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 24 / 26
![Page 50: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/50.jpg)
Future Work
Much to still be done!
Data decomposition by reconstructed volume
bottleneck: size of volume in memoryFFT: real-valued transformsMPI: communication improvement via “ring” methodGPU: hardware interpolation
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 24 / 26
![Page 51: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/51.jpg)
Future Work
Much to still be done!
bottleneck: size of volume in memory
FFT: real-valued transforms
MPI: communication improvement via “ring” method
GPU: hardware interpolation
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 24 / 26
![Page 52: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/52.jpg)
Future Work
Much to still be done!
bottleneck: size of volume in memory
FFT: real-valued transforms
MPI: communication improvement via “ring” method
GPU: hardware interpolation
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 24 / 26
![Page 53: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/53.jpg)
Future Work
Much to still be done!
bottleneck: size of volume in memory
FFT: real-valued transforms
MPI: communication improvement via “ring” method
GPU: hardware interpolation
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 24 / 26
![Page 54: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/54.jpg)
Acknowledgements
University of Tennessee, Knoxville (UTK)Oak Ridge National Laboratory (ORNL)Joint Institute for Computational Sciences (JICS)National Science FoundationExtreme Science and Engineering Discovery Environment (XSEDE)Bridges system, Pittsburgh Supercomputing Center (PSC)
Dr. Rick Archibald (ORNL)Dr. Kwai Wong (UTK, JICS)
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 25 / 26
![Page 55: High Performance Computing for Neutron Tomography … · 2017-08-09 · High Performance Computing for Neutron Tomography Reconstruction A Parallel Approach to Filtered Backprojection](https://reader033.vdocuments.us/reader033/viewer/2022042212/5eb49691d1de2b57f92044c2/html5/thumbnails/55.jpg)
References
Zeng, Gensheng.Revisit of the Ramp FilterIEEE Trans Nucl Sci., 62(1):131–136, 2015.
A. Myagotin, et al.Efficient Volume Reconstruction for Parallel-Beam ComputedLaminography by Filtered Backprojection on Multi-Core ClustersIEEE Trans. Image Process., 22(12):5438–5439, 2013.
Zongpu Li, Cain Gantt HPC Neutron Tomography RECSEM 26 / 26