gpu surgical training planning · • colorectal cancer is a priority research focus for the csiro...
TRANSCRIPT
![Page 1: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/1.jpg)
Harnessing the GPU for Surgical Training and Preoperative PlanningJosh Passenger, David Conlan, Hans de Visser, Mario Cheng, David Hellier, Juan Borda, Tobias Reichl, Christoph Russ, John Taylor and Olivier Salvado
![Page 2: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/2.jpg)
Introduction
• Surgical simulation and preoperative planning systems
• Need high performance computing
• Solve complex spatial and physically based problems• Often consume huge datasets
• CUDA provides the raw power we need
• Success stories so far
• Colonoscopy Simulator
• Non-linear soft tissue model
• Free-hand ultrasound from Computed Tomography (CT)• Large data-set visualisation
![Page 3: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/3.jpg)
Why is CSIRO developing a colonoscopy simulator?
• Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects
• Australia has the second highest incidence of colorectal cancer in the world, with less than 40% early detection
• Screening programs are being phased in slowly as positives usually require a colonoscopy with limited resources available
• Colonoscopy is a difficult procedure to master with hundreds of cases required to reach an expert skill level
![Page 4: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/4.jpg)
Why is CSIRO developing a colonoscopy simulator?
• Training currently happens on real patients with increased risk and cost over patient-free simulation based training
• Existing simulators rate poorly for realism and complexity and are under-utilised
• Development of a high fidelity, portable simulator has the potential to significantly reduce cost and risk of training and provide a platform for future certification
![Page 5: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/5.jpg)
The challenges
• Surgical simulation for colonoscopy poses many computational andtechnical challenges including:
• High density colon organ meshes required for realism• Physically based modelling of the colon (highly mobile)• Physically based modelling of the colonoscope (long and flexible)• Tool and organ tissue interactions (at haptic frequencies)• Realistic organ and tool surface shading (wet surfaces)• Collision detection between many dynamic systems• Simulation and interaction of viscous fluids within the colon• Realistic force rendering to the user
• With limited CPU resources we must rely heavily on the computational power of the GPU
• Not all processes are suitable for GPU acceleration
![Page 6: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/6.jpg)
The colon and colorectal cancer
![Page 7: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/7.jpg)
Anatomy of an Olympus Colonoscope
Insertion tip
Control handle
Insertion shaft
Umbilical socket
Umbilical
![Page 8: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/8.jpg)
Generating realistic cases from CT data
Patient CT scans are processed The colon surface is extracted
![Page 9: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/9.jpg)
Mesh Generation using Marching Cubes
• Mesh generated using Marching Cubes• Smoothed and decimated• No texture coordinates!
![Page 10: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/10.jpg)
Mesh Texture Coordinates
• Texture coordinates are required for surface shading• Allow mapping of 2D textures onto 3D meshes• We use tricky maths to pre-calculate texture coordinates
U
V
Texture mapping
![Page 11: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/11.jpg)
High fidelity visualisation GLSL shading
Virtual colonReal colon
![Page 12: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/12.jpg)
Integrated haptic device - EPFL
![Page 13: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/13.jpg)
Fluid and tissue simulation
![Page 14: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/14.jpg)
Soft tissue interaction
![Page 15: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/15.jpg)
Modelling of tools and surface bleeding
![Page 16: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/16.jpg)
Colon physical model editor
A fully featured colon physical model editor is under development to allow ranged definition of model parameters for inclusion in the clinical database.
![Page 17: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/17.jpg)
Mass Spring Model of the Colon
High to low mapping
• Simulates insufflation and preserves soft tissue appearance using high low mapping
• Solved completely on the GPU using CUDA
![Page 18: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/18.jpg)
We need to simulate endoscopic looping
• Requires an accurate endoscope model interacting with virtual colon model at haptic rates (>300 hertz)
![Page 19: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/19.jpg)
Dynamic endoscope intubation
![Page 20: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/20.jpg)
Software framework for CUDA memory
• CUDA Data Pointer system implemented as C++ templates• Provides management of CUDA device pointers with
associated host memory• Provides a low code overhead for uploading and downloading
data from the device• Bind() and Unbind() methods invoked before and after use• Allows binding of data to OpenGL vertex buffers automatically• Data pointers can be wrapped with textures in CUDA for better
cache utilisation• Greatly simplifies management of complex memory structures • Previous implementations showed that a large amount of code
was devoted to managing device data buffers• Most bugs were related to buffer misuse
![Page 21: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/21.jpg)
Mapping surface meshes to physical models
low resolution physical model
The problem: Mesh stretching and discontinuity during mapping
high resolution mesh
![Page 22: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/22.jpg)
Mapping surface meshes to physical models
• “Batch movement” of high resolution vertices in which low resolution mesh is recognizable
• Solution:• Calculate smooth spline (surface)
through low resolution nodes (“predict” high res. surface)
• Map high resolution vertices onto smooth surface
![Page 23: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/23.jpg)
Mapping surface meshes to physical models
• The benefit of our GPU mapping kernel is that small surface details are preserved while gross low resolution movements are smoothly interpolated onto the high resolution mesh.
• The mapping results are written directly to an OpenGL vertex buffer for rendering, no CPU transfer for high resolution meshes.
![Page 24: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/24.jpg)
Mass Spring Solver on the GPU
• Using masses and springs to represent flexible structures is cheap to compute but has limited physical accuracy.
• CUDA kernel structure:1. Calculate the spring forces based on
spring length2. Calculate the mass accelerations
based on forces exerted by attached springs
• Stores mass positions and spring configurations as textures on the GPU.
• The CUDA solver easily computes a 2K node structure in 0.2 ms which takes 10.6 ms to solve on the CPU.
Simple mass-spring network
![Page 25: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/25.jpg)
Non-Linear Soft Tissue Physical Model
• The mechanical response of organs is known to be non-linear
• Less expensive and simpler models have been used previously including Linear Finite Element
• Linear FEM solvers not only have limited accuracy but are only valid for small deformations (<10%)
• A better solution would capture all tissue properties and still solve at haptic rates (~1000 Hz)
• Previously these solutions were only available offline• Solution: use the GPU!
![Page 26: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/26.jpg)
TLED Non-Linear Soft Tissue Solver
• Previously an efficient CPU formulation for a non-linear FEM was proposed by Karol Miller
• This was accurate but did not run in real-time for large models
• Reformulated the solver into a GPGPU solver using CUDA with support for tetrahedral and hexahedral elements
• CUDA Kernel organisation• Kernel 1 (Elements) computes element stresses based on
current configuration and converts them into nodal force contributions
• Kernel 2 (Nodes) sums the previously calculated element force contributions at each node
![Page 27: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/27.jpg)
TLED Non-Linear Soft Tissue Solver
• Accurately representing soft tissue properties allows us to tackle all non-linear tissue properties
• This includes viscoelasticity where tissue strain is time dependent
Isotropic viscoelasticity Anisotropic viscoelasticity
![Page 28: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/28.jpg)
TLED Non-Linear Soft Tissue Solver
• Ratio of CPU to GPU solution time
• For larger models the GPU solver is up to 56.3 times faster than the CPU solver!
• Our solver can compute a hexahedral model with 30K nodes in 1 millisecond!
(DOF = 3 * node count)
![Page 29: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/29.jpg)
Prostate Cancer
• In developed countries 15% of all male cancers are prostate cancer
• Second most common cancer in Australia in males after skin cancer
• Second most common cause of cancer death• 14% prevalence for men over 70 years
![Page 30: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/30.jpg)
Transrectal needle biopsy
Sampling areas of the gland, where tumors are more prevalent
Use of transrectal ultrasound• Guide the position of the needle• Volume Measurement
![Page 31: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/31.jpg)
Simulation of transrectal needle biopsy
• Difficult procedure that requires training• A simulator will improve skills for a surgeon• Minimises potential risk for the patient
• Biopsy protocol yet to be defined• A simulator will potentially help to develop ways to
improve the procedure• Improve diagnosis efficacy
![Page 32: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/32.jpg)
Simulation of transrectal needle biopsy
TRUS simulator from 3D Computed Tomography (CT) Images
![Page 33: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/33.jpg)
TRUS Simulation
Ultrasound and Interaction with tissues• Acoustic properties
Probe (linear arrray)
Boundary tissueReflection
Refraction
Attenuation
Transmitted
x
Intensity
Tissue 2
Tissue1AbsorptionDiffusion
Transducer
A[dB]f[Hz]
2-18Mhz
![Page 34: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/34.jpg)
TRUS Simulation
Ultrasound and Interaction with tissues• Acoustic properties
Real TRUS ImageA[dB]f[Hz]
Tissue 2
Tissue1
Transducer2-18Mhz
Non-homogeneous tissues
Specular reflection
Scattering
Diffusion
![Page 35: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/35.jpg)
Linear scanning
Tiltscanning
Tilt Scanning with side-firing TRUS transducer
Rotational scanning
TRUS Simulation
• Geometry of acquisition
![Page 36: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/36.jpg)
Simulation of acquisition
![Page 37: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/37.jpg)
CT scan
Probe Position(x,y,z,φ)
![Page 38: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/38.jpg)
Geometry of acquisition
(0,0,0.9)
(0,1,0.4)(0,1,0)
v
(0,0,0)
(1,1,0)
(1,0,1)(0,0,1)w(1,0,0.5)
(1,1,0.2)
(1,1,1)
(1,0,0)u
CT scan
Probe Position(x,y,z,φ)
![Page 39: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/39.jpg)
Transmission
• Ray casting to compute the transmitted intensity
![Page 40: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/40.jpg)
Transmission
Transmission Absorption ReflectionNoise
![Page 41: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/41.jpg)
Method
Probe Position(x,y,z,φ)
Absorption
2D slice CT
AcousticImpedance
3D scanner CT
Reflection
Gradient 3D Noise
2D Noise
Blending
Acquisition Settings(gain, frequency,
geometry)
Transmission Image synthesis
Simulated US image
![Page 42: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/42.jpg)
Preoperative planning
• Using the GPU we can visualise large 3D medical image data sets
• Loads medical images onto GPU as 3D OpenGL textures• Slices the textures using quads aligned with the screen but
modulated 3D texture coordinates• Real-time ray casting is used for iso-surfacing, maximum
intensity projection and transfer function 3D rendering• Platform developed which allows point picking from any view• This data is then exported for use in building patient specific
finite element models and planning surgical procedures• Data is pre-processed to calculate an image gradient and this,
along with the original image intensity is uploaded as a 3D RGBA OpenGL image
![Page 43: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/43.jpg)
Interactive iso-surface generation using GLSL raycasting
![Page 44: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/44.jpg)
Transfer function rendering using GLSL
![Page 45: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/45.jpg)
Conclusion
• NVIDIA CUDA is allowing us to solve problems in real time• We are now using denser meshes, higher fidelity solvers, larger
data sets• We are tackling problems that were not possible previously due
to computational constraints• The possibilities include:
• High fidelity, patient free training systems for surgeons• Better preoperative data for surgeons
• Expected outcomes:• Reduced cost and improved safety with training in repeatable
virtual environments
• Improved surgical outcomes through advanced preoperative planning
• Increased patient safety!
![Page 46: GPU Surgical Training Planning · • Colorectal cancer is a priority research focus for the CSIRO National Preventative Health Flagship within a wide range of projects • Australia](https://reader035.vdocuments.us/reader035/viewer/2022071013/5fcbe71e6f71c530f0017fa2/html5/thumbnails/46.jpg)
Thank you
Project Team
Josh Passenger, Project Leader
Olivier Salvado, Team Leader
Contact UsPhone: 1300 363 400 or +61 3 9545 2176
Email: [email protected] Web: www.csiro.au
Hans de Visser, Scientist
David Conlan, Software Engineer
Olivier Comas, PhD student
Mario Cheng, PhD student
David Hellier, PhD student
Tobias Reichl, Trainee
Juan Borda, Trainee
Christoph Russ, Trainee