hybrid 3d/2d fully convolutional network design for ... · pros/cons of available architectures ......
TRANSCRIPT
![Page 1: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/1.jpg)
#CMIMI18#CMIMI18
Dr. Peter Chang, MDPresented By: Chanon Chantaduly
University of California Irvine
Hybrid 3D/2D Fully Convolutional Network Design for Volumetric Medical Datasets
![Page 2: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/2.jpg)
#CMIMI18
Challenges with High-Res Input Data
![Page 3: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/3.jpg)
#CMIMI18
Pros/Cons of Available Architectures
• 2D Only
• Easy and fast
• Lose 3D spatial information
• 3D only
• Slow and inefficient
• Too much spatial information
CNN
one slice
entire volume
CNN one slice
entire volume
![Page 4: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/4.jpg)
#CMIMI18
Hybrid 3D/2D Architecture
• Intuition
• Takes in a 3D input slice and predicts a 2D slice
• Gives enough 3D information for a more accurate prediction
• Takes advantage of the fact that convolutional neural networks (CNN) can take in any amount of slices
slab
one sliceCNN
![Page 5: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/5.jpg)
#CMIMI18
Hybrid 3D/2D Architecture
• Architecture
• Subsample X/Y with a ‘SAME’ padding convolutional operation of stride 2 (1x2x2)
• Subsample Z with a ‘VALID’ padding convolutional operation of stride 1 (2x1x1)
2x1x1
Z = 4Z = 5
X = 256Y = 256
1x2x2
X = 128
Y = 128‘SAME’
‘VALID’
stride 2
stride 1
![Page 6: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/6.jpg)
#CMIMI18
Hybrid 3D/2D Architecture
• Architecture
• Residual projection operation to match the 3D arm with 2D arm
Z = 5
‘VALID’
5x1x1 Z = 1
+
From the left 3D Contracting Arm
From the right 2D Expanding Arm
Z = 1
stride 1
![Page 7: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/7.jpg)
#CMIMI18
Hybrid 3D/2D Architecture
• Inference
• Pass in the entire volume, pad the top and bottom, then predict block by block
Z = 5predict prediction slab
slice we want to predict
padding
entire volume
![Page 8: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/8.jpg)
#CMIMI18
Hybrid 3D/2D Architecture
• Inference
• Pass in the entire volume, pad the top and bottom, then predict block by block
Z = 5 predict
entire volume
padding
prediction slab
slice we want to predict
![Page 9: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/9.jpg)
#CMIMI18
Hybrid 3D/2D Architecture
• Inference
• Pass in the entire volume, pad the top and bottom, then predict block by block
Z = 5
predict
entire volume
slice we want to predict
padding
prediction slab
![Page 10: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/10.jpg)
#CMIMI18
U-Net Hybrid 3D/2D Architecture
![Page 11: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/11.jpg)
#CMIMI18
FP Hybrid 3D/2D Architecture
![Page 12: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/12.jpg)
#CMIMI18
Results and Conclusion
• Outperformed the corresponding 2D version in all datasets tested
• Huge gain in detecting small findings and the edges of top/bottom slices
• Slight increase in training time and no significant increase in inference time
![Page 13: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/13.jpg)
#CMIMI18
Summary
• Any CNN with a contracting-expanding (encoder-decoder) structure can take in a 3D input of an arbitrary size to give a 2D output
• Subsample in the X/Y direction with a 1x2x2 with ‘SAME’ padding and a stride of 2
• Subsample in the Z direction with a 2x1x1 with ‘VALID’ padding and a stride of 1
• Residual projection operation
![Page 14: Hybrid 3D/2D Fully Convolutional Network Design for ... · Pros/Cons of Available Architectures ... • Takes in a 3D input slice and predicts a 2D slice • Gives enough 3D information](https://reader031.vdocuments.us/reader031/viewer/2022013003/5ff81b61096eab6b770c5b7b/html5/thumbnails/14.jpg)
#CMIMI18
Questions?
Chanon [email protected]
(714) 509-2524
Center for AI in Diagnostic MedicineUC Irvine Medical Centerhttp://caidm.som.uci.edu/@TheCAIDM