![Page 1: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/1.jpg)
A Novel Image Registration Pipeline for 3-A Novel Image Registration Pipeline for 3-D Reconstruction from Microscopy ImagesD Reconstruction from Microscopy Images
Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS;Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS;Tony Pan, MS; Metin Gurcan, PhD; Joel Saltz, MD, PhDTony Pan, MS; Metin Gurcan, PhD; Joel Saltz, MD, PhD
Department of Biomedical InformaticsOhio State University
![Page 2: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/2.jpg)
Creating Geometry from Images
Placenta
H+E Slides Alignment
SegmentationVisualization/Surface
Extraction
AperioScanner
![Page 3: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/3.jpg)
Registration
Registration between different modalities (e.g, MRI and PET)
Mapping of different samples to the same reference (e.g., brain mapping)
3-D reconstruction
![Page 4: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/4.jpg)
An optimization problem Initialization Point feature
matching Automatic vs.
manual
![Page 5: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/5.jpg)
Issues with automatic registration
Initialization Landmark- or image- based? Linear or nonlinear? Error metric / Meaningful morphology /
Domain specific knowledge Computation Structural constraints
![Page 6: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/6.jpg)
Fast initialization using landmarks
![Page 7: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/7.jpg)
S1
Fast initialization
S2
S2’
S1’
S3’Matching pairs:
(S1, S1’) (S1, S3’) (S2, S2’)
S1
S2 S2’
S1’
S3’
d12
d12’d13’
![Page 8: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/8.jpg)
Fast initialization
Maximum Clique Maximum Cyclic Structure
(S1, S1’, S2, S2’, θ, T)
![Page 9: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/9.jpg)
Fast initialization
Difference between two images.
Difference after the automatic initialization using region features.
Difference after the MMI algorithm.
![Page 10: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/10.jpg)
Fast initialization
![Page 11: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/11.jpg)
Registration of Large Images Using Landmarks
![Page 12: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/12.jpg)
Registration of Large Images Using High-Level Features
• No need to globally transform the image• Multi-level registration – rigid to nonrigid• Parallelizable – local operations
![Page 13: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/13.jpg)
Registration of Large Images Using Landmarks
![Page 14: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/14.jpg)
Registration of Large Images Using High-Level Features
• Point feature does NOT contain global information• For global transformation (e.g., rotation and
translation), we need “global” features such as high-level features.
• For nonlinear transformation, which is local, we need “local” features such as point features.
Global first, local second.
![Page 15: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/15.jpg)
Stacks of microscopy images
Principal component analysis (PCA) – based rigid registration
![Page 16: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/16.jpg)
Stacks of microscopy images
![Page 17: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/17.jpg)
Stacks of microscopy images
![Page 18: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/18.jpg)
Stacks of microscopy images
![Page 19: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/19.jpg)
Stacks of microscopy images
![Page 20: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/20.jpg)
Stacks of microscopy images
![Page 21: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/21.jpg)
Stacks of microscopy images
![Page 22: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/22.jpg)
3-D reconstruction vs. registration
• The current metric for registration is between two images and is just for the sake of perfect “registration”.
• We do “registration” for the sake of 3-D reconstruction.
• The structural constraint should be incorporated in the “cost function” instead of just used as a post processing or validation criterion.
New multiple image registration algorithm is needed!
![Page 23: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/23.jpg)
3-D reconstruction via registration
Feature extraction
Feature matching/ tracking
Trajectory generation
Trajectory smoothing and
adjustment
New location for the features in every image
Nonlinear transformation
for every image
Collective adjustment of trajectories
![Page 24: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/24.jpg)
3-D reconstruction via registration
Tracked trajectory
Smoothed trajectory
Registration moves the landmarks to the new locations.
![Page 25: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/25.jpg)
3-D reconstruction via registration
![Page 26: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/26.jpg)
3-D reconstruction via registration
![Page 27: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/27.jpg)
3-D reconstruction via Registration
![Page 28: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/28.jpg)
Summary, future work and discussion
• Technical issues related to automatic registration.• Two step approach to achieve “good” nonlinear
registration.• The paradigm for 3-D reconstruction is different with
pure registration.• New registration pipeline is proposed and implemented.
![Page 29: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/29.jpg)
Summary, future work and discussion
• Parallelization – especially in nonlinear transformation stage.
• Multiresolution / hierarchical approach.
![Page 30: A Novel Image Registration Pipeline for 3- D Reconstruction from Microscopy Images Kun Huang, PhD; Ashish Sharma, PhD; Lee Cooper, MS; Kun Huang, PhD;](https://reader035.vdocuments.us/reader035/viewer/2022062806/56649ee75503460f94bf8a2d/html5/thumbnails/30.jpg)
Acknowledgement
• BMI• Imaging group• Collaborators
Thank you !