The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
Automatic Image Alignment for 3D Environment Modeling
Nathaniel WilliamsKok-Lim LowChad HantakMarc PollefeysAnselmo Lastra
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
2
Motivation: Real World Models
Forensics
Historical Preservation
Education
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
3
The Problem: Multiple Sensors• Digital Camera:
2D color images• Laser Scanner:
2D range map stores reflectance and depth
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
4
The Problem: Alignment
• Manual alignment is very time consuming♦ 5-10 minutes per image
• Modeling one room may require 10 scans and 100 images
• Multi-sensor alignment is difficult to automate♦ Differences in sampling EM spectrum,
illumination, occlusion, etc.
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
5
Our Approach
• Obtain an initial estimate of the correct alignment
• Recast 2D to 3D registration into a fast 2D image-based process
• Refine the initial alignment by optimizing the chi-square test
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
6
Previous Approaches
• Align medical images (e.g. CT, MR) by maximizing mutual information♦ Viola & Wells [1995], Collignon et al, [1995], etc.
• Correlate edges in image & range map♦ McAllister, Nyland, Popescu, Lastra, & McCue [1999]
• Align by comparing object silhouettes♦ Lensch, Heidrich, & Seidel [2000]
• Global optimization of chi-square test♦ Boughorbal et al [1999, 2000]
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
7
Data Acquisition
• Acquire range maps and color images of the environment♦ Need more scans in complex scenes
• Annotate all data with initial estimates of the alignment
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
8
Initial Pose Estimation [1]• Constrain the sensors’ positions
♦ Rigidly mount camera above scanner♦ Acquire from same center of projection
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
9
Initial Pose Estimation [2]• Track the sensors’ positions
♦ Use an optical tracker to measure the pose of the camera relative to the scanner
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
10
Camera & Tracker Calibration• Calculate the orientation of the
camera and scanner in the tracker’s coordinate frame
• Find the camera’s intrinsic parameters♦ Tape the lens in place
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
11
Data Preprocessing
• Correct for image distortion• Convert all range maps into a
single polygonal model♦ Texture map model with laser
reflectance
• Simplify polygonal model♦ Reduce millions of triangles by 99% or
more
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
12
Multi-Sensor Data Alignment• Recast 2D to 3D alignment into a
fast 2D image-based process• Visualize by projectively texture
mapping color image, given pose T
+
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
13
Image Comparison Framework
Reference Image r
Floating Image f
Extract intensity & down-sample
- performed once -
Extract from model given pose
T
- performed often -
Color Image
3D Model
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
14
Chi-Square Test
• Statistical measure of dependence between random variables
• Estimate joint probability density from a joint histogram
Floating ImageR
efe
rence
Im
age
Reference
Floating
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
15
Optimization
• Powell’s multidimensional direction set methods♦ Performs line minimizations given an initial
pose estimate and search direction
• The optimization is unconstrained, but the search is local given good initial estimates
TfrT T |,maxargˆ 2
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
16
Video of 3D Alignment
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
17
Results
• UNC Laboratory Model + 2 color images♦ Data captured from 3 different points of view♦ 6D optimization: 344 iterations, 28.5sec♦ Rendering=16% Readback=33% Chi-
square=51%Image Model Model + 2
Images
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
18
Results
• Global optimization can fail on complicated scenes
Monticello Library
UNC LaboratoryCorrect
Alignment
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
19
Conclusions
• Initial pose estimation improves the robustness of automatic alignment
• Acquiring data from a common COP♦ No occlusion makes the alignment more robust♦ Inflexible: camera is mounted on the scanner♦ Inexpensive: requires a simple bracket
• Decoupling the sensors♦ Flexible: collect more surface information♦ Expensive: tracking sensors takes more effort
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
20
Future Work
• Determine the ideal tracking method for initial alignment estimation♦ Criteria: portability, accuracy, and expense
• Experiment with other information metrics and optimization schemes
• Investigate error sources♦ Camera calibration, tracker calibration, etc.
• Implement image comparison on graphics hardware
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
21
Acknowledgements
• Kurtis Keller and John Thomas (UNC)
• Rich Holloway and 3rdTech, Inc.• The U.S. National Science
Foundation
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
22
The End
• Questions?
The UNIVERSITY of NORTH CAROLINA at CHAPEL HILL
23
References [1] P. K. Allen, I. Stamos, A. Troccoli, B. Smith, M. Leordeanu,and Y. C. Hsu. 3d modeling of historic sites using range andimage data. In Proceedings of International Conference onRobotics and Animation (ICRA), May 2003.[2] F. Boughorbal, D. L. Page, C. Dumont, and M. A. Abidib.Registration and integration of multi-sensor data for photorealisticscene reconstruction. In Proceedings of SPIE Conferenceon Applied Imagery Pattern Recognition, Oct. 1999.[3] C. Buehler, M. Bosse, L. McMillan, S. J. Gortler, and M. F.Cohen. Unstructured lumigraph rendering. In Proceedingsof ACM SIGGRAPH 2001, Computer Graphics Proceedings,Annual Conference Series, pages 425–432, Aug. 2001.[4] W.-C. Chen, L. Nyland, A. Lastra, and H. Fuchs. Acquisitionof large-scale surface light fields. In Proceedings of theSIGGRAPH 2003 Conference on Sketches & Applications, 2003.[5] R. O. Duda, P. E. Hart, and D. G. Stork. Pattern Classification(2nd Edition). Wiley-Interscience, 2000.[6] M. D. Elstrom. A stereo-based technique for the registrationof color and ladar images. Master’s thesis, University ofTennessee, Knoxville, Aug. 1998.[7] H. Lensch,W. Heidrich, and H.-P. Seidel. Automated textureregistration and stitching for real world models. In Proceedingsof Pacific Graphics 2000, pages 317–326, Oct. 2000.[8] K.-L. Low. Calibrating the hiball wand. Technical ReportTR02-018, Department of Computer Science, University ofNorth Carolina at Chapel Hill, Apr. 2002.[9] F. Maes, A. Collignon, D. Vandermeulen, G. Marchal, andP. Suetens. Multimodality image registration by maximizationof mutual information. In IEEE Transactions on MedicalImaging, volume 16, pages 187–198, Apr. 1997.[10] D. McAllister, A. Lastra, and W. Heidrich. Efficient renderingof spatial bi-directional reflectance distribution functions.In Proceedings of the 17th Eurographics/SIGGRAPHworkshop on graphics hardware, pages 79–88, Sept. 2002.[11] D. McAllister, L. Nyland, V. Popescu, A. Lastra, and C. Mc-Cue. Real-time rendering of real world environments. InProceedings of the 10th Eurographics Rendering Workshop ,pages 145–160, 1999.
[12] L. Nyland. Capturing dense environmental range informationwith a panning, scanning laser rangefinder. TechnicalReport TR98-039, Department of Computer Science, Universityof North Carolina - Chapel Hill, Jan. 5 1999.[13] W. H. Press, B. P. Flannery, S. A. Teukolsky, and W. T. Vetterling.Numerical Recipes in C: The Art of Scientific Computing .Cambridge University Press, Cambridge (UK) andNew York, 2nd edition, 1992.[14] M. Sallinen and T. Heikkil¨a. A simple hand-eye calibrationmethod for a 3d laser range sensor. In Advances in NetworkedEnterprises, pages 421–430, 2000.[15] I. Stamos and P. K. Allen. Automatic registration of 2-Dwith 3-D imagery in urban environments. In Proceedingsof the Eighth International Conference On Computer Vision(ICCV-01), pages 731–737, July 2001.[16] E. Trucco and A. Verri. Introductory Techniques for 3-DComputer Vision. Prentice Hall, 1998.[17] P. Viola and W. M. Wells III. Alignment by maximization ofmutual information. In Proceedings of the 5th InternationalConference on Computer Vision, pages 16–23, 1995.[18] R. Wang and D. Luebke. Efficient reconstruction of indoorscenes with color. In Proceedings of the 4th InternationalConference on 3D Imaging and Modeling (3DIM), 2003.[19] G. Welch, G. Bishop, L. Vicci, S. Brumback, K. Keller, andD. Colucci. The hiball tracker: high-performance wide-areatracking for virtual and augmented environments. In Proceedingsof the ACM symposium on Virtual reality softwareand technology, 1999.[20] S. You, U. Neumann, and R. Azuma. Orientation trackingfor outdoor augmented reality registration. IEEE ComputerGraphics and Applications, 19(6):36–42, Nov./Dec. 1999.[21] Z. Zhang. Flexible camera calibration by viewing a planefrom unknown orientations. In Proceedings of the 7th IEEEInternational Conference on Computer Vision (ICCV), pages666–673, 1999.