automatic registration of color images to 3d geometry computer graphics international 2009 yunzhen...
TRANSCRIPT
![Page 1: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/1.jpg)
Automatic Registration of Color Images to 3D Geometry
Computer Graphics International 2009
Yunzhen Li and Kok-Lim Low
School of ComputingNational University of Singapore
* Presented by Binh-Son Hua
![Page 2: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/2.jpg)
Problem StatementRange images
Color images from untracked camera
. . .
3D model Colored 3D model
Automatically register color images to 3D model
2
![Page 3: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/3.jpg)
MotivationsApplications of active range sensing
Manufacturing, cultural heritage modeling, etc.Photometric properties needed for visually-
realistic modelsOnly some range scanners can capture colorColor may not have required resolution
E.g. for close-up or zoomed-in views of paintingsView-dependent reflection requires many color
images from different directionsTherefore, better to capture color separately
However, impractical to manually register color images to 3D geometry
3
![Page 4: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/4.jpg)
Previous WorkFeature-based approaches
Match corresponding features in both color images and 3D model
Can be fully automatedRestricted to certain types of objects[Stamos & Allen, ICCV 2001], [Liu & Stamos, CVPR 2005]
Statistics-based approachesUsed only if reflected intensities of range sensing light
were recorded with range dataSensing light often not in visible light spectrum
Compute statistical dependence between color images and sensing light intensitiesMutual information, chi-square, cross-correlation
Camera calibrated & tracked, or co-locate with scanner[Williams et al, 2004], [Hantak & Lastra, 3DPVT 2006]
4
![Page 5: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/5.jpg)
Our ApproachColor images
. . .
Detailed scanned 3D model
Colored 3D model
Color mapping
Registration
Multiview geometry
reconstruction
Sparse 3D model
5
![Page 6: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/6.jpg)
Steps
1. Data acquisition
2. Multiview geometry reconstruction
3. Approximate registration of sparse model to detailed model
4. Registration refinement
5. Color mapping
6
![Page 7: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/7.jpg)
1. Data AcquisitionRange data
Laser range scanner
Color images Uncalibrated and untracked
digital cameraProject special light pattern
on large textureless surfacesImprove image feature
detection and MVG reconstruction
7
![Page 8: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/8.jpg)
2. MVG ReconstructionDetect and match features in color images
Use SIFT
Compute MVGStructure-from-motionIncrementally add a new image and apply
sparse bundle adjustment (SBA)
Result is a sparse 3D model3D point cloudCamera parameters
8
![Page 9: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/9.jpg)
2. MVG ReconstructionExample sparse 3D model
9
![Page 10: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/10.jpg)
3. Approximate RegistrationTo align sparse model with detailed model
Unknown relative scale and poseRegister one image in MVG to 3D model
User input 6 point correspondencesEstimated transformation propagated to other
views and 3D points in MVGSparse model only approximately aligned to
detailed modelError in user inputsError in MVGGeometric distortion in detailed model
10
![Page 11: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/11.jpg)
4. Registration RefinementNeed non-rigid alignment of MVG with detailed model
To overcome geometric distortion in range images
Registration refinementAutomatically detect planes in detailed modelIdentify 3D points in MVG near the planesRefine MVG to minimize distance
between 3D points and planesEasily incorporated into
sparse bundle adjustment
Better than using ICP algorithmTwo models are treated as rigid shapesCannot refine MVG
11
![Page 12: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/12.jpg)
4. Registration RefinementExample result
12
Before registration refinement
Afterregistration refinement
![Page 13: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/13.jpg)
5. Color MappingColors from different views can be used for
view-dependent renderingView-dependent texture mappingSurface light field
We simply want to assign a single color to each surface point, butSimple averaging blurs out detailsDifferent exposuresOcclusionsDepth boundariesVignetting and view-dependent reflection
13
![Page 14: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/14.jpg)
5. Color MappingUse weighted blending
Use lower weights near image and depth boundaries
Preserve fine details
14
With details
preservation
Without details
preservation
![Page 15: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/15.jpg)
5. Color MappingSmooth color and intensity transitions
With weighted blending
Without weighted blending
15
![Page 16: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/16.jpg)
ResultOffice scene
30 color images (7 with projected pattern)
16
![Page 17: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/17.jpg)
ConclusionAchieve accuracies within 3–5 pixels
everywhere on each imageNot reliant on detection of any specific type of
features in both color images and geometric model
Project light pattern to improve robustness of MVG
Better registration accuracy in face of geometric distortion
Effective color mapping method
17
![Page 18: Automatic Registration of Color Images to 3D Geometry Computer Graphics International 2009 Yunzhen Li and Kok-Lim Low School of Computing National University](https://reader036.vdocuments.us/reader036/viewer/2022062422/56649e695503460f94b65db0/html5/thumbnails/18.jpg)
AcknowledgementsThe Photo Tourism team
For sharing part of their code on MVGPrashast Khandelwal
For contribution to preliminary workSingapore Ministry of Education
For the funding
18