lecture g: multiple-view reconstruction from scene knowledge
DESCRIPTION
Breakthroughs in 3D Reconstruction and Motion Analysis. Lecture G: Multiple-View Reconstruction from Scene Knowledge. Yi Ma. Perception & Decision Laboratory Decision & Control Group, CSL Image Formation & Processing Group, Beckman Electrical & Computer Engineering Dept., UIUC - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/1.jpg)
September 15, 2003 ICRA2003, Taipei
Lecture G: Multiple-View Lecture G: Multiple-View Reconstruction from Scene KnowledgeReconstruction from Scene Knowledge
Yi MaYi MaPerception & Decision Laboratory
Decision & Control Group, CSLImage Formation & Processing Group, BeckmanElectrical & Computer Engineering Dept., UIUC
http://decision.csl.uiuc.edu/~yima
Breakthroughs in 3D Reconstruction and Motion Analysis
![Page 2: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/2.jpg)
September 15, 2003 ICRA2003, Taipei
Symmetry is ubiquitous in man-made or natural environments
INTRODUCTION: Scene knowledge and symmetry
![Page 3: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/3.jpg)
September 15, 2003 ICRA2003, Taipei
INTRODUCTION: Scene knowledge and symmetryParallelism (vanishing point) Orthogonality
Congruence Self-similarity
![Page 4: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/4.jpg)
September 15, 2003 ICRA2003, Taipei
INTRODUCTION: Wrong assumptions
Ames room illusion Necker’s cube illusion
![Page 5: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/5.jpg)
September 15, 2003 ICRA2003, Taipei
INTRODUCTION: Related literature
Cognitive Science: Figure “goodness”: Gestalt theorists (1950s), Garner’74, Chipman’77, Marr’82, Palmer’91’99…
Computer Vision (isotropic & homogeneous textures): Gibson’50, Witkin’81, Garding’92’93, Malik&Rosenholtz’97, Leung&Malik’97…
Mathematics: Hilbert 18th problem: Fedorov 1891, Hilbert 1901, Bieberbach 1910, George Polya 1924, Weyl 1952
Detection & Recognition (2D & 3D): Morola’89, Forsyth’91, Vetter’94, Mukherjee’95, Zabrodsky’95, Basri & Moses’96, Kanatani’97, Sun’97, Yang, Hong, Ma’02
Reconstruction (from single view): Kanade’81, Fawcett’93,Rothwell’93, Zabrodsky’95’97, van Gool et.al.’96, Carlsson’98, Svedberg and Carlsson’99, Francois and Medioni’02, Huang, Yang, Hong, Ma’02,03
![Page 6: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/6.jpg)
September 15, 2003 ICRA2003, Taipei
MUTIPLE-VIEW MULTIPLE-OBJECT ALIGNMENT• Scale alignment: adjacent objects in a single view• Scale alignment: same object in multiple views
SUMMARY: Problems and future work
ALGORITHMS & EXAMPLES• Building 3-D geometric models with symmetry • Symmetry extraction, detection, and matching • Camera calibration
SYMMETRY & MULTIPLE-VIEW GEOMETRY• Fundamental types of symmetry• Equivalent views• Symmetry based reconstruction
Multiple-View Reconstruction from Scene Knowledge
![Page 7: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/7.jpg)
September 15, 2003 ICRA2003, Taipei
SYMMETRY & MUTIPLE-VIEW GEOMETRY
• Why does an image of a symmetric object give away its structure?
• Why does an image of a symmetric object give away its pose?
• What else can we get from an image of a symmetric object?
![Page 8: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/8.jpg)
September 15, 2003 ICRA2003, Taipei
Equivalent views from rotational symmetry
90O
![Page 9: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/9.jpg)
September 15, 2003 ICRA2003, Taipei
Equivalent views from reflectional symmetry
![Page 10: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/10.jpg)
September 15, 2003 ICRA2003, Taipei
Equivalent views from translational symmetry
![Page 11: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/11.jpg)
September 15, 2003 ICRA2003, Taipei
GEOMETRY FOR SINGLE IMAGES – Symmetric Structure
Definition. A set of 3-D features S is called a symmetric structureif there exists a nontrivial subgroup G of E(3) that acts on it such that for every g in G, the map
is an (isometric) automorphism of S. We say the structure S has agroup symmetry G.
![Page 12: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/12.jpg)
September 15, 2003 ICRA2003, Taipei
GEOMETRY FOR SINGLE IMAGES – Multiple “Equivalent” Views
![Page 13: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/13.jpg)
September 15, 2003 ICRA2003, Taipei
GEOMETRY FOR SINGLE IMAGES – Symmetric Rank Condition
Solving g0 from Lyapunov equations:
with g’i and gi known.
![Page 14: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/14.jpg)
September 15, 2003 ICRA2003, Taipei
THREE TYPES OF SYMMETRY – Reflective Symmetry
Pr
![Page 15: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/15.jpg)
September 15, 2003 ICRA2003, Taipei
THREE TYPES OF SYMMETRY – Rotational Symmetry
![Page 16: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/16.jpg)
September 15, 2003 ICRA2003, Taipei
THREE TYPES OF SYMMETRY – Translatory Symmetry
![Page 17: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/17.jpg)
September 15, 2003 ICRA2003, Taipei
SINGLE-VIEW GEOMETRY WITH SYMMETRY – Ambiguities
“(a+b)-parameter” means there are an a-parameter family of ambiguity in R0 of g0 and a b-parameter family of ambiguity in T0 of g0.
P
Pr
NPr
![Page 18: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/18.jpg)
September 15, 2003 ICRA2003, Taipei
Symmetry-based reconstruction (reflection)
Reflectional symmetry
Virtual camera-camera
1 2
3 4(3)(4)
(2) (1)
![Page 19: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/19.jpg)
September 15, 2003 ICRA2003, Taipei
Epipolar constraint
Homography
1 2
3 4(3)(4)
(2) (1)
Symmetry-based reconstruction
![Page 20: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/20.jpg)
September 15, 2003 ICRA2003, Taipei
2 pairs of symmetric image points
Decompose or to obtain
Solve Lyapunov equation
to obtain and then .
Recover essential matrix or homography
Symmetry-based reconstruction (algorithm)
![Page 21: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/21.jpg)
September 15, 2003 ICRA2003, Taipei
Symmetry-based reconstruction (reflection)
![Page 22: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/22.jpg)
September 15, 2003 ICRA2003, Taipei
Symmetry-based reconstruction (rotation)
![Page 23: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/23.jpg)
September 15, 2003 ICRA2003, Taipei
Symmetry-based reconstruction (translation)
![Page 24: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/24.jpg)
September 15, 2003 ICRA2003, Taipei
ALIGNMENT OF MULTIPLE SYMMETRIC OBJECTS
?
![Page 25: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/25.jpg)
September 15, 2003 ICRA2003, Taipei
Pick the image of a point on the intersection line
Correct scales within a single image
![Page 26: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/26.jpg)
September 15, 2003 ICRA2003, Taipei
Correct scale within a single image
![Page 27: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/27.jpg)
September 15, 2003 ICRA2003, Taipei
Correct scales across multiple images
![Page 28: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/28.jpg)
September 15, 2003 ICRA2003, Taipei
Correct scales across multiple images
![Page 29: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/29.jpg)
September 15, 2003 ICRA2003, Taipei
Correct scales across multiple images
![Page 30: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/30.jpg)
September 15, 2003 ICRA2003, Taipei
Image alignment after scales corrected
![Page 31: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/31.jpg)
September 15, 2003 ICRA2003, Taipei
ALGORITHM: Building 3-D geometric models
1. Specify symmetric objects and correspondence
![Page 32: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/32.jpg)
September 15, 2003 ICRA2003, Taipei
Building 3-D geometric models
2. Recover camera poses and scene structure
![Page 33: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/33.jpg)
September 15, 2003 ICRA2003, Taipei
Building 3-D geometric models
3. Obtain 3-D model and rendering with images
![Page 34: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/34.jpg)
September 15, 2003 ICRA2003, Taipei
ALGORITHM: Symmetry detection and matching
Extract, detect, match symmetric objects acrossimages, and recover the camera poses.
![Page 35: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/35.jpg)
September 15, 2003 ICRA2003, Taipei
1.Color-based segmentation (mean shift) 2. Polygon fitting
Segmentation & polygon fitting
![Page 36: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/36.jpg)
September 15, 2003 ICRA2003, Taipei
Symmetry verification & recovery 3. Symmetry verification (rectangles,…)
4. Single-view recovery
![Page 37: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/37.jpg)
September 15, 2003 ICRA2003, Taipei
Symmetry-based matching
5. Find the only one set of camera poses that are consistent with all symmetry objects
![Page 38: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/38.jpg)
September 15, 2003 ICRA2003, Taipei
MATCHING OF SYMMETRY CELLS – Graph Representation
123
123
36 possible matchings
(1,2,1, g)Cell in image 1
Cell in image 2
# of possible matching
Camera transformation
The problem of finding the largest set of matching cells is equivalent to the problem of finding the maximal complete subgraphs (cliques) in the matching graph.
![Page 39: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/39.jpg)
September 15, 2003 ICRA2003, Taipei
Camera poses and 3-D recovery Side view Top view Generic view
Length ratio Reconstruction Ground truthWhiteboard 1.506 1.51Table 1.003 1.00
![Page 40: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/40.jpg)
September 15, 2003 ICRA2003, Taipei
Multiple-view matching and recovery (Ambiguities)
![Page 41: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/41.jpg)
September 15, 2003 ICRA2003, Taipei
Multiple-view matching and recovery (Ambiguities)
![Page 42: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/42.jpg)
September 15, 2003 ICRA2003, Taipei
ALGORITHM: Calibration from symmetry
(vanishing point)
Calibrated homography
Uncalibrated homography
![Page 43: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/43.jpg)
September 15, 2003 ICRA2003, Taipei
ALGORITHM: Calibration from symmetry
Calibration with a rig is also simplified: we only need to know that there are sufficient symmetries, not necessarily the 3-D coordinates of points on the rig.
![Page 44: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/44.jpg)
September 15, 2003 ICRA2003, Taipei
SUMMARY: Multiple-View Geometry + Symmetry
Multiple (perspective) images = multiple-view rank condition
Single image + symmetry = “multiple-view” rank condition
Multiple images + symmetry = rank condition + scale correction
Matching + symmetry = rank condition + scale
correction + clique identification
![Page 45: Lecture G: Multiple-View Reconstruction from Scene Knowledge](https://reader035.vdocuments.us/reader035/viewer/2022070500/5681684c550346895dde4a43/html5/thumbnails/45.jpg)
September 15, 2003 ICRA2003, Taipei
Multiple-view 3-D reconstruction in presence of symmetry
• Symmetry based algorithms are accurate, robust, and simple.
• Methods are baseline independent and object centered.
• Alignment and matching can and should take place in 3-D space.
• Camera self-calibration and calibration are simplified and linear.Related applications
• Using symmetry to overcome occlusion.
• Reconstruction and rendering with non-symmetric area.
• Large scale 3-D map building of man-made environments.
SUMMARY