![Page 1: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/1.jpg)
Recovering Geometric, Photometric and Kinematic Properties from
Images
Jitendra MalikComputer Science DivisionUniversity of California at
Berkeley
Work supported by ONR, Interval Research, Rockwell, MICRO, NSF, JSEP
Jitendra MalikComputer Science DivisionUniversity of California at
Berkeley
Work supported by ONR, Interval Research, Rockwell, MICRO, NSF, JSEP
![Page 2: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/2.jpg)
Physics of Image Formation
•Lighting•BRDFs•Shape and Spatial layout•Internal DOFs
•Lighting•BRDFs•Shape and Spatial layout•Internal DOFs
ImagesImages
![Page 3: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/3.jpg)
Solving inverse problems requires models
• Define suitable parametric models for geometry, lighting, BRDFs, and kinematics.
• Recover parameters using optimization techniques.
• Humans better at selecting models; computers at recovering parameters.
• Define suitable parametric models for geometry, lighting, BRDFs, and kinematics.
• Recover parameters using optimization techniques.
• Humans better at selecting models; computers at recovering parameters.
![Page 4: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/4.jpg)
But there will always be unmodeled detail…..
• Models are always approximate.• Adding more parameters doesn’t help;
data will be insufficient to recover these parameters.
• Models are always approximate.• Adding more parameters doesn’t help;
data will be insufficient to recover these parameters.
![Page 5: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/5.jpg)
Hybrid Approaches are best!• ANALYSIS
– use images to recover a subset of object parameters. These are chosen judiciously so that they can be recovered robustly
• SYNTHESIS
– render using appropriately selected images or subimages, transformed using the model.
• ANALYSIS
– use images to recover a subset of object parameters. These are chosen judiciously so that they can be recovered robustly
• SYNTHESIS
– render using appropriately selected images or subimages, transformed using the model.
![Page 6: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/6.jpg)
Talk Outline
• Geometry– Debevec, Taylor and Malik, SIGGRAPH 96
• Photometry– Yu and Malik, SIGGRAPH 98– Debevec and Malik, SIGGRAPH 97
• Kinematics– Bregler and Malik, CVPR 98
• Geometry– Debevec, Taylor and Malik, SIGGRAPH 96
• Photometry– Yu and Malik, SIGGRAPH 98– Debevec and Malik, SIGGRAPH 97
• Kinematics– Bregler and Malik, CVPR 98
![Page 7: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/7.jpg)
Modeling and Rendering Architecture from Photographs
Paul DebevecCamillo TaylorJitendra Malik
Computer Vision GroupComputer Vision GroupComputer Science Division Computer Science Division
University of California at BerkeleyUniversity of California at Berkeley
George Borshukov
Yizhou Yu
![Page 8: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/8.jpg)
Overview• Photogrammetric Modeling
– Allows the user to construct a parametric model of the scene directly from photographs
• Model-Based Stereo– Recovers additional geometric detail through
stereo correspondence
• View-Dependent Texture-Mapping– Renders each polygon of the recovered
model using a linear combination of three nearest views
![Page 9: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/9.jpg)
Our Modeling Method:• The user represents the scene as a
collection of blocks
• The computer solves for the sizes and positions of the blocks according to user-supplied edge correspondences
![Page 10: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/10.jpg)
Block ModelBlock Model User-Marked EdgesUser-Marked Edges Recovered ModelRecovered Model
![Page 11: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/11.jpg)
Arc de Triomphe
Modeled from five photographsby George Borshukov
![Page 12: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/12.jpg)
Surfaces of Revolution
Taj MahalTaj Mahalmodeled frommodeled from
one photographone photographby G. Borshukovby G. Borshukov
![Page 13: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/13.jpg)
Recovered ModelRecovered Model
PhotographPhotograph Synthetic ViewSynthetic View
![Page 14: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/14.jpg)
Recovering Additional Detailwith Model-Based Stereo
• Scenes will have geometric detail not captured in the model
• This detail can be recovered automatically through model-based stereo
![Page 15: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/15.jpg)
Scene with Geometric DetailScene with Geometric Detail
Approximate Block ModelApproximate Block Model
![Page 16: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/16.jpg)
Model-Based Stereo• Given a key and an offset image,
– Project the offset image onto the model
– View the model through the key camera Warped offset image
• Stereo becomes feasible between key and warped offset images because:
– Disparities are small
– Foreshortening is greatly reduced
![Page 17: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/17.jpg)
Key ImageKey Image Warped Offset ImageWarped Offset Image Offset ImageOffset Image
Disparity Disparity MapMap
![Page 18: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/18.jpg)
Synthetic Views of
Refined ModelFour images composited with
View-Dependent Texture Mapping
![Page 19: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/19.jpg)
Rendering with View-Dependent
Texture Mapping• Triangulate the view hemisphere
• For each polygon, determine which images viewed it from which angles
• Label each triangle vertex according to best viewed image
11
22 55
44
33
view hemisphere
![Page 20: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/20.jpg)
• To render, determine to which triangle the viewpoint belongs
• Compute Barycentric weights for the triangle vertices
• Render the polygon with a weighted average of the three vertex images
Rendering with View-Dependent
Texture Mapping
11
22 55
44
33
view hemisphere
![Page 21: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/21.jpg)
The Campanile (Debevec et al)
• 20 photographs used• approx. 1-2 weeks of modeling time.• Real time rendering
• 20 photographs used• approx. 1-2 weeks of modeling time.• Real time rendering
![Page 22: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/22.jpg)
![Page 23: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/23.jpg)
![Page 24: Recovering Geometric, Photometric and Kinematic Properties from Images](https://reader035.vdocuments.us/reader035/viewer/2022070405/56814023550346895dab834f/html5/thumbnails/24.jpg)
Recovered Campus Model
Campanile + 40 Buildings