interactive point-based modeling of complex objects from images
DESCRIPTION
Interactive Point-based Modeling of Complex Objects from Images. Pierre Poulin ( a,b ) Marc Stamminger ( a,c ) François Duranleau ( b ) Marie-Claude Frasson ( a ) George Drettakis ( a ) ( a ) REVES, INRIA Sophia Antipolis ( b ) DIRO, Université de Montréal ( c ) University of Erlangen. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/1.jpg)
Interactive Point-based Modeling of Complex Objects from Images
Pierre Poulin (a,b)Marc Stamminger (a,c)François Duranleau (b)
Marie-Claude Frasson (a)George Drettakis (a)
(a) REVES, INRIA Sophia Antipolis(b) DIRO, Université de Montréal
(c) University of Erlangen
![Page 2: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/2.jpg)
Modeling Complex Objects
![Page 3: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/3.jpg)
Modeling Complex Objects
• High visual complexity
• Time consuming
• Algorithms for specialized objects– e.g., plants, mountains, etc.
• Adaptive rendering
• Many applications need such objects
![Page 4: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/4.jpg)
Key Observations
• Extracting complex models from photos is a very powerful approach
• Point-based representation is very effective for complex models– Efficient display and storage
• User interaction is beneficial when extracting quality models– Specify where details are needed
– Resolve some ambiguities
![Page 5: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/5.jpg)
Image-based Point Modeling
• Images are very flexible– Reality-based (photos)– Acquisition is easy
![Page 6: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/6.jpg)
Image-based Point Modeling
• Points are very flexible– Fast rendering (hardware support)– Adaptive rendering for interactive display
Stamminger
![Page 7: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/7.jpg)
Image-based Point Modeling
• Points are very flexible– Hierarchical organization and levels of detail
Q-splat
![Page 8: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/8.jpg)
Image-based Point Modeling
• Points are very flexible– Visual quality– Many recent advances
Deussen
![Page 9: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/9.jpg)
Automatic ReconstructionImages
ReconstructionProcess
Constraints
3D Model
Image
![Page 10: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/10.jpg)
Interactive ReconstructionImages
ReconstructionProcess
Constraints
3D Model
Image
User
new imagesrequantizerecalibrate
![Page 11: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/11.jpg)
Interactive ReconstructionImages
ReconstructionProcess
Constraints
3D Model
Image
User
color comparisonsplausibility threshold
new depth mapszone of interest
![Page 12: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/12.jpg)
Interactive ReconstructionImages
ReconstructionProcess
Constraints
3D Model
Image
Userrevalidate the pointsrequest more pointsdecimate the points
jitter the pointssample with patterns
hole filling
![Page 13: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/13.jpg)
Interactive ReconstructionImages
ReconstructionProcess
Constraints
3D Model
Image
User
undo changesremove pointsadd polygons
![Page 14: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/14.jpg)
Interactive Reconstruction
• Interactive display– 6 M points/sec. on a PIII 1GHz with GeForce3
• Efficient reconstruction algorithm– Test more than 1K points/sec.
• Simple and intuitive controls– Direct interaction with the points
![Page 15: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/15.jpg)
Computer Vision Contributions
• 3D scanners
• Structured light
• Stereo – N-views
• Shape-from-X
• Volumetric
![Page 16: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/16.jpg)
Volumetric Reconstruction
• Voxel coloring and Space carving– If a voxel is impossible, carved out of object– Silhouettes, transparency, shading– Photo-consistency
SeitzKutulakos
![Page 17: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/17.jpg)
Image-based Polygon Modeling
• Academic: Façade, Rekon, Reality
• Industry: RealViz, Canoma, Photomodeler
Façade
![Page 18: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/18.jpg)
Image-based Polygon Modeling
• Small polygonal scene (30-100 polygons)
• Extracted textures and illumination
Boivin
![Page 19: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/19.jpg)
![Page 20: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/20.jpg)
Input Images (4/14)
![Page 21: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/21.jpg)
Input Images
• Digital camera: Canon EOS-DS30
• 1080x720 and 2166x1440
• Fixed aperture and shutter speed
• Try not to change zoom
• OpenGL and ray traced test scenes
![Page 22: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/22.jpg)
Camera Calibration
![Page 23: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/23.jpg)
Camera Calibration
• ImageModeler from RealViz
• Fiduciary marks placed around the object
• Interactive system
• Intrinsic and extrinsic camera parameters
![Page 24: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/24.jpg)
3D Zone of Interest
![Page 25: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/25.jpg)
Initial Random Points
![Page 26: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/26.jpg)
Initial Random Points
• Generated randomly within the envelope
• More specific patterns discussed later
• Projection of a point in each photo
• Gather colors
![Page 27: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/27.jpg)
Color Comparison
• Euclidean distance– RGB, CIE xy, CIE Luv, CIE Lab
– Speed vs. accuracy
• Color quantized images– Precomputed (ppmquantall or more sophisticated)
– Quantization only on projected zone of interest
– 32 to 128 colors
– Reduce shading variations
– Efficient test for color equality
![Page 28: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/28.jpg)
C: 25%B: 50%
Dominant Color
A: 100%
Plausibility
100%33%
with visibility
![Page 29: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/29.jpg)
Random Points with Depth Maps
![Page 30: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/30.jpg)
Depth Maps
• Computed from the current set of points
• Updated on user demand
• With depth maps, can raise the plausibility threshold
• Generate more points within the object
• Re-evaluation of previously generated points
![Page 31: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/31.jpg)
Clean-up Points
![Page 32: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/32.jpg)
Clean-up Points
• In general– Increase color threshold and re-evaluate
• With good depth maps– Project in each image– Reject if point visible and color too different
![Page 33: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/33.jpg)
Generate More Points
![Page 34: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/34.jpg)
Generate More Points
• Randomly
• Stratified sampling based on voxels
• Point decimation based on voxels
![Page 35: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/35.jpg)
Guide the Points
![Page 36: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/36.jpg)
Guide the Points
• Smaller 3D sphere of interest– Generate more points– Eliminate all points
• 3D flood fill for branching patterns
• Patterns for planar surfaces
• Patterns for boundary surfaces
![Page 37: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/37.jpg)
Filling with no Leaves
![Page 38: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/38.jpg)
Filling with Leaves
![Page 39: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/39.jpg)
Jitter the Points
![Page 40: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/40.jpg)
Reprojection
![Page 41: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/41.jpg)
Stepping through it again
![Page 42: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/42.jpg)
Results
Scene Images Resolution Colors Points
Fruit bowl 13 512x512 - -
Soldier 13 2160x1440 64 118K
Snack 8 1440x960 64 120K
Ficcus 13 2160x1440 64 150K
![Page 43: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/43.jpg)
Synthetic Fruit Bowl
ray tracingcolor points reprojection
![Page 44: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/44.jpg)
Toy Soldier
color pointscolor points reprojection
![Page 45: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/45.jpg)
Snack
![Page 46: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/46.jpg)
Snack
![Page 47: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/47.jpg)
Ficcus
![Page 48: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/48.jpg)
Conclusions
• Point-based reconstruction of complex objects from images
• Tight integration– 3D color point representation– User-driven and/or automatic reconstructions– Interactive display
• Flexible to integrate most advances in computer vision
![Page 49: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/49.jpg)
Findings
• First steps are encouraging, but objects are still of limited realism
• Information in photos is inspiring, but also difficult to analyse correctly
• How many things in a pixel?
• How many pixels and colors for an object?
![Page 50: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/50.jpg)
Future Work
• Video sequences
• High dynamic range photos
• Shadows and shading in color comparison
• Extraction of limited BRDFs
• 3D texture synthesis of materials
![Page 51: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/51.jpg)
Questions
• Did you…
• Is it…
• Can you…
• When…
• What…
• Where…
![Page 52: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/52.jpg)
User Interaction in Modeling
• Specify regions of interest, thresholds, validity
• Control the visual quality
• Iterative refining process
• Guide the solution
• Automatic or interactive process
• Interactive display (6 M points/sec. GeForce3)
![Page 53: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/53.jpg)
Image-based Point Modeling
• Difficulties with points – Visibility
• Holes in surfaces, size of points
• Filtering the representation and the texture
• Not our goal to fix these difficulties for now
![Page 54: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/54.jpg)
LOD in Graphics
• Environment maps
• Billboards
• Textured polygons
• Layer-depth images
• Light field / lumigraph
![Page 55: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/55.jpg)
3D Scanners
• Very good results in general
• Size of the scanner wrt object
• Costs
• Fixed illumination
![Page 56: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/56.jpg)
Stereo - N views
• Camera calibration• Epipolar constraints• Color matching• 3D position and color• Difficulties
– Holes and occlusions– Sharp edges, noise, shading
• Infinity of shapes in general• Targeted for object recognition and collision avoidance• Only recently goal of photo-realism
![Page 57: Interactive Point-based Modeling of Complex Objects from Images](https://reader035.vdocuments.us/reader035/viewer/2022062517/56813b5c550346895da45260/html5/thumbnails/57.jpg)
Shape-from-X
• Silhouettes
• Shadows
• Focus/defocus
• Motion
• Shading
• etc.