shape from contours and multiple stereo a hierarchical, mesh-based approach hendrik kück, wolfgang...
Post on 19-Dec-2015
214 views
TRANSCRIPT
Shape from Contours and Multiple Stereo
A Hierarchical, Mesh-Based Approach
Hendrik Kück, Wolfgang Heidrich, Christian Vogelgsang
The goalThe goal
The goalThe goal
Our approachOur approach
• Perform reconstruction using• Color• Object’s silhouettes
1. Create initial approximation based on silhouettes (Visual Hull)
2. Improve mesh using color information within an optimization approach
• Perform reconstruction using• Color• Object’s silhouettes
1. Create initial approximation based on silhouettes (Visual Hull)
2. Improve mesh using color information within an optimization approach
Talk outlineTalk outline
• Image data & preprocessing• Visual Hull
• Definition• Computation as a triangle mesh
• Image based mesh optimization• Results
• Image data & preprocessing• Visual Hull
• Definition• Computation as a triangle mesh
• Image based mesh optimization• Results
Image data & preprocessingImage data & preprocessing
• Calibrated images
• Foreground/ background segmentation
• Calibrated images
• Foreground/ background segmentation
Shape from SilhouettesShape from Silhouettes
• Silhouette + camera information: Silhouette Cone
• Completely contains real object
• Silhouette + camera information: Silhouette Cone
• Completely contains real object
Shape from SilhouettesShape from Silhouettes
• Silhouette + camera information: Silhouette Cone
• Completely contains real object
• Silhouettes can have holes
• Silhouette + camera information: Silhouette Cone
• Completely contains real object
• Silhouettes can have holes
Visual HullVisual Hull
• Definition: Largest volume that produces the same silhouettes as the object
• Construction:Intersection of the silhouette cones
• Definition: Largest volume that produces the same silhouettes as the object
• Construction:Intersection of the silhouette cones
Visual HullVisual Hull
Computing the Visual HullComputing the Visual Hull
• Extract Visual Hull using Extended Marching Cubes algorithm(Kobbelt, Botsch, Schwanecke, Seidel, 2001)
• Vertices lie exactly on isosurface• Can preserve sharp discontinuities
• Requires signed directed distance functionD(VH,x,d)• Distance from x to VH surface along direction d• Positive, if x outside VH, negative if inside
• Extract Visual Hull using Extended Marching Cubes algorithm(Kobbelt, Botsch, Schwanecke, Seidel, 2001)
• Vertices lie exactly on isosurface• Can preserve sharp discontinuities
• Requires signed directed distance functionD(VH,x,d)• Distance from x to VH surface along direction d• Positive, if x outside VH, negative if inside
Computing the Visual HullComputing the Visual Hull
• D(VH,x,d) = D( SCi ,x,d) = maxi D(SCi ,x,d)
• D(SCi ,x,d) can be efficiently computed in image space
• D(VH,x,d) = D( SCi ,x,d) = maxi D(SCi ,x,d)
• D(SCi ,x,d) can be efficiently computed in image space
Computing the Visual HullComputing the Visual Hull
14000 triangles 1400 triangles
The Optimization StageThe Optimization Stage
• Evolve triangle mesh into a shape that is• Consistent with color in the images• Consistent with silhouettes in images• Free from self-intersections• Smooth (low curvature)• Composed of well shaped triangles
• Evolve triangle mesh into a shape that is• Consistent with color in the images• Consistent with silhouettes in images• Free from self-intersections• Smooth (low curvature)• Composed of well shaped triangles
OptimizationOptimization
• Only geometry is optimized, not topology
• 3 Nv degrees of freedom
• Global optimization hopeless Use local per-vertex optimizations
• Locally minimize energy function E(vi) using 3D Simplex Method
• Iterate over vertices
• Only geometry is optimized, not topology
• 3 Nv degrees of freedom
• Global optimization hopeless Use local per-vertex optimizations
• Locally minimize energy function E(vi) using 3D Simplex Method
• Iterate over vertices
Per Vertex Energy FunctionPer Vertex Energy Function
color
silhouette
triangle shape
local curvature
self penetration
Color consistencyColor consistency
• Assumption: Lambertian reflectance• Surface points appear the same from all
viewing directions• Points on the real surface will project
onto pixels of the same color in all images that see them
• Projecting images onto the mesh• If surface is consistent, the color from
different images will match• Color cost term color mismatch (L2 norm)
• Assumption: Lambertian reflectance• Surface points appear the same from all
viewing directions• Points on the real surface will project
onto pixels of the same color in all images that see them
• Projecting images onto the mesh• If surface is consistent, the color from
different images will match• Color cost term color mismatch (L2 norm)
Color consistencyColor consistency
• Use OpenGL for color projection• Projective texture
mapping
• And for determining visibility• Shadow mapping
• Use OpenGL for color projection• Projective texture
mapping
• And for determining visibility• Shadow mapping
v v
v v
Color consistencyColor consistency
• Set up orthographic view of triangle fan around vertex v• Choose scale
according to sampling rate in the images
• Render fan to get samples of color and occlusion, once for each (relevant) image
• Set up orthographic view of triangle fan around vertex v• Choose scale
according to sampling rate in the images
• Render fan to get samples of color and occlusion, once for each (relevant) image
Silhouette consistencySilhouette consistency
• No part of the geometry may project outside any silhouette (must stay inside the visual hull)
Strongly penalize distance outside Visual Hull¼ maxi ( distance outside silhouettes in image i )
• Geometry may be smaller than Visual Hull• Where color does not provide enough
information, use Visual Hull as fallback solution Slightly penalize distance inside Visual Hull
¼ mini ( distance inside silhouettes in image i )
• No part of the geometry may project outside any silhouette (must stay inside the visual hull)
Strongly penalize distance outside Visual Hull¼ maxi ( distance outside silhouettes in image i )
• Geometry may be smaller than Visual Hull• Where color does not provide enough
information, use Visual Hull as fallback solution Slightly penalize distance inside Visual Hull
¼ mini ( distance inside silhouettes in image i )
Silhouette consistencySilhouette consistency
• Encode distance from silhouette in alpha channel of OpenGL textures• Project onto triangle fan along with
color & visibility
• Encode distance from silhouette in alpha channel of OpenGL textures• Project onto triangle fan along with
color & visibility
Multi-Resolution OptimizationMulti-Resolution Optimization
• Local minima are a problem,especially when• Triangle size is small compared to
geometric error• Texture frequencies are high
compared to geometric error
• Solution: Perform optimization at multiple resolutions
• Local minima are a problem,especially when• Triangle size is small compared to
geometric error• Texture frequencies are high
compared to geometric error
• Solution: Perform optimization at multiple resolutions
Multi-Resolution OptimizationMulti-Resolution Optimization
• Start with low resolution Visual Hull mesh
• Start with low resolution Visual Hull mesh
Multi-Resolution OptimizationMulti-Resolution Optimization
• Start with low resolution Visual Hull mesh
• Optimize until convergence
• Start with low resolution Visual Hull mesh
• Optimize until convergence
Multi-Resolution OptimizationMulti-Resolution Optimization
• Start with low resolution Visual Hull mesh
• Optimize until convergence
• Subdivide & optimize more
• Start with low resolution Visual Hull mesh
• Optimize until convergence
• Subdivide & optimize more
Multi-Resolution OptimizationMulti-Resolution Optimization
• Start with low resolution Visual Hull mesh
• Optimize until convergence
• Subdivide & optimize more
• Do it again, …
• Start with low resolution Visual Hull mesh
• Optimize until convergence
• Subdivide & optimize more
• Do it again, …
Final ResultsFinal Results
before
Final ResultsFinal Results
before after
Final ResultsFinal Results
before after
Thank you