lecture 3 multi-view shape reconstruction dana cobzas pims postdoc
Post on 18-Dec-2015
220 views
TRANSCRIPT
![Page 1: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/1.jpg)
Lecture 3
Multi-view shape reconstruction
Dana CobzasPIMS Postdoc
![Page 2: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/2.jpg)
Shape reconstructionGiven A set of images (views) of an object / scene Camera calibration information [Light calibration information]
Find the surface that best agrees with the input images.
…
Approach: chose a surface representation define a photo-consistency function
[in practice photo-consistency+regularization]
solve the following minimization
X,S)(X
SXS
dXX )(min
![Page 3: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/3.jpg)
Photo-consistency function(al)
Based on image cues (shading, stereo, silhouettes, …)
)(X
Extension SFS/PS to multi-view:
SFSMove camera Move object
PS
Needs camera/light calibration !
![Page 4: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/4.jpg)
Surface representation
Image-centered Depth/disparity w.r. to image
plane
Image plane
3D point
Voxels
Level sets (implicit)
Mesh
Depth with respect to a base mesh Local patches
3D plane
Object-centered
time
Partial object reconstr.
Limited resolution
Viewpoint dependent
![Page 5: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/5.jpg)
Comparison of different methods
2 datasets
No light
(moving camera)
![Page 6: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/6.jpg)
Volumetric representation
Object : collection of voxelsNormals : ?Method: carve away voxels that are not photo-consistent with images
(purely discrete)Regularization: no way of ensuring smoothnessVisibility: ensured by the order of traversal (in general a problem!)
![Page 7: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/7.jpg)
Disparity/Depth map
Reference image plane
3D pointReferenceimage
Object (surface) :
Normals :
Method: find f that best agrees with the input images (minimize the cost functional integrated over the surface)
Regularization: smoothness on
Visibility: ? (mesh defined on image plane + Zbuffering)
)),(,,(),( yxfyxyxs
),( yx
),( yxf
T
y
f
x
f
y
s
x
s
1,,n
xyf
dxdyffyx ),,,(min
),( yx
),( yxf
2),( yxfsmooth f
![Page 8: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/8.jpg)
Depth w.r. base mesh
Object (surface) : , d – displacement direction (displacement map)
Normals : local (per triangle) transform to global CS
Method:
Regularization: smoothness on (local / global)
Visibility: ? (fine mesh to connect points on each plane + Zbuffering)
How to deal with boundaries ?
dXX fB
X
BX
df
T
y
f
x
f
1,,n
dxdyffX B
Xf B
),,(min
f
![Page 9: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/9.jpg)
Mesh
Object (surface) : mesh vertices
Normals : interpolated
Method: move vertices along interpolated normals based on photo-consistency of neighboring triangles.
Regularization: smooth normals
Visibility: Zbuffering
332211 nnnn
)()()(,,
jiikj
jk
j
jvvvvn
v
332211 vvvX 11 nv
22 nv 33 nv
Topologiocal changes ?
![Page 10: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/10.jpg)
Mixed Representations:Local patches
Mixed approaches : patches on voxels for a finer surface representation mesh on pre-computed voxel correlation (potential fields) [Esteban and Schmitt CVIU 2005] (depth on base mesh)
Patches : arbitrary : [Zheng, Paris et al. IJCV2006] quadratic : surfels [Carceroni and Kutulakos IJCV 2002]
![Page 11: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/11.jpg)
Optimization
Disparity map
Depth w.r.plane
Level sets
Mesh
Voxels
Discrete methods
Graph-cutsBelief propagationDynamic programming…
Voxel carving (coloring)
Continuous methods
Non-linear optimizationVariational calculus
![Page 12: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/12.jpg)
Discrete Voxel carving
Graph cut techniques
Continuous Variational and level set techniques
Mesh-based reconstruction
Summary
![Page 13: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/13.jpg)
Volumetric reconstruction
Method : Carve voxels that are not consistent with images (according to a chosen photo-consistency score).
OR
Assign colors to voxels consistent with the input images.
(color + opacity)
N3 voxelsC colors
3NCall scenes
Photo-consist. scenes
True scene
Not unique solution
Order of traversal v. important
![Page 14: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/14.jpg)
Shape from Silhouette
Binary images
Back project each silhouette 3D cone.Carve all voxels outside the cone. Result:
Reconstruction contains the true scene, but not the same (no concavities) Not photo-consistent (only to binary images).In the limit : visual hull
[Martin PAMI 91][Szeliski 93]
![Page 15: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/15.jpg)
Voxel coloring
Each voxelProject in images and correlateColor if consistent
Visibility !
Depth ordering : Single visibility ordering for each view(Restriction in camera placement)
Traversal order
[Seitz,Dyer CVPR97]Plane sweep [Collins CVPR 96]
No scene point contained within the convex hull of the cameras
![Page 16: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/16.jpg)
Voxel coloring : results
Input image Results
[Seitz,Dyer CVPR97]
![Page 17: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/17.jpg)
Space carvingIn general a view independent order might not exist
Space carving [Kutulakos, Seitz ICCV 99, IJCV 2002] initialize a volume containing the scene Choose a voxel on the surface of the scene Project in all visible images Carve if not consistent Repeat until convergence
Consistency:The resulting shape is photo-consistent(all inconsistent voxels are removed)
Convergence:Carving converges to a non-empty shape(a point on the true surface is never removed)
Photo-hull
![Page 18: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/18.jpg)
Space carving : photo hull
Initial volume andtrue scene
Photo-hull
Photo-hull = union of all photo-consistent shapes
Basic algorithm : requires a difficult update procedure (visibility computation after carving a voxel)
Multi-pass plane sweep : Sweep plane in each 6 directionsConsider active only cameras on one side of the plane
![Page 19: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/19.jpg)
Space carving : results
[Kutulakos, Seitz ICCV 99, IJCV 2002]
![Page 20: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/20.jpg)
Other volumetric representations
- Inaccurate
+ Triangulate w. marching cubes
+ Accurate + Moderatly accurate
+ Fast
+ Marching intersections
Voxel-based Image ray based Axis-aligned
![Page 21: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/21.jpg)
Graph cuts for multi-view reconstruction Discrete surface reconstruction Graph cut Graph cuts as hypersurfaces Example: [Paris, Sillion, Quan IJCV 05]
Types of energies minimized with graph cut
[Kolmogorov Zabih ECCV 2002]
Graph cuts for multi-labeling
![Page 22: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/22.jpg)
Reconstruction as labeling
S XN
smooth
S
dataS
dYdXYXdXX)(
),()(min
Discrete formulation: surface representation
labels
Find a set of labels that minimize
Pp
),...,,...,( 1 Pp ffff
p Nqp
qpsmoothpdata ffffE},{
),()()(
pf
Ex: disparity map voxels pixels voxels disparities occupancy
,...}1,0{ }1,0{
Notes: NP hard Can be solved using MRF energy minimization methods
graph cuts, dynamic programming, belief propagation,
simulated annealing …
Photo-consistency Smoothness
![Page 23: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/23.jpg)
Graph cutsOriented graph nodes edges
EVG ,
V , source s , sink t E , edge capacity w(p,q) (non-negative)
Cut C={S,T} partition of nodes into two disjoint sets such that sS, t T
Cost of the cut
Minimum cut cut that has minimum cost among all cuts (binary labeling)
Maximum flow maximum amount of liquid that can be sent from the source to the sink interpreting edges as pipes with capacity w.
Polynomial time algorithmsAugmenting paths [Ford & Fulkerson, 1962]Push-relabel [Goldberg-Tarjan, 1986]
TqSp
qpw,
),(
![Page 24: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/24.jpg)
Energy minimization via graph cutsMotivation: Geometric interpretation cut = hypersurface in N-D space embedding the corresponding graph
used to compute optimal hypersurface Powerful energy minimization tool for a large class of binary
and non-binary energiesglobal minimum; strong local minimum
Surface reconstruction: Chose a surface representation Define a graph (nodes, weights) such that the cost of a cut corresponds to the surface energy function.
How to find global labeling using graph cut ? What kind of energy can be minimized with a graph cut ?
![Page 25: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/25.jpg)
Graph Cuts as Hypersurfaces
s
tcut
p
“cut”
x
y
lab
els
Graph fully embedded in the working geometric space Feasible cut = separated hypersurface in the embedding
continuous manifold
![Page 26: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/26.jpg)
Example of geometric graph
p Nqp
qppp fffDfE},{
)()(
Disparity map: pixel label = disparity),( yxp
[Paris, Sillion, Quan: A surface reconstruction method using global graph cut optimization IJCV 05]
source
sink
d1
d3
D1(d1)
d2
d4
D4(d1)
)()()( 21 ppp fpIpIfD
D3(d2) D4(d2)
D4(d3)
D4(d4)
D2(d2)D1(d2)
’ ’’
’’ ’
Graph Surface
12' ff
)(
)()()(
4434
33323441min
dDdd
dDdDdddDE
pfp
f1=d4f2=d3f3=d3f4=d4
![Page 27: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/27.jpg)
ResultsConvex smoothing – global solution
[Paris, Sillion, Quan IJCV 05, ACCV 04]
![Page 28: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/28.jpg)
Graph-cuts and hypersurfaces Geometric graph for stereo
Convex smoothing – global convergence
[Ishikawa, Geiger ECCV 1998][Roy,Cox ICCV 98, IJCV 1999]
Connection between cuts and hypersurfaces in continuous spaces [ Boykov, Kolmogorov: Computing geodesics and minimal surfaces via
graph cuts, ICCV 2003]Show how to build a grid graph and sets the weights such that the cost of cuts is arbitrarily close to the area of corresponding surface for any anisotropic Riemannian metric. Graph cut to find globally minimum surfaces (like level sets) under arbitrary Riemannian metric.
Riemannian metric (varying tensor)Euclidian metric (ct)
A
Bdistance
map
A
Bdistance
map
![Page 29: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/29.jpg)
Types of energies
p Nqp
qpp ffVfDfE},{
),()()(
Convex+ global convergence- oversmooth
false
trueb
ffffV qppqqp
,0
,1)(
][),(
2),(
),(
qpqp
qpqp
ffffV
ffffV
2,min),(
,min),(
qpqp
qpqp
ffTffV
ffTffV
V(Δf)
Δf=fp-fq
Pottspiecewise planarbinary graph
Preserves discontinuitiesNP hard? convergence
V(Δf)
Δf=fp-fq
V(Δf)
Δf=fp-fq
linear
Potts
![Page 30: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/30.jpg)
Exact multi-labelingLinear and convex smoothing (interaction) energy Geometric graphs.
s
tMulti-scan-line stereo
[Roy & Cox 1998, 1999]
[Ishikawa & Geiger 1999] (occlusion handling)
“Linear” interaction energy
[Ishikawa & Geiger 1998]
[Boykov, Veksler, Zabih 1998]
Convex interaction energy
[Ishikawa 2000, 2003]
![Page 31: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/31.jpg)
Binary graphs[Kolmogorov, Zabih : What energy functions can be minimized via graph cuts? ECCV 2002]
Npq
fqfppqp
pp fDfE )()(
},{ tsf p
Complete characterization of energies that can be minimized with graph cut : E(f) can be minimized by s-t graph cuts
pqw
n-links
s
t a cut)(tDp
t-lin
k
t-link
)(sDp
),(),(),(),( stVtsVttVssV
Large class of energies (Potts, metric …)BUT multi-view reconstruction is a multi-labeling problem !
![Page 32: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/32.jpg)
Approximate multi-labeling
Binary Potts energyExtended to multi-labelingNP hard ( 3 labels)
pqw
n-links
s
t cut
t-lin
k
t-link
currentlabel
a
[ Boykov et al: Fast approximate energy via graph cuts, PAMI 2001]Expansion approximate
solution Ideea: break optimization into a
set of binary s-t cut problems Each iteration – consider one
label a Binary cut: some labels are
relabeled with a; the others remain unchanged
![Page 33: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/33.jpg)
expansionProperties
Guaranteed approximation quality within a factor of 2 from the global minima (Potts model)
Applies to a wide class of energies with robust interactions Potts model
“Metric” interactions
“Submodular” (regular) interactions
simulated annealing, 19 hours, 20.3% err
a-expansions (BVZ 89,01)90 seconds, 5.8% err
normalized correlation,start for annealing, 24.7% err
![Page 34: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/34.jpg)
Graph cuts : state of the art
Optimization of first-order properties of segmentation boundary (Riemannian length/area, flux of a vector field)
Can’t optimize curvature of the boundary (for now)
Class of energies that can be minimized exactly binary energies with regular (sub-modular) interactions multi-label (non-binary) energies with “convex” interactions excludes robust discontinuity-preserving interactions
Guaranteed quality approximation algorithms for multi-label energies with discontinuity-preserving interactions… Potts model of interactions Metric interactions Regular (sub-modular) interactions
[ Boykov CVPR 05 Tutorial]
![Page 35: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/35.jpg)
Graph cut Example[ Vogiatzis, Hermandez-Esteban, Torr, Cipolla PAMI 2007, CVPR 2005 ]
Surface representation: voxels; no need for bounding inner/outer surfaceRegularization : weighted volume – balloon force Minimization : graph cutOcclusions : accounted using a voting photo-consistency score
(occluded pixels are treated as outliers)
S = surfaceV(S) = foreground
Photoconsistency Foreground/background cost
(x)=-λ balloon force
silhouette cue – make (x) very large outside VH
![Page 36: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/36.jpg)
Photo-consistency metric
Account occlusions ρ(x,S) [continuous, level set formulations]S determines visibility
Problem : Not suitable for graph-cutSolution : ρ(x) that accounts for occlusions using NCC
xOptic ray
Correlation scores
Vote
ci
d
![Page 37: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/37.jpg)
Account occlusions
![Page 38: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/38.jpg)
Graph structure
Data (photo-consistency)
Ballooning
6 neighboring system
![Page 39: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/39.jpg)
Results
Vogiatzis2 0.50mmFurukawa2 0.54mm
![Page 40: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/40.jpg)
Competitor – Yasu Furukawa
Video
![Page 41: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/41.jpg)
Multi-view stereo
Disparity map
Depth w.r.plane
Level sets
Mesh
Voxels
Discrete methods
Graph-cutsBelief propagationDynamic programming…
Voxel carving (coloring)
Continuous methods
Non-linear optimizationVariational calculus
![Page 42: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/42.jpg)
Continuous multi-view methods Regular surface and surface evolution Level set methods Example of mesh-based reconstruction
![Page 43: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/43.jpg)
Surface evolution
SXS
dXX )(min
Continuous formulation
recover shape (surface) by minimizing cost functional integrated over the surface.
Cost functional : photo-consistency + regularization (smoothness)
Numerical methods: gradient descent, conjugate gradient, level sets …
Natural extension of curve evolution (2D) [Caselles ICCV95] to 3D
[Robert,Deriche ECCV 96][Faugeras Kerivan 98]
![Page 44: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/44.jpg)
Regular surfaceDefinition: Regular surface (manifold)
is a regular surface if for each point there exist a neighborhood V and a map of an open set such that (X = parametrization):
1. X differentiable
2. X homeomorphism ( continuous)
3. the differentiable is one to one
3S SPSVU :X
2U
USV :1X
Uvu ),( 32),( :| vudX
![Page 45: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/45.jpg)
Regular surface - propertiesRegular surface S tangent vector
normal
area
curvature )(
,
,
,
,
2
1
)()(
)(),(
1
vuvv
vv
uu
uu
Q R
vu
vu
vuP
Pvu
H
RQdAdudvRA
STspanw
XXXX
Xn
XX
Xn
XXX
XX
XXn
XX
![Page 46: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/46.jpg)
Surface evolutionCost functional
Energy of the surface
S
dAE ),(
: 33
nX
Evolution flow (Euler-Lagrange equations) 0)0(
,2
)(
SS
HS
dAE
Xt
S
nn
X
OBS:
problem is intrinsic (independent on parameterization)
automatic regularization H
motion in the normal direction
whole surface is evolving in time (reference frame attached to the object)
dA
1. Discretization
2. Choice of cost functional
![Page 47: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/47.jpg)
Photo-consistency functional
ii
imi
i
im
EE
duuE
uXfuIu
)(
))(()()(
X
u
I Image point u π(X)
Surface point X(u)=π-1(u,S) X
On Image On surfaceπ-1(u,S)
S-surface, f(X)-color (radiance) of X
π(X)
ii
S
Si
i
S
EE
dAXE
XfXIX
)(
)())(()(
Image i:
![Page 48: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/48.jpg)
Where to integrate image/surface ?On Image On surface
NZ
XfIf
dt
dX
NXXI
fIffII
NZ
X
Z
XN
Z
X
dt
dX
dAZ
XNXE
Z
XNdu
duuE
uXfuIu
im
im
S
im
imim
im
3
333
3
3
)(
0,0
)()(
)(
)(
))(()()(
NHNdt
dX
dAXE
XfXIX
S
S
S
S
)(
)())(()(
0fConstant f
Impossible to reconstruct
2.Automatic regularization
1.Depends on image derivative
3.Doesn’t account for image discretization
![Page 49: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/49.jpg)
Surface evolution example
[Faugeras Kerivan 98]
![Page 50: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/50.jpg)
Modeling correct visibility
CC’
[Gargallo et al ICCV 07]
Horizon termUsual term
else
xnx
x
xx
x
nxH
nxHxv
0
}2/,2/{1)(
}2/,2/{2/1
2/,2/0
)2/,2/(1
)(
)()(
Heaviside
func.
Dirac func.
![Page 51: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/51.jpg)
Surface discretization
Depth / disparity map
Image plane
3D point
Explicit representation
t
x y
reg
x y
data
f
dxdyffyxdxdyfyxfE
yxfyxX
),,,(),,()(
)),(,,(
[Robert and Deriche: Dense depth recovery from stereo images, ECAI1992][Strecha et al: Dense matching of multiple wide-baseline views ICCV 2003]Depth with respect to plane
3D plane[Birkbeck et al 2006]
(Euler-Lagrange eq.)
Mesh
Move points along displacement direction
vv
nnnvvvvvv
/
),()( 332211332211},,{ },,{321 321
E
SE
t
11 nv
22 nv 33 nv[Fua and Leclerc 1993][Birkbeck et al ECCV 2006]
![Page 52: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/52.jpg)
Level set representation
Surface = Zero level set of a higher dimensional function
[Osher and Sethian 88]
[Faugeras Keriven 98] (stereo)
[Soatto Yezzi Jin ICCV 03] (specular refl.)S
XX
XX
XX
for0)(
)ofcloser(for0)(
for0)( 3
normal
curvature
),( tX
2
1kH
nEvolution cost function
If
then 0)),((,
)0( 0
ttS
SSS
t
t
n
n
Efficient numerical schemes
[Sethian 96][Osher 02]
[Birkbeck]Implicit representation
![Page 53: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/53.jpg)
Cost functional
Regularization : linear
quadratic + improves convergence
- penalizes large variation
- over-smoothing
non-quadratic, anisotropic
[Robert, Deriche 96] [Alvarez, Deriche 00] [Strecha 02]
(ex. Nagel-Enkelmann diff. oper.)
+ preserves discontinuities
Photo-consistency : image cues (SFS,PS,stereo,silhouette)
dxdyzEreg
Where to integrate? : surface + makes the problem intrinsec
+ automatic regularization (multiplicative)
- over-smoothing
- not account for image discretization
image + can add a regularizer
![Page 54: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/54.jpg)
Surface/depth regularization
ff 2
fIDfIDf
fIgfIgT
)(div)(
)(div)(222
fffff
gfffTT
'divtr
''div22
Depth map regularization f
1. HomogenousDiffusion (heat eq)
2. Image-based regularization – align depth discontinuities with the image discontinuities
3. Depth map regularization Not smooth across surface discontinuities
energy Euler Lagrange eq.
g – decreasing function g(s2)->0 when s big (high gradients) – inhibit diffusion
D – tensor
D
TffD
![Page 55: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/55.jpg)
Example regularization
fIDfIDf
fIgfIgT
)(div)(
)(div)(222
Image-based regularizationcorrelation i-b regul.
homog
Depth based
depth-based regularization
fffff
gfffTT
'divtr
''div22
![Page 56: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/56.jpg)
Example regularization
i-b regul.homog.
![Page 57: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/57.jpg)
Minimal surfaces and graph cut
S
dAE
Riemannian metric (varying tensor)Euclidian metric (ct)
A
Bdistance
map
A
Bdistance
map
S
dAE )(X
Can the minimal surface energy be minimized with graph cut ?
[Boykov, Kolmogorov ICCV03,05]
![Page 58: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/58.jpg)
Cost of a cut
C
Cut metric is determined by the graph topology and by edge weights.
Ce
eC ||||||
Cost of a cut can be interpreted as a geometric “length” (in 2D) or “area” (in 3D) of the corresponding contour/surface.
![Page 59: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/59.jpg)
Riemanian metric
ddnC L21||||Euclidean length of C
Cauchy-Crofton formula
the number of times line L intersects C
C
![Page 60: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/60.jpg)
Cut Metric on gridscan approximate Euclidean Metric
C
k
kkknC 21||||
Euclidean length
2kk
kw
gcC ||||graph cut cost
for edge weights:the number of edges of family k intersecting C
Edges of any regular neighborhood system
generate families of lines
{ , , , }
Graph nodes are imbeddedin R2 in a grid-like fashion
8-neighborhoods
4-neighborhoods(Manhattan metric)
![Page 61: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/61.jpg)
Example [Boykov]
![Page 62: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/62.jpg)
Summary: representations Image centered
Depth/disparity map
Object centeredImplicit (level sets) Mesh Voxels
Image plane
3D point
+ Natural extension of SFS, PS, stereoStrong min with graph cuts
Handles topological changesImplied normalsVisibility
Graphics basedImplied normalsVisibility
Handles large structuresArbitrary topology
- Limited resolution Partial obj. reconstrViewpoint dependent
Too smoothSlowClosed surfaces
Topological changes
Ignores regularization
(sensitive to noise)Limited resolutionOcclusion handling
time
![Page 63: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/63.jpg)
Discrete vs. continuousLevel Sets Graph Cuts
variational optimization method for fairly general continuous energies
combinatorial optimization for a restricted class of energies [e.g. KZ’02]
finds a local minimum near given initial solution
finds a global minimum for a given set of boundary conditions
numerical stability has to be carefully addressed [Osher&Sethian’88]
continuous formulation -> “finite differences”
numerical stability is not an issue
discrete formulation ->min-cut algorithms
anisotropic metrics are harder to deal with (e.g. slower)
anisotropic Riemannian metrics are as easy as isotropic ones
Gradient descent method VS. Global minimization tool(restricted class of energies)
[Boykov CVPRTut 2005]
![Page 64: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/64.jpg)
A complete systemNeil Birkbeck
Motivation
method that works for objects with general reflectance textured and uniform regions shading + texture
light variation (multiview PS)
specular materials general BRDF – parametric
System
Camera calibration: pattern
Light calibration: specular white sphere
Light variation: rotating table, fixed cam.
![Page 65: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/65.jpg)
i
iiii RPIPh ),,())((),(),( LnXXXnX
Cost functional
dudvdAES u v
vu XXnXnX ),(),(Cost functional
vu
vu
XX
XXn
camera proj. image light
Per-point cost function
Visibility+sampling reflectance
1. Lambertial reflectance Xln ,, diiilamb kalR
light color
light dir.
ambient color
BRDFalbedo
2. Non-Lambertian reflectance BRDF = Phong
In practice : filter specular highlights
fit full reflectance only at the end
![Page 66: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/66.jpg)
Surface discretization
Surface = triangles, move vertices
332211 vvvX 11 nv
22 nv 33 nv
Normals interpolated 332211 nnnn
Albedo implied by the shape (closed form solution)
(knowing light dir+color)
)()()(,,
jiikj
jk
j
jvvvvn
v
Visibility/shadows Z buffering
Cost function ),()( 332211332211},,{ },,{321 321
nnnvvvvvv
SE
![Page 67: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/67.jpg)
Surface discretization
Regularizer mean curvature
Gradient finite differences
nn,2
kt
S
Multi-resolution (image pyramid)
Mesh handles topological changes
Initial shape = visual hull
![Page 68: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/68.jpg)
Results : refinement
![Page 69: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/69.jpg)
Results : shape + reflectance
![Page 70: Lecture 3 Multi-view shape reconstruction Dana Cobzas PIMS Postdoc](https://reader035.vdocuments.us/reader035/viewer/2022062320/56649d235503460f949f9541/html5/thumbnails/70.jpg)
Improvements Better light model – no need for calibration
Model background ? no need to extract silhouette (foreground) – Mumford-Shah functional
Discontinuities – anisotropic regularization; discontinuous mesh
Incorporate noise ?