ontarioyuri/presentations/tricks_08.pdf · the university of ontario optimization of surface...
TRANSCRIPT
![Page 1: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/1.jpg)
The University of
Ontario
How to fit a surface to a point cloud?or
optimization of surface functionals in computer vision
Yuri Boykov
TRICS seminarComputer Science Department
![Page 2: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/2.jpg)
The University of
Ontario
Optimization of surface
functionals in computer vision
Computer vs. human vision
Model fitting in computer vision• templates, pictorial structures, trees, deformable models, contours/snakes, meshes,
surfaces, complexes, graphs, weak-membrane model, Mumford-Shah, Potts
model,……
Optimization in computer vision• dynamic programming, gradient descent, PDEs, shortest paths, min. spanning
trees, linear and quadratic programming, primal-dual schema, network flow
algorithms, QPBO, ...
Applications
• segmentation, stereo, multi-view reconstruction, optical flows
• surface fitting
![Page 7: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/7.jpg)
The University of
Ontario+Texture
The New Yorker Album of Drawings, The Viking Press, NY, 1975
recognizing 3D perspective
![Page 8: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/8.jpg)
The University of
OntarioWhat do humans get by „looking‟?
J. VermeerThe Guitar Player
Contours
Shading
Color
Texture
…
basic image cues:
![Page 9: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/9.jpg)
The University of
OntarioWhat do humans get by „looking‟?
Contours
Shading
Color
Texture
…
basic image cues:
Segmentation
Motion
3D shape perception
3D scene geometry
Detection/Recognition
…
higher-level perception:
![Page 10: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/10.jpg)
The University of
OntarioWhat do computers get by „looking‟?
x
y
3D plot of image
intensity I(x,y)x
y
I(x,y)
x
y
![Page 11: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/11.jpg)
The University of
OntarioWhat do computers get by „looking‟?
P. PicassoThe Guitar Player
Intensity discontinuities(contours)
Intensity gradients (shading)
Multi-valued intensities (color)
Filtering (e.g. texture)
…
basic image cues: higher-levelgrouping?
![Page 12: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/12.jpg)
The University of
Ontario
model
Bayesian approach
Prior + Data Low-level cues
(local info)high-level knowledge
(global picture)
Fit some prior model into data
![Page 13: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/13.jpg)
The University of
OntarioRigid Template Matching
• In matching we estimate “position” of a rigid template in the image • “Position” includes global location parameters of a rigid template:
- translation, rotation, scale,…
Face template
image
image
translation,rotation,scaling
![Page 14: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/14.jpg)
The University of
OntarioNon-rigid (parametric) matching
1. Pick one image (red)
2. Warp the other images to match it (homographic transform)
3. Blend
panorama mosaicing
![Page 16: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/16.jpg)
The University of
Ontarioe.g…. using flexible templates
• In flexible template matching we estimate “position” of each rigid component of a template• For tree-structured models, efficient global optimization is possible via DP
(Felzenswalb&Huttenlocher 2002)
![Page 17: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/17.jpg)
The University of
Ontariotracking parameters => activity recognition
Bottom-up tracker
![Page 18: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/18.jpg)
The University of
Ontario
5-18
deformable contours (“snakes”)
2D curve which matches to image data
Initialized near target, iteratively refined
Can restore missing data
initial intermediate final
Optimization gets harder when a loop is introduced. DP does not apply.
One solution: gradient descent
Kass, Witkin, Terzopoulos 1987
![Page 19: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/19.jpg)
The University of
Ontario
6-19Cremers, Tischhäuser, Weickert, Schnörr, “Diffusion Snakes”, IJCV '02
local minima, fixed contour topology
![Page 20: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/20.jpg)
The University of
Ontario
6-20
A contour may be approximated from
u(x,y) with near sub-pixel accuracy
C
-0.8 0.2
0.5
0.70.30.6-0.2
-1.7
-0.6
-0.8
-0.4 -0.5
),( ppp yxuu
• Level set function u(x,y) is normally discretized/stored over image pixels
• Values of u(p) can be interpreted as distances or heights of image pixels
Implicit representation of contours
Osher&Sethian 1989
![Page 21: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/21.jpg)
The University of
Ontario
6-21
[Visualization is courtesy of O. Juan]
Simple evolution
Morphological Operation:Erosion
1p
ppp NCd
|| ppp udu
![Page 22: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/22.jpg)
The University of
Ontario
6-22
Visualization is courtesy of O. Juan
Example of gradient descent evolution
2
2
2
2
)(y
u
x
up dtu
Gradient descent
w.r.t.
Euclidean length
![Page 23: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/23.jpg)
The University of
Ontario
6-23
Example of gradient descent evolution
Laplacian
Osher&Sethian 1989
Gradient descent
w.r.t.
Euclidean length
2
2
2
2
)(y
u
x
up dtu
![Page 24: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/24.jpg)
The University of
Ontario
6-24
[example from Goldenberg, Kimmel, Rivlin, Rudzsky, IEEE TIP ‟01]
Geodesic Active Contours via Level-sets
y
ug
x
ug
p
yxdtu)()(
)(
C
dsgCE )()(
![Page 25: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/25.jpg)
The University of
Ontario
6-25
Other geometric energy functionals
besides length [courtesy of Ron Kimmel]
weighted length
C
dsgCE )()(
Functional E( C ) gradient descent evolution
,~ Ngg
weighted area dafCE )( ~ f
alignment(flux)
C
dsNCE ,)(
v )div(~ v
Geometric measures commonly used in segmentation
NdC
![Page 26: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/26.jpg)
The University of
Ontario
in 3D…
deformable meshes, level-sets, …
Estimation of positionfor mesh points
Many loops.optimization - gradient
descent
GOALS: global optima (?)“right” functional (?)
Typical problems:- local minima (clutter, outliers)
-over-smoothing
![Page 28: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/28.jpg)
The University of
Ontario
More generally...
Estimate labels for graph nodes
I
p
L
along one scan line in the image
observed noisy image I image labeling L(restored intensities)
NOTE:similar to robust regression modelestimation
![Page 29: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/29.jpg)
The University of
Ontario
(simple example)
Piece-wise smooth restoration
Markov Random Fields (MRF) approach
weak membrane model (Geman&Geman‟84, Blake&Zisserman‟83,87)
pL qL
Nqp
qp
p
pp LLVILLE),(
2 ),()()(
discontinuity preserving prior
optimizing E(L) is NP hard!
T T
(Continuous analogue: Mamford-Shah functional, 1989)
![Page 30: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/30.jpg)
The University of
Ontario
I
p
L
observed noisy image I image labeling L(restored intensities)
(simple example)
Piece-wise constant restoration
along one scan line in the image
![Page 31: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/31.jpg)
The University of
Ontario
(simple example)
Piece-wise constant restoration
Potts model Boykov Veksler Zabih ’01
Greig et al.’89 (for 2 labels)
Nqp
qp
p
p
i
i
LLILLEi ),(
2 )()()(
global optimization is still NP hard, but there are fast provably good
combinatorial approximation algorithms, linear and quadratic programming,
QPBO, primal-dual schema
pL qL
0}2L:p{ p2
“perceptual grouping”
}1L:p{ p1
}0L:p{ p0
![Page 32: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/32.jpg)
The University of
OntarioPerceptual grouping from stereo
(Birchfield &Tomasi‟99)
constant label = plane
![Page 33: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/33.jpg)
The University of
Ontario
Binary labeling(binary image restoration)
)|Pr()()( 2
pppppp LIILLD
}1,0{pL
original binary image I optimal binary labeling L
Nqp
qp
p
pp LLLDLE),(
)()()(
Greig Porteous Seheult ‟89
Globally optimal solution is possible using combinatorial graph cut algorithms• pseudo-boolean optimization Hammer‟65, Picard&Ratlif’75
![Page 34: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/34.jpg)
The University of
Ontario
Binary labeling(object extraction)
1pL
0pL
object segmentation
left ventricle of heart
}1,0{pL
![Page 35: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/35.jpg)
The University of
Ontario
Binary labeling(object extraction)
1
0
C
Globally optimal solution is possible using graph cut algorithms• pseudo-boolean optimization (Hammer‟65, Picard&Ratlif‟75)
surface extraction
Nqp
qppq
p
pp LLwLDLE),(
)()()(
Boykov&Jolly‟01
left ventricle of heart
}1,0{pL
![Page 36: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/36.jpg)
The University of
OntarioImplicit surface representation via graph-cuts
• Any contour (or surface in 3D) satisfying labeling of exterior/interiorpoints (pixel centers) is acceptable if some explicit surface has to be output.
0 1
1
1110
0
0
0
0 0
![Page 37: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/37.jpg)
The University of
Ontario
)int(
)(,)()(CC
x
C
dpxfdsdsgCE vN
Geometric lengthany convex,
symmetric metric (e.g. Riemannian)
Flux
any vector field v
Regional bias
any scalar function f
“edge alignment”
Tight characterization for geometric functionals of contour C that can
be globally optimized by graph cut algorithms (Kolmogorov&Boykov’05)
disclaimer: for pairwise interactions only
Global optimization of
geometric surface functionals
N)q,p(
qp
p
pp )L,L(V)L(D)L(E
![Page 38: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/38.jpg)
The University of
OntarioGlobally optimal surfaces in 3D
Volumetric segmentation(BJ01,BK’03,KB’05)
![Page 39: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/39.jpg)
The University of
Ontario
Binary labeling(object extraction)
Blake et al.‟04, Rother et al.‟04
}1,0{pL
iteratively re-estimate color models e.g. using mixture of Gaussians
![Page 42: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/42.jpg)
The University of
OntarioOptimal surfaces in 3D
3D reconstruction
Vogiatzis, Torr, Cippola‟05
Local cues: voxel’s photoconsistency
Prior:smoothness,
projective geometry constraints
![Page 43: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/43.jpg)
The University of
OntarioGlobally optimal surfaces in 3D
Lempitsky&Boykov, 2006
from a cheap digital camera
![Page 44: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/44.jpg)
The University of
Ontario
3D model
multi-view reconstruction set up
Furukawa&Ponce 2006
Globally optimal surfaces in 3D
(texture mapped)
![Page 45: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/45.jpg)
The University of
OntarioSurface fitting to point cloud
a cloud of 3D points (e.g. from a laser scanner)
3D model:
surface fitting:
![Page 47: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/47.jpg)
The University of
OntarioSurface fitting functional
CC
dsdsCE 1,)( vN
Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
ip
in
S
data fit prior
![Page 48: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/48.jpg)
The University of
OntarioOptimal surfaces in 3D
Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
From 10 views
No initialization is needed
CC
dsdsCE 1,)( vN
![Page 49: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/49.jpg)
The University of
OntarioGlobal vs. local optimization
regional potentials
)(vdivf
CC
dsdpdivCE 1)()int(
v
Fitting a surface into a cloud of oriented points (Lempitsky&Boykov, 2007)
initial solution local minima global minima
![Page 53: Ontarioyuri/Presentations/tricks_08.pdf · The University of Ontario Optimization of surface functionals in computer vision Computer vs. human vision Model fitting in computer vision](https://reader033.vdocuments.us/reader033/viewer/2022041701/5e41e1ea53bd290f3645055d/html5/thumbnails/53.jpg)
The University of
OntarioSummary
Global optimization-Your solution is as good as your functional
-No need to worry about initial guess or convergence issues
-Polynomial algorithms, but many practical
issues (efficient data structures, memory limitations,
parallelization, dynamic applications,…)
- Many useful functionals are NP hard (lots of approximation methods are developed)
- New approaches allowing global optimization
are introduced (including new version of level-sets)