introduction to image...
TRANSCRIPT
![Page 1: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/1.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Introduction to Image Segmentation:
Part 1: binary image labeling
discrete (and other) methods
Yuri Boykov
University of Western Ontario
![Page 2: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/2.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Introduction to Image Segmentation
motivation for optimization-based approach
active contours, level-sets, graph cut, etc.
implicit/explicit representation of boundaries
objective functions (energies)
• physics, geometry, statistics, information theory
• set functions and submodularity (graph cuts)
part II: from binary to multi-label problems
![Page 3: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/3.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Thresholding
T
![Page 4: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/4.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Thresholding
S={ p : Ip < T }
T
![Page 5: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/5.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Background Subtraction
?
Thresholding
- = I= Iobj - Ibkg
Threshold intensities above T
segment’s region property S={ p : Ip > T }
![Page 6: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/6.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Region growing
- = I= Iobj - Ibkg
| Ip|
Breadth First Search (seeds) :
| Ip| < T
seeds
p S | Ip | > T segment’s boundary property
![Page 7: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/7.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Region growing
“leakage”
![Page 8: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/8.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Good segmentation S ?
Objective function must be specified
Quality function
Cost function
Loss function E(S) : 2P
“Energy”
Regularization functional
Segmentation becomes an optimization problem: S = arg min E(S)
![Page 9: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/9.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Good segmentation S ?
Objective function must be specified
Quality function
Cost function E(S)
“Energy”
Regularization functional
combining different constraints e.g. on region and boundary
= E1(S)+…+ En(S)
![Page 10: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/10.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
optimization-based
Common segmentation techniques
region-growing
intelligent scissors
(live-wire)
active contours
(snakes)
watersheds
boundary-based region-based both region & boundary
thresholding geodesic
active contours
(e.g. level-sets)
MRF
(e.g. graph-cuts)
random walker
![Page 11: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/11.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Active contours - snakes
[Kass, Witkin, Terzopoulos 1987]
Given: initial contour (model) near desirable object
Goal: evolve the contour to fit exact object boundary
![Page 12: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/12.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
5-12
Tracking via active contours
Tracking Heart Ventricles
![Page 13: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/13.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
5-13
Active contours - snakes
Parametric Curve Representation (continuous case)
A curve can be represented by 2 functions
open curve closed curve
1s0sy,sxs ))()(()(ν
]}[{ 10ssC ,|)(ν
parameter
![Page 14: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/14.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
5-14
Snake Energy
)C(E)C(E)C(E exin
internal energy encourages smoothness or any particular shape
external energy encourages curve onto
image structures (e.g. image edges)
![Page 15: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/15.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
5-15
Active contours - snakes (continuous case)
internal energy (physics of elastic band)
external energy (from image)
elasticity / stretching stiffness / bending
1
0
2
221
0
2
in dssd
ddsdsd)C(E νν
1
0
2
ex ds|s(I|)C(E ))(v
proximity to image edges
![Page 16: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/16.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Active contours – snakes (discrete case)
5v
4v3v
2v
1v6v
7v
8v
10v
9v
elastic energy (elasticity)
2
11 iiv
ds
d
bending energy (stiffness)
1ii1i1iii1i2
2
2)()(ds
d
)( iii y,xν
2n)( 1n210 ,....,,, ννννC
![Page 17: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/17.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
5-17
Basic Elastic Snake
continuous case
discrete case
1
0
2
1
0
2 ds|))s(v(I|ds|ds
dv|E
1n
0i
2
i
1n
0i
2
i1i |)v(I||vv|E
elastic smoothness term (interior energy)
image data term (exterior energy)
]}1,0[s|)s({ νC
}ni0|{ i νC
![Page 18: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/18.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
5-18
Snakes - gradient descent
simple elastic snake energy
tE' CC
update equation for the whole snake
t...
y
x
...
y
x
'y
'x
...
'y
'x
1n
1n
0
0
yE
xE
yE
xE
1n
1n
0
0
1n
1n
0
0
C
2
1
1
0
2
1 )()( ii
n
i
ii yyxx
2
iiy
1n
0i
2
iix1n01n0 |)y,x(I||)y,x(I|)y,,y,x,,x(E
here, energy is a function of 2n variables
C
![Page 19: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/19.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
5-19
Snakes - gradient descent
simple elastic snake energy
i
i
yE
xE
iF
iF tF' iii
νν
update equation for each node
C t...
y
x
...
y
x
'y
'x
...
'y
'x
1n
1n
0
0
yE
xE
yE
xE
1n
1n
0
0
1n
1n
0
0
2
1
1
0
2
1 )()( ii
n
i
ii yyxx
2
iiy
1n
0i
2
iix1n01n0 |)y,x(I||)y,x(I|)y,,y,x,,x(E
here, energy is a function of 2n variables
C
![Page 20: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/20.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
5-20
Snakes - gradient descent
energy function E(C) for contours C E(C)
0c
EtCC i1i
gradient descent steps
1c2c
local minima
for E(C)
c
n2C
second derivative of image intensities
step size could be tricky
![Page 21: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/21.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Snakes – dynamic programming (DP)
Energy E can be minimized via Dynamic Programming
[Amini, Weymouth, Jain, 1990]
),(...),(),(),...,,( 1132221121 nnnn vvEvvEvvEvvvE Elastic energy - pairwise interactions
)( iii y,xν
possible states for each control point
Energy E can be minimized via Dynamic Programming
),(...),(),(),...,,( 1132221121 nnnn vvEvvEvvEvvvE Elastic energy - pairwise interactions
optimal state
![Page 22: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/22.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Snakes – dynamic programming (DP)
Energy E can be minimized via Dynamic Programming
),(...),(),(),...,,( 1132221121 nnnn vvEvvEvvEvvvE Elastic energy - pairwise interactions
Iterate… until optimal position for each point is the center of the box,
(local minimum condition)
optimal state
[Amini, Weymouth, Jain, 1990]
Advantages: no 2nd derivatives explicit step size control
![Page 23: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/23.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Another example of DP “Live wire” or “intelligent scissors”
[Barrett and Mortensen 1996]
| I|
)( |I|w
|I|
image-based edge weights pixels
A
B
shortest path algorithm (Dijkstra)
![Page 24: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/24.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Shortest paths approach
shortest path on a 2D graph graph cut
Example: find the shortest
closed contour in a given domain of a graph
Compute the shortest path p ->p for a point p.
p
Graph Cuts approach
Compute the minimum cut that
separates red region from blue region
Repeat for all points on the gray line. Then choose the
optimal contour.
![Page 25: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/25.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
graph cuts vs. shortest paths
On 2D grids graph cuts and shortest paths give optimal 1D contours.
A Cut separates regions
A
B
A Path connects points
Shortest paths still give optimal 1-D contours on N-D grids
Min-cuts give optimal hyper-surfaces on N-D grids
![Page 26: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/26.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cut
n-links
s
t a cut hard constraint
hard constraint
Minimum cost cut can be computed in polynomial time
(max-flow/min-cut algorithms)
22
exp
pq
pq
Iw
pqI
[Boykov and Jolly 2001]
![Page 27: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/27.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Minimum s-t cuts algorithms
Augmenting paths [Ford & Fulkerson, 1962]
- heuristically tuned to grids [Boykov&Kolmogorov 2003]
Push-relabel [Goldberg-Tarjan, 1986] - good choice for denser grids, e.g. in 3D
Preflow [Hochbaum, 2003] - also competitive
![Page 28: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/28.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
“Augmenting Paths”
Find a path from S to
T along non-saturated
edges
“source”
A graph with two terminals
S T
“sink”
Increase flow along
this path until some
edge saturates
![Page 29: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/29.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
“Augmenting Paths”
Find a path from S to
T along non-saturated
edges
“source”
A graph with two terminals
S T
“sink”
Increase flow along
this path until some
edge saturates
Find next path…
Increase flow…
![Page 30: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/30.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
“Augmenting Paths”
Find a path from S to
T along non-saturated
edges
“source”
A graph with two terminals
S T
“sink”
Increase flow along
this path until some
edge saturates
Iterate until … all
paths from S to T have at
least one saturated edge MAX FLOW MIN CUT
![Page 31: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/31.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Optimal boundary in 2D
“max-flow = min-cut”
![Page 32: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/32.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Optimal boundary in 3D
3D bone segmentation (real time screen capture)
![Page 33: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/33.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
example of soft regional constraints
Graph cut (region + boundary)
pqw
n-links
s
t a cut )(sDp
)(tDp
NOTE: seeds were hard constrains on segment’s region
assume are known “expected” intensities
of object and background
ts II and
|II|tD t
pp )(
|II|sD s
pp )(
S
S
segmentation
[Boykov and Jolly 2001]
![Page 34: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/34.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
example of soft regional constraints
Graph cut (region + boundary)
pqw
n-links
s
t a cut )(sDp
)(tDp
in general, assume known intensities distributions
of object and background )()( t|IPrlntD pp
)()( s|IPrlnsD pp
S
S
segmentation
[Boykov and Jolly 2001]
NOTE: seeds were hard constrains on segment’s region
![Page 35: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/35.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cut (region + boundary)
![Page 36: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/36.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts as energy optimization for S
pqw
n-links
s
t a cut )(sDp
)(tDp
segmentation cut
S
S
cost of severed t-links
Sp
p
Sp
p DD )0()1(cost(cut) = + E(S)
}10{ ,S p
unary terms pair-wise terms
cost of severed n-links
Npq
pq SSw ][ qpp
pp SD )(
regional properties of S boundary smoothness for S
[Boykov and Jolly 2001]
![Page 37: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/37.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Unary potentials as linear term wrt.
p
pp SD )(
unary terms
Sp
p
Sp
p DD )0()1(
)( pg
S,g p
p
pp SDDconst )0()1(
}10{ ,S p
p
ppp SDSD )-(1)0()1( p
![Page 38: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/38.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
In general,…
pS
Npq
pq SSw ][ qp
pair-wise terms
Npq
pq SSSSw qpqp )1()1(
quadratic polynomial wrt.
k-arity potentials are k-th order polynomial
![Page 39: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/39.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Sg,E(S) B(S)Sg,E(S)
Graph cuts vs Thresholding
I
Fg)|Pr(I Bg)|Pr(I
Sp
g(p)E(S)
bg)|Pr(I(p)
fg)|Pr(I(p)lng(p)
S
thresholding graph cut [BJ, 2001]
![Page 40: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/40.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts vs Region Growing
like “region growing”
![Page 41: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/41.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts vs Region Growing
like “region growing”
![Page 42: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/42.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts vs Region Growing
like “region growing”
![Page 43: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/43.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts vs Region Growing
like “region growing”
![Page 44: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/44.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts
![Page 45: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/45.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts 2
![Page 46: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/46.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts 2
Any paths would work, but shorter paths give faster algorithms
(in theory and practice)
![Page 47: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/47.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts 3
![Page 48: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/48.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts 3
Finds optimal boundary (least number of holes)
=> Energy minimization
![Page 49: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/49.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
‘Smoothness’ of segmentation boundary
snakes (physics-based contours)
geodesic contours
graph cuts
Note: many common distance-based methods do not optimize segmentation boundary directly
(fuzzy connectivity, geodesic Voronoi cells, random walker)
![Page 50: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/50.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Discrete vs. continuous energies
Geodesic contours
S
s
S
dswdppgSE )()(
Both incorporate segmentation cues:
• Regional bias
• Boundary smoothness and alignment to image edges
q,p
qppq
Sp
p SSwgSE )()(
Graph cuts
}10{ ,S p
[Caselles, Kimmel, Sapiro, 1997] (level-sets) [Boykov and Jolly 2001]
C
[Chan, Esidoglu, Nikolova, 2006] (convex)
![Page 51: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/51.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts on a complex and boundary of S
John Sullivan’90, Kirsanov&Gortler’04
![Page 52: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/52.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts on a grid and boundary of S
Severed n-links can approximate geometric length of contour C [Boykov&Kolmogorov, ICCV 2003]
This result fundamentally relies on ideas of Integral Geometry (also
known as Probabilistic Geometry) originally developed in 1930’s.
• e.g. Blaschke, Santalo, Gelfand
}10{ ,S p
Ce
|e|SB )(
![Page 53: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/53.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Integral geometry approach to length
C 2
0
a set of all lines L
CL
a subset of lines L intersecting contour C
ddnC L21||||Euclidean length of C :
the number of times line L intersects C
Cauchy-Crofton formula
probability that a “randomly drown” line intersects C
![Page 54: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/54.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts and integral geometry
C
k
kkknC 21||||
Euclidean
length 2
kkkw
gcC ||||
graph cut cost
for edge weights: the number of edges of family k intersecting C
Edges of any regular
neighborhood system
generate families of lines
{ , , , }
Graph nodes are imbedded
in R2 in a grid-like fashion
Length can be estimated without computing any derivatives
![Page 55: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/55.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Metrication errors
“standard”
4-neighborhoods
(Manhattan metric)
larger-neighborhoods 8-neighborhoods
Euclidean metric
Riemannian metric
![Page 56: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/56.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Metrication errors
4-neighborhood 8-neighborhood
![Page 57: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/57.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
implicit (region-based) representation of contours
Differential vs. integral approach to length
ddnC L21||||
Cauchy-Crofton formula
1
0|||| dtCC t
Inte
gra
l g
eo
me
try
Dif
fere
nti
al
ge
om
etr
y
Parametric (explicit) contour
representation
dxuC
||||||
Level-set function
representation
![Page 58: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/58.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Implicit (region-based) surface
representation via level-sets
),( yxuz
}0),(:),{( yxuyxC(implicit contour representation)
[Dervieux, Thomasset, 79, 81] [Osher, Sethian, 89]
Note: 0.5-level set for u : n[0,1] in convex formulations [Chan, Esidoglu, Nikolova, 2006] (convex)
![Page 59: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/59.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Implicit (region-based) surface
representation via level-sets
|| ppp udu NCd
For example, level sets can compute contour’s gradient descent
for geometric energies of contour C where E is collinear with normal
EdtCd
[Dervieux, Thomasset, 79, 81] [Osher, Sethian, 89]
The scaling by is easily
verified in one dimension
dC
dCudu ||
x
)(xu || u
p
N
![Page 60: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/60.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Implicit (region-based) surface
representation via level-sets
weighted length C
dsgCE )()(
Functional E( C ) gradient descent evolution
,~ Ngg
weighted area
dafCE )( ~ f
alignment (flux)
C
dsNCE ,)(
v )div(~ v
Geometric measures commonly used in segmentation
NdC
![Page 61: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/61.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
pEpN
Gradient descent of snake C produces
tangential motion, which is “invisible” in segmentation.
Note: physic-based energy of snake
depends on contour parameterization
In contrast, geometric energies give E collinear with
can be minimized via level-sets
N
Q: in what applilcations tangential motion of segment boundary matters?
![Page 62: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/62.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
A contour may be approximated from
u(x,y) with sub-pixel accuracy
C
-0.8 0.2
0.5
0.7 0.3 0.6 -0.2
-1.7
-0.6
-0.8
-0.4 -0.5
)()( y,xupu
• Level set function u(p) is normally stored on image pixels
• Values of u(p) can be interpreted as distances or heights of image pixels
Implicit (region-based) surface
representation via level-sets
![Page 63: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/63.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Implicit (region-based) surface
representation via graph-cuts
)( y,xSS p
• Graph cuts represent surfaces via binary labeling Sp of each graph node
• Binary values of Sp indicate interior or exterior points (e.g. pixel centers)
There are many contours satisfying
interior/exterior labeling of points
Question: Is this a contour to be reconstructed from binary labeling Sp ?
Answer: NO
0 1
1
1 1 1 0
0
0
0
0 0
C
![Page 64: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/64.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Contour/surface representations (summary)
Implicit (area-based) Explicit (boundary-based)
Level sets (geodesic active contours,
convex geometric energies)
Graph cuts (grids or complexes)
Graph cuts on complexes
Live-wire (shortest paths on graphs)
Snakes (physics-based band model)
![Page 65: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/65.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Different ways to look at
energy of graph cuts
1: Posterior energy (MAP-MRF)
2: Approximating continuous surface functional
3: Submodular set function
)()()(
Npq
qppq
p
pp S,SVS|DPrlnSE
SS
s
S
dsswds,dsgSE )()()( vN
log-likelihoods log of prior
regional term flux boundary length
A
AA SESE )()( }{ Ap|SS pA for
factors
![Page 66: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/66.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Submodular functions
Edmonds 1970
LLL, TSTSTS
Lattice - set of elements with inf and sup operations )( ,L,
Function s called submodular if for any L:E
)()()()( TESETSETSE
L, TS
![Page 67: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/67.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Lattice of sets and set functions
Assume set Ω, then is a lattice of subsets
S T
Ω
)( ,,2
2
NOTE: if Ω is a set of pixels then
any (binary) segmentation energy E(S)
is a set function E:
S
Ω
![Page 68: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/68.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Submodular set functions
Set function is submodular if for any 2:E
)()()()( TESETSETSE
TS,
Significance: any submodular set function can be globally optimized in polynomial time
[Grotschel et al.1981,88, Schrijver 2000] 9O ||
S T
Ω
![Page 69: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/69.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Submodular set functions
Sets are conveniently represented by binary indicator variables
pSS p |,1}{0
2:EThus, set functions can be represented as
)()( ||,...,, SSSESE 21
ApSS pA | ADefine , a restriction of S to any subset
and consider projections of energy E onto subsets A )( AA SSE \|
Ω
1pS
0pS
Set function E(S) is submodular iff for any pair
),(),(),(),( pqpqpqpq SESESESE \\\\ |||| 10011100
qp,
![Page 70: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/70.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts for minimization of
submodular set functions
Assume set Ω and 2nd-order (quadratic) function
Function E(S) is submodular if for any
)()()()( 10011100 ,,,, pqpqpqpq EEEE
Nqp )( ,
Significance: submodular 2nd-order boolean (set) function can be globally optimized in polynomial time by graph cuts
[Hammer 1968, Pickard&Ratliff 1973]
2NO ||||
Npq
qppq SSESE)(
, )()( }{ 10SS qp ,, Indicator variables
[Boros&Hammer 2000, Kolmogorov&Zabih2003]
![Page 71: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/71.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts for approximating
continuous surface functionals
[Kolmogorov&Boykov, ICCV 2005]
[Boykov&Kolmogorov, ICCV 2003]
SS
x
S
dppfdsdsgSE )()()( vN,
Geometric length any convex,
symmetric metric g e.g. Riemannian
Flux
any vector field v
Regional bias
any scalar function f
Submodular quadratic boolean functions on a grid
can approximate continuous geometric functionals
![Page 72: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/72.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts for minimization of
posterior energy (MRF)
}{ 10Sp ,
}{ Ap|SS pA factors
A
AAn1 SEexpSSPr )(),...,(
Assume Gibbs distribution over binary variables
A
AA SESE )()(
Theorem [Boykov, Delong, Kolmogorov, Veksler in unpublished book 2013?]
Any pair of random variables Sp and Sp are positively correlated iff function
boolean (set) function is submodular
That is, submodularity implies MRF with “smoothness” prior
![Page 73: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/73.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts for minimization of
posterior energy (MRF)
),()()(
Npq
qppq
p
2
pp LLVLILE },{ 10Lp
binary image restoration
Spatial prior (regularization)
Log-Likelihood (data term)
[Greig at. al., IJRSSB, 1989]
},{ 10I p
![Page 74: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/74.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Graph cuts for minimization of
posterior energy (MRF)
][)|(ln),|(
Npq
qppq
p
Lp10 LLwILEp
Pr },{ 10Lp
Spatial prior (regularization)
assuming known
[Boykov&Jolly, ICCV2001]
image segmentation, graph cut
RGBI p
Log-Likelihood (data term)
![Page 75: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/75.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Beyond given appearance models
][)|(ln),,(
Npq
qppq
p
Lp10 LLwILEp
Pr },{ 10Lp
Spatial prior (regularization)
extra variables
[Rother, et al. SIGGRAPH2006]
iterative image segmentation, Grabcut (block coordinate descent L )
RGBI p
Log-Likelihood (data term)
10 ,
Models . can be iteratively
re-estimated
10 ,
![Page 76: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/76.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Beyond submodularity
Many useful non-submodular set functions E(S) • in binary segmentation with learned interaction potentials
• in the context of binary moves for multi-label (later)
QPBO (partial optimality) [survey Kolmogorov&Rother, 2007]
LP relaxations [Schlezinger, Komodakis, Kolmogorov, Savchinsky,…]
Message passing, e.g. TRWS [Kolmogorov]
active area of research…
![Page 77: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/77.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Beyond submodularity
Deconvolution
][)( ||
Npq
qp
p
2
Bq
qB1
p LLwLILEp
)(
image I blurred with mean kernel
QPBO
TRWS
“secret method”
submodular quadratic term
non-submodular quadratic term
![Page 78: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/78.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Beyond loglikelihoods and
length-based smoothness
Shape bias • star-shape (one click) [Veksler 2008]
• shape statistics [Cremers 2003]
• box prior [Lempitsky 2009]
Curvature of the boundary (like bending in snakes)
Cardinality constraints
Distribution constraints
Sparsity or MDL prior, label costs
Many others….
E(S)= E1(S)+…+ En(S)
![Page 79: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/79.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Beyond linear combination of terms
Ratios are also used
• Normalized cuts [Shi, Malik, 2000]
• Minimum Ratio cycles [Jarmin Ishkawa, 2001]
• Ratio regions [Cox et al, 1996]
• Parametric max-flow applications [Kolmogorov et al 2007]
)(
)()(
SE
SESE
2
1
![Page 80: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/80.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Segmentation principles
Boundary seeds
• Livewire (intelligent scissors)
Region seeds
• Graph cuts (intelligent paint)
• Distance (Voronoi-like cells)
Bounding box
• Grabcut [Rother et al]
Center seeds
• Star shape [Veksler]
Many other options…
Normalized cuts [Shi Malik]
Mean-shift [Comaniciu]
MDL [Zhu&Yuille]
Entropy of appearance
Add enough constraints:
Saliency
Shape
Known appearance
Texture
interactive vs. unsupervised
![Page 81: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/81.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Differences maybe minor
interactive ? unsupervised
NOTE: Grabcut converges to a local minima near the initial box.
][)|(ln),,(
Npq
qppq
p
Sp10 SSwISEp
Pr
Grabcut energy [Rother et al]
][||||)(
Npq
qppq SSwSHSSHSSE )()(
unsupervised image segmentation energy
entropy entropy
![Page 82: Introduction to Image Segmentationhelper.ipam.ucla.edu/publications/gss2013/gss2013_11413.pdfIntroduction to Image Segmentation motivation for optimization-based approach active contours,](https://reader030.vdocuments.us/reader030/viewer/2022040905/5e79be22b7bdc42e850d6944/html5/thumbnails/82.jpg)
IPAM Graduate Summer School: Computer Vision, July 2013 Yuri Boykov, UWO
Implicit surface representation Global optimization is possible
Summary
Thresholding, region growing
Snakes, active contours
Geodesic contours
Graph cuts (binary labeling, MRF)
Covered basics of: Not-Covered:
Ratio functionals
Normalized cuts
Watersheds
Random walker
Many others…
High-order models
Multi-label segmentation
Model fitting
To be covered later: