chapter 7 – segmentation ii 7.1. mean shift segmentation 7.2. active contour models – snakes...
TRANSCRIPT
Chapter 7 – Segmentation II
7.1. Mean Shift Segmentation
7.2. Active Contour Models – Snakes
7.3. Geometric Deformable Models – Level
Sets and Geodesic Active Contours
7.4. Fuzzy Connectivity
7.5. Towards 3D Graph-Based Image Segmentation
7.6. Graph Cut-Segmentation
7.7. Optimal Single and Multiple Surface
Segmentation
7-1
1. For each data point
Fix a window around the data point.
Compute the mean of data within the window.
2. Shift data points to their computed means
simultaneously
3. Repeat till convergence.
7.1. Mean Shift Segmentation
Idea of mean shift:
7-2
Window
Center ofmass
Mean Shiftvector
7-3
7-4
7-5
7-6
7-7
Density distribution
Data points
Find modes
Densityestimator ,
1
1( ) ( )
ni
h K di
f Khnh
x x
x
1 , ,{ } , di i n i R x x
Find clusters
Objective:
Density kernel K(x)
Uniform
21( ) exp( )
2x xNK c
2(1 ) if 1
( )0 otherwise
x xxE
cK
1 if 1( )
0 if 1
xx
xUK
7-8
Normal
Epanechnikov
In practice, radially symmetric kernels are used, i.e.,2
( ) ( )x xK ck where k : profile of kernel K
7-9
Uniform1 if 1
( )0 if 1
U
xk x
x
Profile of kernel K(x)
Epanechnikov
Normal
7-10
Given n data points in d-D space , the density
estimation at point x : ,1
1( ) ( )
ni
h K di
f Khnh
x x
x
,1
1( ) ( )
x xx
ni
h K di
f Khnh
2
( ) ( )x x x xi i
kK c kh h
We are interested in locating x for which , ( ) 0h Kf x
2
,1
2
21
( ) ( )
2 ( ) ( )
nk i
h K di
nk i
idi
cf k
hnh
ck
hnh
x xx
x xx x
Use profile, i.e.,
The density gradient:
(See next page)
7-11
( ( )) ( ) ( ) ( )( ( )) ( )
dk f dk f df dfk f k f
d df d d
x x xx
x x x2
( ) ifh
x xxLet
Then2 2 2
i i idk k
h h d h
x x x x x x
x
2
2
( ) ( )
2 2 ( )
i i iT
ii
d d
d h d h h
h h h
x x x x x x
x x
x xx x
2 2
2
2( )i i
ik kh hh
x x x xx x
7-12
2( ( ) / )x xi ig g h let
Assume the derivative of k is ( ) ( )k x g x 2
, 21
2
21
2( ) ( ) ( ( ) / )
2 ( ) ( ( ) / )
nk
h K i idi
nk
i idi
cf k h
nh
cg h
nh
x x x x x
x x x x
2 21 1
21 1
2 2= ( ) = ( )
2( )
n nk k
i i i i id di i
n nk
i i idi i
c cg g g
nh nh
cg g
nh
x x x x
x x
7-13
21 1 1
2( ) ( ) /( )
n n nk
i i i idi i i
cg g g
nh
x x
Let g(x) be the profile of kernel G(x), i.e.,
2( ) ( )gG c gx x
The first term is proportional to a density
estimator 2
,1
( ) ( )n
g ih G d
i
cf g
hnh
x x
x computed with
kernel ( )G x
21
2 nk
idi
cg
nh
7-14
The second term is the mean
shift vector 1 1
( ) /( )x xn n
i i ii i
g g
The successive location of the kernel G are2 2
11 1
( ) / ( )y x y x
y xn n
j i j ij i
i i
g gh h
2
1 1, 2
1 1
( )
( )
( )
x xx x
m x x xx x
n ni
i i ii i
h G n ni
ii i
g gh
g gh
7-15
7-16
Color image
(L u v)- color space
Example: Mode Detection
Feature vector of pixel i, ( )i L u vx
7-17
Example: Image Segmentation
Feature vector of pixel i,
where : spatial-domain part, : range-domain part
( )s ri x x x
sx rxe.g.,
7-18
Two steps of mean-shift image segmentation:
1) Discontinuity Preserving Filtering
-- Preserves discontinuities of images
7-19
Let : pixels of the original image
: pixels of the filtered imageix
iz
2) Mean Shift Clustering
Given ( , ),i jMz : the mode associated withjM izknown from Step 1.
-- Regularizes regions
7-20
BOA : Basin Of Attraction, the set of all locations that converge to the same mode
where : segmentation label of pixel i : convergence points known from Step 1
iL,i cony
7-21
Examples: (1) Image Segmentation
(2) Image Smoothing and Segmentation
Original image Smoothed image Segmentation image
Gary levels of original image The path of mean shift
Gary levels after segmentationGary levels after smoothing7-22
7.2. Active Contour Models – Snakes
7-23
-- An energy-minimization approach
-- (i) A snake is a deformable model whose energy
depends on its shape and location in the image
(ii) Local minima of the energy correspond to
desired image properties.
(iii) Initial snake position should be provided.
7-24
7-25
7.2.1 Traditional Snakes and Balloons
Snake:
The energy functional of the snake, which is to be
minimized is a weighted combination of internal
and external forces
( ) [ ( ), ( )], [0,1]v s x s y s s
The internal forces emanate from the shape of the snake
The external forces come from the image and
constraints
7-26
The image forces may come from lines, edges, and
terminations
3/ 2/g g g
where ( , , ( , )),
( , ) : Gaussian smoothed ( , )
x y g x y
g x y f x y
g
7-27
The constraint forces come from user
specified properties to be imposed on the snake,
e.g., smmothness.
7-28
2 2
2 20
v vvext
d d d ds s E s
ds ds ds ds
written this equation in an evolution form
0,s
dE E
ds v v
To minimize
This leads to the Euler-Lagrange motion equation
, let 0.dE
ds
Substitute int extE E E into the equation
where , , s s
s
E d EE E
ds
v v
vv
v v
22
2 2
, , ,
,ext
s t s t s ts s
t s s s s
E s t
v v v
vStop when( , )
0s t
t
v
7-29
A balloon, which can be inflated, is extended from
the snake by including an additional pressure force
so that it can overcome small isolated barriers
7-30
7.2.3 Gradient Vector Flow Snakes
Difficulties of previous approaches:
(i) initialization , (ii) concaves of boundaries
GVF: An external force field points toward
boundaries when in their proximity
7-31
2 22 2 2 2x y x yE u y v v f f dxdy g
GVF is derived from image by minimizing an energy functional
( , ) ( ( , ), ( , ))g x y u x y v x y
2 2 2 0,x x yu u f f f 2 2 2 0y x yv v f f f
g can be obtained by solving the following Euler equations, which are diffusion equations
Rewrite the above equations in an evolution form
2
2 2
, , , , ( , ,
, )( , , )
t
x x y
u x y t u x y t u x y t
f x y f x y f x y
2
2 2
, , , , ( , ,
, )( , , )
t
y x y
v x y t v x y t v x y t
f x y f x y f x y
7-32
Let in Eqs. (7.15) and (7.22) forming the
GVF snake equation
extE g
, , ,v v v gt s t s t s t
7-33
2 2 2
dE f f d
g g x
22 0,f f g g
22t f f g g g
3D GVF:
7-34
7-35
7.2.2 ExtensionsDeformable models, Fourier deformable model,Finite element snakes, B-snakes, united snakes.
7.3. Geometric Deformable Models (GDM) – Level Sets and Geodesic Active Contours
7-36
1. Parametric deformable model: Borders are represented in a parametric form, e.g., snakes2. Geometric deformable model : Borders are represented by partial differential equations, e.g., level sets, geodesic active contours
2 main groups of deformable models:
Advantages of GDM:(i) Curves are evolved using only geometric
computations, independent of any parameterization
7-37
[ ( ), ( )]C x t y tLet a curve be denoted by
(ii) Curves are represented as level sets of higher dimensional functions yielding seamless treatment of topological changes(iii) Multiple object can be detected simultaneously
( ) ( )t
as t d F
( ) ( )s
t V tt
F
( ) ( ) ( )t x t y t F i j
The associated length function:
positional vector
: speed function
or by
[ ( ( )), ( ( ))] [ ( ), ( )] ( )C x t s y t s C X s Y s s X
7-38
Assume the curve moves only in a direction normal
(N) to itself, i.e.
( )( )
sV t s
t
X
N
Assume also the speed function is a function of
curvature c, ( )( )
sV c s
t
X
N
( ) ( ) ( ) ( ) ( )[ , ] [ , ]
[ ( ), ( )] ( )
s X s Y s X s s Y s s
t t t s t s ts
X s Y s s V tt
X
X
( ) ( )s s X N
7-39
Constant Deformation Equation
-- Deformation is similar to inflation balloon force and
may introduce singularities (e.g., sharp corners)
0/X Nt V
/X Nt c Curvature Deformation Equation
7-40
The above geometric deformable model starts with an initial curve and evolves its shape using a speed function. Such an evolution can be implemented using level sets technique.
Combined constant and curvature deformation:
0( ) .k c Vt
X
N
7-41
1
.1
kG I
Let
Then, k -> 0 when the curve approaches edges. Strong edges will stop the curve propagation.
7-42
Level set function is a higher-dimensional
function, which represents a boundary as a level set.
The curve at time t is the set of image points, for
which the value of the level set function at time t
is equal to zero , 0x y
,x y
7-43
Using the level set representation of a curve allows its evolution by updating the level set function.
7-44
( )0,
V c
t
( )V c
t
2 2
32 2 2
2
( )
xx y x y xy yy x
x y
c
A level set function with contour
as its zero-level set
Differentiate w.r.t. t
,x y sX
0s X
0X
t t
,
N
( )( ),
sV c s
t
X
N ( ),
sV c
t
X
1tI tI
Experiment:
7-45
0 if
( ) if
if in
out
x
x x
x( , )
0t t
t
t t t
( ) x : the initial contour
Level set function:
( )t F F
n
( ) 0t tF
max{0, },int extF F F ,intF ( )extF E
( )E : the edge magnitude
7-46
7.4. Fuzzy Connectivity
7-47
Uncertainties for image segmentation: Noise, uneven illumination, limited spatial resolution, partial occlusion, etc.
Fuzzy connectivity segmentation keeps considering the likelihood (i.e., hanging togetherness) of whether nearby pixels belong to the same object
Fuzzy affinity : a strength of hanging togetherness of nearby pixels, which is a function of distance between two adjacent pixels.
e.g., 2
( , )( , )
1 ( ) ( )
c dc d
k f c f d
: fuzzy adjacency
f : image properties
7-48
2
12 21 1
11 if 2
( , ) 1 ( )
0 otherwise
i ii
i ii
c dc d k c d
Fuzzy adjacency :
Fuzzy connectedness :
7-49
Connectedness map : c : seed; : any pixel
( , )ic d id
7-50
2
12 21 1
11 if 2
( , ) (7.51)1 ( )
0 otherwise
i ii
i ii
c dc d k c d
2
( , )( , ) (7.54)
1 ( ) ( )
c dc d
k f c f d
( ) ( 1)
0 1( ) min ( , )
( , ) max ( )
n n
n N
M
c c
c d
7-51
7-52
Thresholding yields the segmentation result.
7-53
Extensions: Relative fuzzy connectivity,
7-54
Multi-seeded fuzzy connectivity
Scale-based fuzzy connectivity
7.6. Graph Cut-Segmentation
7-55
Idea:Seeds may be
interactively or
automatically
identified.
7-56
(1) Construct a graph with n+2 nodes
(2) Each node has 4 n-links, connected to neighbors. Each link is assigned a cost derived from smoothness term , where f: image valuesmooth ( )C f
Steps: Given an image of size n
(3) Each node has 2 t-links, connected to source S and sink T. Each link is assigned a cost derived from data term data ( )C f
7-57
(5) Find the minimum cut that minimizes
data smooth( ) ( ) ( )C f C f C f
(4) Identify seed pixels and define “hard constraint” for their links
(6) Obtain the segmentation result.
7-58
7-59
( ) ( )p pp I
R R L
L
( , )( , )
( ) ( , )p q p qp q N
B B L L
L
1 if ( , )
0 otherwisep q
p q
L LL L
: the resulting labeling vector
Let I : the set of pixels
Example: Given an image
N: a set of neighborhood pixel pairs ( , ), ,p q p q I
1 2( , , , ),IL L L L {object, background}iL
L is obtained by finding the cut that minimizes the cost where
: regional property term
: boundary property term
( ) ( ) ( ),C R B L L L
7-60
The costs of arcs are defined in the following table
e.g.,
2
2
(object) ln ( | ), (background) ln ( | )
( )1( , ) exp
, 2
p p p p
p q
R P I O R P I B
I IB p q
p q
7-61
Example:
7-62