texture based image segmentation - mit csailpeople.csail.mit.edu/jchang7/pubs/presentations/... ·...

Texture Based

Image Segmentation

Jason Chang

Massachusetts Institute of Technology

Master’s Degree Candidate

October 15, 2008

Outline

• The Goal

• Level Set Methods

• Previous Work

• Extensions / Improvements• Extensions / Improvements

– Bias Field Estimation

– Texture Based Segmentation

Image Segmentation

• Separate the image into separate regions

• Focus on Binary Segmentation (two regions, one curve)

The Goal

Level Set Methods

Previous Work

Bias Field Estimation

Texture Segmentation

Level Set Methods

• Curve evolution is defined by an energy functional to minimize over

• Allows for easy manipulation

• Implicitly define a curve on the image with a surface in 3D

The Goal

Level Set Methods

Previous Work



Level Set Methods

• Define a height at every pixel in the image

The Goal

Level Set Methods

Previous Work



The SurfaceThe Level Sets / Contours

of the Surface

Level Set Methods

• The zero level set represents the 2D curve

The Goal

Level Set Methods

Previous Work



Some Notation

Notation Meaning Comments

The pixel location

The intensity value at pixel j

j

( ) jx j x≡

{ }1,2,...,j N∈

{ }0,1,...,255j

x ∈

The Goal

Level Set Methods

Previous Work



The level set function at pixel j

The label assigned to pixel j

The segmented regions

The curve that segments the

image (zero level set)

( ) ( )signj jL j L φ≡ = { }1, 1j

L ∈ + −

{ }| 1j

j LR± = = ±

R+

R−

( ) jjφ φ≡

{ }| 0j

C j φ= =

C

Segmentation Criterion

• Maximize mutual information between pixel intensity and labeling

• Approximate MI by using a Kernel Density Estimate to find the PDFs

( );J J

I x L

( ) ( )|ˆ

1ˆ

J

j s

jx J R x

s

j

R

x xp x K

hp x

hR±

±

±

∈∈±

− ≡ =

∑

The Goal

Level Set Methods

Previous Work



{ }~ 1, ,J U N…

• Use the Gaussian Kernel±

( )2

2

2

1 x

K x eπ

−=

J. Kim, J.W. Fisher III, A. Yezzi, M. Cetin, and A.S. Willsky. A nonparametric statistical method for image segmentation using information theory and

curve evolution. Image Processing, IEEE Transactions on Image Processing, 14(10):1486-1502, Oct. 2005.

• Minimize the energy functional

• Gradient flow that minimizes the energy

Curve Length Penalty

Junmo’s Algorithm

j C∀ ∈

( )( )

( )( )

( )( )

1 1lo

ˆ

ˆ ˆ ˆg

j i j i jj

R R

xx K x K xp x x

N Nt p p p

di diR Rx x x

ακφ

+ −

+

−−

+ −+

∂= + − −

− −

∂

∫ ∫

� �

( ) ( );ˆJ J

CE C N LI x dsα= − ⋅ + ∫�

The Goal

Level Set Methods

Previous Work



• Approximate Gradient Descent

( ) ( ) ( )ˆ ˆ ˆRjx

Rj jx x

t p p pR Rx x x+ −−+

∂

∫ ∫

Computationally Intensive

( )( )

lˆ

ˆog

jx

x

j

j

pN N

x

t xp

φακ

+

−

∂−

∂≈

� �

j C∀ ∈



Junmo’s AlgorithmThe Goal

Level Set Methods

Previous Work





Comments

• Why is the Scalar Segmentation Algorithm Good?

– Segmentation based on non-parametric densities

– No training required

• What needs to be improved?

The Goal

Level Set Methods

Previous Work



– Does not perform well on images with lighting effects

– Only supports grayscale image segmentation

– Textured images can not be segmented


• Assume the observed image is the product of an intrinsic image and

a multiplicative gain field

x =

The Goal

Level Set Methods

Previous Work



Basic Intrinsic Image Gain Field Observed Image

x =

b x g = x


• Bias field is the log of the Gain field

• Assume that the intrinsic image pixels, bj, are i.i.d. conditioned on

knowing the regions R±

• Find the MAP estimate of β for a given segmentation

j j jx b g= ×

The Goal

Level Set Methods

Previous Work



( ) ( ) ( ) ( )log log log logj j j j j jy x b g b β= = + = +

• Find the MAP estimate of β for a given segmentation

• Use a fixed-point iteration to find β

( ) ( )( )

( )MAP

ˆPr | ,ˆ ,

ˆPr | ,

jj y j j ji

jj y j j ji

L i p y L

L i p y L

β

β

β

β

∂∂ = = Λ = =

∑∑

β f β f β

( ) ( )( )1ˆ ˆk k

β

+= Λβ f β

W.M. Wells, III, W.E.L. Grimson, R. Kikinis and F.A. Jolesz. Adaptive segmentation of MRI data. IEEE Transactions on Medical Imaging, Vol. 15, pp.

429-442, 1995.


Segmentation Algorithm with Bias Field Estimation

1. Assume that the bias field, β, is zero and the intrinsic image is just

the observed image

2. Segment the estimated intrinsic image

3. Estimate the bias field, β, given the current segmentation

The Goal

Level Set Methods

Previous Work



3. Estimate the bias field, β, given the current segmentation

4. Find the estimated intrinsic image, b, from the estimated bias field

5. Repeat from Step 2 until convergence

• Alternate between segmentation and bias field estimation

Bias Field EstimationThe Goal

Level Set Methods

Previous Work



Vector Segmentation

• Extending the formulation to vector values (notice the bold vector ri,

instead of the scalar xi)

• Vector-valued images can be segmented

( )( )

ˆlog

ˆ

jx

x

j

j

pN N

t p

φακ

+

−

∂≈ − ∂

v

v

� �

The Goal

Level Set Methods

Previous Work



• Vector-valued images can be segmented

– Color images are segmented using

– Texture images can be segmented by representing each pixel

with a texture vector

[ ], ,j

R G B=v

Steerable Pyramid

Recursively replace the gray box at the red dot

The Goal

Level Set Methods

Previous Work



Φ0

Φ2

Φ1

x

…

ΦD-1

xΘ0

Θ2

Θ1

ΘD-1

z0

zD-1

z1

z2

…

+

+

+

v0

vD-1

v1

v2

Eero P. Simoncelli and William T. Freeman. The steerable Pyramid: A flexible architecture for multi-scale derivative computation. In International

Conference on Image Processing, volume 3, pages 444-447, 23-26 Oct. 1995, Washington DC, USA, 1995

Steerable Pyramid

Varying OrientationOriginal

High-Pass (v0)

The Goal

Level Set Methods

Previous Work



VaryingScale

Low-Pass (vD)

(v1, v2, v3, … , vD-1)

Dimensionality Reduction

• Estimating PDFs using the Improved Fast Gauss Transform (Yang

et al.) has complexity:

D – dimensionality M – # target points N – # source points

( )( )cO D M N+

The Goal

Level Set Methods

Previous Work



• Problems with high dimensionality

– Takes ~1 hour to do a KDE on 14 dimensions

– Sparse data in 14 dimensions provide for poor estimate

C. Yang, R. Duraiswami, N. Gumerov, and L. David. Improved fast gauss transform and efficient kernel density estimation. IEEE International

Conference on Computer Vision, pages 464-471, 2003.

Dimensionality Reduction

• We can reconstruct the original image from the outputs

• Approximately reconstruct x by using a subset of the outputs

• Define the error of the reconstruction as the MSE

0 0

D D

i i i

i i= =

= Θ =∑ ∑x z v

{ }1

0

ˆ 0,1D

D

i i

i

u−

=

= ∈∑x v u

The Goal

Level Set Methods

Previous Work



• Define the error of the reconstruction as the MSE

• Perform the following optimization

( ) ( ) ( )2

2ˆ ˆ ˆ ˆ,

Te = − = − −x x x x x x x x

{ }( )

0,1

1

0

ˆarg min ,

. .

D

D

i

i

e

s t u d

∗

∈

−

=

=

= =∑

u

u x x

u

Pyramid Subset Results

Ground Truth

Oriented Stripes Scaled Stripes

The Goal

Level Set Methods

Previous Work



Scaled Checkerboard Different Textures

(Using d=3)

Smoothly Varying Textures

• Try to capture a texture that varies smoothly in orientation and scale

The Goal

Level Set Methods

Previous Work



Smoothly Varying TexturesThe Goal

Level Set Methods

Previous Work



Smoothly changing

orientation

Smoothly Varying TexturesThe Goal

Level Set Methods

Previous Work



Outputs at 4 orientations and 1 scale

Smoothly Varying Textures

Junmo’s Scalar Segmentation

The Goal

Level Set Methods

Previous Work



Vector Segmentation (Pyramid Subset)

Vector Segmentation (Smoothly Varying Textures)

Thanks!

Questions / Comments?

texture based image segmentation - mit csailpeople.csail.mit.edu/jchang7/pubs/presentations/... ·...

Documents