4/15/2008 1

Lecture 10 (4.14.07)

Image Representation and Description

Shahram Ebadollahi

DIP ELEN E4830

4/15/2008 2

Image

Description

Recognition

High-level Image Representation

Image Understanding

4/15/2008 3

Lecture Outline

� Image Description� Shape Descriptors� Texture & Texture Descriptors� SIFT� Motion Descriptors� Color Descriptors

4/15/2008 4

Shape Description

� Shape Represented by its Boundary� Shape Numbers, � Fourier Descriptors, � Statistical Moments

� Shape Represented by its Interior� Topological Descriptors� Moment Invariants

4/15/2008 5

Boundary Representation: (Freeman) Chain Code

Sub-sampled boundary Chain code of boundaryOriginal boundary

Chain code for 4-neighborhood

Chain code for 8-neighborhood

Boundary representation = 0766666453321212

4/15/2008 6

8-directional chain code � 00006066666666444444242222202202

Rotation normalized chain code � 0006200000006000006260000620626

Chain Code: example

Starting point normalized chain code � 00006066666666444444242222202202

First difference of chain code

4/15/2008 7

Shape Number –A boundary descriptor

4/15/2008 8

1,,2,1,0)()()( −=+= Kkkjykxks �

1,,2,1,0)()(1

0

/2 −== ∑−

=

− KueksuaK

k

Kukj�

π

1,,2,1,0)(1

)(1

0

/2 −== ∑−

=

KkeuaK

ksK

u

Kukj�

π

Fourier Descriptor

Boundary descriptor – Fourier

4/15/2008 9

50% 10% 5%

2.5% 1.25%

100%

0.63% 0.28%

Only 8 descriptors

2868 descriptors

Boundary Reconstruction using Fourier Descriptors

4/15/2008 10

Boundary Representation: Signatures

• Represent 2-D boundary shape using 1-D signature signal

4/15/2008 11

Boundary Representation: Signatures

4/15/2008 12

Boundary Description using Statistical Moments

( )∑−

=

−=1

0

)()(A

ii

nin vpmvvµ n-th moment of v

∑−

=

=1

1

)(A

iii vpvm

4/15/2008 13

� Area� Perimeter� Compactness� Circularity Ratio� Mean/Median intensity� Max/Min intensity� Normalized area

Region Descriptors - Simple

(perimeter)2/Area

r rπ

2r

b

a=2b

π4 16π5

π4/2P

ARc =

Area of circle with same perimeter as the shape

:cR

:C

1 8.054 ≈ 78.04 ≈π

4/15/2008 14

4/15/2008 15

Topological Region Descriptors• Topological properties: Properties of image preserved under rubber-sheet distortions

H: # holes in the image

C: # connected components

E = C-H: Euler Number

H=2, C=1, E=-1

H=0, C=3, E=3

H=1, C=1, E=0 H=2, C=1, E=-1

EHCFQV =−=+−

4/15/2008 16

Geometric Moment Invariants

∫ ∫= dxdyyxfyxm qppq ),( (p+q)-th 2D geometric moment

Projection of f(x,y) onto monomial qp yx

• Why use moments?

• Geometric moments of different orders represent spatial characteristics of the image intensity distribution

00m

00010

00100

/

/

mmy

mmx

==

Total intensity of image. For binary image � area

Intensity centroid

binary image � geometrical center

∑∑−

=

−

=

=1

0

1

0

),(M

x

N

y

qppq yxfyxm

4/15/2008 17

Central Moments

∑∑−

=

−

=

−−=1

0

1

000 ),()()(

M

x

N

y

qppq yxfyyxxµ

[Translation invariance]

0000 m=µ00110 == µµ

2002,µµ

11µ

Variance about the centroid

covariance

2/)2(00 )'/(' ++= qp

pqpq µµλ Scale and translation invariant

Scaled Central Moment

2' ++=

qp

pqpq α

µµ

2/)2(00)/( ++= qp

pqpq µµη

Normalized Un-Scaled Central Moment

4/15/2008 18

Moment Invariants (translation, scale, mirroring, rotation)

ηηϕ += 201

02201 ηηφ +=211

202202 4)( ηηηφ +−=

20321

212303 )()( ηηηηφ −+−=

20321

212304 )()( ηηηηφ +++=

�=5φ

�=6φ�=7φ

4/15/2008 19

Affine Transform & Affine Moment Invariants

),(' yxTx x=

),(' yxTy y=

∑∑=

−

=

=m

r

rm

k

krrk yxax

0 0

'

∑∑=

−

=

=m

r

rm

k

krrk yxby

0 0

'

In practice: bilinear transformxyayaxaax 3210' +++=xybybxbby 3210' +++=4 pairs of corresponding points

needed to find coefficients

In practice: affine transform

3 pairs of corresponding points needed to find coefficients

yaxaax 210' ++=ybxbby 210' ++=

φφ sincos' yxx +=φφ cossin' yxy +−=

axx ='byy ='

φtan' yxx +=yy ='

Rotation: Scale: Skew:

4/15/2008 20

2

]4)[()(

2

]4)[()(

2/1211

202200220

2

2/1211

202200220

1

µµµµµ

µµµµµ

+−−+=

+−++=

I

I

Principal moment of inertia:

θ

)2

(tan5.00220

111

µµµθ−

= −

2/1002

2/1001 )/(2)/(2 µµ IbIa ==

Image ellipse characterizes fundamental shape features and also 2D position and orientation

20021 /)( mII +

)/()( 2112 IIII +−

spreadness

elongation

Elliptical Shape Descriptors

4/15/2008 21

Texture - Definition

4/15/2008 22

Texture – Quantification Methods

� Statistical: compute local features at each point in image and derive a set of statistics from the distribution of local features

� 1st, 2nd, and higher-order statistics based on how many points are used to define local features

� Structural: texture is considered to be composed of “texture elements”. Properties of the “texture elements” and their spatial placement rules characterizes the texture

� Original texture can be reconstructed from its structural description

4/15/2008 23

Statistical Texture Analysis1st order statistics

image histogramfhf →

• Obtain statistics of the histogram:

Mean:

Variance:

Skewness:

Entropy:

∑−

=

1

0

)(L

i

iih

∑−

=

−1

0

2 )()(L

i

ihi µ

∑−

=

−1

0

3 )()(L

i

ihi µ

∑−

=

−1

0

)(log)(L

i

ihih Measure of variability of intensity

: average intensity

: measure of intensity contrast

4/15/2008 24

4/15/2008 25

Statistical Texture Analysis1st order statistics

imag

ehi

stog

ram

Skewness = 2.08

Entropy = 0.88

Skewness = 2.44

Entropy = 0.77

Skewness = -0.092

Entropy = 0.97stat

istic

s

4/15/2008 26

Statistical Texture Analysis2nd order statistics: Co-occurrence

� Joint gray-level histogram of pairs of pixels� 2D histogram

inmf =),( 11

jnmf =),( 22

θd

1

2

L

1 2 L

i

j

]),(,),(Pr[),( 2211),( jnmfinmfjiPd ==≈θ

),(),( jiP d θ

4/15/2008 27

Statistical Texture Analysis2nd order statistics: Co-occurrence

j

k

°0

°45

°90°135

|}),(,),(,||,0

:)()()),(),,{((|),()0,(

jnmfilkfdnlmk

NMNMnmlkjiPd

===−=−

×××∈=°=θ

|}),(,),(),,(),(

:)()()),(),,{((|),()45,(

jnmfilkfdnldmkdnldmk

NMNMnmlkjiPd

===−−==∨−=−=−

×××∈=°=θ

),()90,(

jiPd °=θ

),()135,(

jiPd °=θ

{ }� is set cardinality

4/15/2008 28

Statistical Texture Analysis2nd order statistics: Co-occurrence (example)

image Co-occurrence matrix

4/15/2008 29

Statistical Texture Analysis2nd order statistics: Co-occurrence (statistics)

Angular 2nd moment (energy):

Maximum Probability:

Contrast:

Correlation:

),(1 1

2),( jiP

L

i

L

jd∑∑

= =θ

),(max ),(,

jiPdji

θ

),(log),( ),(1 1

),( jiPjiP d

L

i

L

jd θθ∑∑

= =

−Entropy:

∑∑= =

−L

i

L

jd jiPji

1 1),( ),(λ

θκ

yx

L

iyx

L

jd jiijP

σσ

µµθ∑∑= =

−1 1

),( )],([

∑ ∑= =

=L

i

L

jdx jiPi

1 1),( ),(θµ ∑ ∑

= =

−=L

i

L

jdxx jiPi

1 1),(

2 ),()( θµσ

(measure of image homogeneity)

(measure of local variations)

(measure of image linearity)

4/15/2008 30

4/15/2008 31

Statistical Texture Analysis2nd order statistics: Difference Statistics

Angular 2nd moment (energy):

Mean:

Contrast:

)(1

0

2),( kP

L

kd∑

−

=θ

∑−

=

1

0),( )(

L

kd kkP θ

∑−

=

−1

0),(),( )(log)(

L

kdd kPkP θθEntropy:

∑−

=

1

0),(

2 )(L

kd kPk θ

∑=−

∈=

kji

Ljidd jiPkP

||

},,1{,),(),( ),()(

�

θθ is a subset of co-occurrence matrix

4/15/2008 32

Statistical Texture Analysis2nd order statistics: Autocorrelation

∑∑

∑∑

= =

−

=

−

=

++

−−=

M

k

N

l

pM

k

qN

lff

lkf

qlpkflkf

qNpM

MNqpC

1 1

2

1 1

),(

),(),(

))((),(

Large texture elements � autoccorrelation decreases slowly with increasing distance

Small texture elements � autoccorrelation decreases rapidly with increasing distance

Periodic texture elements � periodic increase & decrease in autocorrelation value

4/15/2008 33

Statistical Texture Analysis2nd order statistics: Fourier Power Spectrum

),(),( vuFyxf ↔

2|),(|),( vuFvuP =

∑=

=π

θθ

0

),(2)( rPrP

∑=

=2/

0

),()(L

r

rPP θθ

Power Spectrum

u

v

u

v

Indicator for size of dominant texture element or texture coarseness

Indicator for the directionality of the texture

Note:

}|),({| 21 vuFFC ff−=

4/15/2008 34

Law’s Texture Energy Measures

]1,2,1[3 =L ]1,0,1[3 −=E ]1,2,1[3 −−=S

]1,2,0,2,1[

]1,4,6,4,1[

]1,0,2,0,1[

]1,2,0,2,1[

]1,4,6,4,1[

5

5

5

5

5

−−−=−−=

−−=−−=

=

W

R

S

E

L

−−−−−−−−−−

=×

10201

40804

601206

40804

10201

55 SLT•Convolute different Law’s masks with image

• Compute energy statistics

4/15/2008 35

4/15/2008 36

4/15/2008 37

4/15/2008 38

4/15/2008 39

4/15/2008 40

Motion – object

1f 2f

object motion

ε≤−= |),(),(|0),( 21 jifjififjid

otherwise1

Difference image:

Cumulative difference image

absolute positive negative

No motion direction information !

∑=

−=n

kkkcum jifjifajid

11 |),(),(|),(

Tells us how often the image gray level was different from gray-level of reference image

4/15/2008 41

Motion Field

• A velocity vector is assigned to each pixel in the image

• Velocities due to relative motion between camera and the 3D scene

• Image change due to motion during a time interval dt

• Velocity field that represents 3-dimensional motion of object points across 2-dimensional image

Motion field

4/15/2008 42

Optical flow

• Motion of brightness patterns in image sequence

• Assumptions for computing optical flow:

• Observed brightness of any object point is constant over time

• Nearby points in the image plane move in a similar manner

)(),,(),,( 2∂+∂∂+

∂∂+

∂∂+=+++ Odt

t

fdy

y

fdx

x

ftyxfdttdyydxxf

)(),,( 2∂++++= Odtfdyfdxftyxf tyx

dt

dyf

dt

dxfftyxfdttdyydxxf yxt +=−⇒=+++ ),,(),,(

),(),( vudt

dy

dt

dxc ==

cfvfuff yxt .∇=+=−

Gray-level difference at same location over time is

equivalent to product of spatial gray-level difference and

velocity

known unknown

4/15/2008 43

Optical Flow Constraints

• no spatial change in brightness, induce no temporal change in brightness � no discernible motion

• motion perpendicular to local gradient induce no temporal change in brightness � no discernible motion

• motion in direction of local gradient, induce temporal change in brightness � discernible motion

• only motion in direction of local gradient induces temporal change in brightness and discernible motion

vfuff yxt +=−

4/15/2008 44

Optical flow != Motion Field

0

0

=≠

OF

MF0

0

≠=

OF

MF

4/15/2008 45

Which descriptors?Image Feature Evaluation

1. Prototype Performance� Classify (Segment) image using

different features� Evaluate which feature is optimal

(minimum classification error)

2. Figure of Merit� Establish functional distance

measurements between set of image features (large distance �low classification error)

� Bhattacharyya distance