contourlets: construction and...

Contourlets: Construction and Properties

Minh N. Do

Department of Electrical and Computer EngineeringUniversity of Illinois at Urbana-Champaign

www.ifp.uiuc.edu/∼[email protected]

Joint work with Martin Vetterli (EPFL and UC Berkeley),Arthur Cunha, Yue Lu, Jianping Zhou (UIUC)

Outline

1. Motivation

2. Discrete-domain construction using filter banks

3. Contourlets and directional multiresolution analysis

4. Contourlet approximation

5. Contourlet filter design with directional vanishing moments

1. Motivation 1

What Do Image Processors Do for Living?

Compression: At 158:1 compression ratio...

1. Motivation 2


Denoising (restoration/filtering)

Noisy image Clean image

1. Motivation 3


Feature extraction (e.g. for content-based image retrieval)

Featureextraction

Featureextraction

Images

Query

Similarity Measurement

Signatures

N best matched images

1. Motivation 4

Fundamental Question: Parsimonious Representation of

Visual Information

A randomly generated image A natural image

Natural images live in a very tiny bit of the huge “image space” (e.g.R

512×512×3)

1. Motivation 5

Mathematical Foundation: Sparse Representations

Fourier, Wavelets... = construction of bases for signal expansions:

f =∑

n

cnψn, where cn = 〈f, ψn〉.

Non-linear approximation:

fM =∑

n∈IM

cnψn, where IM : indexes of best M components.

Sparse representation: How fast ‖f − fM‖ → 0 as M →∞.

1. Motivation 6

Wavelets and Filter Banks

2

2

2

2x

analysis synthesis

H1

H0

G1

G0

y1

y0 x0

x1

+ xMallat

G12

G02

G12

G02

++ Daubechies

1. Motivation 7

The Success of Wavelets

• Wavelets provide a sparse representation for piecewise smooth signals.

• Multiresolution, tree structures, fast transforms and algorithms, etc.

• Unifying theory ⇒ fruitful interaction between different fields.

1. Motivation 8

Fourier vs. Wavelets

Non-linear approximation: N = 1024 data samples; keep M = 128coefficients

0 100 200 300 400 500 600 700 800 900 1000−20

0

20

40

Original

0 100 200 300 400 500 600 700 800 900 1000−20

0

20

40

Using wavelets (Daubechies−4): SNR = 37.67 dB

0 100 200 300 400 500 600 700 800 900 1000−20

0

20

40

Using Fourier (of size 1024): SNR = 22.03 dB

Approximation movie!

1. Motivation 9

Is This the End of the Story?

1. Motivation 10

Wavelets in 2-D

• In 1-D: Wavelets are well adapted to abrupt changes or singularities.

• In 2-D: Separable wavelets are well adapted to point-singularities (only).

But, there are (mostly) line- and curve-singularities...

1. Motivation 11

The Failure of Wavelets

Wavelets fail to capture the geometrical regularity in images andmultidimensional data.

1. Motivation 12

Edges vs. Contours

Wavelets (with nonlinear approximations) cannot “see” the differencebetween these two images.

• Edges: image points with discontinuity

• Contours: edges with localized and regular direction [Zucker et al.]

1. Motivation 13

Goal: Efficient Representation for Typical Images

with Smooth Contours

��

��regions

smooth

boundary

smooth

Goal: Exploring the intrinsic geometrical structure in natural images.

⇒ Action is at the edges!

1. Motivation 14

Wavelet vs. New Scheme

Wavelets New scheme

For images:

• Wavelet scheme... see edges but not smooth contours.

• New scheme... requires challenging non-separable constructions.

1. Motivation 15

And What The Nature Tells Us...

• Human visual system:

– Extremely efficient: 107 bits −→ 20-40 bits (per second).– Receptive fields are characterized as localized, multiscale and oriented.

• Sparse components of natural images (Olshausen and Field, 1996):

16 x 16 patches

from natural images

Search forSparse Code

1. Motivation 16

Recent Breakthrough from Harmonic Analysis:

Curvelets [Candes and Donoho, 1999]

• Optimal representation for functions in R2 with curved singularities.

• Key idea: parabolic scaling relation for C2 curves:

width ∝ length2

uv

l

w

u = u(v)

1. Motivation 17

“Wish List” for New Image Representations

• Multiresolution ... successive refinement

• Localization ... both space and frequency

• Critical sampling ... correct joint sampling

• Directionality ... more directions

• Anisotropy ... more shapes

Our emphasis is on discrete framework that leads toalgorithmic implementations.

1. Motivation 18

Outline

1. Motivation





2. Discrete-domain construction using filter banks 19

Challenge: Being Digital!

Pixelization:

Digital directions:


Proposed Computational Framework: Contourlets

In a nutshell: contourlet transform is an efficient directional multiresolutionexpansion that is digital friendly!

contourlets = multiscale, local and directional contour segments

• Starts with a discrete-domain construction that is amenable to efficientalgorithms, and then investigates its convergence to a continuous-domainexpansion.

• The expansion is defined on rectangular grids ⇒ seamless translationbetween the continuous and discrete worlds.


Discrete-Domain Construction using Filter Banks

Idea: Multiscale and Directional Decomposition

• Multiscale step: capture point discontinuities, followed by...

• Directional step: link point discontinuities into linear structures.


Analogy: Hough Transform in Computer Vision

Input image Edge image “Hough” image

=⇒ =⇒

Challenges:

• Perfect reconstruction.

• Fixed transform with low redundancy.

• Sparse representation for images with smooth contours.


Multiscale Decomposition using Laplacian Pyramids

decomposition reconstruction

• Reason: avoid “frequency scrambling” due to (↓) of the HP channel.

• Laplacian pyramid as a frame operator → tight frame exists.

• New reconstruction: efficient filter bank for dual frame (pseudo-inverse).


Directional Filter Banks (DFB)

• Feature: division of 2-D spectrum into fine slices using tree-structuredfilter banks.

37

6

5

4 7

6

5

4

0123

0 1 2

ω1

ω2 (π, π)

(−π,−π)

• Background: Bamberger and Smith (’92) cleverly used quincunx FB’s,modulation and shearing.

• We propose:

– a simplified DFB with fan FB’s and shearing– use DFB to construct directional bases


Multichannel View of the Directional Filter Bank

+x

H0

H1

H2l−1

G0

G1

G2l−1

S0S0

S1S1

S2l−1S2l−1

y0

y1

y2l−1

x

Use two separable sampling matrices:

Sk =

[

2l−1 0

0 2

]

0 ≤ k < 2l−1 (“near horizontal” direction)

[

2 0

0 2l−1

]

2l−1 ≤ k < 2l (“near vertical” direction)


General Bases from the DFB

An l-levels DFB creates a local directional basis of l2(Z2):

{

g(l)k [· − S

(l)k n]

}

0≤k<2l, n∈Z2

• G(l)k are directional filters:

• Sampling lattices (spatial tiling):

2

2

2l−1

2l−1


Pyramidal Directional Filter Banks (PDFB)

Motivation: + add multiscale into the directional filter bank+ improve its non-linear approximation power.

image

subbandsdirectionalbandpass

directionalsubbands

bandpass

(2,2) ω1

ω2 (π, π)

(−π,−π)

Properties: + Flexible multiscale and directional representation for images(can have different number of directions at each scale!)

+ Tight frame with small redundancy (< 33%)

+ Computational complexity: O(N) for N pixels.


Wavelets vs. Contourlets

50 100 150 200 250

50

100

150

200

250

50 100 150 200 250

50

100

150

200

250


Examples of Discrete Contourlet Transform


Outline

1. Motivation





3. Contourlets and directional multiresolution analysis 31

Multiresolution Analysis: Laplacian Pyramid

w1

w2

Vj

Wj

Vj−1

Vj−1 = Vj ⊕Wj,

L2(R2) =⊕

j∈Z

Wj.

Vj has an orthogonal basis {φj,n}n∈Z2, where

φj,n(t) = 2−jφ(2−jt− n).

Wj has a tight frame {µj−1,n}n∈Z2 where

µj−1,2n+ki= ψ

(i)j,n, i = 0, . . . , 3.


Directional Multiresolution Analysis: LP + DFB

��

��

��

��

replacements

Vj

Wj

Vj−1

W(lj)

j,k

ω1

ω2

2j

2j+lj−2

ρ(lj)

j,k,n

Wj =2

lj−1⊕

k=0

W(lj)

j,k

W(lj)

j,k has a tight frame{

ρ(l)j,k,n

}

n∈Z2where

ρ(l)j,k,n(t) =

∑

m∈Z2

g(l)k [m− S

(l)k n]

︸︷︷︸DFB basis

µj−1,m(t)︸︷︷︸

LP frame

= ρ(l)j,k(t− 2j−1

S(l)k n).


Contourlet Frames

Theorem (Contourlet Frames) [DoV:03].{

ρ(lj)

j,k,n

}

j∈Z, 0≤k<2lj , n∈Z2

is a tight frame of L2(R2) for finite lj.

Theorem (Connection with Filter Banks) [DoV:04]Suppose x[n] = 〈f, φL,n〉, n ∈ Z

2, for some function f ∈ L2(R2).Furthermore, suppose

xPDFB−→ (aJ , d

(lj)

j,k )j=1,...,J; k=0,...,2

lj−1

where aJ is the lowpass subband, and d(lj)

j,k are bandpass directionalsubbands. Then

aJ [n] = 〈f, φL+J,n〉

d(lj)

j,k [n] = 〈f, ρ(lj)

L+j,k,n〉

for j = 1, . . . , J ; k = 0, . . . , 2lj − 1, n ∈ Z2.


Sampling Grids of Contourlets

(a) (b)

(c) (d)

w

w

l

l

w/4

w/4

l/2

l/2


Contourlet Features

• Defined via iterated filter banks ⇒ fast algorithms, tree structures, etc.

• Defined on rectangular grids ⇒ seamless translation between continuousand discrete worlds.

• Different contourlet kernel functions (ρj,k) for different directions.

• These functions are defined iteratively via filter banks.

• With FIR filters ⇒ compactly supported contourlet functions.


Contourlet Packets

• Adaptive scheme to select the “best” tree for directional decomposition.

ω1

ω2 (π, π)

(−π,−π)

• Contourlet packets ⇒ directional multiresolution elements with differentshapes (aspect ratios).

– They do not necessary satisfy the parabolic relation.– They can include wavelets!


Outline

1. Motivation





4. Contourlet approximation 38

Contourlets with Parabolic Scaling

Support size of the contourlet function ρljj,k: width ≈ 2j and length ≈ 2lj+j

To satisfy the parabolic scaling (for C2 curved singularities):width ∝ length2, simply set:

the number of directions in the PDFB is doubled at every other finer

scale.

uv

l

w

u = u(v)

ω1

ω2 (π, π)

(−π,−π)


Supports of Contourlet Functions

*

Contourlet

=*

DFBLP

=

Key point: Each generation doubles spatial resolutionas well as angular resolution.


Contourlet Approximation

see a wavelet

ΠLΠL

Desire: Fast decay as contourlets turn away from the discontinuity direction

Key: Directional vanishing moments (DVMs)


Geometrical Intuition

At scale 2j (j ≪ 0):

width ≈ 2j

length ≈ 2j/2

#directions ≈ 2−j/2

��

��

2j

2j/2 dj,k,n

d2j,k,n

θj,k,n

|〈f, ρj,k,n〉| ∼ 2−3j/4 · d3j,k,n

dj,k,n ∼ 2j/ sin θj,k,n ∼ 2j/2k−1 for k = 1, . . . , 2−j/2

=⇒ |〈f, ρj,k,n〉| ∼ 23j/4k−3


How Many DVMs Are Sufficient?

d

d^2

��

��

alpha

��

��

��

��

ω1

ω2

Sufficient if the gap to a direction with DVM:

α . d ∼ 2j/2k−1 for k = 1, . . . , 2−j/2

This condition can be replaced with fast decay in frequency across directions.

It is still an open question if there is an FIR filter bank that satisfies thesufficient DVM condition


Experiments with Decay Across Directions using

Near Ideal Frequency Filters

0 0.5 1 1.5 2 2.5 3 3.5 4−18

−16

−14

−12

−10

−8

−6Disc 256 coefficients decaying plot (average)

Log2

(|<

f, h>

|)

Log2(k)

0 0.5 1 1.5 2 2.5 3 3.5 4 4.5 5−22

−20

−18

−16

−14

−12

−10

−8

−6Disc 256 coefficients decaying plot (average)

Log2

(|<

f, h>

|)

Log2(k)


Nonlinear Approximation Rates

��

��C2 regions C2 boundary

Under the (ideal) sufficient DVM condition

|〈f, ρj,k,n〉| ∼ 23j/4k−3

with number of coefficients Nj,k ∼ 2−j/2k. Then

‖f − f(contourlet)M ‖2 ∼ (logM)3M−2

While ‖f − f(Fourier)M ‖2 ∼ O(M−1/2) and ‖f − f

(wavelet)M ‖2 ∼ O(M−1)


Non-linear Approximation Experiments

Image size = 512× 512. Keep M = 4096 coefficients.

Original image Wavelets: Contourlets:PSNR = 24.34 dB PSNR = 25.70 dB


Detailed Non-linear Approximations

Wavelets

M = 4 M = 16 M = 64 M = 256

Contourlets

M = 4 M = 16 M = 64 M = 256


Denoising Experiments

original image noisy image (SNR = 9.55 dB)

wavelet denoising (SNR = 13.82 dB) contourlet denoising (SNR = 15.42 dB)


Embedded Structure for Compression and Modeling

• So far, best-M term approximation:

fM =∑

λ∈IM

cλρλ, where IM is the set of indexes of the M -largest |cλ|.

• For compression, additional cost required to specify IM

– Naive approach: M · log2N bits– With embedded tree (wavelets): M bits

• Embedded trees for wavelets are crucial in state-of-the-art imagecompression (EZW,...), rate-distortion analysis (Cohen et al.), andmultiscale statistical modeling (Baraniuk et al.)


Contourlet Embedded Tree Structure

Embedded tree data structure forcontourlet coefficients:

successively locate the positionand direction of image contours.

Since significant contourlet coefficientsare organized in trees, best M -treeapproximation (using M -node tree):

‖f − f(contourlet)M−tree ‖2 ≈ (logM)3M−2

⇒ D(R) ≈ (logR)3R−2


Outline

1. Motivation





5. Contourlet filter design with directional vanishing moments 51

Filter Bank Design Problem

ρ(l)j,k(t) =

∑

m∈Z2

c(l)k [m]φj−1,m(t)

ρ(l)j,k(t) has an L-order DVM along direction (u1, u2)

⇔ C(l)k (z1, z2) = (1− zu2

1 z−u12 )LR(z1, z2)

So far: Use good frequency selectivityto approximate DVMs.

Draw back: long filters...

��

��

��

��

ω1

ω2

Next: Design short filters that lead to many DVMs as possible.


Filters with DVMs = Directional Annihilating Filters

Input image and after being filtered by a directional annihilating filter


Perfect Reconstruction Two-Channel FBs with DVMs

Filter bank with order-L horizontal or vertical DVM:

T0

Q

Q

Q

Q

T1

Filter design amounts to solve

P (z) + P (−z) = 2,

where P (z) = H0(z)G0(z) = (1− z1)LR(z).

To get DVMs at other directions: Shearing or change of variables

≡

H0(ω) H0(RT0 ω)

R0 R1


Complete Characterization

Proposition [Cunha-D., 04]. Any FIR solution of

(1− z1)LR(z) + (1 + z1)

LR(−z) = 2 (1)

can be written as

R(z) = AL(z1) + (1 + z1)LB(z)

where AL(z1) is the minimum degree 1-D polynomial in z1 that solves (1)

AL(z1) =

L−1∑

i=0

(L+ i− 1

L− 1

)

2−(L+i−1)(1− z1)i,

and B(z) satisfy B(z) +B(−z) = 0.

Proof. Applying the Bezout theorem for each z2.


Design via Mapping (Cunha-D., 2004)

To avoid 2D factorization...

1. Start with a 1-D solution:

P (1D)(z) + P (1D)(−z) = 2 and P (1D)(z) = H(1D)0 (z)G

(1D)0 (z)

2. Apply a 1-D to 2-D mapping to each filter:

F (1D)(z) → F (2D)(z) = F (1D)(M(z1, z2))

• If M(z) = −M(−z) then PR is preserved

P (2D)(z) + P (2D)(−z) = 2

• Suppose H(1D)(z) = (1 + z)kR1(z) and M(z) + 1 = (1− z1)lm(z) then

H(2D)(M(z)) = (1− z1)klR2(z)


How To Design the Mapping

The mapping has to satisfy

M(z) = −M(−z), and

M(z) = (1− z1)lm(z)− 1

Thus,(1− z1)

lm(z) + (1 + z1)lm(−z) = 2

The last proposition tell us exactly how to solve this!


Design Examples


Directional Vanishing Moments Generated After

Iteration

Different expanding rules lead to different set of directions with DVMs.


Gain by using Filters with DVMs

0.01 0.02 0.03 0.04 0.05 0.06 0.07 0.08 0.09 0.1

10

15

20

25

% coeff

SN

R [d

B]

dvmpkva

0 0.02 0.04 0.06 0.08 0.1 0.124

6

8

10

12

14

16

% coeff

SN

R [d

B]

dvmpkva


Summary

• Image processing relies on prior information about images.

– Geometrical structure is the key!

• Strong motivation for more powerful image representations:scale, space, and direction.

– New desideratum beyond wavelets: localized direction

• New two-dimensional discrete framework and algorithms:

– Flexible directional and multiresolution image representation.

– Effective for images with smooth contours ⇒ contourlets.

• Dream: Another fruitful interaction between harmonic analysis, computervision, and signal processing.

61

References

• M. N. Do,“Directional multiresolution image representations,” Ph.D.thesis, EPFL, December 2001.

• M. N. Do and M. Vetterli, “Contourlets,” Beyond Wavelets, G. V.Welland ed., Academic Press, 2003.

• M. N. Do and M. Vetterli, “The contourlet transform: an efficientdirectional multiresolution image representation,” IEEE Trans. on

Image Proc., submitted 2003.

• A. Cunha and M. N. Do, “Biorthogonal two-channel filter bankswith directional vanishing moments: characterization, design andapplications,” IEEE Trans. on Image Proc., submitted 2004.

• Software and downloadable papers: www.ifp.uiuc.edu/∼minhdo

62

contourlets: construction and...

Documents