study coded aperture

Image and Deoth from a Conventional Camera

with a Coded Apertrue

Anat Levin, Rob Fergus, Frédo Durand, William

Freeman

MIT CSAIL

study

Single input image

real objects

output #1: Depth mapoutput #2: all-infocused image

Coded Aperture

Conventional aperture and depth of field

Small aperture

Big aperture

Focal plane

Object

LensCamera sensor

Point spread

function

http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.pptFocal plane

Depth from defocus

LensObject

Camera sensor

Point spread

function

http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.pptFocal plane

Depth from defocus

Focal plane

LensCamera sensor

Point spread

function

Object

http://groups.csail.mit.edu/graphics/CodedAperture/CodedAperture-LevinEtAl-SIGGRAPH07.ppt

Depth from defocus

Defocus as local convolution

Input defocused image

xfy k Local

observed sub-

window

Calibrated blur

kernels at depth K

Sharp sub-

window

Depth k=1

Depth k=2

Depth k=3

Introduction Estimation of depth – a branch of Computational Photography Most challenges of y = fk * x

Input Ringing with the traditional Richardson-Lucy algorithm

• Hard to de-convolve even when kernel is known

• Hard to identify correct scale:

?? Correct

scale

Smaller scale

? Larger scale

Related work – depth estimation Active methods – additional illumination sources• Structured light methods

Nayar et al. ICCV 95Zhang and Nayar, SIGGRAPH 06

Projection Defocus Analysis for Capture and Image Display, Zhang and Nayar, 06

Related work – depth estimation

Passive methods – changes of focus • Depth from defocus (DFD)

Pentland, IEEE 87Chaudhuri, Favaro et al. , 99

• Blind Deconvolution – image prior, maximum likelyhood

Kundur and Hatzinakos , IEEE 96Levin, NIPS 06

• Coded apertures for light gatheringFenimore and Cannon, Optics 78

Related work – depth estimation

• Plenoptic /light field cameraAdelson and Wang, IEEE 92Ng et al., 05

• Wavefront codingCathey & Dowski, Optics 94, 951.Rays don't converge anymore2.Image blur is the same for all

depth3.Blur spectrum does not have

too many zeros

Passive methods – changes of viewpoints

CompPhoto06/html/lecturenotes/25_LightField_6.pdf

OverviewTry deconvolving local input windows with different

scaled filters:

?? Correct

scale

Smaller scale

? Larger scale

Somehow: select best scale

yxfk

Challenges & contributions

Hard to de-convolve even when kernel is knownIDEA 1: Natural images prior

Hard to identify correct scaleIDEA 2: Coded Aperture

? =

=?

Solution 1:

Solution 2:

Deconvolution is ill posed

IDEA 1: Natural images prior

Image

gradient

put a penalty on gradients

Natural images have sparse gradients

Natural

Unnatural

What makes images special?

Deconvolution with prior

i ik xyxf )(min arg x

2 Convolution

errorDerivatives

prior

High

Low ?

?

2

2Equal convolution error

Comparing deconvolution algorithms

Input

Richardson-Lucy

Gaussian prior

“spread” gradients

Sparse prior

“localizes” gradients

8.0)( xx 2

)( xx

Statistical Model of Images

“Deconvolution using natural image priors”, Levin et. al., ETAI 07

Spatial domain

Frequency domain

Maximum a-posteriori P(x|y)

Image prior (gradient here)

Gradient operator

For Gaussian priors

For sparse priors

likelyhood

Minimize deconvolution error

Deconvolution using a Gaussian prior

Note: solved in the frequency domain in a few seconds for MB size file

Using an iterative reweighted least squares process (IRLS) [Meer 2004; Levin and Weiss to appear]

Cannot solve in frequency domain

Deconvolution using a sparse prior

Note: solved in the frequency domain around 1 hour on 2.4Ghz CPR for 2MB file

Iterative reweighted least squares process (IRLS)

Recall: OverviewTry deconvolving local input windows with different

scaled filters:

?? Correct

scale

Smaller scale

? Larger scale

Somehow: select best scale

yxfk

Challenge: smaller scale not so different than correct

IDEA 2: Coded Aperture

Mask (code) in aperture planeMake defocus patterns different from natural

images and easier to discriminate

Conventional aperture

Our coded aperture

Lens with coded aperture

Lens with coded

aperture

Camera sensor

Point spread

function

Image of a defocused point light

source

Aperture pattern

Object

Focal plane

Why coded ?

Convention

al Coded

Coded aperture- reduce uncertainty in scale identification

Correct

scale

Smaller scale

Larger scale

Why coded ?

Convention

al Coded

Coded aperture- reduce uncertainty in scale identification

Correct

scale

Smaller scale

Larger scale

Fourier transforms of 1D slide through the blur pattern

Coded aperture: Scale estimation and division by zero

?

?

=

=

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

Estimated image

spatial ringing

Division by zero

Estimated image

Filter, wrong scale

Filter, correct scale

Observed image

Division by zero with a conventional aperture ?

?

?

=

=

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

0 Frequency

sp

ectr

um

Estimated image

no spatial ringing

Tiny value at ω

Estimated image

Filter, wrong scale

Filter, correct scale

Observed image

No zero at ω !

No zero at ω !

ω is zero !

Filter Selection Criterion

The filter f has good depth discrimination - blurry image distributions Pk1(y) and Pk2(y) at depths k1 and k2 should not be similar

KL-divergence scores y

kkkkk dyyPyPyPyPyP 21121KL loglog,D

Filter Design Practical constrains

1. Binary filter to construct accurately2. Cut the filter from a single piece3. Avoid excessive radial distortion4. Avoid using the full aperture5. Diffraction impose a min size on the holes

in the file Spec.

13x13 patterns with 1 mm holesEach pattern, 8 different scales Varying between 5~15 pixels in width

Filter Design

Conventional

Conventional

Blur scale identification

Not robust at high-frequency noise

Un-normalized energy term

λk learn to minimize the scale misclassification error on a set of traning images

Ek is approximate by the reconstruction error by ML solution

x* is the deblurred image

Regularizing depth estimation

Results

Applications Digital refocusing from a single image

e.g. Synthesis an all-focus imagee.g. Post-exposure

Conclusion Pros.

All-infocus image and depth at a single shotNo loss of image resolution (compared with Plenoptic camera) Simple modificationCoded apertureConventional aperture

Cons.50 % light is blockedDepth is coarseMay need manual

correction