spectral focal sweep (sfs): extended depth of field from chromatic aberrations iccp 2010 ollie...

Spectral Focal Sweep (SFS):

Extended Depth of Field from Chromatic Aberrations

ICCP 2010

Ollie Cossairt, Shree NayarColumbia University

Conventional Camera (F/4)

Spectral Focal Sweep: ICCP 2010

BLURRY

Conventional Camera (F/16)

Spectral Focal Sweep: ICCP 2010

NOISY

EDOF Camera (F/4)

Spectral Focal Sweep: ICCP 2010

CAPTURED

EDOF Camera (F/4)

Spectral Focal Sweep: ICCP 2010

DEBLURRED

Focus Sweep[Nagahara et al. ’08][Hausler ’72] Lens Focal PlaneSensor

Coded Aperture[Welford ’60] [Levin et al. ‘07][Veeraraghavan et al. ’07] [Zhou and Nayar ’09]

Related Work

Spectral Focal Sweep: ICCP 2010

Phase Plates[Dowski and Cathey ’95] [Chi and George ’01] [Levin et al. ’09]

Color[Guichard et al. ’09]

LensFocal Planes

RGB Sensor RedGreenBlue

Spectral Focal Sweep Idea

Spectral Focal Sweep: ICCP 2010

No moving parts or custom elements

PoorlyCorrected

Lens

+

B&WSensor

+

Focus Sweep

Lens

Focal Plane

Sensor

=

Broadband reflectance

spectra

Broadband reflectance spectra• Works exactly for black and white scenes• Many naturally occurring reflectance spectra

are also broadband [Parkkinen ‘89]

Poorly corrected lens• Simpler lens design optimization

Black & White Sensor• Can also approximate with luminance channel

from color sensor

Spectral Focal Sweep Idea

Spectral Focal Sweep: ICCP 2010

Conventional Focus Sweep PSF

Spectral Focal Sweep: ICCP 2010

v1u

u = object depth (fixed)v = sensor distance (varying)

LensEFL = f

FocalPlane

InstantaneousPSF Sensor

Scene Depth Range

Conventional Focus Sweep PSF

Spectral Focal Sweep: ICCP 2010

u = object depth (fixed)v = sensor distance (varying)

LensEFL = f

FocalPlane

InstantaneousPSF Sensor

v2u

Scene Depth Range

d2

Conventional Focus Sweep PSF

Spectral Focal Sweep: ICCP 2010

v3u

d3

u = object depth (fixed)v = sensor distance (varying)

LensEFL = f

FocalPlane

InstantaneousPSF Sensor

Scene Depth Range

Conventional Focus Sweep PSF

Spectral Focal Sweep: ICCP 2010

Varying either object/sensor distance, or focal length will produce the same

PSF

Integrate over time to get final PSF

Circ/Top-hat function

Defocus disk diameter (A = aperture diameter)

Refractive Dispersion (prism)

Spectral Focal Sweep: ICCP 2010

Snell’s Law of refraction

Refraction depends

on wavelengt

h

Refractive Dispersion (lens)

Spectral Focal Sweep: ICCP 2010

uv

SensorPSFFocalPlanes

Scene Depth Range

For an uncorrected lens, Focal length depends on wavelength

R1 , R2 = lens radii of curvature

Spectral Focal Sweep PSF

Spectral Focal Sweep: ICCP 2010

Each wavelength produces a different defocus disk

A B&W Sensor integrates over wavelength, weighted by the spectral reflectance of the scene

For white objects, a SFS PSF is identical to a mechanical focal sweep PSF

SFS Lens Design

Spectral Focal Sweep: ICCP 2010

Maximum defocus diameter 100um at 400nm and 700nm

Design Criteria• Wavelength range 400-

700nm• 75mm EFL F/4• 1/3” Sensor, 10um pixel size

Optimization Criteria• Zemax Optical Design

Software• Maximize chromatic defocus

across wavelength range• Minimize spot size at 550nm• Fit to off-the-shelf lenses

Zemax SFS PSFs(white spectrum)

dep

th

field

SFS Lens Corrected lens

Spectral Focal Sweep: ICCP 2010

15x

Zemax SFS PSFs(white spectrum)

dep

th

Corrected lens

Spectral Focal Sweep: ICCP 2010

field

SFS Lens

1

15x

• Munsell database of 1250 measured reflectance spectra

• Zemax simulation of SFS PSF at 12 depth locations (1.5m-3m, 100um max defocus dia.)

• For each Munsell spectrum– Calculate average deblurring error when a

white spectrum is used for deblurring

Simulating Real Spectra

Spectral Focal Sweep: ICCP 2010

Average Deblurring Error

Spectral Focal Sweep: ICCP 2010

Input Image:

Munsell color PSFs:

White PSF at center depth:

PSF Distance Metric:

Average Deblurring Error:

[Zhou and Nayar ‘09]

Averaged over many images

Munsell Spectra PSFs

Spectral Focal Sweep: ICCP 2010

Corrected lens

dep

th

Most Munsell spectra show little PSF variation

1

15x

Average Deblurring Error(%95 of Munsell spectra)

Spectral Focal Sweep: ICCP 2010

Most Munsell PSFs introduce negligible deblurring artifacts

Doublet SFS Implementation

Spectral Focal Sweep: ICCP 2010

Edmund Optics #481841” Dia. 100mm FL Plano-convex asphere

Thorlabs #LE19291” Dia. 300mm FL positive meniscus

Zemax Design Implementation

Experimental Setup

Spectral Focal Sweep: ICCP 2010

B&W Examples(Basler 1/3” sensor)

Color Examples(Canon 450D sensor)

Corrected Lens SFS Doublet Lens

F/1.4

75

mm

EFL C

osm

icar

F/2.8

10

0m

m

EFL C

annon

F/4 S

FS 7

5m

mEFL D

ouble

tF/4

SFS

75

mm

EFL D

ouble

t

Measured SFS PSFs(white spectrum)

Spectral Focal Sweep: ICCP 2010

Measured SFS PSF approximately depth-invariant

SFS

lens

Cosm

ica

r

depth

Deblurring

Deblur with measured white PSF

B&W Images• Wiener deconvolution

Color Images• Transform from RGB to YUV space• Wiener deconvolution of Y image• Transform back to RGB space

Spectral Focal Sweep: ICCP 2010

EDOF Examples

Spectral Focal Sweep: ICCP 2010

F/4 Corrected Lens

Spectral Focal Sweep: ICCP 2010

8ms exposure

F/4 SFS Camera

Spectral Focal Sweep: ICCP 2010

8ms exposure

(Captured)

F/4 SFS Camera

Spectral Focal Sweep: ICCP 2010

8ms exposure

(Deblurred)

F/4 Corrected Lens

Spectral Focal Sweep: ICCP 2010

8ms exposure

Background Foreground

F/4 SFS Lens

Spectral Focal Sweep: ICCP 2010

8ms exposure

(Captured)

F/4 SFS Lens

Spectral Focal Sweep: ICCP 2010

(Deblurred)

8ms exposure

F/4 Corrected Lens

Spectral Focal Sweep: ICCP 2010

16ms exposure

Background

Foreground

F/4 SFS Lens

Spectral Focal Sweep: ICCP 2010

16ms exposure

(Captured)

F/4 SFS Lens

Spectral Focal Sweep: ICCP 2010

16ms exposure

(Deblurred)

F/4 Corrected Lens

Spectral Focal Sweep: ICCP 2010

16ms exposure

BackgroundForeground

F/4 SFS Lens

Spectral Focal Sweep: ICCP 2010

16ms exposure

(Captured)

F/4 SFS Lens

Spectral Focal Sweep: ICCP 2010

16ms exposure

(Deblurred)

F/4 Corrected Lens

Spectral Focal Sweep: ICCP 2010

16ms exposure

Background

Foreground

F/4 SFS Lens

Spectral Focal Sweep: ICCP 2010

16ms exposure

(Captured)

F/4 SFS Lens

Spectral Focal Sweep: ICCP 2010

16ms exposure

(Deblurred)

F/4 Corrected Lens

Spectral Focal Sweep: ICCP 2010

16ms exposure

Background

Foreground

F/4 SFS Lens

Spectral Focal Sweep: ICCP 2010

16ms exposure

(Captured)

F/4 SFS Lens

Spectral Focal Sweep: ICCP 2010

16ms exposure

(Deblurred)

Limitations

Spectral Focal Sweep: ICCP 2010

• Broadband assumption does not always hold (e.g. %5 of Munsell Spectra)– deblurring artifacts introduced

• Color deblurring method does not correct blur in chrominance channels

• Works exactly for B&W scenes• Works well for many naturally occurring

spectra• Results for color images “look” good• Easy to build EDOF camera

Conclusion

Spectral Focal Sweep: ICCP 2010

Broadbandspectra

Chromaticaberrations

B&Wsensor

Focussweep+ + ==