depth from diffusion

Depth from Diffusion

Supported by ONR

Changyin Zhou Shree NayarOliver Cossairt

Columbia University

Optical Diffuser

Optical Diffuser

Micrograph of a Holographic Diffuser(RPC Photonics)

~ 10 micron

[Gray, 1978] [Chang et al., 2006] [Garcia-Guerrero et al. 2007]

Diffusers as Accessories

Diffuser to preview the image

(B&H)Diffusers to soften the image

Diffusers for illumination(B&H)

CameraDiffuser

Object

CameraDiffuser

Object

The amount of diffusion varies with depth.

Diffusion Encodes Depth

Geometry of Diffusion: A Pinhole Camera

Sensor

Pinhole

Object

P

Q

Miss

Object

P

Geometry of Diffusion: A Pinhole Camera

Sensor

θ

Pinhole

Diffuser

Geometry of Diffusion: A Pinhole Camera

Pinhole

θ

θ

Sensor

Diffuser

Object

PA

B

Geometry of Diffusion: A Pinhole Camera

O

V U Z

θ

θPinhole

Diffusion Law:

Sensor

A

B

Object

2r

P

ObjectDiffuser

Geometry of Diffusion: A Pinhole Camera

O

V U Z

Pinhole

Sensor

A

B

Object

2r

P

Diffusion Size and Depth:

ObjectDiffuser

θ

θ

Geometry of Diffusion: A Pinhole Camera

O

V U

Pinhole

SensorObject

2r

Diffuser as a proxy object

Diffusion Size and Depth:

PZ

Diffuser

Diffusion as Convolution: A Pinhole Camera

Latent clear image

Captured Image

Diffusion Size

Diffusion PSF

Assume field angle and depth are constant for small Assume field angle and depth are constant for small

image patches, we have:image patches, we have:

Geometry of Diffusion: A Lens Camera

Geometry of Diffusion: A Lens Camera

O

V U

Pinhole

SensorObject

2r

Diffuser as a proxy object

PZ

Diffuser

Geometry of Diffusion: A Lens Camera

V USensorObject

2r

Diffuser as a proxy object

PZ

Diffuser

Lens

The captured image can be

further blurred due to defocus.

Diffusion as Convolution: A Lens Camera

The Final PSF

Defocus PSF

Diffusion PSF

For a lens camera with a diffuser, we have:

is the diffusion PSF if a pinhole were used.

is the defocus PSF if the diffuser were removed.

Depth from Diffusion (DFDiff) Algorithm

Same form as in DFD

2. Estimate

Blur Size r

3. Compute

Depth Z

1. Capture

Two Images

With a diffuser Without a diffuser

Depth from Diffusion vs. Depth from Defocus

[Pentland, 1987] [Subbarao, 1988] [Watanabe & Nayar, 1996]

[Chaudhuri & Rajagopalan, 1999] [Favaro & Soatto, 2005] [Schechner & Kiryati, 2000]

Depth from Defocus

Aperture pattern

P

Lens Focal PlaneSensor

Zr

Depth from Diffusion

Diffusion pattern

Sensor

Pinhole

θ

θ

DiffuserZ

P

r

Depth from Diffusion vs. Depth from Defocus

Depth from Diffusion

P

Any lens is fine!

Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL

Object distance = 1000 mm

ObjectA Diffuser of 21.8

o

Depth precision is about 0.1 mm.

Field of View

Depth from Diffusion vs. Depth from Defocus

Depth from Defocus

P

Lens

Aperture diameter

?

Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL

Depth precision is about 0.1 mm.

Object distance = 1000 mm

Object

Field of View

Depth from Diffusion vs. Depth from Defocus

Depth from Defocus

Object distance = 1000 mm

Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL

Depth precision is about 0.1 mm.

Aperture diameter

800 mm

Object

P

Depth from Diffusion vs. Depth from Defocus

Depth from DiffusionSuppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL

P

Depth precision is about 1.0 mm.

Object distance = 5000 mm

ObjectAny lens is fine!A Diffuser of 11.2

o

Depth from Diffusion vs. Depth from Defocus

Depth from Defocus

P

Lens

Aperture diameter

?

Suppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL

Depth precision is about 1.0 mm.

Object distance = 5000 mm

Object

Depth from Diffusion vs. Depth from Defocus

Depth from DefocusSuppose 22.5x15mm Sensor, 10 um pixel, 100 mm EFL

Depth precision is about 1.0 mm.

Aperture diameter

2000 mmP

Object distance = 5000 mm

Object

PSF Measurement: A Pinhole Camera

F/22 , Field Angle = 0 o

Z = 2 mm Z = 5 mm

Captured Modeled

- Canon EOS T1i; EF 50mm F/1.8 Lens;

- Luminit Holographic Diffuser (10o Gaussian);

- Diffuser distance: U = 1m

Z = 2 mm Z = 5 mm

PSF Measurement: A Pinhole Camera

Captured Modeled

F/22 , Field Angle = 10 o

- Canon EOS T1i; EF 50mm F/1.8 Lens;

- Luminit Holographic Diffuser (10o Gaussian);

- Diffuser distance: U = 1m

PSF Measurement: A Lens Camera

F/1.8 , Field Angle = 10 o

Captured Modeled

- Canon EOS T1i; EF 50mm F/1.8 Lens;

- Luminit Holographic Diffuser (10o Gaussian);

- Diffuser distance: U = 1m

Experiments

Five playing cards,

0.29mm thick each

Canon 20D + 50mm Lens

Luminit Diffuser (20o)

Experiments

Captured WITHOUT a Diffuser Captured WITH a Diffuser

Experiments

Computed Depth Map

(~ 0.1 mm precision)

(mm)Five playing cards,

0.29mm thick each

Experiments

A small sculpture of

about 4mm thickness

Canon G5 Compact Camera

Luminit Diffuser (5o)

Experiments

Captured WITHOUT a Diffuser Captured WITH a Diffuser

Experiments

Computed Depth Map

A 3D View of Depth Map

A small sculpture of

about 4mm thickness

Experiments

Canon 20D; Gaussian Diffuser (10o)

450

mm

650 mm

Experiments

Stitched Depth Map

(precision)

(mm)

Summary

Formulated the image formation Formulated the image formation with optical diffuserswith optical diffusers

Proposed Depth from DiffusionProposed Depth from Diffusion- Require a diffuser on the object side - Require a diffuser on the object side

+ High-precision depth estimation+ High-precision depth estimation

+ Distant objects+ Distant objects

+ Less sensitive to lens aberrations+ Less sensitive to lens aberrations

Demonstrated high-precision Demonstrated high-precision depth estimationdepth estimation

CameraDiffuser

Object

Depth from Diffusion

Supported by ONR

Changyin Zhou Shree NayarOliver Cossairt

Columbia University