Lensless Imaging

Richard Baraniuk

Rice University

Ashok VeeraraghavanRice

Aswin SankaranarayananCMU

John RogersUIUC

Re-Imagining Imaging

• Conventional camera design– based on human visual system model– objective lens: directs a cone of light into the camera– 1-to-1 correspondence between scene points and camera

pixels makes imaging easy

Re-Imagining Imaging

• Conventional camera design– based on human visual system model– objective lens: directs a cone of light into the camera– 1-to-1 correspondence between scene points and camera

pixels makes imaging easy

• Goal: A large, potentially flexible, imaging platform capable of distributed acquisition of light fields– inspired by distributed light sensing in cephalopod skin

• Approach: Lensless imaging– leverage recent progress in coded aperture and

compressive sensing– exciting opportunities for flat and flexible cameras

• With incoherent light, no phase information– all photo-detectors measure roughly the same information,

the average light level of the scene

sensor scene

photo-detector 1

photo-detector 2

Problem

Solution

• Add a mask in front of the sensor/photo-detector– attenuates certain rays of light

• How does this help?– same scene point is attenuated differently at different photo-

detectors– each photo-detector sees a different linear combination of the

scene points– can design mask(s) such that we can recover a high-resolution

version of the scene (compressive sensing)

mask

sensor scene

photo-detector 1

photo-detector 2

Mask Design

• Random mask– rich theory and algorithms available

from compressive sensing– provable recovery bounds

measurements sparsesignal

nonzeroentries

super sub-Nyquist measurement

• Random mask– rich theory and algorithms available

from compressive sensing– provable recovery bounds– major impact in a variety of DOD

and industrial sensing systems:medical, radar, sonar, hyperspectral, IR, THz imaging, …

Mask Design

mask(s)sensor scene X

10 u

nit

s

10 units

1/10 unit

Simulations

= f(

sensor measurements

mask

)X

noiseless systemPSNR = 19dB

noisy systemPSNR = 15dB

image

Planar Prototype

walls to limit FOV

mask

gap (0.5mm)

mount

sensor

• Sensor: Flea3 Point Grey camera, 1024x1280 pixels (5.3µm each)

• Mask: Random binary mask (1)135x135 features (85µm each)10% of mask “pixels” transparent

• Target projected on screen 15cm from camera (target height/width 24cm)

sensor-mask assembly

target on monitor

sensor image

Planar Prototype Results 1

targeton screen

recoveredimage• Sensor: Flea3 Point Grey camera,

1024x1280 pixels (5.3µm each)

• Mask: Random binary mask (1)135x135 features (85µm each)

10% of mask “pixels” transparent

• Target projected on screen 15cm from camera (target height/width 24cm)

• Reconstruction: Least-squares

• Ongoing: High-resolution reconstruction leveraging sparsity and priors on scene structure

Planar Prototype Results 2

targeton screen

recoveredimage• Sensor: Flea3 Point Grey camera,

1024x1280 pixels (5.3µm each)

• Mask: Random binary mask (1)270x270 features (42µm each)

10% of mask “pixels” transparent

• Target projected on screen 15cm from camera (target height/width 24cm)

• Reconstruction: Least-squares

• Ongoing: High-resolution reconstruction leveraging sparsity and priors on scene structure

Color Images

From Challenge …

• With a planar sensor, must limit the field of view

• Outside field of view, image recovery becomes increasingly ill-posed

mask(s)sensor

scene X

… To Opportunity

• Without the need for a lens, we can make the mask and sensor curved

• Example: (Hemi)spherical camera– 180/360 degree field of view– no field-of-view ill-posedness!

with John Rogers

sensorarray

mask(s)

… To Opportunity

• Without the need for a lens, we can make the mask and sensor curved

• Example: (Hemi)spherical camera– 180/360 degree field of view– no field-of-view ill-posedness!

spherical sensor array

scene projected on sphere

un-warped recovered image

Simulations

noiseless scene 𝜎=0.1 𝜎=0.5 𝜎=1

512

256

256

256

128

128

128

Number of pixels in reconstruction

additive noise

Spherical Prototype• Sensor: White paint on a spherical shell

acts as diffuser

• Mask: random binary mask 68x68 features (170µm each)10% of mask “pixels” transparent

• Target projected on screen 18cm away (target height/width 24cm)

plastic shell with diffuser inner surface

(proxy for a spherical sensor)

planar mask in front of shell(flexible PDMS masks on shell)

current prototype uses a Grasshopper point grey camera to capture image formed on a 1.2 cm2

area (400x400 pixels)

targetson screen

recoveredimages

32x32 recoveredimages

Spherical Prototype Results• Sensor: White paint on a spherical

shell acts as diffuser

• Mask: random binary mask 68x68 features (170µm each)10% of mask “pixels” transparent

• Target projected on screen 18cm away (target height/width 24cm)

• Reconstruction: Least-squares

• Ongoing: High-resolution reconstruction leveraging sparsity and priors on scene structure

Spherical photo-detector to improve SNR

Planned Research

• Radically new kinds of cameras – flat, flexible, (hemi)spherical cameras– beyond visible (IR, THz, …)– numerous potential DOD and industrial applications

• New theory and algorithms– mask design (light throughput versus invertibility)– dynamic masks– new recovery algorithms needed for 0/1 masks

• Can perform exploitation directly on compressive measurements (detection/classification, etc.) without numerical scene reconstruction

• Sensing light fields instead of images