image destabilization: programmable defocus using lens and sensor motion
DESCRIPTION
Image Destabilization: Programmable Defocus using Lens and Sensor Motion. Ankit Mohan, Douglas Lanman, Shinsaku Hiura, Ramesh Raskar MIT Media Lab. MIT Media Lab Camera Culture. Defocus Blur. Lots of glass; Heavy; Bulky; Expensive. Image Destabilization. Camera. Lens. Sensor. - PowerPoint PPT PresentationTRANSCRIPT
![Page 1: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/1.jpg)
MIT Media Lab Camera Culture
Image Destabilization:Programmable Defocus using
Lens and Sensor Motion
Ankit Mohan, Douglas Lanman,Shinsaku Hiura, Ramesh Raskar
MIT Media Lab
MIT Media Lab Camera Culture
![Page 2: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/2.jpg)
MIT Media Lab Camera Culture
Lots of glass; Heavy; Bulky; Expensive
Defocus Blur
![Page 3: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/3.jpg)
MIT Media Lab Camera Culture
Image Destabilization
Lens Sensor
Camera
Static Scene
![Page 4: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/4.jpg)
MIT Media Lab Camera Culture
Image Destabilization
Static Scene
Lens Motion Sensor Motion
Camera
![Page 5: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/5.jpg)
MIT Media Lab Camera Culture
Related Work
[Bae and Durand 2007]
[Hasinoff and Kutulakos 2007]
f8
f/4
f/2
f/1
extrapolated aperture
[Hiura et al. 2009][Vaish et al. 2004]
![Page 6: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/6.jpg)
MIT Media Lab Camera Culture
LaminographyX-Ray Source
X-Ray Sensor
Plane of focus
Related technique: Time Delay and Integration (TDI)
Motion direction
Motion direction
![Page 7: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/7.jpg)
MIT Media Lab Camera Culture
A
B A’
B’
Lens based Focusing
Lens Sensor
![Page 8: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/8.jpg)
MIT Media Lab Camera Culture
A
B A’
B’
Lens based Focusing
Lens Sensor
![Page 9: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/9.jpg)
MIT Media Lab Camera Culture
A
B
B’
A’
Lens
Smaller aperture Smaller defocus blur
Sensor
![Page 10: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/10.jpg)
MIT Media Lab Camera Culture
A
B A’
B’
Pinhole
Pinhole: All In-Focus
Sensor
![Page 11: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/11.jpg)
MIT Media Lab Camera Culture
A
B
A’
B’
Pinhole
Shifting Pinhole
Sensor
vp
![Page 12: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/12.jpg)
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
Shifting Pinhole
Sensor
vp
![Page 13: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/13.jpg)
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
Shifting Pinhole
Sensor
vp
![Page 14: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/14.jpg)
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
Shifting Pinhole
Sensor
vp
![Page 15: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/15.jpg)
MIT Media Lab Camera Culture
B
Pinhole Sensor
A’
B’
Shifting Pinhole
A
da
db ds
tp
vp
![Page 16: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/16.jpg)
MIT Media Lab Camera Culture
A
B
A’
B’
Pinhole
Shifting Pinhole and Sensor
vp
Sensor
vs
da
db dsFocus Here
![Page 17: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/17.jpg)
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’vp
Sensor
vs
da
db dsFocus Here
Shifting Pinhole and Sensor
![Page 18: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/18.jpg)
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
vp
Sensor
vs
da
db dsFocus Here
Shifting Pinhole and Sensor
![Page 19: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/19.jpg)
MIT Media Lab Camera Culture
A
B
Pinhole
A’
B’
vp
Sensor
vs
da
db dsFocus Here
Shifting Pinhole and Sensor
![Page 20: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/20.jpg)
MIT Media Lab Camera Culture
A Lens in Time!
Analogous to shift and sum basedLight field re-focusing.
Lens Equation:
Virtual Focal Length:
Virtual F-Number:
![Page 21: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/21.jpg)
MIT Media Lab Camera Culture
Our Prototype
![Page 22: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/22.jpg)
MIT Media Lab Camera Culture
Adjusting the Focus Plane
all-in-focus pinhole image
![Page 23: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/23.jpg)
MIT Media Lab Camera Culture
Adjusting the Focus Plane
focused in the front using destabilization (10 second exposure)
![Page 24: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/24.jpg)
MIT Media Lab Camera Culture
Adjusting the Focus Plane
focused in the middle using destabilization (5 second exposure)
![Page 25: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/25.jpg)
MIT Media Lab Camera Culture
Adjusting the Focus Plane
focused in the back using destabilization (10 second exposure)
![Page 26: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/26.jpg)
MIT Media Lab Camera Culture
Adjusting the Virtual Aperture
focused in the middle using destabilization (5mm pinhole translation)
![Page 27: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/27.jpg)
MIT Media Lab Camera Culture
Adjusting the Virtual Aperture
focused in the middle using destabilization (30mm pinhole translation)
![Page 28: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/28.jpg)
MIT Media Lab Camera Culture
Shifting Lens and Sensor Defocus• Physical vs. synthetic focus• Similar physical and synthetic focus
aperture
sensor
real focus
virtual focus
Defocus Exaggeration
![Page 29: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/29.jpg)
MIT Media Lab Camera Culture
Defocus Exaggeration
static lens with an f/2.8 aperture
![Page 30: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/30.jpg)
MIT Media Lab Camera Culture
Defocus Exaggeration
destabilization simulates a reduced f-number
![Page 31: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/31.jpg)
MIT Media Lab Camera Culture
Defocus Invariance
aperture
sensor
real focus
virtual focus
• Differing physical and synthetic focus
![Page 32: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/32.jpg)
MIT Media Lab Camera Culture
* =real PSF virtual PSF overall PSF
Defocus Invariance• Related work
– [Nagahara et al. 2008]– [Cathey and Dowski 1995]– [Levin et al. 2008]
• PSF not depth invariant– only size is depth invariant
• Gaussian special case– depth invariant PSF – inversion is ill-conditioned* =
real PSF virtual PSF overall PSF
[Nagahara et al. 2008]
![Page 33: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/33.jpg)
MIT Media Lab Camera Culture
Defocus Invariance
depth-invariant blur size (horizontal slit + destabilization)
![Page 34: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/34.jpg)
MIT Media Lab Camera Culture
Defocus Invariance
Richardson-Lucy deconvolution result
![Page 35: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/35.jpg)
MIT Media Lab Camera Culture
Tilted Sensor
C
DC′
D′
focus plane aperture plane sensor plane
Scheimpflug intersection
dC
dD
d′Cd′D
CC ddf 111
![Page 36: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/36.jpg)
MIT Media Lab Camera Culture
Tilted Sensor
C
D
vs?
focus plane aperture plane sensor plane
vp
CC ddf 111
dC
dD
![Page 37: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/37.jpg)
MIT Media Lab Camera Culture
Tilted Sensor
C
D
vp
C′ D′
vs
focus plane aperture plane sensor plane
dC
dD
d′Cd′D
s
p
DD
D
CC
C
vv
ddd
ddd
![Page 38: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/38.jpg)
MIT Media Lab Camera Culture
Tuning the PSF
aperture
sensor
real focus
pinhole image (static f/22 aperture)
![Page 39: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/39.jpg)
MIT Media Lab Camera Culture
Tuning the PSF
real focus
aperture
sensor
large aperture image (static f/2.8 aperture)
![Page 40: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/40.jpg)
MIT Media Lab Camera Culture
Tuning the PSF
virtual focus
aperture
sensor
destabilized image using a pinhole (translated f/22 aperture)
![Page 41: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/41.jpg)
MIT Media Lab Camera Culture
Tuning the PSF
aperture
sensor
real focusvirtual focus
destabilized image using a large aperture (translated f/2.8 aperture)
![Page 42: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/42.jpg)
MIT Media Lab Camera Culture
Tuning the PSFreal focus
virtual focus
aperture
sensor
simulated aspheric lens using a vertical slit aperture and destabilization
![Page 43: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/43.jpg)
MIT Media Lab Camera Culture
Extension to 2D Displacements
ellipticallinear circular
“figure 8” hypocycloidal trispiral
![Page 44: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/44.jpg)
MIT Media Lab Camera Culture
Large apertures with tiny lenses?Benefits• No time or light inefficiency
wrt cheap cameras• Exploits unused area around
the lens• Compact design• With near-pinhole apertures
(mobile phones) many possibilities
Limitations• Coordinated mechanical
movement required• Diffraction (due to small aperture)
cannot be eliminated[Zhang and Levoy, tomorrow]
[Our group: augmented LF for wave analysis]
• Scene motion during exposure
![Page 45: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/45.jpg)
MIT Media Lab Camera Culture
Acknowledgements
Grace WooMIT CSAIL
Quinn SmithwickMIT Media Lab
Gabriel TaubinBrown University
Jaewon KimMIT Media Lab
MIT Media Lab: Camera Culture
![Page 46: Image Destabilization: Programmable Defocus using Lens and Sensor Motion](https://reader035.vdocuments.us/reader035/viewer/2022062323/56815d93550346895dcbaf9b/html5/thumbnails/46.jpg)
MIT Media Lab Camera Culture
SLRs with tiny lenses?
• Analysis of space of relative lens/sensor displacement• Destabilization as virtual focusing mechanism• Shallower depth of field than physical aperture• Depth-independent defocus blur size