mit camera culture group update july 2009
TRANSCRIPT
![Page 1: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/1.jpg)
Raskar, Camera Culture, MIT Media Lab
Camera Culture
Ramesh Raskar
Camera CultureAssociate Professor, MIT Media Lab http://raskar.info
![Page 2: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/2.jpg)
How can we create an entirely new class of imaging platforms
that have an understanding of the world that far exceeds human ability
and produce meaningful abstractions that are well within human comprehensibility ?
Ramesh Raskar http://raskar.info
![Page 3: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/3.jpg)
Can you look around a corner ?
![Page 4: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/4.jpg)
Can you decode a 5 micron feature Can you decode a 5 micron feature from 3 meters away from 3 meters away
with an ordinary camera ?with an ordinary camera ?
![Page 5: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/5.jpg)
Beyond Multi-touch: Mobile 3D Interfaces?
![Page 6: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/6.jpg)
6D DisplayLight sensitive 4D display
One Pixel of a 6D Display = 4D Display Raskar, Saakes, Fuchs, Siedel, Lensch, 2008
![Page 7: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/7.jpg)
![Page 8: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/8.jpg)
A. Introduction 5 minutes ‐‐
B. Cameras of the future (Raskar, 30 minutes) * Form factors, Modalities and Interaction * Enabling Visual Social Computing
C. Reconstruction the World (Seitz, 30 minutes) * Photo tourism and beyond * Image based modeling and rendering on a massive scale ‐* Scene summarization
D. Understanding a Billion Photos (Efros, 30 minutes) * What will the photos depict? * Photos as visual content for computer graphics * Solving computer vision
E. Discussion 10 minutes ‐‐
Course: Next Billion CamerasWedn at 3:30pm
![Page 9: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/9.jpg)
Looking around a corner via Transient Imaging
Coded AperturePost-capture Focus
Control
Bokode: Long Distance Barcode
Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.info
BiDi Screen
Light Field
Augmented Light Field
WDF
Theory of Light Propagation
![Page 10: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/10.jpg)
International Conference on Computational Photography
Papers due November 2,
2009
http://cameraculture.media.mit.edu/iccp10
![Page 11: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/11.jpg)
![Page 12: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/12.jpg)
Traditional PhotographyTraditional Photography
Lens
Detector
Pixels
Image
Mimics Human Eye for a Single Snapshot:
Single View, Single Instant, Fixed Dynamic range and Depth of field for given Illumination in a Static world Courtesy: Shree
Nayar
![Page 13: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/13.jpg)
Computational Illumination
Computational Camera
Scene: 8D Ray Modulator
Display
GeneralizedSensor
Generalized OpticsProcessing
4D Ray BenderUpto 4D
Ray Sampler
Ray Reconstruction
Generalized Optics
Recreate 4D Lightfield
Light Sources
Modulators
4D Incident Lighting
4D Light Field
Computational PhotographyComputational Photography
![Page 14: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/14.jpg)
Computational Photography [Raskar and Tumblin]
1. Epsilon Photography– Low-level vision: Pixels– Multi-photos by perturbing camera parameters– HDR, panorama, …– ‘Ultimate camera’
2. Coded Photography– Mid-Level Cues:
• Regions, Edges, Motion, Direct/global– Single/few snapshot
• Reversible encoding of data– Additional sensors/optics/illum– ‘Scene analysis’
3. Essence Photography– High-level understanding
• Not mimic human eye• Beyond single view/illum
– ‘New artform’
![Page 15: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/15.jpg)
Digital
Epsilon
Coded
Essence
Goal and Experience
Low Level Mid Level HighLevel
Hyper realism
Raw
Angle, spectrum
aware
Non-visual Data, GPS
Metadata
Priors
Comprehensive
Computational Photography aims to make progress on
both axis
Camera ArrayHDR, FoV Focal stack
Decomposition problems
Depth
Spectrum
LightFields
Human Stereo Vision
8D reflectance field
Dylan’s bicycle
Transient Imaging
Virtual Object Insertion
Relighting
Augmented Human
Experience
Material editing from single photo
Scene completion from photos
Motion Magnification
![Page 16: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/16.jpg)
• Ramesh Raskar and Jack Tumblin
• Book Publishers: A K Peters• Siggraph 2009 booth: 20% off • Booth #2527
• ComputationalPhotography.org
• Meet the Authors• Thursday at 2pm-2:30pm
![Page 17: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/17.jpg)
Can you look around a corner ?
![Page 18: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/18.jpg)
Can you look around a corner ?
Kirmani, Hutchinson, Davis, Raskar 2009Accepted for ICCV’2009, Oct 2009 in Kyoto
Impulse Response of a Scene
![Page 19: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/19.jpg)
Femtosecond Laser as Light SourcePico-second detector array as
Camera
![Page 20: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/20.jpg)
Coded Aperture CameraCoded Aperture Camera
The aperture of a 100 mm lens is modified
Rest of the camera is unmodifiedInsert a coded mask with chosen binary pattern
![Page 21: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/21.jpg)
Captured Blurred Photo
![Page 22: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/22.jpg)
Refocused on Person
![Page 23: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/23.jpg)
MIT media lab camera culture
Bokode:
ankit mohan, grace woo, shinsaku hiura,quinn smithwick, ramesh raskar
camera culture group, MIT media lab
imperceptible visual tags for camera based interaction from a distance
![Page 24: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/24.jpg)
MIT media lab camera culture
Barcodesmarkers that assist machines in
understanding the real world
![Page 25: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/25.jpg)
• Smart Barcode size : 3mm x 3mm• Ordinary Camera: Distance 3 meter
Computational Probes: Computational Probes: Long Distance Bar-codesLong Distance Bar-codes
Mohan, Woo,Smithwick, Hiura, RaskarAccepted as Siggraph 2009 paper
![Page 26: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/26.jpg)
MIT Media Lab Camera Culture
Bokode
![Page 27: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/27.jpg)
MIT Media Lab Camera Culture
Defocus blur of Bokode
![Page 28: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/28.jpg)
MIT Media Lab Camera Culture
Image greatly magnified.
Simplified Ray Diagram
![Page 29: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/29.jpg)
MIT Media Lab Camera Culture
Our Prototypes
![Page 30: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/30.jpg)
MIT media lab camera culture
street-view tagging
![Page 31: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/31.jpg)
MIT media lab camera culture
tabletop/surface interaction
![Page 32: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/32.jpg)
MIT media lab camera culture
multi-user interaction
![Page 33: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/33.jpg)
MIT media lab camera culture
Varying Exposure VideoAmit Agrawal MERL, Yi Xu Purdue, Ramesh Raskar, MIT
![Page 34: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/34.jpg)
Deblurred Result
Blurred Photo
![Page 35: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/35.jpg)
Varying Exposure Video
DFT
Exposure Time
Exposure Time
Exposure Time
![Page 36: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/36.jpg)
Can we deal with particle-wave duality of light with modern Lightfield theory ?
45
Young’s Double Slit Expt
first null (OPD = λ/2)
Diffraction and Interferences modeled using Ray representation
![Page 37: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/37.jpg)
Light Fields
•Radiance per ray•Ray parameterization:•Position : x •Direction : θ Reference
plane
positiondirection
Goal: Representing propagation, interaction and image formation of light using purely position and angle
parameters
![Page 38: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/38.jpg)
Light Fields for Wave Optics EffectsLight Fields for Wave Optics Effects
Wigner Distribution Function
Light Field
LF < WDF
Lacks phase propertiesIgnores diffraction, phase masks
Radiance = Positive
Light Field
Augmented Light Field
WDF
ALF ~ WDF
Supports coherent/incoherent
Radiance = Positive/NegativeVirtual light sources
![Page 39: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/39.jpg)
Limitations of Traditional Lightfields
Wigner Distribution Function
TraditionaTraditional Light l Light FieldField
ray optics basedsimple and powerful
rigorous but cumbersomewave optics based
limited in diffraction & interference
holograms
beam shaping
rotational PSF
![Page 40: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/40.jpg)
Example: New Representations Augmented Lightfields
Wigner Distribution Function
TraditionaTraditional Light l Light FieldField
WDF
TraditionaTraditional Light l Light FieldField
Augmented LF
Interference & DiffractionInteraction w/ optical elements
ray optics basedsimple and powerfullimited in diffraction & interference
rigorous but cumbersomewave optics based
Non-paraxial propagation
http://raskar.scripts.mit.edu/~raskar/lightfields/
![Page 41: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/41.jpg)
(ii) Augmented Light Field with LF Transformer
50
WDF
Light Light FieldField
Augmented LF
Interaction at the optical elements
LF propagatio
n
(diffractive)
optical element
LF LF LF LF
LF propagatio
n
light field transformer
negative radiance
Augmenting Light Field to Model Wave Optics Effects , [Oh, Barbastathis, Raskar]
![Page 42: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/42.jpg)
Virtual light projector with real valued (possibly negative radiance) along a ray
51
real projector
real projector
first null (OPD = λ/2)
virtual light projector
![Page 43: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/43.jpg)
(ii) ALF with LF Transformer
52
![Page 44: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/44.jpg)
Converting LCD Screen = large Camera for 3D Interactive HCI and Video Conferencing
Matthew Hirsch, Henry HoltzmanDoug Lanman, Ramesh Raskar
BiDi Screen*
![Page 45: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/45.jpg)
Light Sensing Pixels in LCD
Dis
play
with
em
bedd
ed o
ptic
al s
enso
rs
Sharp Microelectronics Optical Multi-touch Prototype
![Page 46: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/46.jpg)
Beyond Multi-touch: Hover Interaction
• Seamless transition of multitouch to gesture
• Thin package, LCD
![Page 47: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/47.jpg)
Beyond Multi-touch: Mobile
Laptops
Mobile
![Page 48: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/48.jpg)
Design Vision
Object Collocated Captureand Display
Bare Sensor
Spa
tial L
ight
Mod
ulat
or
![Page 49: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/49.jpg)
Touch + Hover using Depth Sensing LCD Sensor
![Page 50: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/50.jpg)
Overview: Sensing Depth from Array of Virtual Cameras in
LCD
![Page 51: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/51.jpg)
Design Overview
Dis
play
with
em
bedd
ed o
ptic
al s
enso
rs
LCD , displaying mask
Opt
ical
sen
sor a
rray
~2.5 cm~50 cm
![Page 52: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/52.jpg)
International Conference on Computational Photography
Papers due November 2,
2009
http://cameraculture.media.mit.edu/iccp10
![Page 53: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/53.jpg)
![Page 54: MIT Camera Culture Group Update July 2009](https://reader035.vdocuments.us/reader035/viewer/2022081521/53fc74fe8d7f723a2d8b4f0a/html5/thumbnails/54.jpg)
Looking around a corner via Transient Imaging
Coded AperturePost-capture Focus
Control
Bokode: Long Distance Barcode
Camera Culture Group, MIT Media Lab Ramesh Raskar http://raskar.info
BiDi Screen
Light Field
Augmented Light Field
WDF
Theory of Light Propagation