cornar: looking around corners using trillion fps imaging
DESCRIPTION
We have built a camera that can look around corners and beyond the line of sight. The camera uses light that travels from the object to the camera indirectly, by reflecting off walls or other obstacles, to reconstruct a 3D shape.TRANSCRIPT
![Page 1: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/1.jpg)
Raskar, Camera Culture, MIT Media Lab
Camera Culture
Ramesh Raskar
MIT Media LabRamesh Raskar
Computational Light Transport:
CORNAR: Looking Around Corners using
Trillion FPS Imaging
http://raskar.info/cornar
![Page 2: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/2.jpg)
What isaround the corner ?
![Page 3: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/3.jpg)
Can you look around the corner ?
![Page 4: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/4.jpg)
Can you look around the corner ?
![Page 5: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/5.jpg)
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
![Page 6: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/6.jpg)
![Page 7: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/7.jpg)
![Page 8: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/8.jpg)
Bit Hacking
Phot
on H
acki
ng
Computer Vision
Optics
Sensors
Signal Processing
Displays
Machine Learning
Computational Light Transport
Computational PhotographyIllumination
Co-designing Optical and Digital Processing
![Page 9: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/9.jpg)
2nd Bounce
Multi-path Analysis
1st Bounce
3rd Bounce
![Page 10: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/10.jpg)
CORNAR: Femto-PhotographyFemtoFlash
Trillion FPS camera
Computational Optics
Serious Sync
•2012: 3D around a corner (NatureComm, Velten, et. al.)•2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala)•2011: DARPA Young Faculty Award•2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar)•2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)•2008: Indirect depth (Hirsch, Raskar)•2008: Transient Light Transport (Raskar, Davis, March 2008)
With M Bawendi,
MIT Chemistry
![Page 11: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/11.jpg)
Inverting Light Transport
Multiple Scattering Direct/Global
Dual Photography
[Seitz , Kutulakos, Matsushita 2005] [Nayar, Raskar et al 2006]
[Sen et al 2005]
LIDAR
[Atcheson et al 2008][Kutulakos, Steger 2005]
![Page 12: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/12.jpg)
Collision avoidance, robot navigation, …
![Page 13: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/13.jpg)
…, bronchoscopies, …
![Page 14: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/14.jpg)
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
![Page 15: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/15.jpg)
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
![Page 16: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/16.jpg)
Streak Photo
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
![Page 17: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/17.jpg)
Multi-Dimensional Light Transport
5-D Transport
![Page 18: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/18.jpg)
s1
Streak-camera
Occluder
C
3rd bounce
Why Pico-second Resolution?
s2ToF Diff = 0.15 mm
p1
p21st bounce
1cm
Curse of Pythagoras
![Page 19: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/19.jpg)
Streak Photo
L
xz
s
S
Streak-camera
Laser beam
Occluder
CB
Echoes of Light
3rd bounce
R
![Page 20: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/20.jpg)
Trillion FPS
ToF Streak Tube = Inverse of CRT
1D camera: Single scan line stretched vertically in time~2 ps resolution, 480 lines ~= 1 ns
But for small samples in biochemistry
Very accurate sync
![Page 21: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/21.jpg)
Time-Image
Time Profile for a single pixel
R
![Page 22: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/22.jpg)
Third Bounce (First bounce not shown)
Space, 640 pixels
Tim
e, ~
2ns e
ach
row
Time Image of a single point
![Page 23: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/23.jpg)
![Page 24: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/24.jpg)
![Page 25: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/25.jpg)
Forward
Wavefront Propagation
Reconstruction
Non-linear Inversion
Invertibility
Scene Priors
Analysis
Resolution and dimensions
3D shape result from synthetic data
![Page 26: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/26.jpg)
Steady State 4D
Impulse Response, 5D
[Raskar and Davis, 2007]
[Kajiya, 1986] [Seitz.., 2005]
![Page 27: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/27.jpg)
Scene with hidden elements
Raw Time profiles
t
Signal Proc.
Novel light transport models and inference
algorithms
Photo, geometry, reflectance
beyond line of sight
3D Time images
Ultra fast illumination and camera
5D Capture
Time Resolved Multi-path Imaging
![Page 28: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/28.jpg)
Third Bounce (First bounce not shown)
![Page 29: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/29.jpg)
Third Bounce (First bounce not shown)
![Page 30: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/30.jpg)
Third Bounce (First bounce not shown)
![Page 31: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/31.jpg)
Third Bounce (First bounce not shown)
![Page 32: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/32.jpg)
Third Bounce (First bounce not shown)
![Page 33: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/33.jpg)
Capture Setup
Photos from Streak Camera
Hidden Scene
![Page 34: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/34.jpg)
Capture Setup
Photos from Streak Camera
Hidden Scene OverlayReconstruction
![Page 35: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/35.jpg)
![Page 36: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/36.jpg)
Hidden 3D Shape Space-time Photos
![Page 37: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/37.jpg)
![Page 38: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/38.jpg)
![Page 39: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/39.jpg)
Motion beyond line of sight
Pandharkar, Velten, Bardagjy, Lawson, Bawendi, Raskar, CVPR 2011
![Page 40: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/40.jpg)
BRDF (reflectance) from Single Viewpoint and Time Images
Naik, Zhao, Velten, Raskar, Bala, (SIGGRAPH Asia 2011)
![Page 41: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/41.jpg)
![Page 42: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/42.jpg)
…, cardioscopies, …
Participating Media
![Page 43: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/43.jpg)
Space-time Photos
![Page 44: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/44.jpg)
![Page 45: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/45.jpg)
Trillion Frames Per Second Imaging
http://raskar.info/trillionfps
![Page 46: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/46.jpg)
Each frame = ~2ps = 0.6 mm of Light Travel
![Page 47: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/47.jpg)
Camera Time
![Page 48: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/48.jpg)
Camera Time
raskar.info/trillionfps
![Page 49: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/49.jpg)
World Time
![Page 50: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/50.jpg)
Lorentz transformation: distances, velocities, ordering
raskar.info/trillionfps
![Page 51: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/51.jpg)
Pixel Ray Wave Photons(ampli+phase)
Steady State TransientImpulse/Step•2012: 3D around corner (Nature Comm, Velten et al)•2011: Material Sensing (Siggraph Asia, Naik, Zhao, Velten, Raskar, Bala)•2011: DARPA Young Faculty Award•2011: Motion Sensing (CVPR, Pandharkar, Velten, Bardagjy, Bawendi, Raskar)•2009: Hidden barcode (Kirmani, Hutchinson, Davis, Raskar, ICCV’2009)•2008: Indirect depth (Hirsch, Raskar)•2008: Transient Light Transport (Raskar, Davis, March 2008)
FemtoFlash
Trillion FPS camera
Computational OpticsSerious Sync
![Page 52: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/52.jpg)
• Collaborations Welcome• Dataset Available• Propose Configurations
raskar(at)media.mit.eduhttp://www.media.mit.edu/~raskar/cornar/
![Page 53: CORNAR: Looking Around Corners using Trillion FPS Imaging](https://reader033.vdocuments.us/reader033/viewer/2022061104/5404fb038d7f72a6768b495f/html5/thumbnails/53.jpg)
Femto-PhotographyLooking Around the Corner
BRDF Detection
Trillion FPS Movies
Space-time Transforms
http://raskar.info