eg 2011 | computational plenoptic imaging star | vi. high speed imaging1 computational plenoptic...

34
EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 1 Computational Plenoptic Imaging don Wetzstein 1 Ivo Ihrke 2 Douglas Lanman 3 Wolfgang Heidr ersity of British Columbia 2 Saarland University 3 MIT Medi Eurographics 2011 – State of the Art Report VI. Multiplexing Time

Upload: abigayle-wilcox

Post on 29-Jan-2016

215 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 1

Computational Plenoptic ImagingGordon Wetzstein1 Ivo Ihrke2 Douglas Lanman3 Wolfgang Heidrich1

1University of British Columbia 2Saarland University 3MIT Media Lab

Eurographics 2011 – State of the Art Report

VI. Multiplexing Time

Page 2: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 2

History – Eadweard Muybridge 1830-1904

http://en.wikipedia.org/wiki/Eadweard_Muybridge

Page 3: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 3

History – Étienne-Jules Marey 1830-1904

www.wikipedia.org

Page 4: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 4

VI.I Time Lapse Photography

Page 5: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 5

BBC Time Lapse – Look It!

ND 3.0 filter, f22, 1 minute exposure

• Long exposures to avoid temporal aliasing

Page 6: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 6

VI.II High-Speed Imaging

Page 7: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 7

High-Speed Cameras

Vision Research Phantom Flex (CMOS)2570 fps at HD resolution

Photron FASTCAM SA5 (CMOS)7500 fps at megapixel resolutionone million fps at 64x64 pixels

Casio Exilim Series (consumer cam)1000 fps at reduced resolution

Shimadzu HyperVision HPV-2 (CCD)one million fps at 312x260 pixels

Page 8: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 8

Assorted Pixels

[Narasimhan & Nayar 05]

Page 9: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 9

Temporal Mosaic with DMD

[Bub et al. 10]

DMD aligned with CCD in microscope

Page 10: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 10

Non-Destructive Sensor Readout & Pixim

Cypress Semiconductor LUPA 30003 megapixels, 485 fps

www.pixim.com

Page 11: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 11

Coded Rolling Shutter

[Gu et al. 10]

Page 12: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 12

Reinterpretable Imager

• Moving pinhole over time in aperture

• Capture with light field camera

[Agrawal et al. 10]

Page 13: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 13

Bullet Time Effect

from ‘The Matrix’

Page 14: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 14

Stanford Multi-Camera Array

[Wilburn et al. 04]

Page 15: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 15

Coded Temporal Sampling

[Agrawal et al. 10]

Page 16: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 16

High-Speed Illumination – Electronic Strobes

Harold ‘Doc’ Edgerton1903-1990

Page 17: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 17

Temporal Dithering with DLP Illumination

[Narasimhan et al. 08]

Page 18: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 18

Coded Strobing Photography

[Reddy et al. 11]

Page 19: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 19

Streak Cameras

www.hamamatsu.com

C5680$200K

Page 20: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 20

VI.I Motion Deblurring

Page 21: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 21

Motion Deblurring Overview

• Motion blur is velocity-dependent

• Can be described as convolution, where– Kernel shape is motion trajectory– Trajectory is modulated by exposure function

http://en.wikipedia.org/wiki/Motion_blur

Page 22: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 22

Deconvolution is Still Hard

• Again – problems:

– Camera noise

– Spatially varying kernel (velocity-dependent)

– Unknown motion trajectory

– Ill-posed problem, kernel of box integration function is not invertible (optical cancellation of image frequencies)

Page 23: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 23

Approaches to Improve Motion Deblurring

• Make PSF invertible coded exposure

• Make PSF velocity-invariant shift-invariant deconvolution

• Automatize PSF estimation

Page 24: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 24

Flutter Shutter

[Raskar et al. 06]

Page 25: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 25

Optimal Motion PSFs

• Optimality criteria PSF invertibility & estimation

[Agrawal & Xu 07]

Page 26: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 26

Motion Invariant Photography

• Engineer PSF to be motion invariant

• Only for 1D motion

[Levin 08]

Page 27: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 27

Hybrid Cameras

• Combined high-speed low-quality & low-speed high-quality camera

Input images Computed PSF

Deblurred Result Ground Truth

[Ben-Ezra & Nayar 04]

Page 28: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 28

Motion Blur in Video

• Coded exposure &

super-resolution in

successive video frames

[Agrawal et al. 09]

Page 29: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 29

Next: Further Light Properties

Page 30: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 30

Flexible Voxels

• Flexible space-time resolution as post-processing

[Gupta et al. 10]

Page 31: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 31

Synthetic Shutter Speed Imaging

• Combine multiple short exposures to reduce noise

• Align with optical flow

[Tel

leen

07]

Page 32: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 32

Hybrid Cameras

• Motion deblurring & super-resolution

[Tai

et a

l. 08

]

Page 33: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 33

Hybrid Cameras

• Motion deblurring & depth from two low-resolution high-speed camers

[Li e

t al.

08]

Input images

Deblurred result Recovered Depth

Page 34: EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging1 Computational Plenoptic Imaging Gordon Wetzstein 1 Ivo Ihrke 2 Douglas Lanman

EG 2011 | Computational Plenoptic Imaging STAR | VI. High Speed Imaging 34

Analysis

• Analysis of optimal coded, single image deblurring

• MIP becomes worse when velocities exceed expectations

[Agrawal & Raskar 09]

Coded Exposure Motion Invariant Photography