light field photography and...

68
Light field photography and videography Marc Levoy Computer Science Department Stanford University

Upload: others

Post on 19-Sep-2020

2 views

Category:

Documents


0 download

TRANSCRIPT

Page 1: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Light fieldphotography and videography

Marc Levoy

Computer Science DepartmentStanford University

Page 2: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

List of projects

• high performance imagingusing large camera arrays

• light field photographyusing a handheld plenoptic camera

• dual photography

Page 3: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

High performance imagingusing large camera arrays

Bennett Wilburn, Neel Joshi, Vaibhav Vaish, Eino-Ville Talvala, Emilio Antunez,Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy

(Proc. SIGGRAPH 2005)

Page 4: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Stanford multi-camera array

• 640 × 480 pixels ×30 fps × 128 cameras

• synchronized timing• continuous streaming• flexible arrangement

Page 5: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Ways to use large camera arrays

• widely spaced light field capture• tightly packed high-performance imaging• intermediate spacing synthetic aperture photography

Page 6: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Intermediate camera spacing:synthetic aperture photography

Σ

Page 7: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Example using 45 cameras[Vaish CVPR 2004]

Page 8: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc
Page 9: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Tiled camera array

• world’s largest video camera• no parallax for distant objects• poor lenses limit image quality• seamless mosaicing isn’t hard

Can we match the image quality of a cinema camera?

Page 10: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Tiled panoramic image(before geometric or color calibration)

Page 11: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Tiled panoramic image(after calibration and blending)

Page 12: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Tiled camera array

• world’s largest video camera• no parallax for distant objects• poor lenses limit image quality• seamless mosaicing isn’t hard• per-camera exposure metering• HDR within and between tiles

Can we match the image quality of a cinema camera?

Page 13: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

same exposure in all cameras

individually metered

checkerboardof exposures

Page 14: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

High-performance photography as multi-dimensional sampling

• spatial resolution• field of view• frame rate• dynamic range• bits of precision• depth of field• focus setting• color sensitivity

Page 15: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Spacetime aperture shaping

• shorten exposure time to freeze motion → dark

• stretch contrast to restore level → noisy

• increase (synthetic) aperture to capture more light →decreases depth of field

Page 16: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

• center of aperture: few cameras, long exposure →high depth of field, low noise,but action is blurred

• periphery of aperture: many cameras, short exposure →freezes action, low noise,but low depth of field

Page 17: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc
Page 18: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc
Page 19: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Light field photography using a handheld plenoptic camera

Ren Ng, Marc Levoy, Mathieu Brédif,Gene Duval, Mark Horowitz and Pat Hanrahan

(Proc. SIGGRAPH 2005and TR 2005-02)

Page 20: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Conventional versus light field camera

Page 21: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Conventional versus light field camera

uv-plane st-plane

Page 22: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Conventional versus light field camera

uv-planest-plane

Page 23: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Prototype camera

4000 × 4000 pixels ÷ 292 × 292 lenses = 14 × 14 pixels per lens

Contax medium format camera Kodak 16-megapixel sensor

Adaptive Optics microlens array 125µ square-sided microlenses

Page 24: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Mechanical design

• microlenses float 500µ above sensor• focused using 3 precision screws

Page 25: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc
Page 26: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Prior work

• integral photography– microlens array + film– application is autostereoscopic effect

• [Adelson 1992]– proposed this camera– built an optical bench prototype using relay lenses– application was stereo vision, not photography

Page 27: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Digitally stopping-down

• stopping down = summing only the central portion of each microlens

Σ

Σ

Page 28: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Digital refocusing

• refocusing = summing windows extracted from several microlenses

Σ

Σ

Page 29: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

A digital refocusing theorem

• an f / N light field camera, with P × P pixels under each microlens, can produce views as sharp as an f / (N × P) conventional camera

– or –

• it can produce views with a shallow depth of field ( f / N ) focused anywhere within the depth of field of an f / (N × P) camera

Page 30: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Example of digital refocusing

Page 31: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Refocusing portraits

Page 32: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Action photography

Page 33: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Extending the depth of field

conventional photograph,main lens at f / 22

conventional photograph,main lens at f / 4

light field, main lens at f / 4,after all-focus algorithm

[Agarwala 2004]

Page 34: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Macrophotography

Page 35: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Digitally moving the observer

• moving the observer = moving the window we extract from the microlenses

Σ

Σ

Page 36: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Example of moving the observer

Page 37: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Moving backward and forward

Page 38: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Implications

• cuts the unwanted link between exposure(due to the aperture) and depth of field

• trades off (excess) spatial resolution for ability to refocus and adjust the perspective

• sensor pixels should be made even smaller, subject to the diffraction limit

36mm × 24mm ÷ 2.5µ pixels = 266 megapixels20K × 13K pixels4000 × 2666 pixels × 20 × 20 rays per pixel

Page 39: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Can we build a light field microscope?

• ability to photograph moving specimens

• digital refocusing → focal stack →deconvolution microscopy → volume data

Page 40: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Dual Photography

Pradeep Sen, Billy Chen, Gaurav Garg, Steve Marschner,Mark Horowitz, Marc Levoy, Hendrik Lensch

(Proc. SIGGRAPH 2005)

Page 41: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Helmholtz reciprocity

scene

light

camera

Page 42: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Helmholtz reciprocity

scene

camera

light

Page 43: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

photocell

scene

Measuring transport along a set of pathsprojector

Page 44: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

scene

point light

Reversing the pathscamera

Page 45: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Forming a dual photograph

scene

photocellprojector “dual” light“dual” camera

Page 46: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Forming a dual photograph

scene

image ofscene

“dual” light“dual” camera

Page 47: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Physical demonstration

• light replaced with projector• camera replaced with photocell• projector scanned across the scene

conventional photograph,with light coming from right

dual photograph,as seen from projector’s position

and as illuminated from photocell’s position

Page 48: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Related imaging methods

• time-of-flight scanner– if they return reflectance as well as range– but their light source and sensor are typically coaxial

• scanning electron microscope

Velcro® at 35x magnification,Museum of Science, Boston

Page 49: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

The 4D transport matrix

scene

photocellprojector camera

Page 50: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

camera

The 4D transport matrix

scene

projector

P

pq x 1

C

mn x 1

T

mn x pq

Page 51: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

T PC =

The 4D transport matrix

pq x 1mn x 1

mn x pq

Page 52: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

TC =

The 4D transport matrix

10000

mn x pq

pq x 1mn x 1

Page 53: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

TC =

The 4D transport matrix

01000

mn x pq

pq x 1mn x 1

Page 54: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

TC =

The 4D transport matrix

00100

mn x pq

pq x 1mn x 1

Page 55: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

The 4D transport matrix

T PC =

pq x 1mn x 1

mn x pq

Page 56: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

The 4D transport matrix

applying Helmholtz reciprocity...

T PC =

pq x 1mn x 1

mn x pq

T P’C’ =

mn x 1pq x 1

pq x mn

T

Page 57: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Example

conventional photographwith light coming from right

dual photographas seen from projector’s position

Page 58: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Properties of the transport matrix

• little interreflection → sparse matrix• many interreflections → dense matrix• convex object → diagonal matrix• concave object → full matrix

Can we create a dual photograph entirely from diffuse reflections?

Page 59: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Dual photographyfrom diffuse reflections

the camera’s view

Page 60: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

The relighting problem

• subject captured under multiple lights• one light at a time, so subject must hold still• point lights are used, so can’t relight with cast shadows

Paul Debevec’sLight Stage 3

Page 61: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

The 6D transport matrix

Page 62: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

The 6D transport matrix

Page 63: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

The advantage of dual photography

• capture of a scene as illuminated by different lights cannot be parallelized

• capture of a scene as viewed by different cameras can be parallelized

Page 64: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

scene

Measuring the 6D transport matrix

projector

camera arraymirror arraycamera

Page 65: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Relighting with complex illumination

• step 1: measure 6D transport matrix T• step 2: capture a 4D light field• step 3: relight scene using captured light field

scene

camera arrayprojector

TT P’C’ =

mn x uv x 1pq x 1

pq x mn x uv

Page 66: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoy

Running time

• the different rays within a projector can in fact be parallelized to some extent

• this parallelism can be discovered using a coarse-to-fine adaptive scan

• can measure a 6D transport matrix in 5 minutes

Page 67: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

Can we measure an 8D transport matrix?

scene

camera arrayprojector array

Page 68: Light field photography and videographygraphics.stanford.edu/talks/lightfields-UVa-oct05/light...Adam Barth, Andrew Adams, Mark Horowitz, Marc Levoy (Proc. SIGGRAPH 2005) ©2005Marc

© 2005 Marc Levoyhttp://graphics.stanford.edu

T PC =

pq x 1mn x 1 mn x pq