Light  Field  Montages

Vera Dadok

Problem Given multiple light field shots of the a scene with minor

differences, how can you successfully combine the best parts of each image?

Challenges: Many representations of light fields (format) Large file sizes, higher dimensionality Creation and display of light fields

Image  Source:  Light  Field  Rendering,  1996

Motivation Why use Light Fields? •  Hand-held

plenoptic cameras •  Light Fields contain

more information than images (4-d vs. 2-d sample)

•  Few tools to modify light fields exist, and many interesting operations to develop Image  Source:  Lytro.com

Motivation Why montage Light Fields? •  Single Light Field may not

capture ideal scene o  Want to remove/change elements

•  Automatic montage method necessary o  Large image stacks or

representations

Refined  Problem  Statement

•  Input o  Two Light Fields of the same scene with minor differences represented as

an array of images taken at different (u,v) coordinates

o  User selection of pixels fixed to each source

•  Output o  Light Field Montage containing the best of the two images

Image  Source:  Light  Field  Rendering,  1996

Refined  Problem  Statement

•  Input o  Two Light Fields of the same scene with minor differences represented as

an array of images taken at different (u,v) coordinates

o  User selection of pixels fixed to each source

•  Output o  Light Field Montage containing the best of the two images

Approach

•  Ingredients: o  Light Fields o  Light Field Viewer o  User Interaction Algorithm o  Montage Algorithm

Input: 2  Light  Fields

User  interaction:  Select  pixels  from  

sources

Montage  Algorithm

Result:   Light  Field  Montage

User  Interaction •  Light field is represented as a matrix of images,

each positioned at a (u,v) coordinate. •  User interacts only with the 4 corners

V-­‐‑position

U-­‐‑position

V-­‐‑position

U-­‐‑position

Screenshot

Light  field  montage Based on Graph Cut (2d and 3d) •  Minimize a Cost Function (Energy Function)

1

2

o  Data Term •  Based on user-selected pixels and

surrounding pixels o  Augmented user-selection with inverse

square law based on distance •  Interpolated pixels at (u,v) coordinates from

corner pixel selections

o  Smoothness Term •  Several possible functions: Color, Color &

Gradient, Gradient, Color & Edges (see Agarwala et. al, SIGGRAPH 2004)

•  Color cost – related to difference in pixel colors between 2 source light fields

Image:  UC  Berkeley  CS294-­‐‑69  Course  Notes,  2011

min  E  =  min  (Data  Term  +  Smoothness  Term)

Graph-­‐‑cut  at  work  (2-­‐‑d)  

3d  Graph  Cut

Graph  cut  on  ordered  stack  of  images  from  single  line  of  u  or  v

Results

Demos

Plant  Demos: Originals: file:///Users/Vera/Desktop/apertureupdateb/aperture.swf?lightfield=cdswithoutplantscroppedsmaller&xmlzip=cdswithplants.zip   file:///Users/Vera/Desktop/apertureupdateb/aperture.swf?lightfield=cdswithplantscroppedsmaller&xmlzip=cdswithplants.zip  

Graph  Cuts: file:///Users/Vera/Desktop/apertureupdateb/aperture.swf?lightfield=plants2d&xmlzip=cdswithplants.zip   file:///Users/Vera/Desktop/apertureupdateb/aperture.swf?lightfield=plants3d&xmlzip=cdswithplants.zip  

Jellybean  Demo: Originals: file:///Users/Vera/Desktop/apertureupdateb/aperture.swf?lightfield=preview&xmlzip=previewc.zip   file:///Users/Vera/Desktop/apertureupdateb/aperture.swf?lightfield=newColoredBeans2&xmlzip=previewc.zip  

Graph  Cut: file:///Users/Vera/Desktop/apertureupdateb/aperture.swf?lightfield=previewnewColoredBeans2montage&xmlzip=previewc.zip  

Future  Work •  4d graph-cut •  Improved user interactions

o  What is the most intuitive way to edit a light field? o  Automatic reality checks using known geometry/depth o  Speed o  Multiple light fields montage (just layer via user interface) o  Additional montage modes (lighting, etc.)

•  New cost functions for graph cut o  Use depth maps, geometry

•  Obtain more light fields to test this on o  Or create synthetic light fields.

•  Manipulating different representations of light fields o  Depth array of images (full aperture)

Thank  You!

Selected  references •  Images :

o  UC Berkeley CS294-69 Course Notes, 2011 o  Lytro.com o  Light Field Rendering, Marc Levoy and Pat Hanrahan, 1996.

•  Light Fields: o  Stanford Repository, http://lightfield.stanford.edu/, December 2011

•  Code supplements: o  Graphcut:, Matlab wrapper for robust higher order potentials. Shai Bagon,

http://www.wisdom.weizmann.ac.il/~bagon/matlab.html 2009. •  See also: http://www.wisdom.weizmann.ac.il/~bagon/matlab_code/robustpn_wrapper.bib

o  Light Field Renderer: From http://lightfield.stanford.edu/, o  FZip

•  Code supplement references: o  Efficient Approximate Energy Minimization via Graph Cuts. Yuri Boykov, Olga Veksler, Ramin Zabih,

IEEE transactions on PAMI, vol. 20, no. 12, p. 1222-1239, November o  What Energy Functions can be Minimized via Graph Cuts? Vladimir Kolmogorov and Ramin Zabih.

European Conference on Computer Vision (ECCV), May 2002. o  An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Vision. Yuri

Boykov and Vladimir Kolmogorov. In IEEE Transactions on Pattern Analysis and Machine Intelligence (PAMI), September 2004

For  a  more  thorough  list  of  references,  please  contact  Vera  Dadok