![Page 1: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/1.jpg)
1
Plenoptic Imaging
Chong Chen
Dan Schonfeld
Department of Electrical and Computer Engineering
University of Illinois at Chicago
May 7 2009
![Page 2: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/2.jpg)
2
Plenoptic Function (1)
• From plenus (complete or full) and optic.• An idealized function to express the image of a
scene from any possible viewing position at any viewing angle at any point in time.
( , , , , , , )x y zP V V V t
![Page 3: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/3.jpg)
3
Plenoptic Function (2)
![Page 4: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/4.jpg)
4
Plenoptic Camera (1)
R. Ng et al., Stanford University 2005
![Page 5: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/5.jpg)
5R. Ng et al., Stanford University 2005
Plenoptic Camera (2)
![Page 6: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/6.jpg)
6
Image-Base Rendering
L. McMillan and G. Bishop, SIGGRAPH 1995
![Page 7: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/7.jpg)
7
LightfieldThe lightfield data is composed of six 4D functions, where the plane of the inner box is indexed with coordinate (u, v) and that of the outer box with coordinate (s, t).
M. Levoy and P. Hanrahan, SIGGRAPH 1996
![Page 8: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/8.jpg)
8
Sampling and Reconstruction• The lightfield reconstruction is computed as
• The sampled lightfield ls(u,v,s,t) is represented by
( , , , ) ( , , , )[ ( , , , ) ( , , , )]I u v s t R u v s t L u v s t P u v s t
( , , , ) ( , , , ) [ ( , , , ) ( , , , )]i u v s t r u v s t l u v s t p u v s t
1 2 1 2
1 2 1 2, , ,
( , , , ) ( , , , ) ( ) ( ) ( ) ( )sn n k k Z
l u v s t l u v s t u n u v n v s k s t k t
1 2 1 2
1 2 1 2
, , ,
2 2 2 2( , , , ) ( ) ( ) ( ) ( )s
n n k k Z
n n k kL u v s t L u L v L s L t
u v s t
H.Y. Shum et al., SIGGRAPH 2000
![Page 9: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/9.jpg)
9
Plenoptic Sampling (1)
( , ) ( ,0)( , , , )
ftl v t l v
z u v s t
Assumptions: Lambertian surfaces and no occlusion.
H.Y. Shum et al., SIGGRAPH 2000
![Page 10: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/10.jpg)
10
Plenoptic Sampling (2)( )
( ) ( )
0 0
( , , , ) ( , , , )
( , ,0,0)
u v s t
u v s t
j u v s tu v s t
j u v j s t
L l u v s t e dudvdsdt
fs ftl u v e dudv e dsdt
z z
2
0 0
4 '( , ) ( ) ( )u v s u t v
f fL
z z
The spectral support of a lightfield signal is bounded by the minimum and maximum depths only, no matter how complicated the spectral support might be because of depth variations in the scene.
where L’ is the 2D Fourier transform of l(x,y,0,0)
H.Y. Shum et al., SIGGRAPH 2000
![Page 11: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/11.jpg)
11
Unstructured Lumigraph
M. Cohen et al., SIGGRAPH 2001
![Page 12: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/12.jpg)
12
Parallel Cameras
1
( , )
2 '( ) ( )
v t
v t v
L
d fLd d
Plenoptic signals taken by parallel cameras will be bandlimited, and their spectral support is bounded by the minimum and maximum depths.
![Page 13: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/13.jpg)
13
Unstructured Cameras (1)( )
2 2
21 1 1
( , ) ( , )
( 1) 1( ,0)
1
v tj v tv tL l v t e dvdt
d d f a fl td a d a dv
a
1 2[1,2 ]
[ ]
0
j vvv
BF e
0v
Plenoptic signals taken by unparallel cameras will not be bandlimited.
0v
( )v tj v te dvdt
1 1 1/2 '( ) [ ]t t
d qj jaf v a
t
d dL e F e
f f
[ , ]B n x is the Bessel Function.
![Page 14: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/14.jpg)
14
Unstructured Cameras (2)
Plenoptic signals taken by unparallel cameras with limited FOV and rotations can be approximated to be bandlimited
( )
1 1
( , ) ( , )
( ( tan ) ,0)
v tj v tv tL l v t e dvdt
d fl v f td d
tan tan( ) tan Assuming
( )v tj v te dvdt
1 tan1 12 '( ) ( )
vd f
jd
v t v
d d fe L
d d d
![Page 15: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/15.jpg)
15
Concentric Mosaic (1)
• for the constant depth concentric mosaic, the spectrum lies on a line with slope
After linearization
( )
( )
( , ) ( , )
( ,0)
2 '( ) ( )
v
v
j vv
j v
v v
L l v e dvd
rl v e dvd
r Rr
Lr R
C. Zhang and T. Chen, Carnegie Mellon University 2001
![Page 16: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/16.jpg)
16
Bernstein's inequality: if is a bounded function on with supported in the ball . Then for all multi-indices , there exist constants (depending only on and on the dimension n) such that
Before linearization
Concentric Mosaic (2)
( )
( )
sin
cos
( , ) ( , )
sin( ,0)
cos
2 '( ) [ ]
v
v
v
j vv
j v
rj
r Rv
L l v e dvd
rl v e dvd
r R
L F e
,nC
,b n b LLf C R f
bfnR
0,wB ˆbf
![Page 17: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/17.jpg)
17
Unoccluded Image Constraints
If any point (f(x), g(x)) on the surface of the scene is differentiable, there is no occlusion with the surface if and only if
'
'
( )( ) ( )
( )
f xf x g x L
g x
which can be proved by Cauchy's Mean-Value Theorem.
![Page 18: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/18.jpg)
18
![Page 19: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/19.jpg)
19
![Page 20: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/20.jpg)
20
Conclusions and Future Work
• Plenoptic signals taken by unparallel cameras will not be bandlimited.
• Plenoptic signals taken by unparallel cameras with limited FOV and rotations can be approximated to be bandlimited.
• Sampling (light conditions, surface luminance)
• Reconstruction (integral imaging)
![Page 21: Chong Chen Dan Schonfeld Department of Electrical and Computer Engineering](https://reader036.vdocuments.us/reader036/viewer/2022062322/5681500c550346895dbdea5f/html5/thumbnails/21.jpg)
21
Thank you !