siggraph 2014 course on computational cameras and displays (part 3)
DESCRIPTION
Recent advances in both computational photography and displays have given rise to a new generation of computational devices. Computational cameras and displays provide a visual experience that goes beyond the capabilities of traditional systems by adding computational power to optics, lights, and sensors. These devices are breaking new ground in the consumer market, including lightfield cameras that redefine our understanding of pictures (Lytro), displays for visualizing 3D/4D content without special eyewear (Nintendo 3DS), motion-sensing devices that use light coded in space or time to detect motion and position (Kinect, Leap Motion), and a movement toward ubiquitous computing with wearable cameras and displays (Google Glass). This short (1.5 hour) course serves as an introduction to the key ideas and an overview of the latest work in computational cameras, displays, and light transport.TRANSCRIPT
Compressive Display Systems
Gordon Wetzstein
MIT / Stanford University
media.mit.edu/~gordonw
displayblocks.org
This slide has a 16:9 media windowEvolution?
1928 2014
Nature
Image: National Geographic
Evolution!
uberpixel
Next-generation Devices
computation optics & electronics human visual
system
interaction
optics (compressive) computationsensing
Computational &
Compressive Displays
Computational
Imaging
displayblocks.org
Images: Wikipedia, Shinya Yoshioka
This slide has a 16:9 media windowNature
Image: Desafio Monteverde and Arenal Volcano Tours Costa Rica
Three-layer Tensor Display
This slide has a 16:9 media windowCompressive Light
Field Displays
This slide has a 16:9 media window
This slide has a 16:9 media window
viewer moves right
vie
wer
moves d
ow
n
4D Light Field
Display OpticsComputational
Processing
Compressive Displays
2D display
barrier
Parallax Barriers – Ives 1903
• low resolution & very dim
Integral Imaging – Lippmann 1908
• low-res, but brighter than parallax barriers
2D display
len
sle
ts
patent drawings - early 20th century
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
MN
O
C
U C
C
N
Structural Formula for Compressive Displays
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
MN
O
C
U C
C
N
Nonlinear Pixel Interaction & Coupling
layer 2
layer 1
Conventional Parallax Barriers
layer 2
layer 1
Conventional Parallax Barriers
blocked!
p1
p2
l = p1*p2
nonlinear
layer 2
layer 1
Most Volumetric Displays / Additive Layers
e.g., LEDs or transparent OLEDs
not blocked!
p1
p2
l = p1+p2
linear
Holography – Nonlinear Interaction
plane wave
hologram
emitted wavefront:
screen or retina
received intensity:
I(x) = T U(x){ }2
= Re T U(x){ }{ }2
+ Im T U(x){ }{ }2
U(x)
Holography – Coupling
plane wave
hologram
W (x,u = l sin(q )) = t x+x '
2
æ
èç
ö
ø÷t x -
x '
2
æ
èç
ö
ø÷e
2pix 'u dx 'ò
Fourier transform of all points interacting
with each other!
nonlinear interaction & pixel coupling!
attenuation layers with
spacers
backlight
Layered 3D – SIGGRAPH 2011
Layered 3D – SIGGRAPH 2011
layer 2
layer K
layer 1
…
Layered 3D – SIGGRAPH 2011
layer 2
layer K
layer 1
…
p1
p2
pK
l = p1*p2*…*pK
Layered 3D – SIGGRAPH 2011
…
layer 2
layer K
layer 1
Tensor Displays – SIGGRAPH 2012
directional backlight three layer
Reconfigurable
time
Tensor Displays – Directional Backlight
Perceptual Integration
layer 2
microlens array
layer 1
LCD with directional backlight, rank 6
LCD + directional BL
conventional lenslets
view from above
LCD with directional backlight, rank 6 (as seen by observer)
front LCD
directional backlight
Filmed with High-speed Camera
four stacked liquid crystal panels
two crossed polarizers
LCD 1
LCD 2
LCD 3
backlight
Polarization Fields – SIGGRAPH ASIA 2011
polarizer
polarizercolor filter array
polarizercolor filter array
polarizercolor filter array
backlight
Polarization Fields – SIGGRAPH ASIA 2011
LCD 1
LCD 2
LCD 3
polarizer
polarizer
image formation
f1
f2
f3
L(x,q) = sin2 Q(x,q)( )
backlight
Polarization Fields – SIGGRAPH ASIA 2011
LCD 1
LCD 2
LCD 3
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
MN
O
C
U C
C
N
Compression
Which Light Field is More Compressible?
a b
Which Light Field is More Compressible?
a b
This slide has a 16:9 media window
4D Light Field
Uniform or
Directional BacklightStacked Layers
Nonnegative Tensor
Factorization
Display-adaptive
Compression
Compressive
Computed Tomography
(LCDs or Transparencies)
Optics
Observer = Decoder
Give those Pixels a Break!
Applied Mathematics
• sparse optimization
• low-rank factorization
• computed tomography
• …
Benefits for Optics & Electronics
• fewer pixels
• relaxation on refresh rate
• thinner form factors
• …
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
t
HR3D
SIG Asia 2010
t tLayered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
Conventional Parallax Barriers
layer 2
layer 1
From Conventional to Compressive Displays
…
…
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
t
HR3D
SIG Asia 2010
t tLayered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
…
time
Perceptual Integration
Tensor Displays – Multilayer & Directional Backlighting
From Conventional to Compressive Displays
Parallax Barriers
1903
Time-Shifted
Parallax Barriers 2007
t
HR3D
SIG Asia 2010
t tLayered 3D
SIGGRAPH 2011
Tensor Displays
SIGGRAPH 2012
Perceptual Integration
Tensor Displays – Directional Backlighting
time
thin
!
From Conventional to Compressive Displays
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
MN
O
C
U C
C
N
Real-time Optimization
This slide has a 16:9 media window
4D Light Field
Display-adaptive
Compression
CompressiveOptics
via optimization
attenuation layers with
spacers
backlight
Layered 3D – SIGGRAPH 2011
?
• limited baseline tomography
• use algebraic approaches!
Layered 3D – SIGGRAPH 2011
This slide has a 16:9 media windowComputed Tomography (CT)
Image: W
ikip
edia
x-ray source
x-ray sensor
3D Reconstruction
Reconstructed 2D Slices
Tomographic Light Field Synthesis
q
2D light field
x
xq
backlight
attenuation volume
virtual planesimage formation
L(x,q ) = e- m (r )dr
cò
log L x,q( )( ) = - m(r)drc
ò
2
20
P)log(argmin
L
tomographic synthesis
Limits of Tomographic Light Field Synthesis
image formation
L(x,q ) = e- m (r )dr
cò
log L x,q( )( ) = - m(r)drc
ò
2
20
P)log(argmin
L
tomographic synthesis
log space
…
p1
p2
pK
…
p1
p2
pK
time
l = p1*…*pK
…
p1
p2
pK
log(l) = log(p1)+…+log(pK)
l = (p1*p2*…*pK)1+…+(p1*p2*…*pK)N
log
lin
???
backlight
rear layer
front layer
two-layer light field display
Low-rank Light Field Factorization
light field
fm(1)(x1)
fm(2)(x2)
x1
x2
L(x1, x2)
backlight
rear layer
front layer
two-layer light field display
fm(1)(x1)
fm(2)(x2)
x1
x2
L(x1, x2)
`
front layer
rear
laye
r
rank-1
Lanman et al. – SIGGRAPH Asia 2010
Low-rank Light Field Factorization
backlight
rear layer
front layer
two-layer light field display
fm(1)(x1)
fm(2)(x2)
x1
x2
L(x1, x2)
` rank-4
Lanman et al. – SIGGRAPH Asia 2010
high-speed LCDs = perceptual average
Low-rank Light Field Factorization
` rank-4F
G
L~
Lanman et al. – SIGGRAPH Asia 2010
Low-rank Light Field Factorization
high-speed LCDs = perceptual average
arg minF,G
L-FGW
2, for F,G ³ 0
objective function:
fm(1)(x1)
fm(2)(x2)
fm(3)(x3)
x1
x2
x3
L(x2, x3)
multi-layer light field display
backlight
rear layer
middle layer
front layer
light field
Light Field Slice Representation
fm(1)(x1)
fm(2)(x2)
fm(3)(x3)
x1
x2
x3
L(x1,x2,x3)
Rear
Layer
x3
x2
x1
L(x1,x2,x3)
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation
fm(1)(x1)
fm(2)(x2)
fm(3)(x3)
x1
x2
x3
Rear
Layer
x3
x2
x1
L(x1,x2,x3)
L(x1,x2,x3)
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation
fm(1)(x1)
fm(2)(x2)
fm(3)(x3)
Rear
Layer
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation
fm(1)(x1)
fm(2)(x2)
fm(3)(x3)
Rear
Layer
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation
fm(1)(x1)
fm(2)(x2)
fm(3)(x3)
Rear
Layer
light field tensor
backlight
rear layer
middle layer
front layer
light field
multi-layer light field display
Light Field Tensor Representation
frame M
+ ... +
Target Light Field Tensor
frame 1
nonnegative tensorfactorization (NTF)
Rank-M Approximation
perceptualintegration
frame 2
+
Low-rank Tensor Factorization
iterative update rules
nonlinear (multilinear)
optimization problem
Low-rank Tensor Factorization
• standard form – Tensor Compendium
• multiplicative update keep factors positive
• basically steepest descent with fixed step length
“standard” formulationalternate formulation
Low-rank Tensor Factorization
alternate formulation
forward projection (multiview rendering) back projection (projective texture mapping)
Efficient GPU Implementation
Tensor Factorization - Implementation
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
MN
O
C
U C
C
N
Who cares?
Vision-correcting Displayperceived image
displayed image
SIGGRAPH 2014 - Display Session, Tue morning
Vision-correcting Display
iPod Touch prototype printed transparency
SIGGRAPH 2014 - Display Session, Tue morning
prototype construction
300 dpi or higher
vision-correcting displayconventional display
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
MN
O
C
U C
C
N
What’s next?
resolution
contrast
3D display capabilities
Compressive Multi-mode DisplayOptics Express 2014
This slide has a 16:9 media window
High-speed LCD
High-speed LCD
Electronically-switchable Diffuser
High-speed LCD
High-speed LCD
Electronically-switchable Diffuser OFF
3D Display Mode
Light Field Factorization – LCD Patterns
Front Layer Rear Layer
Light Field Factorization – Results
High Dynamic Range Display Mode
High-speed LCD
High-speed LCD
Electronically-switchable Diffuser OFF
Target HDR Image
Solver = Light Field without Parallax
Front Layer Rear Layer
ConventionalHigh Dynamic Range
Superresolution Display Mode
High-speed LCD
High-speed LCD
Electronically-switchable Diffuser ON
Results from Prototype
Results from Prototype
Light Field Projection SIGGRAPH 2014 – ETech & Paper
Compressive Light Field Projection
multiplexing
nonlinear
compression
optimization
user experience
content
community
next
N
O
C
U C
C
N
Structural Formula for Compressive Displays
M
Mobile Displays Optics Express 2014
Projection Displays SIGGRAPH 2014
Coded Illumination for
Microscopy & LithographyHead Mounted Displays
CGF 2010, SIGGRAPH 2014
Monitors / TVs SIG2011,2012,2013, SIGAsia 2009,2011
Cool Displays at SIGGRAPH 2014
Technical Papers Sessions
Emerging Technologies
• Displays, Tuesday 10:45-12:15, Hall A
• Computational Sensing and Display, Tue 3:45-5:15, Hall A
• AR & VR Displays
• Light Field Projection, HDR Projection
• much more!
Gordon Wetzstein
Massachusetts Institute of Technology
media.mit.edu/~gordonw
displayblocks.org
Matt Hirsch (MIT)
Doug Lanman (MIT/NVIDIA/Oculus VR)
Andrew Maimone (UNC)
Felix Heide (UBC)
Fu-Chung Huang (UC Berkeley)
Belen Masia (University of Zaragoza)
collaborators
sources of funding & hardware
Wolfgang Heidrich (UBC/KAUST)
Ramesh Raskar (MIT)
Diego Gutierrez (University of Zaragoza)
Brian Barsky (UC Berkeley)
Henry Fuchs (UNC)
Gordon Wetzstein
Massachusetts Institute of Technology
media.mit.edu/~gordonw
displayblocks.org
Stanford looking for:
• students, postdocs, interns
• (industry) collaborators