siggraph 2014 course on computational cameras and displays (part 3)

Compressive Display Systems

Gordon Wetzstein

MIT / Stanford University

media.mit.edu/~gordonw

displayblocks.org

This slide has a 16:9 media windowEvolution?

1928 2014

Nature

Image: National Geographic

Evolution!

uberpixel

Next-generation Devices

computation optics & electronics human visual

system

interaction

optics (compressive) computationsensing

Computational &

Compressive Displays

Computational

Imaging

displayblocks.org

Images: Wikipedia, Shinya Yoshioka

This slide has a 16:9 media windowNature

Image: Desafio Monteverde and Arenal Volcano Tours Costa Rica

Three-layer Tensor Display

This slide has a 16:9 media windowCompressive Light

Field Displays

This slide has a 16:9 media window

This slide has a 16:9 media window

viewer moves right

vie

wer

moves d

ow

n

4D Light Field

Display OpticsComputational

Processing

Compressive Displays

2D display

barrier

Parallax Barriers – Ives 1903

• low resolution & very dim

Integral Imaging – Lippmann 1908

• low-res, but brighter than parallax barriers

2D display

len

sle

ts

patent drawings - early 20th century

multiplexing

nonlinear

compression

optimization

user experience

content

community

next

MN

O

C

U C

C

N

Structural Formula for Compressive Displays

multiplexing

nonlinear

compression

optimization

user experience

content

community

next

MN

O

C

U C

C

N

Nonlinear Pixel Interaction & Coupling

layer 2

layer 1

Conventional Parallax Barriers

layer 2

layer 1

Conventional Parallax Barriers

blocked!

p1

p2

l = p1*p2

nonlinear

layer 2

layer 1

Most Volumetric Displays / Additive Layers

e.g., LEDs or transparent OLEDs

not blocked!

p1

p2

l = p1+p2

linear

Holography – Nonlinear Interaction

plane wave

hologram

emitted wavefront:

screen or retina

received intensity:

I(x) = T U(x){ }2

= Re T U(x){ }{ }2

+ Im T U(x){ }{ }2

U(x)

Holography – Coupling

plane wave

hologram

W (x,u = l sin(q )) = t x+x '

2

æ

èç

ö

ø÷t x -

x '

2

æ

èç

ö

ø÷e

2pix 'u dx 'ò

Fourier transform of all points interacting

with each other!

nonlinear interaction & pixel coupling!

attenuation layers with

spacers

backlight

Layered 3D – SIGGRAPH 2011

Layered 3D – SIGGRAPH 2011

layer 2

layer K

layer 1

…

Layered 3D – SIGGRAPH 2011

layer 2

layer K

layer 1

…

p1

p2

pK

l = p1*p2*…*pK

Layered 3D – SIGGRAPH 2011

…

layer 2

layer K

layer 1

Tensor Displays – SIGGRAPH 2012

directional backlight three layer

Reconfigurable

time

Tensor Displays – Directional Backlight

Perceptual Integration

layer 2

microlens array

layer 1

LCD with directional backlight, rank 6

LCD + directional BL

conventional lenslets

view from above

LCD with directional backlight, rank 6 (as seen by observer)

front LCD

directional backlight

Filmed with High-speed Camera

four stacked liquid crystal panels

two crossed polarizers

LCD 1

LCD 2

LCD 3

backlight

Polarization Fields – SIGGRAPH ASIA 2011

polarizer

polarizercolor filter array

polarizercolor filter array

polarizercolor filter array

backlight

Polarization Fields – SIGGRAPH ASIA 2011

LCD 1

LCD 2

LCD 3

polarizer

polarizer

image formation

f1

f2

f3

L(x,q) = sin2 Q(x,q)( )

backlight

Polarization Fields – SIGGRAPH ASIA 2011

LCD 1

LCD 2

LCD 3

multiplexing

nonlinear

compression

optimization

user experience

content

community

next

MN

O

C

U C

C

N

Compression

Which Light Field is More Compressible?

a b

Which Light Field is More Compressible?

a b

This slide has a 16:9 media window

4D Light Field

Uniform or

Directional BacklightStacked Layers

Nonnegative Tensor

Factorization

Display-adaptive

Compression

Compressive

Computed Tomography

(LCDs or Transparencies)

Optics

Observer = Decoder

Give those Pixels a Break!

Applied Mathematics

• sparse optimization

• low-rank factorization

• computed tomography

• …

Benefits for Optics & Electronics

• fewer pixels

• relaxation on refresh rate

• thinner form factors

• …

Parallax Barriers

1903

Time-Shifted

Parallax Barriers 2007

t

HR3D

SIG Asia 2010

t tLayered 3D

SIGGRAPH 2011

Tensor Displays

SIGGRAPH 2012

Conventional Parallax Barriers

layer 2

layer 1

From Conventional to Compressive Displays

…

…

Parallax Barriers

1903

Time-Shifted

Parallax Barriers 2007

t

HR3D

SIG Asia 2010

t tLayered 3D

SIGGRAPH 2011

Tensor Displays

SIGGRAPH 2012

…

time

Perceptual Integration

Tensor Displays – Multilayer & Directional Backlighting

From Conventional to Compressive Displays

Parallax Barriers

1903

Time-Shifted

Parallax Barriers 2007

t

HR3D

SIG Asia 2010

t tLayered 3D

SIGGRAPH 2011

Tensor Displays

SIGGRAPH 2012

Perceptual Integration

Tensor Displays – Directional Backlighting

time

thin

!

From Conventional to Compressive Displays

multiplexing

nonlinear

compression

optimization

user experience

content

community

next

MN

O

C

U C

C

N

Real-time Optimization

This slide has a 16:9 media window

4D Light Field

Display-adaptive

Compression

CompressiveOptics

via optimization

attenuation layers with

spacers

backlight

Layered 3D – SIGGRAPH 2011

?

• limited baseline tomography

• use algebraic approaches!

Layered 3D – SIGGRAPH 2011

This slide has a 16:9 media windowComputed Tomography (CT)

Image: W

ikip

edia

x-ray source

x-ray sensor

3D Reconstruction

Reconstructed 2D Slices

Tomographic Light Field Synthesis

q

2D light field

x

xq

backlight

attenuation volume

virtual planesimage formation

L(x,q ) = e- m (r )dr

cò

log L x,q( )( ) = - m(r)drc

ò

2

20

P)log(argmin

L

tomographic synthesis

Limits of Tomographic Light Field Synthesis

image formation

L(x,q ) = e- m (r )dr

cò

log L x,q( )( ) = - m(r)drc

ò

2

20

P)log(argmin

L

tomographic synthesis

log space

…

p1

p2

pK

…

p1

p2

pK

time

l = p1*…*pK

…

p1

p2

pK

log(l) = log(p1)+…+log(pK)

l = (p1*p2*…*pK)1+…+(p1*p2*…*pK)N

log

lin

???

backlight

rear layer

front layer

two-layer light field display

Low-rank Light Field Factorization

light field

fm(1)(x1)

fm(2)(x2)

x1

x2

L(x1, x2)

backlight

rear layer

front layer

two-layer light field display

fm(1)(x1)

fm(2)(x2)

x1

x2

L(x1, x2)

`

front layer

rear

laye

r

rank-1

Lanman et al. – SIGGRAPH Asia 2010

Low-rank Light Field Factorization

backlight

rear layer

front layer

two-layer light field display

fm(1)(x1)

fm(2)(x2)

x1

x2

L(x1, x2)

` rank-4

Lanman et al. – SIGGRAPH Asia 2010

high-speed LCDs = perceptual average

Low-rank Light Field Factorization

` rank-4F

G

L~

Lanman et al. – SIGGRAPH Asia 2010

Low-rank Light Field Factorization

high-speed LCDs = perceptual average

arg minF,G

L-FGW

2, for F,G ³ 0

objective function:

fm(1)(x1)

fm(2)(x2)

fm(3)(x3)

x1

x2

x3

L(x2, x3)

multi-layer light field display

backlight

rear layer

middle layer

front layer

light field

Light Field Slice Representation

fm(1)(x1)

fm(2)(x2)

fm(3)(x3)

x1

x2

x3

L(x1,x2,x3)

Rear

Layer

x3

x2

x1

L(x1,x2,x3)

light field tensor

backlight

rear layer

middle layer

front layer

light field

multi-layer light field display

Light Field Tensor Representation

fm(1)(x1)

fm(2)(x2)

fm(3)(x3)

x1

x2

x3

Rear

Layer

x3

x2

x1

L(x1,x2,x3)

L(x1,x2,x3)

light field tensor

backlight

rear layer

middle layer

front layer

light field

multi-layer light field display

Light Field Tensor Representation

fm(1)(x1)

fm(2)(x2)

fm(3)(x3)

Rear

Layer

light field tensor

backlight

rear layer

middle layer

front layer

light field

multi-layer light field display

Light Field Tensor Representation

fm(1)(x1)

fm(2)(x2)

fm(3)(x3)

Rear

Layer

light field tensor

backlight

rear layer

middle layer

front layer

light field

multi-layer light field display

Light Field Tensor Representation

fm(1)(x1)

fm(2)(x2)

fm(3)(x3)

Rear

Layer

light field tensor

backlight

rear layer

middle layer

front layer

light field

multi-layer light field display

Light Field Tensor Representation

frame M

+ ... +

Target Light Field Tensor

frame 1

nonnegative tensorfactorization (NTF)

Rank-M Approximation

perceptualintegration

frame 2

+

Low-rank Tensor Factorization

iterative update rules

nonlinear (multilinear)

optimization problem

Low-rank Tensor Factorization

• standard form – Tensor Compendium

• multiplicative update keep factors positive

• basically steepest descent with fixed step length

“standard” formulationalternate formulation

Low-rank Tensor Factorization

alternate formulation

forward projection (multiview rendering) back projection (projective texture mapping)

Efficient GPU Implementation

Tensor Factorization - Implementation

multiplexing

nonlinear

compression

optimization

user experience

content

community

next

MN

O

C

U C

C

N

Who cares?

Vision-correcting Displayperceived image

displayed image

SIGGRAPH 2014 - Display Session, Tue morning

Vision-correcting Display

iPod Touch prototype printed transparency

SIGGRAPH 2014 - Display Session, Tue morning

prototype construction

300 dpi or higher

vision-correcting displayconventional display

multiplexing

nonlinear

compression

optimization

user experience

content

community

next

MN

O

C

U C

C

N

What’s next?

resolution

contrast

3D display capabilities

Compressive Multi-mode DisplayOptics Express 2014

This slide has a 16:9 media window

High-speed LCD

High-speed LCD

Electronically-switchable Diffuser

High-speed LCD

High-speed LCD

Electronically-switchable Diffuser OFF

3D Display Mode

Light Field Factorization – LCD Patterns

Front Layer Rear Layer

Light Field Factorization – Results

High Dynamic Range Display Mode

High-speed LCD

High-speed LCD

Electronically-switchable Diffuser OFF

Target HDR Image

Solver = Light Field without Parallax

Front Layer Rear Layer

ConventionalHigh Dynamic Range

Superresolution Display Mode

High-speed LCD

High-speed LCD

Electronically-switchable Diffuser ON

Results from Prototype

Results from Prototype

Light Field Projection SIGGRAPH 2014 – ETech & Paper

Compressive Light Field Projection

multiplexing

nonlinear

compression

optimization

user experience

content

community

next

N

O

C

U C

C

N

Structural Formula for Compressive Displays

M

Mobile Displays Optics Express 2014

Projection Displays SIGGRAPH 2014

Coded Illumination for

Microscopy & LithographyHead Mounted Displays

CGF 2010, SIGGRAPH 2014

Monitors / TVs SIG2011,2012,2013, SIGAsia 2009,2011

Cool Displays at SIGGRAPH 2014

Technical Papers Sessions

Emerging Technologies

• Displays, Tuesday 10:45-12:15, Hall A

• Computational Sensing and Display, Tue 3:45-5:15, Hall A

• AR & VR Displays

• Light Field Projection, HDR Projection

• much more!

Gordon Wetzstein

Massachusetts Institute of Technology

media.mit.edu/~gordonw

displayblocks.org

Matt Hirsch (MIT)

Doug Lanman (MIT/NVIDIA/Oculus VR)

Andrew Maimone (UNC)

Felix Heide (UBC)

Fu-Chung Huang (UC Berkeley)

Belen Masia (University of Zaragoza)

collaborators

sources of funding & hardware

Wolfgang Heidrich (UBC/KAUST)

Ramesh Raskar (MIT)

Diego Gutierrez (University of Zaragoza)

Brian Barsky (UC Berkeley)

Henry Fuchs (UNC)

Gordon Wetzstein

Massachusetts Institute of Technology

media.mit.edu/~gordonw

displayblocks.org

Stanford looking for:

• students, postdocs, interns

• (industry) collaborators