video textures arno schödl richard szeliski david salesin irfan essa microsoft research, georgia...
TRANSCRIPT
Video TexturesVideo Textures
Arno SchödlRichard Szeliski
David SalesinIrfan Essa
Microsoft Research, Georgia Tech
Still photosStill photos
Video clipsVideo clips
Video texturesVideo textures
Related workRelated work
Image-based rendering
• Novel still views from still images
• View Morphing [Seitz 96]
• Lightfields / Lumigraph [Levoy 96, Gortler 96]
• Façade [Debevec 96]
• Virtualized Reality [Kanade 97]
• many more…
Image-based rendering (IBR)Image-based rendering (IBR)
• From View Morphing by Seitz and Dyer
Slide added by D.Brogan
Related workRelated work
Texture synthesis
• Infinite extension of 2D pattern
• [Heeger 95]
• [de Bonet 97]
• [Efros 99]
Related workRelated work
Temporal texture synthesis
• Video as a spatio-temporal texture
• Statistical Learning of Multidimensional Textures [Bar-Joseph 99][Wei 00]
Related workRelated work
Video Rewrite
• Assemble video snippets to generate lip motion
• [Bregler 97]
Periodic motion analysis
• Uses similar analysis tools
Related workRelated work
• [Niyogi 94]
• [Seitz 97]
• [Polana 97]
• [Cutler 00]
Problem statementProblem statement
video clip video texture
Our approachOur approach
How do we find good transitions?
Finding good transitions Finding good transitions
Compute L2 distance Di, j between all frames
Similar frames make good transitions
frame ivs.
frame j
Markov chain representationMarkov chain representation
2 3 41
Similar frames make good transitions
Transition costs Transition costs
Transition from i to j if successor of i is similar to j
Cost function: Cij = Di+1, j
i
j
i+ 1
j-1
i j D i+ 1, j
Transition probabilitiesTransition probabilities
Probability for transition Pij inversely related
to cost:Pij ~ exp ( – Cij / 2 )
high low
Preserving dynamicsPreserving dynamics
Preserving dynamics Preserving dynamics
Preserving dynamics Preserving dynamics
Cost for transition ij
Cij = wk Di+k+1, j+kk = -N
N -1
i
j j+ 1
i+ 1 i+ 2
j-1j-2
i jD i, j - 1 D Di+ 1, j i+ 2, j+ 1
i-1
D i-1, j-2
Preserving dynamics – effect Preserving dynamics – effect
Cost for transition ij
Cij = wk Di+k+1, j+kk = -N
N -1
2 3 41
Dead endsDead ends
No good transition at the end of sequence
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
Fij = Cij + mink Fjk
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
Fij = Cij + mink Fjk
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
Fij = Cij + mink Fjk
2 3 41
Future costFuture cost
• Propagate future transition costs backward
• Iteratively compute new cost
Fij = Cij + mink Fjk
2 3 41
• Propagate future transition costs backward
• Iteratively compute new cost
Fij = Cij + mink Fjk
• Q-learning
Future costFuture cost
Future cost – effectFuture cost – effect
Fixed-length videosFixed-length videos
• Alternative to random transitions
• Precompute set of loops up front
Slide edited by D.Brogan
Fixed-length videosFixed-length videos
• Creating fixed-length videos• Enumerate all sets of transitions with a total
length L (multiloops)• Select the ones that are legal (the loops
overlap so video can jump from one to another)
• Find the best one
• Exponential in number of transitions
Slide added by D.Brogan
Fixed-length videosFixed-length videos
• Create an L x N table
• L = length of longest loop being considered
• N = number of loops being considered
Slide added by D.Brogan
Fixed-length videosFixed-length videos
• Populate the table
• For each (loopn, lengthl) tuple, find best loop (possibly compound) of length l that includes at least one instance of loop n
Slide added by D.Brogan
Fixed-length videosFixed-length videos
• Populate the table with dynamic programming
• Start from row one, find best loops
• Proceed through the rows
• (length5, loopC) is found from (length3, loopC) + (length2, loopD)
Slide added by D.Brogan
Fixed-length videosFixed-length videos
• Each cell requires inspection of at most L-1 compound loops from same column and N-1 entries from other columns
• Total Cost: O(L2N2)
Slide added by D.Brogan
Fixed-length videosFixed-length videos
• Organizing the loops
• You cannot jump from any loop to any other loop… sequence order matters
• Start with loop with largest last frame num.
• Remove it from the pool
• Identify isolated loops
• Add loops that require first loop for access
Slide added by D.Brogan
Visual discontinuitiesVisual discontinuities
• Problem: Visible “Jumps”
Crossfading Crossfading
• Solution: Crossfade from one sequence to the other.
A i-2
A i-2
B j-2
15
…
…
3
1 2
2 1
3
4
4 4
4 4
4+ + +
A i-1
A i-1 / B j-2 A i-1 / B j-2 A i-1 / B j-2
B j-1
A i
B j
A i+1
B j+1
B j+1
Morphing Morphing
• Interpolation task:
A25 B2
5 C15+ +
Morphing Morphing
• Interpolation task:
• Compute correspondencebetween pixels of all frames
• Optical flow
A25 B2
5 C15+ +
Slide edited by D.Brogan
Morphing Morphing
• Interpolation task:
• Compute correspondence between pixels of all frames
• Interpolate pixel position andcolor in morphed frame
• based on [Shum 2000]
A25 B2
5 C15+ +
Results – crossfading/morphingResults – crossfading/morphing
Results – crossfading/morphingResults – crossfading/morphing
Jump Cut Crossfade Morph
CrossfadingCrossfading
Frequent jump & crossfadingFrequent jump & crossfading
Video portraitVideo portrait
Useful for web pages
Combine with IBR techniques
Video portrait – 3DVideo portrait – 3D
Region-based analysisRegion-based analysis
• Divide video up into regions
• Generate a video texture for each region
Automatic region analysisAutomatic region analysis
User selects target frame range
User-controlled video texturesUser-controlled video textures
slow variable fast
Lengthen / shorten video without affecting speed
Time warpingTime warping
shorter original longer
Video-based animationVideo-based animation
• Like spritescomputer games
• Extract spritesfrom real video
• Interactively control desired motion
©1985 Nintendo of America Inc.
Video sprite extractionVideo sprite extraction
blue screen m attingand velocity estim ation
C i j = + angle C i j
vector tom ouse pointer
S im ilarity term Contro l term
velocity vector
Animation{ {
Video sprite controlVideo sprite control
• Augmented transition cost:
F i j
F i j
F i j F i j
F i j
F i jF i j
S W
W
N W
N
N E
E
S E
S
G oal
Video sprite controlVideo sprite control
• Need future cost computation
• Precompute future costs for a few angles.
• Switch between precomputed angles according to user input
• [GIT-GVU-00-11]
Interactive fishInteractive fish
Summary Summary
• Video clips video textures
• define Markov process
• preserve dynamics
• avoid dead-ends
• disguise visual discontinuities
Summary Summary
• Extensions
• regions
• external constraints
• video-based animation
Discussion Discussion
• Some things are relatively easy
Discussion Discussion
• Some are hard
Future workFuture work
• Better analysis of complex motions
• New optimality metrics
• More powerful video-based animation
• Other kinds of video-based rendering
A final exampleA final example