texture optimization for example-based synthesis vivek kwatra irfan essa aaron bobick nipun kwatra

Texture Optimization for Example-based Synthesis

Vivek Kwatra

Irfan EssaAaron BobickNipun Kwatra

Image vs. Texture

Example-based Texture Synthesis

Input Example

Related Work

• Local region-growing method

-Pixel-based

-Patch-based

• Global optimization-based method Heeger and Bergen sig95,Pyramid-based texture synthesis

Paget and Longstaff IEEE Tran… 98,Texture synthesis via a noncausal nonparametric multiscale markov random field

…..

• Physical Simulation et al

Example-based Texture Synthesis

1. Dynamic texturing (animation/video)

2. Greater control: external knobs, e.g., flow,

shape, illumination

3. Texture quality evaluation and refinement

Motivation

+

InputTexture

InputFlow Field

OutputSequence

Limitations of previous work

• Pixel-based

can not maintain global structure of the texture

inconsistencies

• Patch-based

have no control over individual pixel values

• Global methods:

pixel-to-pixel interactions

insufficient to capture large scale structures

complex formulations difficult to optimize

Texture Energy

• Pixel-by-pixel comparison of source and target not possible

• Compare texture “elements”– Local pixel neighborhoods

• Want each target neighborhood to be similar to some source neighborhood

Texture Energy

X(target frame)

Z(source)

Energy for Single Neighborhood

p (pixel)Z(source)

X(target frame)

Energy for Single Neighborhood

Xp (neighborhood)Z(source)

X(target frame)

Energy for Single Neighborhood

Xp (neighborhood)Z(source)

X(target frame)

Energy for Single Neighborhood

(nearest neighbor) Zp

Xp (neighborhood)Z(source)

X(target frame)

Energy for Single Neighborhood

X

Z

(nearest neighbor) Zp

Xp (neighborhood)

Texture Energy(single neighborhood)

2

t P P PE X X Z

Texture Energy for Entire Image

X

Z

Texture Energy for Entire Image

X

Z

individual neighborhood energy

2;t P P P

P X X

E X Z X Z

4.Repeat iteratively until set {ZP} stops changing

3.Update X to be texture that minimizes the energy

Texture Synthesis

1.Initialize texture X

2

t P P PE X X Z

2.Find closest input neighborhood ZP corresponding to each output neighborhood XP

2;t P P P

P X X

E X Z X Z

Texture Synthesis

Gradient-based Energy

D is the differentiation operator and µ is a relative weighting coefficient

Muti-level Synthesis

• Multi-resolution

• Multi-scale/multiple neighborhood sizes

Reasons a good initialization of the texture avoid undesirable local minima avoid inconsistencies capture large scale structures

• Random Initialization

• Multiple– Resolution Levels– Neighborhood

Sizes

• Progressively refined output

Texture Energy Graph

Level 1

Level 2

Level 3

Level 4

Level 5

Level 6

Level 7

0 5 10 15 20 25 30 35 40 45 500

50

100

150

200

250

300

350

400

Number of Iterations

Controllable Synthesis

2; c c

ck

E X X X k X k

; ; ct P cE X E X Z E X X

• Flow consistency– Perceived motion similar to flow

• Texture similarity– Shape, size, orientation of texture

elements similar to source

Flow-guided Texture Animation

Target FlowFlowing Target

Source Texture

TextureSimilarity

FlowConsistency

Naive Approach

Target Flow

• Ignore Texture Similarity

• Warp each frame via flow field

X1 Xn

TargetFrames X0

Warp

Flowing Target

FlowConsistency

Source Texture

TextureSimilarity

Source Texture

TextureSimilarity

Naive Approach

Texture structure not maintained

Target FlowFlowing Target

FlowConsistency

Source Texture

TextureSimilarity

Warp

Instead of…

WarpXnX0 X1

TargetFrames

Warp

Instead of…

WarpXnX0 X1

TargetFrames

Warp + Correct

“Correct” Operation

TargetFrames XX–

Warp

W

Correct

WarpedFrame

SynthesizedFrame

Link to original goals

“Correct” Operation

XTargetFrames X–

W

Correct

Warp

Link to original goals

“Correct” Operation

XTargetFrames X–

W

Correct

Flow ConsistencyWarp

Link to original goals

“Correct” Operation

XTargetFrames X–

W

Correct

Source

Flow Consistency

Texture Similarity

Warp

Energy = Flow Energy + Texture Energy

Optimization of Energy

XTargetFrames X–

W

Correct

Source

Flow Consistency

Texture Similarity

Warp

Flow Energy

1 2 1, , , Lf f f f

2D flow field

:if frame i frame i+1

pixel (p,i) pixel (q,i+1)

2

1: 1 ,

; , , 1 ,i

c ii L p i X

E X f X p i X q i q f p

Results: Texture Synthesis

Results: Texture Synthesis

Input Wei-Levoy[Wei’00]

Image Quilting[Efros’01]

Graph-cuts[Kwatra’03]

Texture Optimization

[Kwatra’05]

Time Cost

• Static texture synthesis– 5-10 iterations per resolution level

• Flowing texture synthesis– 20-60 seconds per frame

Resolution One iteration Total time

64x64 2.5 sec 20 sec

128x128 10 sec 2 min

256x256 25 sec 8.5 min

Limitations

• Optimization finds local minima

• Blurry / misaligned texture elements

• Texture scale should be compatible

with flow field

Thanks a lot!

Pixel-based Methods

• Compare local causal neighbourhoods Efros and Leung (ICCV ’99)

Wei and Levoy (Siggraph 2000)Ashikhmin (I3D 2001)

Input Output

Patch-based Methods

• Copy patches of pixels rather than single pixels

Chaos Mosaic, Xu et al, 1997

Patch-Based Sampling, Liang et al(ACM 2001)

Image Quilting, Efros and Williams(Siggraph 2001)

Back