andrew nealen and marc alexa, discrete geometric modeling group, tu darmstadt, 2003 hybrid texture...

Andrew Nealen and Marc Alexa, Discrete Geometric Modeling Group, TU Darmstadt, 2003

Hybrid Texture SynthesisHybrid Texture Synthesis

Andrew NealenAndrew NealenMarc AlexaMarc Alexa

Discrete Geometric Modeling Group (DGM)Discrete Geometric Modeling Group (DGM)Technische Universität DarmstadtTechnische Universität Darmstadt


2D Texture Synthesis

nxm Input Texture

NxM Output Texture

► The goal: Synthesize an output texture which is perceptually similar to the input texture. Also ensure that the result contains sufficient variation.


Our Inspiration

► Existing (and Impressive) Technology• Pixel-Based:

─ Non-parametric Sampling [Efros and Leung 1999]─ Tree-structured Vector Quantization [Wei and Levoy 2000]─ Synthesizing Natural Textures [Ashikhmin 2001]─ Image Analogies [Hertzmann et al. 2001]

• Patch-Based:─ Patch-Based Sampling [Guo et al. 2001]─ Image Quilting [Efros and Freeman 2001]─ Hierarchical Pattern Mapping [Soler et al. 2002]


Our Inspiration

► Possible Drawbacks ?• Loss of global structure• Loss of scale• Boundary mismatch• Blurring


Our Inspiration


Wei/Levoy Algorithm:Can occur when using structured Textures with rich histograms.This is due to the L2 norm.


Our Inspiration


Patch-Based Sampling Algorithm:Can occur when texture features are not well-aligned across patch boundaries


Our Inspiration


► Is it possible to reduce all deficiencies for an even larger class of textures ?


Pixel-Based Texture Synthesis

Input Output

Step n in Algorithm



Input Output

Build L-shaped Neighborhood



Input Output

Find Matching Neighborhood(s) in Input



Input Output

Select Best Neighborhood from all Candidates



Input Output

Copy Selected Pixel to Output



Input Output

Repeat for all Pixels in this Row...



Input Output

...and all other Output Rows



► Main difference between Efros/Leung (EL), Wei/Levoy (WL) and Ashikhmin (AS): Per-Pixel Search Strategy.

► Ashikhmin’s insight: “WL tends to produce textures with smoothed out edges in some cases.”

► Ashikhmin’s solution: consider only coherent pixels in the search, which encourages region growing.

► Problem: inconsistencies can still occur for highly-structured textures.


Patch-Based Texture Synthesis

Input Output

Step n in Algorithm



Input Output

Patch to be Synthesized in this Step



Input Output

Build Neighborhood (Image Mask)



Input Output

Find Matching Neighborhood(s) in Input



Input Output

Select Best Neighborhood from all Candidates



Input Output

Copy Selected Patch to Output



Input Output

Apply some Overlap Repair Strategy ...



Input Output

... And we are done with this patch



Input Output

Repeat for all Patches in Output


Patch-Based Texture SynthesisGeneralization of Pixel-Based Synthesis

► Main (visual) difference between Image Quilting (IQ) and Patch-based Sampling (PBS): Overlap Repair Strategy.

► IQ performs a minimum-error-boundary-cut (MEBC) through the error surface (least cost path; Dijkstra).

► PBS feathers the overlap region (alpha-blending).

► Pro: patch-based algorithms are good at maintaining global structure.

► Problems: boundary mismatch (MEBC, top) and blurring (PBS, bottom).


Hybrid Texture Synthesis (HTS)

Algorithm

► Hybrid, two-fold approach to texture synthesis• Adaptive patch sampling [Soler et al. 2002]

─ Start with a uniform quadrilateral grid of patches (the output)─ Recursively split a patch if the best fit taken from the input

texture is not good enough (a user-defined tradeoff max)

• Overlap re-synthesis (novel overlap repair strategy)─ Mark invalid pixels in the overlap region

─ Validity map is defined by the error surface E(x)=(xsrc – xdst)2

and a user-defined tolerance parameter max

─ Define an ordering for the invalid pixels and re-synthesize them using a per-pixel strategy


Adaptive Patch SamplingPrinciple

► Essentially: Hierarchical Pattern Mapping applied to the plane [Soler et al. 2002]

► Analogy: Assembling a quadtree puzzle

p > max Split PatchSynthesize White Patch p < max


Adaptive Patch SamplingTrade-off

► Trade-off between preserving global structure and avoiding detail artifacts

max = 1.0max = 0.01 max = 0.04


In Search of Good PatchesSynthesizing a single patch

Synthesize Black Patch i


In Search of Good PatchesExtracting the image mask

Synthesize Black Patch i

► Grow patch by overlap (toroidally)


In Search of Good PatchesExtracting the image mask

Image Mask (Ii)Synthesize Black Patch i Binary Support (Ji)

► Extract image mask (Ii) and binary support (Ji), and circularly shift to upper left corner


In Search of Good PatchesComputing the error image

Image Mask (Ii)Input Texture (T) Binary Support (Ji)

► Compute error Ei(x0) between Ii and T for each circular shift x0=(x,y) of the input texture T


In Search of Good PatchesComputing the error image

Error Image (Ei)Input Texture (T)

x

xxxxxc

cicicii

i TIwJκ

E 20,,0 ))()(()(

1)(


In Search of Good PatchesSelecting a Patch

Error Image (E)Input Texture (T) Selected Patch (P)

x

xxxxxc

cicicii

i TIwJκ

E 20,,0 ))()(()(

1)(


In Search of Good PatchesSelecting a Patch

Error Image (E)Input Texture (T) Selected Patch (P)

► Note: error image can be computed efficiently in the Fourier domain. Complexity: O(n log n) per patch.


Overlap Re-synthesisOverlap error and pixel invalidation

Patch and Image Mask Selected Patch (P)



Image Mask (I) Selected Patch (P)

c

cicicii PIwJS 2,, ))()(()()( xxxx



Image Mask (I) Selected Patch (P)Error Surface (S)

c

cicicii PIwJS 2,, ))()(()()( xxxx



Patch and Image Mask P with Invalid Pixels (blue)Error Surface (S)

► Trade-off: user-defined max is the pixel error tolerance. Setting to 1 results in pure feathering.


Overlap Re-synthesisCompositing

Intermediate Result Patch with Invalid Pixels

P OVER Ri-1


Overlap Re-synthesisCompositing

Composited Result Patch with Invalid Pixels

► Simple scanline re-synthesis: possibly insufficient causal neighborhood within valid pixel region

► Solution: alternative ordering (Pixel Traversal Map)


Overlap Re-synthesisPixel Traversal Map (Ordering)

Composited Result Patch with Invalid PixelsPixel Traversal Map (M)

► Pixel Traversal Map: repeated morphological dilation of the binary support for valid pixels with a euclidian disk of radius 1 (city-block distance transform).




► Pixel Traversal Map: step 1




► Pixel Traversal Map: step 2 ...




► Pixel Traversal Map: ... step n




► Art Restorer Analogy: stepwise restoration of the hole from the boundary of the existing image.


Overlap Re-synthesisPer-Pixel Re-synthesis

Pixel Traversal Map (M)

► Synthesize red pixel from valid (or already re-synthesized) pixels

Intermediate Result



Image Mask (Ij)

Binary Support (Jj)

► Extract image mask (Ij) and binary support (Jj) for best-pixel search in the input texture. Complexity: O(n log n) per pixel.

Intermediate Result



Intermediate Result Pixel Traversal Map (M)



Intermediate Result Shifted




Intermediate Result with Traversal Map




Overlap Re-synthesisStep 1




Overlap Re-synthesisResult



ResultsOverlap Repair Comparisons

No Repair HTSPBSIQ


ResultsSynthesis Comparisons

Input

Efros/Leung Wei/Levoy

IQ PBS HTS


ResultsSynthesis Comparisons

Input

PBS HTS

PBS HTS

Input


Results


Conclusions and Future Work

► Improve Computational Complexity• Pixel neighborhoods of patch and pixel stage are not

known a priori, so precomputation is not straightforward• We can solve this in the pixel stage by employing k-

coherence search [Tong et al. 2002] [Ashikhmin 2001]

► Improve Error Metric• Still using the L2 norm due to its simplicity

• Develop a metric which takes feature mismatch into account

• Texton approach looks promising [Zhang et al. 2003]


Questions ?

► Contact Information

Andrew [email protected]

Marc [email protected]

http://www.dgm.informatik.tu-darmstadt.de

Matlab code and extended Thesis:http://www.nealen.com

andrew nealen and marc alexa, discrete geometric modeling group, tu darmstadt, 2003 hybrid texture...

Documents

tu darmstadt

output texture

algorithm slide

input slide

output slide

texture features

d texture synthesis

shaped neighborhood