Download - Overview of Texture Synthesis
Overview of Texture Synthesis
Ganesh Ramanarayanan
Cornell Graphics Seminar
The Problem
small texture
???
big texture
A Simple Solution …
small texture
???
big texture
???
big texture
A Simple Solution … Tiling
small texture
A Simple Solution … Tiling
Tiling is obvious!big texture?
small texture
Ideal Solution?
The Problem … in Words
Given texture I, generate a texture J which– Looks like the same texture– Has no obvious copying or tiling from I– Difference between I and J should be the same as
the way I “differs from itself” [DeBonet97]
Things to watch for:– ‘Looks the same’: what is the texture model?– ‘Obvious copying’: how is it avoided?– Underlined text: indicates algorithm parameter
Classes of Algorithms
Multiresolution pyramids– [HeegerBergen95]
Pixel-by-pixel synthesis– [EfrosLeung99]
Multiresolution pixel-by-pixel– [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01],
[Ashikhmin01] Patch quilting
– [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04] Geometric feature matching
– [WuYu04], [Liu et.al. 04]
Heeger Bergen 1995
Seminal paper that introduced texture synthesis to the graphics community
Algorithm:– Initialize J to noise– Create multiresolution pyramids for I and J– Match the histograms of J’s pyramid levels with I’s
pyramid levels– Loop until convergence– Can be generalized to 3D
Heeger Bergen 1995 - Algorithm
Image pyramids– Gaussian– Laplacian
Steerable pyramids [SimoncelliFreeman95]
– b): multiple scales of oriented filters
– c): a sample image– d): results of filters in b)
applied to c)
Heeger Bergen 1995 - Results
I J
Successes Failures
I J
Heeger Bergen 1995 - Results
Heeger Bergen 1995 - Verdict
Texture model:– Histograms of responses to various filters
Avoiding copying:– Inherent in algorithm
No user intervention required Captures stochastic textures well Does not capture structure
– Lack of inter-scale constraints
De Bonet 1997
Propagate constraints downwards by matching statistics all the way up the pyramid
Feature vector: multiscale collection of filter responses for a given pixel
Algorithm:– Initialize J to empty image– Create multiresolution pyramids for I and J– For each pixel in level of J, randomly choose pixel
from corresponding level of I that has similar feature vector
De Bonet 1997 - Algorithm
6 feature vectors shown
Notice how they share parent information
De Bonet 1997 - Results
De Bonet 1997 - Verdict
Texture model:– Feature vector containing multiscale responses to
various filters
Avoiding copying:– Random choice of pixels with ‘close’ feature vectors,
but copying still frequent on small scale
Individual per-filter thresholds are cumbersome Feature vectors used in later synthesis work
Efros Leung 1999
Markov Random Fields: pixels synthesized probabilistically based on causal neighborhood
Algorithm:– Initialize J to empty– Synthesize new pixels in a spiral pattern– Select new pixel from I based on conditional
probabilities given the current neighborhood in J and similar neighborhoods in I
Efros Leung 1999 - Algorithm
I J
‘causal’neighborhood
Efros Leung 1999 - Results
Efros Leung 1999 – Verdict
Texture model:– MRF
Avoiding copying:– MRF
Neighborhood size = largest feature size Markov model is surprisingly good
– “I spent an interesting evening recently with a grain of salt.”
Search is very slow with large neighborhoods
Wei Levoy 2000
Adds tree-structured vector quantization speedup and multiresolution matching
Algorithm:– Initialize J to noise– Synthesize new pixels in scanline order– Select new pixel from I that has the closest
matching neighborhood to J– In multiresolution case, look instead at
neighborhood of feature vectors
Wei Levoy 2000 - Algorithm
Scanline synthesis with neighborhoods
– a) input texture– b) algorithm start– c) midway point– d) ending
Wei Levoy 2000 - Results
Wei Levoy 2000 - Verdict
Texture model:– Multiscale filters and MRF
Avoiding copying:– Not explicitly, but in practice inherently avoided;
TSVQ adds additional unpredictability
Results comparable to [EfrosLeung99] TSVQ speedup comes at a price (quality loss)
Ashikhmin 2001
Adds more verbatim image copying and user-constrained synthesis
Algorithm:– New array J* stores location in I of J’s pixels– Initialize J to empty, J* to random– Synthesize new pixels in scanline order– Select new pixel by copying from patches of I, as
determined by locations in J*– In user-constrained case, also include error
comparison with user-specified target image T
Ashikhmin 2001 - Algorithm
J*
x=3y=4
x=4y=4
x=12y=11
x=17y=16
Candidates:-x=4, y=5-x=18, y=16-x=11, y=12
x=18y=16
Ashikhmin 2001 - Results
Ashikhmin 2001 - Verdict
Texture model:– MRF
Avoiding copying:– Actually, here it is encouraged on a small scale, but
in practice it doesn’t occur on a large scale
Recognized that copying preserves fine detail Features can get ‘cut off’ because of the abrupt
jumping from patch to patch
Hertzmann et. al. 2001
Framework for solving image analogies: learning an image filter f by example– Given A, f(A), and B, generate f(B)
To handle texture synthesis, ignore A, B, and let f(A) = I, f(B) = J
Algorithm:– Bring image histograms into correspondence– Combine [WeiLevoy00] and [Ashikhmin01]
Each search method is weighted appropriately and then the best choice is used
Hertzmann et. al. 2001 - Algorithm
A provides set of constraints
Histogram matching critical because of color differences between A and B sets of images
Hertzmann et. al. 2001 - Results
I J I J
Hertzmann et. al. 2001 - Results
Hertzmann et. al. 2001 - Verdict
Texture model:– Multiscale filters and MRF
Avoiding copying:– See [WeiLevoy00] and [Ashikhmin01]
Unique approach to image manipulation and texture synthesis
Same fundamental limitations as [WeiLevoy00] and [Ashikhmin01]
Efros Freeman 2001
Inspired by chaos mosaics [Xu et. al. 00] Make an ‘image quilt’ by pasting together
patches of the input and fixing seams Algorithm:
– Initialize J to empty– Copy new patches of certain size in scanline order,
with fixed overlap width– Randomly select new patch that has error in overlap
region less than given threshold– Copy pixels based on minimal error boundary cut
Efros Freeman 2001 - Algorithm
Input texture
block
B1 B2
Neighboring blocksconstrained by overlap
B1 B2
Minimal errorboundary cut
Efros Freeman 2001 - Results
Efros Freeman 2001 - Verdict
Texture model:– ??
Avoiding copying:– Randomized patch selection, but still noticeable
Patch size is a hard parameter to understand Results are surprisingly good given algorithm Multiscale goes on a brief hiatus
Kwatra et. al. 2003
Generalizes seam computation in overlap regions as a graph cut problem– Based on [Boykov et. al. 99] (with Ramin Zabih)
Algorithm:– Initialize J to empty– Copy pieces of I to J using a variety of methods– Formulate flow graph in overlap region based on
errors and compute minimum cut– Copy sink-side pixels to J– Variety of strategies to further hide seams
Kwatra et. al. 2003 - Algorithm
(assume cut region is 3x3 for simplicity)
Kwatra et. al. 2003 - Results
Kwatra et. al. 2003 - Results
Kwatra et. al. 2003 - Verdict
Texture model:– MRF
Avoiding copying:– Even with a multitude of patch selection methods,
still noticeable when it happens repeatedly
Paper presents a bag of synthesis tricks without much intuition for when to use what
Graph cut formalization is useful and powerful
Wu Yu 2004
Synthesizes a feature image J* in parallel Tries to match features in overlap region Algorithm:
– Compute feature image I*– Initialize J* to empty– Synthesize J and J* in parallel– When searching for new patch, add in weighted
feature matching error– Prior to copying, warp selected patch to match
features exactly
Wu Yu 2004 - Algorithm
Match feature pixels based on
– Tangent orientation– Distance– Bijectivity
Example:– L1
out matches L1in
– L2out matches L3
in
– Warping will move matching features closer to each other
Feature correspondence
Wu Yu 2004: Results
Wu Yu 2004: Verdict
Texture model:– Feature lines and MRF
Avoiding copying:– No; in fact, mirroring is often very noticeable
Unclear how to combine this metric with others Insight to use warping is valuable, although the
next paper does a much better job with it
Liu et. al. 2004
Regular textures: can be tiled Near-regular textures: statistical distortion of a
regular texture Algorithm (simplified):
– Mark lattice points in I– Transform I into regular texture R– Express transformation as deformation texture I*– Synthesize larger deformation J*– Synthesize J by combining J* with R
Liu et. al. 2004 - Algorithm
RI
Deformation Texture I*
Liu et. al. 2004 - Results
Liu et. al. 2004 - Results
original texture transfer
Liu et. al. 2004 - Verdict
Texture model:– Regular texture with modulated geometry and color
Avoiding copying:– Depends on the synthesis algorithm
Proposes specific model to handle specific types of textures
Interestingly, this method depends on previous texture synthesis methods
Classes of Algorithms - Recap
Multiresolution pyramids– [HeegerBergen95]
Pixel-by-pixel synthesis– [EfrosLeung99]
Multiresolution pixel-by-pixel– [DeBonet97], [WeiLevoy00], [Hertzmann et.al. 01],
[Ashikhmin01] Patch quilting
– [EfrosFreeman01], [Kwatra et.al. 03], [WuYu04] Geometric feature matching
– [WuYu04], [Liu et.al. 04]
Evolution of Texture Model
Responses to image filters / decompositions Markov Random Fields Copies of small pieces of image Feature data Various combinations of the above
What’s Next?
Texture synthesis is really hard Rather than creating a silver bullet, focus on
algorithms for more specific problems Superresolution:
+ = ???
What’s Next?
Texture synthesis is really hard Rather than creating a silver bullet, focus on
algorithms for more specific problems Superresolution:
+ =
[Hertzmann et. al. 01]