eero simoncelli howard hughes medical institute, and new
TRANSCRIPT
![Page 1: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/1.jpg)
Perceptually-Driven Statistical Texture Modeling
Eero SimoncelliHoward Hughes Medical Institute, and
New York University
Javier PortillaUniversity of Granada, Spain
![Page 2: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/2.jpg)
What is “Visual Texture”?
Homogeneous, with repeated structures....
1/02
![Page 3: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/3.jpg)
What is “Visual Texture”?
Homogeneous, with repeated structures....
“You know it when you see it”
1/02
![Page 4: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/4.jpg)
Perceptual Texture Description
All Images
Texture Images
Equivalence class (visually indistinguishable)
Perceptual model:
• Set of texture images divided into equivalence classes (metamers)
• Perceptual “distance” between classes
1/02
![Page 5: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/5.jpg)
Julesz’s Conjecture (1962)
Hypothesis: two textures with identical Nth-order pixel statistics look the same(for some N).
• Explicit goal of capturing perceptual definition with a statistical model
• Statistical measurements should be:
– universal (for all textures)
– stationary (translation-invariant)
– a minimal set (necessary and sufficient)
• Julesz (and others) constructed counter-examples for N=2 and N=3, dis-missing the hypothesis...
1/02
![Page 6: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/6.jpg)
Julesz’s Conjecture, Revisited
Why did the early attempts fail?
• Right hypothesis, wrong model: A set of measurements equivalent to thevisual processes used for texture perception should satisfy the hypothesis.
• Lacked a powerful methodology for testing whether a model satisfies thehypothesis
• We can benefit from advances of the past few decades:
– scientific: better understanding of early vision
– engineering/mathematical: “wavelets”, statistical estimation, statisticalsampling
– technological: availability of powerful computers, digital images
1/02
![Page 7: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/7.jpg)
Testing a Texture Model
• As with most scientific test, we seek counter-examples
• Fundamental problem: we usually work with a small number of examples(tens or hundreds).
• Classification is an important application, but a weak test
• Synthesis can provide a much stronger test...
1/02
![Page 8: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/8.jpg)
Testing a Model via Synthesis
Example�����ture
Image
Random
Seed
Statistical
Image
Sampler
Statistical
Parameter
Estimator
Perceptual
Comparison
• Positive results are compelling, assuming:
– reference texture set contains a sufficient variety– statistical sampler generates “typical” examples
• Negative results are definitive: A single failure indicates insufficiency ofconstraints!
• Partial necessity test: remove a constraint and find a failure example
• Studying failures allows us to refine the model
1/02
![Page 9: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/9.jpg)
Methodological Ingredients
1. Representative set of example texture images: Brodatz, VisTex, our own
2. Method of estimating parameters: sample mean
3. Method of generating sample images from model: primary topic of thiswork
4. Perceptual test: informal viewing
1/02
![Page 10: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/10.jpg)
Iterative Synthesis Algorithm
Synthesis
Analysis
Transform MeasureStatistics
ExampleTexture
RandomSeed
SynthesizedTexture
Transform
MeasureStatistics
Adjust InverseTransform
Heeger & Bergen, ’95
1/02
![Page 11: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/11.jpg)
Transform: Steerable Pyramid
Example basis function Spectra
Linear basis: multi-scale, oriented, complex.
Basis functions are oriented bandpass filters, related by translation, dilation,rotation (directional derivatives, order K−1).
Tight frame, 4K/3 overcompleteness for K orientations.
Translation-invariant, rotation-invariant.
Motivation: image processing, computer vision, biological vision.
1/02
![Page 12: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/12.jpg)
Steerable Pyramid: Example Decomposition
Real part of coefficients complex magnitude of coefficients
Decomposition of a “disk” image
1/02
![Page 13: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/13.jpg)
Parameters: Marginal Statistics
Distribution of intensity values is captured with the first through fourth mo-ments of both the pixels and the lowpass coefficients at each pyramid scale.
Note: A number of authors have used marginal histograms:
Faugeras ’80 (pixels), Heeger & Bergen ’95 (wavelet), Zhu etal. ’96 (Gabor).
15 parameters
1/02
![Page 14: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/14.jpg)
Parameters: Spectral
Periodicity and globally oriented structure is best captured by frequency-domainmeasures (Francos, ’93).
Can be captured by autocorrelation measurements (included in most texturemodels).
In our model: central 7×7 region of the autocorrelation of each subband pro-vides a crude measure of spectral content within each subband.
125 parameters
1/02
![Page 15: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/15.jpg)
Parameters: Magnitude Correlation
Coefficient magnitudes are correlated both spatially and across bands. We cap-ture this with local autocorrelation and cross-correlation measurements.
472 parameters
1/02
![Page 16: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/16.jpg)
Parameters: Phase Correlation
Phases of complex responses at adjacent scales are aligned near image “fea-tures”.
We capture this using a novel measure of relative phase:
φ( f ,c) =c2 · f ∗
|c|,
where f is a fine-scale coefficient, c is a coarse-scale coefficient at the samelocation.
96 parameters
Total parameters: 708
1/02
![Page 17: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/17.jpg)
Phase Correlation Example
input
real/imag mag/phase real/imag mag/phase
coarse
realimag
phase mag x18
realimag
phase mag x18
fine
realimag
phase mag x18
realimag
phase mag x18
rphase
realimag
phase mag x20
realimag
phase mag x18
1/02
![Page 18: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/18.jpg)
Implementation
(high)
(low)
(mid)buildcomplexsteerablepyramid
imposeautoCorr
impose subbandstats & reconstruct(coarse-to-fine)
imposevariance
Gaussiannoise
+
imposeskew/kurt impose
pixelstatistics
synthetictexture
Each statistic, φk(~I), is imposed by gradient projection:
~I′ =~I +λk~∇φk(I), s.t. φk(~I) = mk,
where mk are the parameter values estimated from the example texture.
1/02
![Page 19: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/19.jpg)
Example Synthesis Sequence
Initial 1 4 64
We cannot prove convergence. But in practice, algorithm converges rapidly(typical: 50 iterations).
Run time: 256×256 image takes roughly 20 minutes (500 Mhz Pentium work-station, matlab code)
1/02
![Page 20: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/20.jpg)
Examples: Artificial
1/02
![Page 21: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/21.jpg)
Examples: Photographic, Quasi-periodic
1/02
![Page 22: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/22.jpg)
Examples: Photographic, Aperiodic
1/02
![Page 23: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/23.jpg)
Examples: Photographic, Structured
1/02
![Page 24: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/24.jpg)
Examples: Color
Color is incorporated by transforming to YIQ space, and including cross-bandmagnitude correlations in the parameterization.
1/02
![Page 25: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/25.jpg)
Examples: Non-textures?
1/02
![Page 26: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/26.jpg)
Necessity: Marginal Statistics
original with without
Needed for proper distribution of intensity values (at each scale).
1/02
![Page 27: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/27.jpg)
Necessity: Autocorrelation
original with without
Needed for capturing periodicity and global orientation.
1/02
![Page 28: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/28.jpg)
Necessity: Magnitude Correlation
original with without
Needed for capturing periodicity local structure.
1/02
![Page 29: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/29.jpg)
Necessity: Relative Phase
original with without
Needed for capturing details of local structure (edges vs. lines), and shading.
1/02
![Page 30: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/30.jpg)
Julesz Counter-Examples
Examples with identical 3rd-order pixel statistics
Left: Julesz ’78; Right: Yellott ’93
1/02
![Page 31: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/31.jpg)
Spatial Extrapolation
Modification: incorporate an additional projection operation in the synthesisloop, replacing central pixels by those of the original.
1/02
![Page 32: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/32.jpg)
Scale Extrapolation
Modification: incorporate an additional projection operation in the synthesisloop, replacing coarse-resolution coefficients by those of the original.
1/02
![Page 33: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/33.jpg)
Texture Mixtures
Modification: choose parameter vector that that is the average of those associ-ated with two example textures.
1/02
![Page 34: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/34.jpg)
Conclusions
• A framework for texture modeling, based on that originally proposed byJulesz
• New texture model:
– based on biologically-inspired statistical measurements
– includes methodology for testing
– provides heuristic methodology for refinement
– can be applied to a wide range of problems
Further information: http://www.cns.nyu.edu/∼lcv/texture
1/02
![Page 35: Eero Simoncelli Howard Hughes Medical Institute, and New](https://reader031.vdocuments.us/reader031/viewer/2022013001/61cbd5ff2abc25033c4afb03/html5/thumbnails/35.jpg)
To Do
• Adaptive front-end transformation (e.g., Zhu et al ’96, Manduchi & Portilla’99)
• Eliminate redundancy of parameterization
• Applications: compression, super-resolution, texture interpolation, texturepainting...
1/02