Patch-based Image Patch-based Image Interpolation: Interpolation:
Algorithms and Algorithms and ApplicationsApplicationsXin LiXin Li
Lane Dept. of CSEELane Dept. of CSEE
West Virginia UniversityWest Virginia University
Where Does Patch Come Where Does Patch Come from?from?
Neuroscience: Neuroscience: receptive fields of receptive fields of neighboring cells in neighboring cells in human vision system human vision system have severe have severe overlappingoverlapping
Engineering: patch Engineering: patch has been under the has been under the disguise of many disguise of many different names such different names such as as windowswindows in digital in digital filters, filters, blocksblocks in JPEG in JPEG and the and the supportsupport of of wavelet bases, wavelet bases,
Cited from D. Hubel, “Eye, Brain and Vision”, 1988
Patch-based Image Patch-based Image ModelsModels
Local modelsLocal models Markov Random Field (MRF) and higher-order extensiMarkov Random Field (MRF) and higher-order extensi
ons (e.g., Field-of-Expert)ons (e.g., Field-of-Expert) Transform-based: PCA, DCT, waveletsTransform-based: PCA, DCT, wavelets
Nonlocal modelsNonlocal models Bilateral filtering (Tomasi et al. ICCV’1998)Bilateral filtering (Tomasi et al. ICCV’1998) Texture synthesis via Nonparametric resampling (EfroTexture synthesis via Nonparametric resampling (Efro
s&Leung ICCV’1999)s&Leung ICCV’1999) Exemplar-based inpainting (Criminisi et al. TIP’2004)Exemplar-based inpainting (Criminisi et al. TIP’2004) Nonlocal mean denoising (Buades et al.’ CVPR’200Nonlocal mean denoising (Buades et al.’ CVPR’200
5)5) Total Least-Square denoising (Hirakawa&Parks TIP’2Total Least-Square denoising (Hirakawa&Parks TIP’2
006)006) Block-matching 3D denoising (Dabov et al. TIP’2007)Block-matching 3D denoising (Dabov et al. TIP’2007)
A Bayesian Formulation of A Bayesian Formulation of Image Interpolation Image Interpolation
ProblemProblem
( | , ) ( | )( | , )
( | )
p H p Hp H
p H
y x xx y
y
Unobservabledata Observable
data
Image prior(e.g., sparsity-based)
Likelihood(our focus here)
Model class(e.g., local vs. nonlocal)
A Simple Extension of A Simple Extension of BM3DBM3D
1( [ ( ) | ])T S T x x
Hard thresholding3D transform of similar patches
Basic idea: combine BM3D with progressive thresholding (Guleryuz TIP’2006)
x y bicubic NEDI1 this work
28.70dB6 27.34dB 28.19dB
31.76dB 32.36dB 32.63dB
34.71dB 34.45dB 37.35dB
18.81dB 15.37dB 16.45dB
Interpolation of LR Interpolation of LR ImagesImages
1X. Li and M. Orchard, “New edge directed interpolation”, IEEE TIP, 2001
Go Back to Biology Go Back to Biology
Spatially random distribution of rod/cone cells keeps aliasing artifacts out of our vision
cone
rods
29.06dB 31.56dB 34.96dB
x y DT KR this work
28.46dB 31.16dB 36.51dB
17.90dB 18.49dB 29.25dB
26.04dB 24.63dB 29.91dB
Interpolation of Interpolation of Nonuniformly-sampled Nonuniformly-sampled
ImagesImages
DT- DelauneyTriangle-based(griddata under MATLAB)
KR- KernalRegression-based(Takeda et al.IEEE TIP 2007)
Modeling Spatial Modeling Spatial RandomnessRandomness
Extensively studied in geostatistics and Extensively studied in geostatistics and environmental statistics (e.g., spatial environmental statistics (e.g., spatial distribution of animals and plants)distribution of animals and plants)
Mathematically modeled by Mathematically modeled by homogeneous Poisson process (density homogeneous Poisson process (density parameterparameterλλ)) Lack of positional differentiationLack of positional differentiation Lack of scale differentiationLack of scale differentiation
Empirically there exist quadrant-based Empirically there exist quadrant-based and distance-based randomness metrics and distance-based randomness metrics
Monte-Carlo Based Monte-Carlo Based Optimization Optimization
Iterative procedure: randomly pick two locations (one black and the other white), ifswapping them decreases the energy, accept it; otherwise accept it with some probability
The lower energythe more random
Importance of Locations Importance of Locations
before optimization
after optimization
Identical reconstruction algorithm; only differ on sampling locations
In biological world:evolution + development
RandomSampling
Pattern
interpolationquantization channel
sensor node
S
Application intoApplication intoCompressive ImagingCompressive Imaging
How is it different from conventional image coding system? No bits are spent on coding the location information (random=no cost).
Coding ResultsCoding Results
R=0.21bpporiginal ours
PSNR=27.85dBSSIM=0.8750
SPIHT PSNR=28.82dBSSIM=0.8637
R=0.81bpporiginal ours
PSNR=28.10dBSSIM=0.9182
SPIHT PSNR=22.98dBSSIM=0.7512
Error Resilience ResultsError Resilience Results
ConclusionsConclusions
A good image prior is useful to many A good image prior is useful to many processing tasks involving incomplete or noisy processing tasks involving incomplete or noisy observationobservation
As we move from local to nonlocal models, the As we move from local to nonlocal models, the location of sampling points becomes important location of sampling points becomes important – “location (address) and intensity (data) are – “location (address) and intensity (data) are the same thing” cited from T. Kohonen “the same thing” cited from T. Kohonen “Self-Self-Organization and Associative MemoryOrganization and Associative Memory””
Image processing is at the intersection of Image processing is at the intersection of science and engineering- will BM3D lead to a science and engineering- will BM3D lead to a new class of SOM? new class of SOM?