patch-based image interpolation: algorithms and applications

Patch-based Image Patch-based Image Interpolation: Interpolation:

Algorithms and Algorithms and ApplicationsApplicationsXin LiXin Li

Lane Dept. of CSEELane Dept. of CSEE

West Virginia UniversityWest Virginia University

Where Does Patch Come Where Does Patch Come from?from?

Neuroscience: Neuroscience: receptive fields of receptive fields of neighboring cells in neighboring cells in human vision system human vision system have severe have severe overlappingoverlapping

Engineering: patch Engineering: patch has been under the has been under the disguise of many disguise of many different names such different names such as as windowswindows in digital in digital filters, filters, blocksblocks in JPEG in JPEG and the and the supportsupport of of wavelet bases, wavelet bases,

Cited from D. Hubel, “Eye, Brain and Vision”, 1988

Xin Li

This slide describes the scientific basis of using patches as the units of modeling images: human vision system processes the stimuli through overlapping receptive fields;and engineering concepts of patch: it has appeared in many different forms.

Patch-based Image Patch-based Image ModelsModels

Local modelsLocal models Markov Random Field (MRF) and higher-order extensiMarkov Random Field (MRF) and higher-order extensi

ons (e.g., Field-of-Expert)ons (e.g., Field-of-Expert) Transform-based: PCA, DCT, waveletsTransform-based: PCA, DCT, wavelets

Nonlocal modelsNonlocal models Bilateral filtering (Tomasi et al. ICCV’1998)Bilateral filtering (Tomasi et al. ICCV’1998) Texture synthesis via Nonparametric resampling (EfroTexture synthesis via Nonparametric resampling (Efro

s&Leung ICCV’1999)s&Leung ICCV’1999) Exemplar-based inpainting (Criminisi et al. TIP’2004)Exemplar-based inpainting (Criminisi et al. TIP’2004) Nonlocal mean denoising (Buades et al.’ CVPR’200Nonlocal mean denoising (Buades et al.’ CVPR’200

5)5) Total Least-Square denoising (Hirakawa&Parks TIP’2Total Least-Square denoising (Hirakawa&Parks TIP’2

006)006) Block-matching 3D denoising (Dabov et al. TIP’2007)Block-matching 3D denoising (Dabov et al. TIP’2007)

Xin Li

Althoug not a common view, it is possible to interpret various image models under a patch-based framework. The main difference between local and nonlocal models lies in the Markovian assumption they made: is it in the domain or the range? Such range-domain duality is the basis for bilateral filtering (arguably the first nonlocal model).

A Bayesian Formulation of A Bayesian Formulation of Image Interpolation Image Interpolation

ProblemProblem

( | , ) ( | )( | , )

( | )

p H p Hp H

p H

y x xx y

y

Unobservabledata Observable

data

Image prior(e.g., sparsity-based)

Likelihood(our focus here)

Model class(e.g., local vs. nonlocal)

Xin Li

This talk is more about likelihood term than image prior (since I am just using BM3D to regularize the reconstruction).

A Simple Extension of A Simple Extension of BM3DBM3D

1( [ ( ) | ])T S T x x

Hard thresholding3D transform of similar patches

Basic idea: combine BM3D with progressive thresholding (Guleryuz TIP’2006)

Xin Li

It might be fair to mention you guys' ICIP2007 work though the targeting application is different (I did not do anything like stochastic approximation here).

x y bicubic NEDI1 this work

28.70dB6 27.34dB 28.19dB

31.76dB 32.36dB 32.63dB

34.71dB 34.45dB 37.35dB

18.81dB 15.37dB 16.45dB

Interpolation of LR Interpolation of LR ImagesImages

1X. Li and M. Orchard, “New edge directed interpolation”, IEEE TIP, 2001

Xin Li

This slides leads to the motivational observation about the limitation of uniform sampling - despite severe aliasing in the last example, the reconstructed image is visually very convincing.

Go Back to Biology Go Back to Biology

Spatially random distribution of rod/cone cells keeps aliasing artifacts out of our vision

cone

rods

Xin Li

If human eyes also do uniform sampling, we will have tremendous difficulty with understanding the world.

29.06dB 31.56dB 34.96dB

x y DT KR this work

28.46dB 31.16dB 36.51dB

17.90dB 18.49dB 29.25dB

26.04dB 24.63dB 29.91dB

Interpolation of Interpolation of Nonuniformly-sampled Nonuniformly-sampled

ImagesImages

DT- DelauneyTriangle-based(griddata under MATLAB)

KR- KernalRegression-based(Takeda et al.IEEE TIP 2007)

Xin Li

Nothing new here - just confirm nonuniform sampling could work better with our reconstruction algorithm.

Modeling Spatial Modeling Spatial RandomnessRandomness

Extensively studied in geostatistics and Extensively studied in geostatistics and environmental statistics (e.g., spatial environmental statistics (e.g., spatial distribution of animals and plants)distribution of animals and plants)

Mathematically modeled by Mathematically modeled by homogeneous Poisson process (density homogeneous Poisson process (density parameterparameterλλ)) Lack of positional differentiationLack of positional differentiation Lack of scale differentiationLack of scale differentiation

Empirically there exist quadrant-based Empirically there exist quadrant-based and distance-based randomness metrics and distance-based randomness metrics

Xin Li

If you have heard about the the Waitomo glowworm caves on Lake Roturura in New Zealand, you would know what spatial randomness means - in God's will.

Monte-Carlo Based Monte-Carlo Based Optimization Optimization

Iterative procedure: randomly pick two locations (one black and the other white), ifswapping them decreases the energy, accept it; otherwise accept it with some probability

The lower energythe more random

Xin Li

This is the standard Monte-Carlo optimization. Runs not very fast but can be done offline.

Importance of Locations Importance of Locations

before optimization

after optimization

Identical reconstruction algorithm; only differ on sampling locations

In biological world:evolution + development

Xin Li

I am also a little surprised by the gain achieved by randomness optimization. If 6dB is what animal survival requires, evolution surely finds its way to implement the optimization.

RandomSampling

Pattern

interpolationquantization channel

sensor node

S

Application intoApplication intoCompressive ImagingCompressive Imaging

How is it different from conventional image coding system? No bits are spent on coding the location information (random=no cost).

Xin Li

Encoder has little complexity; while decoder does most job. It is a paradigm shift from redundancy reduction to redundancy exploitation.

Coding ResultsCoding Results

R=0.21bpporiginal ours

PSNR=27.85dBSSIM=0.8750

SPIHT PSNR=28.82dBSSIM=0.8637

R=0.81bpporiginal ours

PSNR=28.10dBSSIM=0.9182

SPIHT PSNR=22.98dBSSIM=0.7512

Xin Li

Only some preliminary results, but good enough to show the competence of new coding system.

Error Resilience ResultsError Resilience Results

Xin Li

When partial samples are received (due to channel errors), we can still obtain a reconstructed image with graceful quality degradation.

ConclusionsConclusions

A good image prior is useful to many A good image prior is useful to many processing tasks involving incomplete or noisy processing tasks involving incomplete or noisy observationobservation

As we move from local to nonlocal models, the As we move from local to nonlocal models, the location of sampling points becomes important location of sampling points becomes important – “location (address) and intensity (data) are – “location (address) and intensity (data) are the same thing” cited from T. Kohonen “the same thing” cited from T. Kohonen “Self-Self-Organization and Associative MemoryOrganization and Associative Memory””

Image processing is at the intersection of Image processing is at the intersection of science and engineering- will BM3D lead to a science and engineering- will BM3D lead to a new class of SOM? new class of SOM?

Xin Li

I don't know how many in audience know Prof. Kohonen in person. But to me, he is one of my idols - a true pioneer in the field of neural networks.

patch-based image interpolation: algorithms and applications

Documents