image super-resolution using sparse representation by: michael elad single image super-resolution...
Post on 19-Dec-2015
232 views
TRANSCRIPT
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Single Image Super-Resolution Using Sparse Representation
Michael EladThe Computer Science DepartmentThe Technion – Israel Institute of technologyHaifa 32000, Israel
MS45: Recent Advances in Sparse and Non-local Image Regularization - Part III of
III Wednesday, April 14, 2010
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
*
* Joint work with Roman Zeyde and Matan Protter
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
The Super-Resolution Problem
2
hz y v SH
hyz
Blur (H) and
Decimation (S)
+WGN
v
Our Task: Reverse the process – recover the high-resolution image from the
low-resolution one
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Single Image Super-Resolution
3
The reconstruction process should rely on:The given low-resolution imageThe knowledge of S, H, and statistical properties of v,
and Image behavior (prior).
Recovery Algorithm
hyz ?
In our work: We use patch-based sparse and redundant
representation based prior, and We follow the work by Yang, Wright, Huang, and Ma
[CVPR 2008, IEEE-TIP – to appear], proposing an improved algorithm.
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Core Idea (1) - Work on Patches
4
z
hy
Interp.Q
y
Enhance Resolution
Every patch from should go through a process of resolution enhancement. The improved patches are then merged together into the final image (by averaging).
y
y zQ
We interpolate the low-res. Image in order to align the coordinate systems
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Core Idea (2) – Learning [Yang et. al. ’08]
5
Enhance Resolution?
We shall perform a sparse decomposition of the low-res. patch, w.r.t. a learned dictionary A
We shall construct the high res. patch using the same sparse representation, imposed on a dictionary
hA
and now, lets go into the details …
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
The Sparse-Land Prior [Aharon & Elad, ’06]
6
hy Extraction of a patch from y
h in location k is
performed by hk k hp yRn
n
Model Assumption: Every such patch can be represented sparsely over the dictionary Ah:
Position k
hk k
hk h k k 0
p q such that
p q and q nA
m
n Akq
hkp
h
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Low Versus High-Res. Patches
7
hy
n
n
Position k
z
InterpolationQ
y
hk k 2
p p L
L = blur + decimation + interpolation. This expression relates the low-res.
patch to the high-res. one, based on
hz y v SH
hkp
n
n
kp?
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Low Versus High-Res. Patches
8
hk h kp qAh
k k 2p p L
We interpolate the low-res. Image in order to align the coordinate systems
Interpolation
z
hy
n
n
Position ky
n
n
h k k 2q p LA qk is ALSO the
sparse representation of the low-resolution patch,
with respect to the dictionary LAh.
A
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Training the Dictionaries – General
9
Training pair(s)
Interpolation
hk
k k
p
p
We obtain a
set of matching
patch-pairs
Pre-process(low-res)
Pre-process(high-res)
k kp
hk k
p These will be
used for the
dictionary training
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Alternative: Bootstrapping
10
z
The given image to be scaled-up
hx z vSH
xBlur (H)
and Decimation
(S)+WGN
v
Simulate the degradation process
Use this pair of images to generate the patches for the
training
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Pre-Processing High-Res. Patches
11
z
x
Interpolation
-
High-Res
Low-Res
hk k
p
Patch size: 9×9
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Pre-Processing Low-Res. Patches
12
x
Interpolation
Low-Res
* [0 1 -1]
* [0 1 -1]T
* [-1 2 -1]* [-1 2 -1]T
We extract patches of size 9×9 from each of these images, and concatenate them to form one vector of length 324
k kp
Dimensionality Reduction
By PCA
k kp
Patch size: ~30h h
k kp pB
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Training the Dictionaries:
13
A
K-SVD Am=1000
N=30
40 iterations, L=3
k k
pFor an image of size 1000×1000 pixels, there are ~12,000 examples to train on
kk k k 02q
min p q s.t. q LA
Given a low-res. Patch to be scaled-up, we start the resolution enhancement by sparse coding it, to find qk:
Remember
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Training the Dictionaries:
14
hA
kk k k 02q
min p q s.t. q LA
Given a low-res. Patch to be scaled-up, we start the resolution enhancement by sparse coding it, to find qk:
Remember
And then, the high-res. Patch is obtained by h
k h kp qA
Thus, Ah should be designed such that
h
2hk h k 2
k
p q minAA
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Training the Dictionaries:
15
hA
h
2hk h k 2
k
p q minAA
However, this approach disregards the fact that the resulting high-res. patches are not used directly as the final result, but averaged due to overlaps between them.
A better method (leading to better final scaled-up image) would be – Find Ah such that the following error is minimized:
h h
Tk
22
h h h kh2k 2
ˆmin y y min y y qA A
R AW
The constructed image
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Overall Block-Diagram
16
Training the low-res. dictionary
zA
Training the high-res. dictionary
hA
The recovery algorithm
(next slide)
On-line or off-line
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
The Super-Resolution Algorithm
17
z
Interpolation
y
multiply by B to reduce dimension
* [0 1 -1]
* [0 1 -1]T
* [-1 2 1]
* [-1 2 1]T
Extract patches
k kp
Sparse coding using Aℓ to compute
kq
k k
pMultiply by Ah and combine by averaging the overlaps
hy
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Relation to the Work by Yang et. al.
18
Training the low-res. dictionary
zA
Training the high-res. dictionary
hA
The recovery algorithm
(previous slide)
We can train on the given image or a training set
We reduce dimension prior
to training
We train using the K-SVD algorithm
We use a direct approach to the
training of Ah
We train on a difference-
image Bottom line: The proposed algorithm is much simpler,
much faster, and, as we show next, it also leads to better
results
We use OMP for sparse-
coding
We avoid post-processing
We interpolate to avoid coordinate
ambiguity
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Results (1) – Off-Line Training
19
The training image: 717×717 pixels, providing a set of 54,289 training patch-pairs.
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
20
Results (1) – Off-Line Training
Ideal Image
Given Image
SR ResultPSNR=16.95dB
Bicubic interpolation PSNR=14.68
dB
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
21
Results (2) – On-Line Training
Given image
Scaled-Up (factor 2:1) using the proposed algorithm, PSNR=29.32dB (3.32dB improvement over bicubic)
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
22
Results (2) – On-Line Training
The Original Bicubic Interpolation SR result
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
23
Results (2) – On-Line Training
The Original Bicubic Interpolation SR result
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Comparative Results – Off-Line
24
Our alg. Yang et. al. Bicubic
SSIM PSNR SSIM PSNR SSIM PNSR
0.78 26.77 0.76 26.39 0.75 26.24 Barbara
0.66 27.12 0.64 27.02 0.61 26.55 Coastguard
0.82 33.52 0.80 33.11 0.80 32.82 Face
0.93 33.19 0.91 32.04 0.91 31.18 Foreman
0.88 33.00 0.86 32.64 0.86 31.68 Lenna
0.79 27.91 0.77 27.76 0.75 27.00 Man
0.94 31.12 0.93 30.71 0.92 29.43 Monarch
0.89 34.05 0.87 33.33 0.87 32.39 Pepper
0.91 25.22 0.89 24.98 0.87 23.71 PPT3
0.84 28.52 0.83 27.95 0.79 26.63 Zebra
0.85 30.04 0.83 29.59 0.81 28.76 Average
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
25
Comparative Results - Example
Yang et. al.Ours
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Summary
26
Single-image scale-up –
an important
problem, with many
attempts to solve it in
the past decade.
The rules of the game:
use the given image,
the known degradation,
and a sophisticated
image prior.
Yang et. al. [2008] – a
very elegant way to
incorporate sparse &
redundant
representation prior
We introduced
modifications to Yang’s
work, leading to a simple,
more efficient alg. with
better results. More work is required to
improve further the results
obtained.
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
Michael Elad
A New Book
Image Super-Resolution Using Sparse RepresentationBy: Michael Elad
27
SPARSE AND REDUNDANT REPRESENTATIONS
From Theory to Applications in Signal
and Image Processing
Michael Elad
Thank You Very Much !!