single image blind deconvolution
DESCRIPTION
Single Image Blind Deconvolution. Presented By: Tomer Peled & Eitan Shterenbaum. Agenda. Problem Statement Introduction to Non-Blind Deconvolution Solutions & Approaches Image Deblurring PSF Estimation using Sharp Edge Prediction / Neel Joshi et. Al. MAP x,k Solution Analysis - PowerPoint PPT PresentationTRANSCRIPT
Single Image Blind Deconvolution
Presented By:Tomer Peled
&Eitan Shterenbaum
Agenda
1. Problem Statement2. Introduction to Non-Blind Deconvolution3. Solutions & Approaches
A. Image Deblurring PSF Estimation using Sharp Edge Prediction / Neel Joshi et. Al.
B. MAPx,k Solution AnalysisUnderstanding and evaluating blind deconvolution algorithms / Anat Levin et. Al.
C. Variational Method MAPkRemoving Camera Shake from a Single Photograph / Rob Fergus et. Al
4. Summary2
Problem statement
• Blur = Degradation of sharpness and contrast of the image, causing loss of high frequencies.
• Technically - convolution with certain kernel during the imaging process.
3
Camera Motion blur
4
Defocus blur
5
Defocus blur
6
Defocus blur
7
Blur – generative model
=
=
Point Spread Function
Optical Transfer Functionfft(Image)
Sharp image Blured image
fft(Blured image)
Object Motion blur
9
Local Camera Motion
10
Depth of field – Local defocus
11
Lucy Richardson
Evolution of algorithms
?
Camera motion blur
Simple kernels
Non blind deconvolution
12
1972
Wiener1949
Joshi2008
Shan2008
Volunteers ?
Fergus2006
Introduction to Non-Blind Deconvolution
blur kernelblurred image sharp image
Deconvolution Evolution:
Simple no-Noise Case
Noise Effect Over Simple Solution
Wiener Deconvolution RL Deconvolution
noise
13
Simple no-Noise Case:
BlurreBlurredd
ImageImage
RecoveredRecovered
14
Noisy case:
15
Original (x) Blured + noise (y) Recovered x
Original signalOriginal signalFT of original signalFT of original signal
Convolved signals w/o noiseConvolved signals w/o noiseFT of convolved signalsFT of convolved signals sd
Reconstructed FT of the Reconstructed FT of the signalsignal
High Frequency Noise Amplified16
Noisy case, 1D Example:
Noisy SignalOriginal Signal
Wiener Deconvolution
Blurred noisy Blurred noisy imageimage
Recovered imageRecovered image
18
Non Blind Iterative Method : Richardson –Lucy Algorithm
Assumptions: Blurred image yi~P(yi), Sharp image xj~P(xj) i point in y, j point in x
Target: Recover P(x) given P(y) & P(y|x)
From Bayes theorem Object distribution can be expressed iteratively:
Richardson, W.H., “Bayesian-Based Iterative Method of Image Restoration”, J. Opt. Soc. Am., 62, 55, (1972).Lucy, L.B., “An iterative technique for the rectification of observed distributions”, Astron. J., 79, 745, (1974).
19
where
Richardson-Lucy ApplicationSimulated Multiple Star
measurement PSF Identification reconstruction of 4th Element
20
Solution Approaches
A. Image Deblurring PSF Estimation using Sharp Edge PredictionNeel Joshi Richard Szeliski David J. Kriegman
B. MAPx,k Solution AnalysisUnderstanding and evaluating blind deconvolution algorithmsAnat Levin, Yair Weiss, Fredo Durand, William T. Freeman
C. Variational Method MAPkRemoving Camera Shake from a Single PhotographRob Fergus, Barun Singh, Aaron Hertzmann, Sam T. Roweis, William T. Freeman
23
PSF Estimation by Sharp Edge Prediction
Given edge steps, debluring can be reduced to Kernel Optimization
Suggested in PSF Estimation by Sharp Edge Prediction \ Neel Joshi et. el. in
Select Edge Step (Masking)
Estimate Blurring Kernel
Recover Latent Image
24
PSF Estimation by Sharp Edge Prediction - Masking
Original Image Edge Prediction Masking
Min
Max
Valid Region
25
Masking, Cont.Which is Best the Signals?
Edge
Impulse
Original Blurred
26
Blurr Model: y=x*k+n, n ~ N(0,σ2)
Bayseian Framework: P(k|y) = P(y|k)P(k)/P(y)
Map Model:argmaxk P(k|y) = argmink L(y|k) + L(k)
PSF Estimation by Sharp Edge Prediction – PSF Estimation
27
PSF Estimation by Sharp Edge Prediction – Recovery
Recovery through Lucy-Richardson Iterations given the PSF kernel
28
Blurred Recovered
PSF Estimation by Sharp Edge, Summary & Improvements
1. Handle RGB Images – perform processing in parallel
2. Local Kernel Variations:Sub divide image into sub-image units
Limitations:– Highly depends on the quality of the edge detection– Requires Strong Edges in multiple orientations for
proper kernel estimation– Assumes knowledge of noise error figure.
29
blur kernel
MAPx,k , Blind Deconvolution Definition
blurred image sharp image
noise
Input (known)
Unknown, need to estimate
?
?Courtesy of Anat Levin CVPR 09 Slides30
MAPx,k Cont. - Natural Image Priors
Derivative histogram from a natural image
Parametric models
Derivative distributions in natural images are sparse:
Log
prob
xx
Gaussian:
-x2
Laplacian:
-|x||-x|0.5
|-x|0.25
Courtesy of Anat Levin CVPR 09 Slides31
Naïve MAPx,k estimation
Given blurred image y,
Find a kernel k and latent image x minimizing:
Should favor sharper x explanations
Convolution constraint
Sparse prior
Courtesy of Anat Levin CVPR 09 Slides32
The MAPx,k paradox
P( , )>P ),( Let be an arbitrarily large image sampled from a sparse prior , and
Then the delta explanation is favored
Latent imagekernel
Latent imagekernel
Courtesy of Anat Levin CVPR 09 Slides33
?
The MAPx,k failure sharp blurred
Courtesy of Anat Levin CVPR 09 Slides34
The MAPx,k failure
Red windows = [ p(sharp x) >p(blurred x) ]
15x15 windows 25x25 windows 45x45 windows
simple derivatives
-]1,1-],[1;1[
FoE filters
)Roth&Black(
35
P(blurred step edge)
sum of derivatives: cheaper
The MAPx,k failure - intuition
P(blurred impulse) P(impulse)
sum of derivatives:
cheaper
>P(step edge)
>
k=[0.5,0.5]
Courtesy of Anat Levin CVPR 09 Slides36
P(blurred real image)
MAPx,k Cont. - Blur Reduces Derivative Contrast
Noise and texture behave as impulses - total derivative contrast reduced by blur
>P(sharp real image)
cheaper
Courtesy of Anat Levin CVPR 09 Slides37
MAPx,k Reweighting Solution
Alternating Optimization Between x & k
Minimization term:
MAPx,k
High Quality Motion Debluring From Single Image / Shan et al.
39
MAPx,k Reweighting - Blurred
39
4040
MAPx,k Reweighting - Recovered
Solution Approaches
A. Image Deblurring PSF Estimation using Sharp Edge PredictionNeel Joshi Richard Szeliski David J. Kriegman
B. MAPx,k Solution AnalysisUnderstanding and evaluating blind deconvolution algorithmsAnat Levin, Yair Weiss, Fredo Durand, William T. Freeman
C. Variational Method MAPkRemoving Camera Shake from a Single PhotographRob Fergus, Barun Singh, Aaron Hertzmann, Sam T. Roweis, William T. Freeman
47
MAPk estimation
Given blurred image y, Find a kernel minimizing:
Again, Should favor sharper x explanations
Convolution constraint
Sparse prior Kernel prior
48
Superiority of MAPk over MAPk,x
Toy Problem : y=kx+n
The joint distribution p(x, k|y). Maximum for x → 0, k → ∞.
p(k|y) produce optimum closer to true k .∗
uncertainty of p(k|y) reduces given multiple observations yj =kxj + nj .
49
Evaluation on 1D signals
MAPk variational approximation (Fergus et al.)
Exact MAPk MAPx,kFavors delta solution
MAPk Gaussian prior
Favor correct solution despite
wrong prior!
Courtesy of Anat Levin CVPR 09 Slides50
Intuition: dimensionality asymmetry
MAPx,k– Estimation unreliable. Number of measurements always lower than number of unknowns: #y<#x+#k
MAPk – Estimation reliable. Many measurements for large images: #y>>#k
Large, ~105 unknowns Small, ~102 unknowns
blurred image ykernel k
sharp image x
~105 measurements
Courtesy of Anat Levin CVPR 09 Slides51
Courtesy of Rob Fergus Slides52
Three sources of information
Courtesy of Rob Fergus Slides53
Image prior p(x)
Courtesy of Rob Fergus Slides55
Blur prior p(b)
Courtesy of Rob Fergus Slides56
The obvious thing to do
Courtesy of Rob Fergus Slides57
Variational Bayesian approach
Courtesy of Rob Fergus Slides58
Variational Bayesian methods
• Variational Bayesian = ensemble learning, • A family of techniques for approximating intractable
integrals arising in Bayesian inference and machine learning. • Lower bound the marginal likelihood (i.e. "evidence") of
several models with a view to performing model selection.
59
Setup of Variational Approach
Ensemble Learning for Blind Source Separation / J.W. Miskin , D.J.C.
MacKay
Small synthetic
blurs
large real world blurs
Cartoon images
Gradients of natural images
Independent Factor Analysis \ H. AttiasAn introduction to variational methods for graphical models \ JORDAN M. et al.
61
63
64
65
Courtesy of Rob Fergus Slides66
Example 1
67
Output 1
68
Example 2
69
Output 2
70
Example 3
71
Output 3
72
Achievements
• Work on real world images• Deals with large camera motions
(up to 60 pixels)• Getting close to practical generic solution
of an old problem .
73
Limitations• Targeted at camera motion blur
– No in plane rotation– No motion in picture– Out of focus blur
• Manual input– Region of Interest– Kernel size & orientation– Other parameters e.g. scale offset, kernel TH & 9 other semi-fixed
parameters
• Sensitive to image compression, noise(dark images) & saturation
• Still contains artifacts (solvable by upgrading from Lucy Richardson)
74
Evaluation
Cumulative histogram of deconvolution successes:
bin r = #{ deconv error > r }
MAPk, Gaussian prior
Shan et al. SIGGRAPH08Fergus, variational MAPk
MAPx,k sparse prior
100
80
60
40
20
Su
cces
ses
per
cen
t
75
Summary
MethodQuasi-MAPK
JoshiReweighted MAPKX
ShanVariational MAPk
Fergus
Distortion modelDefocus blursimple PSF
Camera motion blurComplex sparse PSF
Camera motion blurComplex sparse PSF
Region of interestEdge regionEdge regionUser selected
Optimization modelQuasi-MAPKMAPKXVariational Bayes for K estimation (MAPk equivalent)
Degrees of freedomO(K)O(K+X)O(K+Xprior+PRIOR)
SchemeGradient based least squares
Alternating iterativeMultiscale iterative(internal altering)
78
Debluring is underconstrained
Debluring single image under constrained
problem
?Blured imageRecovered image
Recovered kernel
Priors do the trick
?Blured image
Image prior
Recovered kernel
Kernel marginalization
?Blured image
Recovered kernel
Image prior
Back to non-blind deconvolution
?Recovered image Blured image
Recovered kernel
Existing challenges and potential research
• Robustness to user’s parameters & initial priors
• Solutions to spatially varying kernels
84
Thank You Eitan & Tomer
The End