single image blind deconvolution

Single Image Blind Deconvolution

Presented By:Tomer Peled

&Eitan Shterenbaum

Agenda

1. Problem Statement2. Introduction to Non-Blind Deconvolution3. Solutions & Approaches

A. Image Deblurring PSF Estimation using Sharp Edge Prediction / Neel Joshi et. Al.

B. MAPx,k Solution AnalysisUnderstanding and evaluating blind deconvolution algorithms / Anat Levin et. Al.

C. Variational Method MAPkRemoving Camera Shake from a Single Photograph / Rob Fergus et. Al

4. Summary2

Problem statement

• Blur = Degradation of sharpness and contrast of the image, causing loss of high frequencies.

• Technically - convolution with certain kernel during the imaging process.

3

Camera Motion blur

4

Defocus blur

5

Defocus blur

6

Defocus blur

7

Blur – generative model

=

=

Point Spread Function

Optical Transfer Functionfft(Image)

Sharp image Blured image

fft(Blured image)

Object Motion blur

9

Local Camera Motion

10

Depth of field – Local defocus

11

Lucy Richardson

Evolution of algorithms

?

Camera motion blur

Simple kernels

Non blind deconvolution

12

1972

Wiener1949

Joshi2008

Shan2008

Volunteers ?

Fergus2006

Introduction to Non-Blind Deconvolution

blur kernelblurred image sharp image

Deconvolution Evolution:

Simple no-Noise Case

Noise Effect Over Simple Solution

Wiener Deconvolution RL Deconvolution

noise

13

Simple no-Noise Case:

BlurreBlurredd

ImageImage

RecoveredRecovered

14

Noisy case:

15

Original (x) Blured + noise (y) Recovered x

Original signalOriginal signalFT of original signalFT of original signal

Convolved signals w/o noiseConvolved signals w/o noiseFT of convolved signalsFT of convolved signals sd

Reconstructed FT of the Reconstructed FT of the signalsignal

High Frequency Noise Amplified16

Noisy case, 1D Example:

Noisy SignalOriginal Signal

Wiener Deconvolution

Blurred noisy Blurred noisy imageimage

Recovered imageRecovered image

18

Non Blind Iterative Method : Richardson –Lucy Algorithm

Assumptions: Blurred image yi~P(yi), Sharp image xj~P(xj) i point in y, j point in x

Target: Recover P(x) given P(y) & P(y|x)

From Bayes theorem Object distribution can be expressed iteratively:

Richardson, W.H., “Bayesian-Based Iterative Method of Image Restoration”, J. Opt. Soc. Am., 62, 55, (1972).Lucy, L.B., “An iterative technique for the rectification of observed distributions”, Astron. J., 79, 745, (1974).

19

where

Richardson-Lucy ApplicationSimulated Multiple Star

measurement PSF Identification reconstruction of 4th Element

20

Solution Approaches

A. Image Deblurring PSF Estimation using Sharp Edge PredictionNeel Joshi Richard Szeliski David J. Kriegman

B. MAPx,k Solution AnalysisUnderstanding and evaluating blind deconvolution algorithmsAnat Levin, Yair Weiss, Fredo Durand, William T. Freeman

C. Variational Method MAPkRemoving Camera Shake from a Single PhotographRob Fergus, Barun Singh, Aaron Hertzmann, Sam T. Roweis, William T. Freeman

23

PSF Estimation by Sharp Edge Prediction

Given edge steps, debluring can be reduced to Kernel Optimization

Suggested in PSF Estimation by Sharp Edge Prediction \ Neel Joshi et. el. in

Select Edge Step (Masking)

Estimate Blurring Kernel

Recover Latent Image

24

PSF Estimation by Sharp Edge Prediction - Masking

Original Image Edge Prediction Masking

Min

Max

Valid Region

25

Masking, Cont.Which is Best the Signals?

Edge

Impulse

Original Blurred

26

Blurr Model: y=x*k+n, n ~ N(0,σ2)

Bayseian Framework: P(k|y) = P(y|k)P(k)/P(y)

Map Model:argmaxk P(k|y) = argmink L(y|k) + L(k)

PSF Estimation by Sharp Edge Prediction – PSF Estimation

27

PSF Estimation by Sharp Edge Prediction – Recovery

Recovery through Lucy-Richardson Iterations given the PSF kernel

28

Blurred Recovered

PSF Estimation by Sharp Edge, Summary & Improvements

1. Handle RGB Images – perform processing in parallel

2. Local Kernel Variations:Sub divide image into sub-image units

Limitations:– Highly depends on the quality of the edge detection– Requires Strong Edges in multiple orientations for

proper kernel estimation– Assumes knowledge of noise error figure.

29

blur kernel

MAPx,k , Blind Deconvolution Definition

blurred image sharp image

noise

Input (known)

Unknown, need to estimate

?

?Courtesy of Anat Levin CVPR 09 Slides30

MAPx,k Cont. - Natural Image Priors

Derivative histogram from a natural image

Parametric models

Derivative distributions in natural images are sparse:

Log

prob

xx

Gaussian:

-x2

Laplacian:

-|x||-x|0.5

|-x|0.25

Courtesy of Anat Levin CVPR 09 Slides31

Naïve MAPx,k estimation

Given blurred image y,

Find a kernel k and latent image x minimizing:

Should favor sharper x explanations

Convolution constraint

Sparse prior

Courtesy of Anat Levin CVPR 09 Slides32

The MAPx,k paradox

P( , )>P ),( Let be an arbitrarily large image sampled from a sparse prior , and

Then the delta explanation is favored

Latent imagekernel

Latent imagekernel

Courtesy of Anat Levin CVPR 09 Slides33

?

The MAPx,k failure sharp blurred

Courtesy of Anat Levin CVPR 09 Slides34

The MAPx,k failure

Red windows = [ p(sharp x) >p(blurred x) ]

15x15 windows 25x25 windows 45x45 windows

simple derivatives

-]1,1-],[1;1[

FoE filters

)Roth&Black(

35

P(blurred step edge)

sum of derivatives: cheaper

The MAPx,k failure - intuition

P(blurred impulse) P(impulse)

sum of derivatives:

cheaper

>P(step edge)

>

k=[0.5,0.5]

Courtesy of Anat Levin CVPR 09 Slides36

P(blurred real image)

MAPx,k Cont. - Blur Reduces Derivative Contrast

Noise and texture behave as impulses - total derivative contrast reduced by blur

>P(sharp real image)

cheaper

Courtesy of Anat Levin CVPR 09 Slides37

MAPx,k Reweighting Solution

Alternating Optimization Between x & k

Minimization term:

MAPx,k

High Quality Motion Debluring From Single Image / Shan et al.

39

MAPx,k Reweighting - Blurred

39

4040

MAPx,k Reweighting - Recovered

Solution Approaches

A. Image Deblurring PSF Estimation using Sharp Edge PredictionNeel Joshi Richard Szeliski David J. Kriegman

B. MAPx,k Solution AnalysisUnderstanding and evaluating blind deconvolution algorithmsAnat Levin, Yair Weiss, Fredo Durand, William T. Freeman

C. Variational Method MAPkRemoving Camera Shake from a Single PhotographRob Fergus, Barun Singh, Aaron Hertzmann, Sam T. Roweis, William T. Freeman

47

MAPk estimation

Given blurred image y, Find a kernel minimizing:

Again, Should favor sharper x explanations

Convolution constraint

Sparse prior Kernel prior

48

Superiority of MAPk over MAPk,x

Toy Problem : y=kx+n

The joint distribution p(x, k|y). Maximum for x → 0, k → ∞.

p(k|y) produce optimum closer to true k .∗

uncertainty of p(k|y) reduces given multiple observations yj =kxj + nj .

49

Evaluation on 1D signals

MAPk variational approximation (Fergus et al.)

Exact MAPk MAPx,kFavors delta solution

MAPk Gaussian prior

Favor correct solution despite

wrong prior!

Courtesy of Anat Levin CVPR 09 Slides50

Intuition: dimensionality asymmetry

MAPx,k– Estimation unreliable. Number of measurements always lower than number of unknowns: #y<#x+#k

MAPk – Estimation reliable. Many measurements for large images: #y>>#k

Large, ~105 unknowns Small, ~102 unknowns

blurred image ykernel k

sharp image x

~105 measurements

Courtesy of Anat Levin CVPR 09 Slides51

Courtesy of Rob Fergus Slides52

Three sources of information

Courtesy of Rob Fergus Slides53

Image prior p(x)

Courtesy of Rob Fergus Slides55

Blur prior p(b)

Courtesy of Rob Fergus Slides56

The obvious thing to do

Courtesy of Rob Fergus Slides57

Variational Bayesian approach

Courtesy of Rob Fergus Slides58

Variational Bayesian methods

• Variational Bayesian = ensemble learning, • A family of techniques for approximating intractable

integrals arising in Bayesian inference and machine learning. • Lower bound the marginal likelihood (i.e. "evidence") of

several models with a view to performing model selection.

59

Setup of Variational Approach

Ensemble Learning for Blind Source Separation / J.W. Miskin , D.J.C.

MacKay

Small synthetic

blurs

large real world blurs

Cartoon images

Gradients of natural images

Independent Factor Analysis \ H. AttiasAn introduction to variational methods for graphical models \ JORDAN M. et al.

61

63

64

65

Courtesy of Rob Fergus Slides66

Example 1

67

Output 1

68

Example 2

69

Output 2

70

Example 3

71

Output 3

72

Achievements

• Work on real world images• Deals with large camera motions

(up to 60 pixels)• Getting close to practical generic solution

of an old problem .

73

Limitations• Targeted at camera motion blur

– No in plane rotation– No motion in picture– Out of focus blur

• Manual input– Region of Interest– Kernel size & orientation– Other parameters e.g. scale offset, kernel TH & 9 other semi-fixed

parameters

• Sensitive to image compression, noise(dark images) & saturation

• Still contains artifacts (solvable by upgrading from Lucy Richardson)

74

Evaluation

Cumulative histogram of deconvolution successes:

bin r = #{ deconv error > r }

MAPk, Gaussian prior

Shan et al. SIGGRAPH08Fergus, variational MAPk

MAPx,k sparse prior

100

80

60

40

20

Su

cces

ses

per

cen

t

75

Summary

MethodQuasi-MAPK

JoshiReweighted MAPKX

ShanVariational MAPk

Fergus

Distortion modelDefocus blursimple PSF

Camera motion blurComplex sparse PSF

Camera motion blurComplex sparse PSF

Region of interestEdge regionEdge regionUser selected

Optimization modelQuasi-MAPKMAPKXVariational Bayes for K estimation (MAPk equivalent)

Degrees of freedomO(K)O(K+X)O(K+Xprior+PRIOR)

SchemeGradient based least squares

Alternating iterativeMultiscale iterative(internal altering)

78

Debluring is underconstrained

Debluring single image under constrained

problem

?Blured imageRecovered image

Recovered kernel

Priors do the trick

?Blured image

Image prior

Recovered kernel

Kernel marginalization

?Blured image

Recovered kernel

Image prior

Back to non-blind deconvolution

?Recovered image Blured image

Recovered kernel

Existing challenges and potential research

• Robustness to user’s parameters & initial priors

• Solutions to spatially varying kernels

84

Thank You Eitan & Tomer

The End