image deblurring - university of california, santa...
TRANSCRIPT
![Page 1: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/1.jpg)
Image Deblurring
October, 18th 2007Orazio Gallo
Removing Camera Shake from a Single PhotographRob Fergus, Barun Singh, Aaron Hertzmann, Sam T. Roweis, and William T. Freeman
SIGGRAPH 2006
Single Image Motion Deblurring Using TransparencyJiaya Jia
CVPR 2007
![Page 2: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/2.jpg)
Motivation
Camera Motion
![Page 3: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/3.jpg)
Removing Camera Shake from a Single Photograph
Rob Fergus, Barun Singh, Aaron Hertzmann, Sam T. Roweis, and William T. Freeman
SIGGRAPH 2006
![Page 4: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/4.jpg)
Camera Motion: Hardware Solution
Commercial stabilization
Ben-Ezra and Nayar PAMI 2004
![Page 5: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/5.jpg)
Camera Motion
=
I=L∗ f n
⊗
I is the degraded image
L is the latent (unblurred) image
f can be Point Spread Function or motion Kernel
![Page 6: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/6.jpg)
The Problem
● Many pairs of L and f can be convolved resulting in the same I!
● On the other hand, the distribution of gradients in different natural images are similar across very different images![Field D., “What is the goal of sensory coding?” Neural Computation 6, 559-601, 1994]
![Page 7: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/7.jpg)
Gradients Distribution
![Page 8: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/8.jpg)
Gradients Distribution
![Page 9: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/9.jpg)
Gradients Distribution
The parametric model is built with mixture of Gaussians
![Page 10: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/10.jpg)
Estimation of the Blur Kernel
But the statistics we have access to are on the gradients so:
P=LP∗Kn
LP , P∇ LP ,∇ P
Likelihood term Mixture of zero-meanGaussians
Encourages sparsity and positiveness of the kernel
![Page 11: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/11.jpg)
Estimation of the Blur KernelThe straightforward MAP solution to the problem fails so they use a Variational Bayesian approach.[J. Miskin and D. MacKay, “Ensemble Learning for Blind Image Separation and Deconvolution”, Adv in Independent Comp. Analysis, 2000]
Variational Bayes
Maximum a-Posteriori (MAP)
Ä
Score
![Page 12: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/12.jpg)
Algorithm Outline● User input:
– Manual selection of a patch rich in edge structure– User input of the maximum size of the blur kernel– Initial estimate of the kernel (horizontal vs. vertical)
● Estimation of the blur kernel with the variational Bayes● Multi-scaling● Deconvolve with standard deconvolution algorithm (Lucy-
Richardson)
![Page 13: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/13.jpg)
Results
![Page 14: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/14.jpg)
Results
![Page 15: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/15.jpg)
Results
![Page 16: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/16.jpg)
Single Image Motion Deblurring Using Transparency
Jiaya Jia
CVPR 2007
![Page 17: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/17.jpg)
Motivation
Object Motion(and camera motion)
![Page 18: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/18.jpg)
Object Motion: Hardware Solution
Raskar et al. SIGGRAPH 2006
![Page 19: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/19.jpg)
1-D Object Blur
0∈{ 0,1}i∈[ 0,1 ]
What happens when a solid, non-noisy, 1-D object is blurred?
IDEA:Information about the blur can be extracted from the
transparencies!
![Page 20: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/20.jpg)
1-D Object Blur
The convolution can be seen as applied to the alpha matte!
![Page 21: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/21.jpg)
2-D Object Blur
The bands can be shown to be a close upper bound for the filter (most other methods require user interaction).
The first problem is the size of the filter!
![Page 22: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/22.jpg)
2-D Object BlurIn this case the shape is unknown so we need to calculate the alpha matte together with f !
Likelihood term Uniform distribution
Two terms:• Alpha matte is binary• Neighboring pixels belong to the same region (either object or non-object)
![Page 23: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/23.jpg)
2-D Object Blur● Solution of the MAP problem provides both the
un-blurred alpha matte and the blur filter● Solution is found by iterative optimization:
– Conjugate gradient– Refinement with Belief Propagation
![Page 24: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/24.jpg)
2-D Object Blur
![Page 25: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/25.jpg)
Natural Images
![Page 26: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/26.jpg)
Natural Images● For a blurred object on a partially occluded
background the transparency map is unique.● For patches on natural images what is
background?● They prove mathematically that the
identification of background does not affect the filter estimation!
![Page 27: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/27.jpg)
Implementation● For moving objects: user-drawn strokes to
determine foreground and background● For camera motion: user-selected patches
around boundaries● Calculate the transparency map with standard
algorithms● Estimate blur filter and deconvolve with
standard deconvolution algorithm (Lucy-Richardson)
![Page 28: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/28.jpg)
Results
![Page 29: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/29.jpg)
Results
![Page 30: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/30.jpg)
Results
![Page 31: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/31.jpg)
Results
![Page 32: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/32.jpg)
Results
![Page 33: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/33.jpg)
Summary● Fergus et al.
– Use the finding that gradients distribution across natural images is heavy-tailed
– Variational Bayes results in increased robustness● Jia
– Transparency carries information about blur– Is the MAP approach robust enough?
![Page 34: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/34.jpg)
That's all!
![Page 35: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/35.jpg)
![Page 36: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/36.jpg)
Backup: Convolution f ∗g =∑
nf n⋅g −nFor the 1-D case:
Point Spread Function PSF:
http://en.wikipedia.org/wiki/Point_spread_function
![Page 37: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/37.jpg)
Backup: Alpha Matte
Pictures from Chang et al. “A Bayesian Approach to Digital Matting”, CVPR 2001
![Page 38: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/38.jpg)
Backup: Maximum A Posteriori
Formulas images from Wikipedia at: http://en.wikipedia.org/wiki/Maximum_a_posteriori
![Page 39: Image Deblurring - University of California, Santa Cruzalumni.soe.ucsc.edu/~orazio/CMPE280V_Presentations/DeblurSingle... · Image Deblurring October, 18th 2007 Orazio Gallo Removing](https://reader035.vdocuments.us/reader035/viewer/2022070615/5c65f60209d3f230488b66c9/html5/thumbnails/39.jpg)
Backup: Terms Explained
Constrains that the unblurred alpha matte is a two-tone image
Alpha difference between neighboring pixels(8-neighborhood?)