localized and computationally efficient approach to …
TRANSCRIPT
![Page 1: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/1.jpg)
LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO
SHIFT-VARIANT IMAGE DEBLURRING
ICIP, Oct. 2008, San Diego
Murali SubbaRao, Youn-sik Kang, Satyaki Dutta*, Xue Tu
{murali , yskang, tuxue}@ece.sunysb.edu,*[email protected]
Dept. of Electrical and Computer Engineering*Dept. of Mathematics
State University of New York at Stony Brook, NY 11794
![Page 2: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/2.jpg)
Problem: Shift/Space-Variant Image Deblurring
• Example: Actual Image of a slanted planar object captured by a cell-phone camera
• Blurring Point Spread Function (PSF) is different at each pixel.• Computationally efficient shift-variant image deblurring is
needed.edgeleft near 55.2}Max{ ≈σ
Image Captured by a Cellular phone Image Restored by RT
areacenter near 1}min{ ≈σ
![Page 3: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/3.jpg)
Causes and Applications
• Causes of Shift-Variant Blur– 3D shape of objects and limited depth of field of cameras result in
shift-variant defocus blur.– Motion of 3D objects causes shift-variant motion blur.– Aberrations of Optical Imaging Systems: e.g.: defocus, coma,
astigmatism, and spherical.– atmospheric turbulence in astronomical telescopes.
• Applications of Shift-Variant Deblurring– Digital Image/Video Cameras and Cellphone cameras: Image
deblurring.– Machine Vision: Shape-from-Defocus, Inverse Imaging– Medical Imaging: Microscopes, Endoscopes– Scientific: Telescopes (atmospheric turbulence causes shift-
variant blur in astronomy)– Solution of Integral and Differential Equations.
![Page 4: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/4.jpg)
COMPARISON• Previous approaches
– Partial (not full) localization: e.g. divide a large image into 32x32 blocks and carry-out shift variant deblurring in each block separately.
– Often does not exploit the locality and limited support domain of the blurring kernel PSF,
– Often does not exploit the spatial smoothness of focused images (formulation includes redundant unknown parameters).
– Computationally exorbitant and less accurate.
• New Approach– Avoids the disadvantages listed above for the previous
approaches.– Represents a fundamental theoretical and computational advance.– relevant to solving integral/differential equations and shape from
defocus.– Extension of S Transform for shift-invariant deblurring proposed
by Subbarao in 1989.
![Page 5: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/5.jpg)
Shift-Variant Blurring Model
Conventional Model: Fredholm Integral Equation of the First Kind
f ( ): Focused imagek( ) : Shift-Variant PSF (blurring kernel)g( ) : blurred image
Problems: – k( ) is in global form.– inverting the equation to obtain f( ) is difficult.– Discrete form of the above equation is often used to solve for f( )
in small (e.g. 32x32) image blocks using a Singular Value Decomposition (SVD) technique (e.g. spectral filtering).
– Computational cost is exorbitant, and accuracy may be limited.
( ) ( ) ( )b d
a cg x y k x y u v f u v d u d v, = , , , , .∫ ∫
![Page 6: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/6.jpg)
Rao Transform (RT) Theory and Algorithm
Define a completely localized kernel
Reformulate the problem in an equivalent form that definesRao Transform (RT):
In this form, it becomes possible to invert this integral transform locally under the assumption of analyticity of images.
f( ) is expanded in a Taylor-series at f(x,y), and order or integration and summation are interchanged.
Various order derivatives of g(x,y) are consedered.
( , , , ) ( , , )h x y u v k x u y v x y= + , +
( ) ( ) ( )x a y c
x b y dg x y h x u y v u v f x u y v du dv
− −
− −, = − , − , , − , − .∫ ∫
![Page 7: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/7.jpg)
RT Theory and Algorithm
Blurred image can be expressed as the weighted sum of the derivatives of the focused images where the weights are determined by the given shift-variant PSF (SV-PSF):
By considering the derivatives of g( ), the following linear system of equations can be obtained:
( )
0 0( )
N nn i i
n in i
g x y S f − ,,
= =
, ≈ ∑ ∑
( 0 , 0 ) ( 0 , 0 )0 0 0 1
( 1 , 0 ) ( 1 , 0 )1 0 1 1
( 0 , ) ( 0 , )N N
g fr rr rg f
g f
⎡ ⎤ ⎡ ⎤⎡ ⎤⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥=⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥⎢ ⎥⎢ ⎥ ⎢ ⎥⎣ ⎦⎣ ⎦ ⎣ ⎦
, , ,x y x y x y=g R f
![Page 8: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/8.jpg)
Inverse RT
Forward RT:
Inverse RT:
Focused Image:
A regularization technique (e.g. SVD based spectral filtering) can be used to obtain a better estimate of inverse RT while inverting . This yields a new localized regularization technique.
, , ,x y x y x y=g R f
, , ,x y x y x y′=f R g 1, ,( )x y x y
−′ =R R
( )
0 0( )
N nn i i
n in i
f x y gS − ,,
= =
, = ′∑∑
,x yR
![Page 9: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/9.jpg)
RT/IRT Example: Symmetric SV-PSFBlurred image:
Closed-form solution to Focused Image:
We are not aware of any previous work that presents such a localized closed-form solution, or one that has similar
computational advantages.
( 0 0 ) ( 0 0 ) ( 1 0 ) ( 1 0 ) ( 0 1 ) ( 0 1 )2 0 0 2
( 2 0 ) ( 0 0 ) ( 0 2 ) ( 0 0 )2 0 0 2
1 1 . . .2 2
g f f h f h
f h f h H O T s
, , , , , ,, ,
, , , ,, ,
= + +
+ + +
( 0 0 ) ( 0 0 ) ( 1 0 ) ( 1 0 ) ( 0 1 ) ( 0 1 )2 0 0 2
( 2 0 ) ( 1 0 ) 2 ( 0 1 ) ( 0 1 ) ( 0 0 )2 0 0 2 2 0 2 0
( 0 1 ) 2 ( 1 0 ) ( 1 0 ) ( 0 0 )( 0 2 )0 2 2 0 0 2 0 2
3 1 1( )2 2 2
3 1 1( )2 2 2
. . . ( H i g h e r O r d e r T e r m s
f g g h g h
g h h h h
g h h h h
H O T s
, , , , , ,, ,
⎛ ⎞⎜ ⎟, , , , ,⎜ ⎟, , , ,⎜ ⎟⎝ ⎠
⎛ ⎞⎜ ⎟, , , ,,⎜ ⎟, , , ,⎜ ⎟⎝ ⎠
= − −
+ + −
+ + −
+ )
![Page 10: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/10.jpg)
Analysis of RT/IRTIf higher order derivatives of g( ) used in the computation of IRT
are erroneous (due to noise, usually so), then, solution for f ( )will have errors, even if a localized regularization technique is used.
The region of significant support of the blurring kernel often satisfies the condition (e.g. blur circle radius or sigma)
The error will be low if, within the region of significant support of the blurring kernel, Taylor-series expansion of f( ) in RT provides a good approximation to the actual product under the integral:
Further research has provided some improved solutions to theseproblems. They will be presented in a future publication.
( ) ( )h x u y v u v f x u y v− , − , , − , −
( ) 0 for | | or | |h x u y v u v u R v R− , − , , ≈ > >
![Page 11: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/11.jpg)
Analysis of Computational Complexity
For a conventional SVD method based on spectral filtering, a 32x32 image requires inverting a matrix, i.e. O(32^6)=O(2^30).
In the RT approach, computations are dominated by the inversionof NxN size matrix , once at each of the 32^2 pixels. N is the order up to which the derivatives of g( ) are used.Setting N=4, the computational complexity becomes O(16^3 X 32^2)=O(2^22).
In this example, approximate computational advantage is a factorof 2^8=256. In our MATLAB implementation, without code optimization, we have observed a speed up of 2 to 4.
2 232 32×
,x yR
![Page 12: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/12.jpg)
Experiment on 1-D RT: Simulated blur
Input: sine wave,PSF: Gaussian, sigma =0.5+0.1x , x_max =10, N=8, M=2
Input Shift variant blurred input
Restored output
![Page 13: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/13.jpg)
Experiment on 2-D RT: Simulated blur
• Input image size :469x188,• PSF: Gaussian, • N=4 M=2• Sigma_max =2.8,
Sigma_min = 0.2
Input focused image
Result of image restoration by RT
Shift-variant Gaussian blurred image
![Page 14: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/14.jpg)
Experiment on 2-D RT: Real Image
Input image size :640x480PSF: Gaussian, N=4 M=2Sigma_max =2.5, Sigma_min = 0.5
Result of image restoration by RTShift-variant real input image
![Page 15: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/15.jpg)
RT on Cell-phone image
• Image size: 640 x 480 • Shift Variant PSF
plate of edgeright in 7.2}Max{ ≈σ
Image captured by a Cell phone Image restored by RT
area left wallin 35.1}min{ ≈σ
![Page 16: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/16.jpg)
RT on Cell-phone image
• Image size: 640 x 480 • Shift Variant PSF
edgeleft near 55.2}Max{ ≈σ
Image captured by a Cell phone Image restored by RT
areacenter near 1}min{ ≈σ
![Page 17: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/17.jpg)
Example of Ongoing Research: Extending RT for Aberration PSFs.
Aberration PSF: Coma
Cylindrical PSF: Defocus Gaussian PSF: Defocus
Aberration PSF: Astigmatism
![Page 18: LOCALIZED AND COMPUTATIONALLY EFFICIENT APPROACH TO …](https://reader030.vdocuments.us/reader030/viewer/2022012715/61ae2a7bd30d8e0697218d4b/html5/thumbnails/18.jpg)
Conclusion• A novel approach has been presented for shift-variant image
deblurring. It naturally exploits the spatial locality of blurring PSFs and smoothness of underlying focused images.
• A central idea is the formulation of the problem in a completelylocalized form. It represents a fundamental theoretical and computational advance.
• A closed-form solution is provided that is computationally efficient and usually accurate.
• The RT formulation provides new insights, and a new local regularization technique.
• RT provides a new method of solving integral equations (and therefore differential equations).
• This approach can be extended to shape-from-defocus techniques in the case of shift-variant blur.
• This approach is being extended in several ways through further research.