image restoration and super-...
TRANSCRIPT
![Page 1: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/1.jpg)
ImageRestoration and Super-
Resolution
Manjunath V. Joshi
Professor
Dhirubhai Ambani Institute of Information and Communication Technology, Gandhinagar,
Gujarat
email:[email protected]
![Page 2: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/2.jpg)
Overview
Image Restoration
Different approaches for Image Restoration
Image Super-Resolution (SR)
Super-Resolution using motion as a cue
Learning based Super-Resolution- Use of Wavelet Transform (WT) and Discrete Cosine Transform (DCT)
Our recent Works on Restoration and Super-Resolution
![Page 3: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/3.jpg)
Image Restoration
Goal of image restoration is to improve the quality of the image from it’s degraded version
Different from image enhancement
Enhancements are heuristic procedures designed to manipulate an image and are image dependent. No strong mathematical reasoning exists.
![Page 4: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/4.jpg)
Restoration techniques are based on modeling the degradation which takes place during image capture.
Degradation may be due to camera defocus blur, motion blur that may occur when there is relative motion between camera and the object, atmospheric turbulence, sensor noise, etc
![Page 5: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/5.jpg)
Restoration- to apply inverse process on the degradation to obtain the estimate of the original
Contrast stretching is an enhancement technique while removal of blur by deconvolution is a restoration process.
![Page 6: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/6.jpg)
Restoration applications:
Medical imaging-X ray images noisy-better diagnosis
Film industry- old film prints- scratches
Law enforcement- license plates motion blurred (numbers not clear in the image)
Remote sensing
![Page 7: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/7.jpg)
),(),(),(),(
),(),(*),(),(
vuNvuHvuFvuG
OR
yxnyxhyxfyxg
• The objective of restoration is to obtain an estimate of
f(x,y) given some knowledge of the H and N. This is an
inverse problem
H
n(x,y)
g(x,y)f(x,y)
Restoration model
![Page 8: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/8.jpg)
Restoration and blind deconvolution are different
Restoration – degradation is known- PSF (H) and noise statistics are known or assumed depending on application
Blind deconvolution – estimate both the original signal as well as the degradation function when partial knowledge on degradation is known.
![Page 9: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/9.jpg)
Restoration and Super-Resolution - Comparison
Restoration – The model do not consider the aliasing. The restored image is of same size as the degraded image
Super-resolution – Aliasing is taken care by suitably modeling the aliasing due to under sampling. The size of the super-resolved image is bigger than the degraded one. Difficult problem to solve.
![Page 10: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/10.jpg)
Different Approaches: Restoration
Inverse Filtering
Weiner Filtering
Constrained optimization
Unconstrained approaches.
![Page 11: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/11.jpg)
OO Original Image Blurred and noisy image
(degraded image)
Restored image using Weiner filter
![Page 12: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/12.jpg)
Resolution: Smallest measurable detail in a visual presentation. Tells about the fineness of detail that can be distinguished in an image
Spatial Resolution: spacing of pixels in an image measured in pixels per inch (ppi)
High Spatial Resolution: Pixel density is high. (Larger no of pixels in an image)
Image Super-Resolution (SR)
![Page 13: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/13.jpg)
HR Applications : Medical Imaging,
remote sensing, robot vision, industrial inspection, etc
Super-Resolution (SR): Obtain high resolution
from several low resolution observations
of the same scene. ( minimizes aliasing
and blurring).
![Page 14: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/14.jpg)
Example on SR:
Observations Super-Resolved Cha
Deepu and Chaudhuri
![Page 15: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/15.jpg)
Why SR ?
1. Cost 2. Shot noise
Conventional Interpolation Methods :
Nearest Neighbor or zero order hold
or pixel replica, Bilinear, Bicubic
Disadvantage : Single image used.
Do not consider the aliasing or blurring.
![Page 16: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/16.jpg)
Motion
Zoom
Blur
Photometry
Learning based techniques
Different approaches to Solve Super-resolution Problem
![Page 17: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/17.jpg)
MPEG sequence A key frame
Super-Resolved
Use of Motion CueDipti and Chaudhuri
SuperSS
![Page 18: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/18.jpg)
The Idea!
© Moon Gi Kang
![Page 19: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/19.jpg)
Learning based Super-ResolutionUSE OF DISCRETE WAVELET TRANSFORM (DWT)
AND DISCRETE COSINE TRANSFORM (DCT)
Prakash Gajjar, PhD student
![Page 20: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/20.jpg)
Problem Definition
Given a low resolution image and a set of high resolution training images learn the high frequencies from the training data set and obtain SR.
![Page 21: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/21.jpg)
Approach
Learn the wavelet coefficients at finer
scales of the unknown high resolution image
from high resolution training set.
![Page 22: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/22.jpg)
An image and its Wavelet Transform (DB4)
Image
![Page 23: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/23.jpg)
Learning:
![Page 24: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/24.jpg)
Results
Low resolution image
Bilinear interpolation Super-resolved image
![Page 25: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/25.jpg)
Learning using DCT
Objective: Given a set consisting of pair of LR and HR images of a camera obtain high resolution image for an LR image captured using the same camera.
Motivation: HR Video with a very low memory.
![Page 26: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/26.jpg)
Sample training pairs of images
![Page 27: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/27.jpg)
A 4x4 DCT block
![Page 28: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/28.jpg)
DCT coefficients related to high frequency details in a block
![Page 29: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/29.jpg)
DCT based Learning from LR-HR pairs:
![Page 30: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/30.jpg)
Results
Observation Bicubic Interpolation DCT based Super-resolved
MSE=0.003784 MSE=0.003245
![Page 31: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/31.jpg)
Conclusions
A learning based technique for super-resolution using a single low resolution image is described
Advantage: 1. No cue used 2. Single LR observation used
Learning represents the next challenging
frontier for computer vision.
![Page 32: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/32.jpg)
Our recent works in SR and Restoration
Super-resolution in real time- Graph cuts optimization- Swati Sharma (M.tech student)
Particle Swarm Optimization for SR (B.tech student)
SR based on histograms of different filters as priors (B.Tech students).
Learning based methods for restoration–Kishor (PhD student)
Super-resolution of multi-spectral images in remote sensing (B.tech student)
![Page 33: Image Restoration and Super- Resolutionieee.daiict.ac.in/dip/slides/PRESENTATION-IEEE-29-02-09_mvj.pdf · Different from image enhancement ... Blind deconvolution –estimate both](https://reader033.vdocuments.us/reader033/viewer/2022050102/5f40e4908695585ea171dcb4/html5/thumbnails/33.jpg)
THANK YOU