8/16/2019 Khan 2015

1/6

 

A Novel Multi-frame Super Resolution algorithm for

Surveillance Camera Image Reconstruction  Aunsia Khan, Muhammad Aamir Khan,Faisal Obaid, Sultanullah Jadoon,

 Mudassar Ali Khan and Misba Sikandar{aunsiakhan,mudaser,aamir,sultan,misbasikandar}@uoh.edu.pk, faisalobaid74@gmail.com

 Department of Information Technology, University of Haripur, Khyber Pakhtunkhwa, Pakistan 

 Abstract  —  This paper gives a loom towards the  

growing spatial resolution necessary to beat the limitations

of the imaging technology in surveillance and security

disciplines. It has been observed that metropolis cities

worldwide invest huge sum of money in surveillance

camera system but few are closely observing the benefits

and the costs of those investments and to measure the

overall impact of surveillance cameras on crime rates. The

low resolution coupled with poor quality optics is not be

enough to identify the subject of interest in crowd, from a

distance, in bad weather and any other limiting factor. In

this paper we have introduced multi-frame super-

resolution technique that does not require explicit motion

estimation and will be useful for producing imagery

evidence that the police might reasonably accept as proof

of someone's identity. Mostly the research is done in this

area by taking a SR image and then after adding their own

noise patterns where as our algorithm are working on

actual LR images of surveillance camera and getting a SR

image while removing the original blur and noise. Our

algorithm requires the training set of Low resolution (LR)

images from a still camera to produce High resolution

(HR) image data and enhances it using anisotropic

Diffusion and De-noising. In the image based

representations, this technique of super resolution providesa great step towards resolution independence. The

application of this method was successfully demonstrated

for the restoration from a short low resolution set of

images into a super resolved image. This super resolution

algorithm works best when the Diffusion is applied and

noise reduction filters are applied.

 Keywords —Super Resolution, Surveillance camera

images, Multi frame super resolution, Gray images, Low

Resolution images, Training Set.

I. INTRODUCTION

Image-based models intended for computer graphics

are resolution dependent, they cannot be zoomed a lot

 beyond the pixel resolution on which they were sampled

at without causing degradation of overall quality. The process of obtaining a high resolution image by increasing the number of pixels is known as Super

resolution. Vande walle et al defined higher resolution

image as an image constructed by up- sampling and

interpolation having more resolution and large number

of pixels than actual low resolution image but have no

extra details. Capel's proposed definition of resolution

enhancement depicts that the low resolution image

should result in an improved detailed contents in super

resolution image by restoring the high frequency

contents. Which causes the overall increase in the

number of pixels [10]. 

Super  

Resolution image reconstruction (SRIR) is one of

the capable techniques of digital image processing in

which challenges are made by reconstructing a HighResolution imagery by blending some of the information

enclosed inside countable numbers of under sampled

low-resolution images of a particular scene. This

technique engage many groups by filtering out the blur

and noise and Up sampling the under sampled images.

In contrast with diverse image enhancement techniques,

SRIR technique filter outs the distortions and increasing

their spatial resolution can enhance the worth of such

low resolution and under sampled images [1]. Two

approaches are used to perform super-resolution

enhancement: that are single frame and multi-frame

enhancement. Since low resolution images inherently

contain less information than higher resolution images,the process of constructing high resolution images form

an input low resolution images requires the missing high

resolution data to be calculated. 

The target of SRIR method is to recuperate an image of

high resolution from low resolution images from a

surveillance camera provided as an input. Two families

of methods are used to classify the SR i.e. i) Classical

multiple image super resolution and ii) Super resolution

Based on Example. In the classical method (e.g., [12, 5,

8]) a group of images of a stationary scene having low-

resolution are taken. All of the low resolution images

inflict a set of linear constraints on the unidentified highresolution intensity values. If considerable number of

low-resolution images are taken from the source, then

high resolution image can easily be reconstructed. 

978-1-4799-7620-1/15/$31.00 ©2015 IEEE

8/16/2019 Khan 2015

2/6

 

In this paper we focus on multi-frame image

reconstruction in which LR images are captured

simultaneously by a surveillance camera. As one of

constrain with image reconstruction is the missing data.

So in the multi-frame super-resolution reconstruction,

Observed images are registered to a specific high

resolution reference frame in order to formulate the

fusion problem.

The major processing steps that characterize the Super

resolution consist of these following steps: 1.

 

Low Resolution images acquisition: LR images are

acquired from the same scene with non-integer (in terms

of inter-pixel distances) geometric displacements

 between any two of the images.

2. Motion compensation and Image registration: For the

reference to high resolution desirable grid, the sub-pixel

geometric transformation is estimated for every source

image.

3. High resolution image reconstruction: constructing a

high resolution output image as a solution to the actual

 problem.

4. 

 Noise Reduction: Reducing blur and noise from the

final image.

One would expect that the affluence of the real- world

images seem difficult to capture analytically. This

encourage an approach based on learning: that is; in a

training set, The pixel details of different images at a

low resolution are learnt; then the details of final image

are predicted based on the learned relationships. For the

 past several years [2, 6], researchers have been

exploring this approach for enlarging images. To

motivate why this approach should work at all, we notedthat a set of completely random variables have much

high variability as compared to the set of image pixels.

Many research studies have been conducted on

mammalian visual system for their early processing

stages [4, 15]. We take advantage of these regularities of

registration in our algorithms, as we use multiple images

of one scene, with a still camera, to create an image

having information more visible then LR images.

Correct high resolution images cannot be generated

without a specific training data set. 

II. ALGORITHM

CONCEPTUAL IMAGE RESAMPLING

After reading over the different approaches to perform

super-resolution, we have decided to focus on multi-

frame. We selected the multi frame approach because

we believe it will produce more robust results above

wider collection of input images. One of the main

limitations of the single-frame approach is that the

 process will only be effective if the database contains a

high resolution image which is similar to the image

which is trying to be enhanced. The multi-frame

approach on the other hand uses the data which is

contained in the multiple input frames to interpolate the

HR image pixel values, this allows the multi-frame

approach to be applied to any input image, even if a

similar image was not used during the design of the

algorithm. 

At first, as we don't have high resolution data so this

task may seem impossible. However, we know that the

edges of the low resolution images should be kept sharp

in the next resultant resolution level. It is extremely

difficult to estimate the arbitrary motion without

guarantee of the estimators performance in the image of

real world scenes. The Super resolution performance can

have disastrous implications if the motion estimation is

incorrect. 

We aim of generating visually plausible details of

images e.g. sharp edges and credible texture. The set ofdata is then used to create a new image having increased

number of pixels and, on the same time, increase in

resultant image size.

This paper is to be considered of having the following

steps: 1.

 

Set of low resolution images captured via still

Surveillance camera.

2. 

Mapping pixels of each image of low resolution

to a grid of high resolution.

Fig. 1. LR images to HR image

3.  Anisotropic diffusion applied to the HR image.

4.   Noise Reduction using Noise Reducing Filter.

III. IMPLEMENTATION

 A. Generation of Training set  We started the generation of training set by collecting

low resolution images and arranged the images that we

 process all in way for up gradation that we planned. On

average, low resolution image with original number of

 pixels is created by sub-sampling and blurring. But we

collected a cluster of same low resolution images to

reconstruct an up-graded image.

8/16/2019 Khan 2015

3/6

 

Fig. 2. Single LR image 

 B. Mapping LR pixels to HR grid  We apply an initial step of mapping t

resolution image. We needed to store t

values of different low resolution imag 

true image of high resolution. We suphaving grids as the number of pixels

The size of each HR grid be the same

number of LR images used. We put al

LR images in the HR grid respectivel

figure 

Fig. 3. LR image to HR grid 

We want to store each pixel that corr 

 possible pixel of the low-resolution

 pixels are taken from set of 25 image

shown in fig. (3) whereby interlacing

we get that the desired resolution is o

 perfect reconstruction is guaranteed.

camera resolution, we sample using a

grid. We restricted ourselves to reaso

of the image, there is still a large numb

that we have to stock up, and therefo

generally applicable training sets. Fig.

image which is created by the said mAs per our believe, for the prediction o

 present in the original image, the co

resolution image with highest resolu

important role.

he pixels of low-

he different pixel

s to form the 

 pose a HR imagein the LR image.

size as that of the

l the pixels of the

, as shown in the

esponds to every

image; here the

s, respectively as

the four images,

 btained, and thus

Due to limited

n insufficient 2D

able information

er of information

re we formulated

(4) shows the HR

apping of pixels.f very fine details

mponents of low

ion  play  a  very 

Fig. 4. Grid of HR image having

Fig. 5. HR image

C. Image Enhancement using Diffus We applied anisotropic diffusion o

image obtained as HR image so th

mosaic pattern in our resultant im

diffusion helped in favoring the hig

the low contrast edge. We believe

contrast affects the overall relations

and high resolution image pixels,

anisotropic diffusion helps in recontrast resultant image of high reso 

LR images Pixels

on the LR image and

t we don’t have the

ge. The anisotropic

contrast edge over

that the local image

hip between the low

and essentially the

taining the overalllution. 

8/16/2019 Khan 2015

4/6

 

Fig. 6. Diffused LR image 

Fig. 7. Diffused HR image

  D. Noise Reduction We filter out the Diffused HR image b

filter in the frequency domain, to reduresultant image. There was no noise i

the overall degradation is caused onl

number of pixels. 

Fig. 8. Applied Filter in Frequency

a noise reducing

e the noise in then the LR image;

 because of less

omain

Fig. 9. De-Noised Diffused HR i

 

IV. RESULTS

Fig. (9) shows results of the algorit

LR images. Training set wassurveillance camera. A significantly

resulting image was generated

acceptable than the original LR

details and sharp edges were pre

depicts that our low level training s

 produce an acceptable HR accura

algorithm interprets that resultant H

more via diffusion. The fig. (7) sho

upgrades the HR image. In the mea

diffused HR image fig. (9) using th

(8) showed sharper and better res

LR image in fig. (6). Nonetheless; t

its best to depict the low-resolution

high resolution detailed image, fig. ( There is a surprising regularity acro

a training set made from the imag

can be used to invent missing detai

images. While training set of the im

works best, a training set of generic

very broad class of inputs. Finally,

 best when the Diffusion is applied

filters are applied.

TABLE ISampling rate 10 samp

Image array dimensions 721 X 5

 No. of Samples 25

Zoom in Factor 17 X

Table I enumerate the sampling rate of dat

samples taken for generating the output an

which pictures were zoomed for comparison.

age

m we applied to our

aken from a stillclear and acceptable

that was highly

images, the image

erved. The fig. (5)

t is not sufficient to

te image data. The

image is enhanced

s that the diffusion

while the de-noised

filter shown in fig.

lt than the diffused

he algorithm proved

image, resulting in a

9). 

ss images, such that

s of low resolution

s in many other LR

ages to be processed

images can handle a

the algorithm works

and noise reduction

les/sec

2

set, the total number of

d the zoom in factor by

8/16/2019 Khan 2015

5/6

 

TABLE IIFigure Algorithm applied Result

a No Poor

 b Wiener filter Deficient

c Bicubic interpolation Good

d Our algorithm Best

Table II enlists the application and result of different algorithms when

applied on the same input and the sub figures shows the zoomed

images by factor of 17.

Fig. 10. Comparing our SR with the wiener fi lter and BicubicInterpolation constraints. Note that the bicubic interpolation, when

applied to LR patches (a) , results in a high-resolution image (c)

which is sharper and cleaner than the wiener filtered image (b), but is

not able to recover the fine and plausible details. In contrast, our high-

resolution image (d) produced these fine details while maintaining the

edges and line sharpness.

V. CONCLUSIONS

In our this paper, we have introduced multi-frame

super-resolution technique that does not require explicit

motion estimation and will be useful for producing

something that the police might reasonably accept as

 proof of someone's identity. Our algorithm was inspired by the algorithms [2, 4]. It is unclear in many studies

that appear to show success whether surveillance

cameras had a positive impact in combination with

improved lighting, or whether the improved lighting

might accomplish the positive outcome on its own.

Studies vary on the degree to which they take

confounding factors into account. The computational

 burden being a difficult task prevents the iteration

scheme to produce improved results. Since there are

many parameters to be considered, it seems that a fair

comparison with other super resolution algorithms

cannot be made, resulting in no comparisons presented

in this paper. Experiments with a number of sets of

 parameters suggest that our algorithm yields results

which are quite comparable if not superior to some of

the algorithms [17, 16] especially when the training set

images are of very low resolution. 

We have constructed a super-resolution algorithm that

is based on training set [7], and introduced a faster,

simpler, and, we believe, better super-resolution

algorithm of single pass. The algorithm requires the

training set of LR images from a still surveillance

camera to produce HR image data and enhances it using

anisotropic Diffusion and De-noising. In the image

 based representations, this technique of super resolution

 provides a great step towards resolution independenceThe application of this method was successfully

demonstrated for the restoration from a short low

resolution set of images into a super resolved image. 

RECOMMENDATIONS

The approach we described here possibly will be very

useful in image processing techniques and graphics

applications (see [6, 13]). A recent Home Office study

found that 80 per cent of images from CCTV cameras

were of such poor quality that they were worthless as

evidence. For the enlargement of images, removal of

noise, estimation of shapes of 3-D surfaces and forattacking the other imaging application, Training sets

can be built and used. We assumed the gray images that

were registered and were spatially static for our

restoration of images in multi frame super resolution.

Reviewing previous work on image re-sampling theory

draw from the literature of computer, we formed a

relationship between contrasting research areas. In our

research, we demonstrated that Resampling theory of

8/16/2019 Khan 2015

6/6

 

images can be handy to imaging systems that are

aligned. The supremacy of this approach is fabricated in

the larger training sets, proper noise reduction models,

allowing rendering models and rich prior probabilities

and, allowing efficient scene inference. This algorithm

of super resolution provided the state of the art results

when applied. Furthermore, this work can be expanded

 by using colored images of low resolution as a data set

hence producing comparable results.

ACKNOWLEDGMENT 

This paper is made possible through the help and

support from everyone, including: parents, teachers, and

friends. 

The product of this research paper would not be

 possible without all of them.

REFERENCES

[1] Vivek Bannore, 2009, Volume 195/2009 " Iterative- Interpolation Super-Resolution Image Reconstruction,Studies"

in Computational Intelligence.

[2] W. T. Freeman and E. C. Pasztor MA, 1999. MIT Press"

Learning to estimate scenes from image".

[3] S. Baker and T. Kanade. 2000. " Limits on super-

resolution and how to break them". In IEEE Conf.

Computer Vision and Pattern  Recognition,. 

[4] D. J. Field 6:559–601, 1994. "What is the goal of sensorycoding". Neural Computation. 

[5] Dallas County, Texas: "Repeat offender shows

surveillance cameras' limitations," by Tanya Eiserer, The

 Dallas Morning News. 

[6] W. T. Freeman, E. C. Pasztor, and O. T. Carmichael.

40(1):25–47, 2000 " Learning lowlevel vision". Intl. J.

Computer Vision.

[7] J. S. DeBonet and P. Viola 1998. "Texture recognition

using a non-parametric multi-scale statistical model". In Proc.

IEEE Computer Vision and Pattern Recognition.

[8] S. Farsiu, M. Robinson, M. Elad, and P. Milanfar T-IP,

(10), 2004. "Fast and robust multiframe super resolution".

[9] 

William T. Freeman, Thouis R. Jones, and Egon C.Pasztor_ MERL (A MITSUBISHI ELECTRIC RESEARCH

LABORATORY) http://www.merl.com Example-Based

Super-Resolution

[10]  Denis Chekhlov

http://www.cs.bris.ac.uk/~chekhlov/sr.html

[11] M. Irani and S. Peleg CVGIP, (3), 1991. "Improving

resolution by image registration".

[12] H. Sidenbladh and M. J. Black. Learning image

statistics for Bayesian tracking. In  Intl. Conf. on

Computer Vision, 2001.

[13] D. J. Heeger and J. R. Bergen ACM SIGGRAPH, pages

229{236, 1995. "Pyramid-based texture analysis/synthesi". In

Computer Graphics Proceedings, Annual Conference Series.

[14] 

E. P. Simoncelli. Statistical models for images:

Compression, restoration and synthesis. In 31st Asilomar

Conf. on Sig., Sys. and   Computers, Pacific Grove, CA,

1997. 

[15] 8. S. Farsiu, D. Robinson, M. Elad, and P. Milanfar

August 2004, Volume 14, no 2, pp. 47-57, " Advances and

Challenges in Super-Resolution", International Journal of

Imaging Systems and Technology.

[16] S. Farsiu, D. Robinson, M. Elad, and P. Milanfar, Fast and

Robust Multi-Frame Super-Resolution, IEEE Transactions on

Image Processing, October 2004, Volume 13, no 10, pp. 1327-

1344.

[17] E. P. Simoncelli Paci c Grove, CA, 1997. "Statistical

models for images: Compression, restoration and synthesis". In

31st Asilomar Conf. on Sig., Sys. and Computers,.

[18] S. C. Zhu and D. Mumford. Prior learning and Gibbs

reaction-dilusion. IEEE Pattern Analysis and Machine

Intelligence, 19(11), 1997.