solving a two-colour problem by applying probabilistic approach to a full-colour multi-frame image...

Contents lists available at SciVerse ScienceDirect

Signal Processing: Image Communication

Signal Processing: Image Communication 28 (2013) 509–521

0923-59http://d

n CorrE-m

mirjana

journal homepage: www.elsevier.com/locate/image

Solving a two-colour problem by applying probabilisticapproach to a full-colour multi-frame image super-resolution

Barbara Džaja n, Mirjana Bonković, Ljubomir MaleševićUniversity of Split, Livanjska 5, Split, Croatia

a r t i c l e i n f o

Article history:Received 9 July 2012Accepted 7 March 2013Available online 16 March 2013

Keywords:Super-resolutionDemosaicingMAPTwo colour prior

65/$ - see front matter & 2013 Elsevier B.V.x.doi.org/10.1016/j.image.2013.03.001

esponding author. Tel.: þ385 989123602.ail addresses: [email protected] (B. Dž[email protected] (M. Bonković), [email protected]

a b s t r a c t

Reconstruction based algorithms play an important role in the multi-frame super-resolution problem. A group of images of the same scene are fused together to producean image with higher spatial resolution, or with more visible details in the high spatialfrequency features. Demosaicing algorithms interpolate missing pixels in a raw imagetaken from one Charged Coupled Device (CCD) array, upsampling the number of the pixelspresent in the image. Since super-resolution (SR) and demosaicing are the two faces of thesame problem it is natural to address them together. In this paper it is: (i) shown thatcorrect modelling of the Bayer pattern in the generative process improves the super-resolution performance for colour images, and (ii) an algorithm that incorporates the twocolour prior into the probabilistic model is designed. The algorithm presented in thispaper focuses on the classes of images that have two dominant colours, i.e. most of theareas in the image are uniformly coloured. A convex optimization procedure for jointsuper-resolution and demosaicing is developed which outperforms state-of-the-artalgorithms.

& 2013 Elsevier B.V. All rights reserved.

1. Introduction and related work

In multi-frame image super-resolution (SR) a group ofimages of the same scene are fused to produce an image witha higher spatial resolution (HR), or with more visible detailsin the high spatial frequency features [1]. Limits on theresolution of the original imaging device can be improvedby exploiting the relative sub-pixel motion between thescene and the imaging plane.

SR algorithms can be categorized into four classes [2–4]:interpolation-based algorithms (IBAs), frequency-based algo-rithms (FBAs), learning-based algorithms (LBAs), and reconstruction-based algorithms (RBAs). IBA applies nonuniforminterpolation to produce HR image which is then deblurred,what gives a simple method which tends to blur the highfrequency details. FBA uses phase differences between the LR

All rights reserved.

),r (L. Malešević).

images to de-alias LR images. They are simple to implementand computationally cheap, but according to Borman [5] areextremely sensitive to model errors, hence limiting their use.LBA [6–9] employs application-dependent priors to recap theunknown HR image. The approach highly relies on thesimilarity between the training set and the test set. RBA[10–14] relies on the relationship between the LR images andHR image and hires various kinds of priors on the HR image inorder to regularize the ill posed inverse problem.

Algorithms can be also divided with respect to the dataset referring to the number of the LR input images.Algorithms running on the data set containing only oneimage are usually LBAs. Conversely, algorithms running onmore than just one image require pre-computing themapping between the images in their overlapping regionwhich is called image registration. If the image registrationis of high accuracy, the many overlapping images maybe used to increase the spatial sampling density of thescene, thus allowing recovery of image frequencies abovethe Nyquist limit of any single image [15]. In between allthese methods, the RBAs outperform all other algorithms,

www.elsevier.com/locate/image

www.elsevier.com/locate/image

http://dx.doi.org/10.1016/j.image.2013.03.001



http://crossmark.dyndns.org/dialog/?doi=10.1016/j.image.2013.03.001&domain=pdf



mailto:[email protected]




B. Džaja et al. / Signal Processing: Image Communication 28 (2013) 509–521510

according to Ben-Ezra [16], and by far are the best choice andtherefore are most frequently used. Lately an attempt hasbeen made to improve the result by combining the methods,i.e. using the learning based priors in the MAP approach likeSun et al. did [17,16,9]. RBAs among themselves also differ inthe number and type of the priors used. Hence some likeFarsiu et al. [18] use not less than several different types ofpriors, or have a content-aware image prior like Cho [11].

Conventional colour digital cameras also suffer fromlimited low spatial resolution and undergo the colourfiltering process. CCD is a photosensor, i.e. silicon chip,which lies in the focal plane of the camera, beyond whichColour Filter Array (CFA) is sited. At each pixel position CFAfilters out either the red, green or blue value resulting inimage that only measures one colour per pixel, known asthe raw image. To provide a full colour image, demosaicingalgorithms interpolate the missing pixel values in a rawimage. Besides SR algorithms, the process of demosaicingalso results with higher spatial resolution in an image.Regardless of whether SR and demosaicing are resolvedseparately [19,20,8,21–25] they are the two faces of thesame problem, hence it is natural to address them jointly.Fast dynamic image fusion methods have been defined forthe translational motion model and MAP estimation methodhas been proposed to incorporate dynamic SR method in ageneral algorithm for addressing the multi-frame demosai-cing problem in Gunturk et al. [26] and Farsiu et al. [18].Farsiu [18] applied the multi-frame demosaicing methoddirectly on CFA data and then applied the robust super-resolution method of Farsiu et al. [27] on each resultingcolour channel. Gotoh et al. [28] aimed to produce a high-resolution colour image directly from a raw image, obtainedby a single CCD. The authors tried to avoid colour artefactsusing YCbCr colour space in which the planes are lesscorrelated.Thus the possibility of emerging colour artefactsstill exists, and the resulting HR image highly dependson the number of the LR raw images used. Vandewalleet al. [29] also recognized the opportunity to improve theSR results by joining demosaicing and super-resolutionalgorithms. According to Vandewalle, artefacts introducedby demosaicing (such as colour aliasing) are a part of thesignal in the subsequent SR algorithm. Aliasing artefactsintroduced during the demosaicing process cannot be

Fig. 1. Left: low-resolution raw image. Right:

removed anymore in the SR algorithm. This results in alower performance of the reconstruction algorithm. There-fore, Vandewalle et al. presented an algorithm for jointdemosaicing and SR, based on the separation of luminanceand chrominance in the Fourier transform of the Bayer CFA.The authors separately computed the high-resolution lumi-nance and chrominance information using the informationfrom all the input images. Finally, they combined the HRluminance and chrominance images again to construct a HRimage with less colour aliasing artefacts.

The purpose of this paper is to obtain the SR image fromsub-pixel shifted low-resolution (LR) raw images resolvingSR and demosaicing jointly using MAP approach, similar toFarsiu [18], Gotoh and Vandewalle. The paper is concernedonly with images that are related by an eight degree-of-freedom (dof) plane projective transformation (or homo-graphy), and also restricts itself even more to translationand rotation. As opposed to Vandewalle, our algorithmworks with RGB values which are correlated by the newintroduced two colour prior. That way colour planes arecalculated depending on each other resulting in a lowernumber of input low-resolution raw images and higherquality. The algorithm presented in this paper was per-formed on the classes of images that have two dominantcolours, i.e. most of the areas in the image are uniformlycoloured. Some of our results are shown in Figs. 1 and 2.The images on the left are low-resolution raw images fromthe data set and on the right super resolved demosaicedimage (SRD) made by the approach proposed in this paper.This paper treats SR and demosaicing jointly using MAPprobabilistic approach in which different types of priorregularize the problem. More specifically the paper washeavily inspired by the work of Bennett and Joshi. The workof Bennett et al. [30] applied an earlier version of the twocolour prior used in Joshi [31] for super-resolution.

In this paper two contributions to this area are made:(i) it is shown that correct modelling of the Bayer patternin the generative process improves the super-resolutionperformance for colour images. This is demonstratedquantitatively on synthetic examples and qualitatively onthe real image sequences; (ii) the idea of the two colourprior of Joshi [31] is adopted and incorporated into theprobabilistic model and a convex optimization procedure

super resolved demosaiced image (SRD).

Fig. 2. Left: low-resolution raw image. Right: super resolved demosaiced image (SRD).

R

G

B x

s

pα

φ

Fig. 3. 3D representation of the points in space. pi is “primary” and si“secondary” colour centre in the 5�5 neighbourhood surrounding pixelxi . The colour of the cluster centre closer to xi (in terms of L2 distance) isassigned to pi , and the other centre colour is assigned to si . At each pixellocation i, a latent variable αi is introduced as a blending coefficientbetween pi and si .

B. Džaja et al. / Signal Processing: Image Communication 28 (2013) 509–521 511

is developed. Again, quantitative and qualitative improve-ments are demonstrated.

The rest of the paper is composed of Sections 2–5 andAppendix A. In Section 2 Maximum A Posteriori (MAP)multi-frame super-resolution is explained and accompaniedby the generative model, smoothness prior, two colour prior,objective function and optimization technique. Sections 3and 4 explain the incorporation of the two colour prior andthe Bayer pattern, respectively, supported by the results.In Section 5 a discussion regarding contributions is made,followed by Appendix A in which analytic gradient of theobjective function is derived.

2. MAP multi-frame super-resolution

Maximum a posteriori (MAP) approach has objectivefunction composed of a data error term and a weightedset of priors. The goal is to obtain a high-resolution image xwith given K low-resolution images yðkÞ. Next subsectionsexplain thoroughly the generative model of image formationprocess, smoothness prior, two colour prior, full objectivefunction and optimization technique, respectively.

2.1. The generative model

The generative model is used to describe the imageformation process in terms of a motion model, illumina-tion changes and camera limitations. K low-resolutioncolour images, yðkÞ, are assumed to have been generatedfrom the same high-resolution scene x, by the applicationof warping, blur, subsampling, illumination changes, andthe addition of image noise. The scene is presumed to be avectorized stack of pixels from red, green and blue colourchannels, with every channel having a value at everylocation in the image. The low-resolution scene is assumedto be a collection of colour measurements at some knownspacing, but not necessarily sampled so that each of thethree colour channels is measured at each location.

The generative model for a single colour channel, C, is

yðk,CÞ ¼ λðk,CÞ1 Wðk,CÞxðCÞ þλðk,CÞ2 1þϵðk,CÞ, ð1Þ

where the addition and multiplication across all pixels byλ1 and λ2 define a global affine photometric correction over

each image [32], ϵðk,CÞ is a zero-mean i.i.d. Gaussian noisevector with std sN , and Wðk,CÞ is the sparse matrix model-ling the warping, subsampling and blur. For a given colourchannel C, Wðk,CÞ will be of size Mðk,CÞ � N, where Mðk,CÞ isthe number of measurements in colour C in the kth image,and N is the number of pixels of any one colour in thehigh-resolution scene. Each row of this system matrix,Wðk,CÞ, is constructed according to the description given in[33]. The description is cited here:

Wij ¼ exp −12ðvj−u′iÞTCPSF ðvj−u′iÞ

n oð2Þ

where

CPSF ¼ s−2PSF ð∇HÞI2x2ð∇HÞT ð3Þ

is general 2D Gaussian with covariance sPSF , because thelocal deformation of any blur kernel in the HR image isapproximately affine, even for the projective case. u′i is 2Dcentre location of the ith pixel projected into the HR imageframe, while vj is 2D centre location of the jth HR pixel inits own frame and sPSF is the standard deviation of the PSF.The homography matrix H is 3�3 matrix parameterizedwith registration parameters θ. The motion model gives

u′i ¼HðθÞui ð4Þ

where ui is the pixel centre in the LR image. u′i and uiare given with homogeneous coordinates. The necessary


image registration (geometric and photometric) and point-spread functions for this computation have been found inadvance by an algorithm for finding an estimate of thehomography relating two images using the method ofmaximum likelihood used in [15].

2.2. Smoothness prior

The Huber function used as a prior ρð�Þ is a goodselection for image super-resolution [34,32], particularly in

I2460 I2530

B169012 B1200

I2542 I2633

Fig. 4. Images used as examples in this p

Fig. 5. Error (L2 norm) graph of SR. Errors for all images from the base which hadfunction are below the equal error line, what means that they are lower than erro

greyscale images. It is quadratic for small values of inputand linear for larger values. That is why it penalizes edges(discontinuities) less severely than a Gaussian prior, whosepotential function is purely quadratic [33]. Therefore it canbe stated that the Huber function models the statistics ofreal images more closely than a purely quadratic function,because real images contain edges, and thus have muchheavier-tailed first-derivative distributions than can bemodelled by a Gaussian [33,10,35]. Usually, the Huber prioris defined over pairwise image gradient estimates in the

I2548

3 B3008

I2674

aper with corresponding numbers.

Huber (H) and two colour priors (TCP) included together in the objectivers when Huber (H) prior was the only constraint in the objective function.

Table 1Comparison of errors for super-resolution with Huber and Huber withTCP, according to Fig. 5.

Imagenumber

Huber prior Huber and TCP

L2 PSNR SSIM L2 PSNR SSIM

I2460 20.7111 28.0349 0.7832 20.3382 28.1927 0.7877I2530 25.2303 26.5946 0.7078 25.0015 26.6854 0.7185I2674 20.9489 26.3805 0.7625 19.9446 26.8072 0.7940I2548 20.1215 28.7016 0.7800 18.7284 29.3248 0.8278I2542 21.2877 25.5313 0.7521 21.1130 25.6029 0.7628I2633 26.2801 26.2032 0.7249 26.0460 26.2809 0.7339

Table 2Errors for SRD and AHD for image shown in Fig. 6 with only Huberprior used.

Number of LRraw images

SRD (H) AHD (H)


3 18.7344 20.0808 0.6681 20.8597 19.1474 0.66085 16.7827 21.0364 0.7391 20.1414 19.4518 0.7120

10 17.1657 20.8404 0.7358 19.2071 19.8644 0.711620 16.7827 21.0364 0.7513 20.6578 19.2319 0.7201

Table 3Errors for SRD and AHD for image shown in Fig. 6 with Huber and TCPpriors used.


SRD (HTCP) AHD (HTCP)


3 18.3565 20.2578 0.6716 20.8647 19.1454 0.66035 16.2431 21.3202 0.7469 20.1444 19.4505 0.7119

10 16.6488 21.1060 0.7483 19.2053 19.8652 0.710920 16.2974 21.2912 0.7630 20.6586 19.2316 0.7200

Table 4Errors for SRD and AHD for image shown in Fig. 7 with only Huberprior used.


SRD (H) AHD (H)


3 20.4152 22.9280 0.6613 32.9282 18.7757 0.60065 19.6171 23.2744 0.6840 32.9632 18.7665 0.6193

10 19.8244 23.1831 0.6790 31.5789 19.1392 0.615820 19.2216 23.4513 0.7142 29.7675 19.6523 0.6645

Table 5Errors for SRD and AHD for image shown in Fig. 7 with Huber and TCPpriors used.


SRD (HTCP) AHD (HTCP)


3 20.1388 23.0465 0.6635 27.9529 20.1986 0.59645 19.3384 23.3987 0.6871 28.1415 20.1402 0.6126

10 19.5721 23.2944 0.6814 28.5152 20.0256 0.601920 18.9997 23.5522 0.7154 28.1900 20.1252 0.6726


horizontal, vertical and two diagonal directions, leading toan eight-neighbour graph structure over the image, denotedGðxÞ. The Huber function of each gradient in turn is used asthe energy term in a Gibbs prior:

PðxÞ ¼ ∑C∈fR,G,Bg

1Z exp −ν1 ∑

g∈GðxðCÞÞρðg,τÞ

( )" #ð5Þ

ρðz,τÞ ¼ z2 if jzjoτ

2τjzj−τ2 otherwise

(ð6Þ

where τ is the single Huber parameter specifying thegradient value at which the penalty switches from beingquadratic to being linear. The parameter ν1 represents theprior strength which is constant through the SR algorithm.Z is the partition function, found by integrating the functionover every possible high-resolution image. Using only theHuber function as a constraint with colour images is notenough. In practice, it is possible to obtain reasonable coloursuper-resolution image results just by applying the Huberprior separately in the three colour channels for a widevariety of input data sets. However, it does not introducecorrelation between the colour channels. Hence, the objec-tive function is upgraded with the prior that will relatecolour intensities across the three colour channels andsuppress colour artefacts, which is explained next.

2.3. Two-colour prior (TCP)

Heavily inspired by the work of Joshi et al., and alsoBennett et al., we used the two-colour image prior in theobjective function. It is assumed that at any point in thesuper-resolution reconstruction, the colour of the pixel underconsideration (xi) should be a blend of the two dominant (i.e.most prevalent) colours (pi, si) within a neighbourhood of thepixel. The underlying statistical image model treats all coloursin the local area as a linear combination of no more thanthose two representative colours [30]:

xi ¼ ð1−αiÞpiþαisi, ð7ÞFor the ith pixel of x, the 5�5 pixel neighbourhoodsurrounding is extracted. The two dominant colours inthat spatial neighbourhood are initialized by 10 iterationsof k-means clustering. The colour of the cluster centrecloser to xi (in terms of L2 distance) is assigned to pi,and the other centre colour is assigned to si. This way thecentral pixel always corresponds to the “primary” colour.This is followed by soft clustering Expectation Maximiza-tion (EM) algorithm, resulting in the fact that central pixelis no longer only a part of the “primary” colour, but amixture of the “primary” and “secondary” dominantcolours with the blending coefficient between the twocluster centres α. This explanation is represented graphi-cally in Fig. 3, where pixels are in the RGB colour space,and the condition is set that the vector jpisi

��!j is normalized:jpisi��!j¼ 1. Two further images the same size as x arecreated: p and s, for the “primary” and “secondary” localimage colours. At each pixel location i, a latent variable αiis introduced as a blending coefficient between pi and si,so that

di ¼ xi−ð1−αiÞpi−αisi, ð8Þ


is minimized for each pixel. From the scalar product of thevectors follows

pisi��! � pixi

��!¼ jpisi��!j � jpixi

��!j � cos φ, ð9Þ

and the mathematical expression for parameter α can bederived as follows:

αi ¼ðsi−piÞ

ðsi−piÞT ðsi−piÞ

!T

ðxi−piÞ: ð10Þ

In general there is no guarantee that the colour of xi willlie exactly on the pi−si line. Note that while x, d, p and s allhave 3� N values (three colours at each of N pixels), there

Fig. 7. Top left: raw image. Top right: original image.

Fig. 6. Top left: raw image. Top right: original image.

are only N elements in α, thus coupling the three colourchannels using only one blending parameter.

Alpha matting, usually addressed in literature as in He[36], refers to the problem of softly and accurately extract-ing the foreground from an image. The image in alphamatting is modelled as the linear composite of the fore-ground and background image using the alpha mattingparameter (the opacity for the foreground). Alpha mattinguses the same form of Eq. (7), where p and s represent thebackground and foreground image, respectively. So the αhere differs from α in He. In this paper α is used as thematting parameter of the primary and secondary colours,like Bennett and Joshi did [30,31].

Bottom left: AHD result. Bottom right: SRD result.

Bottom left: AHD result. Bottom right: SRD result.

Fig. 9. Error graph of super-resolution and demosaicing (SRD) and AHDwith Huber (H) and Huber with TCP (HTCP) prior terms for image shownin Fig. 6. As shown by the graph, error reduces notably when TCP isincluded.

Fig. 10. Error graph of super-resolution and demosaicing (SRD) and AHDwith Huber (H) and Huber with TCP (HTCP) prior terms for the imageshown in Fig. 7. As shown by the graph error reduces notably when TCP isincluded.


Using Eq. (8) as a prior, p and s can be computed inadvance, e.g. from an accurate estimate of the super-resolution image. At each step of the optimization, α canbe found, and the image error d can then be computed ateach pixel. The first colour prior term in Eq. (11) is simplythe weighted L2 norm of d. The second colour prior term inEq. (11) arises from a hyperprior over α. It is probable thatmost pixels in a piecewise constant image are close toeither the primary or secondary colours in their neighbour-hood, so αi ought to be either 0 or 1. Since pixels are flippedin p and s so that p is always closer to x, it is sufficient topenalize the absolute magnitude of the elements of α.By introducing the two colour prior, the possibility of colourartefacts has been suppressed as much as possible, since thecolour channels are coupled in an organized manner andthe resulting HR image is less sensitive to the number of LRinput images.

2.4. Objective function

The estimate of high-resolution image x is found usingan iterative optimization. The full objective function takesthe form

F ¼ ∑C∈fR,G,Bg

∑K

k ¼ 1∥rðk,CÞ∥22þν1 ∑

g∈GðxðCÞÞρðg,τÞ

"

þν2∥xðCÞ−ðαpðCÞ−ð1−αÞsðCÞÞ∥22þν3∥α∥1

#, ð11Þ

where

rðk,CÞ ¼ yðk,CÞ−λðk,CÞ1 Wðk,CÞxðCÞ−λðk,CÞ2 1 ð12Þ

is the residual for a given low-resolution image colourchannel C. This first term of the objective function comesfrom the data likelihood, and the remaining three termscome from the priors over the super-resolution image x(the Huber and two-colour priors, respectively).

Fig. 8. Error graph of SR and demosaicing into one process (L2 norm). Errors forincluded together in the objective function for SRD are below the equal error linonly constraint in the objective function.

2.5. Optimization

The three prior terms are added to the objective functionwith weights ν1, ν2 and ν3, which represent the ratios of thenormalization constants for the priors with the noise

all images from the base which had Huber (H) and two colour priors (TCP)e. It means that they are lower than errors when Huber (H) prior was the


precision 1=s2N on the data likelihood. In practice, thesevalues can either be set by hand empirically, or can be fittedto the data set presented using a scheme such as the oneproposed in [37]. In this paper these values were set

Table 6Comparison of errors between SRD and other demosaicing algorithms for a cou

Image BILINEAR ALT


I2460 105.4965 16.9657 0.7207 107.6910 16.7869 0.700I2548 73.4112 17.4596 0.8047 73.4802 17.4515 0.774I2542 67.0990 15.5595 0.7039 75.3737 14.5495 0.635B12003 63.5051 18.1400 0.6061 61.3312 18.4428 0.596

Table 7Comparison of errors between SRD and other demosaicing algorithms for a cou

Image BILINEAR ALT


I2460 104.0844 17.0828 0.7339 106.7782 16.8608 0.686I2548 68.5694 18.0523 0.8000 68.6489 18.0422 0.695I2542 61.3735 16.3342 0.7471 70.4269 15.1391 0.658B12003 62.6051 18.2643 0.6112 60.1282 18.6149 0.599

Fig. 11. Results of SRD and other demosaicing algorithms with Huber and TCP prdemosaicing gives a blurred result, (d) ALT also over smoothes the result and theresolved and demosaiced image (SRD) separates colours on the edges quite wground truth.

empirically. ν1 was in the range of [0.2, 3], ν2 was in between[0.01, 4], and ν3 in [0.001, 5]. s2N is a standard deviation ofGaussian noise level, set to the value from four to ten greylevels for each colour channel, and zoom factor is 2.

ple of images with only Huber prior term in SR algorithm.

AHD SRD


9 47.2623 23.9402 0.7223 41.7284 25.0219 0.77200 32.1665 24.6268 0.7745 26.5856 26.2819 0.81428 43.9455 19.2356 0.7335 30.5024 22.4072 0.76246 36.1778 23.0276 0.6754 39.2105 22.3284 0.6779

ple of images with Huber and TCP prior term in SR algorithm.

AHD SRD


4 46.6863 24.0467 0.7473 40.5480 25.2711 0.77842 29.9042 25.2602 0.8022 25.0454 26.8003 0.85129 42.6733 19.4907 0.7692 29.3083 22.7540 0.76729 36.1609 23.0317 0.6755 38.6638 22.4504 0.6795

ior terms for I2542 image. (a) LR raw image, (b) ground truth, (c) bilinearedges are quite jagged, (e) AHD blurs the edges between colours, (f) superell and outperforms all other algorithms giving the image closest to the


This objective function of Eq. (11) describes an convexoptimization problem, and is solved with respect to x toobtain the MAP estimate of the super-resolution image.Scaled Conjugate Gradients (SCG) from the Netlab toolbox[38] is an effective iterative solver of these problems. Theanalytic form for the gradient of F with respect to x isgiven in Appendix A. A good starting point for the itera-tions is the average image, a:

a¼D−1WTλ−11 ðy−λ2Þ, ð13Þwhere W, y, λ1 and λ2 are the stacks of the K groups ofWðkÞ, yðkÞ, λðkÞ1 I, and λðkÞ2 1, respectively, and D is a diagonalmatrix whose elements are the column sums of W. Noticethat both inverted matrices are diagonal, so a is simple tocompute. There is also no coupling between the colourchannels here, so each colour can be found independently.

Using this average image as the starting point for theMAP SR algorithmwith only Huber prior term is enough toget a sharp estimate from which we can finally calculatethe values of p and s, the primary and secondary coloursused in the two-colour image prior. These p and s can alsobe computed in other ways, from the early stopped MLimage or the average image itself, but the best results arealways gained if the primary and secondary colours arecalculated from the image that is closer to the goal.The values of p and s can be fixed for the duration of aMAP iterative optimization, meaning that at each step,only α and d need to be re-estimated from the currentsuper-resolution estimate x.

The idea of the experiments was to show that by solvingdemosaicing with MAP approach which uses generativemodel to estimate missing pixels, gives better results thanmost of the existing algorithms. In order to validate thisapproach, the tests on synthetic and real data sets wereperformed. Regarding the synthetic data, the images weremosaiced with different masks of the Bayer pattern [39] sothat they would look like the raw images taken by the digitalcamera. The Bayer patternwas incorporated in the generativeprocess by alternating the corresponding values in Wðk,CÞ tothe value of zero, and afterwards upsampled by the proposedapproach. To make sure that the two colour prior will causethe desired effect, its efficiency was previously tested on theplain set of LR images, i.e. not raw LR images. Therefore,hereinafter in Section 3 the incorporation of the two colourprior is described and quantified, and the effect of optimizingprocedure on the raw images is given in Section 4.

3. Incorporating the two colour prior (TCP)

This section investigates how adding TCP to the objectivefunction improves SR result (with no demosaicing included).In order to obtain a reliable algorithm, the synthetic andafterwards the real data sets were used. For the syntheticdata sets Berkeley and ICIDAR data bases were used andimages are named correspondingly to the base they weretaken from B for Berkeley and I for ICIDAR, followed by thenumber they have in the base. Some images from thesebases used in this paper are shown in Fig. 4. Experimentsshowed that having the hyperprior on alpha (fourth term inEq. (11)) does not require more than just a few input images.Hence, testing is reduced to the set of only 3 LR images. By

adding a large amount of noise the data sets resulted insignificantly low-quality images. To test the plain SR algo-rithm upgraded with TCP (i.e. without any raw images, justsimple LR images), the calculation of the primary andsecondary colours was carried out after performing SR withonly Huber prior term. Afterwards SR was performed onceagain with TCP added as an extra constraint to the objectivefunction. Testing regarding the net of the strength para-meters was done quite widely and the results for the plainSR with Huber and TCP are on the graph and in the table(Fig. 5, Table 1). Table 1 also contains PSNR values (in dB)calculated in the RGB colour space and also SSIM valuescalculated according to the paper [40].

The final results of SR process are expressed through theL2 norm, PSNR and SSIM values in the sense of differencebetween the ground truth and SR result. Errors obtained fromthe use of Huber and TCP priors together were comparedwith errors obtained from the use of only Huber prior term.The scatter plot of the L2 norms shows that errors for allimages from the base which had Huber and TCP priorsincluded together in the objective function are below theequal error line. That means that they are lower than errorswhen TCP was not included in the objective function. This isalso supported by Table 1, where all errors are shown. WithTCP included, errors are smaller than when only Huber prioris included in the objective function. The contribution thatTCP brings is more evident when the images are raw, i.e.more degraded, what will be shown in the next section.

4. Incorporating the Bayer pattern

Once it has been proved that TCP does actually improvethe results, the combined algorithm of SR and demosaicing(SRD) based on the MAP approach is now addressed.The goal is to produce a HR image from the set of LR rawimages gained directly from the CCD array. Considering thecost function in the first term of Eq. (11), the generativepart has to be modified. Excluding the correspondingvalues in Wðk,CÞ by setting them to the value of zero, wemodel the correct generative part where we use only themeasured raw LR pixels. Just like in Gotoh [28] andVandewalle [29], the aim is to produce high-resolutioncolour image directly from a raw image, obtained by oneCCD, but in the RGB colour space so that none of theconversion between colour spaces is necessary. Addingthat extra constraint of the two colour prior and thehyperprior on α results in a method that improves the SRresults especially if the number of the LR input images isquite low. The correct modelling of the generative partwhere only the measured raw LR pixels were used iscompared to the one where the LR image was firstdemosaiced and then all pixels used. For the purpose ofcomparison of our combined SR and demosaicing algo-rithm (SRD), several demosaicing algorithms were used:simple bilinear, ALT [26] and AHD [25]. The testing wasdone on the set of three LR images with a bigger amount ofnoise. As the error is significantly lower, addition of thethird and the fourth term to the objective function in (11)is more evident now when the images are raw. It can beseen in Tables 2–5. Acronym H refers only to Huber priorbeing used in the objective function, and acronym HTCP


refers to Huber and TCP being parts of the objectivefunction. The results shown in tables are accompanied bycorresponding images in Figs. 6 and 7, which show thecomparison of SRD and AHD using Huber and TCP for theinput of three LR raw images.

The top left image in these figures represents one of thethree low-resolution raw images used, while the top rightrepresents the original image from which LR raw imagesare generated. The bottom left is the result after perform-ing AHD demosaicing first followed by SR afterwards,while the bottom right represents the result after performingSR and demosaicing (SRD) algorithm together. Followingthe fact that the SR and demosaicing are the two faces ofthe same problem and therefore combining them into oneprocess, the results obtained in such a way give a lowererror and are much better than those obtained when theimages were first demosaiced and then pulled through SRalgorithm, no matter which demosaicing algorithm wasused. The experiment results that verify this statement arepresented in Sections 4.1 and 4.2.

The graph of errors for super-resolution and demosaicingcombined as one process is shown in Fig. 8. All errors for allimages are once again below the equal error line, what meansthat they are lower than errors when TCP was not included inthe objective function. So, by adding that extra constraint ofTCP to the objective function the lower error is gained.

Conclusions that can be drawn: using the TCP withHuber prior in joint SRD algorithm results in smaller errors,and the colours of such images are visually closer to thosein the ground truth image.

Fig. 12. Results of SRD and other demosaicing algorithms with Huber and TCP pdemosaicing gives a blurred result, (d) ALT gives also a blurred result, (e) AHD b(SRD) separates colours on the edges better than any other algorithm.

4.1. Synthetic data sets and results

To evaluate the power and sensitivity of the proposedalgorithm with respect to the number of input LR rawimages, the testing was done again on the synthetic(Berkeley and ICIDAR data base of images) and afterwardson real data sets consisting of 3, 5, 10 and 20 low-resolutionraw images. The synthetic results of SRD algorithm werecompared to the results gained by AHD, ALT and bilineardemosaicing followed by the SR algorithm. The error wasalways smaller for combined SR and demosaicing into oneprocess (SRD). Furthermore, the error is smaller as thenumber of input images increases. For AHD the number ofinput images is insignificant and the error varies very little.

The difference between SRD and AHD is more visible onthe graphs shown in Figs. 9 and 10. The comparison of all fouralgorithms is shown for the tested images in Tables 6 and 7,expressed through the L2 norm, PSNR (in dB) calculated in theRGB colour space and also SSIM values calculated according tothe paper [40].

SRD performs super-resolution and demosaicing together,other algorithms do not. Bilinear, ALT and AHD are demosai-cing algorithms, and raw images are first demosaiced andthen routed through the SR algorithm which incorporatesHuber and TCP priors into the objective function. This meansthat TCP will also impact the SR results of images demo-saiced with bilinear, ALT and AHD algorithms. According tothe results, SRD outperforms other methods. The artefactsintroduced by demosaicing are incorporated into the signal.The affected signal is routed through the SR algorithm. Once

rior terms for I2460 image. (a) LR raw image, (b) ground truth, (c) bilinearlurs the edges between colours, (f) super resolved and demosaiced image


incorporated, demosaicing artefacts can no longer beremoved from the signal and SR algorithm cannot removethem anymore. The images that show the difference of allfour algorithms are in Figs. 11 and 12.

Fig. 13. Results for comparison of the algorithms for the real data set. (a) Originimage magnified part, (d) original low-resolution jpeg image (magnified part), (super resolved and demosaiced image (SRD) separates colours on the edges qu

As seen in Fig. 11, SRD outperforms all other algorithmssince the TCP separates colours on the edges better thanany other algorithm. Although the improvement in Fig. 12is minor, it is still an improvement and even a small

al low-resolution raw image taken by Nikon D50, (b) SRD result, (c) rawe) AHD introduces a new (non-existing) colour on the edges (yellow), (f)ite well and does not introduce non-existing colours.


progress in the super-resolution problem is a signi-ficant step.

4.2. Real data sets and results

Testing on the real data base began by using the sets of3, 5, 8, 10 and 16 LR raw images, captured by Nikon D50camera, from slightly different positions. The image regis-tration (geometric and photometric) and point-spreadfunctions have been found by an algorithm for finding anestimate of the homography relating two images using themethod of maximum likelihood used in [15]. Using com-bined SRD algorithm on the real raw data visually gives nodifference in the number of used LR raw images whatleads to the conclusion that performing SR on the raw datais actually more definite. Furthermore, SR on the raw dataleaves no possibility for emerging jpeg and colour arte-facts. The gained results were compared with the resultsobtained by the first demosaiced images (by bilinearinterpolation or AHD) and thereafter routed through theSR algorithmwith and without that extra constraint of TCP.The results of performing SRD on the real raw data can beseen in Fig. 13. Figure shows comparison with anotherdemosaicing algorithm and a magnified part so that thedifference can be more noticeable.

This is how we interpret the results: bilinear demosaicingis proved to be worse than the AHD and gives quite a blurredimage. AHD algorithm (like other demosaicing algorithms)apparently introduces a new colour into a demosaiced image(yellow) which obviously does not exist there. Those demo-saicing artefacts, once incorporated into the signal, can nolonger be removed from it and SR can only make them morevisible. This is why joint algorithm for SR and demosaicing isadvantageous. Using TCP with Huber prior term is betterthan just using Huber prior. This is more evident on the rawimage since TCP models the colour neighbourhood quiteaccurately. This results in the accurate separation of thecolours on the edges without emerging some new coloursthat do not actually exist there.

5. Discussion

This paper proves that the two colour prior contributes tothe quality of the estimated HR images in poor conditionssuch as a small number of LR input images, or when the LRimages are highly degraded with significant presence ofnoise. Furthermore, demosaicing based on the MAP approachshowed to be better than most of the demosaicing algo-rithms, frequently named in the literature.

By introducing the two colour prior, the colour artefactsare successfully suppressed by the virtue of the blendingparameter α. α in essence makes correlation betweencolour planes possible, thus reducing the need for manyLR input images.

The work presented in this paper proved that thedominant colour concept works nicely with uniformlycoloured areas, i.e. an image with locally only two colours.What will happen with more colours, where three or morecoloured areas meet in a single point or with shaded andtextures areas, is a part of the further research. Investiga-tion regarding the optimization of the strength parameters

ν1, ν2 and ν3 is also needed. The goal is to optimize them ina shorter time thus making the whole algorithm less timeconsuming. A similar procedure has been done by Pickup[37], and will also be a part of the future work.

Acknowledgement

The authors would like to express their gratitude toProfessor Andrew Zisserman and Dr Lyndsey Pickup fortheir invaluable help, assistance and guidance through therealization of this work.

Appendix A. Analytic gradient of the objective function

The gradient of the objective function (11) with respectto high-resolution image pixels is

∂F∂x

¼−2 ∑K

k ¼ 1λðk,CÞ1 W ðk,CÞT rðk,CÞ þν1G

Tρ′ðGxðCÞ,τÞ

þ2ν2 ðαsðCÞ þð1−αÞpðCÞ−xðCÞÞðsðCÞ−pðCÞÞ ðsðCÞ−pðCÞÞTjsðCÞ−pðCÞj

"

−ðαsðCÞ þð1−αÞpðCÞ−xÞ#

þν3sðCÞ−pðCÞ

ðsðCÞ−pðCÞÞT ðsðCÞ−pðCÞÞ

� �T

sgnðαÞ, ðA:1Þ

where

ρ′ðzÞ ¼z if jzjoτ

2τ sgnðzÞ otherwise

(ðA:2Þ

and G is Capel's [15] a version of the gradient operator.

References

[1] M. Irani, S. Peleg, Improving resolution by image registration, GMIP53 (1991) 231–239.

[2] S. Borman, R. Stevenson, Spatial Resolution Enhancement of Low-resolution Image Sequences—A Comprehensive Review with Direc-tions for Future Research, University of Notre Dame, TechnicalReport, 1998.

[3] S.C. Park, M.K. Park, M.G. Kang, Super-resolution image reconstruc-tion: a technical overview, IEEE Signal Processing Magazine 20 (3)(2003) 21–36.

[4] S. Farsiu, D. Robinson, M. Elad, P. Milanfar, Advances and challengesin super-resolution, International Journal of Imaging Systems andTechnology 14 (2) (2004) 47–57.

[5] S. Borman, R.L. Stevenson, Super-resolution from image sequences—a review, in: Proceedings of the 1998 Midwest Symposium onCircuits and Systems, IEEE, Notre Dame, IN, USA, August 1998,pp. 374–378.

[6] W. Freeman, E.C. Pasztor, O.T. Carmichael, Learning low-level vision,in: ijcv, vol. 40, October 2000, pp. 25–47.

[7] W.T. Freeman, T.R. Jones, E.C. Pasztor, Example-based super-resolu-tion, IEEE Computer Graphics and Applications 22 (March/April (2))(2002) 56–65.

[8] M. Tappen, B. Russell, W. Freeman, Exploiting the sparse derivativeprior for super-resolution and image demosaicing, in: Proceedings ofIEEE Workshop on Statistical and Computational Theories of Vision,2003, pp. 1–24.

[9] S. Baker, T. Kanade, Limits on super-resolution and how to breakthem, in: cvpr, 2000.

[10] L.C. Pickup, D.P. Capel, S.J. Roberts, A. Zisserman, Bayesian methodsfor image super-resolution, The Computer Journal (2007). http://dx.doi.org/10.1093/comjnl/bxh000.

[11] T.S. Cho, N. Joshi, C. Zitnick, S.B. Kang, R. Szeliski, W.T. Freeman,A content-aware image prior, in: cvpr, June 2010, pp. 169–176.


[12] S. Dai, M. Han, W. Xu, Y. Wu, Y. Gong, Soft edge smoothness prior foralpha channel super resolution, in: IEEE Conference on ComputerVision and Pattern Recognition, June 2007, pp. 1–8.

[13] S. Dai, M. Han, W. Xu, Y. Wu, Y. Gong, A. Katsaggelos, Softcuts: a softedge smoothness prior for color image super-resolution, IEEETransactions on Image Processing 18 (May (5)) (2009) 969–981.http://dx.doi.org/10.1109/TIP.2009.2012908.

[14] S. Babacan, R. Molina, A.K. Katsaggelos, Variational Bayesian superresolution, IEEE Transactions on Image Processing 20 (4) (2011)984–999.

[15] D.P. Capel, Image Mosaicing and Super-resolution, Ph.D. Disserta-tion, University of Oxford, 2001.

[16] M. Ben-Ezra, Z. Lin, B. Wilburn, W. Zhang, Penrose pixels for super-resolution, IEEE Transactions on Pattern Analysis and MachineIntelligence 33 (7) (2011) 1370–1383.

[17] J. Sun, Z. Xu, H.Y. Shum, Image super-resolution using gradientprofile prior, in: IEEE Conference on Computer Vision and PatternRecognition, June 2008, pp. 1–8.

[18] S. Farsiu, M. Elad, P. Milanfar, Multi-frame demosaicing and super-resolution of color images, IEEE Transactions on Image Processing 15(2006) 41–159.

[19] D. Menon, G. Calvagno, Regularization approaches to demosaicking,IEEE Transactions on Image Processing 18 (10) (2009) 2209–2210.

[20] J. Mairal, M. Elad, G. Sapiro, Sparse representation for color imagerestoration, IEEE Transactions on Image Processing 17 (January (1))(2008) 53–69.

[21] F. Zhang, X. Wu, X. Yang, W. Zhang, L. Zhang, Robust colordemosaicking with adaptation to varying spectral correlations, IEEETransactions on Image Processing 18 (December (12)) (2008)2706–2717.

[22] J. Li, S. Randhawa, Color filter array demosaicking using high-orderinterpolation techniques with a weighted median filter for sharpcolor edge preservation, IEEE Transactions on Image Processing18 (September (9)) (2009) 1946–1957. http://dx.doi.org/10.1109/TIP.2009.2022291.

[23] P. Chatterjee, N. Joshi, S.B. Kang, Y. Matsushita, Noise suppression inlow-light images through joint denoising and demosaicing, in: cvpr,June 2011.

[24] J. Takamatsu, Y. Matsushita, T. Ogasawara, K. Ikeuchi, Estimatingdemosaicing algorithms using image noise variance, in: cvpr, June2010, pp. 279–286.

[25] K. Hirakawa, T.W. Parks, Adaptive homogeneity-directed demosai-cing algorithm, IEEE Transactions on Image Processing 14 (Mar (3))(2005) 360–369.

[26] B.K. Gunturk, Y. Altunbasak, R.M. Mersereau, Color plane interpola-tion using alternating projections, IEEE Transactions on ImageProcessing 11 (September (9)) (2002) 997–1013. http://dx.doi.org/10.1109/TIP.2002.801121.

[27] S. Farsiu, D. Robinson, M. Elad, P. Milanfar, Dynamic demosaicingand color super-resolution of video sequences, in: Proceedings ofSPIE Conference on Image Reconstruction from Incomplete Data III,vol. 5562, August 2004, Denver, CO, pp. 169–178.

[28] T. Gotoh, M. Okutomi, Direct super-resolution and registration usingraw cfa images, in: CVPR, 2004, pp. 600–607.

[29] P. Vandewalle, K. Krichane, D. Alleysson, S. Süsstrunk, Joint demo-saicing and super-resolution imaging from a set of unregisteredaliased images, in: Proceedings of the IS&T/SPIE Electronic Imaging:Digital Photography III, vol. 6502, 2007.

[30] E.P. Bennett, M. Uyttendaele, C.L. Zitnick, R. Szeliski, S.B. Kang, Videoand image Bayesian demosaicing with a two color image prior, in:eccv, 2006, pp. 508–521.

[31] N. Joshi, C.L. Zitnick, R. Szeliski, D. Kriegman, Image deblurring anddenoising using color prior, in: cvpr, 2009.

[32] D. Capel, Image Mosaicing and Super-Resolution (Cphc/Bcs Distin-guished Dissertations), Springer Verlag, 2004.

[33] L.C. Pickup, Machine Learning in Multi-frame Image Super-resolu-tion, Ph.D. Dissertation, University of Oxford, February 2008.

[34] S. Borman, R.L. Stevenson, Linear models for multi-frame super-resolution restoration under non-affine registration and spatiallyvarying psf, in: Computational Imaging II, vol. 5299, January 2004,pp. 234–245.

[35] L.C. Pickup, D.P. Capel, S.J. Roberts, A. Zisserman, Overcomingregistration uncertainty in image super-resolution: maximize ormarginalize?, EURASIP Journal on Advances in Signal Processing2007 (2007) 14, Article ID 23 565.

[36] K. He, C. Rhemann, C. Rother, X. Tang, J. Sun, A global samplingmethod for alpha matting, in: cvpr, June 2011.

[37] L.C. Pickup, S.J. Roberts, A. Zisserman, Optimizing and learning forsuper-resolution, in: bmvc, 2006.

[38] I.T. Nabney, NETLAB. Algorithms for Pattern Recognition, Advancesin Pattern Recognition, Springer, 2002.

[39] B.E. Bayer, Color imaging array, US Patent 3,971,065, July 1976.[40] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Perceptual image

quality assessment: from error visibility to structural similarity, IEEETransactions on Image Processing 13 (April (4)) (2004) 600–612.(recipient, IEEE Signal Processing Society Best Paper Award, 2009).

solving a two-colour problem by applying probabilistic approach to a full-colour multi-frame image...

Documents