Contents lists available at ScienceDirect

Signal Processing: Image Communication

Signal Processing: Image Communication 36 (2015) 53–62

http://d0923-59

n CorrE-m

ekkma@chcai@h

journal homepage: www.elsevier.com/locate/image

SIFT-flow-based color correction for multi-view video

Huanqiang Zeng a,n, Kai-Kuang Ma b, Chen Wang b, Canhui Cai a

a School of Information Science and Engineering, Huaqiao University, Xiamen, Chinab School of Electrical and Electronic Engineering, Nanyang Technological University, Singapore

a r t i c l e i n f o

Article history:Received 31 July 2013Received in revised form25 April 2015Accepted 28 May 2015Available online 11 June 2015

Keywords:Multi-view videoColor variationColor correctionSIFT flowMVC

x.doi.org/10.1016/j.image.2015.05.00865/& 2015 Elsevier B.V. All rights reserved.

esponding author.ail addresses: zeng0043@hqu.edu.cn (H. Zenntu.edu.sg (K.-K. Ma), wa0002en@e.ntu.eduqu.edu.cn (C. Cai).

a b s t r a c t

During the multi-view video acquisition, color variation across the views tends to beincurred due to different camera positions, orientations, and local lighting conditions.Such color variation will inevitably deteriorate the performance of the follow-up multi-view video processing, such as multi-view video coding (MVC). To address this problem, aneffective color correction algorithm, called the SIFT flow-based color correction (SFCC), isproposed in this paper. First, the SIFT-flow technique is used to establish point-to-pointcorrespondences across all the views of the multi-view video. The average color is thencomputed based on those identified common corresponding points and used as thereference color. By minimizing the energy of the difference yielded between the color ofthose identified common corresponding points in each view with respect to the referencecolor, the color correction matrix for each view can be obtained and used to correct itscolor. Experimental results have shown that the proposed SFCC algorithm is able toeffectively eliminate the color variation inherited in multi-view video. By furtherexploiting the developed SFCC algorithm as a pre-processing for the MVC, extensivesimulation results have shown that the coding efficiency of the color-corrected multi-viewvideo can be greatly improved (on average, 0.85 dB, 1.27 dB and 1.63 dB gain for Y, U, and Vcomponents, respectively), compared with that of the original multi-view video withoutcolor correction.

& 2015 Elsevier B.V. All rights reserved.

1. Introduction

With the rapid development of camera, display andnetwork communication techniques, multi-view video sys-tems, such as three-dimensional TV (3DTV) and free view-point TV (FTV), have been emerging [1,2], as these servicesprovide customers with depth perception and freedom ofchoosing the viewpoint. Due to better interactivity andreality, the multi-view video systems are expected to bewidely exploited in various domains, such as home enter-tainment, education, medical field, to name a few.

g),.sg (C. Wang),

Unlike the single-view video, the multi-view video iscomposed of more than one view, and these views areacquired by using multiple cameras distributed in differentviewpoints to synchronously capture the same scene.Consequently, noticeable color variation is often incurredacross the views due to different camera positions, orien-tations, characteristics, and local lighting conditions dur-ing the multi-view video acquisition. In order to efficientlycompress the huge amount of multi-view video data, theJoint Video Team (JVT) standardization body has standar-dized multi-view video coding (MVC) as an amendment ofH.264/AVC—Annex H [3]. The MVC not only exploits thespatial and temporal correlation within a single view (i.e.,intra-view) but also employs the correlation between theneighboring views (i.e., inter-view) for further improvingthe coding efficiency. However, more color variation across

H. Zeng et al. / Signal Processing: Image Communication 36 (2015) 53–6254

the views means that less inter-view correlation will bepresented among them. Obviously, this will reduce thecoding efficiency of the MVC. For that, an efficient colorcorrection algorithm for mitigating color variation inher-ited among various views of the multi-view video wouldbe highly desirable and can be used as a pre-processingstage for facilitating the follow-up multi-view videoprocessing tasks.

Multiple color correction methods for multi-view videocan be found in the literature. Based on the assumptionthat the DC component of each macroblock is affected bythe local illumination change, Hur et al. [4] suggested amacroblock-based adaptive local illumination change com-pensation method. Fecker et al. [5] utilized a histogrammatching approach for conducting color variation reduc-tion based on the fact that the histogram of the target viewshould be similar to that of its reference view. For that, thecumulative histograms of the target view and its referenceview are computed to establish a color mapping function.Shi et al. [6] presented a two-stage color correctionmethod. In the first stage, a colorization process isexploited to expand the color of each 8�8 block in thetarget view relative to that of the corresponding block inthe reference view. A coarse-to-fine scheme is thenemployed for those blocks that still present large colorvariation after the previous stage. By using the block-baseddisparity estimation based on a normalized cross-correlation criterion, Doutre et al. [7] identified the com-mon corresponding points shared by all the views in orderto compute the average color. For each view, a third-orderpolynomial color correction function is then established inreference to the computed average color so as to correct itscolor. Chen et al. [8] presented a histogram-offset-basedcolor correction method. In this method, the histograms ofthe reference view and the target view are first computedwithin their maximally matched regions, which are iden-tified by conducting disparity estimation in the rank-transformed domain. A histogram offset is then calculatedto correct the target view by using an iterative threshold-ing approach. Shao et al. [9] jointly exploited color correc-tion and chrominance reconstruction as the pre- and post-processing stages of MVC to improve its coding efficiency.

In addition, since the inception of scale invariant featuretransformation (SIFT) [10], some SIFT-based color correc-tion methods for multi-view video are developed due toits effectiveness on finding correspondences. Among them,Yamamoto et al. [11] proposed a method to correct theluminance and the chrominance of other view frames forinter-view prediction and view-interpolation predictionbased on the lookup tables of each RGB color channel.Yamamoto et al. [12] exploited the SIFT to detect thecorrespondences, calculated lookup tables with anenergy-minimization approach and then corrected themulti-view video with these tables. Tehrani et al. [13]randomly chose a view and the next view as the referenceview and the target view, respectively, for performingiterative color correction. In this method, the color correc-tion transformation is calculated based on the correspond-ing points obtained by using modified SIFT with outliersuppression. The iteration will be stopped, when theaverage change between the reference and target views

is smaller than a threshold. Lu et al. [14] presented a colorcorrection method by jointly using the SIFT and generalregression neural network. Jung et al. [15] proposed a colorcorrection method based on the camera characteristiccurve. This camera characteristic curve is first modeledby using nonlinear regression (with outlier removal inadvance) based on the correspondences between thereference and target views and is then utilized to generatethe lookup tables to correct the color distribution of thetarget views. Fezza et al. [16] exploited the combination ofthe SIFT and RANSAC methods to effectively define thecommon regions across views, and then developed animproved histogram matching using only common regionsand temporal sliding window to achieve a robust colorcorrection performance.

In this paper, an effective color correction algorithm formulti-view video, called the SIFT-flow-based color correction(SFCC), is proposed. In our approach, the SIFT flow [17]technique (where SIFT stands for scale invariant feature trans-form [10]) is firstly exploited to identify all common corre-sponding points across all the views of the multi-view video.The average color is then computed based on these identifiedcommon corresponding points and used as the reference color.The color of each view is then corrected by multiplying a colorcorrection matrix, which is generated by minimizing theenergy of the difference yielded between the color of thoseidentified common corresponding points in each view withrespect to the reference color using the least mean-squareapproach. Experimental results have demonstrated the accu-racy and effectiveness of the proposed SFCC algorithm on themitigation of color variation inherited in the multi-view video,compared with the state-of-the-art color correction methods.

The rest of this paper is organized as follows. Theproposed color correction algorithm, SFCC, is presentedin Section 2 in detail. Extensive simulation results aredocumented and discussed in Section 3. Finally, conclusionis drawn in Section 4.

2. Proposed sift-flow-based color correction (SFCC)

The proposed color correction algorithm, SFCC, consists oftwo stages as described in the following two sub-sections,respectively. In the first stage, the SIFT flow technique [17] isexploited to identify all the common corresponding pointsacross all the views of the multi-view video. In the secondstage, the average color is computed based on those detectedcommon corresponding points across all the views. By usingthis computed average color as the reference color, the colorcorrection matrix for each view is then established andemployed to correct the color of each view, respectively. Itshould be pointed out that the proposed SFCC algorithm isperformed picture by picture and view by view for the entiremulti-view video sequence. In other words, the pictures fromdifferent viewpoints at the same time instant are individuallycorrected.

2.1. Finding common corresponding points

Intuitively, it is fairly essential to identify accuratecorresponding points as many as possible for color correc-tion. In our view, approaches to find the corresponding

H. Zeng et al. / Signal Processing: Image Communication 36 (2015) 53–62 55

points between two views can be classified into two broadcategories: the block-based approach and the feature-basedapproach. In the first category, both the current andreference views (or pictures) are divided into smallerblocks such that each block in the current picture issubject to be matched at all locations within the pre-imposed search window of the reference picture accordingto a criterion (e.g., sum of absolute differences, SAD).Although the block-based approach can generate densecorresponding points, however, this approach is sensitiveto illumination change since such illumination changewithin each block might not be uniform [18]. Conse-quently, inaccurate corresponding points are resulted.Furthermore, the block contents of two views undermatching are normally not exactly identical due to smallviewpoint variation. This also contributes inaccurateestablishment of corresponding points. For the feature-based approach, on the other hand, features are firstextracted (say, using the SIFT technique [10]) for both thecurrent and reference pictures, individually. The point-to-point correspondence process is then conducted based onthe extracted features. Although not as much dense asblock-based approach, the number of established corre-sponding points by the feature-based approach is stillquite large and these resulted corresponding points tendto be more accurate [19]. Therefore, the feature-basedapproach is further investigated here.

To identify accurate corresponding points between twoviews as many as possible, the SIFT flow technique [17] isexploited in this paper. Inspired by the optical flow [20,21],which can produce dense corresponding points, SIFT flowfollows the same principles on the computing of theoptical flow, except that the matching process is conductedbased on the SIFT descriptors extracted on both currentand reference pictures, rather than using the pixel inten-sity as performed in the optical flow. Therefore, it com-bines the advantages of both SIFT features and optical flowon providing accurate and dense corresponding points.Another unique and highly desirable merit of exploitingSIFT flow is that it is especially suitable for those scenariosthat undergo a small viewpoint variation. This is in linewith the fundamental nature of the content across theviews of multi-view video.

In the SIFT flow, a SIFT descriptor is first established ateach pixel position to characterize the local image struc-tures for both the current and reference pictures. Similarto the optical flow [20,21], the SIFT descriptors from thecorresponding frames of two views are then matched toobtain the point-to-point correspondences and their flowvectors. This is achieved by minimizing the followingenergy function for SIFT flow, EðwÞ [17]:

EðwÞ ¼XpminðJs1ðpÞ�s2ðpþwðpÞÞJ1; tÞ ð1Þ

þXpηðjuðpÞjþjvðpÞjÞ ð2Þ

þX

ðp;qÞAε½minðα � juðpÞ�uðqÞj; dÞ

þminðα � jvðpÞ�vðqÞj;dÞ� ð3Þ

where s1ð�Þ and s2ð�Þ denote the SIFT descriptors extracted ateach pixel position on the current and reference pictures,respectively, p¼(x,y), q¼ðx0; y0Þ are the spatial coordinates,wðpÞ¼ðuðpÞ; vðpÞÞ is the flow vector at p, ε represents thefour-neighbors of p, η and α are the regularization parametersthat provide tradeoff among (1)–(3) and are empiricallydetermined, t and d are thresholds, which are utilized toeliminate matching outliers and flow discontinuities, respec-tively. Note that the default parameter setting of SIFT flow [17]is used in this work, that is, α¼2 � 255, η¼0.005 � 255,d¼40 � 255, and t is disabled.

Furthermore, it can be seen that the energy function EðwÞ isformulated as the similar form of the optical flow and consistsof three terms: the data term, the small displacement term andthe smoothness term. First, the data term shown in (1) is usedto reflect the similarity of the local image structures betweenthe current and reference pictures. This can be accuratelymeasured in terms of the difference incurred between s1ð�Þand s2ð�Þ. Therefore, to find more accurate correspondingpoints, the data term is preferred to be as small as possible.Second, although the flow vector is allowed to be as large asthe picture itself, the small displacement term shown in (2) isexploited to constrain the flow vector to be as small as possible,which encourages the smoothness of the flow field. Finally, thesmoothness term shown in (3) is utilized to penalize anyunsmoothness that might be incurred in the flow field. This canbe simply achieved by minimizing the difference of the flowvectors of the adjacent pixels. Since all the above-mentionedthree terms are favored to be as small as possible, the energyfunction EðwÞ is subject to be minimized.

As mentioned above, SIFT flow can be directly appliedto obtain dense corresponding points between two views.In this light, we further explore SIFT flow for the scenarioof multi-view video to identify common correspondingpoints across all views as below and an illustration isfurther presented in Fig. 1. For the ease of explanation,let N be the total number of views and Vi represent the ithview of the multi-view video, where i ranges from 1 to N.

1.

For each two consecutive views of the given multi-viewvideo Vi and Viþ1 (e.g., View 1 and View 2 in Fig. 1), SIFTflow is performed between them and the resulted flowvector of view Vi is denoted as wiðpÞ¼ðuiðpÞ; viðpÞÞ, fori¼1, 2, …, N � 1, respectively.

2.

By using the flow vectorwiðpÞ¼ðuiðpÞ; viðpÞÞ obtained inthe previous step, for each point in the first view V1

(denoted as p1 ¼ ðx1; y1Þ), its corresponding point in allthe other views Vi (denoted as pi ¼ ðxi; yiÞ) can bederived as follows: for i¼2, 3, …, N, respectively:

xi ¼ x1þXi�1

k ¼ 1

ukðxk; ykÞ

yi ¼ y1þXi�1

k ¼ 1

vkðxk; ykÞ

subject to

1rxirH1ryirW ð4Þ

where H �W is the frame resolution of the multi-view video.

View 1 View 2 View 3 View 4

View 5View 6View 7View 8

Fig. 1. An illustration of finding the common corresponding points across all views of the multi-view video by performing SIFT flow between each twoconsecutive views, individually. The 45th frame of multi-view video sequence “Race1” is presented here as an example.

H. Zeng et al. / Signal Processing: Image Communication 36 (2015) 53–6256

If point p1 can find its corresponding points pi in all theother views Vi based on (4), these corresponding points (i.e., pi, for i¼1, 2, …, N) will be selected as one of thecommon corresponding points across all the views (e.g.,the yellow circle in Fig. 1). On the contrary, if any point pi

falls out of the image range, it means that p1 fails to find itscorresponding point in view Vi and thus these correspond-ing points (i.e., pi, for i¼1, 2, …, N) will not be consideredas a common corresponding point (e.g., the red square inFig. 1).

2.2. Establishing color correction matrix

Since most cameras deliver RGB color signal directly,the proposed SFCC algorithm is conducted in the RGB colorspace. In this work, the color of each pixel of the uncor-rected picture in view Vi, for i¼1, 2, …, N, respectively, canbe corrected via a linear transformation; that is,

Ri

Gi

Bi

264

375¼Ai

Ri

Gi

Bi

264

375 ð5Þ

where Ri, Gi, and Bi are the original R-, G-, and B-component values of each pixel of the current picture inview Vi to be corrected, respectively; Ri, Gi, and Bi denotethe corrected counterpart components of the same pixel,respectively; the matrix Ai is a 3�3 color correctionmatrix of view Vi with real-valued entries, for i¼1, 2, …,N, respectively.

Now the question boils down to how to determine thecolor correction matrix Ai for view Vi. For this, theproposed SFCC algorithm makes full use of the detectedcommon corresponding points across all views. Firstly, theaverage color is calculated as the reference color by simplyaveraging each color component (i.e., R, G, B) of thedetected common corresponding points across all views

individually. The color correction matrix Ai is then com-puted by minimizing the energy of difference yieldedbetween the color of the detected common correspondingpoints in view Vi and the average color.

Let matrix Ci¼½Ri Gi Bi�T and matrix C i¼½R i Gi B i�Trepresent the original and corrected color of the commoncorresponding points located in view Vi, and let matrixC¼½R G B�T denote the average color of Ci. The Ri, Gi, andBi are the vectors that contain the original R-, G-, and B-component values of the detected common correspondingpoints located in view Vi, respectively. The R i, G i, and B i

are the vectors that contain the corrected counterpartcomponents of the same point, respectively. The R , G,and B are the vectors that contain the average R-, G-, andB-component values of the detected common correspond-ing points identified from all views, respectively. There-fore, the average color C can be calculated as

C ¼RGB

264

375¼

1N

XN

i ¼ 1

Ri

1N

XN

i ¼ 1

Gi

1N

XN

i ¼ 1

Bi

2666666666664

3777777777775

ð6Þ

The goal of color correction is to correct the color of allviews so that the color of each view becomes similar to thereference color so as to eliminate color variation. In otherwords, by using the average color as the reference color,the difference yielded between the corrected color of thedetected common corresponding points in each view C i

and this average color C should be as minimum aspossible. Hence, for view Vi, the corresponding difference(i.e., matrix di) can be defined as

di ¼ C� Ci ð7Þ

H. Zeng et al. / Signal Processing: Image Communication 36 (2015) 53–62 57

According to (5), di can be further expressed as

di ¼ C�AiCi ð8Þ

Hence, the color correction matrix Ai can be determinedby exploiting the least mean square, which is to minimizethe energy of difference, dT

i di, as

Ai ¼ arg minðdTi diÞ ð9Þ

Consequently, the relative analytic solution towards (9)can be derived as

Ai ¼ CCiðCiCTi Þ�1 ð10Þ

With the obtained color correction matrix Ai, the color ofview Vi can now be corrected based on (5), for i¼1, 2,…, N,respectively.

Table 1Multi-view video sequences.

Sequences From Resolution Views Frames

Race1 KDDI 640�480 8 250Flamenco2 KDDI 640�480 5 250Crowd KDDI 640�480 5 250Rena Tanimoto Lab 640�480 8 250

Fig. 3. Color correction result of multi-view video sequence “Flamenco2” (the 1salgorithm.

Fig. 2. Color correction result of multi-view video sequence “Race1” (the 25thalgorithm.

3. Experimental results and discussion

In this work, the proposed color correction algorithm,SFCC, has been experimented using multiple multi-viewvideo sequences as listed in Table 1 [22], and the corre-sponding results are described in the following three sub-sections. Since these multi-view video sequences havebeen stored in the YUV format while the proposed algo-rithm requires to be conducted in the RGB color space,they need to be converted to the RGB color space firstly.

3.1. Visual quality evaluations

First, the visual quality comparison on multiple multi-view video sequences before and after color correction areshown in Figs. 2–5. To have a better illustration, the zoom-in areas indicated by the yellow bounding box in twomulti-view video sequences, “Race1” and “Flamenco2,” (i.e., Figs. 2 and 3) are enlarged and presented in Figs. 6 and7 as examples, respectively.

One can see that each original (i.e., uncorrected) multi-view video sequence has noticeable color variation acrossthe views. For example, there is a significant change inbrightness from view V1 to view V2 in “Race1” and so doesthe brightness of view V2 of “Flamenco2” when comparedwith that of the other views. After applying the proposed

t frame): (a) original pictures; (b) corrected pictures by the proposed SFCC

frame): (a) original pictures; (b) corrected pictures by the proposed SFCC

Fig. 4. Color correction result of multi-view video sequence “Rena” (the 89th frame): (a) original pictures; (b) corrected pictures by the proposed SFCCalgorithm.

Fig. 5. Color correction result of multi-view video sequence “Crowd” (the 128th frame): (a) original pictures; (b) corrected pictures by the proposed SFCCalgorithm.

Fig. 6. An enlarged portion of the yellow bounding box region indicated in Fig. 2: (a) original pictures; (b) corrected pictures by the proposed SFCCalgorithm.

H. Zeng et al. / Signal Processing: Image Communication 36 (2015) 53–6258

SFCC algorithm, it can be observed that the color across allthe views of the corrected multi-view video sequenceslooks much more consistent (i.e., with much less varia-tions), compared with that of the uncorrected ones.

To further evaluate the color correction performance, theuncorrected multi-view video and the corrected one resultedfrom different color correction methods are evaluated by

utilizing four visual quality assessment metrics—structuralsimilarity metric, denoted as SSIM in [23], feature similaritymetrics, denoted as FSIM and FSIMC in [24], and motion-basedvideo integrity evaluation metric, denoted as MOVIE in [25],since these metrics are correlated well with the human visualperception and provide good subjective evaluation. Note thatSSIM, FSIM and MOVIE employ only the luminance

Table 2The visual quality comparison results: (A) Uncorrected, (B) HM [5], (C)BDEPF [7], (D) FBCC [16] and (E) our proposed SIFT-flow-based colorcorrection (SFCC) algorithm.

Sequences Method SSIM FSIM FSIMC MOVIE

Race1 (A) 0.4609 0.6740 0.6634 0.0125(B) 0.4637 0.6755 0.6658 0.0123(C) 0.4670 0.6765 0.6683 0.0120(D) 0.4745 0.6802 0.6726 0.0115(E) 0.4938 0.6860 0.6786 0.0110

Flamenco2 (A) 0.4361 0.6517 0.6362 0.0151(B) 0.4397 0.6554 0.6428 0.0146(C) 0.4414 0.6538 0.6403 0.0145(D) 0.4426 0.6562 0.6431 0.0139(E) 0.4429 0.6577 0.6439 0.0137

Rena (A) 0.8733 0.9013 0.8926 0.0027

H. Zeng et al. / Signal Processing: Image Communication 36 (2015) 53–62 59

information while FSIMC exploits both luminance and chro-minance information. Besides, the higher SSIM, FSIM andFSIMC scores while the lowerMOVIE score indicates the bettervisual quality. In our experiments, SSIM, FSIM, FSIMC andMOVIE are individually computed across two consecutiveviews (i.e., Vi and Viþ1) of the given multi-view videosequentially, where View Vi is used as the reference view ofView Viþ1. Then, the computed results are averaged as thesubjective result, which can effectively reflect the colorconsistency across the views of the multi-view video beforeand after applying various color correction methods.

Table 2 documents the SSIM, FSIM, FSIMC and MOVIEvalues of the uncorrected multi-view video sequences(denoted as “Uncorrected”) and the corrected ones bythe methods recently proposed by Fecker et al. [5](denoted as HM), Doutre et al. [7] (denoted as BDEPF),Fezza et al. [16] (denoted as FBCC), and the proposed SFCCalgorithm. It can be seen that the proposed SFCC algorithmconstantly achieves the highest SSIM, FSIM, FSIMC and thelowest MOVIE average scores, showing that the proposedcolor correction algorithm is able to yield better visualquality than the Uncorrected, HM, BDEPF and FBCC,respectively.

(B) 0.8747 0.9025 0.8953 0.0024(C) 0.8758 0.9036 0.8960 0.0022(D) 0.8750 0.9028 0.8952 0.0022(E) 0.8782 0.9068 0.8998 0.0019

Crowd (A) 0.1455 0.5732 0.5605 0.0208(B) 0.1478 0.5769 0.5648 0.0191(C) 0.1579 0.5816 0.5729 0.0182(D) 0.1658 0.5842 0.5756 0.0176(E) 0.1766 0.5881 0.5804 0.0169

Average (A) 0.4789 0.7001 0.6882 0.0128(B) 0.4815 0.7026 0.6922 0.0121(C) 0.4855 0.7039 0.6944 0.0117(D) 0.4895 0.7059 0.6966 0.0113(E) 0.4979 0.7096 0.7007 0.0109

3.2. Coding efficiency evaluations

Color correction is normally considered as an imagepre-processing scheme to facilitate the main task ofintended image application. In this work, we shall furtherdemonstrate that the color correction could be appreciablybeneficial to multi-view video coding (MVC), on the aspectof coding efficiency.

To show the effect on coding efficiency contributed bycolor correction, experiments have been conducted bothon the original (before correction) and on the corrected

Fig. 7. An enlarged portion of the yellow bounding box region indicated in Falgorithm.

multi-view video sequences based on the MVC referencesoftware—joint multi-view video coding (JMVC 8.5) [26]. Inour experiments, various color correction methods areapplied to the original multi-view video sequence to

ig. 3: (a) original pictures; (b) corrected pictures by the proposed SFCC

H. Zeng et al. / Signal Processing: Image Communication 36 (2015) 53–6260

reduce the color variation among the views as thepre-processing stage, followed by conducting MVC. Thetest conditions are set as follows:

1.

TabAve(B)corfromcom

S

R

F

R

C

A

Each test sequence as listed in Table 1 is encoded usingthe hierarchical B picture prediction structure [27]under a group of picture length¼16.

2.

The quantization parameter QP is set at 24, 28, 32, and36, respectively.

3.

The rate distortion optimization (RDO) is enabled. 4. The CABAC entropy coding is used. 5. The search range of motion estimation and disparity

estimation is 764.

In addition to directly compressing those uncorrectedmulti-view video sequences (denoted as “Uncorrected”),the coding result of the proposed color correction algo-rithm, SFCC, is compared with that of methods proposedby HM [5], BDEPF [7] and FBCC [16]. Fig. 8 presents theirrate distortion (RD) performance of Y, U, and V componentsexperimented on several commonly-used multi-viewvideo sequences as listed in Table 1, namely, “Race1,”“Flamenco2,” “Rena,” and “Crowd.” Note that the ordinateof those RD curves in Fig. 8, PSNR, is calculated accordingto the commonly-used way in the area of color correctionfor multi-view video—using the corrected multi-viewvideo as the reference, which is also utilized by HM [5],BDEPF [7] and FBCC [16]. Moreover, Table 3 furthersummarizes the average PSNR gains of HM, BDEPF, FBCC,and the proposed SFCC algorithm for comparison. Note

le 3rage PSNR gains of Y, U, V components of four methods: (A) HM [5],BDEPF [7], (C) FBCC [16], and (D) our proposed SIFT-flow-based colorrection (SFCC) algorithm. All the incremental differences are resulted

comparing each method with respect to the result of directlypressing the uncorrected multi-view video sequences, individually.

equences Method BDPSNR (dB)

Y U V

ace1 (A) þ0.43 þ0.58 þ0.70(B) þ0.76 þ0.92 þ1.04(C) þ0.89 þ1.22 þ1.36(D) þ1.67 þ1.64 þ1.88

lamenco2 (A) þ0.19 þ0.59 þ0.39(B) þ0.39 þ0.63 þ0.64(C) þ0.59 þ0.82 þ0.69(D) þ0.57 þ0.90 þ0.75

ena (A) þ0.48 þ0.91 þ0.63(B) þ0.58 þ1.20 þ1.45(C) þ0.52 þ0.93 þ1.25(D) þ0.68 þ1.13 þ1.36

rowd (A) þ0.20 þ0.56 þ0.43(B) þ0.27 þ0.83 þ1.14(C) þ0.36 þ1.02 þ1.39(D) þ0.47 þ1.41 þ2.52

verage (A) þ0.33 þ0.66 þ0.54(B) þ0.50 þ0.90 þ1.07(C) þ0.59 þ1.00 þ1.17(D) þ0.85 þ1.27 þ1.63

that this average PSNR gains of each color correctionmethod are respectively made with respect to the resultof the Uncorrected and measured by the commonly-usedBDPSNR as suggested in [28].

From the experimental results as shown in Fig. 8 andTable 3, it can be seen that the proposed SFCC algorithmconstantly achieves 0.85 dB PSNR gain averaged over fourtest sequences in the Y component, compared with that ofdirectly compressing the uncorrected multi-view videosequence. It is important to observe that higher PSNR gainis yielded on chrominance components U (1.27 dB onaverage) and V (1.63 dB on average) by using the proposedSFCC algorithm. This reflects the effectiveness and poten-tial of the proposed color correction approach. Experi-mental results clearly indicate that the proposed SFCCalgorithm obtains the highest PSNR improvement onaverage in Y, U, and V components respectively, showingthat the proposed color correction algorithm consistentlyoutperforms the HM [5], BDEPF [7] and FBCC [16].

3.3. Computational complexity

To evaluate the computational complexity, extensivesimulation experiments have been conducted based on aset of multi-view video sequences as listed in Table 1 toobtain the average running time. The PC used for conduct-ing these experiments is made up of 2.66 GHz Intel Core2processor and 4 GB memory. The proposed color correc-tion algorithm, SFCC, is implemented in the Matlab.

Experimental results have shown that the proposedSFCC algorithm takes 18 s per frame on average for themulti-view video with a frame resolution of 640�480.Although the current computational complexity does notallow for a real-time implementation for the proposedalgorithm, it can be performed off-line, since it is meant tobe utilized as a pre-processing stage to facilitate thefollow-up multi-view video processing as mentioned ear-lier. Moreover, it should be pointed out that the computa-tional complexity of the proposed algorithm can be furtherreduced, say, through an implementation by using the Clanguage.

4. Conclusion

In this paper, a novel color correction algorithm formulti-view video, called the SIFT-flow-based color correc-tion (SFCC), is proposed. In our approach, a set of commoncorresponding points across all the views in the multi-view video under correction are first identified by utilizingthe SIFT flow technique. Based on the detected commoncorresponding points, the average color is calculated andused as the reference color so that the color correctionmatrix for each view can be thus obtained by minimizingthe energy of difference between the color of the detectedcommon corresponding points in each view and thereference color, respectively. With the obtained colorcorrection matrix, the color correction is then performedfor each view. Experimental results have demonstrated

0 200 400 600 80032

34

36

38

40

42

Bit rate (Kbit/s)

PSN

R (d

B)

Y Component

UncorrectedHMBDEPFFBCCProposed

0 200 400 600 80034

36

38

40

42

Bit rate (Kbit/s)

PSN

R (d

B)

U Component

UncorrectedHMBDEPFFBCCProposed

0 200 400 600 80036

37

38

39

40

41

42

Bit rate (Kbit/s)

PSN

R (d

B)

V Component

UncorrectedHMBDEPFFBCCProposed

0 200 400 600 800 100032

34

36

38

40

42

Bit rate (Kbit/s)

PSN

R (d

B)

Y Component

UncorrectedHMBDEPFFBCCProposed

0 200 400 600 800 100036

37

38

39

40

41

42

Bit rate (Kbit/s)

PSN

R (d

B)

U Component

UncorrectedHMBDEPFFBCCProposed

0 200 400 600 800 100037

38

39

40

41

42

Bit rate (Kbit/s)

PSN

R (d

B)

V Component

UncorrectedHMBDEPFFBCCProposed

0 100 200 300 400 50034

36

38

40

42

44

Bit rate (Kbit/s)

PSN

R (d

B)

Y Component

UncorrectedHMBDEPFFBCCProposed

0 100 200 300 400 50036

37

38

39

40

41

42

Bit rate (Kbit/s)

PSN

R (d

B)

U Component

UncorrectedHMBDEPFFBCCProposed

0 100 200 300 400 50035

36

37

38

39

40

41

Bit rate (Kbit/s)

PSN

R (d

B)

V Component

UncorrectedHMBDEPFFBCCProposed

0 500 1000 1500 200030

32

34

36

38

40

Bit rate (Kbit/s)

PSN

R (d

B)

Y Component

UncorrectedHMBDEPFFBCCProposed

0 500 1000 1500 200032

34

36

38

40

Bit rate (Kbit/s)

PSN

R (d

B)

U Component

UncorrectedHMBDEPFFBCCProposed

0 500 1000 1500 200034

36

38

40

42

Bit rate (Kbit/s)

PSN

R (d

B)

V Component

UncorrectedHMBDEPFFBCCProposed

Fig. 8. The RD performance of multiple multi-view video sequences: (a) “Race1,” (b) “Flamenco2,” (c) “Rena,” and (d) “Crowd.”

H. Zeng et al. / Signal Processing: Image Communication 36 (2015) 53–62 61

that the proposed color correction algorithm, SFCC, caneffectively reduce the color variation inherited in themulti-view video. By further exploiting the proposed SFCCalgorithm as a pre-processing for the MVC, the coding

efficiency of the corrected multi-view video has beenappreciably increased (on average, 0.85 dB, 1.27 dB and1.63 dB increments in PSNR of Y, U and V components,respectively), compared with that of the uncorrected one.

H. Zeng et al. / Signal Processing: Image Communication 36 (2015) 53–6262

Acknowledgments

This work was supported in part by the NationalNatural Science Foundation of China under the Grants61372107 and 61401167, and in part by the High-LevelTalent Project Foundation of Huaqiao University under theGrants 14BS201 and 14BS204.

References

[1] P. Benzie, J. Watson, P. Surman, I. Rakkolainen, K. Hopf, H. Urey,V. Sainov, C. Kopylow, A survey of 3DTV displays: techniques andtechnologies, IEEE Trans. Circuits Syst. Video Technol. 17 (11) (2007)1647–1658.

[2] M. Tanimoto, M.P. Tehrani, T. Fujii, T. Yendo, Free-viewpoint TV, IEEESignal Process. Mag. 28 (1) (2011) 67–76.

[3] ITU-T Recommendation H.264, Advanced Video Coding for GenericAudiovisual Services, March 2010.

[4] J.-H. Hur, S. Cho, Y.-L. Lee, Adaptive local illumination changecompensation method for H.264/AVC-based multiview video cod-ing, IEEE Trans. Circuits Syst. Video Technol. 17 (11) (2007)1496–1505.

[5] U. Fecker, M. Barkowsky, A. Kaup, Histogram-based prefiltering forluminance and chrominance compensation of multiview video, IEEETrans. Circuits Syst. Video Technol. 18 (9) (2008) 1258–1267.

[6] B.X. Shi, Y.X. Li, L. Liu, C. Xu, Block-based color correction algorithmfor multi-view video coding, in: IEEE International Conference onMultimedia and Expo (ICME), June 2009, pp. 65–68.

[7] C. Doutre, P. Nasiopoulos, Color correction preprocessing for multi-view video coding, IEEE Trans. Circuits Syst. Video Technol. 19 (9)(2009) 1400–1405.

[8] Y.B. Chen, K.-K. Ma, C.H. Cai, Histogram-offset-based color correctionfor multi-view video coding, in: IEEE International Conference onImage Processing (ICIP), September 2010, pp. 977–980.

[9] F. Shao, M. Yu, G. Jiang, K. Chen, Z. Peng, Color correction pre-processing and chrominance reconstruction post-preprocessing formulti-view video coding, IET Image Process. 6 (2) (2012) 129–138.

[10] D.G. Lowe, Object recognition from local scale-invariant features, in:IEEE International Conference on Computer Vision (ICCV), Septem-ber 1999, pp. 1150–1157.

[11] K. Yamamoto, M. Kitahara, H. Kimata, T. Yendo, T. Fujii, M. Tanimoto,S. Shimizu, K. Kamikura, Y. Yashima, Multiview video coding usingview interpolation and color correction, IEEE Trans. Circuits Syst.Video Technol. 17 (11) (2007) 1436–1449.

[12] K. Yamamoto, R. Oi, Color correction for multi-view video usingenergy minimization of view networks, Int. J. Autom. Comput. 5 (3)(2008) 234–245.

[13] M.P. Tehrani, A. Ishikawa, S. Sakazawa, A. Koike, Iterative colourcorrection of multicamera systems using corresponding featurepoints, J. Vis. Commun. Image Represent. 21 (5–6) (2010) 377–391.

[14] C. Lu, Y. Zhang, Y. Shen, Color correction based on SIFT and GRNN formulti-view video, in: 2011 Fourth International Joint Conference onComputational Sciences and Optimization, April 2011, pp. 857–860.

[15] J.I. Jung, Y.-S. Ho, Color correction algorithm based on cameracharacteristics for multi-view video coding, Signal Image VideoProcess. (May 2012) 1–12.

[16] S.A. Fezza, M.-C. Larabi, K.M. Faraoun, Feature-based color correctionof multiview video for coding and rendering enhancement, IEEETrans. Circuits Syst. Video Technol. 24 (9) (2014) 1486–1498.

[17] C. Liu, J. Yuen, A. Torralba, SIFT Flow: dense correspondence acrossscenes and its applications, IEEE Trans. Pattern Anal. Mach. Intell. 33(5) (2011) 978–994.

[18] D. Scharstein, R. Szeliski, A taxonomy and evaluation of dense two-frame stereo correspondence algorithms, Int. J. Comput. Vis. 47 (1)(2002) 7–42.

[19] K. Mikolajczyk, C. Schmid, A performance evaluation of localdescriptors, IEEE Trans. Pattern Anal. Mach. Intell. 27 (10) (2005)1615–1630.

[20] T. Brox, A. Bruhn, N. Papenberg, J. Weickert, High accuracy opticalflow estimation based on a theory for warping, in: EuropeanConference on Computer Vision (ECCV), May 2004, pp. 25–36.

[21] A. Bruhn, J. Weickert, C. Schnorr, Lucas/Kanade meets Horn/Schunk:combining local and global optical flow methods, Int. J. Comput. Vis.61 (3) (2005) 211–231.

[22] Y.P. Su, A. Vetro, A. Smolic, Common Test Conditions for MultiviewVideo Coding, ISO/IEC JTC1/SC29/WG11 and ITU-T SG16 Q.6, JVT-U211, Hangzhou, China, October 2006.

[23] Z. Wang, A.C. Bovik, H.R. Sheikh, E.P. Simoncelli, Image qualityassessment: from error visibility to structural similarity, IEEE Trans.Image Process. 13 (4) (2004) 600–612.

[24] L. Zhang, L. Zhang, X.Q. Mou, D. Zhang, FSIM: a feature similarityindex for image quality assessment, IEEE Trans. Image Process. 20(8) (2011) 2378–2386.

[25] K. Seshadrinathan, A.C. Bovik, Motion tuned spatio-temporal qualityassessment of natural videos, IEEE Trans. Image Process. 19 (2)(2010) 335–350.

[26] Joint Video Team, Multi-view Video Coding Reference Software–Joint Multi-view Video Coding (JMVC 8.5). Available: garcon.ient.rwth-aachen.de, March 2011 (Online).

[27] P. Merkle, A. Smolic, K. Muller, T. Wiegand, Efficient predictionstructure for multiview video coding, IEEE Trans. Circuits Syst. VideoTechnol. 17 (11) (2007) 1461–1473.

[28] G. Bjontegaard, Calculation of Average PSNR Differences betweenRD-Curves, Document VCEG-M33, VCEG 13th meeting, Austin,Texas, USA, April 2001.