Image Processing Theory, Tools & Applications

de e ase oetection in old movies

H. Ammar-Badri, A. Benazza-BenyahiaUnite de Recherche en Imagerie Satellitaire et ses Applications,

SUP'COM Tunis,e-mail: heyfa.amar@ gmail.com, benazza.amel supcom.rnu.tn

Abstract-In this paper, we are interested in detectingblotches in old movies. The novelty of our approach istwofold. Firstly, we study the pertinency of operating thedetection in the wavelet transform domain. Secondly, wepropose to resort to statistical outlier test in order tolocalize the underlying artifacts.

Index Terms-Old movies restoration, wavelet trans-form, blotch, multiple outliers detection, Hampel test, Boxand Cox transform.

I. INTRODUCTION

Old movies suffer from several deteriorations due tothe storage media, the archiving conditions, excessivemishandlings and the aging of films. Very often, they areso important that they dramatically reduce the interestand the pleasure of watching the movies. It is importantto note that these movies represent an amount of valuabledata for scientific, cultural, social and economical pur-poses. Therefore, it is necessary to preserve them thanksto restoration techniques in order to produce versions atincreased visual quality. Analog restoration techniquesexist but they involve overwhelming costs and require amanual how-to-know of experts and very long processingdelays. This is the reason why a great attention waspaid to digital techniques after the successful digitalrestoration of the Walt Disney's movie "Snow White"in 1993 [1]. The objective is to design fully digital andautomatic restoration techniques which could process theold movies at a reasonable financial cost. To this respect,several works were conducted as the pionneering onesof A. C. Kokaram [2], the European project BRAVA(Broadcat Restoration of Archives through Video Anal-ysis). In the context of the latter project, all the defaultswhich could contaminate the old movies were exhaus-tively listed with an accurate nomenclature: a referencecodebook containing around 190 defaults was generated.The diversity of the defaults makes difficult and delicatethe image processing task. Therefore, it is used to firststudy the problem of detection of a single artifact ata time. In this work, we are interested in detecting

specific artifacts called blotches. They correspond tobright and dark spots introduced either by dirt particlesas dust, hair or the loss of gelatin due to mishandling oraging of film. Blotches present three main characteristics.Firstly, they are local artifacts since they contaminatelimited coherent areas with almost the same brightness.However, the shape and the size of these areas arerandom. Secondly, blotches are temporally independentand hence, they hardly ever occur at the same positionin successive images. Finally, the intensity of a blotchgreatly differs from those of the uncorrupted neighbors.Several blotch detection methods were developed as theheuristic ones [2], [3], [4], [5]. Bayesian detectors werealso designed [3]. The subsequent review of the state-of-art will indicate that the reported methods operate in thespatial domain. Our contribution relies on the choice ofthe Wavelet Transform (WT) domain for performing theblotch detection. Our choice is motivated by the abilityof the WT to localize transitions of the signal at differentscales.This paper is organized as follows. In Section II, we givea brief review of the main blotch detection methods. InSection III, we describe very useful background on WTand multiple outlier statistical tests. In Section IV, thenew method we propose is detailed. Finally, in SectionV. we provide the performances of our method on bothartificial and real sequences and in Section VI, someconclusions are drawn.

II. A BRIEF REVIEW ON BLOTCH DETECTION

In order to digitally process the images, the film is firstof all digitized according to an appropriate protocol (e.g.accurate adjustment of the spatial and temporal samplingparameters, suitable choice of the quantization step).Therefore, the goal is to restore the digital video: anestimate I- of the original video 1, should be computedfrom the observed degraded video I, In the case ofblotches, the observation model at each frame k at aspatial location n could be expressed as:

Ir (n, k) =I1(n, k) + b(n,k) (1)

978-1-4244-3322-3/08/$25.00 ©2008 IEEE

where b(n, k) corresponds to a blotch which takes ran-dom values within [0,255] with a probability Pb. Gen-erally, a two-step procedure is followed to remove theblotches. Firstly, the blotches are detected: a binary maskwhich indicates if each spatial location in each framewhether or not is a blotch is then generated. Secondly,the corrupted intensities at the detected locations shouldbe corrected through a spatio-temporally interpolation.The first detectors were heuristically developed by ex-ploiting the three aforementioned characteristics and thetemporal evolution of the sequence. The most simpleblotch detector is the pixel-based Spike-Detector-Index(SDI)[2], [5]. It starts by exploiting the temporal evolu-tion of the sequence by computing the backward motionfield db(n, k) = (db,I (n, k), db,2(n, k))T and the forwardmotion field df(n, k) - (df,(n,k> df,2(n, k))T. Therelated displaced frame differences DFDb and DFDfare easily derived as follows:

DFDb(n, k) = Ir(fl, k) -Ir(fl - db(n, k), k - 1)DFDf(n, k) Ir,(l, k) - (n- df(n, k), k + 1)

(2)Then, the SDI matrix at time k is defined by:

SDI(n, k) - I - (DFDb(n, k) - IDFDf(n, k))DFDb(n, k) + DFDf (n, k)

(3)if IDFDb(n, k) > t1 or DIFDf(n, k) > t1 andotherwise, SDI(n, k) - 0 where t1 is a low thresholdwhich overcomes problems when DFDb(n, k) andDFDf (n, k) tend to zero. The SDI is limited to valuesbetween 0 and 1 and the pixel (n, k) is judged corruptedif it presents a temporal discontinuity w.r.t. both the pastand the future, or equivalently, if:

SDI(n, k) > (4)

where T is a given threshold [2]. In [4], R. Storey em-ploys criteria reflecting the temporal adjacency to detectblotches. However, the performances are only satisfyingin the stationary areas of the sequence. Besides, non-linear spatio-temporal filters have been proposed whoseresponses are adjusted according to the local anisotropicintensity continuity [6], [7]. For instance, in [6], a com-bination of a set of median filters is employed withoutexploiting the motion information. Methods employingmathematical morphology tools have been also investi-gated [8], [9], [10]. Pardas et al. have applied openingand closing operators. In fact, the initial goal was theremoval of impulse noise but the authors have mentionedthe extension to the restoration of old movies [8]. In asimilar manner in [9], openings and closings controlled

allow to detect local extrema of the intensity. Then,a refinement of the artifact localization has been alsoobtained by a further analysis of the temporal continuity.A Bayesian framework has also been retained: a 3Dauto-regressive model is envisaged to describe the spatio-temporal evolution of the sequence Ii [3].However, it is worth noting that all the reported methodsoperate in the image domain. The objective of this paperis to investigate if the use of the WT domain couldimprove the blotch detection performances.

III. REQUIRED BACKGROUND

A. Motivation

The blotches are local artifacts randomly located inthe image. Bright spots occur in dark areas and appearas local maxima whereas dark spots correspond to localminima in bright regions of the frame. Besides, thespatial support and its shape randomly vary both in spaceand in time. Consequently, the problem boils down todetect an information which is well spatially and tem-porally localized with a variable spatial resolution. Ourrationale is to use the WT for its appealing properties ofscale-space localization. Before presenting our method,we will give a brief sketch on the WT.

B. Reminders of WT

WT is a pyramidal transform which consists in decom-posing any original image in frequency subbands givingrise to a multiresolution representation [11]. A dyadicorthogonal WT in 2 (R) is entirely defined by a mother-wavelet function (t) in 2(IR) or a scaling function 0b(t)in 22(R) that satisfy the following equations:

Vt c R, 0(t) -- 2E h (k) 3(2t - 1),loWi

t) -/2 1, hI(1)02t -1),1CEz

(5)

(6)

where [t, It E p2(Z). The set {2-j/2 b(2-it- hllconstitutes an orthonormal basis of 2 (IR) if the para-unitary conditions hold for every couple (in, in') in{0, 1}1:

W

jHT(w + P 27)Hm(+P 2w) 26Trl-mnL 2 +p 2p=O,1

(7)

where Hm denotes the Fourier transform of hm. Con-sequently, ho is a low-pass filter whereas h, is a high-pass filter. Thus, the decomposition of any ID signal

f E L(IR) over J resolution levels could be written asfollows..

f(t) - E aj(1)2J/2 (2jt - 1)lE - -j/2d7 t 0T/j_7(8)+ Z L L 2i/Gd(1)f(2t - 1). (j>J l7z

The ID coefficients aj correspond to a coarse versionof f and the dj are its details at scale j. Concernig2D signals of L2 (R 2), the decomposition is appliedin a separable way. At each resolution level j, 4 sub-images are obtained: the approximation subband cq 0)the detail subbands oriented horizontally Cf1), verticallycj' 2) and diagonally Cf'3. Then, the decomposition isiterated on C.'°) An extension to the case of imagesspatially sampled belonging to 12(Z2) leads to the samekind of multiresolution representation [12]. It is worthnoting that a great choice of filters (or mother-wavelets)is offered to the user [11].

C. Background on outliers statistical tests

1) Definitions: An outlier-region corresponds to sev-eral observations that come from several distributionsdiffering from a given target distribution. Many workshave been devoted to the case of a normal target distribu-tion which leads to the following definition [13]. For anyreal-valued parameter a within 10, 1 [, the a-outlier regionof the normal target distribution NV(p, (T2) is the set ofobservations {x : X - > ZI-a/2} where Zq denotesthe q-th quantile of the I\(0, 1) distribution. Given asample XN - {XI,... XN of size N and a specifiedreal aN lying in ]0, 1 [ a Multiple Outlier Detector(MOD) aims at identifying the K observations that areaN-outliers w.r.t. K(,u, a2) where the positive numberK, p and a2 are assumed to be unknown although Kshould not exceed N/2.

2) Principle of multiple outliers detectors: Generally,based on XN, a MOD proceeds by specifying a lowerbound L(XN, aN) and an upper bound R(XN, aN)such as the set of all the aN-outliers is given by

O(XN, aN) (-oc, L(XN, aN)1 U [R(XN, aN), °°).(9)

Consequently, a key issue is the computation of thebounds L(XN, aN) and R(XN, aN). To this respect,many methods have been investigated giving rise toseveral MODs [13].

3) The Hampel identifier: Among the most knownMODs, more attention was paid to the Hampel MOD

associated with the following choice:

L(XN, aN) med[XN1 - mad[g(N, aN)XN1R(XN, aN) med[XNI + mad[g(N, aN)XNI

(10)where medQ.) and mad(.) respectively denote the medianand the median absolute deviation operators':

m X(Fr(N+1)/21 ) +X(FrN/2 +1)med[XN]A 2

mad[XN]-med[ XI - med[XN ,..., |XN - med[XNI I(1 1)

The positive univariate function g(.) allows to standard-ize the MOD by requiring that the probability that in a iidnormal sample, no outliers (or regular) are identified witha probability 1-av where a is chosen by the statistician.This amounts to set g so as:

-IX- med[XN]Prob r<g(N, aN))1 1-a (12)mad[X2NIwith aN 1 - (1 _ a)I/NI. Very often, it is used to seta = 0.05. Therefore, it is easy to derive the values ofg( ) w.r.t. N.

IV. PROPOSED APPROACH

In this section, we present an approach for blotch de-tection based only on spatial information. Unlike the SDIapproach, we do not exploit the temporal information ofthe movie.

A. PreprocessingWe come back to the problem of blotch detection by

resorting to a MOD on a multiresolution representationof the input frame I, More precisely, a WT is appliedto the kth frame I of the digitized old film. For thisframe, at each resolution level j, the wavelet coefficientsc(rok) are generated for o _ 1... 3. The principle is tocarry out a statistical analysis in each subband (j, o, k) inorder to locate the irregular wavelet coefficients having asignificant magnitude regarding to those encountered intheir neighborhood. This analysis is based on the HampelMOD. However, a preprocessing has to be performed tomake the input frame of the MOD normally distributed.To this respect, the power transform Box-Cox is appliedseparately to the magnitudes of the wavelet coefficientsc(r,ok) with 1 < o < 3 [14]. It is worth noting that the

symmetry of the distribution of the coefficients c 'rok)allows us to only consider the magnitude information.In what follows, we will designate the transform coeffi-cients by C(-rk)

ithe operator -] denotes the rounding operation.

B. Level-dependent detection

The blotch detection problem could be considered asa multiple outliers detection problem. We assume thata blotch corresponds to observations (rk)which havelocally a high magnitude. As a consequence, by definingan accuracy level aN1, and a neighborhood XN(°) of

size N(0 of the positive observations <~~,a blotchis identified to be an aN( -outlier only if it exceedsthe upper bound R(XN(O) aN(J)). More precisely, thecurrent neighborhood XN(° iS associated with a rect-

3

angular sliding window of size N =. x___where N( and N2 are adjusted by the user. It isimportant to emphasize that an horizontal shape couldbe obtained by selecting N{1) < N in order to fit thedominant orientation of the coefficients in the subbands(j, 1, k). Similarly, vertical windows (N{2) > N 2 ) arerecommended for the subbands (j, 2, k) and square ones(N(2)= N(2)) for the diagonal subbands (j, 3, k). Fur-thermore, by varying the size N(0) of the neighborhoodwith the resolution level j, we aim at adapting the scaleof the blotches to the scale of the analyzed subband.As a result of such adaptive Hampel MOD, we obtain abinary mask indicating the outliers positions.

C. Gathering the multiscale information

Many strategies could be selected to merge the infor-mation of the binary masks of all the (j, o, k) subbands inorder to build the binary mask at the full resolution. Forinstance, coarse-to-fine approaches could be designed.However, for the sake of simplicity, we will resort toa very basic strategy which consists in retaining all theidentified coefficients. Formally, a pixel n in I, (n; k) isa blotch if at least one of its related preprocessed waveletcoefficients c(r'o') is identified as an outlier.

V. EXPERIMENTAL RESULTS

A. Experimental setupExperimental tests were conducted on two kinds of

sequences. The first one includes artificially degradedimages whereas the second type corresponds to reallydegraded ones 2. An example of the first category, takenfrom the Frankstein sequence, is displayed in Figure 1.Figures 2 and 5 show really blotched frames extractedrespectively from the Frankstein and Biplane movies. Forour benchmarking, we have retained the SDI approach.

2The test frames were directly taken from the CD-Rom includedin [2].

Concerning the tests performed on artificially degradedframes, we report the results under the hypothesis of ex-act motion compensation, since SDI is a spatio-temporaldetector. The SDI threshold t1 used for these tests isthe same for all b. For the test performed on the realdegraded frames, the parameters of our detector andthe SDI one were adjusted in order to give the bestperformances. The blotch detector based on the HampelMOD uses the Daubechies wavelet with 10 vanishingmoments and we report the results for a two-level WT(J - 2). The sizes of the underlying sliding windowswere empirically fixed as follows:

N(1) -128 N(1)- 90|1 - : 2 -j N(2)90, N(2)- 128 (13)

1 - 2 -

NN(3) =107, N 3) =107for the frames of size 256 x 256, and:

J (1) _ 256 N(1) -180t1 21N(2)_180 N(2) - 256 (14)1 - 2-

N(3)-214 N(3) -2141 2,

for the frames of size 512 x 512.

B. Performance evaluation

Concerning the really degraded image, we only per-form a visual inspection of the binary masks provided byour approach and the SDI one in Figures 3, 4, 6 and 7.We can note that the blotches are detected. However, wecan note that many false alarms occur with our detectorprobably due to the very basic and "tolerant" detectionwe have retained to gather the results of the MOD acrossthe scales.Objective performance evaluation are carried out onthe artificially degraded frames. Table I provides thedetection performances in terms of correct detectionprobability P, when applying our approach and the SDIone. We note that our method yields to a significantimprovement of the correct detection probability P, forall the considered frames whatever is the value of theblotch probability P The average performance of ourdetector over the 10 frames is about 0.26; we note animprovement of about 0.13 compared to the SDI detectorwhose average performance was about 0.13.

VI. CONCLUSION

This paper is a preliminary study of the relevanceof the wavelet transform domain for the detection ofblotches in degraded movies. The novelty of our ap-proach relies on the use of a statistical outlier detectiontest. Indeed, we have accounted for the properties of the

wavelet coefficients to detect signal jumps at differentscales, and the efficiency of the statistical analysis tolocalize these jumps. Experimental results indicate thatthe blotch detection we proposed yields to high correctprobabilities even if no temporal information is available.Besides, unlike the conventional methods as the SDI, nothreshold adjustment is required. There are many aspectsof this preliminary study which can be improved. Inparticular, it seems interesting to refine procedure ofinformation fusion through the scales and incorporatethe temporal information within the statistical multipleoutlier detector.

REFERENCES

[1] D. Turner, "Engineers developing technology movie classics,"Los Angeles Business vol. 17 no. 32 pp. 30 August 1995.

[2] A. C. Kokaram. Motion picture restoration: digital algorithmsfor artefact suppression in degraded motion picture film andvideo, Springer Verlag, 1998.

[3] A. C. Kokaram "On missing data treatment for degraded videoand film archives: a survey and a new Bayesian approach" IEEETrans. on Image Processing, vol. 13, no. 3, pp. 397-415, 2004.

[4] R. Storey, "Electronic detection and concealment of film dirt,"SMPTE Jouarnl, pp. 642-647, June 1985.

[5] A. Hanjalic, G. C. Langelaar P. M. B. van Roosmalen, J.Biemond and R. L. Lagendijk, Image and Video DatabasesRestoration, Watermarking and Retrieval

[6] G. R. Arce,"Multistage order statistics filters for image se-quence processing" IEEE Trans. on ASSP vol. 39 no. , pp.1146-1163, May 1991.

[7] T. Saito, T. Komatsu, T. Ohuchi and T. Hoshi, "Practicalnonlinear filtering for removal of blotches from old film," Proc.of the IEEE ICIP, Kobe, Japan, pp. 164-168 October 1999.

[8] M. Pardas, J. Serra and L. Torres, "Connectivity filters for imagesequences" Proc. of SPIE Image Algebra and MorphologicalImage Processing III vol. 1769 P. D. Gader E. R. DoughertyJ. C. Serra, Editors, pp. 318-329, June 1992.

[9] E. Decenciere Ferrandiere, S. Marshall and J. Serra, "Appli-cation of the morphological geodesic reconstruction to imagesequence analysis" IEE Proc. of Vision Image a;nd SignalProcessing, vol. 144, no. 6, pp. 339-344, 1997.

[10] L. Tenze, G. Ramponi and S. Carrato, "Blotches correction andcontrast enhancement for old film pictures," Proc. of the IEEEICIP, Vancouver, Canada, September 2000.

[ 1] S. G. Mallat A wavelet tour of signal processing, AcademicPress San Diego 1998.

[12] 0. Rioul, "A discrete-time multiresolution theory unifyingoctave-band filter banks, pyramid and wavelet transforms",IEEE Trans. ASSP, June 1990.

[13] L. Davies and U. Gathers, "The identification of multipleoutliers," Journal of the American Statistical Associatio', pp.782-792, 1993.

[14] E. P. G. Box and D. R. Cox "An analysis of transformations,"Journal ofRoyal Statistical Society, Series B (Methodological),vol. 26, no. 2, pp. 211-252, 1964.

TABLE ICORRECT DETECTIONS PROBABILITIES ON ARTIFICIALLYDEGRADED IMAGES: SDI APPROACH VS OUR APPROACH

Pb SDI Our approach0.01 0.07 0.290.02 0.16 0.320.03 0.19 0.340.04 0.13 0.260.05 0.14 0.310.06 0.1 0.220.07 0.18 0.240.08 0.12 0.20.09 0.14 0.230.1 0.09 0.24

Fig. 1. Artificially degraded frank frame of size 256 x 256, Pb =0.03.

Fig. 2. Really degraded frame Frankestein of size 256 x 256.

Fig. 3. Blotch positions detected in the really degraded Frankensteinframe by our approach.

Fig. 4. Blotch positions detected in the really degraded Frankensteinframe by SDI approach.

Fig. 5. Really degraded Biplane frame of size 512 x 512.

Fig. 6. Blotches positions detected in the really degraded Biplaneframe by our approach.

Fig. 7. Blotch positions detected in the really degraded Biplaneframe by SDI approach.