nonlocal variational models for inpainting and interpolation
TRANSCRIPT
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 1–65
Mathematical Models and Methods in Applied SciencesVol. 22, Suppl. 2 (2012) 1230003 (65 pages)c© World Scientific Publishing CompanyDOI: 10.1142/S0218202512300037
NONLOCAL VARIATIONAL MODELS FOR INPAINTINGAND INTERPOLATION
PABLO ARIAS∗, VICENT CASELLES†, GABRIELE FACCIOLO‡,VANEL LAZCANO§ and RIDA SADEK¶
Department of Information and Communication Technologies,Pompeu Fabra University, Tanger 122-140,
Barcelona 08018, Spain∗[email protected]
†[email protected]‡[email protected]
§[email protected]¶[email protected]
Received 8 February 2012Accepted 29 February 2012
Published 23 July 2012Communicated by N. Bellomo and F. Brezzi
In this paper we study some nonlocal variational models for different image inpaintingtasks. Nonlocal methods for denoising and inpainting have gained considerable attentiondue to their good performance on textured images, a known weakness of classical localmethods which are performant in recovering the geometric structure of the image. Wefirst review a general variational framework for the problem of nonlocal inpainting thatexploits the self-similarity of natural images to copy information in a consistent wayfrom the known parts of the image. We single out two particular methods dependingon the information we copy: either the gray level (or color) information or its gradient.We review the main properties of the corresponding energies and their minima. Thenwe discuss three other applications: we consider the problem of stereo inpainting, somesimple cases of video inpainting, and the problem of interpolation of incomplete depthmaps knowing a reference image. Incomplete depth maps can be obtained as a result
of stereo algorithms, or given for instance by Time-of-Flight cameras (in that case theinterpolated result can be used to generate the images of the stereo pair). We discuss thebasic algorithms to minimize the energies and we display some numerical experimentsillustrating the main properties of the proposed models.
Keywords: Image inpainting; variational models; exemplar-based; stereo inpainting;video inpainting.
AMS Subject Classification: 68U10, 35A15, 65D05, 65C50
†Corresponding author
1230003-1
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 2–65
P. Arias et al.
1. Introduction
Image inpainting, also known as image completion or disocclusion, is an activeresearch area in image processing. The purpose of inpainting is to obtain a visuallyplausible image interpolation in a region in which data are missing due to problemsoccurred during acquisition, damage or occlusion. It has become a standard toolin digital photography for image retouching and restoration (e.g. the removal ofscratches in old photographies) and it is the object of intensive research in order todispose of an effective tool for video editing (e.g. for object elimination).
The recent commercial interest in exhibiting 3D movies or other events likesports or music in theaters has motivated the development of inpainting tools toassist in many tasks like the acquisition of 3D data or post-production, e.g. to elim-inate unwanted objects like rigs or cables which may be unavoidable during filming.This application is a source of new problems like the interpolation of depth data,either acquired by Time-of-Flight cameras or computed using stereo algorithms,in order to have complete depth maps for depth-based image rendering and 3Dvideo content generation. Another example is given by the problem of stereo imageinpainting where the reconstructed missing information in pairs of stereo images hasto look like the projection of a real 3D object and produce a depth-consistent percep-tion. Besides its numerous applications to image and video editing, the inpaintingproblem is of theoretical interest since its analysis involves the understanding of theself-similarity present in natural images, visible in repetitive geometric and texturepatterns that appear in almost any image.
Our purpose in this paper is to review some basic variational models for imageinpainting and video editing and extend them to the new contexts of applica-tion. We will pay attention to the precise mathematical formulation of the mod-els. But before that, let us give a brief review of the most popular inpaintingmethods.
Review of inpainting methods. Although the number of works is too big tomake justice to all of them, most inpainting methods found in the literature canbe classified into two groups: geometry- and texture-oriented methods. Both typesof techniques were introduced almost at the same time,61,35 but texture-orientedmethods have proved to be more adapted to produce natural looking images, evenif ideas of both of them reveal fruitful in order to produce state-of-the-art results.
This classification represents also the two main ideas underlying most imageinpainting works. In geometry-oriented methods images can be modeled as functionswith some degree of smoothness, expressed for instance in terms of the curvature ofthe level lines or the total variation of the image. Originally introduced in Ref. 61,they take advantage of this structure and interpolate the inpainting domain by con-tinuing the geometric structure of the image (its level lines, or its edges), usuallyas the solution of a (geometric) variational problem or by means of a partial differ-ential equation (PDE). Such PDE can be derived from variational principles, as forinstance in Refs. 61, 10, 27, 28, 38, 60, or inspired by phenomenological modeling
1230003-2
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 3–65
Nonlocal Variational Models for Inpainting and Interpolation
as in Refs. 13, 79, 18. They are local in the sense that the associated PDEs, oncediscretized, only involve interactions between neighboring pixels on the image grid.An implication of this is that among all the data available in the image, they onlyuse that around the boundary of the inpainting domain. For instance, in the case ofvariational second-order energy functionals such as the Elastica,63,61 the boundaryinformation is given by the restriction of the image to the boundary of the inpaint-ing domain together with the direction of the incoming level lines. The use of thisinformation is of relevance for a good continuation of the geometric structure of theimage. This idea has been exploited in some way or another in the above references.
These methods show good performance in propagating smooth level lines orgradients, but fail in the presence of texture. They are often named as structure orcartoon inpainting.
Texture-oriented (also called exemplar-based) inpainting was initiated by thework of Efros and Leung35 on texture synthesis using non-parametric samplingtechniques (parametric models have also been considered, e.g. Ref. 58). In theseworks texture is modeled as a two-dimensional probabilistic graphical model, inwhich the value of each pixel is conditioned by its neighborhood. They exploitthe self-similarity prior by directly sampling the desired texture to perform thesynthesis. The value of each target pixel x ∈ O is copied from the center of a(square) patch in the sample image, chosen to match the available portion of thepatch centered at x. As opposed to geometry-oriented inpainting, these so-calledexemplar-based approaches, are nonlocal: To determine the value at x, the wholeimage may be scanned in the search for a matching patch. The prior self-similarity isone of the most influential ideas underlying the recent progress in image processing.In particular, it had a strong influence in image denoising through the paper Ref. 23(see also Refs. 42, 68–70).
Exemplar-based methods (with various modifications) have been extensivelyused for inpainting.17,34,30,66 They provide impressive results in recovering tex-tures and repetitive structures. However, their ability to recreate the geometrywithout any example is limited and not well understood. Different strategieshave been proposed for combining geometry and texture inpainting. Some relyon human intervention for constraining the geometry.77 Others usually decom-pose the image in structure and texture components. The structure is recon-structed using some geometry-oriented scheme, and this is used to guide the textureinpainting.14,34,51,26
Variational models. The synthesis of both types of methods is a trend in currentresearch. Variational models in particular are appropriate for combining the mainfeatures of local and nonlocal methods.
As pointed out in Ref. 31 the problem of exemplar-based inpainting can bestated as that of finding a correspondence map ϕ which assigns to each location xin the inpainting domain O (a subset of the image domain Ω, usually a rectanglein R
2) a corresponding location ϕ(x) ∈ Oc := Ω\O, the domain where the image
1230003-3
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 4–65
P. Arias et al.
Fig. 1. (Color online) Inpainting problem. Left: on a rectangular image domain Ω, missing datau in a region O has to be reconstructed using the available image u over Oc := Ω\O. A patchcentered at x ∈ O is denoted by pu(x). The set of centers of incomplete patches is eO := O + Ωp,the latter being the patch domain. The center image shows a completion found with patch NL-medians, a scheme derived from the general formulation presented in this work. Right: the resultingcompletion is a patchwork built by copying arbitrarily shaped regions from Oc. The red curvesshow the boundaries between the copied regions.
u is known (see Fig. 1). The unknown part of the image u|O is then synthesizedusing the correspondences ϕ by u|O(x) = u(ϕ(x)), x ∈ O. The filling-in strategy ofRefs. 35, 82 can be regarded as a greedy procedure (each hole pixel is visited onlyonce) for computing a correspondence map. The results obtained are very sensitiveto the order in which the pixels are processed.30,66,43
To address this issue, in Ref. 31 the authors proposed to compute the corre-spondence map by minimizing the energy functional
E(ϕ) =∫
O
∫Ωp
|u(ϕ(x + h)) − u(ϕ(x) + h)|2dhdx, (1.1)
where Ωp is the patch domain (centered at (0, 0)), and u is the known image definedin Oc. The unknown image is computed as u(x) = u(ϕ(x)), for x ∈ O. Thus ϕshould map a pixel x and its neighbors in such a way that the resulting patch isclose to the one centered at ϕ(x). Additionally, in Ref. 31 the authors proposed toadd a term penalizing the deviation of the correspondence map from translations,thus imposing a rigidity constraint on ϕ. These ideas have been the subject offurther analysis by Aujol et al.,8 proposing extensions and proving the existenceof solutions (of a related model) in the set of piecewise constant roto-translationmaps.
The energy (1.1) is non-convex and no effective way to minimize it is known. Thesame could be said for the models proposed in Ref. 8. Other authors have addressedthe determination of a correspondence map looking for simpler optimization prob-lems. In Ref. 55 the problem is formulated as a probabilistic inference on a graphicalmodel. Using a message passing algorithm the authors efficiently compute a coarsecorrespondence map.
One of the most effective optimization strategies relies on a relaxation of theproblem of (1.1),83,68,52,5,7 where the unknown variables are the correspondencemap and the image to be reconstructed, which appears as an auxiliary variable. Theresulting algorithm can be regarded as an alternate optimization in both variables
1230003-4
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 5–65
Nonlocal Variational Models for Inpainting and Interpolation
of the energy
E(u, ϕ) =∫
eO
∫Ωp
|u(x+ h) − u(ϕ(x) + h)|2dhdx, (1.2)
where O := O+Ωp refers to the set of centers of patches that intersect the inpaintingdomain O (see Fig. 1). The unknown image is now determined as part of theoptimization process, and is not constrained to be u(x) = u(ϕ(x)). Although thisrelaxation is still non-convex, the alternating minimization scheme coincides withthe well-known Expectation–Maximization (EM) algorithm which converges to acritical point of the energy.84,6 This approach was also used in the context of texturesynthesis in Ref. 56.
The works on texture synthesis of Refs. 35, 82 have also influenced the devel-opment of nonlocal methods for other applications, such as denoising,23,9 super-resolution,72 and regularization of inverse problems.42,69 As opposed to the case ofinpainting, in these contexts the denoised pixel value is estimated from many loca-tions in the image, typically as a (nonlocal) average. This results in replacing thecorrespondence map by a weight function w : Ω × Ω → R, with Ω being the imagedomain. For each x, w(x, ·) weights the contribution of each image location to theestimation of x. Inspired by regularization techniques used in the context of graphsor discrete data and trying to formulate the nonlocal means denoising method23 asa variational model, Gilboa and Osher42,41 proposed the following functional
Ew(u) =∫
Ω
∫Ω
w(x, y)(u(x) − u(y))2dydx, (1.3)
which can be considered as a nonlocal version of the Dirichlet integral. The weightsw(x, y) are considered as known and measure the similarity of the given imagein the neighborhoods of x and y. With a proper choice for them (see (2.2)), thenon-local means formula can be seen as the first step of Jacobi’s iterative methodfor solving the Euler–Lagrange equation of (1.3). Partly motivated by the ideaof getting automatically the exponential weights that appear in nonlocal means, aslightly different formulation was proposed in Ref. 53 where the energy is written as
Eκ,g(u) =∫
Ω
∫Ω
κ(|x− y|)g(
(u(x) − u(y))2
τ2
)dydx, (1.4)
where κ is a positive localization kernel, g : R+ → R is an increasing function, andτ > 0. By appropriately choosing the kernel κ and the function g one can get thebilateral filter,78 the Yaroslavsky neighborhood filters,86 or the SUSAN filter.75 Bytaking
g
(Gσ ∗ (u(x+ ·) − u(y + ·))2
τ2
)= 1 − exp
(−Gσ ∗ (u(x+ ·) − u(y + ·))2
τ2
),
where Gσ is a Gaussian kernel and Gσ ∗ (u(x + ·) − u(y + ·))2 =∫
R2 Gσ(h)· (u(x + h) − u(y + h))2 dh, (using a periodic extension of u to give sense to theintegral), instead of g
( (u(x)−u(y))2
τ2
)in (1.4), one automatically gets exponential
1230003-5
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 6–65
P. Arias et al.
weights like in the nonlocal means filter, although averaged by Gσ. Other variationalapproaches for nonlocal denoising have also been proposed in Refs. 9, 21, 69, 70.
1.1. The content and the organization of the paper
The paper is essentially divided into three parts. In the first part (Secs. 2–7) wereview a general nonlocal variational framework for image inpainting, studying someof its qualitative features, its algorithmic aspects, and showing some experiments. Inthe second part (Secs. 8 and 9), we study the problem of simultaneous inpainting ofstereo images and the inpainting of disparity maps, a tool both for stereo inpaintingand for the interpolation of complete disparity maps from sparse disparity data. Inthe third part (Sec. 10), we consider a simple case of the problem of video inpaintingbased on the propagation of an inpainted frame.
Inpainting models and their analysis. In Refs. 5, 7, 6 we proposed a varia-tional framework for image inpainting. As in the model (1.3) we encode the imageredundancy as a nonlocal weight function w(x, y) measuring the similarity of thepatch around x ∈ O with the patch around y ∈ Ω\O. Recall that in (1.3) weightsw are considered as known and remain fixed through all the iterations. While thismight be appropriate in applications where they can be estimated from the noisyimage, in the image inpainting scenario, the weights are not available and have tobe inferred together with the image (as in Refs. 69, 72, 5, 7, 6). Thus, we considertwo unknowns, the image u and the weights w. Instead of prescribing explicitly theGaussian functional dependence of w with respect to u, we do it implicitly, as acomponent of the optimization process. In Sec. 2 we review the proposed models,which are written in terms of a Gibbs energy functional with temperature parameterT > 0, or its limit case T = 0, which corresponds to a relaxation of (1.2).
The proposed framework is rather general and allows the derivation of differentinpainting models. By choosing different distances to compare patches we singledout in Ref. 7 four inpainting models and we compared them experimentally. Thiswas complemented in Ref. 6 with a theoretical study of two models contained in thegeneral Gibbs energy formulation. The two models are the patch NL-means, that isbased on a patch-based formulation of (1.3), and the patch NL-Poisson model whichreplaces the functions by their gradients in the patch NL-means energy. In Sec. 3 wereview the main results in Ref. 6. In particular, the existence of minima and theirregularity for both models. We also review the existence of optimal solutions of theΓ-limit as T → 0 of the patch NL-means energy, which is a relaxation of the energy(1.2). In particular, there exist optimal solutions of the limit functional which arecorrespondence maps which also minimize (1.2) (a model computationally studiedin Refs. 83, 5, 52, 7). This case is the most relevant for the image inpainting appli-cation. Of particular interest is the regularity of the correspondence map obtainedby minimizing (1.2). We review the mild regularity result proved in Ref. 6, namelythe existence of optimal correspondence maps ϕ which are uniform limits of mapsof bounded variation.
1230003-6
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 7–65
Nonlocal Variational Models for Inpainting and Interpolation
Finally, in Sec. 5 we review the analysis of the alternating optimization algo-rithm and we study an efficient algorithm to minimize (1.2) with respect to ϕ
keeping u fixed, called PatchMatch.11 The most time-consuming step in the mini-mization of (1.2) is the computation of optimal matchings between patches in theinpainting domain and patches in the region of available data. Recently Barneset al.11 introduced the PatchMatch, an efficient algorithm based on heuristics tosolve the problem of matching patches between images. We proved in Ref. 6 itsprobabilistic convergence giving a bound on its convergence rate.
To conclude the presentation of inpainting models, we display in Sec. 7 someexperiments to illustrate the main features of the algorithms and the resultsobtained with them.
Other applications. The rest of the paper is devoted to the application of theabove ideas to three different problems: the inpainting of stereo images, the problemof video inpainting and the interpolation of depth data.
Stereo image inpainting81,45,46 addresses the reconstruction of missing informa-tion in pairs of stereo images with the constraint that their reconstructed partshave to look like the projection of a real 3D object and produce a depth-consistentperception. In Sec. 8 we discuss the approach to this problem introduced in Ref. 46where the problem is separated in two steps: the interpolation of the depth mapfollowed by the depth-consistent inpainting of the stereo pair. This permits thesimultaneous and depth coherent inpainting of stereo images, with disparity relatedcorresponding points being inpainted with the same color. We discuss the con-nections of the first step with the interpolation of depth maps studied in Sec. 9.Finally, we discuss the possibility to address the second step using a variationalframework.59
In Sec. 9 we consider the problem of interpolation of depth, or disparity (whichis inversely proportional to depth), data u given a partial depth, or disparity, mapand a reference image I, which is used to assist in the inpainting task. If the depthdata has been acquired by a Time-of-Flight camera,47 it may be noisy or unreliableand one needs to filter it and interpolate a complete one. Typically, an image isalso acquired with a regular camera afixed to the Time-of-Flight camera. Takingthis image as the left one of the stereo pair, the depth map can be used to generatethe right view of the scene for 3D display. Incomplete depth maps also appearas a result of stereo algorithms and its interpolation becomes necessary.33 In thisproblem the similarity weights are computed using the reference image and theenergy functional imposes that depths should be similar for two nearby pixels thatbelong to the same object. This is the basic principle of the bilateral filter78 whichhas been adapted to address the depth interpolation problem.87,85,33 Inspired bythe Poisson model for inpainting5,7,6 we extend it to copy not the disparity map butthe disparity gradients, enabling the interpolation of planes. The proposed energy isan extension of the energy of the bilateral filter (see (1.4)) to the gradient domain.We show in Sec. 9 some experiments illustrating its ability to interpolate planes in
1230003-7
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 8–65
P. Arias et al.
contrast to the bilateral filter. We prove the Γ-convergence of the proposed energy,which is nonlocal, as we localize the neighborhood where pixels are compared,obtaining the energy∫
Ω
Trace(D2u(x)Q(∇I(x))D2u(x))dx, (1.5)
where D2u denotes the Hessian of u and Q(∇I(x)) is a tensor inhibiting the inter-polation across edges (discontinuities) of the image I. This shows that the nonlocalPoisson energy for disparity interpolation is a nonlocal version of a second-orderfunctional. Moreover, thanks to the anisotropic tensor Q(∇I(x)) we are able tointerpolate planes locally. The interphases separating two different planes are givenby the edges of the reference image. Notice that the previous result is an exten-sion of the asymptotic expansion in Ref. 22 that identifies the anisotropic diffusionequation underlying the bilateral filter. The proof is based on the results of Brezisand Bourgain19 approximating Sobolev norms by their nonlocal version. The appli-cation of this energy model to the interpolation of disparity maps acquired using aTime-of-Flight camera together with the generation of the right image of the stereopair is the object of current research.57
In Sec. 10 we consider some simple cases of the problem of inpainting videosequences. The enormous variety of situations make this problem very difficult andinteresting at the same time. The automatic or semiautomatic choice of the inpaint-ing region, the development of techniques that help to impose time coherence inthe inpainted result, and to give it illumination effects which are coherent withthe surrounding parts of the image,65,40 are just some of the difficulties that arisein video inpainting. First we discuss the possible ingredients needed to generatea semiautomatic tool for video inpainting. Then we focus our attention on one ofthem: a gradient-domain editing technique extending the classical Poisson editingtool to video,40,80 which helps to get time coherent inpainted results and illumina-tion effects coherent with the surrounding parts of the image.65,40 We also pointout the utility of this tool for propagating an inpainted frame along a video, or forobject insertion in video. There is still a lot of work to do to provide an effectiveediting tool for the video and cinema post-production industry.
Concluding remarks and future work are discussed in Sec. 11.
2. A Variational Framework
In this section we review the variational framework for nonlocal image inpaintingproposed in Refs. 5, 7, 6. The proposed scheme can be seen as a generalization ofthe model (1.2) of Refs. 83, 52 (see also Refs. 5, 7) to the case of a probabilistic cor-respondence. As a limit case of our model we obtain an energy which is a relaxationof the correspondence map case (1.2). Although this particular case is the most rele-vant for the image inpainting application, the general framework allows to establishinteresting links between the models for nonlocal image regularization23,53,41 and
1230003-8
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 9–65
Nonlocal Variational Models for Inpainting and Interpolation
exemplar-based inpainting. On the other hand, the use of a probabilistic correspon-dence simplifies in the continuous case the analysis of the functional, allowing forinstance to prove the existence and regularity of minima and the convergence ofthe alternating minimization scheme to critical points of the energy. It permits alsoto relate the functional to the classical Gibbs functional in statistical mechanics.36
2.1. Notation
Images are denoted as functions u : Ω → R, where Ω denotes the image domain,usually a rectangle in R
N . We will commonly refer to points in Ω as pixels. Pixelpositions are denoted by x, y or h, the latter for positions inside the patch. A patchof u centered at x is denoted by pu(x) = pu(x, ·) : Ωp → R where Ωp is a rectanglecentered at O. The patch is defined by pu(x, h) = u(x+h), with h ∈ Ωp. Let O ⊂ Ωbe the hole or inpainting domain, and Oc = Ω\O. We assume that O is an open setwith Lipschitz boundary. We still denote by u the part of the image u inside thehole, while u is the part of u in Oc : u = u|Oc .
Let us define some domains in Ω that are useful to work with patches. We denoteby Ω the set of centers of patches contained in the image domain, i.e. Ω = x ∈Ω :x+ Ωp ⊆ Ω. We take O as the set of centers of patches that intersect the hole,i.e. O = O + Ωp = x ∈ Ω : (x + Ωp) ∩ O = ∅. For a simplified presentation, weassume that O ⊆ Ω, i.e. every pixel in O is the center of a patch contained in Ω.We denote Oc = Ω\O. Thus, patches pu(y) centered at points y ∈ Oc are containedin Oc (see Fig. 1). Further notation will be introduced in the text.
2.2. A motivating example
Our variational framework is inspired by the following nonlocal functional
Fw(u) =∫
O
∫Oc
w(x, y)(u(x) − u(y))2dydx, (2.1)
where w :O × Oc → R+ is a weight function that measures the similarity between
patches centered in the inpainting domain and in its complement.Let us assume for the moment that the weights are known. The minimum of
(2.1) should have a low pixel error (u(x)− u(y))2 whenever the similarity w(x, y) ishigh. In this way the similarity weights drive the information transfer from knownto unknown pixels. A similar functional was proposed in Ref. 41 as a nonlocalregularization energy in the context of image denoising. It models the nonlocalmeans filter23 when the weights are Gaussian
w(x, y) ∝ exp(− 1T‖pu(x) − pu(y)‖2
). (2.2)
Here, ‖ · ‖ is a weighted L2-norm in the space of patches and T is a parameter thatdetermines the selectivity of the weights w.
One of the novelties of the proposed framework5,7 is the inclusion of adaptiveweights in a variational setting, considering the weight function w as an additional
1230003-9
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 10–65
P. Arias et al.
unknown. Instead of prescribing explicitly the Gaussian functional dependence ofw with respect to u, we will do it implicitly, as a component of the optimizationprocess. On the one hand this permits to write a simpler functional which has theform of a Gibbs free energy; on the other hand, this is in consonance with thealternating optimization algorithm, one of the algorithms usually used in practice.We will come back to this later on in Sec. 5 when we identify this scheme with theEM algorithm.
In our formulation, we will constrain w(x, ·) to be a probability density function,which can be seen as a relaxation of the correspondence maps in Refs. 8, 31, provid-ing a fuzzy correspondence between each x ∈ O and the points in the complementof the inpainting domain.
2.3. The proposed formulation
In this setting, we propose an energy which contains two terms, one of them isinspired by (2.1) and measures the coherence between patches in O and thosein Oc, for a given similarity weight function w : O × Oc → R. This permits theestimation of the image u when the weights w are known. The second term allowsus to compute the weights given by the image. The complete proposed functional is
Eε,T (u,w) = Uε(u,w) − T
∫eO
H(w(x, ·))dx,
subject to∫
eOc
w(x, y)dy = 1,(2.3)
where
Uε(u,w) =∫
eO
∫eOc
w(x, y)ε(pu(x) − pu(y))dydx, (2.4)
ε(·) is an error function for image patches (such as the squared L2-norm), and
H(w(x, ·)) = −∫
eOc
w(x, y) logw(x, y)dy
is the entropy of the probability w(x, ·).Let us now make some additional comments on the functional. We observe
that the term (u(x) − u(y))2 in (2.1), that penalizes differences between pixels, issubstituted in (2.4) by the patch error function ε(pu(x) − pu(y)). This has severalimplications. First, observe that minimizing (2.4) with respect to the image u willforce the patch pu(x) to be similar to pu(y) whenever w(x, y) is high. Second, weobserve that for a given completion u, and for each x ∈ O, the optimum weightsminimize the mean patch error for pu(x), given by∫
eOc
w(x, y)ε(pu(x) − pu(y))dy,
while maximizing the entropy. This can be related to the principle of maximumentropy,48 widely used for inference of probability distributions. According to it,
1230003-10
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 11–65
Nonlocal Variational Models for Inpainting and Interpolation
the best representation for a distribution given a set of samples is the one thatmaximizes the entropy, i.e. the distribution which makes less assumptions aboutthe process. Taking ε as the squared L2-norm of the patch, then the resultingweights are given by formula (2.2). The parameter T in (2.3) controls the trade-offbetween both terms and is also the selectivity parameter of the Gaussian weights.Note also that by restricting w(x, ·) to be a probability, trivial minima of E withw(x, y) = 0 everywhere are discarded.
2.3.1. Examples of patch error function ε
Patches are functions defined on Ωp, and if P denotes a suitable space of patches,we consider error functions ε : P → R
+ defined as the weighted sum of pixelwiseerrors
ε(pu(x) − pu(y)) = g ∗ e(u(x+ ·) − u(y + ·)), (2.5)
where e : R → R+, or gradient errors
ε(pu(x) − pu(y)) = g ∗ e(∇u(x+ ·) −∇u(y + ·)),where e :R2 → R
+, and g : RN → R+ denotes a suitable localizing kernel function,
like a Gaussian or a function with compact support.Let us give some examples of patch error functions:
Patch nonlocal means. We consider P = L2(Ωp), e(r) = |r|2, and the patch errorfunction
ε(pu(x) − pu(y)) = ‖pu(x) − pu(y)‖2g,2 := g ∗ |u(x+ ·) − u(y + ·)|2.
Patch nonlocal medians. We consider P = L1(Ωp), e(r) = |r|, and the patch errorfunction
ε(pu(x) − pu(y)) = ‖pu(x) − pu(y)‖g,1 := g ∗ |u(x+ ·) − u(y + ·)|.
Patch nonlocal Poisson. We define P = W 1,2(Ωp), e(r) = |r|2, and the patch errorfunction
ε(pu(x) − pu(y)) = ‖pu(x) − pu(y)‖2∇,g := g ∗ |∇u(x+ ·) −∇u(y + ·)|2.
Patch nonlocal gradient medians. Let us take P as the space of bounded variationfunctions in Ωp,2 e(r) = |r|, and the patch error function
ε(pu(x) − pu(y)) = ‖pu(x) − pu(y)‖∇,g,1 := g ∗ |∇u(x+ ·) −∇u(y + ·)|.
The last two patch error functions are based on the gradient of the image. As itwill be discussed below, the patch error function determines not only the similaritycriterion but also the image synthesis, and thus is a key element in the proposedframework. Let us mention that the use of nonlocal energies with gradient termsfor deblurring and denoising problems has been proposed in Ref. 53.
1230003-11
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 12–65
P. Arias et al.
2.3.2. Getting a correspondence
Finally, let us note that we can get formally a correspondence map by taking thelimit as T → 0. The resulting energy is dominated by the image term and can bewritten as
E(u,w) =∫
eO
∫eOc
w(x, y)ε(pu(x) − pu(y))dydx. (2.6)
We will describe more precisely this limit process in Sec. 4 and show that themap x → w(x, ·) has to be replaced by a Young measure (a general measurableprobability-valued map which can be disintegrated with respect to the Lebesguemeasure). In that context there are minima of (2.6) with respect to the weightsw(x, ·) which can be written as a Dirac’s delta function on a point ϕ(x) which isa nearest neighbor of the patch pu(x) with respect to the patch error function ε,i.e. w(x, y) = δ(y − ϕ(x)). We notice that the set arg miny∈ eOc ε(pu(x) − pu(y)) isnot necessarily made of a single point. In the case that it is a singleton for anyx ∈ O, the problem of minimizing the energy (2.6) can be rewritten in terms of acorrespondence map
E(u, ϕ) =∫
eO
ε(pu(x) − pu(ϕ(x)))dx. (2.7)
The model of Refs. 52, 83 (Eq. (1.2)) is obtained as a particular case when ε is thesquared L2-norm. An equivalent formulation has been proposed by Peyre,68 wherethe energy is interpreted as a regularization model based on the distance to themanifold formed by the known patches.
Although the case T → 0 is the most relevant for the image inpainting appli-cation5,7 and most of the experiments shown correspond to it, we will present theframework for T ≥ 0. This will give us a broader view of the model and the mainideas underlying it, allowing us to relate it with other models recently proposed fornonlocal image regularization. It also shows the connections with the Gibbs energyfunctional in statistical mechanics as we describe in Sec. 2.4.
2.4. Connections with statistical mechanics
Let us mention here that the formulation above can be rephrased as a Gibbs energyfunctional in statistical mechanics. For simplicity, we restrict ourselves to the dis-crete case where O is a subset of a domain Ω in Z
2.In the context of statistical mechanics we consider that for each x ∈ O there is
a set of possible configurations indexed by the parameter y ∈ Oc with a probabilitydensity w(x, y). We follow the tradition in statistical mechanics and use the notationβ = 1
T . If we consider that each configuration has an energy
Uε(u)(x, y) := ε(pu(x) − pu(y)),
1230003-12
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 13–65
Nonlocal Variational Models for Inpainting and Interpolation
then the Boltzmann distribution gives the probability that the “particle” x is inthe state y. This distribution is given by
pε,β(u)(x, y) =rε,β(u)(x, y)Zε,β(u)(x)
. (2.8)
where
rε,β(u)(x, y) := e−βUε(u)(x,y)
and the normalization factor is given by the partition function
Zε,β(u)(x) =∑
y∈ eOc
e−βUε(x,y).
Then the energy Eε,1/β(u,w) can be written as
Eε,1/β(u,w) = −β−1∑x∈ eO
∑y∈ eOc
w(x, y) log rε,β(u)(x, y)
+ β−1∑x∈ eO
∑y∈ eOc
w(x, y) logw(x, y)
= β−1∑x∈ eO
∑y∈ eOc
w(x, y) logw(x, y)
pε,β(u)(x, y)− β−1
∑x∈ eO
logZε,β(u)(x)
= β−1∑x∈ eO
KL(w(x ·), pε,β(u)(x, ·)) +∑x∈ eO
Fε,β(u)(x),
where KL(P,Q) denotes the Kullback–Leibler divergence between two probabilitydistributions P and Q in Oc:
KL(P,Q) =∑
y∈ eOc
P (y) log(P (y)Q(y)
),
and Fε,β(u)(x) denotes the Helmholtz free energy
Fε,β(u)(x) := − 1β
logZε,β(u)(x).
The expression
Gε,β(u,w)(x) :=1β
KL(w(x ·), pε,β(u)(x, ·)) + Fε,β(u)(x)
is called in statistical mechanics the Gibbs free energy and we may write ourenergy as
Eε,1/β(u,w) =∑x∈ eO
Gε,β(u,w)(x).
1230003-13
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 14–65
P. Arias et al.
With u fixed, the minimum of Eε(u,w) with respect to w is attained when w(x, ·) =pε,β(u)(x, ·), i.e. when w(x, ·) coincides with the Boltzmann distribution for all x.Then
Eε,1/β(u, pβ) =∑x∈ eO
Gε,β(u, pε,β(u), x) =∑x∈ eO
Fε,β(u)(x),
which is the sum of the free energies for all x. Finally, notice that the Helmholtzfree energy can be written as
Fε,β(u)(x) = 〈U(x, ·)〉pε,β(u)(x,·) −1βH(pε,β(u)(x, ·)),
where 〈U(x, ·)〉pε,β(u)(x,·) is the average energy and H(pε,β(u)(x, ·)) is the entropyof the Boltzmann distribution pε,β(u)(x, ·). We refer to Sec. 5 for a complementarypoint of view.
Let us finally mention that statistical mechanics has also been the frameworkfor a variational approach to many problems like clustering, pattern recognitionand classification, regression, or coding theory (see Ref. 73 and references therein).
3. Existence of Minima
We review in this section the results on the existence of solutions of the NL-meansmodel and the NL-Poisson models as proved in Refs. 7 and 6.
Let Cc(RN ) be the set of continuous functions with compact support in RN . By
Cc(RN )+ we denote the set of non-negative functions in Cc(RN ). As usual, if Q is anopen set we denote by W 1,p(Q), 1 ≤ p ≤ ∞, the space of functions v ∈ Lp(Q) suchthat ∇v ∈ Lp(Q)N . By W 1,p(Q)+ we denote the set of non-negative functions inW 1,p(Q). We denote by W 2,p(Q) (respectively, by W 2,p
loc (Q)), 1 ≤ p ≤ ∞, the spaceof functions v ∈ Lp(Q) such that ∇v ∈ Lp(Q)N andD2v ∈ Lp(Q)N×N (respectively,the functions v ∈W 2,p(Q′) for any subdomain Q′ included in a compact set of Q).
Let us assume in the rest of the paper that g ∈ L1(RN )+ and∫
RN g(h)dh = 1.
3.1. Existence of minima for the patch NL-means model
We assume that Ω is a rectangle in RN and u :Oc → R with u ∈ L∞(Oc). We
assume that u : Ω → R is such that u|Oc = u. We also assume that u is extendedby symmetry and then by periodicity to R
N .In this section we consider the patch NL-means model
E2,T (u,w) =∫
eO
∫eOc
w(x, y)‖pu(x) − pu(y)‖2g,2
+T
∫eO
∫eOc
w(x, y) logw(x, y)dydx. (3.1)
We implicitly understand that E2,T (u,w) = +∞ in case that the second integral isnot defined.
1230003-14
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 15–65
Nonlocal Variational Models for Inpainting and Interpolation
Let
W :=w ∈ L1(O × Oc) :
∫eOc
w(x, y) dy = 1 a.e. x ∈ O
.
Let us consider the admissible class of functions
A2 := (u,w) :u ∈ L∞(Ω), u = u in Oc, w ∈ W.Our purpose is to prove the following result stating the existence of minima of
min(u,w)∈A2
E2,T (u,w). (3.2)
Proposition 3.1. Assume that g ∈ Cc(RN )+ has support contained in Ωp, ∇g ∈L1(RN ) and u ∈ BV(Oc) ∩ L∞(Oc). There exists a minimum (u,w) ∈ A2 of E2,T .For any minimum (u,w) ∈ A2 we have that u ∈ W 1,∞(O) and w ∈W 1,∞(O× Oc).
In other words, there are smooth minima and smooth probability distributionsrepresenting the fuzzy correspondences between O and Oc. To prove Proposition3.1 we need the following lemma proved in Ref. 7.
Lemma 3.1. Assume that g ∈ Cc(RN )+ has support contained in Ωp, ∇g ∈L1(RN ) and u ∈ BV(Oc) ∩ L∞(Oc). Assume that u ∈ L∞(O + Ωp). Then thefunctions
∇xg ∗ (u(x+ ·) − u(y + ·))2 and ∇yg ∗ (u(x+ ·) − u(y + ·))2 (3.3)
are uniformly bounded in O × Oc by a constant that depends on ‖∇g‖L1, ‖u‖∞,‖u‖∞.
The weight function is the source of the bounds stated in last lemma. Its prooffollows by direct computation.7
Proof. Let us give a sketch of the proof of Proposition 3.1.
Step 1. Basic observations. If (u,w) ∈ A2 and u is a minimum of E2,T (·, w) wehave
u(z) =∫
RN
g ∗ (w eO, eOc(z − ·, z′ − ·))u(z′)dz′, z ∈ O, (3.4)
where
g ∗ (w eO, eOc(z − ·, z′ − ·)) :=∫
RN
g(h)χ eOc(z′ − h)χ eO(z − h)w(z − h, z′ − h)dh.
(3.5)
If w is a minimum of E2,T (u, ·) we have that
w(x, y) = w2,T (u)(x, y) =1
Z2,T (u;x)exp(− 1Tg ∗ (u(x+ ·) − u(y + ·))2
). (3.6)
Thus, both Eqs. (3.4) and (3.6) hold if (u,w) ∈ A2 is a minimum of E2,T .
1230003-15
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 16–65
P. Arias et al.
Step 2. There are minimizing sequences (un, wn) ∈ A2 for E2,T such that un isuniformly bounded.
For that, let (u′n, w′n) ∈ A2 be a minimizing sequence for E2,T . Let
un = argminu
E2,T (u,w′n), (3.7)
wn = argminw
E2,T (un, w). (3.8)
Since
E2,T (un, wn) ≤ E2,T (un, w′n) ≤ E2,T (u′n, w
′n),
(un, wn) ∈ A2 is also a minimizing sequence of (3.2). By (3.4) we have
un(z) =∫
RN
g ∗ ((w′n) eO, eOc(z − ·, z′ − ·))u(z′)dz′, z ∈ O, (3.9)
and we deduce that ‖un‖L∞(O) ≤ ‖u‖∞.
Step 3. Existence of minima. If (un, wn) ∈ A2 is a minimizing sequence for E2,T
such that un is uniformly bounded, then we may extract a subsequence convergingto a minimum of E2,T .
Since O is a bounded domain and E2,T (un, wn) is bounded, wn(1 + log+ wn) isbounded in L1(O× Oc), i.e. wn is bounded in LLog+L(O× Oc). Then the sequencewn is weakly relatively compact in L1 and, modulo a subsequence, we may assumethat wn converges weakly in L1(O × Oc) to some w ∈ W .
Since un is bounded, by Lemma 3.1, modulo the extraction of a subsequence, wemay assume that un → u weakly in all Lp, 1 ≤ p < +∞ and g∗(un(x+·)−u(y+·))2converges strongly in all Lp spaces and also in the dual of LLog+L to some functionW . Taking test functions ψ(x, y), integrating in O× Oc and using the convexity ofthe square function, we have∫
RN
g(h)(u(x+ h) − u(y + h))2 dh ≤W (x, y).
Then by passing to the limit as n→ ∞ we have
E2,T (u,w) ≤ lim infn
E2,T (un, wn).
Step 4. Regularity of minima of E2,T . The regularity of minima of E2,T follows byanalyzing Eqs. (3.4), (3.6) and using Lemma 3.1.
3.2. Existence of minima for the patch NL-Poisson model
In this section we consider the model
E∇,T (u,w) :=∫
eO
∫eOc
w(x, y)‖pu(x) − pu(y)‖2∇,gdydx
+T
∫eO
∫eOc
w(x, y) logw(x, y)dy, (3.10)
1230003-16
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 17–65
Nonlocal Variational Models for Inpainting and Interpolation
where
‖p‖2∇,g =
∫RN
g(h)‖∇p(h)‖22dh, p ∈ P = W 1,2(Ωp).
Recall that we assume that u|Oc = u. Let
A∇ := (u,w) ∈ A2 :u ∈W 1,2(O), u|∂O = u|∂Oc.Our purpose is to prove the following result states the existence of minima of
min(u,w)∈A∇
E∇,T (u,w). (3.11)
Proposition 3.2. Assume that u ∈ W 2,2(Oc)∩L∞(Oc) and g ∈ W 1,∞(RN )+ hascompact support in Ωp. There exists a solution of the variational problem (3.11).Moreover for any solution (u,w) ∈ A∇ we have u ∈ W 1,2(O) ∩W 2,p
loc (O) ∩ L∞(O)for all p ∈ [1,∞) and w ∈ W 1,∞(O × Oc).
The proof of (3.2) is based on the following lemma, analogous of Lemma 3.1.
Lemma 3.2. Assume that u ∈ W 2,2(Oc), u ∈ W 1,2(O), u|∂O = u|∂Oc and g ∈W 1,∞(RN )+ has compact support on Ωp. Then
∇x
∫RN
g(h)|∇xu(x+ h) −∇yu(y + h)|2 dh
and
∇y
∫RN
g(h)|∇xu(x+ h) −∇yu(y + h)|2 dh
are bounded in L∞(O × Oc) with a bound depending on ‖u‖W 2,2(Oc), ‖g‖W 1,∞ and‖∇u‖L2(O).
4. Existence of Optimal Correspondence Maps
In this section we consider a relaxation of the correspondence map approach (1.2). Inturn, the energy (1.2) can be considered as a relaxation of the energy considered inRef. 31 (see also Ref. 8). We review the existence of solutions of the relaxed problemand the existence of optimal correspondence maps. We also review the existence ofcorrespondence maps which are uniform limits of bounded variation functions withfinitely many values. We restrict our presentation to the patch NL-means model.Analogous results hold for the NL-Poisson model (see Ref. 6).
Let us first recall the notion of measurable measure-valued map.
Definition 4.1. (Measurable measure-valued map) Let X ⊆ RN , Y ⊆ R
M be opensets, µ be a positive Radon measure in X and x → νx be a function that assignsto each x in X a Radon measure νx on Y. We say that the map is µ-measurable ifx→ νx(B) is µ-measurable for any Borel set B in Y.
By the disintegration theorem, if ν is a positive Radon measure in X × Y suchthat ν(K ×Y) <∞ for any compact set K ⊆ X and µ = πν (i.e. µ(B) = ν(B×Y
1230003-17
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 18–65
P. Arias et al.
for any Borel set B ⊆ X ), where π :X ×Y → X is the projection on the first factor,then there exists a measurable measure-valued map x → νx such that νx(Y) = 1µ-a.e. in X and for any ψ ∈ L1(X × Y, ν) we have
ψ(x, ·) ∈ L1(Y, νx) for µ-a.e. x ∈ X
x→∫Yψ(x, y)dνx(y) ∈ L1(X , µ)∫
X×Yψ(x, y)dν(x, y) =
∫X
∫Yψ(x, y)dνx(y)dµ(x).
Let us consider MP the set of measurable measure-valued maps ν ≥ 0 in O×Oc
such that πν = LN | eO, where LN | eO denotes the Lebesgue measure restricted to O.We assume that g ∈ Cc(RN ) has support contained in Ωp, ∇g ∈ L1(RN ), andu ∈ BV(Oc) ∩ L∞(Oc). Let
A2,0 := (u, ν) :u ∈ L∞(Ω), u = u in Oc, ν ∈ MP.For (u, ν) ∈ A2,0, define
E2,0(u, ν) :=∫
eO
∫eOc
g ∗ (u(x+ ·) − u(y + ·))2dν(x, y). (4.1)
Notice that, by Lemma 3.1, the above integral is well defined.Let ϕ : O → Oc be a measurable map. Then x ∈ O → νx = δϕ(x) is measurable.
Similarly if the map x ∈ O → νx = δϕ(x) is measurable then ϕ is measurable. Letus denote by νϕ the measure determined by ϕ.
Theorem 4.1. There exists a minimum (u, ν) ∈ A2,0 of E2,0. Moreover, thereexists a minimum (u∗, ν∗) ∈ A2,0 of E2,0 such that ν∗ = νϕ where ϕ : O → cl(Oc)is a measurable map.
The second assertion can be proved using Kuratowski–Ryll–Nardzewski theo-rem76 or as a consequence of the following lemma.6
Lemma 4.1. The set of extreme points of the convex set MP coincides with theset of measures νϕ :ϕ is a measurable map.
The relation between the patch NL-means functional for T > 0 and E2,0 is givenin the following proposition (see Ref. 6).
Proposition 4.1. The energies E2,T Γ-converge to the energy E2,0. In particular,the minima of E2,T converge to minima of E2,0.
4.1. Existence of regular correspondence maps
Of particular interest is the existence of regular or piecewise regular correspondencemaps. Given a correspondence map ϕ(x), let τ(x) = ϕ(x) − x, x ∈ O, denote
1230003-18
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 19–65
Nonlocal Variational Models for Inpainting and Interpolation
the offsets. The offset maps found in experiments usually correspond to piecewiseconstant translations (see Sec. 7). We are not able to prove such a result. Neitherwe know which is the optimal regularity of offset maps. We are only able to prove amild regularity result, namely the existence of optimal correspondence maps (hence,offsets) which are uniform limits of functions of bounded variation. The question ifcurrent algorithms produce such class of maps is open.
Let us first recall the definition of functions of bounded variation. Let Q be anopen subset of R
N . Let u ∈ L1loc(Q). The total variation of u in Q is defined by
V (u,Q) := sup∫
Q
u div σ dx : σ ∈ C∞c (Q; RN ), |σ(x)| ≤ 1 ∀x ∈ Q
, (4.2)
where C∞c (Q; RN) denotes the vector fields with values in R
N which are infinitelydifferentiable and have compact support in Q. For a vector v = (v1, . . . , vN ) ∈ R
N
we denoted |v|2 :=∑N
i=1 v2i . Following the usual notation, we will denote V (u,Q)
by |Du|(Q).Let u ∈ L1(Q). We say that u is a function of bounded variation inQ if V (u,Q) <
∞. The vector space of functions of bounded variation in Q will be denoted byBV(Q). Recall that BV(Q) is a Banach space when endowed with the norm ‖u‖ :=∫
Q |u|dx+ |Du|(Q).
Theorem 4.2. Let X be an open bounded subset of RN with Lipschitz boundary
and Y be a compact subset of Rm. Let U :X × Y → R be a Lipschitz continuous
function. For each x ∈ X, let M(x) := y ∈ Y :U(x, y) = miny∈Y U(x, y). Thenthere exists a selection of the multifunction x ∈ X → M(x) ⊆ Y, i.e. a functionS :X → Y such that S(x) ∈ M(x) ∀x ∈ X, which is a uniform limit of functionsin BV(X)m.
The result can be immediately applied to our case with X = O, Y = Oc. Indeed,it can be applied to the patch NL-means model and to the patch NL-Poisson modelwhen the assumptions of Lemma 3.2 hold. Theorem 4.2 implies that the offsets mapτ(x) = ϕ(x) − x is a uniform limit of maps of bounded variation (in this case fromO to R
2), since the identity map x→ x also is.Notice that the result does not say that all optimal correspondence maps are
regular. In view of Propositions 3.1 and 3.2, this raises the question if the solutionobtained by annealing, i.e. by solving E2,T (or E∇,T ) and letting T → 0+ is indeeda regular solution in the sense described in Theorem 4.2. We are not able to answerthis question, at present. A related question will be considered in Sec. 6.
Its proof is an adaptation of the Kuratowski–Ryll–Nardzewski theorem76 (The-orem 5.2.1). A different geometric insight based on the manifold of patches will begiven in Sec. 4.2.
As a uniform limit of functions in BV(X)m, S inherits some of its properties.In particular, HN−1-a.e. x ∈ X is either a point of approximate continuity of S,or a jump point.2 Moreover given a point x in the jump set of S, denote it byJS , x is in the jump set of Sn, denote it by JSn , for n large enough. That is
1230003-19
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 20–65
P. Arias et al.
JS =⋂
k
⋃n≥k JSn = lim supn JSn . In particular, JS is a countably rectifiable set.2
Outside it, the function is approximately continuous modulo an HN−1 null set.2
4.2. Remarks on the existence of regular correspondence maps:The point of view of the manifold of patches
Let us explain the role of the selection theorem in Theorem 4.2. For that we con-sider the problem from the point of view of the manifold of patches. For simplicity,let us consider the case of sampled images. In that case, we assume that u : Ω → R
is acquired by an optical system whose point spread function is given by a ker-nel K so that u = K ∗ v where v is the original radiation field. Assuming thenthat u is continuous, the patch of u centered at x is given by a sampling of uon a neighborhood of x. Let Π = Ωp ∩ Z
2, which is a discrete neighborhood of0 ∈ Z
2. Let D be the cardinal of Π. Thus pu(x) = u(x + h) :h ∈ Π ∈ RD
and the map pu : Ω → RD. Notice that pu is a parametrization of the mani-
fold of patches and it is defined in the continuous domain Ω. Its discretizationto the sampling grid of the image would give a sampling of the manifold ofpatches.
Let Mu be the manifold of patches given by the image of pu. By suitable assump-tions on the kernel K we may assume that pu is of class C1.
Given a closed set C ⊂ RD, we denote by dC the distance of a point p ∈ R
D toC, that is dC(p) = infq∈C ‖p− q‖.
Let us consider the following problem. Given two images u : Ωu → R, u : Ωu → R
with the corresponding manifolds of patches Mu,Mu, let us consider the mapprojMu
(p) that associates to each point p ∈ RD the set of points of Mu of minimal
distance to p, that is
projMu(p) := q ∈ Mu : dMu(p) = ‖p− q‖.
What is the regularity of correspondence maps ϕ that associate to each point x ∈ Ωu
a point ϕ(x) ∈ Ωu such that pu(ϕ(x)) ∈ projMu(pu(x))?
The regularity of correspondence maps is related to the regularity of theparametrizations pu and pu of the manifolds Mu and Mu and to the regularityof the projection map projMu
(x). Our purpose is to describe what is the genericgeometric situation and justify the need of using selection theorems in order toensure the existence of mildly regular correspondences. For that, let us recall someresults on the differentiability and the singular set of semiconcave functions.25
A function f defined on an open set A ⊂ RD is semiconcave if there is a nonde-
creasing upper semicontinuous function ω : R+ → R+ such that limρ→0+ ω(ρ) = 0and
λf(p) + (1 − λ)f(q) − f(λp+ (1 − λ)q) ≤ λ(1 − λ)|p− q|ω(|p− q|)
for any pair p, q such that the segment [p, q] is contained in A and for any λ ∈ [0, 1].
1230003-20
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 21–65
Nonlocal Variational Models for Inpainting and Interpolation
The distance dC to a closed set C ⊂ RD is a locally semiconcave function in
RN\C with modulus ω(ρ) = ρ. The function d2
C is semiconcave in RD with modulus
ω(ρ) = ρ.
Proposition 4.2. (Ref. 25, Chap. 3) If f is semiconcave in A, then the superdif-ferential D+f(p) = ∅, ∀ p ∈ A. If D+f(p) is a singleton, then f is differentiable atp. If D+f(p) is a singleton for any p ∈ A, then f ∈ C1(A).
Proposition 4.3. (Ref. 25, Proposition 4.1.2) Suppose that f :A→ R is semicon-cave with a linear modulus. Then the gradient of f is a function in BVloc(A,RD)and the set Σ(f) of singularities of f, i.e. the points where f is not differentiable,coincides with the jump set SDf of the gradient of f, which is an HD−1-rectifiableset.
Proposition 4.4. (Ref. 25, Proposition 4.4.1) Let C ⊂ RD be a closed set. Then
dC is semiconcave with linear modulus in RD\C. dC is differentiable at p ∈ C if
and only if projC(p) is a singleton. In that case
projC(p) = p− dC(p)DdC(p).
Let us apply these results to the distance map dMu . If p ∈ Σ(dMu), then dMu
is differentiable at p and projMu(p) is a single point on Mu. Thus, the function
dMu is differentiable at the points Mu\Σ(dMu). Hence the mapping projMuis well
defined on Mu\Σ(dMu) and is given by
p ∈ Mu\Σ(dMu) → projMu(p) = p− dMu(p)DdMu(p).
Hence it is a BV function (see Proposition 4.3). Thus, the mapping x ∈ x ∈Ωu : pu(x) ∈ Σ(dMu) to projMu
(pu(x)) is a BV function, since the map x ∈ Ωu →pu(x) is differentiable. By Proposition 4.2 we may assume that it is C1 at the pointsin Mu\Σ(dMu).
Generically, we may assume that Mu intersects Σ(dMu) transversally, thus wemay assume that the intersection Mu ∩ Σ(dMu) is a rectifiable curve in Mu.
Let MP2(u) be the set of points y ∈ Ωu where the rank of the differential of pu aty is 2. Since the differential of pu is given by the map dpu(y)(v) = v ·∇u(y+h) :h ∈Π and
‖dpu(y)(v)‖22 =
∑h∈Π
〈∇u(y + h) ⊗∇u(y + h)(v), v〉,
the rank of dpu(y) is the rank of the 2 × 2 matrix∑
h∈Π ∇u(y + h) ⊗∇u(y + h).Since we can give the structure of a differentiable manifold to pu(MP2(u)), if
we assume that Ωu = MP2(u) and pu is injective, then x ∈ Ωu → ϕ(x) ∈ Ωu is a(locally) BV function. A more detailed analysis of the manifold of patches will begiven in Ref. 4.
Thus, we have seen that there are several obstructions to the regularity of ϕ. Thefirst is the position of the manifold Mu relative to the singularities of the distance
1230003-21
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 22–65
P. Arias et al.
map to Mu. Intuitively, if they intersect, generically this intersection is a rectifiablecurve. At those points, we have to select the correspondence map so that it is aBV map. More significantly, the map pu may not be injective and then we needagain a selection. We can only guarantee the regularity of ϕ if projMu
(pu(x)) is apoint in pu(MP2(u)) (that is, in the image of the set of points where pu is locallyinjective) where the inverse of pu is differentiable. Thus, regularity without usingthe selection theorem can only be guaranteed at the points of MP2(u) where pu
is injective. But we only know that pu is locally injective at the points of MP2(u)and we can only expect a BV regularity result on ϕ in all Ωu by a proper selectionof the inverse map of pu.
For that reason, we used the extension to the BV case of the Kuratowski–Ryll–Nardewsky selection theorem in order to have a global statement in all Ωu.
5. Convergence of the Alternating Optimization Scheme
To minimize the energy, we use an alternating minimization scheme. At each iter-ation, two optimization steps are solved: the minimization of E with respect to wwhile keeping u fixed; and the minimization with respect to u with w fixed. In thissection we state the convergence of such a scheme to a critical point of the energyboth for the case of patch NL-means and the NL-Poisson models.6
Let Eε,T be one of the energies E2,T or E∇,T . Similarly, Aε denotes A2 or A∇.
Algorithm 1. Alternate minimization of Eε,T .
Initialization: Choose u0 with ‖u0‖∞ ≤ ‖u‖∞.
For each k ∈ N solve
wk+1 = arg minw∈W
Eε,T (uk, w), (5.1)
uk+1 = argmin(u,wk+1)∈Aε
Eε,T (u,wk+1). (5.2)
Proposition 5.1. The iterated optimization algorithm converges (modulo asubsequence) to a critical point (u∗, w∗) ∈ Aε of Eε,T . For the energy E2,T
(respectively E∇,T ) the solution obtained has the smoothness described in Propo-sition 3.1 (respectively 3.2), that is u∗ ∈ W 1,∞(O) and w∗ ∈ W 1,∞(O × Oc)(respectively u∗ ∈ W 1,2(O) ∩ W 2,p
loc (O) ∩ L∞(O) for any p ∈ [1,∞) and w ∈W 1,∞(O × Oc)).
Let us point out that the convergence of the alternating optimization (Algo-rithm 1) holds both in the continuous and the discrete domains.
Let us notice that the iterations of the alternating optimization algorithm coin-cide with the EM algorithm.16 Indeed, (5.1) is the E-step, while (5.2) is the M-step.
1230003-22
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 23–65
Nonlocal Variational Models for Inpainting and Interpolation
Using the notation of Sec. 2.4, if
wε,T (u)(x, y) =1
Zε,T (u)(x)exp(− 1T‖pu(x) − pu(y)‖2
g,ε
). (5.3)
where Zε,T (u)(x) is the corresponding normalization factor, then we may write
1TEε,T (u,w) =
∫eO
KL(wε,T (u)(x, ·), w(x, ·))dx − Lε(u),
where
Lε(u) =∫
eO
logZε,T (u)(x)dx (5.4)
corresponds to the so-called marginal likelihood in the context of EM. The alternat-ing optimization algorithm converges (modulo subsequences) to stationary pointsof Lε(u). Notice that given u, the solution of minw Eε,T (u,w) is given by wε,T (u).Note that
−TLε(u) = Eε,T (u,wε,T (u)) ≤ Eε,T (u,w) ∀ (u,w)
and
min(u,w)
Eε,T (u,w) = minu
minw
Eε,T (u,w) = minu
Eε,T (u,wε,T (u)) = minu
−TLε(u).
Thus, functional Eε,T (u,w) is equivalent to −Lε(u) in the sense that both have thesame minima. The alternating optimization algorithm converges to a critical pointof both of them. The convergence of the EM algorithm was proved in Ref. 84.
We can interpret the marginal likelihood (5.4) by noticing that Zε,T (u)(x) is adensity estimate (in the patch space) of the set of patches in Oc: it corresponds tothe total unnormalized similarity of patch pu(x). The minimizer (u∗, w∗) is obtainedwhen for all (x, y) ∈ O × Oc, w∗(x, y) = pε,β(u∗)(x, y) (i.e. normalized Gaussianweights), and the patches of the inpainted image are in regions of high density in thepatch space. This provides a geometric intuitive interpretation of our variationalformulation. The image is considered as an ensemble of overlapping patches. Knownpatches in Oc are fixed, forming a patch density model used to estimate the patchesin O.
5.1. The relaxed correspondence model (discrete case)
The translation of the previous approach to the relaxed correspondence model Eε,0
described in Sec. 4 has to face, in its continuous version, the difficulty of the lack ofestimates ensuring the compactness of the sequence of solutions of (5.5) and (5.6)in Algorithm 2. In the discrete case, the convergence can be proved thanks to theconvexity of Eε,0(u, ν) in each variable when the other is fixed. Thus, we work onlyin the discrete case.
Proposition 5.2. There exists a subsequence which converges to a critical pointof the energy Eε,0.
1230003-23
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 24–65
P. Arias et al.
Algorithm 2. Alternate minimization of Eε,0.
Initialization: Choose u0 with ‖u0‖∞ ≤ ‖u‖∞.
For each k ∈ N solve
νk = arg minν∈MP
Eε,0(uk, ν), (5.5)
uk+1 = arg minu
Eε,0(u, νk). (5.6)
Proof. Since uk and νk are bounded, there is a subsequence (ukj , νkj ) converg-ing to (u, ν). Notice that if ν is fixed, then the solution of minu Eε,0(u, ν) isunique. Clearly ν is a minimum of ν → Eε,0(u, ν). Proceeding as in Ref. 15, onecan prove that ukj+1 − ukj converges to 0 and deduce that u is a minimum ofu→ Eε,0(u, ν).
The convergence of the algorithm giving the solution of the first step (5.5) ofAlgorithm 2 is the object of Sec. 6. The solution of (5.6) is explicitly given by thediscrete analogous of (3.4) for the energy E2,0 and requires the solution of a discreteversion of Poisson equation for E∇,0.
6. Computation of the Nearest Neighbor Field
Throughout the section we consider discretized versions of the inpainting domainand its complement, O = O ∩ Z
2 and Oc = Oc ∩ Z2. To avoid a cumbersome
notation, in some cases the arguments of functions will be denoted as subindices.Most of the computational load in Algorithms 1 and 2 is caused by the updating
of the weights. In this section we discuss the convergence properties of PatchMatch,an algorithm recently introduced by Barnes et al.,11 which we use to speed up thecomputation of the similarity weights. For other aspects of the numerical imple-mentation we refer to Sec. 7. More details can be found in Ref. 7.
For T > 0, the computation of the weight function w is of order O(|O||Oc||Ωp|).This is also the case in the limit T = 0, namely for E2,0. In that case, as shownin Theorem 4.1, there are minima given by measure-valued maps ν determinedby a measurable correspondence map ϕ : O → Oc. This allows us to express theenergy directly in terms of the unknown map ϕ, instead of the measure-valuedmap ν. Thus, when considering the optimization of E2,0 the weights update step issubstituted by a minimization with respect to a correspondence map ϕ,
ϕx ∈ arg minξ∈Z2
Ux(ξ), for all x ∈ O,
where the energy Ux corresponds to the patch error function
Ux(ξ) =
ε(pu(x) − pu(ξ)) if ξ ∈ Oc,
+∞ otherwise.
1230003-24
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 25–65
Nonlocal Variational Models for Inpainting and Interpolation
Although the patch error does not have to be a metric, we will refer to pu(ϕx) asthe nearest patch or nearest neighbor of pu(x). Following Ref. 11, we denote thecorrespondence map ϕ : O → Oc as the nearest neighbor field (NNF). A brute forcesearch for the NNF also conveys O(|O||Oc||Ωp|) operations.
PatchMatch is a very efficient algorithm for approximating the NNF.11 Thesearch for the nearest neighbor is performed simultaneously over the points in O
based on the following heuristic: since query patches overlap, the offset ϕx − x ofa good match at x is likely to lead to a good match for the adjacent points of xas well. It is an iterative algorithm which starting from a random initialization,alternates between steps of propagation of good offsets and random search.
To describe it, we need some definitions. Pixels in O = x1, x2, . . . , x| eO| are
sorted according to the lexicographical order in Z2. For any x ∈ O, let N4(x) =
z ∈ O : 0 < |z−x| ≤ 1 be its 4-neighborhood. We consider a transition probabilitykernel Q : Oc × B → [0, 1], where B is a σ-algebra in Oc (the subsets of Oc in ourdiscrete case). Finally, let us define the notation,
η ∧x ξ =
η if Ux(η) ≤ Ux(ξ),
ξ if Ux(η) > Ux(ξ),
where ξ, η ∈ Oc.The PatchMatch algorithm is given in Algorithm 3. For most applications, a
few iterations after a random initialization are often sufficient. The computationalcomplexity is O(|Ωp||O|). This algorithm can be extended to store queues of Loffsets in an L-nearest neighbors field12 (see also Ref. 7). This allows its applicationto the case T > 0, by truncating the support of w(x, ·) to the L-nearest neighborsof pu(x).
Algorithm 3. PatchMatch with propagation of offsets.
Initialization. Choose ϕ0x ∼ U(Oc), i.e. randomly with a uniform distribution.
For each n ∈ N,
Random search. For each x ∈ O draw Sϕnx ∼ Q(ϕn
x , · ). Set ϕn+ 12
x = ϕnx ∧x Sϕ
nx .
Forward propagation. If n is odd, for each i = 1, . . . , |O|, set
ϕn+1xi
= ϕn+ 1
2xi ∧xi
(∧xi
ϕn+1xj
+ xi − xj :xj ∈ N4(xi), j < i).
If n is even, invert the direction of propagation (backward propagation).
6.1. Convergence of the PatchMatch algorithm
In this section we discuss the convergence properties of the PatchMatch algorithm(for L = 1, i.e. without considering queues). Before proceeding to the convergence
1230003-25
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 26–65
P. Arias et al.
result, it is necessary to add some additional structure. We will consider that ele-ments in O correspond to the vertices of a directed acyclic graph (DAG) G = (O, E),where E ⊂ O × O denotes the edge set. We define the edge set as:
E = (x, y) ∈ O × O : y ∈ N4(x), x < y.By x < y we mean that x precedes y in the lexicographical order. Note that thelexicographical order is a topological order for the resulting DAG. Paths in G willbe denoted by c = (c1, . . . , cnc) ∈ Onc , where nc ∈ N is the length of the path.Given any pair of nodes, x, z ∈ O, we will denote by P(z, x) the set of pathsfrom z to x. A node z ∈ O is said to be an ancestor of x if P(z, x) = ∅. Notethat if z is an ancestor of x, then z comes before x in the lexicographical order-ing. Similarly, z is a descendant of x if P(x, z) = ∅ (i.e. x is an ancestor of z).We will write A(x) and D(x) for the set of ancestors and descendants of node x,respectively.
For each pair of connected nodes (z, x) ∈ E let us define Tx,z : Oc → Z2 by
Tx,z(ξ) = ξ− z+ x. We can rewrite the propagation step in Algorithm 3 as follows:
ϕn+1xi
= ϕn+ 1
2xi ∧xi
(∧xi
Txi,xj (ϕn+1xj
) :xj ∈ N4(xi), j < i).
Without loss of generality, for simplicity, we will assume throughout this sectionthat minξ Ux(ξ) = 0 for all x ∈ O. Note that when performing propagation steps,correspondences on the border of Oc might fall outside Oc, and therefore will not bepropagated. To simplify the following discussion, we assume that for every x ∈ O,the minimum of Ux is attained in a position that can be propagated. More precisely,let us denote by e0 = (0, 1) and e1 = (1, 0) the directions of (forward) propagation.The parents of node x ∈ O are x − ei with i = 0, 1 (when they belong to O),and correspondingly Tx,x−ei(ξ) = ξ + ei. Notice that the positions that cannot bepropagated lie on the set Oc\((Oc − e0) ∩ (Oc − e1)). We will assume that thereexist of a positive constant κ such that
minUz(ξ) : ξ ∈ Oc\((Oc − e0) ∩ (Oc − e1)) > κ, for all z ∈ O. (6.1)
For a more general convergence result we refer to Ref. 7.The following proposition proved in Ref. 7 provides a bound on the convergence
rate in probability for the PatchMatch.
Proposition 6.1. Assume that for each pair (x, y) ∈ E, we have that dx,y :=‖Ux Tx,y − Uy‖∞ < +∞, and that it exists a positive constant κ such that (6.1)holds. Assume that Oc is compact (and therefore finite) and that Q(x,A) > 0, forall x ∈ O, A ⊂ Oc, A = ∅. Then, the sequence (ϕn) defined by the PatchMatchalgorithm converges to a minimizer of the total energy U, in the sense that
limn→∞P (Ux(ϕn
x) > ε) = 0, for all ε > 0, x ∈ O.
1230003-26
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 27–65
Nonlocal Variational Models for Inpainting and Interpolation
Moreover, we have that for ε < κ
P (Ux(ϕn+1x ) > ε) ≤
∏z∈A(x)
C(z, ε− z,x)P (Ux(ϕnx) > ε), (6.2)
where z,x is the length of the minimal path from z to x:
z,x :=
min
c∈P(z,x)
nc∑i=2
dci−1,ci if P(z, x) = ∅,
+∞ if P(z, x) = ∅,
and for each z ∈ O, C(z, · ) : R → [0, 1] is a non-increasing function defined by:
C(z, a) := supη∈Uz>a
Q(η, Uz > a).
For a > 0, C(z, a) < 1.
Remark 6.1. The efficiency of the PatchMatch is mostly given by the propagationsteps, when nodes collaborate by sharing their findings. This is reflected by (6.2).For comparison, consider a PatchMatch algorithm without propagation. Each ϕx
is searched for independently for each x ∈ O. In that case, the bound on the rateof convergence (6.2) reduces to
P (Ux(ϕn+1x ) > ε) ≤ C(x, ε)P (Ux(ϕn
x) > ε).
The speed-up given by the propagation corresponds to∏
z∈A(x)z =x
C(z, ε− z,x). Note
that only those z ∈ A(x) with z,x < ε contribute to lower the bound.
Remark 6.2. Let us estimate the bound (6.2) when the energy Ux corresponds toa patch error function. For dx,x−ei we have
d(T )x,x−ei = ‖Ux Tx,x−ei − Ux−ei‖∞= sup
ξ∈ eOc
|g ∗ e(u(Tx,x−ei(x− ei) + ·) − u(Tx,x−ei(ξ) + ·))
− g ∗ e(u(x− ei + ·) − u(ξ + ·))|≤ sup
ξ∈ eOc
|(g Tx,x−ei − g)| ∗ e(u(x− ei + ·) − u(ξ + ·)). (6.3)
We have used that x = Tx,x−ei(x − ei) and that if T is a translation g ∗ (f T ) =(g T ) ∗ f . The bound (6.3) corresponds to a patch error weighted by the kernel
∂+i g(h) := g Tx,x−ei(h) − g(h) = g(h+ ei) − g(h) ≈ ∂ig(h),
which is an approximation of the partial derivative of g. This is an interestingproperty, because if g is smooth, |∂+
i g(h)| is small. This is essentially the Lipschitzestimate of Lemma 3.1 in the present context. The propagation of offsets exploits
1230003-27
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 28–65
P. Arias et al.
the overlap of neighboring patches in the image domain, suggesting that each nodeshould be connected with its neighbors on the image grid. This supports the intu-itions in Ref. 11.
7. Experiments
We display some image inpainting experiments obtained using the energies above,namely the patch NL-means model, the patch NL-Poisson model, and a mixture ofthem (see Sec. 7.2). For inpainting real images we use the models with T = 0, but wealso display an example obtained using the annealing scheme. Let us denote E(u,w)any of the energies used below. The numerical algorithm we use is based on thealternating optimization scheme (or EM scheme). The solution of the image updatestep is given explicitly as a nonlocal average (for patch NL-means) or it requires thesolution of a Poisson-type equation (for patch NL-Poisson or the combination ofboth). For computing the weights we use the PatchMatch algorithm in case T = 0or, for T > 0, an extension of it adapted to store queues of L offsets in an L-nearestneighbors field12 (see also Ref. 7).
7.1. A synthetic example
First we consider the inpainting of the regular texture shown in Fig. 2, with twodifferent mean intensities. The inpainting domain hides all patches on the bound-ary between the dark and bright parts of the image. With this example we cantest the ability of each method to synthesize an interface between both regions.Situations like these are common in real inpainting problems, for instance due toinhomogeneous lighting conditions. We have also added Gaussian noise with stan-dard deviation σ = 10 to show the influence of the selectivity parameter T . Wehave tested each method with T = 0 (top row), and T > 0 (bottom row), cho-sen approximately to match the expected deviation of each patch error due to thepresence of noise.
Fig. 2. (Color online) Inpainting of a synthetic texture. The initial condition is shown in the firstcolumn. The other four columns show a zoom (region in the red rectangle) of the results of patchNL-means, and NL-Poisson. Top row, T = 0, bottom row T = 200 and T = 400, respectively. Theintra-patch weight kernel g is shown in the bottom right corner of the initial condition, it has astandard deviation a = 5 and the patch size is s = 15.
1230003-28
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 29–65
Nonlocal Variational Models for Inpainting and Interpolation
Fig. 3. Profiles of the results in Fig. 2. The profiles are taken from a horizontal line going betweenthe circles in Fig. 2. Top: results with T = 0 and bottom: results with T > 0.
Gradient-based method yields a much smoother shading of the texture. Thisis due to the fact that the image update step is computed as the solution of aPDE which diffuses the intensity values present at the boundary of the inpaintingdomain.
As expected, the results using a higher value of T show some denoising andan undesirable loss of texture quality. This effect can be better appreciated in theprofiles shown in Fig. 3, which depict the image values for a horizontal line betweenthe circles. For that reason for the rest of the inpainting experiments shown in thispaper we will consider T = 0. In other applications such as denoising or imageregularization, the case of T > 0 becomes relevant. Although we do not pursuethem in the present work, it would be interesting to explore the application of thisformalism to more general settings following the line of Ref. 69 and our work inRef. 39 on the reconstruction of sparsely sampled images.
7.2. Results on real images
We consider two inpainting methods, variations of our proposed framework, namelypatch NL-means, and NL-Poisson. As discussed in the preceding section we setT = 0 to prevent from blurring. For more details and a comparison with otherstate-of-the-art methods we refer to Refs. 7 and 6.
Combination of gradients and intensities. For patch NL-Poisson the patchsimilarity weights w are computed based only on the gradients of the image. Inmost cases however, the gradient is not a good feature for measuring the patchsimilarity, and it is convenient to consider also the gray level/color data. For thisreason we consider a convex combination of the patch error functions of patchNL-means and NL-Poisson:
E2,λ,0(u,w) =∫
eO
∫eOc
w(x, y)(λ‖pu(x) − pu(y)‖2g
+ (1 − λ)‖pu(x) − pu(y)‖2∇,g)dydx, (7.1)
where the parameter λ ∈ [0, 1) controls the mixture. In this case the Euler equationwith respect to u becomes:
(1 − λ)∆u(z) − λu(z) = (1 − λ)div v(z) − λf(z), for z ∈ O,
u(z) = u(z), for z ∈ ∂O.(7.2)
1230003-29
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 30–65
P. Arias et al.
Here
f(z) :=∫
R2g ∗ w eO, eOc(z − ·, z′ − ·)u(z′)dz′,
v(z) =∫
R2g ∗ w eO, eOc(z − ·, z′ − ·)∇u(z′)dz′,
where we use the notation g ∗ w eO, eOc introduced in (3.5). Observe that f corre-sponds to a patch NL-means image update, and v is a nonlocal average of thegradients in Oc. The problem is linear and can be solved for instance with a con-jugate gradient method. The theoretical results of the paper hold in this case,under the assumptions of the patch NL-Poisson method. From now on, we willuse the term patch NL-Poisson to refer to this model. The case λ = 1 cor-responds to the patch NL-means and will be considered separately. Typicallywe will set λ ≤ 0.1, in this way we include some intensity information in thecomputation of the weights, without departing too much from the pure Poissonmodel.
Multiscale scheme. Exemplar-based inpainting methods have a critical depen-dence with the size of the patch. Furthermore, when the inpainting domain is largein comparison with the patch, the proposed energies have many local minima, andnot all of them are good inpaintings. It is common practice in the literature (e.g.Refs. 52, 55, 83), to incorporate a multiscale scheme. It consists on applying sequen-tially the inpainting method on a Gaussian image pyramid, starting at the coarsestscale. The result at each scale is upsampled and used as initialization for the nextfiner scale. The patch size is constant through scales. This alleviates the criticaldependence with respect to the size of the patch, helps in avoiding local minima,and alleviates the computational cost. In our experiments, the size of the coars-est scale is a 10–20% of the original size, except for a few cases which requiredless subsampling. The number of scales is set such that the subsampling rate isapproximately 0.8 as in Ref. 83.
The results are shown in Figs. 4–6, classified according to the nature of theinpainting problem. The most important parameters are the patch size, the sizeof the coarsest scale, and λ. For the mixing coefficient λ for the patch NL-Poissonmodel in (7.1) we tested two configurations: λ = 0.01 and λ = 0.1. Recall that lowervalues of λ give a higher weight to the gradient component of the energy. This isappropriate for structure images with strong edges.
Copy regions and transition bands. Let us focus now on the solution of theproposed functional in the limit case when T → 0. In this case, the weights w arereplaced by a map ϕ : O → Oc. ϕ(x) corresponds to the center of the most similarpatch to pu(x) (the nearest neighbor).
In Fig. 7 we show some results obtained with the proposed methods. The redcurves depict the boundaries of the regions with constant offset with respect to thenearest neighbor. This offset is given by t(x) = ϕ(x) − x, for x ∈ O. Since ϕ is the
1230003-30
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 31–65
Nonlocal Variational Models for Inpainting and Interpolation
Fig. 4. Results on structured images. KSY: method of Kawai et al.52 M and P stand for patchNL-means and NL-Poisson. From top to bottom sofa, house and motor.
Fig. 5. Results on textured images. KT: method of Komodakis and Tziritas.55 M and P standfor patch NL-means and NL-Poisson. From top to bottom golf and elephant.
NNF, we will refer to t as the offset-to-the-nearest-neighbor field. We can see thatthese region constitute a rather simple partition of O.
Results with gradient-based methods. The patch NL-Poisson performs wellin images with a strong structure. Each iterate is obtained as the result of copyinggradients from the known portion of the image into the hole and then solving thePDE (7.2). Although we do not have a proof of it, in our experiments we observe
1230003-31
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 32–65
P. Arias et al.
Fig. 6. (Color online) Results on repetitive patterns. KT: method of Komodakis and Tziritas.55
M and P stand for patch NL-means and NL-Poisson. From top to bottom matsuri and building 1.
Fig. 7. (Color online) Copy regions. Some results obtained with the proposed schemes with theboundaries of the copy regions superimposed in red. The last two images show the correspondingdistribution of the patch error (energy density). The energy concentrates along the boundaries ofcopy regions.
that the synthesized image will not have edges which are not copied from Oc. Thisis not the case for patch NL-means, which may present discontinuities (seams) atthe boundary of the hole (see, for instance, the result of patch NL-means in sofa,Fig. 4).
For patch NL-Poisson (7.1), the patch error is a combination of intensity andgradients. With the low values of λ used, the gradient component dominates. Forsome textured images this may cause the method to fail. For instance, in baseball(Fig. 5), segments of the sky have been reproduced in the snow. The result withλ ≈ 1 produces a better reconstruction.
Enriching the set of exemplars. The proposed inpainting formalism can be eas-ily extended to handle a set of transformations of exemplars (such as symmetries,rotations, or affinities of the image). In Fig. 8 we show a preliminary result consid-ering as the orbit the identity, vertical, horizontal, and central symmetries of theimage. From a computational point of view, this is equivalent to having an imagewith a larger Oc. It is interesting to point out, that since the computational cost
1230003-32
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 33–65
Nonlocal Variational Models for Inpainting and Interpolation
Fig. 8. (Color online) Symmetries. Result using patch NL-means. To enrich the set of exemplars,eOc is redefined as the union of the complement of the inpainting domain with its vertical, horizontaland central symmetries.
of the PatchMatch algorithm is O(|Ωp||O|), this has little effect on the computingtime.
Deterministic annealing. The proposed formalism is related to the deterministicannealing framework for clustering of Ref. 73. In this work the author presents adeterministic annealing scheme for finding a global minimum for an energy closelyrelated to ours. In our context, this corresponds to minimization of a series ofenergies (E2,λ,Tn)n where Tn is a decreasing sequence of temperatures (e.g. Tn =αnT0). The minimum find for Tn is used as initialization for Tn+1.
Our context is not equivalent to that of Ref. 73, however it is still interestingto explore the application of such an annealing scheme. Indeed, this has alreadybeen used for nonlocal demosaicking24 and in our previous work on interpolationof sparsely sampled images.39
Figure 9 shows some images of the annealing sequence for T = 500, 154, 65, 61, 1.The red curves on the bottom row depict the boundaries of the regions with constantoffset with respect to the nearest neighbor. This offset is given by t(x) = n(x) − x,for x ∈ O, where n(x) ∈ Oc denotes the position of the nearest neighbor of pu(x).When T = 0, n(x) corresponds to the correspondence map ϕ. However, it is stilluseful to show the properties of the solution with T > 0.
Fig. 9. (Color online) Deterministic annealing. Result using patch NL-means. Top: original imageand inpainting domain. Some iterates of a deterministic annealing process, from left to rightT = 500, 154, 65, 61, 1.
1230003-33
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 34–65
P. Arias et al.
When decreasing T , the results vary smoothly, except at some critical tempera-tures in which the solution changes considerably, suggesting the possibility of phasetransitions. This can be seen in Fig. 9: T = 65 and T = 61 correspond to consecu-tive temperatures in the sequence. Their solutions differ considerably. Notice alsothe corresponding change in the regions of constant offset t.
The results seem to support the idea that the optimal correspondence mapsobtained by annealing have the regularity properties described in Theorem 4.2. Wedo not know the answer to this question.
8. Stereoscopic Image Inpainting
As we mentioned in Sec. 1, the recent commercial interest in exhibiting 3D movies orother events like sports or music in theaters has motivated the development of post-production tools in order to assist their acquisition or to eliminate unwanted objectslike rigs or cables which may be unavoidable during filming. In this context, stereoimage inpainting46 addresses the reconstruction of missing information in pairs ofstereo images. In that case, we have to fill-in the holes in both left and right imagesso that the reconstructed parts of both images look like the projection of real 3Dobjects and produce to the viewer a depth-consistent perception.
While in the classical inpainting of 2D images or video one does not take intoaccount the 3D geometry of objects, in the case of stereoscopic image inpainting oneneeds to take into account depth information to produce coherent results.81,46 InRef. 81, the authors adopted a joint color and depth completion method, where colorand depth maps are inpainted simultaneously and the process is done separatelyfor each view. The view consistency is obtained using an iterative process butno convergence is ensured. In Ref. 45, the authors proposed to produce first a3D coherent filling-in of the depth maps using an energy minimization process,then both images of the stereo pair are inpainted simultaneously using a modifiedversion of Ref. 30 that chooses the best patches in restricted image regions that takeinto account the disparity maps (see next paragraph for the definition). Having atdisposal the disparity map, each patch in the left view has a disparity related one onthe right view and the authors of Ref. 45 add the distance between correspondingpatches in both views. A related approach was proposed in Ref. 46: first the objectssurrounding the hole are extended into it using binary inpainting techniques,61,60,37
then the disparity is interpolated into each extended object; finally these objectsare inpainted separately to generate the image. This is done for one of the images(say, left) and the result transferred to the other (right) using the disparity. Theremaining holes of the right image are filled-in by inpainting. For the moment, letus review the method proposed in Ref. 46.
The input of the algorithm is a stereo pair of images I1 : Ω1 → R, I2 : Ω2 → R
and the associated disparity map d : Ω1 → R2. The disparity is the difference of
coordinates between points in both images corresponding to the projection of a 3Dpoint, and is inversely related to the depth of the 3D point. We notice that disparity
1230003-34
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 35–65
Nonlocal Variational Models for Inpainting and Interpolation
is usually represented by a scalar function, after image rectification.44 We denote byΩ1,Ω2 the corresponding image domains that we assume to be rectangles in R
2, sayΩ1 = Ω2 = [0, 1]2. Let us refer to I1, I2 as the left and right images, respectively. Weassume that the disparity map is referenced to the left image, that is, for each x ∈Ω1, I1(x) and I2(x+d(x)) are images of the same 3D point. The disparity map canbe computed from I1, I2 using a favorite stereo algorithm like Ref. 54. Then we selecta region O1 in Ω1, representing an object that we want to eliminate. This createsthe hole O1 in the domain Ω1 of both the disparity d and the left image I1, and wewant to inpaint both of them. The corresponding hole O2 in I2 can be also selectedmanually or can be computed by mapping Ω1 into Ω2 using the disparity map.
The proposed algorithm has two main steps. Let us first describe them as theywere presented in Ref. 46 and comment on how can they be implemented in a moreautomatic way.59
Step 1. The first step is to inpaint the disparity map in O1. This can be donein several ways. As in Ref. 46 we can segment the color image into regions beingmainly interested in the regions surrounding the hole. Then we extend each regioninto the hole and inpaint the disparity map in each of them.
(1) The segmentation step. Segmentation can be achieved using the multiscale algo-rithm for the simplified Mumford–Shah functional62 or the mean-shift algorithm.29
Then we approximate the disparity map in each region of the segmentation by aplane. This set of planes can be used to refine the segmentation by computing theregions of the disparity map that are fitted by this set of planes.46
(2) Extending each region into the hole. Then we can extend each region into the holeby using a binary inpainting algorithm.79 We can also use the geometry inpaintingalgorithm based on the Euler spiral to complete the boundaries of each region asin Ref. 26. This is the approach followed in Ref. 59. After this, each region hasbeen extended into the hole. Notice that two different regions may overlap. Thisambiguity will be solved after inpainting the disparities.
(3) Inpainting the disparities. We inpaint the disparities on each region separately.Notice that as a result, we may assign several disparities to a pixel in the hole.The final assignment corresponds to the larger disparity, i.e. to the smallest depth,since this criterion determines which objects are visible in the hole. This is whywe proceed to inpaint the disparity of each object separately. Now, concerning thedisparity inpainting itself, if the disparity is represented by a plane, we can justextrapolate it into the hole as in Ref. 79. We can also use a more complex modellike the inpainting algorithm based on gradients. We will give a discussion of thisin Sec. 9 (see also Ref. 57).
The output of Step 1 is an inpainted disparity map in O1.
Remark 8.1. Notice that this algorithm can be implemented in a more directway. Indeed, instead of a segmentation we can get a description of the geometry
1230003-35
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 36–65
P. Arias et al.
surrounding the hole using the level lines of the image, and more concretely thesufficiently contrasted level lines known as significant level lines.32 Each level linecan be closed to a region inside the hole using Euler’s spiral26 and the disparityinpainted in that region, as in Step 3 above. That is, the extension of the disparitymap into each region can be done by simple plane extrapolation or by the moresophisticated inpainting algorithm described in Sec. 9. The underlying model is ahigh-order interpolation model.
Step 2. Inpainting the images. A first inpainting strategy is to look for simultaneouscorrespondences between I1 and I2, that is, we compute the correspondence mapϕ :O1 → Oc
1 by minimizing
λ∑x∈O
‖pI1(x) − pI1(ϕ(x))‖2g,2 + ‖pI2(x+ d(x)) − pI2(ϕ(x) + d(ϕ(x)))‖2
g,2
+ (1 − λ)∑x∈O
‖pI1(x) − pI1(ϕ(x))‖2∇,g
+ ‖pI2(x+ d(x)) − pI2(ϕ(x) + d(ϕ(x)))‖2∇,g , (8.1)
where λ ∈ [0, 1]. That is, disparity information can be used not only to achievea coherent inpainting of the left and right images, but to improve the inpaintingitself. This is essentially the approach followed in Ref. 45 with λ = 1 or in Ref. 59with a different energy formulation. We have seen that the use of 3D informationadds more constraints to the problem and permits to obtain more robust results.Moreover, we may restrict the value of the correspondence map to search in thesame region to which the pixel x belongs to, that is, if x belongs to the region R
reconstructed in Step 1, we could search for ϕ(x) also in R.A more simple approach would be to inpaint the left image and then transfer
the inpainting to the right image by using the disparity map.46 Since there areregions which are visible in I2 but not in I1, this may still leave holes in I2 whichare filled-in by 2D inpainting.
We can see the proposed approach as a mixture of geometric- and texture-based methods, where the geometric step fills-in object labels rather than colors. Inaddition, we see that the depth information greatly helps in the geometric reasoning.
8.1. Experiments
First, we illustrate all the process using a football image, then we show some otherexperiments on stereo inpainting.
The images of the stereo pair are displayed in Figs. 10(a) and 10(b) with theholes represented in black, and the corresponding disparity map is displayed inFig. 10(c). The first step of the process consists in computing a segmentation ofthe image in Oc into regions of uniform color. For that we use the mean-shiftalgorithm.29 The segmentation obtained is shown in Fig. 10(d). We observe thatthere are N = 3 regions adjacent to the hole (the green, light gray and dark gray
1230003-36
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 37–65
Nonlocal Variational Models for Inpainting and Interpolation
(a) (b) (c) (d)
Fig. 10. (Color online) From left to right: (a), (b) a stereoscopic image pair representing thesame scene, with the regions to inpaint in black. (c) The disparity map d corresponding to (a),with unknown disparity in the same black region. (d) Segmentation by mean-shift clustering Cof (a).
ones corresponding to the post). For each of these regions, a plane (not shown inthe figure) is fitted to the disparity map.
As shown in Fig. 11, the proposed segmentation by planes of the disparity datahas fused the two planes representing the light and dark gray regions of the postgiven in the color segmentation (at the left part of the image). Indeed, those tworegions have a similar plane equation since they correspond to the same object. Asthere were only three planes adjacent to the hole, this step has reduced the numberof planes to two. The final disparity segmentation associated to Fig. 10(d) is shownin Fig. 11(a) as black and white regions. As a result we have now a set of imageregions, whose geometry is approximated by a set of planes.
The disparity of each pixel in the hole is determined by the visible region thatcontains it, i.e. the closest to the camera. Notice that several extended regions maycontain the pixel because the disparity extension has been done independently foreach object. While simple, this is an important step of the algorithm. By computingthe visibility of the regions, we determine when one of them becomes hidden byanother. This kind of geometric reasoning is not possible without the disparitymaps and is the reason of the improvements over the 2D inpainting methods. InFig. 11(c), we see the infinite extension of the post layer produced by the smoothingprocedure, while in Fig. 11(d), we can see the post stopping on the ground.
(a) (b) (c) (d)
Fig. 11. (Color online) Illustration of the region extension procedure with two adjacent regions.From left to right: (a) Disparity plane segmentation of the known part Ωc of the image in Fig. 10(a).We have two disparity planes displayed in black S1 and white S2, and the inpainting mask Ω inblue. (b) and (c) Extension of the disparity layers S1 and S2 in the hole Ω using Ref. 79. (d) Visible
part of the extended regions with respect to the two layers plane equations (the gray surroundedregion is the visible part of the white extended region).
1230003-37
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 38–65
P. Arias et al.
(a) (b) (c) (d)
Fig. 12. (Color online) Disparity and disparity based color inpainting of the soccer image(Fig. 10). From left to right: (a) The inpainted disparity map. (b) Inpainting of the first layer(region surrounded in gray Fig. 11(d)). (c) Final result obtained after filling-in the backgroundregion. (d) Result using the 2D exemplar-based algorithm in Ref. 30.
Figure 12 illustrates the inpainting process of image in Fig. 11(a). Figure 12(a)shows the inpainted disparity map. Figure 12(b) is the result of inpainting thedisparity layer corresponding to the dark post (i.e. the gray surrounded region inFig. 11(d)). Figure 12(c) shows the final result, where the background layer has alsobeen inpainted with its corresponding texture. The segmentation and the restrictionof the search to similar patches in the same region are responsible for the fact thatonly grass texture, without white lines, has been copied into the background layer.Figure 12(d) shows the result of the original inpainting algorithm30 without searchrestriction.
Finally, the result of stereoscopic inpainting is shown in Fig. 13. Figure 13(a)shows the inpainting of the right image of the stereo pair displayed in Fig. 10(b)obtained by transferring the inpainted left image in Fig. 12(c) using the inpainteddisparity map in Fig. 12(a). Notice that there are still some unfilled-in regions.Figure 13(b) shows the final inpainting result after filling-in the remaining black areausing the algorithm in Ref. 45. Figures 13(c) and 13(d) show the results obtainedby using Ref. 81 for the left and right images of the stereo pair. Notice that thisalgorithm creates white lines that are not 3D consistent between the two imageswhile the described method ensures perfect correspondence of the inpainted partsof the two images according to the disparity maps.
(a) (b) (c) (d)
Fig. 13. (Color online) Illustration of the stereoscopic inpainting technique for the stereoscopicimage pair of Fig. 10. From left to right: (a) Partial filling-in of Fig. 10(b) (the right image)obtained by transferring the inpainted image in Fig. 12(c) using the inpainted disparity map inFig. 12(a). The unfilled-in region remains in black. (b) Final inpainting result after filling-in theremaining black area using the algorithm in Ref. 45. (c), (d) Results for left and right imagesobtained using Ref. 81.
1230003-38
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 39–65
Nonlocal Variational Models for Inpainting and Interpolation
(a) (b)
(c) (d)
Fig. 14. (Color online) Stereoscopic image inpainting illustrations. First row: (a), (b) A stereo-scopic image pair with the inpainting mask (in blue). Second row: (c) and (d) The inpaintingresults obtained using the method described.
Figures 14(a) and 14(b) show the initial stereo pair with the inpainting mask (inblue). Figures 14(c) and 14(d) show the inpainting results obtained by the methoddescribed in Ref. 46.
9. Gradient-Based Neighborhood Filters for Image Interpolation
The nonlocal Poisson model for image inpainting can be considered as a higher-order model (involving a nonlocal derivative of the gradient). As we have seenit permits to create smooth transitions between given data. Inspired by this, westudy in this section a gradient-domain extension of the bilateral filter78 (namedas neighborhood filter in Ref. 86) for disparity interpolation from incomplete orsparse data for which higher-order interpolation may be required. For instance, wemay need to interpolate disparity in a given region as a step for stereo inpainting.
1230003-39
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 40–65
P. Arias et al.
We may also be given an incomplete disparity map that may be obtained as aresult of a stereo algorithm or could be acquired by a camera sensor, like a Time-of-Flight camera (ToF), and a corresponding reference image. As in the bilateralfilter, the reference image constraints the comparison of disparity gradients (and/orvalues) to neighboring pixels that have a similar color. This is encoded by the weightfunction w(x, y) that this time is given beforehand and is based on pixel (or patch)comparison. By means of w(x, y) the edge information of the image is incorporatedinto the disparity map, ensuring that discontinuities in disparity are consistent withgray level (or color) discontinuities. But in contrast to the bilateral filter, that wouldcompute at a pixel x a weighted average of the given disparity map at pixels y ona neighborhood of x, say Nx, (the weights being given in terms of the distance andthe color similarity between x and y), we solve a Poisson equation that tries to copyat x the weighted average of gradients at pixels y ∈ Nx. We will show that use ofgradient information permits to extend the planes existing in the given disparitydata.
We start by reviewing in Sec. 9.1 the bilateral filter, then we present in Sec. 9.2its gradient-based extension and we display in Sec. 9.3 comparative experiments.In the present formulation, with weights depending only on a reference image, weshow in Sec. 9.4 that the underlying local energy functional obtained as Γ-limitof the nonlocal energy associated to the gradient-domain extension of the bilateralfilter involves second-order derivatives, explaining thus the ability of the proposedmodel to obtain higher-order interpolations.
9.1. The bilateral filter
The bilateral filter78 is a example of neighborhood filter86 which takes into accountthe gray level values of the image to define neighboring pixels. Let I : Ω → R
n bea given image, where n = 1 for a gray level image or n = 3 for a color image. Asusual, the image domain Ω is a rectangle in R
2. Then the bilateral filter is given by
B(I)(x) =1c(x)
∫Ω
wB(x, y)I(y)dy, (9.1)
where wB : Ω × Ω → R+ is given by
wB(x, y) = exp(−|x− y|2
2k2− ‖I(x) − I(y)‖2
2h2
), (9.2)
and c(x) =∫ΩwB(x, y)dy is the normalization factor, k, h > 0. Notice that wB(x, y)
incorporates both the Euclidean distance of y to the reference pixel x and thephotometric distance based on the comparison of I(x) and I(y). The spatial extendof the neighborhood is controlled by k, whereas the photometric extend is controlledby h. Thus the effective extend of the photometric neighborhood is given by a packetof level lines y ∈ Ω : I(x) − 3h ≤ u(y) ≤ I(x) + 3h. When h and k are of the
1230003-40
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 41–65
Nonlocal Variational Models for Inpainting and Interpolation
same order, the behavior is similar to the Perona–Malik equation67 which inhibitsdiffusion along large gradients of the reference image.22
The bilateral filter has found many applications to different problems in imageand surface processing. In particular, to the processing of disparity maps. Indeed,many correlation-based stereo matching algorithms provide a sparse disparity mapof reliable data which can be improved by filtering. In Refs. 87, 85, 33 the authorsperform, respectively, disparity super-resolution and interpolation by iterating abilateral filter on the disparity map u : Ω → R:
uk+1(x) =1c(x)
∫Ω
wB(x, y)uk(y)dy, (9.3)
where the weights are given by (9.2) and incorporate a registered high-quality ref-erence image I. Thus, disparity values are diffused with weights given by the simi-larities of the reference image in a neighborhood of x.
The iterative bilateral filtering can be interpreted as a process to minimize thenonlocal energy functional
RB(u) =∫
Ω
∫Ω
wB(x, y)(u(x) − u(y))2dydx. (9.4)
As we already mentioned in Sec. 1, this type of variational formulation of nonlocalfilters86,78,23 was proposed in Ref. 42.
The energy (9.4) favors constant or piecewise constant minima (on regions sep-arated by high image gradients). This motivates the use of RB as a regularizationterm, to be used together with a data attachment term to the initial disparity:
E(u) = D(u) + λRB(u), λ > 0. (9.5)
In the context of data interpolation we consider the following data attachment
D(u) =∫
Ω
β(x)|u(x) − u0(x)|pdx, (9.6)
where u0 is a given data, β(x) is a confidence mask (or the mask where the data isknown), and p = 1, 2.
The regularization term RB is based on the assumption that nearby locationswith similar colors should correspond to the same object and have similar disparitysince the object’s surface is assumed to vary smoothly. Although this assumptionworks well in many cases, it favors piecewise constant disparity maps (see Fig. 15).
9.2. A gradient-based neighborhood filter
Taking into account the limitations of the bilateral filter, we propose a higher-ordermodel so that nearby locations with similar colors have similar disparity gradient.This is equivalent to assume that objects are locally planar. To achieve this, the
1230003-41
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 42–65
P. Arias et al.
0 20 40 60 80 10030
40
50
60
70
80
90
100
110
120
x coordinate
dept
h
lambda = 1.0lambda = 0.0lambda = 0.0 + gu
0 20 40 60 80 10030
40
50
60
70
80
90
100
110
120
x coordinate
dept
h
lambda = 1.0lambda = 0.0lambda = 0.0 + gu
Fig. 15. (Color online) Interpolation of two planes. First row: ground truth disparity data, refer-ence image, image with the computed edge map in black, and mask displaying in white the regionwhere the disparity is unknown. Second row left: a profile displaying the result of interpolationusing (9.8). We assume that the disparity data is only known outside the mask. The figure displaysthe result obtained with the bilateral filter (λ = 1, in red), the gradient model (λ = 0) withoutthe edge map (in green), and the gradient model with the edge map (in blue). Second row right:a profile displaying the result of interpolation for the same disparity data with a Gaussian noiseof standard deviation 5. The model filters the disparity map outside the mask and interpolates itinside the mask. Again we display the result obtained with the bilateral filter, the gradient modelwithout the edge map, and the gradient model with the edge map.
corresponding regularization term is
RP(u) =∫
Ω
∫Ω
wP (x, y)|∇u(x) −∇u(y)|2dydx. (9.7)
The weights wP (x, y) correspond to a slight modification of the bilateral weightswB with the aim of strengthening the ability of recovering a disparity map withdiscontinuities (or with high gradients). They are located at high gradients of theimage.
Thus, we define a discontinuity map σ : Ω → [0, 1] via
σ(x) = 1 − 11 + exp(−(|∇I(x)|2 − s0)/τ)
,
where I is the smoothed reference image by a Gaussian kernel. This is a (decreasing)soft-thresholding function around s0 where −(2τ)−1 is the slope of the function ats0. We incorporate the discontinuity map into the weights, defining a new weightfunction:
wP (x, y) = wB(x, y)σ(x)σ(y).
1230003-42
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 43–65
Nonlocal Variational Models for Inpainting and Interpolation
Strictly speaking, discontinuities are only allowed when σ(x) = 0. Still, small non-zero values favor high gradients of the disparity map (if needed).
In practice it is convenient to combine both regularization terms and use thefollowing energy:
E(u) = D(u) + λRB(u) + (1 − λ)RP (u), λ ∈ [0, 1], (9.8)
in which the regularization term is a convex combination of both regularizers dis-cussed before.
9.3. Experiments
A synthetic example with two planes. In Fig. 15 we display a synthetic imagewith two planes separated by a discontinuity to test our algorithm. The disparitymap (the data) is only known outside the mask and we know the reference image.We assume that the discontinuities of the disparity map correspond to the discon-tinuities of the reference image. Thus the edge map is computed on the referenceimage and determines the map σ(x). In the second row of Fig. 15 we show a profileof the interpolated disparity in the given mask: the left figure shows the results ofthe bilateral filter (λ = 1, in red), the gradient model (λ = 0) without the edgemap (in green), and the gradient model with the edge map (in blue). The rightfigure shows a profile of the interpolated result for the same disparity data with aGaussian noise of standard deviation 5. The model filters the disparity map outsidethe mask and interpolates it inside. Again we display the result obtained with thebilateral filter, the gradient model without the edge map, and the gradient modelwith the edge map.
A synthetic example of a sphere. In Fig. 16 we display a synthetic image witha sphere. The data is only known outside the mask (middle of the first row) and weknow the reference image from which we have extracted the edge map (right of thefirst row). We used (9.8) to interpolate the data in the given mask. In the second rowof Fig. 16 we show a profile of the results: we display the result obtained using thegradient model (λ = 0) and the result obtained using the bilateral filter (λ = 1). Inboth cases we display the profile of the original sphere and the interpolated result.
An example with a real test image. In Fig. 17 we display a real test imagecorresponding to the left image of a stereo pair. The context of the experiment is theelimination of an object from both images of the stereo pair and the simultaneouslyinpainting of the corresponding holes left by the object. As explained in Sec. 8 wefirst proceed to inpaint the disparity in the hole of Fig. 17(a), displayed Fig. 17(b).To inpaint the disparity, we first extend the objects incoming into the hole. Forthat we extended the curves corresponding to the significantly contrasted levellines (Fig. 17(c)) in the sense of Ref. 32. This corresponds to extend into the holethe objects surrounding it. Then we inpaint the disparity in each region of thehole determined by the extension of these lines. The results corresponding to the
1230003-43
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 44–65
P. Arias et al.
20 40 60 80 100
10
20
30
40
50
60
70
80
90
10020 40 60 80 100
10
20
30
40
50
60
70
80
90
100
0 20 40 60 80 10050
55
60
65
70
75
80
85
0 20 40 60 80 10050
55
60
65
70
75
80
85
Fig. 16. (Color online) Interpolation of a section of a sphere. First row: ground truth data, maskdisplaying in black the region where the data is unknown, and image with the computed edge mapin black. Second row: profiles displaying the results obtained using the gradient model (λ = 0 in(9.8)) and the bilateral filter (λ = 1).
bilateral filter (λ = 1) and the gradient model (λ = 0) are displayed in Figs. 17(d)and 17(e). The profiles of the interpolation are displayed in Fig. 17(f). The profilecorresponds to the line x = 130, displayed as a white line in Fig. 17(e). The profilestarts at y = 0 which corresponds to the top of the line (the origin of coordinates islocated at the upper left corner of the image, the x- and y-axis being the horizontaland vertical ones). The cone ends at y = 241 and then continues the plane of thetable. The profile shows that the gradient-based model captures better the planeprofile.
9.4. The local model obtained as Γ-limit of the gradient-based
neighborhood filter
In order to justify the performance of the gradient-based model to produce higher-order interpolations we prove that the underlying local model obtained by asymp-totic rescaling of the neighborhood functions is a second-order anisotropic energythat incorporates the information of the reference image. This adds further justi-fication to our experiments that exhibit this higher-order interpolation behavior.The proofs is based on the results in Refs. 19 and 71. For the approximation oflocal diffusion equations by their nonlocal counterparts we refer to Ref. 3.
1230003-44
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 45–65
Nonlocal Variational Models for Inpainting and Interpolation
(a) (b)
(c) (d)
0 50 100 150 200 250 300 350 40060
80
100
120
140
160
180
200
220lambda = 1.0lambda = 1E-6
(e) (f)
Fig. 17. (Color online) Interpolating the disparity in a real image. First row: (a) original image,(b) the mask corresponding to the object we want to remove, (c) the completion of the significantlevel lines arriving at the boundary of the hole. Second row: (d) Interpolation results obtainedwith the bilateral filter (λ = 1), (e) the gradient model (λ = 0), and (f) the profile of the solutions.The profile corresponds to the line x = 130 displayed as a white line in (e). The profile starts aty = 0 which corresponds to the top of the line. The cone ends at y = 241 and then continues theplane of the table. The gradient-based model captures better the plane profile.
1230003-45
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 46–65
P. Arias et al.
Let Ω be an open bounded set in RN which represents the image domain. We
assume (H)Ω: there is a constant CΩ > 0 such that given any two points x, y ∈ Ωthere exists a curve γ connecting x to y with L(γ) ≤ C|x− y|, where L(γ) denotesthe length of γ. Let I : Ω → R be a given image. We assume that I ∈ W 2,∞(Ω). Inpractice this means that we have convolved a given and less regular image with aGaussian kernel.
Let ρ ∈ L1(RN ), ρ ≥ 0. Let g ∈W 1,∞(R) be such that
minr∈B
g(r) ≥ αB > 0 for any compact subset B ⊂ R. (9.9)
Let A ∈ W 1,∞(Ω × Ω), A ≥ a > 0.Let us consider the energy P :L2(Ω) → R
+ defined by
P(u) =∫
Ω
∫Ω
ρ(x− y)g(I(x) − I(y))|∇u(x) −∇u(y)|2A(x, y)dxdy, (9.10)
if u ∈W 1,2(Ω), and P(u) = +∞ if u ∈ L2(Ω)\W 1,2(Ω).Let ρε(x) = 1
εN ρ(xε ), x ∈ R
N , ε > 0. Let us rescale this energy as
Pε(u) =1ε2
∫Ω
∫Ω
ρε(x − y)g(I(x) − I(y)
ε
)|∇u(x) −∇u(y)|2A(x, y)dxdy,
(9.11)
if u ∈W 1,2(Ω). If u ∈ L2(Ω)\W 1,2(Ω), we define Pε(u) = +∞.Let
Q(w) =∫
RN
ρ(z)g(〈w, z〉)z ⊗ zdz. (9.12)
We define
P0(u) =∫
Ω
Trace(D2u(x)Q(∇I(x))D2u(x))A(x, x)dx, (9.13)
if u ∈W 2,2(Ω), and P0(u) = +∞ if u ∈ L2(Ω)\W 2,2(Ω).
Theorem 9.1. The energies Pε Γ-converge to P0 as ε→ 0+.
This result will be a consequence of Propositions 9.1 and 9.2.
Proposition 9.1. Let uε ∈W 1,2(Ω), uε → u in L1(Ω). Then
P0(u) ≤ lim infε→0+
Pε(uε). (9.14)
Proof. The result being obviously true if the right-hand side of (9.14) is +∞, wemay assume that Pε(uε) is bounded. By (H)Ω we have that∣∣∣∣I(x) − I(y)
ε
∣∣∣∣ ≤ CΩ‖∇I‖∞ |x− y|ε
. (9.15)
1230003-46
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 47–65
Nonlocal Variational Models for Inpainting and Interpolation
Since the right-hand side is bounded when x−yε is in the support of ρ, using (9.9)
we have that∫Ω
∫Ω
ρε(x − y)|∇uε(x) −∇uε(y)|2
ε2A(x, y)dxdy ≤ 1
αPε(uε) ≤ C, (9.16)
for some α > 0 and some constant C > 0.Let us prove that we may assume that ρ is of compact support in R
N . Otherwisewe replace ρ by ρχB(0,r) and we prove that
P0(ρχB(0,r), u) ≤ lim infε→0+
Pε(ρχB(0,r), uε) ≤ lim infε→0+
Pε(ρ, uε). (9.17)
where we made explicit the dependence of the energies on the kernel ρχB(0,r).Letting r → ∞, we obtain (9.14). Thus, we assume that ρ is of compact support inR
N . Without loss of generality assume that the support of ρ is the ball B(0, 1).
Step 1. Regularization of uε and preliminary inequalities. Let χ : RN → R be asmooth mollifying kernel with support in B(0, 1), χ ≥ 0,
∫RN χ(x)dx = 1. Let
χδ(x) = 1δN χ(x
δ ), x ∈ RN , δ > 0. Let Ωδ := x ∈ Ω : dist(x, ∂Ω) ≥ δ. Let
uεδ = χδ ∗ uε. To avoid a cumbersome notation, let us write A instead of A(x, y),Ah instead of A(x− h, y − h), A instead of A(x, x), χδ
x,y = χΩδ(x)χΩδ
(y), and
gε(x, y) = g
(I(x) − I(y)
ε
),
unless a more explicit notation is necessary. Then∫Ωδ
∫Ωδ
ρε(x− y)gε(x, y)|∇uεδ(x) −∇uεδ(y)|2Adxdy
≤∫
Ωδ
∫Ωδ
∫B(0,δ)
χδ(h)ρε(x− y)gε(x, y)|∇uε(x+ h) −∇uε(y + h)|2Adxdydh
=∫
RN
∫RN
∫B(0,δ)
χδ(h)χδx,yρε(x − y)gε(x, y)|∇uε(x+ h)
−∇uε(y + h)|2Adxdydh
=∫
Ω
∫Ω
∫B(0,δ)
χδ(h)χδx−h,y−hρε(x− y)gε(x− h, y − h)|∇uε(x)
−∇uε(y)|2Ahdxdydh
≤∫
Ω
∫Ω
ρε(x− y)gε(x, y)|∇uε(x) −∇uε(y)|2Adxdy
+∫
Ω
∫Ω
∫B(0,δ)
χδ(h)χδx−h,y−hρε(x− y)Qε(x, y, h)|∇uε(x)
−∇uε(y)|2dxdydh:= T1 + T2,
1230003-47
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 48–65
P. Arias et al.
where
Qε(x, y, h) = gε(x − h, y − h)Ah − gε(x, y)A.
When x, y ∈ Ω, x− h, y− h ∈ Ωδ, the segments [x− h, x], [y − h, y] ⊂ Ω and, using(H)Ω, we have∣∣∣∣I(x− h) − I(y − h)
ε− I(x) − I(y)
ε
∣∣∣∣ ≤ |h|ε
∫ 1
0
|∇I(x− sh) −∇I(y − sh)| ds
≤ CΩ‖D2I‖∞ |x− y|ε
|h|. (9.18)
Writing
gε(x− h, y − h)Ah − gε(x, y)A
= gε(x− h, y − h)(Ah −A) + (gε(x− h, y − h) − gε(x, y))A,
and using (9.16) and (9.18) we have
1ε2|T2| ≤ C1
∫Ω
∫Ω
∫B(0,δ)
χδ(h)|h|ρε(x− y)|∇uε(x) −∇uε(y)|2
ε2dxdydh
+C2
∫Ω
∫Ω
∫B(0,δ)
χδ(h)|h|ρε(x − y)|x− y|ε
|∇uε(x) −∇uε(y)|2ε2
Adxdydh
≤ CC1
aδ + CC2Cρδ,
where C1 = ‖g‖∞‖A‖W 1,∞ , C2 = CΩ‖D2I‖∞‖∇g‖∞, and Cρ is a bound in thecompact support of ρ and some C > 0.
Step 2. Let δ > 0 be fixed. Let uδ = χδ ∗ u and uiδ = ∂uδ
∂xi. We prove that
1ε2
∫Ωδ
∫Ωδ
ρε(x − y)gε(x, y)|∇uεδ(x) −∇uεδ(y)|2Adydx
→N∑
i=1
∫Ωδ
∫RN
ρ(z)g(〈∇I(x), z〉)|〈∇uiδ, z〉|2Adzdx
as ε→ 0+. First, observe that1ε2
∫Ωδ
ρε(x − y)gε(x, y)|∇uεδ(x) −∇uεδ(y)|2Ady
≤ ‖D2uεδ‖2∞‖g‖∞‖A‖∞
∫Ωδ
|x− y|2ε2
ρε(x− y)dy
≤ C(δ)
for any x ∈ Ωδ and ε < δ, where C(δ) is a constant that does not depend on ε.Let ui
εδ = ∂uεδ
∂xi. Let y = x− εz, z ∈ B(0, 1). Since
g
(I(x) − I(x− εz)
ε
)= g
(∫ 1
0
〈∇I(x − sεz), z〉ds),
1230003-48
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 49–65
Nonlocal Variational Models for Inpainting and Interpolation
we have
1ε2
∫Ωδ
ρε(x− y)gε(x, y)|∇uεδ(x) −∇uεδ(y)|2Ady
=N∑
i=1
∫RN
χΩδ(x− εz)ρ(z)gε(x, x− εz)
|uiεδ(x) − ui
εδ(x − εz)|2ε2
A(x, x − εz)dz
=N∑
i=1
∫RN
χΩδ(x− εz)ρ(z)g
(∫ 1
0
〈∇I(x− sεz), z〉ds),
∣∣∣∣∫ 1
0
〈∇uiεδ(x− sεz), z〉 ds
∣∣∣∣2A(x, x − εz)dz
→N∑
i=1
A(x, x)∫
RN
ρ(z)g(〈∇I(x), z〉)|〈∇uiδ, z〉|2dz
as ε→ 0+. Then Step 2 follows by the dominated convergence theorem.
Step 3. Let us prove that
N∑i=1
∫Ω
∫RN
ρ(z)g(〈∇I(x), z〉)|〈∇ui(x), z〉|2Adzdx
≤ lim infε→0+
1ε2
∫Ω
∫Ω
ρε(x− y)gε(x, y)|∇uε(x) −∇uε(y)|2Adxdy.
Let Ω′ ⊂⊂ Ω. Let δ > 0 be small enough so that Ω′ ⊂⊂ Ωδ ⊂⊂ Ω. Using theresults of Steps 1 and 2 and letting ε→ 0+ we have
N∑i=1
∫Ω′
∫RN
ρ(z)g(〈∇I(x), z〉)|〈∇uiδ(x), z〉|2Adzdx
≤ lim infε→0+
1ε2
∫Ω
∫Ω
ρε(x− y)gε(x, y)|∇uε(x) −∇uε(y)|2Adxdy + C(δ),
where C(δ) is a constant independent of ε. Letting δ → 0+ and Ω′ ↑ Ω, Step 3follows.
Step 4. Conclusion. Since
N∑i=1
ρ(z)g(〈∇I(x), z〉)|〈∇ui(x), z〉|2
= Trace
(ρ(z)g(〈∇I(x), z〉) (z ⊗ z)
N∑i=1
(∇ui(x) ⊗∇ui(x))),
1230003-49
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 50–65
P. Arias et al.
then ∫RN
N∑i=1
ρ(z)g(〈∇I(x), z〉)|〈∇ui(x), z〉|2
= Trace
(Q(∇I(x))
N∑i=1
(∇ui(x) ⊗∇ui(x)))
= Trace(D2u(x)Q(∇I(x))D2u(x)).
Then (9.14) follows from this and Step 3.Let us recall the following simple result.19
Lemma 9.1. Assume that Ω is a bounded domain in RN with Lipschitz boundary.
Let w ∈ W 1,p(Ω), 1 ≤ p <∞ and let ρ ∈ L1(RN ), ρ ≥ 0. Then∫Ω
∫Ω
ρ(x− y)|w(x) − w(y)|p
|x− y|p dxdy ≤ C‖w‖W 1,p‖ρ‖1. (9.19)
Proposition 9.2. Let u ∈ W 2,2(Ω). Then
P0(u) = limε→0+
Pε(u). (9.20)
Proof. Since
Trace(D2u(x)Q(∇I(x))D2u(x)) =N∑
i=1
〈Q(∇I(x))∇ui(x),∇ui(x)〉,
it suffices to prove that
limε→0+
∫Ω
∫Ω
ρε(x− y)g(I(x) − I(y)
ε
) |ui(x) − ui(y)|2ε2
A(x, y)dxdy
=∫
Ω
〈Q(∇I(x))∇ui(x),∇ui(x)〉A(x, x)dx, (9.21)
for all i = 1, . . . , N .First we observe that it suffices to assume that u ∈ C2(Ω). Let us write v = ui.
Let w ∈W 1,2(Ω). Let
Vε =√ρε(x − y)
√g
(I(x) − I(y)
ε
) |v(x) − v(y)|ε
√A(x, y),
Wε =√ρε(x − y)
√g
(I(x) − I(y)
ε
) |w(x) − w(y)|ε
√A(x, y).
1230003-50
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 51–65
Nonlocal Variational Models for Inpainting and Interpolation
Then
|‖Vε‖2 − ‖Wε‖2|2 ≤ ‖Vε −Wε‖22
=∫
Ω
∫Ω
ρε(x − y)g(I(x) − I(y)
ε
)
· ||v(x) − v(y)| − |w(x) − w(y)||2ε2
A(x, y)dxdy,
≤∫
Ω
∫Ω
ρε(x − y)g(I(x) − I(y)
ε
)
· |(v(x) − v(y)) − (w(x) − w(y))|2ε2
A(x, y)dxdy,
since ρ has compact support ρε(x−y) = 0 if and only if |x−y|ε ≤ C for some constant
C > 0. In that case 1ε ≤ C
|x−y| . We can continue the above inequalities
≤ C
∫Ω
∫Ω
ρε(x− y)g(I(x) − I(y)
ε
) |(v(x) − v(y)) − (w(x) − w(y))|2|x− y|2 A(x, y)dxdy
≤ C‖g‖∞‖ρ‖1‖A‖∞‖v − w‖W 1,2(Ω),
where the last inequality follows from Lemma 9.1. Thus, by density we may assumethat ui ∈ C1(Ω), i.e. u ∈ C2(Ω). Then proceeding as in Step 2 of Proposition 9.1we obtain (9.21).
Remark 9.1. Let us consider ρ(z) = 1
(2πσ2)N2e−
|z|22σ2 , σ > 0, z ∈ R
N , g(r) = e−r2
2h2 ,
r ∈ R, h > 0. Let Q(w) be given by (9.12). Let B = 1σ2 I + 1
h2w ⊗ w. Then
Q(w) =1
(2πσ2)N2
∫RN
e−〈Bz,z〉
2 z ⊗ zdz
=1
(2πσ2)N2
(detB)−12
∫RN
e−〈z,z〉
2 B− 12 z ⊗B− 1
2 zdz.
Since
detB =1σ2N
(1 +
σ2
h2|w|2
),
and
1(2π)
N2
∫RN
e−〈z,z〉
2 z ⊗ zdz = I,
we have
Q(w) = σ2
(1 +
σ2
h2|w|2
)− 12(I +
σ2
h2w ⊗ w
)−1
.
1230003-51
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 52–65
P. Arias et al.
Thus
Q(∇I(x)) = σ2
(1 +
σ2
h2|∇I(x)|2
)− 12(I +
σ2
h2∇I(x) ⊗∇I(x)
)−1
is an anisotropic tensor.
Remark 9.2. In a similar way, if
Bε(u) =1ε2
∫Ω
∫Ω
ρε(x− y)g(I(x) − I(y)
ε
)|u(x) − u(y)|2A(x, y)dxdy,
u ∈ L2(Ω), (9.22)
and we define
B0(u) =∫
Ω
〈Q(∇I(x))(∇u(x)),∇u(x)〉A(x, x)dx, (9.23)
if u ∈W 1,2(Ω), and B0(u) = +∞ if u ∈ L2(Ω)\W 1,2(Ω), we have
Theorem 9.2. As ε→ 0+, the energies Bε Γ-converge to the energy B0.
Remark 9.3. One can also compute the Γ-limit in L2(Ω) of bilateral filter energiesin the case of a faster rescaling
Qaε (u) =
1ε2
∫Ω
∫Ω
ρε(x− y)max(Cε(x), Cε(y))
g
(I(x) − I(y)
ε1+α
)|u(x) − u(y)|2A(x, y)dxdy,
(9.24)
where u ∈ L2(Ω), α > 0, and
Cε(x) =∫
Ω
ρε(x− y)g(I(x) − I(y)
ε1+α
)dy.
In that case the matrix Q(∇I(x)) is replaced by Qa(∇I(x)) = cPx where Px =I − ∇I(x)⊗∇I(x)
|∇I(x)|2 , if ∇I(x) = 0, and Px = I if ∇I(x) = 0. We assume that the set ofpoints x ∈ Ω where ∇I(x) = 0 is a null set. We notice that it is also not difficultto compute the Γ-limit in W 1,2(Ω) in the case of gradient energies with a fasterrescaling although this result is not fully satisfactory.
10. Some Remarks on Video Inpainting
The problem of video inpainting is of a much more complex nature. The variety ofpossible situations is too large and we do not dispose at the moment of right tools toaddress this problem. The extension of existing 2D methods to 3D (2D+time) hasbeen proposed in Ref. 83 but it is mostly adapted to periodic motions where we haveexamples of the moving object that we want to reconstruct. Moreover, they assumea static background. Other attempts are addressed to background reconstructionand inpainting objects that move slowly with respect to the background.50,64,49
1230003-52
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 53–65
Nonlocal Variational Models for Inpainting and Interpolation
Even for static objects that inherit the camera motion the difficulties raised by thereconstruction of textured areas with noise varying in time and changing illumina-tion conditions or shadows are considerable. Thus the identification of simple andgeneric problems that can be solved would be an important step in order to makeprogress towards disposing of automatic tools for video inpainting. Here we wantonly to mention one of those problems: the problem of propagating forward in timethe inpainting obtained at a given frame (like in object removal) keeping the sharp-ness of the original image and adapting to the changing illumination conditionsalong time. The techniques can also be adapted to other cases: the correction ofartifacts due to illumination changes in inpainted video sequences, or the problemof object insertion and propagation in a video sequence.
We restrict our study to the case where the inserted or removed object is affixedto a surface (for instance a sign on a wall). However, we do not restrict the motionsof the camera nor of the surface in the scene. The method also copes with occlusionsand disocclusions of the inserted/removed object.
10.1. Video editing model (continuous setting)
Let us describe some simple models for the above tasks.40 Let us first discussthe case of forward propagation of a inpainting result obtained at a given frame.As above, let Ω ⊂ R
2 be a rectangular open set representing the image domainand [0, T ] be the temporal domain. A point moving in the real world describes atrajectory s : [0, T ] → Ω when seen in a video. The velocity field v(s(t), t) = d
dts(t)characterizes the motion of all the particles in the video. Then, given any particleand the associated trajectory s, the brightness temporal consistency means thatalong s the gray level of u(s(t), t) is constant, which can be stated as
d
dtu(s(t), t) = 0.
Applying chain’s rule to this equation we obtain
∂vu(x, t) := ∇xu(x, t) · v(x, t) +∂
∂tu(x, t) = 0 ∀ (x, t) ∈ Ω × [0, T ]. (10.1)
The temporal consistency can be stated in terms of the convective derivative:∂vu(x, t) = 0, ∀ (x, t) ∈ Ω × [0, T ].
Let O ⊂ ΩT := Ω × [0, T ] be the spatio-temporal domain where the editing isperformed (Fig. 18 illustrates these domains) and let Ot := x ∈ Ω : (x, t) ∈ O.As usual ∂O and ∂ΩT denote their respective boundaries. Assume that ∂O is aLipschitz boundary. We denote by νO(x, t) the outer unit normal to ∂O at thepoint (x, t).
In order to propagate the information in time we solve
minu
∫O
|∂vu(x, t)|2dxdt, (10.2)
1230003-53
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 54–65
P. Arias et al.
Fig. 18. Illustration of the spatio-temporal domain.
where we add Dirichlet boundary conditions
u(x, 0) = f(x, 0) x ∈ O0
u(x, t) = f(x, t) (x, t) ∈ ∂O\∂ΩT and (v, 1) · νO = 0.
Notice that the Euler–Lagrange equations will be supplemented with Neumann-type boundary conditions in the remaining part of the boundary, that is,
∂vu(x, t) = 0 (x, t) ∈ ∂O ∩ ∂ΩT \O0. (10.3)
Notice that (10.3) gives the boundary condition in OT . If we want to incorporateinformation coming from OT we can put a Dirichlet-type boundary condition there.Although this energy functional propagates information in time, it does not adaptto changing illumination conditions. The simultaneous propagation of informationwith illumination correction has been addressed in Ref. 74.
Let us now discuss the problem of correcting the illumination in inpainted videosequences. That is, we assume that we already have the hole O filled-in with animage whose geometry is correct although its illumination is not consistent withits surroundings. For a single frame, the problem is solved by Poisson editing.30
Let O0 be a hole in Ω (with a Lipschitz boundary) and we know the image I inΩ\O0. Assume that u ∈ W 1,2(Ω\O0). To copy an image u0 in O0 (assume thatu0 ∈ W 1,2(O0)) while adapting to the illumination of I, we describe the geometryof u0 by its gradient ∇u0, and we solve
minu
∫O0
‖∇u−∇u0‖2dx, with u|∂O0 = I|∂O0 ,
where I|∂O0 denotes the trace of I taken from outside O0. Then one redefines I = u
in O0. The boundary condition ensures the good continuation with its surround-ings, the geometry being inherited from u0. This technique shows that the gradientperforms an excellent job to define object’s geometry. The solution of this prob-lem is computed by solving the Poisson equation ∆u = ∆u0 in O0 with Dirichletboundary conditions u|∂O0 = I|∂O0 .
1230003-54
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 55–65
Nonlocal Variational Models for Inpainting and Interpolation
In the case of video, combining the time consistency term with Poisson editingtechniques we are led to minimize the energy
EP (u) =∫
O
|∂vu(x, t)|2 + β|∇xu(x, t) − g(x, t)|2dxdt, (10.4)
where β > 0 and g(x, t) = ∇xu0(x, t) for any (x, t) ∈ O. We assume thatu0 ∈ W 1,2(O). For illumination correction, problem (10.4) is solved with Dirichletboundary conditions
u(x, t) = f(x, t) x ∈ ∂Ot, t ∈ (0, T ), (x, t) ∈ ∂ΩT . (10.5)
The first term of the energy (10.4) represents a regularization along the trajectoriesof the motion field in order to enforce the temporal consistency. The second term in(10.4) is similar to 2D gradient-domain image editing models30 and it is responsiblefor image editing at each frame.
Energy (10.4) is quadratic and its solution is computed by solving the linearproblem
(∂∗v∂v · +β divx∇x·)u = β divx g in O, (10.6)
with boundary conditions (10.5), to which we add the Neumann boundaryconditions
∂vu(x, t) = 0 (x, t) ∈ ∂O ∩ ∂ΩT . (10.7)
Notice that (10.7) holds in O0∪OT . We have denoted by divx the spatial divergenceand ∂∗v is given by ∂∗vf = −∂tf + divx(vf ). Equation (10.6) is of Poisson-type andcan be solved by using the conjugate gradient method.
The energy (10.4) requires some further explanation. Notice that the modelcan be used for several tasks: (a) If we take β = 0 we recover (10.2), the modelfor propagation. In practice, it is better to keep β > 0 although small in orderto add some spatial regularization. In this case, we replace (10.7) by (10.3) andadd Dirichlet boundary conditions in O0 (if required we also replace Neumann byDirichlet in OT ). (b) If we want to correct the illumination we take β > 0 largeenough so that the second term is dominant. In that case, the first term acts as atime regularizer. The boundary conditions are those specified above for illuminationcorrection, namely (10.5) and (10.7). (c) We may use the model advantageously inorder to propagate in time and correct the illumination. For that we first projectin time using β > 0 although small as in (a). Then we use that result to computethe guidance field g(x, t) and solve (10.4) with β > 0 large enough as in (b). Thisalgorithm is a little bit cumbersome and better models are required to transportwhile correcting the illumination. A model for this has recently been proposed inRef. 74. The model is based on the minimization of the energy
EP (u) =∫
O
|∇x∂vu(x, t)|2dxdt
1230003-55
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 56–65
P. Arias et al.
under suitable boundary conditions to ensure propagation and illumination correc-tion. We refer to Ref. 74 for details.
The main difficulties derived from these models are related to their numericalimplementation, namely to the discretization of the convective derivative ∂vu(x, t).
10.2. Discretization of the model
In what follows we work in a discrete setting. The spatial domain is now a squarelattice in Z
2, Ωd = 0, 1, . . . , N2, and the temporal domain is 0, 1, . . . , T; there-fore the video is represented as a stack of T +1 digital images (frames). The spatialgradient is computed using forward differences and is denoted by ∇+
x .Let us consider a given digital video u0 : Ωd × 0, 1, . . . , T → R from
which we compute the optical flow. We define the forward optical flow v+ : Ωd ×0, 1, . . . , T → R
2 between two frames u(·, t) and u(·, t+ 1) as a vector field suchthat (x, t) and (x+ v+(x, t), t+ 1) correspond to the same point in the scene. Simi-larly, the backward optical flow v− relates the points in frame t with those in framet − 1. The optical flow used in our experiments is obtained with the algorithmdescribed in Ref. 20, but any optical flow algorithm with subpixel precision andregularization (or better, with edge preserving regularization) could be used (e.g.see Refs. 1 and 88). If we discretize the convective derivative ∂vu(x, t) using theforward optical flow v+, we define the discrete v+-scheme based on
∂+v u(x, t)
:=
u(x+ v+(x, t), t+ 1) − u(x, t) if t < T,
0 if (x+ v+(x, t), t+ 1) /∈ Ω × 0, . . . , T,(10.8)
where u(x+ v+(x, t), t+1) is the bilinear interpolation of u(·, t+1) at x+ v+(x, t).Notice that we used the continuous domain Ω in (10.8). This is convenient ifwe want to say that the coordinates of x + v+(x, t), which are real-valued, areoutside Ω.
In a similar way, we can also discretize the convective derivative ∂vu(x, t) usingthe backward optical flow v− and define the discrete v−-scheme based on
∂−v u(x, t)
:=
u(x, t) − u(x+ v−(x, t), t − 1) if t > 0,
0 if (x+ v−(x, t), t− 1) /∈ Ω × 0, . . . , T,where u(x+ v−(x, t), t− 1) is the bilinear interpolation of u(·, t− 1) at x+ v−(x, t).The adjoint operators of ∂+
v and ∂−v are denoted by ∂+∗v and ∂−∗
v , respectively.Due to successive interpolations the use of the v−-scheme tends to blur the
results. In order to explain the behavior of the v+-scheme for ∂+v , let us consider
the case of a video with two frames where u(x, 0) is known and we have to computeu(x, 1) in the mask. If we consider Eq. (10.6) with β = 0 involving ∂+∗
v ∂+v acting
1230003-56
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 57–65
Nonlocal Variational Models for Inpainting and Interpolation
on u, we have to compute the values of u(x, 1) on the regular grid x ∈ Ωd usingknowledge of u(·, 1) on the irregular grid obtained by transferring the values ofu(x, 0) by the v+-flow, that is, using u(x+ v+(x, 0), 1) = u(x, 0). This inverse prob-lem sharpens the high frequencies and may generate instabilities. Solving the linearsystem using the conjugate gradient method helps to mitigate these instabilitiesbut it may not be sufficient. For that reason we proposed in Refs. 40, 74 a schemethat alternates the use of the v+- and v−-schemes and we showed experimentallythat it is able to interpolate the information producing sharp and stable results.Let us call it the deblurring scheme for the convective derivative, abbreviated asDSCD.
The DSCD takes advantage of the fact that the v+- and v−-schemes have oppo-site effects on the data (one blurs while the other sharpens) and alternates betweenthem. The v+-scheme permits to recover the frequencies smoothed by the previousstep (v−-scheme). Shortly, if between t = 0 and t = 1 we apply the v+-scheme(implicit), then from frame t = 1 to t = 2 we apply v−-scheme (explicit) and so on.This preserves the transported information for much longer periods of time withoutnoticeable decay. Figure 19 illustrate the behavior of the schemes described.
As a last remark, the optical flow v+ (respectively, v−) is not defined at theoccluded (respectively, disoccluded) areas of a frame. In these cases there may beno correspondence for a pixel in the next (respectively, previous) frame. While someoptical flow algorithms produce occlusion maps, many others do not, so we pre-process the flow to identify the occlusion and disocclusion areas and remove themfrom the energy.74 Thus, we define an occlusion map Occ : Ωd × 0, 1, . . . , T →0, 1 where Occ(x, t) = 1 if |div v(x, t)| is large enough. The discrete energybecomes:
Ep(u) =T∑
t=0
∑x∈Ωd
‖Occ(x, t)∂vu(x, t)‖2 + β∑t∈T
∑x∈Ω
‖∇+x u(x, t) − g(x, t)‖2, (10.9)
0 5 10 15 20 25 30 35 400
5
10
15
20
25
30
35
frame
MS
E
v+–scheme
v−–schemeDSCD
Fig. 19. (Color online) Temporal propagation solving (10.9) with β = 0.01, horizontal motion0.175px/frame, and known g. After 40 frames the results of v−-scheme (explicit scheme) or v+-
scheme (implicit scheme) are more distorted than with DSCD. The plots of the evolution of theerror with respect to the ground truth (right) confirm it.
1230003-57
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 58–65
P. Arias et al.
where ∂v denotes any one of the above discretizations of the convective derivative,i.e. the v+-, the v−-, or the DSCD schemes (see Refs. 40 and 74).
10.3. Experiments
We display some experiments that illustrate (E1) the correction of illuminationin an inpainted domain making it temporally consistent, (E2) the insertion ofan object on a surface while handling situations where the object is partiallyoccluded/disoccluded.
In case (E1) we are given an inpainted video sequence (in this case it has beencomputed by introducing the first frame and propagating it forwards by minimizing(10.9) with β > 0 small). Thus, we can compute the guidance field in all frames. Wewould like to correct this inpainted sequence so that it becomes spatio-temporallyconsistent. In order to achieve it, we use the inpainted video as our input andsolve (10.9) with β = 1 and Occ ≡ 1. Figure 20(a) shows a frame of the originalsequence with a newspaper on a table, the newspaper will be removed and replacedby a different image. Figure 20(b) shows the mask where the new object will beinserted. Figures 20(c)–20(f) show the inpainted sequence which is temporally con-sistent yet not consistent with respect to the illumination changes at each frame.This result is obtained by solving (10.9) with β = 0.01. Figures 20(g)–20(j) showthe result obtained by solving (10.9) with β = 1 and Occ ≡ 1. Notice that theinconsistency which is present in the inpainted sequence has been corrected andthe result is temporally and spatially consistent.
In case (E2), we wish to insert a two-dimensional object into a video sequenceand affix it to a surface. Thus, the optical flow can be computed using the orig-inal sequence and used for incorporating the object. Using (10.9) we are able tocoherently transport information present in one frame into subsequent (or previ-ous) frames in the video. Basically, we start inserting the object into a chosen framethat indicates the first appearance of the object in the video. Let us call this thefirst frame. Then, by minimizing (10.9) with a small value of β (usually 0.01) andsetting the first frame as a Dirichlet temporal boundary condition, we are able totransport the first frame to the others. Since we are affixing an object on a surface,the inserted object inherits the optical flow from this surface. The occlusions arehandled automatically by temporal consistency using the pre-processed optical flowand it suffices to insert the object into the first frame. If we want to handle disoc-clusions, then we need also to insert the object into a later frame in the sequenceand set it as another Dirichlet boundary condition. Let us call this the last frame.We note that in this setting we only need the information in the first and the lastframes. For the intermediate frames we just have a hole that we fill-in with the newinformation.
Figure 21 shows an experiment where we insert a poster on a door replacingan existing one. The area where we insert the poster is being occluded and thendisoccluded by a moving man. The total number of frames of the sequence is 31.
1230003-58
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 59–65
Nonlocal Variational Models for Inpainting and Interpolation
(a) (b)
(c) (d)
(e) (f)
(g) (h)
(i) (j)
Fig. 20. (Color online) Correcting an inpainted domain to impose spatio-temporal consistencyusing Eq. (10.9). (a) Top left: frame of a given video displaying a table with a journal. We wantto replace the journal by a different object. (b) Top right: mask showing the region that we willchange by another one. (c)–(f) Several frames of the video with the result of including a newobject, for that we used (10.9) with β = 0.01. (g)–(j) Result of applying (10.9) to correct theillumination variations (β = 1 and Occ ≡ 1). The guidance gradient field has been computed inimages (c)–(f) and the boundary data is taken from the original video.
1230003-59
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 60–65
P. Arias et al.
(a) (b)
(c) (d)
(e) (f)
(g) (h)
(i) (j)
Fig. 21. (Color online) Object insertion experiment with handling of occlusions and disocclusions.Panels (a)–(d) show four frames of a given video. The white placard will be changed by a differentimage (in our case a fragment of Lena). Notice that the two people occlude the placard. Panels (e),(f) show the mask for images (a) and (d) above, Panels (g)–(j) show the result of changing theplacard by the figure of Lena. The new figure has been inserted in the first frame and transportedby minimizing (10.9) with β = 0.01.
1230003-60
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 61–65
Nonlocal Variational Models for Inpainting and Interpolation
The results shown in Fig. 21 are obtained by solving (10.9). As we can see, ourmethod handles both occlusions and disocclusions.
11. Concluding Remarks
We presented in this work an overview of inpainting problems going from theinpainting of static images to the simultaneous inpainting of stereo pairs or tovideo inpainting. We reviewed some variational models for 2D image inpaintingand we discussed the existence of regular correspondence maps. The problem ofcopying with regular correspondence maps is one of the open questions relative tothe proposed models that do not explicitly include a regularization term. This wouldexplain the observed experimental fact of getting completions that are patchworksof copied regions from the complement of the inpainting domain. We also discussedthe basic ingredients of the problem of stereo image inpainting, and the inpaint-ing of disparity maps, a tool both for stereo inpainting and for the interpolationof complete disparity maps from sparse data. We discussed some issues in videoinpainting, which is a more unexplored area. The main problem is the copying oftextured regions in a time consistent way without propagating an initially inpaintedtexture.
Acknowledgments
We acknowledge partial support by MICINN project, reference MTM2009-08171,and by GRC reference 2009 SGR 773. P.A. is supported by the FPI grant BES-2007-14451 from the Spanish MICINN. V.C. also acknowledges partial support byIP project “2020 3D Media: Spacial Sound and Vision”, financed by EC, and by“ICREA Academia” prize for excellence in research funded both by the Generalitatde Catalunya.
References
1. L. Alvarez, J. Weickert and J. Sanchez, Reliable estimation of dense optical flow fieldswith large displacements, Int. J. Comput. Vis. 39 (2000) 41–56.
2. L. Ambrosio, N. Fusco and D. Pallara, Functions of Bounded Variation and FreeDiscontinuity Problems (Oxford Univ. Press, 2000).
3. F. Andreu, J. M. Mazon, J. D. Rossi and J. Toledo, Nonlocal Diffusion Problems,Mathematical Surveys and Monographs (Amer. Math. Soc., 2010).
4. P. Arias, V. Caselles, G. Facciolo, V. Lazcano and R. Sadek, On the manifold ofpatches of an image, preprint (2012).
5. P. Arias, V. Caselles and G. Sapiro, A variational framework for non-local imageinpainting, in EMMCVPR, Lecture Notes in Computer Science (Springer, 2009),pp. 345–358.
6. P. Arias, G. Facciolo and V. Caselles, Analysis of a variational framework forexemplar-based image inpainting, preprint, University Pompeu Fabra (2011).
7. P. Arias, G. Facciolo, V. Caselles and G. Sapiro, A variational framework for exemplar-based image inpainting, Int. J. Comput. Vis. 93 (2011) 319–347.
1230003-61
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 62–65
P. Arias et al.
8. J. F. Aujol, S. Ladjal and S. Masnou, Exemplar-based inpainting from a variationalpoint of view, SIAM J. Math. Anal. 42 (2010) 1246–1285.
9. S. P. Awate and R. T. Whitaker, Unsupervised, information-theoretic, adaptive imagefiltering for image restoration, IEEE Trans. Pattern Anal. Machine Intell. 28 (2006)364–376.
10. C. Ballester, M. Bertalmıo, V. Caselles, G. Sapiro and J. Verdera, Filling-in by jointinterpolation of vector fields and gray levels, IEEE Trans. Image Process. 10 (2001)1200–1211.
11. C. Barnes, E. Shechtman, A. Finkelstein and D. B. Goldman, PatchMatch: A ran-domized correspondence algorithm for structural image editing, ACM Trans. Graph.28 (2009) 1–11.
12. C. Barnes, E. Shechtman, D. B. Goldman and A. Finkelstein, The generalized Patch-Match correspondence algorithm, in European Conf. Computer Vision (Springer,2010), pp. 29–43.
13. M. Bertalmıo, G. Sapiro, V. Caselles and C. Ballester, Image inpainting, in Proc. 27thAnnual Conf. Computer Graphics and Interactive Techniques (ACM Press, 2000),pp. 417–424.
14. M. Bertalmıo, L. Vese, G. Sapiro and S. Osher, Simultaneous structure and textureinpainting, IEEE Trans. Image Process. 12 (2003) 882–889.
15. D. P. Bertsekas, Nonlinear Programming (Athena Scientific, 1999).16. C. M. Bishop, Pattern Recognition and Machine Learning (Springer, 2006).17. R. Bornard, E. Lecan, L. Laborelli and J. H. Chenot, Missing data correction in still
images and image sequences, in Proc. ACM Int. Conf. Multimedia (ACM Press, 2002),pp. 355–361.
18. F. Bornemann and T. Marz, Fast image inpainting based on coherence transport,J. Math. Imaging Vis. 28 (2007) 259–278.
19. J. Bourgain, H. Brezis and P. Mironescu, Another look at Sobolev spaces, in OptimalControl and Partial Differential Equation: In Honor to Professor Alain Bensoussan’s60th Birthday (IOS Press, 2001), pp. 439–455.
20. T. Brox, A. Bruhn, N. Papenberg and J. Weickert, High accuracy optical flow esti-mation based on a theory for warping, in European Conf. Computer Vision — ECCV(Springer, 2004), pp. 25–36.
21. T. Brox, O. Kleinschmidt and D. Cremers, Efficient nonlocal means for denoising oftextural patterns, IEEE Trans. Image Process. 17 (2008) 1083–1092.
22. A. Buades, B. Coll and J. M. Morel, Neighborhood filters and PDEs, Numer. Math.105 (2006) 1–34.
23. A. Buades, B. Coll and J. M. Morel, A review of image denoising algorithms, with anew one, Multiscale Model. Simulat. 4 (2006) 490–530.
24. A. Buades, B. Coll, J. M. Morel and C. Sbert, Self-similarity driven color demosaick-ing, IEEE Trans. Image Process. 18 (2009) 1192–1202.
25. P. Cannarsa and C. Sinestrari, Semiconcave Functions, Hamilton–Jacobi Equations,and Optimal Control, Progress in Nonlinear Differential Equations and their Appli-cations, Vol. 58 (Birkhauser, 2004).
26. F. Cao, Y. Gousseau, S. Masnou and P. Perez, Geometrically guided exemplar-basedinpainting, SIAM J. Imaging Sci. 4 (2011) 1143–1179.
27. T. Chan, S. H. Kang and J. H. Shen, Euler’s elastica and curvature based inpaintings,SIAM J. Appl. Math. 63 (2002) 564–592.
28. T. Chan and J. H. Shen, Mathematical models for local nontexture inpaintings, SIAMJ. Appl. Math. 62 (2001) 1019–1043.
1230003-62
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 63–65
Nonlocal Variational Models for Inpainting and Interpolation
29. D. Comaniciu and P. Meer, Mean shift: A robust approach toward feature spaceanalysis, IEEE Trans. Pattern Anal. Machine Intell. 24 (2002) 603–619.
30. A. Criminisi, P. Perez and K. Toyama, Region filling and object removal by exemplar-based inpainting, IEEE Trans. Image Process. 13 (2004) 1200–1212.
31. L. Demanet, B. Song and T. Chan, Image inpainting by correspondence maps: Adeterministic approach, Appl. Comput. Math. 1100 (2003) 217–250.
32. A. Desolneux, L. Moisan and J. M. Morel, Edge detection by Helmholtz principle,J. Math. Imaging Vis. 14 (2001) 271–284.
33. J. Digne, M. Dimicoli, N. Sabater and P. Salembier, Neighborhood filters andthe recovery of 3D information, in Handbook of Mathematical Methods in Imaging(Springer, 2010).
34. I. Drori, D. Cohen-Or and H. Yeshurun, Fragment-based image completion, ACMTrans. Graph. 22 (2003) 303–312.
35. A. A. Efros and T. K. Leung, Texture synthesis by non-parametric sampling, in Proc.IEEE Int. Conf. Computer Vision (IEEE Press, 1999), pp. 1033–1038.
36. R. S. Ellis, Entropy, Large Deviations, and Statistical Mechanics (Springer, 2005).37. S. Esedoglu, S. Ruuth and R. Tsai, Threshold dynamics for high order geometric
motions, Interfaces Free Boundaries 10 (2008) 263–282.38. S. Esedoglu and J. H. Shen, Digital image inpainting by the Mumford–Shah–Euler
image model, European J. Appl. Math. 13 (2002) 353–370.39. G. Facciolo, P. Arias, V. Caselles and G. Sapiro, Exemplar-based interpolation of
sparsely sampled images, in Energy Minimization Methods in Computer Vision andPattern Recognition, Lecture Notes in Computer Science (Springer, 2009), pp. 331–344.
40. G. Facciolo, R. Sadek, A. Bugeau and V. Caselles, Temporally consistent gradientdomain video editing, in Energy Minimization Methods in Computer Vision and Pat-tern Recognition, Lecture Notes in Computer Science (Springer, 2011), pp. 59–73.
41. G. Gilboa and S. J. Osher, Nonlocal linear image regularization and supervised seg-mentation, Multiscale Model. Simulat. 6 (2007) 595–630.
42. G. Gilboa and S. Osher, Nonlocal operators with applications to image processing,Multiscale Model. Simulat. 7 (2008) 1005–1028.
43. P. Harrison, Texture tools, Ph.D. thesis, Monash University (2005).44. R. Hartley and A. Zisserman, Multiple View Geometry in Computer Vision (Cam-
bridge Univ. Press, 2000).45. A. Hervieu, N. Papadakis, A. Bugeau, P. Gargallo and V. Caselles, Stereoscopic image
inpainting: Distinct depth maps and images inpainting, in 2010 Int. Conf. PatternRecognition (IEEE Press, 2010), pp. 4101–4104.
46. A. Hervieu, N. Papadakis, A. Bugeau, P. Gargallo and V. Caselles, Stereoscopic imageinpainting using scene geometry, in Int. Conf. Multimedia and Expo (ICME) (IEEEPress, 2011), pp. 1–6.
47. B. Huhle, T. Schairer, P. Jenke and W. Straßer, Fusion of range and color images fordenoising and resolution enhancement with a non-local filter, Computer Vis. Imageand Understanding 114 (2010) 1336–1345.
48. E. T. Jaynes, Information theory and statistical mechanics, Phys. Rev. 106 (1957)620–630.
49. J. Jia, Y. W. Tai, T. P. Wu and C. K. Tang, Video repairing under variable illumi-nation using cyclic motions, IEEE Trans. Pattern Anal. Machine Intell. 28 (2006)832–839.
1230003-63
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 64–65
P. Arias et al.
50. J. Jia, W. Tai-Pang, Y. W. Tai and C. K. Tang, Video repairing: Inference of fore-ground and background under severe occlusion, in Proc. IEEE Computer Society Conf.Computer Vision and Pattern Recognition (IEEE Press, 2004), pp. 364–371.
51. J. Jia and C.-K. Tang, Inference of segmented color and texture description by tensorvoting, IEEE Trans. Pattern Anal. Machine Intell. 26 (2004) 771–786.
52. N. Kawai, T. Sato and N. Yokoya, Image inpainting considering brightness changeand spatial locality of textures and its evaluation, in Adv. in Image and Video Tech.(Springer, 2009), pp. 271–282.
53. S. Kindermann, S. Osher and P. W. Jones, Deblurring and denoising of images bynonlocal functionals, Multiscale Model. Simulat. 4 (2006) 1091–1115.
54. V. Kolmogorov and R. Zabih, Multi-camera scene reconstruction via graph cuts, Euro-pean Conf. Computer Vision (Springer, 2002), pp. 8–40.
55. N. Komodakis and G. Tziritas, Image completion using efficient belief propagationvia priority scheduling and dynamic pruning, IEEE Trans. Image Process. 16 (2007)2649–2661.
56. V. Kwatra, I. Essa, A. Bobick and N. Kwatra, Texture optimization for example-basedsynthesis, ACM Trans. Graph. 24 (2005) 795–802.
57. V. Lazcano, P. Arias and V. Caselles, A gradient based neighborhood filter for depthinterpolation, preprint (2011).
58. A. Levin, A. Zomet and Y. Weiss, Learning how to inpaint from global image statistics,in Proc. IEEE Int. Conf. Computer Vision (IEEE Press, 2003), pp. 305–312.
59. Y. Liu and V. Caselles, A model for stereoscopic image inpainting, preprint (2012).60. S. Masnou, Disocclusion: A variational approach using level lines, IEEE Trans. Image
Process. 11 (2002) 68–76.61. S. Masnou and J.-M. Morel, Level lines based disocclusion, in Proc. IEEE Int. Conf.
Image Processing, Vol. 3 (IEEE Press, 1998), pp. 259–263.62. J. M. Morel and S. Solimini, Variational Methods in Image Segmentation: With Seven
Image Processing Experiments (Birkhauser, 1995).63. M. Nitzberg, D. Mumford and T. Shiota, Filtering, Segmentation, and Depth, Lecture
Notes in Computer Science (Springer, 1993).64. K. A. Patwardhan, G. Sapiro and M. Bertalmio, Video inpainting of occluding and
occluded objects, in Proc. IEEE Int. Conf. Image Processing, Vol. 2 (IEEE Press,2005), pp. 69–72.
65. P. Perez, M. Gangnet and A. Blake, Poisson image editing, ACM Trans. Graph. 22(2003) 313–318.
66. P. Perez, M. Gangnet and A. Blake, PatchWorks: Example-based region tiling forimage editing, Technical Report, Microsoft Research (2004).
67. P. Perona and J. Malik, Scale-space and edge detection using anisotropic diffusion,IEEE Trans. Pattern Anal. Machine Intell. 12 (1990) 629–639.
68. G. Peyre, Manifold models for signals and images, Computer Vis. Image and Under-standing 113 (2009) 249–260.
69. G. Peyre, S. Bougleux and L. Cohen, Non-local regularization of inverse problems, inEuropean Conf. Computer Vision (Springer, 2008), pp. 57–68.
70. L. Pizarro, P. Mrazek, S. Didas, S. Grewenig and J. Weickert, Generalised nonlocalimage smoothing, Int. J. Comput. Vis. 90 (2010) 62–87.
71. A. C. Ponce, A new approach to Sobolev spaces and connections to-convergence, Calc.Var. Partial Differential Equations 19 (2004) 229–255.
72. M. Protter, M. Elad, H. Takeda and P. Milanfar, Generalizing the non-local meansto super-resolution reconstruction, IEEE Trans. Image Process. 18 (2009) 36–51.
1230003-64
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.
2nd Reading
July 19, 2012 13:0 WSPC/103-M3AS 1230003 65–65
Nonlocal Variational Models for Inpainting and Interpolation
73. K. Rose, Deterministic annealing for clustering, compression, classification, regression,and related optimization problems, Proc. IEEE 86 (1998) 2210–2239.
74. R. Sadek, P. Arias, G. Facciolo and V. Caselles, Gradient based models for videoediting, preprint (2012).
75. S. M. Smith and J. M. Brady, SUSAN — A new approach to low level image process-ing, Int. J. Comput. Vis. 23 (1997) 45–78.
76. S. M. Srivastava, A Course on Borel Sets (Springer, 1998).77. J. Sun, L. Yuan, J. Jia and H. Y. Shum, Image completion with structure propagation,
ACM Trans. Graph. 24 (2005) 861–868.78. C. Tomasi and R. Manduchi, Bilateral filtering for gray and color images, in Sixth
Int. Conf. Computer Vision (IEEE Press, 1998), pp. 839–846.79. D. Tschumperle, Fast anisotropic smoothing of multi-valued images using curvature-
preserving PDE’s, Int. J. Comput. Vis. 68 (2006) 65–82.80. H. Wang, X. Xu, R. Raskar and N. Ahuja, Videoshop: A new framework for spatio-
temporal video editing in gradient domain, Graph. Models 69 (2007) 55–70.81. L. Wang, H. Jin, R. Yang and M. Gong, Stereoscopic inpainting: Joint color and
depth completion from stereo images, in IEEE Conf. Computer Vision and PatternRecognition (IEEE Press, 2008), pp. 1–8.
82. L. Y. Wei and M. Levoy, Fast texture synthesis using tree-structured vector quan-tization, in Proc. 27th Annual Conf. Computer Graphics and Interactive Techniques(ACM Press, 2000), pp. 479–488.
83. Y. Wexler, E. Shechtman and M. Irani, Space-time completion of video, IEEE Trans.Pattern Anal. Machine Intell. 29 (2007) 463–476.
84. C. F. J. Wu, On the convergence properties of the EM algorithm, Ann. Statist.11 (1983) 95–103.
85. Q. Yang, R. Yang, J. Davis and D. Nister, Spatial-depth super-resolution for rangeimages, in IEEE Conf. Computer Vision and Pattern Recognition (IEEE Press, 2007),pp. 1–8.
86. L. P. Yaroslavsky, Digital Picture Processing. An Introduction (Springer, 1985).87. K. J. Yoon and I. S. Kweon, Adaptive support-weight approach for correspondence
search, IEEE Trans. Pattern Anal. Machine Intell. 28 (2006) 650–656.88. C. Zach, T. Pock and H. Bischof, A duality based approach for realtime tv-l 1 optical
flow, in Pattern Recognition, Lecture Notes in Computer Science (Springer, 2007),pp. 214–223.
1230003-65
Mat
h. M
odel
s M
etho
ds A
ppl.
Sci.
2012
.22.
Dow
nloa
ded
from
ww
w.w
orld
scie
ntif
ic.c
omby
UN
IVE
RSI
TY
OF
RE
AD
ING
on
12/2
1/14
. For
per
sona
l use
onl
y.