tensorfactorizationwithtotalvariationandtikhonovregulariza ... · exhibit spatial/temporal...

Journal of Mathematical Imaging and Visionhttps://doi.org/10.1007/s10851-019-00933-9

Tensor Factorization with Total Variation and Tikhonov Regularizationfor Low-Rank Tensor Completion in Imaging Data

Xue-Lei Lin1 ·Michael K. Ng2 · Xi-Le Zhao3

Received: 25 April 2019 / Accepted: 26 November 2019© Springer Science+Business Media, LLC, part of Springer Nature 2019

AbstractThe main aim of this paper is to study tensor factorization for low-rank tensor completion in imaging data. Due to theunderlying redundancy of real-world imaging data, the low-tubal-rank tensor factorization (the tensor–tensor product of twofactor tensors) can be used to approximate such tensor very well. Motivated by the spatial/temporal smoothness of factortensors in real-world imaging data, we propose to incorporate a hybrid regularization combining total variation and Tikhonovregularization into low-tubal-rank tensor factorization model for low-rank tensor completion problem. We also develop anefficient proximal alternating minimization (PAM) algorithm to tackle the corresponding minimization problem and establisha global convergence of the PAM algorithm. Numerical results on color images, color videos, and multispectral images arereported to illustrate the superiority of the proposed method over competing methods.

Keywords Tensor factorization · Hybrid regularization · Tensor completion · Proximal alternating minimization

Mathematics Subject Classification 90C26 · 90C30 · 90C90 · 65F22

1 Introduction

As a high-dimensional extension of the matrix, the tensorcan express more complicated intrinsic structures of data,which plays an important role in many real-world applica-tions; see, e.g., [22,31,37,51]. Low-rank tensor completion(LRTC) is one of the most important problems in tensor

The research is supported by research grants HKRGC GRF(12306616, 12200317, 12300519, 12300218), HKU Grant(104005583), and NSFC (61876203, 61772003, 11801479).

B Xi-Le [email protected]

Xue-Lei [email protected]

Michael K. [email protected]

1 Department of Mathematics, Hong Kong Baptist University,Kowloon Tong, Hong Kong

2 Department of Mathematics, The University of Hong Kong,Pokfulam, Hong Kong

3 School of Mathematical Sciences, University of ElectronicScience and Technology of China, Chengdu 611731, Sichuan,People’s Republic of China

processing, which aims at filling in the missing entries of apartially observed low-rank tensor. Indeed, high-dimensionaldata, such as videos and images, usually have a low-rankstructure or approximately so, based on which many workson low-rank tensor completion have been done; see, e.g.,[32,36,42,48].

The fundamental issue of LRTC is the definition of therank of the tensor. However, unlike matrix rank, the defini-tion of the rank of the tensor is not unique. Many researchefforts have been devoted to designing the rank of tensor. TheCANDECOMP/PARAFAC (CP) rank [10,21,50] is definedbased on the CP decomposition, where an nth-order tensoris decomposed as the sum of rank-one tensors, i.e., the outerproduct of n vectors. The CP rank is defined as the minimalnumber of the rank-one tensors required to express the targettensor. However, the corresponding LRTC problems are NP-hard. The Tucker rank is based on the Tucker decompositionwhich decomposes a tensor into a core tensor multiplied bya matrix along each mode [17,26,27,39]. The Tucker rankis defined as a vector, the kth element of which is the rankof the mode-k unfolding matrix. The corresponding LRTCproblems are alsoNP-hard. Liu et al. suggested the sumof thenuclear norm (SNN) of the mode-k unfolding matrix as theconvex surrogate [23,30,31]. However, unfolding the tensor

123

Journal of Mathematical Imaging and Vision

Fig. 1 Low-rank tensorfactorization (tensor–tensorproduct of two factor tensors)

along each mode will inevitably destroy the intrinsic struc-tures of the tensor [25,40].

Recently, the novel tensor–tensor product (t-product) oftwo tensors B ∈ R

n1×r×n3 and C ∈ Rr×n2×n3 is defined

to be a tensor X := B ∗ C ∈ Rn1×n2×n3 [25,46], where the

mode-3fibers ofX are defined asX (i, j, :) =∑rk=1 B(i, k, :

)�C(k, j, :). Here, “�” denotes the circular convolution oftwo vectors. Based on t-product, the tensor singular valuedecomposition (t-SVD), which generalizes the matrix sin-gular value decomposition (SVD), is proposed in [25]. Forany third-order tensor X ∈ R

n1×n2×n3 , its t-SVD is definedas

X = U ∗ S ∗ V,

where U ∈ Rn1×n1×n3 and V ∈ R

n2×n2×n3 are third-orderorthogonal tensors and S ∈ R

n1×n2×n3 is f -diagonal tensorwhose frontal slices are diagonal matrices. Based on t-SVD,the corresponding tensor tubal rank has received an increas-ing attention [32,35,45,46,49,51].

The tubal rank of X is defined as the number of nonzerotubes of S [32,51], i.e.,

rankt(X ) := #{i : S(i, i, :) �= 0}.

The nonzero tubes of S in t-SVD are analogous to nonzerodiagonal elements of the diagonal matrix in matrix SVD. Asthe natural generalization of the matrix SVD, the t-SVD alsohas the optimal tubal rank r approximation property [44].Due to the combinational nature of the tensor tubal rankminimization problems, Semerci et al. [35] suggested thetensor nuclear norm (TNN) as the convex surrogate. Then,Zhang et al. [46] proposed the TNN-based LRTC model,and subsequently, Lu et al. [32] proposed the TNN-basedtensor robust principal component analysis model. In orderto avoid time-consuming t-SVD, Zhou et al. [51] proposedthe low-tubal-rank tensor factorization model for the LRTCproblem, which factorizes the target tensor into the t-productof two factor tensors. Interestingly, many real-world imagesexhibit spatial/temporal smoothness. We observed that thesmoothness of the real-world imaging data is inherited byfactor tensors. For example in Fig. 1, a color video can

be represented as a third-order tensor, whose factor tensorsinherit the spatial and temporal smoothness along the firstmode and the second mode, respectively. Motivated by theclear physical interpretations of factor tensors, we suggestthe total variation- and Tikhonov-regularized tensor factor-ization model for the LRTC problem. Due to its ability topreserve edges, total variation (TV) regularization, firstlyproposed in [34], is introduced to promote the smoothnessof factor tensors. Moreover, Tikhonov regularization is intro-duced to guarantee the stability of the solution [19]. The TVregularization and Tikhonov regularization are complemen-tary to each other, rather than isolated and uncorrelated. Ahybrid regularization combining TV and Tikhonov regular-ization contributes to an excellent completion performance.

The rest of this paper is organized as follows: In Sect. 2, anew low-rank tensor completion model is introduced and theproximal alternatingminimization (PAM) algorithm is devel-oped for solving the model. In Sect. 3, a global convergenceis established for the PAM algorithm. In Sect. 4, some effi-cient solvers and the fast implementations are introduced forthe subproblems arising from the PAM algorithm. In Sect. 5,numerical results are reported to show the performance ofthe proposed method. Finally, some concluding remarks aregiven in Sect. 6.

2 The ProposedModel and the PAMAlgorithm

Before presenting the model, we firstly introduce some nota-tions which will be frequently used later. For a third-ordertensorA, we use theMATLABnotations, such asA(:),A(i, :, :), A(:, i, :), A(:, :, i), and also define ||A||�2 := ||A(:)||2,||A||�1 := ||A(:)||1.

2.1 The Tensor CompletionModel by TensorFactorization with TV–Tikhonov Regularizations

The proposed tensor completion model by tensor factor-ization with TV–Tikhonov (TCTF-TVT) regularization isdescribed as follows:

123


argmin0≤X≤1,B,C

1

2‖X − B ∗ C‖2�2

+ α

m1∑

k=1

||D1k ∗ B||�1 + α

m2∑

k=1

||C ∗ D2k ||�1

+ μ

2||B||2�2 + μ

2||C||2�2 ,

s.t . P�(X ) = P�(M), (2.1)

where X ∈ Rn1×n2×n3 is the tensor to be filled; B ∈

Rn1×r×n3 , C ∈ R

r×n2×n3 ; � is an index set; M is a giventensor; P�(·) is a projection operator preserving entries withindices in � while restricting entries with indices out of �

to be zeros; M is a given tensor such that 0 ≤ M ≤ 1;D1k ∈ R

n1×n1×n3 (k = 1, 2, . . . , m1) and D2k ∈ Rn2×n2×n3

(k = 1, 2, . . . , m2) are some tensor representations ofbackward partial difference operators with homogeneousNeumann boundary conditions, whose specific form will begiven in Sect. 4; r is a given small positive integer such thatr ≤ min{n1, n2}; α and μ are given positive numbers.

Instead of imposing TV and Tikhonov regularizers on theoriginal tensorX , we consider TV and Tikhonov regularizersof the factor tensors B and C in (2.1) for two reasons. First,the smoothness of the original tensor can be inherited byfactor tensors. Second, because the sizes of factor tensorsare smaller than the size of the original tensor, imposing TVand Tikhonov regularizers on the factor tensors instead ofthe original tensor saves the computational cost and storage.Moreover, in this paper, we mainly consider completion ofthe color image, color video, and multispectral image, pixelvalues of which always belong to the interval [0, 1] up to ascaling constant. This is why we set a constraint 0 ≤ X ≤ 1in model (2.1).

For ease of statement, we rewrite (2.1) as the followingequivalent unconstrained form

argminX ,B,C

F(X ,B, C), (2.2)

where

F(X ,B, C) = 1

2‖X − B ∗ C‖2�2 + δS(X ) + α

m1∑

k=1

||D1k ∗ B||�1

+ α

m2∑

k=1

||C ∗ D2k ||�1 + μ

2||B||2�2 + μ

2||C||2�2

S = {W ∈ Rn1×n2×n3 |W(i, j, k) = M(i, j, k) for (i, j, k) ∈ �,

W(i, j, k) ∈ [0, 1] for (i, j, k) /∈ �},

δS(X ) :={0, X ∈ S,

+∞, X /∈ S,.

It is clear that F is coercive with respect to B and C, i.e.,

limB→∞ inf

X ,CF(X ,B, C) = +∞, lim

C→∞ infX ,B

F(X ,B, C) = +∞.

By such coercivity and continuity of F on S × Rn1×r×n3 ×

Rr×n2×n3 , existence of (2.2) can be easily proven.

2.2 The PAMAlgorithm for Solving (2.2)

Note that (2.2) is a multivariate optimization problem. Alter-nating minimization (AM) algorithm is commonly used tosolve multivariate optimization problems due to its simplic-ity and efficiency; see, e.g., [3,5,9,33,38]. To enhance thetheoretical convergence and numerical stability of AM algo-rithm, proximal terms are suggested to add in subproblemsarising from AM algorithm, which is called PAM algorithm;see, e.g., [1,2,4]. In this subsection, a PAMalgorithm is devel-oped for solving (2.2).

Given an initial guess (X k,Bk, Ck) for problem (2.2), thenthe PAM iteration is defined as follows:

X k+1 = argminX

F(X ,Bk, Ck) + ρ

2||X − X k ||2�2 , (2.3)

Bk+1 = argminB

F(X k+1,B, Ck) + ρ

2||B − Bk ||2�2 , (2.4)

Ck+1 = argminC

F(X k+1,Bk+1, C) + ρ

2||C − Ck ||2�2 ,

(2.5)

where ρ > 0 is a given primal parameter.It is easy to see that (2.3)–(2.5) are all strongly convex

optimization problems, whose existence and uniqueness areguaranteed. In Sect. 4, efficient algorithmswill be introducedto solve (2.3)–(2.5).

3 Global Convergence Analysis of PAMIteration

In this section, we will prove the global convergence of PAMalgorithm.

To formalize the discussion, we express the tensor-formvariables X , B and C as vectors in what follows.

For a positive integer n, denote �(n) := {1, 2, . . . , n}.For positive integers, l1, l2, l3, define bijections V[l1,l2,l3] :A ∈ R

l1×l2×l3 → V[l1,l2,l3](A) ∈ Rl1l2l3×1 and I[l1,l2,l3] :

(i, j, k) ∈ �(l1) × �(l2) × �(l3) → I[l1,l2,l3](i, j, k) ∈�(l1l2l3) by

V[l1,l2,l3](A) := A(:),A(i, j, k) = [A(:)](I[l1,l2,l3](i, j, k)) holding for each A ∈ R

l1×l2×l3 .

From the above definition, it is easy to see that I[l1,l2,l3]maps an index (i, j, k) of an entry in A ∈ R

l1×l2×l3 intothe index of the same entry in A(:). Note that V[l1,l2,l3] is abi-continuous linear bijection.

123


In the following,wedenoteV[n1,n2,n3],V[n1,r ,n3],V[r ,n2,n3],I[n1,n2,n3], I[n1,r ,n3], I[r ,n2,n3] byV1,V2,V3, I1, I2, I3, respec-tively. Denote M1 = n1n2n3, M2 = n1rn3, M3 = rn2n3,M = M1+ M2+ M3. Then, (2.2) can be equivalently rewrit-ten as the following vector form

argminv∈RM

F(v), (3.1)

where v = (v1; v2; v3), vi ∈ RMi (i = 1, 2, 3),

F(v) := G(v) + δS(v1) + g2(v2) + g3(v3),

G(v) = 1

2||V−1

1 (v1) − V−12 (v2) ∗ V

−13 (v3)||2�2

+ μ

2

(||v2||22 + ||v3||22

),

g2(v2) := α

m1∑

k=1

||D1k(v2)||�1,

g3(v3) := α

m2∑

k=1

||D2k(v3)||�1,

D1k(v2) := D1k ∗ [V−12 (v2)],

D2k(v3) := V−13 (v3) ∗ D2k,

S := {w ∈ RM1 |w(i) = [V1(M)](i) for i ∈ I1(�),

w(i) ∈ [0, 1] for i /∈ I1(�)},

δS(v1) :={0, v1 ∈ S,

+∞, v1 /∈ S,.

It is clear that v1, v2 and v3 in (3.1) are counterparts ofX , B and C in (2.2), respectively. Due to the bilinearity of“∗” operation, it is clear that D1k : Rn1rn3×1 → R

n1×r×n3

(k = 1, 2, . . . , m1) and D2k : Rrn2n3×1 → R

r×n2×n3

(k = 1, 2, . . . , m2) are both linear operators. Moreover, itis easy to see that S is a non-empty closed set, which meansδS(·) is a proper lower semi-continuous (PLSC) function onR

M1 and F(·) is a PLSC function on RM .

(2.3)–(2.5) is then equivalent to

vk+11 ∈ argmin

v1

F(v1; vk2; vk

3) + ρ

2||v1 − vk

1 ||22, (3.2)

vk+12 ∈ argmin

v2

F(vk+11 ; v2; vk

3) + ρ

2||v2 − vk

2 ||22, (3.3)

vk+13 ∈ argmin

v3

F(vk+11 ; vk+1

2 ; v3) + ρ

2||v3 − vk

3 ||22, (3.4)

provided that (vk1; vk

2; vk3) is an initial guess for (3.1).

The aim of this section is to establish a global conver-gence for the iteration process (3.2)–(3.4) or equivalentlyfor the PAM iteration (2.3)–(2.5). We firstly introduce somenotations and preliminaries.

For a real-extended-valued function f : Rn → R∪{+∞},define

dom( f ) := {x ∈ Rn| f (x) < +∞},

∂ f (x) :=⎧⎨

⎩

{v ∈ R

n∣∣∣ lim inf

y→x

f (y)− f (x)−〈v,y−x〉||x−y||2 ≥ 0

}, x ∈ dom( f ),

∅, x /∈ dom( f ),,

∂ f (x) :={

{v ∈ Rn|∃{xk}+∞

k=1 s.t . xk → x, f (xk) → f (x), vk ∈ ∂ f (xk) → v}, x ∈ dom( f ),

∅, x /∈ dom( f ),,

dom(∂ f ) := {x ∈ Rn|∂ f (x) �= ∅}.

∂ f (x) is called sub-differential of f at x . Here, 〈·, ·〉 denotesthe Euclidean inner product in Rn .

Proposition 1 [1] A necessary condition for x to be a mini-mizer of a PLSC function f is that

0 ∈ ∂ f (x). (3.5)

A point satisfying (3.5) is called limiting critical or simplycritical.

For a PLSC function f : Rn → R ∪ {+∞}, and −∞ <

η1 < η2 ≤ +∞, we denote

[η1 < f < η2] := {x ∈ Rn|η1 < f (x) < η2}.

Definition 1 (Kurdyka–Łojasiewicz property, [1]) A PLSCfunction f : Rn → R ∪ {+∞} is said to have the Kurdyka–Łojasiewicz property at x ∈ dom(∂ f ) if there exists η ∈(0,+∞], a neighborhood U of x and a continuous concavefunction φ : [0, η) → [0,+∞) such that:

123


− φ(0) = 0,

− φ is C1 on (0, η),

− φ′ is positive on (0, η),

− for each x ∈ U ∩ [ f (x) < f < f (x) + η], the Kurdyka − −Ł ojasiewicz inequality hold :φ′( f (x) − f (x))dist(0, ∂ f (x)) ≥ 1.

In the above definition, the norm involved in dist(·, ·) is || · ||2and the convention dist(0,∅) := +∞ is used.

A PLSC function f satisfying Kurdyka–Łojasiewiczproperty at each point of dom(∂ f ) is called a KL function.

We will make use of the convergence theory built in [2]stated in the following to analyze the convergence of PAMiteration.

Lemma 1 ([2])Let f : Rn → R∪{+∞} be a PLSC function.Let {xk}k∈N ⊂ R

n be a sequence such that

H1. (Sufficient decrease condition). For each k ∈ N,f (xk+1)+a||xk+1−xk ||22 ≤ f (xk) holds for a constanta ∈ (0,+∞);

H2. (Relative error condition). For each k ∈ N, ∃wk+1 ∈∂ f (xk+1) such that ||wk+1||2 ≤ b||xk+1 − xk ||2 holdsfor a constant b ∈ (0,+∞);

H3. (Continuity condition). There exists a subsequence{xk j } j∈N and x ∈ R

n such that

xk j → x and f (xk j ) → f (x), as j → ∞.

If f has the Kurdyka–Łojasiewicz property at x appearingin H3, then

(i) xk → x ;(ii) x is a critical point of f , i.e., 0 ∈ ∂ f (x);(iii) the sequence {xk}k ∈ N has a finite length, i.e.,

+∞∑

k=0

||xk+1 − xk ||2 < +∞.

The rest of this section is devoted to proving that theobjective function F in (3.1) and the iterative sequence(vk

1; vk2; vk

3)k∈N generated by PAM iteration (3.2)–(3.4) sat-isfy the assumptions imposed in Lemma 1, by which weestablish the convergence of PAM iteration.

Definition 2 [6]

(a) A subset A ⊂ Rn is called a semi-algebraic set if and

only if there exists a finite number of real polynomialfunctions Pi j , Qi j : Rn → R such that

A =p⋃

j=1

q⋂

i=1

{x ∈ Rn|Pi j (x) = 0, Qi j (x) < 0}.

(b) Let A1 ⊂ Rn and A2 ⊂ R

m be two subsets. A functionf : A1 → A2 is called semi-algebraic if and only if itsgraph {(x, f (x)) ∈ R

n+m |x ∈ A1} is a semi-algebraicset in Rn+m .

We call a function f is semi-algebraic on A if and only if{(x, f (x))|x ∈ A} is a semi-algebraic set.

Lemma 2 ([7,8,28])A semi-algebraic real-valued function fis a KL function, i.e., f satisfies Kurdyka–Łojasiewicz prop-erty at each x ∈ dom( f ).

Proposition 2 ([2,6,14,41])

(i) A subset of R is semi-algebraic if and only if it isfinite union of points and open intervals (bounded orunbounded).

(ii) Complements, finite unions, finite intersections, finiteCartesian products of semi-algebraic sets are semi-algebraic.

(iii) Finite sums and finite products of semi-algebraic func-tions are semi-algebraic.

(iv) The composition g ◦ f of semi-algebraic mappings f :A1 → A2 and g : A2 → A3 is semi-algebraic.

(v) If f : A1 → A2 is semi-algebraic and A3 ⊂ A1 is asemi-algebraic subset, then f (A3) is a semi-algebraicset.

(vi) Polynomials defined on Rn are semi-algebraic func-

tions.(vii) The absolute value function f (x) := |x | is semi-

algebraic on R.

Proposition 3 If f : A1 → A2 is semi-algebraic and A3 ⊂A1 is a semi-algebraic subset, then f is semi-algebraic onA3.

Proof By Proposition 2 (ii), (v), A3 × f (A3) is semi-algebraic. Notice that

{(x, f (x))|x ∈ A3} = {(x, f (x))|x ∈ A1} ∩ (A3 × f (A3)),

which togetherwith Proposition 2 (ii) proves the proposition.��

Denote

S = S × RM2 × R

M3 .

123


Lemma 3 F is semi-algebraic on S. Hence, F satisfies theKurdyka–Łojasiewicz property at each v ∈ S.

Proof On S, F can be expressed as

F(v) := G(v) + g2(v2) + g3(v3), v ∈ S.

By Proposition 2 (i), (ii), it is clear that S is a semi-algebraicset.

Notice thatV−1i (i = 1, 2, 3) are linear mappings between

finite-dimensional spaces. Hence, each element of V−1i (vi )

(i = 1, 2, 3) is actually a linear polynomial of v :=(v1; v2; v3). Additionally, by the definition of || · ||�2 , weknow that G(v) is a polynomial of v. Thus, by Proposition 2(vi) and Proposition 3, G is semi-algebraic on S.

Again, D1k (k = 1, 2, . . . , m1) and D2k (k = 1, 2, . . . , m2)are linear mappings between finite-dimensional spaces.Hence, each element of D1k(v2)’s and D2k(v3) can beregarded as a linear polynomial of v := (v1; v2; v3) for eachk. Therefore, g2(v2) and g3(v3) are simply finite sums ofcompositions of absolute value function and linear polyno-mial of v, which is a semi-algebraic function onRM and thusis semi-algebraic on S by Propositions 2, 3.

Then, Proposition 2 (iii) implies that F is semi-algebraicon S. The proof is complete. ��

For two subsets A1, A2 and an element x of the samevector space, denote

A1 + A2 := {x1 + x2|x1 ∈ A1, x2 ∈ A2},x + A1 = A1 + x := {x + y|y ∈ A1}.

Theorem 4 Let vk = (vk1; vk

2; vk3) be the sequence generated

by the PAM iteration (3.2)–(3.4) with an arbitrary initialguess v0 ∈ S. Then, there exists v ∈ S such that

(i) vk → v;(ii) 0 ∈ ∂ F(v);(iii) {vk}k∈N has a finite length, i.e.,

+∞∑

k=0

||vk+1 − vk ||2 < +∞.

Proof It has beenmentioned above that F is a PLSC functionon R

M . From (3.2)–(3.4), we see that

F(vk+11 ; vk

2; vk3

)+ ρ

2||vk+1

1 − vk1 ||22

≤ F(vk1; vk

2; vk3

), k ∈ N,

F(vk+11 ; vk+1

2 ; vk3

)+ ρ

2||vk+1

2 − vk2 ||22

≤ F(vk+11 ; vk

2; vk3

), k ∈ N,

F(vk+11 ; vk+1

2 ; vk+13

)+ ρ

2||vk+1

3 − vk3 ||22

≤ F(vk+11 ; vk+1

2 ; vk3

), k ∈ N.

Summing over the three inequalities above, we obtain

F(vk+1) + ρ

2||vk+1 − vk ||22 ≤ F(vk), k ∈ N. (3.6)

Hence, H1 (sufficient decrease condition) is satisfied witha = ρ

2 .Recall that F contains the Tikhonov regularization terms

μ2 (||v2||22 + ||v3||22), which together with (3.6) implies that

supk∈N

μ

2(||vk

2 ||22 + ||vk3 ||22) ≤ F(v0) < +∞.

Moreover, from the definition of the indicator function δS(·)and (3.2), we clearly see that {vk

1 |k ∈ N} ⊂ S. Since S is abounded set, supk∈N ||vk

1 ||2 < +∞. Thus, we conclude thatthe sequence {vk}k∈N is bounded, i.e.,

supk∈N

||vk ||2 < +∞.

Let ∂vi and ∇vi denote sub-differential and gradient withrespect to variable vi , respectively, for i = 1, 2, 3. It has beenmentioned in the proof of Lemma 3 that G is a polynomialfunction and thus is infinitely differentiable. Then, (3.2) andProposition 1 imply that

0 ∈ ∂v1

[F(·; vk

2; vk3) + ρ

2|| · −vk

1 ||22]∣∣∣v1=vk+1

1

= ∂v1 F(vk+11 ; vk

2; vk3) + ρ(vk+1

1 − vk1)

= ∇v1G(vk+11 ; vk

2; vk3) + ∂δS(vk+1

1 ) + ρ(vk+11 − vk

1), k ∈ N

(3.7)

Similarly, (3.3)–(3.4) and Proposition 1 imply that

0 ∈ ∇v2G(vk+11 ; vk+1

2 ; vk3) + ∂g2(v

k+12 ) + ρ(vk+1

2 − vk2), k ∈ N,

(3.8)

0 ∈ ∇v3G(vk+11 ; vk+1

2 ; vk+13 ) + ∂g3(v

k+13 ) + ρ(vk+1

3 − vk3), k ∈ N.

(3.9)

(3.7)–(3.9) imply that there exists wk+11 ∈ ∂δS(v

k+11 ),

wk+1i ∈ ∂gi (v

k+1i ) (i = 2, 3) such that

−wk+1 = uk + ρ(vk+1 − vk), k ∈ N,

where

wk+1 = (wk+11 ;wk+1

2 ;wk+13 ),

uk = (∇v1G(vk+11 ; vk

2; vk3); ∇v2G(vk+1

1 ; vk+12 ; vk

3);∇v3G(vk+1

1 ; vk+12 ; vk+1

3 )).

123


Denote wk+1 = wk+1 + ∇G(vk+1). It is clear that wk+1 ∈∂ F(vk+1) and

−wk+1 = uk − ∇G(vk+1) + ρ(vk+1 − vk), k ∈ N,

(3.10)

Denote E = {vk |k ∈ N}. For v = (v1; v2; v3) ∈ RM with

vi ∈ RMi (i = 1, 2, 3), define the coordinate projections by

�i (v) := vi , i = 1, 2, 3.

Denote E = �1(E)×�2(E)×�3(E). Since E is bounded,it is trivial to see that E ⊂ R

M is also bounded. Since Gis a polynomial, it is easy to prove that ∇G is Lipschitz-continuous on any bounded subset ofRM . Hence, there existsa constant c > 0 such that

||∇G(v) − ∇G(w)||2 ≤ c||v − w||2, v, w ∈ E .

Therefore,

||uk − ∇G(vk+1)||2 =[||∇v1G(vk+11 ; vk

2; vk3) − ∇v1G(vk+1)||22

+ ||∇v2G(vk+11 ; vk+1

2 ; vk3)

− ∇v2G(vk+1)||22]12

≤ [c2(||vk2 − vk+1

2 ||22 + ||vk3 − vk+1

3 ||22)+ c2||vk

3 − vk+13 ||22]

12

≤√2c||vk+1 − vk ||2, k ∈ N,

which together with (3.10) implies that

||wk+1||2 ≤ ||uk − ∇G(vk+1)||2 + ρ||vk+1 − vk ||2≤ (

√2c + ρ)||vk+1 − vk ||2, k ∈ N.

Hence, H2 (Relative error condition) is satisfied with b =√2c + ρ.Moreover, since {vk}k∈N ⊂ R

M is bounded and thusrelative compact, there exists a subsequence {vk j } j∈N andv ∈ R

M such that vk j → v as j → +∞. As mentionedabove {vk

1 |k ∈ N} ⊂ S, it holds that {vk}k∈N ⊂ S. SinceS is closed, v ∈ S. Additionally, it is easy to see that F iscontinuous on S. Therefore, F(vk j ) → F(v) as j → +∞.Hence, H3 (Continuity condition) is fulfilled.

By Lemma 3, F is semi-algebraic on S. Hence, byLemma 2, F satisfies the Kurdyka–Łojasiewicz propertyat v ∈ S. Then, the proof is complete by applyingLemma 1. ��Corollary 5 (Global convergence) Let {(X k,Bk, Ck)}k∈N bethe sequence generated by the PAM iteration (2.3)–(2.5) withan arbitrary initial guess (X 0,B0, C0) ∈ S × R

n1×r×n3 ×R

r×n2×n3 . Then, there exists (X , B, C) ∈ S × Rn1×r×n3 ×

Rr×n2×n3 such that

(i) (X k(:);Bk(:); Ck(:)) → (X (:); B(:); C(:));(ii) 0 ∈ ∂ F(X , B, C);(iii) the sequence {(X k,Bk, Ck)}k∈N has a finite length, i.e.,

+∞∑

k=0

√||X k − X ||2�2 + ||Bk − B||2�2 + ||Ck − C||2�2 < +∞.

In the above corollary, ∂ F(X ,B, C) should be understoodas the sub-differential of a multivariate function F whosevariables are entries of X ,B, and C.

4 Implementation

In this section, some efficient algorithms are introduced tosolve subproblems (2.3)–(2.5) arising from PAM method.

Firstly, it is easy to check that (2.3) has an unique closed-form solution as follows:

X k+1(i, j, k)={min

{1,max

{(Bk∗Ck+ρX k )(i, j,k)

1+ρ, 0}}

, (i, j, k) /∈ �,

M(i, j, k), (i, j, k) ∈ �,.

(4.1)

It is clear that (2.4) and (2.5) are both strictly convexoptimization problems with strongly convex objective func-tion, which possess global and unique minimizers. Thereare many efficient solvers for finding global minimizers of(2.4) and (2.5); see, e.g., the Bregman methods [18], prox-imal splitting methods [12], primal-dual methods [11,15],Douglas–Rachfordmethods [13], and alternating direction ofmultipliermethods (ADMM) [16,43,47].We chooseADMMas a solver for the minimization problems (2.4) and (2.5),since the convergence of ADMM for both (2.4) and (2.5) istheoretically guaranteed (see, e.g., [20]).

For introducing the ADMM iterations, we need to specifythe expressions of D1k’s and D2k’s. In general, the back-ward partial difference linear operatorsD1k’s andD2k’s withhomogeneous Neumann boundary conditions have the fol-lowing representations:

D1k(:, :, i) =

⎧⎪⎨

⎪⎩

q1⊗

j=1L

skjn1 j , i = 1,

On1, i > 1,,

q1∏

j=1

n1 j = n1, {n1 j }q1j=1 ⊂ N

+,

q1∑

j=1

sk j = 1,

{sk j }q1j=1 ⊂ N, k = 1, 2, . . . , m1. (4.2)

D2k(:, :, i) =

⎧⎪⎨

⎪⎩

q2⊗

j=1(L

tk jn2 j )

T, i = 1,

On2 , i > 1,,

123


q2∏

j=1

n2 j = n2, {n1 j }q2j=1 ⊂ N

+,

q2∑

j=1

tk j = 1,

{tk j }q2j=1 ⊂ N, k = 1, 2, . . . , m2, (4.3)

where

Lm =

⎡

⎢⎢⎢⎢⎢⎣

0−1 1

−1 1. . .

. . .

−1 1

⎤

⎥⎥⎥⎥⎥⎦

∈ Rm×m, L0

m := Im,

Om and Im denote m × m zero matrix and m × m iden-tity matrix, respectively, “⊗” denotes Kronecker product.Lm(1, 1) = 0 is due to the homogeneousNeumann boundarycondition.

(2.4) is equivalent to the following equality constrainedproblem

argminB,{Q1 j }m1

j=1

1

2||X k+1

− B ∗ Ck ||2�2 + ρ

2||B − Bk ||2�2 + μ

2||B||2�2 + α

m1∑

j=1

||Q1 j ||�1 ,

s.t ., Q1 j = D1 j ∗ B, j = 1, 2, . . . , m1. (4.4)

Note that the objective function in (4.4) is the summa-tion of m1 + 1 single variable functions with B, {Q1 j }m1

j=1as their individual variables. Thus, ADMM is applicable.Let {Y1 j }m1

j=1 be Lagrangian multipliers for (4.4). Then, theADMM iteration scheme for (4.4) can be described as fol-lows:

Q1 j ← argminQ1 j

α1||Q1 j ||�1 + β

2

∣∣∣∣

∣∣∣∣Q1 j − D1 j ∗ B + Y1 j

β

∣∣∣∣

∣∣∣∣

2

�2

,

j = 1, 2, . . . , m1, (4.5)

B ← argminB

1

2||X k+1 − B ∗ Ck ||2�2 + ρ

2||B − Bk ||2�2 + μ

2||B||2�2

+ β

2

m1∑

j=1

∣∣∣∣

∣∣∣∣Q1 j − D1 j ∗ B + Y1 j

β

∣∣∣∣

∣∣∣∣

2

�2

, (4.6)

Y1 j ← Y1 j + β(Q1 j − D1 j ∗ B), j = 1, 2, . . . , m1. (4.7)

Repeating the iteration process (4.5)–(4.7) sufficiently manytimes, the so-obtained B is taken as an iterative solution toBk+1. Note that among thesemany rounds of iterative repeat-ing of (4.5)–(4.7), the first round requires initial guesses ofBand Y1 j ’s, for which we set Bk as the initial guess of B andO (zero tensor) as the initial guess of Y1 j ’s.

By the well-known soft thresholding, (4.5) has a uniquesolution as follows:

Q1 j (l1, l2, l3) ← Tα1β

((

D1 j ∗ B − Y1 j

β

)

(l1, l2, l3)

)

,

(l1, l2, l3) ∈ �(n1) × �(r) × �(n3),

j = 1, 2, . . . , m1, (4.8)

where

Tμ(y) :={

(|y| − μ)sign(y), |y| > μ,

0, |y| ≤ μ,, y ∈ R.

In what follows, we introduce some notations and lemmasfor fast implementation and solution of (4.6). For a third-order tensorZ ∈ C

l×m×n , defineF (Z) ∈ Cl×m×n , B(Z) ∈

Cln×mn as

F (Z)(i, j, k) :=n∑

q=1

Z(i, j, q) exp(−2π i(q − 1)(k − 1)/n),

(i, j, k) ∈ �(l) × �(m) × �(n),

B(Z) := blockdiag(Z(:, :, 1),Z(:, :, 2), . . . ,Z(:, :, n)),

where i = √−1. Then, by the properties of discrete Fouriertransform, the inverse mapping F−1 of F is given by

F−1(Z)(i, j, k) := 1

n

n∑

q=1

Z(i, j, q) exp(2π i(q − 1)(k − 1)/n),

(i, j, k) ∈ �(l) × �(m) × �(n),

for any Z ∈ Cl×m×n . For a block diagonal matrix Z =

blockdiag(Z1, Z2, . . . , Zn) ∈ Cln×mn with Zi ∈ C

l×m (i =1, 2, . . . , n), define T(Z) ∈ C

l×m×n such that

T(Z)(i, j, k) := Zk(i, j), (i, j, k) ∈ �(l)×�(m)×�(n).

By the above definitions, it is clear that F (F−1(Z)) =F−1(F (Z)) = Z , Z = T(B(Z)), Z = B(T(Z)) for anythird-order tensorZ and any block diagonal matrix Z whoseblocks have the same size. Also, from the definition ofF andF−1, we know that the evaluations of F (Z) and F−1(Z)

can be fast implemented by fast Fourier transforms (FFTs)as follows:

F (Z)(i, j, :) = fft(Z(i, j, :)), F−1(Z)(i, j, :) = ifft(Z(i, j, :)),(4.9)

for any third-order tensor Z , where fft and ifft denote FFTand inverse FFT, respectively. Denote F := B ◦ F .

Lemma 6 ([24])

(i) For Z1 ∈ Cl×m×n and Z2 ∈ C

m×l×n, F (Z1 ∗ Z2) =F (Z1)F (Z2).

123


(ii) For Z ∈ Cl×m×n, ||Z||2�2 = 1

n ||F(Z)||2F .

It is clear that F , B are both linear bijections betweenunderlying spaces. Thus, F is also a linear bijection, and itis easy to see that F−1 = F−1 ◦ T. Hence, Lemma 6 (i)implies that (4.1) requires O(n1n2n3(r + log n3)) operationcost. Moreover, by Lemma 6, (4.6) is equivalent to

B ← argminB

1

2||X k+1 − BCk ||2F + ρ

2||B

− Bk ||2F + μ

2||B||2F + β

2

m1∑

j=1

∣∣∣∣∣

∣∣∣∣∣Q1 j − D1 j B + Y1 j

β

∣∣∣∣∣

∣∣∣∣∣

2

F

,

with B = F (B), Bk = F (Bk), X k+1 = F (X k+1), Ck =F (Ck), Q1 j = F (Q1 j ), D1 j = F (D1 j ), Y1 j = F (Y1 j ).Treating real part and imaginary part of B as real variablesof the objective function in the above problem, it is then easyto check that the unique solution of the above problem isactually the solution of the following matrix equation

B[Ck(Ck)∗ + (ρ + μ)Irn3 ] + β

⎛

⎝m1∑

j=1

D∗1 j D1 j

⎞

⎠ B = Rk,

(4.10)

where Rk = X k+1(Ck)∗ + ρ Bk +∑m1j=1 D∗

1 j (β Q1 j + Y1 j ).Notice that matrices appearing in (4.10) all have block diag-onal structures. Additionally, from the definition of D1 j in(4.2), we see that diagonal blocks of D1 j are identical, i.e.,

D1 j = In3 ⊗(⊗q1

m=1 Ls jmn1m

)for j = 1, 2, . . . , m1. There-

fore, (4.10) is equivalent to n3 many matrix equations ofsmaller size as follows:

Bl [Ckl (Ck

l )∗+(ρ+μ)Ir ]+β

⎛

⎝m1∑

j=1

q1⊗

m=1

Hs jmn1m

⎞

⎠ Bl = Rkl , l = 1, 2, . . . , n3,

(4.11)

where Bl , Ckl and Rk

l denote lth diagonal block of B, Ck andRk , respectively,

Hm := LTm Lm =

⎡

⎢⎢⎢⎢⎢⎣

1 −1−1 2 −1

. . .. . .

. . .

−1 2 −1−1 1

⎤

⎥⎥⎥⎥⎥⎦

∈ Rm×m, H0

m := Im .

Let Ckl (Ck

l )∗ + (ρ + μ)Ir = U kl V k

l (U kl )∗ denote the sin-

gular value decomposition (SVD). Notice that r is a smallnumber. Hence, the computation of the SVD, U k

l V kl (U k

l )∗,is economic, which requires O(r3) operation cost. Also, it is

straightforward to verify that Hm has the following orthogo-nal diagonalization form

Hm = Km�m KTm,

Km =√

2

m

[√(1 + δ j,1)−1 cos

(π(2i − 1)( j − 1)

2m

)]m

i, j=1,

δ j,1 ={1, j = 1,

0, otherwise,, Km KT

m = Im,

�m = 4 × diag

(

sin2(

(i − 1)π

2m

))m

i=1.

For a given vector v ∈ Rm , Kmv and KT

mv can be com-puted by two types of discrete cosine transforms inMATLABlanguage such that Kmv = dct(v,‘Type’, 3) and KT

mv =dct(v,‘Type’, 2), which requires O(m logm) operations.Denote �0

m := Im and

K1 =q1⊗

m=1

Kn1m , �1 =m1∑

j=1

q1⊗

m=1

�s jmn1m .

Then, it is clear that �1 is a n1 × n1 nonnegative diagonalmatrix and

∑m1j=1

⊗q1m=1 H

s jmn1m = K1�1 KT

1 .Multiplying KT1

from the left and multiplying U kl from the left on both sides

of (4.11), we see that (4.11) is equivalent to

Bl Vk

l + β�1 Bl = Rkl , l = 1, 2, . . . , n3, (4.12)

where Bl = KT1 BlU k

l , Rkl = KT

1 Rkl U k

l . Since V kl and �1 are

both diagonal matrices, Bl can be fast solved by

Bl(m, n) = Rkl (m, n)

β�1(m) + V kl (n)

, (m, n) ∈ �(n1) × �(r).

(4.13)

Hence, B in (4.6) can be solved by

B = F−1(blockdiag(K1 Bl(Ukl )∗)n3

l=1). (4.14)

To save the operation cost of each PAM iteration, the iter-ative process (4.5)–(4.7) is performed only one time. Then,it is easy to check that approximately solving B-subproblem(2.4) requires O(n1n3(rn2 + n2 log n3 + r log n1)) opera-tions.

Similarly to (4.5)–(4.7), one can perform one time ofADMM iteration with the same fashion of fast imple-mentation to approximately solve (2.5), which requiresO(n2n3(rn1 + n1 log n3 + r log n2)) operations.

To conclude, with the introduced fast implementationin this section, each PAM iteration (2.3)–(2.5) requires

123


21-4

22

23

0

PS

NR

-5

24

-1

25

26

-2-6-3

-7 -4

22

22.5

23

23.5

24

24.5

25

25.5

(α, β) = (0.5, 1.3)

21-4

22

23

0

24

PS

NR

-5

25

-1

26

27

-2-6-3

-7 -4

22

22.5

23

23.5

24

24.5

25

25.5

26

(α, β) = (0.5, 1.5)

21-4

22

23

0

PS

NR

-5

24

-1

25

26

-2-6-3

-7 -4

22

22.5

23

23.5

24

24.5

25

25.5

(α, β) = (0.5, 1.7)

21-4

22

23

0

PS

NR

-5

24

-1

25

26

-2-6-3

-7 -4

22

22.5

23

23.5

24

24.5

25

25.5

(α, β) = (0.5, 1.9)

21-4

22

23

0

PS

NR

-5

24

-1

25

26

-2-6-3

-7 -4

22

22.5

23

23.5

24

24.5

25

25.5

(α, β) = (5, 1.3)

21-4

22

23

0

24

PS

NR

-5

25

-1

26

27

-2-6-3

-7 -4

22

22.5

23

23.5

24

24.5

25

25.5

26

(α, β) = (5, 1.5)

21-4

22

23

0

PS

NR

-5

24

-1

25

26

-2-6-3

-7 -4

22

22.5

23

23.5

24

24.5

25

25.5

(α, β) = (5, 1.7)

21-4

22

23

0

PS

NR

-5

24

-1

25

26

-2-6-3

-7 -4

22

22.5

23

23.5

24

24.5

25

25.5

(α, β) = (5, 1.9)

21-4

22

23

0

PS

NR

-5

24

-1

25

26

-2-6-3

-7 -4

22

22.5

23

23.5

24

24.5

25

25.5

(α, β) = (50, 1.3)

21-4

22

23

0

24

PS

NR

-5

25

-1

26

27

-2-6-3

-7 -4

22

22.5

23

23.5

24

24.5

25

25.5

26

(α, β) = (50, 1.5)

21-4

22

23

0

PS

NR

-5

24

-1

25

26

-2-6-3

-7 -4

22

22.5

23

23.5

24

24.5

25

25.5

(α, β) = (50, 1.7)

21-4

22

23

0

PS

NR

-5

24

-1

25

26

-2-6-3

-7 -4

22

22.5

23

23.5

24

24.5

25

25.5

(α, β) = (50, 1.9)

Fig. 2 The PSNR values of the recovered results by TCTF-TVT on the color image “Panda” with the sampling rate 0.1 and different (α, β, μ, ρ)

(Color figure online)

O(n3(rn1n2 + n1n2 log n3 + rn2 log n2 + rn1 log n1)) oper-ations.

5 Numerical Experiment

In this section, we test the proposed TCTF-TVT model onseveral examples of image and video completion, compar-ing it with the TCTF model proposed in [51], TNN modelproposed in [46], the LRTC-TV-I and LRTC-TV-II modelsproposed in [29]. All numerical experiments presented in thissection are performed via MATLAB R2017a on a PC withthe configuration: Intel(R)Core(TM) i5-6500CPU3.20GHzand 32 GB RAM.

Let p ∈ (0, 1) be a given number. The set of indices ofknown entries,�, is chosen randomly inMATLAB languageas follows:

� := find(rand(n1n2n3, 1) < p). (5.1)

p is also called sampling rate.Let Xre f represent the ground truth of an nb-band video

with n f frames (e.g., the case of n f = 1 and nb = 3 rep-

resents a color image). Let Xi j (1 ≤ i ≤ nb, 1 ≤ j ≤ n f )denote the i th band of j th frame of Xre f . Let X represent arecovered video from some incomplete observation of Xre f

whose i th band of j th frame is denoted by Xi j . Then, twometrics are defined as follows to measure the quality of therecovered result X :

PSNR(X ,Xre f ) := 1

nbn f

nb∑

i=1

n f∑

j=1

PSNR(Xi j , Xi j , ||Xi j (:)||∞),

SSIM(X ,Xre f ) := 1

nbn f

nb∑

i=1

n f∑

j=1

SSIM(Xi j , Xi j ),

where PSNR and SSIMwith images as inputs are two built-infunctions in MATLAB.

Denote by “Time”, the computational time in unit of sec-ond. The maximum iteration number for TCTF-TVT is set tobe 500 if not specified. The initial guess X 0 for TCTF-TVTis set as

X 0(i, j, k) ={M(i, j, k), (i, j, k) ∈ �,

random value, (i, j, k) /∈ �,, (5.2)

123


0 100 200 300 400 500 600 700 800

Iteration number

10

12

14

16

18

20

22

24

26

28P

SN

R

TCTF-TVTTCTF-BTCTF-C

PSNR of first 800 iterations

0 100 200 300 400 500 600 700 800

Iteration number

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

SS

IM

TCTF-TVTTCTF-BTCTF-C

SSIM of first 800 iterations

Fig. 3 The PSNR and SSIM values of the recovered results by TCTF-TVT, TCTF-B, and TCTF-C on the color image “Panda” with the samplingrate 0.1 (Color figure online)

Original Observed (p = 0.1) TCTF-B TCTF-C TCTF-TVT

Fig. 4 The original color image “Panda”, the observed image, and the recovered images by TCTF-B, TCTF-C, and TCTF-TVT (Color figureonline)

Lena Airplane Sailboat Baboon Barbara

Fig. 5 The original color images (Color figure online)

Lena Airplane Sailboat Baboon Barbara

Fig. 6 The observed color images with the sampling rate 0.15 (Color figure online)

123


TCTF TNN LRTC-TV-I LRTC-TV-II TCTF-TVT





Fig. 7 The recovered images by TCTF, TNN, LRTC-TV-I, LRTC-TV-II, and TCTF-TVT from observed data with sampling rate p = 0.15 (Colorfigure online)

123


Table 1 The PSNR and SSIMvalues of recovered images bydifferent methods on colorimages “Lena,” “Airplane,” and“Sailboat” with differentsampling rates

Color image Lena Airplane Sailboat

p Model PSNR SSIM Time PSNR SSIM Time PSNR SSIM Time

0.05 TCTF 11.32 0.051 2.75 10.30 0.049 2.00 10.77 0.060 2.22

TNN 16.36 0.225 5.51 17.01 0.301 5.27 15.32 0.230 5.33

LRTC-TV-I 18.41 0.540 24.07 18.55 0.567 22.73 16.91 0.448 24.26

LRTC-TV-II 22.52 0.705 83.87 21.50 0.711 83.98 20.12 0.617 85.87

TCTF-TVT 22.71 0.647 25.65 21.91 0.647 25.23 20.42 0.576 25.95

0.1 TCTF 14.09 0.133 3.16 12.77 0.118 3.20 11.65 0.103 3.15

TNN 18.86 0.348 4.92 19.05 0.413 4.76 17.40 0.333 4.85

LRTC-TV-I 21.87 0.679 23.50 21.17 0.679 22.76 19.61 0.591 23.74

LRTC-TV-II 25.02 0.796 83.75 23.40 0.785 85.53 21.47 0.691 82.72

TCTF-TVT 24.67 0.743 26.00 23.77 0.751 26.69 22.09 0.674 27.93

0.15 TCTF 17.69 0.306 3.19 17.06 0.351 3.08 15.97 0.315 3.19

TNN 20.81 0.461 4.75 20.78 0.510 4.63 18.99 0.426 5.20

LRTC-TV-I 23.73 0.758 23.41 22.92 0.757 22.60 21.30 0.682 24.35

LRTC-TV-II 26.00 0.833 55.42 23.80 0.808 85.49 21.91 0.717 84.25

TCTF-TVT 25.81 0.796 30.31 24.97 0.798 28.58 23.21 0.731 27.65

The best PSNR and SSIM values are highlighted in bold

Table 2 The PSNR and SSIMvalues of the recovered imagesby different methods on colorimages “Baboon” and “Barbara”with different sampling rates

Color image Baboon Barbara

p Model PSNR SSIM Time PSNR SSIM Time

0.05 TCTF 11.54 0.066 3.07 11.53 0.062 2.90

TNN 15.46 0.171 5.48 15.83 0.228 5.45

LRTC-TV-I 17.29 0.277 24.03 17.20 0.414 24.12

LRTC-TV-II 19.98 0.390 84.59 22.10 0.645 86.37

TCTF-TVT 19.70 0.367 25.52 22.23 0.620 24.05

0.1 TCTF 12.67 0.119 3.27 13.61 0.143 3.15

TNN 17.19 0.262 4.90 18.23 0.359 5.04

LRTC-TV-I 19.61 0.393 23.65 20.90 0.600 23.33

LRTC-TV-II 20.79 0.458 85.32 23.41 0.712 85.35

TCTF-TVT 20.68 0.464 27.57 24.17 0.716 26.69

0.15 TCTF 15.20 0.242 3.15 17.68 0.378 3.22

TNN 18.52 0.347 4.74 20.28 0.478 4.96

LRTC-TV-I 20.79 0.481 23.43 23.26 0.707 23.39

LRTC-TV-II 21.34 0.516 89.51 24.51 0.762 93.14

TCTF-TVT 21.36 0.531 24.01 25.35 0.769 24.73


The initial guesses B0 and C0 for TCTF-TVT are set as

B0 = F−1(B0), C0 = F−1(C0), B0(:, :, k) :=r∑

i=1

σki uki eTi ,

C0(:, :, k) :=r∑

i=1

eivTki , k = 1, 2, . . . , n3,

where σki is the i th largest singular value ofF (X 0)(:, :, k);uki and vki are the left and the right singular vectors corre-sponding to σk,i ; ei denotes the i th column of r × r identity

matrix. The stopping criterion for TCTF-TVT is set as fol-lows:

max

{ ||X k+1 − X k ||2�2||X k+1||2�2

,||Bk+1 − Bk ||2�2

||Bk+1||2�2,||Ck+1 − Ck ||2�2

||Ck+1||2�2

}

< 10−6.

Parameters of other models are specified by following thepapers in which there approaches were proposed and devel-oped.

123


Fig. 8 The 50th frame of the original color video “Salesman” (Colorfigure online)

Parameter Setting of TCTF-TVT: Throughout this sec-tion, we set α = 5, β = 1.5, μ = 0.04, ρ =5e-6, ifnot specified. We mainly focus on the tensor completionfrom poorly observed data. Hence, in the numerical test,the � given in (5.1) is equipped with small p such thatp ∈ {0.05, 0.1, 0.15}.

For a color image with size nv × nh × 3, we set n1 = nv ,n2 = nh , n3 = 3, m1 = m2 = q1 = q2 = 1. To demon-strate the optimality of the parameter setting (α, β, μ, ρ) =(5, 1.5, 0.04, 5e−6), we compare the PSNR of the recovered

results byTCTF-TVTwith different values of (α, β, μ, ρ) oncompletion of a color image. The tested color image “Panda”is of size 256 × 256 × 3, the ground truth of which is listedin “Original” of Fig. 4. The results of the test are plotted inFig. 2. Since changes of SSIM are quite similar to those ofPSNR, we do not present values of SSIM in Fig. 2. Figure 2shows that (i) the performance of TCTF-TVT is not sensi-tive to changes of α and ρ; (ii) performance of TCTF-TVTwith β = 1.5 is better than that with β ∈ {1.3, 1.7, 1.9};(iii) when μ ∈ [0.03, 0.05], TCTF-TVT achieves its bestperformance. For these reasons, we set the parameters as(α, β, μ, ρ) = (5, 1.5, 0.04, 5e − 6).

Let TCTF-B and TCTF-C denote model (2.1) but withoutα∑m2

k=1 ||C ∗D2k ||�1 + μ2 ||C||2�2 and α

∑m1k=1 ||D1k ∗B||�1 +

μ2 ||B||2�2 , respectively. The same PAM iteration is employedto solve TCTF-B and TCTF-C, the parameters of which areset to be the same as the one used for solving TCTF-TVT.To test the effectiveness of the regularization terms on Band C, we apply TCTF-TVT, TCTF-B, and TCTF-C on thecolor image “Panda”, the results of which are presented inFigs. 3 and 4. Figure 3 shows that the recovered result byTCTF-TVT has a better quality than those by TCTF-B andTCTF-C. From Fig. 4, we can observe that the recoveredresult by TCTF-B (TCTF-C) has a staircase effect along thehorizontal (vertical) direction, while such staircase effects

observed(p = 0.05) TCTF TNN LRTC-TV-I TCTF-TVT



Fig. 9 The 50th frame of the observed videos and the recovered videos by TCTF, TNN, LRTC-TV-I, and TCTF-TVT with different sampling rates(Color figure online)

123


Table 3 The PSNR and SSIMvalues of the recovered videosby different methods on thecolor video “Salesman” withdifferent sampling rates

Model p = 0.05 p = 0.1 p = 0.15

PSNR SSIM Time PSNR SSIM Time PSNR SSIM Time

TCTF 13.54 0.153 135.30 18.00 0.368 181.75 27.18 0.784 180.70

TNN 26.83 0.842 39.32 31.44 0.935 39.22 33.75 0.954 35.86

LRTC-TV-I 21.65 0.602 918.34 24.75 0.725 900.09 26.62 0.798 897.47

LRTC-TV-II − − > 72 h − − > 72 h − − > 72 h

TCTF-TVT 31.09 0.926 655.31 34.10 0.958 658.99 35.64 0.968 676.23


0 20 40 60 80 100Frame number

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Pix

el v

alue

Salesman

OriginalTCTF-TVTLRTC-TVTNNTCTF

Fig. 10 The selected fibers along the temporal mode of the originalvideo “Salesman” and the recovered videos by TCTF, TNN, LRTC-TV-I, and TCTF-TVT with the sampling rate 0.05 (Color figure online)

are significantly suppressed in the result by TCTF-TVT.These evidences illustrate that the regularization terms onboth B and C contribute to the performance of TCTF-TVT,i.e., the regularizations on the two factor tensors are botheffective.

Example 1 In this example, we evaluate the performance ofdifferent methods on color images. For a color image withsize nv × nh × 3, we set n1 = nv , n2 = nh , n3 = 3, m1 =m2 = q1 = q2 = 1 and r = 40. We remark that not onlyin this paper and but also in the literature (see, e.g., [29,51]),color image is always considered as a third-order tensor withthe first two dimensions representing spatial dimensions andthe third dimension representing band. All the color imagestested in Example 1 are cited from [29]. The original colorimages and the observed color images of size 256× 256× 3are shown in Figs. 5 and 6. The recovered images by TCTF,TNN, LRTC-TV-I, LRTC-TV-II and TCTF-TVT are dis-played in Fig. 7. The PSNRand SSIMvalues of the recoveredimages are summarized in Tables 1 and 2. Figure 7 andTables 1 and 2 show (i) TCTF-TVT and LRTC-TV-II have

Fig. 11 The original MSI “Pompoms” (Color figure online)

the best performance in terms of visual quality among the fivemodels; (ii) the recovered results by TCTF-TVT and LRTC-TV-II are comparable; (iii) TCTF-TVT is more efficient thanLRTC-TV-II in terms of computational time.

Example 2 We evaluate the performance of different meth-ods on the color video. We first form a third-order tensor byvectorizing each frame of the color video (the fourth-ordertensor). Since the factor tensors of the formed third-ordertensor have clear physical interpretations (the combined twospatial dimensions, the time dimension and the color dimen-sion), Example 2 serves as a good example to justify ourmotivation in the introduction. For a color video of t frameswith each frame of size nv × nh × 3, we set n1 = nvnh ,n2 = t , n3 = 3, r = 30, m1 = q1 = 2, n11 = nh , n12 = nv ,s11 = 0, s12 = 1, s21 = 1, s22 = 0, m2 = q2 = 1. The testedcolor video “Salesman”,1 shown in Fig. 8, has 100 frameswith each frame of size 144 × 176 × 3. Since LRTC-TV-II is too time-consuming for color video completion, thereis no result of LRTC-TV-II presented in Example 2. The50th frame of the recovered videos is displayed in Fig. 9.The PSNR and SSIM values of the recovered videos arereported in Table 3. We can see that the recovered results

1 http://trace.eas.asu.edu/yuv/.

123


Fig. 12 The observed“Pompoms” with differentsampling rates (Color figureonline)

p = 0.05 p = 0.1 p = 0.15




Fig. 13 First row: the recovered results from observed data with p = 0.05. Second row: the recovered results from observed data with p = 0.1.Third row: the recovered results from observed data with p = 0.15 (Color figure online)

Table 4 The PSNR and SSIMvalues of the recovered MSIs bydifferent methods on the MSI“Pompoms” with differentsampling rates

Model p = 0.05 p = 0.1 p = 0.15

PSNR SSIM Time PSNR SSIM Time PSNR SSIM Time

TCTF 18.98 0.316 35.72 24.09 0.534 40.26 26.26 0.636 40.41

TNN 23.30 0.559 63.26 28.14 0.755 63.63 31.34 0.847 62.26

LRTC-TV-I 19.12 0.614 329.68 26.28 0.819 322.35 29.92 0.888 319.80

LRTC-TV-II 21.23 0.656 431.63 24.96 0.788 434.23 27.60 0.855 431.58

TCTF-TVT 27.01 0.783 153.84 32.42 0.888 152.48 33.06 0.894 153.23


123


by TCTF-TVT are superior over those by TCTF, TNN, andLRTC-TV-I both qualitatively and quantitatively. Moreover,we display one selected fiber along the temporal mode ofthe original video and the recovered videos in Fig. 10. Wecan observe that the selected fibers of the recovered resultsby TCTF, TNN, and LRTC-TV-I exhibit strong oscillationphenomenon, while the selected fiber of the recovered resultby TCTF-TVT is close to the true fiber.

Example 3 We evaluate the performance of different meth-ods onmultispectral images (MSIs). For theMSI of nb bandswith each band of size nv × nh , we set n1 = nv , n2 = nh ,n3 = nb, r = 30, m1 = m2 = q1 = q2 = 1. Actually, notonly in this paper, but also in the literature (see, e.g., [40]),MSI data are considered as a third-order tensor with the firsttwo dimensions representing spatial dimensions and the thirddimension representing band. The tested MSI “Pompoms”2

is cited from [40], which has 31 bands with each band ofsize 256 × 256. A composition of 1st, 2nd and 31th bandsof the ground truth, observed data and the recovered MSIs isdisplayed in Figs. 11, 12 and 13, respectively. The PSNR andSSIM values of the recovered MSIs are reported in Table 4.We can conclude from the figure and table that the recov-ered results by TCTF-TVT are superior over those by TCTF,TNN, LRTC-TV-I, and LRTC-TV-II both qualitatively andquantitatively.

6 Concluding Remark

A new tensor completion model based on the tensor factor-ization with TV–Tikhonov regularization has been proposed.The global convergence of the PAMalgorithm for solving theproposedmodel has been established.We have also proposeda fast implementation for the PAM algorithm. Numericalresults reported have shown the better performance of theproposed method compared with the state-of-the-art meth-ods.

References

1. Attouch,H., Bolte, J., Redont, P., Soubeyran,A.: Proximal alternat-ing minimization and projection methods for nonconvex problems:an approach based on the Kurdyka–Łojasiewicz inequality. Math.Oper. Res. 35(2), 438–457 (2010)

2. Attouch, H., Bolte, J., Svaiter, B.-F.: Convergence of descent meth-ods for semi-algebraic and tame problems: proximal algorithms,forward–backward splitting, and regularized Gauss–Seidel meth-ods. Math. Program. 137(1–2), 91–129 (2013)

3. Auslender, A.: Méthodes numériques pour la décomposition et laminimisation de fonctions non différentiables. Numer.Math. 18(3),213–223 (1971)

2 http://www.cs.columbia.edu/CAVE/databases/multispectral/.

4. Auslender, A.: Asymptotic properties of the Fenchel dual func-tional and applications to decomposition problems. J. Optim.Theory Appl. 73(3), 427–449 (1992)

5. Bertsekas, D.P., Tsitsiklis, J.-N.: Parallel andDistributedComputa-tion: Numerical Methods. Prentice Hall, Englewood Cliffs (1989)

6. Bochnak, J., Coste, M., Roy, M.-F.: Real Algebraic Geometry.Springer, Berlin (2013)

7. Bolte, J., Daniilidis, A., Lewis, A.: The Łojasiewicz inequality fornonsmooth subanalytic functions with applications to subgradientdynamical systems. SIAM J. Optim. 17(4), 1205–1223 (2007a)

8. Bolte, J., Daniilidis, A., Lewis, A., Shiota, M.: Clarke subgradientsof stratifiable functions. SIAM J. Optim. 18(2), 556–572 (2007b)

9. Cai, X.: Variational image segmentation model coupled withimage restoration achievements. Pattern Recognit. 48(6), 2029–2042 (2015)

10. Carroll, J.-D., Pruzansky, S., Kruskal, J.-B.: CANDELINC: a gen-eral approach to multidimensional analysis of many-way arrayswith linear constraints on parameters. Psychometrika 45(1), 3–24(1980)

11. Chambolle, A., Pock, T.: A first-order primal-dual algorithm forconvex problems with applications to imaging. J. Math. ImagingVis. 40(1), 120–145 (2011)

12. Combettes, P.-L., Pesquet, J.-C.: Proximal splitting methods insignal processing. In: Bauschke, H., Burachik, R., Combettes, P.,Elser, V., Luke, D., Wolkowicz, H. (eds.) Fixed-Point Algorithmsfor Inverse Problems in Science and Engineering, pp. 185–212.Springer, New York (2011)

13. Combettes, P.-L., Wajs, V.-R.: Signal recovery by proximalforward–backward splitting.MultiscaleModel. Simul. 4(4), 1168–1200 (2005)

14. Coste, M.: An Introduction to O-Minimal Geometry. Istituti edito-riali e poligrafici internazionali, Pisa (2000)

15. Esser, E., Zhang, X., Chan, T.-F.: A general framework for a classof first order primal-dual algorithms for convex optimization inimaging science. SIAM J. Imaging Sci. 3(4), 1015–1046 (2010)

16. Gabay, D., Mercier, B.: A Dual Algorithm for the Solution of NonLinear Variational Problems Via Finite Element Approximation.Institut de recherche d’informatique et d’automatique, Rocquen-court (1975)

17. Gandy, S., Recht, B., Yamada, I.: Tensor completion and low-n-rank tensor recovery via convex optimization. Inverse Probl. 27(2),025010 (2011)

18. Goldstein, T., Osher, S.: The split Bregman method for L1-regularized problems. SIAM J. Imaging Sci. 2(2), 323–343 (2009)

19. Hansen, P.-C., Nagy, J .G., O’leary, D.-P.: Deblurring Images:Matrices, Spectra, and Filtering. SIAM, Philadelphia (2006)

20. Hong, M., Luo, Z.-Q.: On the linear convergence of the alternatingdirection method of multipliers. Math. Program. 162(1–2), 165–199 (2017)

21. Jain, P., Oh, S.: Provable tensor factorization with missing data. In:Proceedings of the International Conference onNeural InformationProcessing Systems (2014)

22. Ji, T.-Y., Huang, T.-Z., Zhao, X.-L., Ma, T.-H., Liu, G.: Tensorcompletion using total variation and low-rank matrix factorization.Inf. Sci. 326, 243–257 (2016)

23. Ji, T.-Y., Huang, T.-Z., Zhao, X.-L., Ma, T.-H., Deng, L.-J.: A non-convex tensor rank approximation for tensor completion. Appl.Math. Model. 48, 410–422 (2017)

24. Kilmer, M., Martin, C.: Factorization strategies for third-order ten-sors. Linear Algebra Appl. 435(3), 641–658 (2011)

25. Kilmer, M.-E., Braman, K., Hao, N., Hoover, R.-C.: Third-ordertensors as operators on matrices: a theoretical and computationalframework with applications in imaging. SIAM J. Matrix Anal.Appl. 34(1), 148–172 (2013)

26. Kolda, T.-G., Bader, B.-W.: Tensor decompositions and applica-tions. SIAM Rev. 51, 455–500 (2009)

123


27. Kressner, D., Steinlechner,M., Vandereycken, B.: Low-rank tensorcompletion by Riemannian optimization. BITNumer. Math. 54(2),447–468 (2014)

28. Kurdyka, K.: On gradients of functions definable in o-minimalstructures. Annales de l’institut Fourier 48, 769–783 (1998)

29. Li, X., Ye, Y., Xu, X.: Low-rank tensor completion with total vari-ation for visual data inpainting. In: Proceedings of Thirty-FirstAAAI Conference on Artificial Intelligence (2017)

30. Li, X.-T., Zhao, X.-L., Jiang, T.X., Zheng, Y.B., Ji, T.Y., Huang,T.Z.: Low-rank tensor completion via combined non-local self-similarity and low-rank regularization. Neurocomputing 367, 1–12(2019)

31. Liu, J., Musialski, P., Wonka, P., Ye, J.: Tensor completion forestimating missing values in visual data. IEEE Trans. Pattern Anal.Mach. Intell. 35(1), 208–220 (2013)

32. Lu, C., Feng, J., Chen, Y., Liu, W., Lin, Z., Yan, S.: Tensor robustprincipal component analysis: exact recovery of corrupted low-rank tensors via convex optimization. In: Proceedings of the IEEEConference on Computer Vision and Pattern Recognition (2016)

33. Ortega, J.-M., Rheinboldt, W.-C.: Iterative Solution of NonlinearEquations in Several Variables. SIAM, Philadelphia (1970)

34. Rudin, L.-I., Osher, S., Fatemi, E.: Nonlinear total variation basednoise removal algorithms. Physica D 60(1–4), 259–268 (1992)

35. Semerci, O., Hao, N., Kilmer, M.-E., Miller, E.-L.: Tensor-basedformulation and nuclear norm regularization for multienergy com-puted tomography. IEEE Trans. Image Process. 23(4), 1678–1693(2014)

36. Tan, H., Cheng, B., Wang, W., Zhang, Y.-J., Ran, B.: Tensorcompletion via amulti-linear low-n-rank factorizationmodel. Neu-rocomputing 133, 161–169 (2014)

37. Tao, D., Li, X., Wu, X., Maybank, S.-J.: General tensor discrimi-nant analysis and gabor features for gait recognition. IEEE Trans.Pattern Anal. Mach. Intell. 29(10), 1700–1715 (2007)

38. Tseng, P.: Convergence of a block coordinate descent method fornondifferentiable minimization. J. Optim. Theory Appl. 109(3),475–494 (2001)

39. Tucker, L.: Some mathematical notes on three-mode factor analy-sis. Psychometrika 31(3), 279–311 (1966)

40. Xie, Q., Zhao, Q., Meng, D., Xu, Z.: Kronecker-basis-representation based tensor sparsity and its applications to tensorrecovery. IEEETrans. PatternAnal.Mach. Intell.40(8), 1888–1902(2018)

41. Xu, Y., Yin,W.: A block coordinate descent method for regularizedmulticonvex optimization with applications to nonnegative tensorfactorization and completion. SIAM J. Imaging Sci. 6(3), 1758–1789 (2013)

42. Xu, W.-H., Zhao, X.-L., Ji, T.-Y., Miao, J.-Q., Ma, T.-H., Wang,S., Huang, T.-Z.: Laplace function based nonconvex surrogate forlow-rank tensor completion. Signal Process. Image Commun. 73,62–69 (2019)

43. Yang, J.-H., Zhao, X.-L., Ma, T.-H., Chen, Y., Huang, T.-Z., Ding,M.: Remote sensing images destriping using unidirectional hybridtotal variation and nonconvex low-rank regularization. J. Comput.Appl. Math. 363, 124–144 (2020)

44. Zhang, Z.: A novel algebraic framework for processing multidi-mensional data: theory and application. Ph.D. dissertation, TuftsUniversity (2017)

45. Zhang, Z., Aeron, S.: Exact tensor completion using t-SVD. IEEETrans. Signal Process. 65(6), 1511–1526 (2017)

46. Zhang, Z., Ely, G., Aeron, S., Hao, N., Kilmer, M.: Novel methodsfor multilinear data completion and de-noising based on tensor-SVD. In: Proceedings of the IEEEConference on Computer Visionand Pattern Recognition (2014)

47. Zhao, Xi-Le, Wang, Fan, Michael, K.Ng: A new convex optimiza-tion model for multiplicative noise and blur removal. SIAM J.Imaging Sci. 7(1), 456–475 (2014)

48. Zheng, Y.-B., Huang, T.-Z., Zhao, X.-L., Jiang, T.-X.,Ma, T.-H., Ji,T.-Y.:Mixed noise removal in hyperspectral image via low-fibered-rank regularization. IEEE Trans. Geosci. Remote Sens. (2019).https://doi.org/10.1109/TGRS.2019.2940534

49. Zhou, P., Feng, J.: Outlier-robust tensor PCA. In: Proceedings ofthe IEEE Conference on Computer Vision and Pattern Recognition(2017)

50. Zhou, M., Liu, Y., Long, Z., Chen, L., Zhu, C.: Tensor rank learn-ing in CP decomposition via convolutional neural network. SignalProcess. Image Commun. 73, 12–21 (2018a)

51. Zhou, P., Lu, C., Lin, Z., Zhang, C.: Tensor factorization for low-rank tensor completion. IEEE Trans. Image Process. 27(3), 1152–1163 (2018b)

Publisher’s Note Springer Nature remains neutral with regard to juris-dictional claims in published maps and institutional affiliations.

Xue-Lei Lin received the BS degreein Information and ComputingScience from School of Mathe-matics and Statistics, Ningxia Uni-versity, Ningxia, China. He recei-ved the MS degree in Mathemat-ics from the Department of Math-ematics, University of Macau,Macau, China. He is currently pur-suing the PhD degree in the Depart-ment of Mathematics, Faculty ofScience, Hong Kong Baptist Uni-versity, Hong Kong, China. Hisresearch interests include scien-tific computation and applications,

numerical algorithms for image processing, numerical linear alge-bra, preconditioning technologies. His website is http://scholar.google.com/citations?user=BqyuMZ8AAAAJ&hl=en.

Michael K. Ng is the Directorof Research Division for Math-ematical and Statistical Science,and Chair Professor of Depart-ment of Mathematics, the Univer-sity of Hong Kong. His researchareas are data science, scientificcomputing, and numerical linearalgebra. His website ishttps://hkumath.hku.hk/~mng/.

123


Xi-Le Zhao received the MS andPhD degrees from the Universityof Electronic Science and Tech-nology of China (UESTC),Chengdu, China, in 2009 and 2012.He is currently a professor withthe School of Mathematical Sci-ences, UESTC. His research inter-ests include image processing,computer vision, and machine learn-ing. His website is https://sites.google.com/site/xilezhao2018/home.

123

tensorfactorizationwithtotalvariationandtikhonovregulariza ... · exhibit spatial/temporal...

Documents